External parameter calibration method and device of infrared camera, vehicle and storage medium

Abstract

The embodiment of the invention discloses an external parameter calibration method and device of an infrared camera, a vehicle and a storage medium. The method comprises the steps that when a vehicle is located between two lane lines, a plurality of driver pupil images are obtained through an in-vehicle camera arranged on the vehicle, and road lane line images are obtained through an infrared camera arranged on the vehicle; determining a target fixation point coordinate of the driver in an image coordinate system corresponding to an in-vehicle camera based on the plurality of driver pupil images; determining intersection point coordinates of two lane lines in an image coordinate system corresponding to an infrared camera based on the road lane line image; determining a target vanishing point coordinate based on the target fixation point coordinate and the intersection point coordinate; and determining an external parameter angle of the infrared camera based on the target vanishing point coordinate. According to the method, the calibration precision of the external reference angle of the infrared camera is improved.

Description

Technical Field

[0001] The present application belongs to the field of vehicle technology, and specifically relates to an external parameter calibration method, device, vehicle and storage medium for an infrared camera. Background Art

[0002] Since the working principle of infrared cameras is different from that of ordinary cameras, and it uses temperature difference to form images, most of the current calibration of infrared cameras uses the calibration plate heating source method, which requires specific equipment and is less convenient. Therefore, it is proposed to use the vanishing point principle of lane lines to calibrate the extrinsic angle of infrared cameras, but this method will result in the calibration accuracy of the extrinsic angle of infrared cameras still needs to be improved. Summary of the invention

[0003] In view of the above problems, the present application proposes an extrinsic parameter calibration method, device, vehicle and storage medium for an infrared camera to improve the above problems.

[0004] In a first aspect, an embodiment of the present application provides an extrinsic parameter calibration method for an infrared camera, the method comprising: when a vehicle is between two lane lines, acquiring multiple driver pupil images by an in-vehicle camera arranged on the vehicle, and acquiring a road lane line image by an infrared camera arranged on the vehicle, wherein the driver pupil image is an image of the driver's pupil captured by the in-vehicle camera when the driver visually looks at the intersection of two lane lines at infinity in front of the vehicle; based on the multiple driver pupil images, determining the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera; based on the road lane line image, determining the intersection coordinates of the intersection of the two lane lines in the image coordinate system corresponding to the infrared camera; based on the target gaze point coordinates and the intersection coordinates, determining the target vanishing point coordinates; based on the target vanishing point coordinates, determining the extrinsic parameter angle of the infrared camera.

[0005] In a second aspect, an embodiment of the present application provides an extrinsic parameter calibration device for an infrared camera, the device comprising: an image acquisition unit, for acquiring a plurality of driver pupil images through an in-vehicle camera disposed on the vehicle when the vehicle is between two lane lines, and acquiring a road lane line image through an infrared camera disposed on the vehicle, wherein the driver pupil image is an image of the driver's pupil captured by the in-vehicle camera when the driver visually looks at the infinitely far intersection of two lane lines in front of the vehicle; a first determination unit, for determining the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera based on the plurality of driver pupil images; a second determination unit, for determining the intersection coordinates of the two lane lines in the image coordinate system corresponding to the infrared camera based on the road lane line image; a third determination unit, for determining the target vanishing point coordinates based on the target gaze point coordinates and the intersection coordinates; an extrinsic parameter determination unit, for determining the extrinsic parameter angle of the infrared camera based on the target vanishing point coordinates.

[0006] In a third aspect, an embodiment of the present application provides a vehicle, comprising an in-vehicle camera, an infrared camera, one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, wherein the above method is executed when the program code is run.

[0008] The embodiment of the present application provides an infrared camera external parameter calibration method, device, vehicle and storage medium. When the vehicle is in the middle of two lane lines, a plurality of driver pupil images are obtained by an in-vehicle camera arranged on the vehicle, and a road lane line image is obtained by an infrared camera arranged on the vehicle, wherein the driver pupil image is an image of the driver's pupil captured by the in-vehicle camera when the driver visually looks at the infinite intersection of the two lane lines in front of the vehicle, and then based on the plurality of driver pupil images, the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera are determined, based on the road lane line image, the intersection coordinates of the two lane lines in the image coordinate system corresponding to the infrared camera are determined, and then based on the target gaze point coordinates and the intersection coordinates, the target vanishing point coordinates are determined, and finally based on the target vanishing point coordinates, the external parameter angle of the infrared camera is determined. Through the above method, the external parameter angle of the infrared camera is determined by combining the driver's pupil image and the road lane line image, thereby improving the calibration accuracy of the external parameter angle of the infrared camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flow chart of an extrinsic parameter calibration method for an infrared camera proposed in one embodiment of the present application is shown;

[0011] Figure 2 A flow chart of an extrinsic parameter calibration method for an infrared camera proposed in another embodiment of the present application is shown;

[0012] Figure 3 A structural block diagram of an external parameter calibration device for an infrared camera proposed in an embodiment of the present application is shown;

[0013] Figure 4 A structural block diagram of a vehicle for executing an external parameter calibration method for an infrared camera according to an embodiment of the present application is shown;

[0014] Figure 5 A storage unit in an embodiment of the present application is shown for storing or carrying a program code for implementing an external parameter calibration method for an infrared camera according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The embodiment of the present application proposes an infrared camera external parameter calibration method, device, vehicle and storage medium. When the vehicle is in the middle of two lane lines, multiple driver pupil images are obtained by an in-vehicle camera arranged on the vehicle, and a road lane line image is obtained by an infrared camera arranged on the vehicle, wherein the driver pupil image is an image of the driver's pupil captured by the in-vehicle camera when the driver visually looks at the infinite intersection of the two lane lines in front of the vehicle, and then based on the multiple driver pupil images, the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera are determined, based on the road lane line image, the intersection coordinates of the two lane lines in the image coordinate system corresponding to the infrared camera are determined, and then based on the target gaze point coordinates and the intersection coordinates, the target vanishing point coordinates are determined, and finally based on the target vanishing point coordinates, the external parameter angle of the infrared camera is determined. Through the above method, the external parameter angle of the infrared camera is determined by combining the driver's pupil image and the road lane line image, thereby improving the calibration accuracy of the external parameter angle of the infrared camera.

[0017] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0018] See also Figure 1 , an embodiment of the present application provides an extrinsic parameter calibration method for an infrared camera, the method comprising:

[0019] Step S110: When the vehicle is between two lane lines, a plurality of driver's pupil images are acquired by an in-vehicle camera disposed on the vehicle, and a road lane line image is acquired by an infrared camera disposed on the vehicle, wherein the driver's pupil image is an image of the driver's pupil captured by the in-vehicle camera when the driver visually looks at the infinitely far intersection of the two lane lines in front of the vehicle.

[0020] In the embodiment of the present application, when the vehicle is in the middle of two lane lines in the preset scene, the vehicle begins to collect images through the in-vehicle camera and the infrared camera installed in the vehicle. The preset scene may be a scene of a three-lane road with plenty of sunshine on a sunny day, few vehicles on the lanes and a straight road. Specifically, since the infrared camera is imaged based on the temperature difference principle, since the layer of the lane line is different from the road surface, a temperature difference is formed, and feature extraction can be performed based on this feature. In order to improve the accuracy of extracting feature points, a scene of a three-lane road with plenty of sunshine on a sunny day, few vehicles on the lanes and a straight road can be selected.

[0021] The vehicle is in the middle of two lane lines means that the vehicle is parked in the middle of two lane lines. Two lane lines are two adjacent lane lines on the road surface, that is, two lane lines are two lane lines that constitute one lane.

[0022] The in-car camera is a camera of a different type from the infrared camera, and the in-car camera is a camera with pre-calibrated parameters; the infinity intersection of the two lane lines refers to the intersection position of the two lane lines at infinity. It can be known that when the driver is in the driving position and looking forward, the lane lines on the road are not parallel, and the two lane lines on the road will have an intersection at infinity, which is the infinity intersection of the two lane lines in the embodiment of the present application.

[0023] As one method, when the vehicle is parked between two lane lines, the driver looks at the intersection of the two lane lines at infinity, and at this time, the driver's pupil image is collected by the in-car camera installed in the vehicle. When the in-car camera starts to collect the driver's pupil image, multiple driver's pupil images can be collected continuously; or, a driver's pupil image is collected at a preset time interval to obtain multiple driver's pupil images; or, multiple driver's pupil images are collected within a preset time period, which is not specifically limited here. Among them, the preset time interval and the preset time period can be determined according to actual conditions. In an embodiment of the present application, the multiple driver's pupil images can include at least four driver's pupil images.

[0024] Optionally, when the in-car camera starts to collect the driver's pupil image, the infrared camera can start to collect the road lane line image in front of the vehicle. Alternatively, when the in-car camera collects the last driver's pupil image among multiple driver's pupil images, the infrared camera starts to collect the road lane line image in front of the vehicle. Alternatively, when the in-car camera collects the driver's pupil image at a specified position among multiple driver's pupil images, the infrared camera starts to collect the road lane line image in front of the vehicle. The specified position refers to the collection time that is after the collection time of the first driver's pupil image and before the collection time of the last driver's pupil image in the collection time sequence.

[0025] Furthermore, when the vehicle is parked between two lane lines, before the in-car camera starts to collect the driver's pupil image, it is necessary to first determine whether the driver has visually viewed the intersection position of the two lane lines at infinity. In the embodiment of the present application, the in-car camera can be used to identify the image in the driver's pupil. If the two lane lines intersect at one point in the image in the driver's pupil, it can be determined that the driver has visually viewed the intersection position of the two lane lines at infinity; conversely, if the two lane lines have not yet intersected at one point in the image in the driver's pupil, it can be determined that the driver has not yet visually viewed the intersection position of the two lane lines at infinity.

[0026] Optionally, when the vehicle is in the middle of two lane lines and the driver is looking at the infinite intersection of the two lane lines in front of the vehicle, in response to the image acquisition instruction, multiple driver pupil images are acquired through the in-vehicle camera, and the road lane line image is acquired through the infrared camera. Among them, the image acquisition instruction can be an instruction sent by other electronic devices that have established a communication connection with the vehicle, and the image acquisition instruction includes the time to instruct the in-vehicle camera and the infrared camera to start acquiring images. Further, the image acquisition instruction may include a first image acquisition instruction and a second image acquisition instruction, the first image acquisition instruction is used to instruct the in-vehicle camera to start acquiring images, and the second image acquisition instruction is used to instruct the infrared camera to start acquiring instructions. The first image acquisition instruction and the second image acquisition instruction can be instructions sent by the same electronic device, or they can be instructions sent by different electronic devices, which are not specifically limited here. Among them, the electronic device can be a mobile phone, a computer, a tablet computer, a smart wearable device and other devices, which are not specifically limited here.

[0027] Step S120: Based on the multiple driver pupil images, determine the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera.

[0028] In an embodiment of the present application, after obtaining multiple driver pupil images, the pupil gaze points corresponding to the multiple driver pupil images can be obtained by performing image recognition processing on the multiple driver pupil images, wherein the driver's pupil gaze point can be understood as the intersection point of the two lane lines at infinity.

[0029] After obtaining the pupil fixation points corresponding to the multiple pupil images of the driver, the target fixation point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera can be determined from the pupil fixation points corresponding to the multiple pupil images of the driver according to the preset rules. The preset rules are pre-set rules for determining the target fixation point of the driver from the pupil fixation points corresponding to the multiple pupil images of the driver, and the target fixation point coordinates refer to the fixation point coordinates in the image coordinate system corresponding to the in-vehicle camera.

[0030] Step S130: Based on the road lane line image, determine the intersection coordinates of the two lane lines in the image coordinate system corresponding to the infrared camera.

[0031] In the embodiment of the present application, when the road lane line image acquired by the infrared camera is a single image, the road lane line image can be directly processed to determine the intersection coordinates of two lane lines in the road lane line image. The intersection coordinates of the two lane lines refer to the intersection coordinates in the image coordinate system corresponding to the infrared camera.

[0032] In the case where the road lane line images acquired by the infrared camera are multiple images, one image may be selected from the multiple road lane line images as the road lane line image in step S130; or one image may be selected from the multiple road lane line images according to a preset rule as the road lane line image in step S130. The preset rule is a pre-set rule for selecting the road lane line image in step S130, for example, the preset rule may be set to select the image with the highest image clarity as the road lane line image in step S130; for another example, the preset rule may be set to select the image with the largest field of view included in the image as the road lane line image in step S130.

[0033] Step S140: Determine the target vanishing point coordinates based on the target gaze point coordinates and the intersection point coordinates.

[0034] In the embodiment of the present application, after the target gaze point coordinates and the intersection point coordinates are determined, the target vanishing point coordinates can be obtained by processing the target gaze point coordinates and the intersection point coordinates. The target vanishing point can be understood as the final intersection of the two lane lines at infinity for the subsequent calculation steps; the target vanishing point coordinates are the pixel coordinates in the pixel coordinate system corresponding to the infrared camera.

[0035] Step S150: Determine the external parameter angle of the infrared camera based on the target vanishing point coordinates.

[0036] In the embodiment of the present application, after obtaining the coordinates of the target vanishing point, the coordinates of the target vanishing point can be substituted into a preset calculation formula to obtain the external parameter angle of the infrared camera. The external parameter angle of the infrared camera may include the yaw angle and the pitch angle.

[0037] After the yaw angle and pitch angle of the infrared camera are calculated, it can be determined whether the calculated yaw angle and pitch angle of the infrared camera meet the preset conditions. If they meet the preset conditions, the external parameter angle of the infrared camera is updated. The preset conditions are the preset maximum thresholds of the yaw angle and the pitch angle. If the calculated yaw angle and the pitch angle of the infrared camera are both less than the corresponding maximum thresholds, it can be determined that the calculated yaw angle and the pitch angle of the infrared camera meet the preset conditions, and the external parameter angle of the infrared camera is updated based on the calculated yaw angle and the pitch angle of the infrared camera.

[0038] The present application provides an extrinsic parameter calibration method for an infrared camera. When a vehicle is in the middle of two lane lines, a plurality of driver pupil images are obtained by an in-vehicle camera arranged on the vehicle, and a road lane line image is obtained by an infrared camera arranged on the vehicle, wherein the driver pupil image is an image of the driver's pupil captured by the in-vehicle camera when the driver visually looks at the infinitely far intersection of the two lane lines in front of the vehicle, and then based on the plurality of driver pupil images, the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera are determined, based on the road lane line image, the intersection coordinates of the two lane lines in the image coordinate system corresponding to the infrared camera are determined, and then based on the target gaze point coordinates and the intersection coordinates, the target vanishing point coordinates are determined, and finally based on the target vanishing point coordinates, the extrinsic parameter angle of the infrared camera is determined. Through the above method, the position constraint compensation of the target gaze point is realized through the driver's pupil image, and the dependence on the hardware such as the calibration board, combined inertial navigation and laser radar is eliminated, the calibration success rate is improved under poor road conditions, and finally the purpose of high-precision calibration of the extrinsic parameters of the infrared camera is achieved at a low cost.

[0039] See also Figure 2 , an embodiment of the present application provides an extrinsic parameter calibration method for an infrared camera, the method comprising:

[0040] Step S210: When the vehicle is between two lane lines, a plurality of driver's pupil images are acquired by an in-vehicle camera disposed on the vehicle, and a road lane line image is acquired by an infrared camera disposed on the vehicle, wherein the driver's pupil image is an image of the driver's pupil captured by the in-vehicle camera when the driver visually looks at the infinitely far intersection of the two lane lines in front of the vehicle.

[0041] Step S220: Based on the multiple driver pupil images, determine the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera.

[0042] As one approach, step S220 may specifically include:

[0043] Step S211: determining a plurality of pupil fixation points corresponding to each of the plurality of driver pupil images, and obtaining a plurality of groups of pupil fixation points, wherein each group of pupil annotation points includes a plurality of pupil fixation points corresponding to each driver pupil image.

[0044] In the embodiment of the present application, the multiple pupil fixation points corresponding to the multiple driver pupil images can be determined by an image recognition algorithm; one driver pupil image can determine multiple pupil fixation points. Specifically, the pupil fixation points identified by different image recognition algorithms may be different. Therefore, each driver pupil image can be identified by different image recognition algorithms to obtain multiple pupil fixation points corresponding to each driver pupil image. The image recognition algorithm can be an RCNN algorithm, a YOLO algorithm, etc., which is not specifically limited here.

[0045] The pupil gaze point corresponding to each determined driver pupil image can be represented by the image coordinates in the image coordinate system corresponding to the in-vehicle camera.

[0046] Step S212: Determine multiple fixation point sets from the multiple groups of pupil fixation points, and determine multiple fitting circles, each of the fixation point sets is a set consisting of at least three pupil fixation points randomly selected from the multiple pupil fixation points of each group of pupil fixation points, and different fixation point sets include at least one different pupil fixation point.

[0047] In the embodiment of the present application, since at least three points can determine a circle, after obtaining multiple sets of pupil fixation points, at least three pupil fixation points can be randomly selected from the multiple sets of pupil fixation points each time as a fixation point set. The multiple fixation point sets are all different, and the difference here can refer to the different number of pupil fixation points included in the fixation point set, or can refer to the different pupil fixation points included in the fixation point set (at least one pupil fixation point is different).

[0048] The fitting circle refers to the smooth curve obtained after fitting each set of fixation points.

[0049] As a method, after obtaining multiple sets of fixation points, pupil fixation points in the multiple sets of fixation points can be fitted by a fitting algorithm to determine the fitting circle corresponding to each set of fixation points. The fitting algorithm can be at least one of the least squares method, kernel method, spline method, maximum likelihood estimation method, lsqcurvefit function, polyfit and polyval functions, and a polynomial fitting algorithm with integral constraints.

[0050] Step S213: determining, from each set of fixation points, the inner point sets corresponding to the plurality of fitted circles respectively.

[0051] In the embodiment of the present application, the inner point set corresponding to each of the multiple fitting circles refers to a set consisting of pupil fixation points whose fitting errors are less than a preset threshold in the fixation point set corresponding to the multiple fitting circles.

[0052] As a method, the fitting errors between the pupil fixation points included in each fixation point set and the corresponding fitting circles are determined respectively; based on the fitting errors, the inner point sets corresponding to the multiple fitting circles are determined respectively from each fixation point set, wherein each of the inner point sets is a set consisting of pupil fixation points in the corresponding fixation point set whose corresponding fitting errors are less than a preset threshold.

[0053] In the embodiment of the present application, after obtaining the set of inner points corresponding to each fitting circle, the fitting error can be determined by the fitting error calculation formula. The fitting error calculation formula is a method used to measure the fitting data, also known as the degree of fit or goodness of fit; it is usually used to measure the difference between the fitting curve and the original data to evaluate the accuracy of the fitting curve.

[0054] Among them, the fitting error calculation formula can be: Where yi is the true value of the coordinate of the pupil gaze point in the gaze point set, is the fitting value in the fitting circle, and n is the number of pupil fixation points.

[0055] After the fitting error between the pupil fixation point in each fixation point set and the fitting circle is calculated by the above fitting error calculation formula, the pupil fixation point in each fixation point set whose fitting error is less than a preset threshold can be added to its corresponding inner point set to obtain the inner point set corresponding to each fitting circle. The preset threshold is the maximum value of the pre-set fitting error.

[0056] Step S215: determining the best fitting circle from the multiple fitting circles based on the inner point sets corresponding to the multiple fitting circles.

[0057] In an embodiment of the present application, after determining the inner point sets corresponding to each fitting circle, the number of pupil fixation points included in each inner point set is counted, and the fitting circle corresponding to the inner point set with the largest number of pupil fixation points among multiple inner point sets is determined as the best fitting circle.

[0058] Step S216: determining the coordinates of the center of the best fitting circle as the coordinates of the target gaze point of the driver in the image coordinate system corresponding to the in-vehicle camera.

[0059] In the embodiment of the present application, the center coordinates of the best fitting circle are the image coordinates in the image coordinate system corresponding to the in-vehicle camera.

[0060] After obtaining the center coordinates of the best fitting circle, the center coordinates of the best fitting circle are compared with the preset vanishing point reference coordinate range. If the center coordinates of the best fitting circle are within the preset vanishing point reference coordinate range, the center coordinates of the best fitting circle are determined as the target gaze point coordinates of the driver. The preset vanishing point reference coordinate range can be determined according to the actual debugging situation.

[0061] If the determined center coordinates of the best fitting circle are not within the preset vanishing point reference coordinate range, steps S211 to S216 are re-executed.

[0062] Step S230: grayscale processing is performed on the road lane line image to obtain a grayscale road lane line image.

[0063] In the embodiment of the present application, the road lane line image is gray-processed by a gray-scale processing algorithm to obtain a gray-scale road lane line image. The gray-scale processing algorithm can be one of a maximum method, an average method, a weighted average method, a desaturation method, and a decomposition method, which is not specifically limited here.

[0064] Specifically, the maximum value method: taking the maximum value of the three RGB components as the value of the grayscale image will make the grayscale of the processed image brighter.

[0065] Average value method: Take the average value of the three RGB components as the value of the grayscale image, and the processed image appears softer.

[0066] Weighted average method: assign different weights to the three components of R, G, and B, and take the weighted average of the three components of RGB as the mean of the grayscale image. Since the human eye is most sensitive to green, followed by red, and least sensitive to blue, Wg>Wr>Wb is usually the case, and the generated image is more in line with the visual perception of the human eye. According to statistical analysis, the most reasonable image is usually obtained when Wr=30%, Wg=59%, and Wb=11%.

[0067] Desaturation method: Convert RGB to HLS, then set the saturation to 0, take a color, and convert it to the least saturated value. After desaturation, the image will have a weaker three-dimensional effect, but will be softer.

[0068] Decomposition method: decompose the image into different color and brightness channels for processing.

[0069] Step S240: Acquire feature points of two lane lines from the grayscale road lane line image.

[0070] In the embodiment of the present application, after obtaining the grayscale road lane line image, the feature points of the two lane lines can be extracted from the grayscale road lane line image by using a SIFT (Scale Invariant Feature Transform) algorithm.

[0071] Step S250: Based on the feature points of the two lane lines, determine the coordinates of the intersection of the two lane lines in the image coordinate system corresponding to the infrared camera.

[0072] In the embodiment of the present application, further, the intersection coordinates of the two lane lines in the image coordinate system corresponding to the infrared camera can be determined based on the feature points of the two lane lines through the SIFT algorithm.

[0073] Step S260: Projecting the target gaze point coordinates into the image coordinate system corresponding to the infrared camera to obtain the gaze point mapping coordinates of the target gaze point in the image coordinate system corresponding to the infrared camera.

[0074] In the embodiment of the present application, the gaze point mapping coordinates refer to the image coordinates of the target gaze point in the image coordinate system corresponding to the infrared camera.

[0075] As a method, the actual installation positions of the in-vehicle camera and the infrared camera are obtained; based on the actual installation positions of the in-vehicle camera and the infrared camera, the transformation matrix between the in-vehicle camera and the infrared camera is determined; based on the transformation matrix, the target gaze point coordinates are projected into the image coordinate system corresponding to the infrared camera to obtain the gaze point mapping coordinates of the target gaze point in the image coordinate system corresponding to the infrared camera.

[0076] In an embodiment of the present application, the actual installation positions of the in-vehicle camera and the infrared camera have been determined after the vehicle leaves the factory. Therefore, the transformation matrix between the in-vehicle camera and the infrared camera can be directly determined according to the actual installation positions of the in-vehicle camera and the infrared camera. The transformation matrix here can be understood as the transformation matrix between the image coordinate system corresponding to the in-vehicle camera and the image coordinate system corresponding to the infrared camera.

[0077] As a method, when the vehicle leaves the factory, both the in-vehicle camera and the infrared camera can be set at a designated position inside the vehicle, the in-vehicle camera is used to collect the driver's pupil image, and the infrared camera is used to collect the road lane line image in front of the vehicle; or, when the vehicle leaves the factory, the in-vehicle camera is set at a designated position inside the vehicle, and the infrared camera is set at a designated position outside the vehicle. The designated position is a position determined based on multiple tests where the driver's pupil image and the road lane line image in front of the vehicle can be well collected.

[0078] After determining the transformation matrix, the target gaze point and the transformation matrix can be substituted into the following formula to calculate the gaze point mapping coordinates. The formula can be: Among them, (x c ,y c ) is the gaze point mapping coordinate, (x e ,y e ) is the target gaze point coordinate, M is the transformation matrix between the in-car camera and the infrared camera, which can be calculated according to the actual installation positions of the in-car camera and the infrared camera.

[0079] Step S270: performing Kalman filtering on the gaze point mapping coordinates and the intersection point coordinates to determine the target vanishing point coordinates.

[0080] In the embodiment of the present application, the gaze point mapping coordinates and the intersection point coordinates can be filtered by a Kalman filter to obtain the target vanishing point coordinates, wherein the target vanishing point coordinates are pixel coordinates in the pixel coordinate system corresponding to the infrared camera.

[0081] Step S280: Substituting the target vanishing point coordinates into a preset calculation formula to determine the external parameter angle of the infrared camera.

[0082] Wherein, the preset calculation formula includes: α=sin -1 R yz , Wherein, K is the camera intrinsic parameter matrix of the infrared camera, P ∞ =(u,v,1) T is the target vanishing point coordinate, (R xz ,R yz ,R zz ) is the rotation matrix of the infrared camera, α is the yaw angle of the infrared camera, and β is the pitch angle of the infrared camera.

[0083] The camera intrinsic parameter matrix of the infrared camera has been calibrated before the infrared camera leaves the factory and is a known parameter. The rotation matrix of the infrared camera is also a known parameter in this step, and the rotation matrix is ​​a part of the parameters in the conversion matrix determined in step S260.

[0084] In the embodiment of the present application, the process described in step S210 to step S280 can be repeated multiple times, and if the yaw angle and pitch angle of the infrared camera finally calculated multiple times meet the set threshold, the external parameter angle of the infrared camera is updated. The set threshold is the maximum threshold in the aforementioned preset conditions.

[0085] As one of the methods, if the yaw angle and pitch angle of the infrared camera finally calculated for multiple consecutive times meet the set threshold, and the values ​​of the yaw angle and pitch angle of the infrared camera calculated multiple times are different, then the external parameter angle of the infrared camera is updated to the yaw angle and pitch angle of the infrared camera calculated for the last time; otherwise, the external parameter angle of the infrared camera can be updated to the yaw angle and pitch angle of the infrared camera calculated any one of the multiple consecutive times.

[0086] The present application provides an extrinsic parameter calibration method for an infrared camera, which combines a driver's pupil image and a road lane line image to determine the extrinsic parameter angle of the infrared camera, thereby reducing the calibration cost and improving the calibration accuracy of the extrinsic parameter angle of the infrared camera.

[0087] See also Figure 3 The embodiment of the present application provides an external parameter calibration device 300 for an infrared camera, and the device 300 includes:

[0088] The image acquisition unit 310 is used to acquire multiple driver's pupil images through an in-vehicle camera arranged on the vehicle when the vehicle is between two lane lines, and to acquire a road lane line image through an infrared camera arranged on the vehicle, wherein the driver's pupil image is an image of the driver's pupil captured by the in-vehicle camera when the driver visually looks at the infinitely far intersection of the two lane lines in front of the vehicle.

[0089] The first determination unit 320 is used to determine the coordinates of the target gaze point of the driver in the image coordinate system corresponding to the in-vehicle camera based on the multiple driver pupil images.

[0090] As a method, the first determination unit 320 is specifically used to determine multiple pupil fixation points corresponding to each of the multiple driver pupil images, and obtain multiple groups of pupil fixation points, wherein each group of pupil fixation points includes multiple pupil fixation points corresponding to each driver pupil image; determine multiple fixation point sets from the multiple groups of pupil fixation points, and determine multiple fitting circles, each of the fixation point sets is a set consisting of at least three pupil fixation points randomly selected from the multiple pupil fixation points of each group of pupil fixation points; determine the inner point sets corresponding to each of the multiple fitting circles from each fixation point set; determine the best fitting circle from the multiple fitting circles based on the inner point sets corresponding to each of the multiple fitting circles; and determine the center coordinates of the best fitting circle as the target fixation point coordinates of the driver.

[0091] Furthermore, the first determination unit 320 is specifically used to respectively determine the fitting errors between the pupil fixation points included in the multiple fixation point sets and the corresponding fitting circles; based on the fitting errors, respectively determine the inner point sets corresponding to the multiple fitting circles from the multiple fixation point sets, wherein each of the inner point sets is a set consisting of pupil fixation points in the corresponding fixation point set whose corresponding fitting errors are less than a preset threshold.

[0092] The second determining unit 330 is used to determine the coordinates of the intersection of two lane lines in the image coordinate system corresponding to the infrared camera based on the road lane line image.

[0093] As a method, the second determination unit 330 is specifically used to perform grayscale processing on the road lane line image to obtain a grayscale road lane line image; obtain feature points of two lane lines from the grayscale road lane line image; and determine the intersection coordinates of the two lane lines in the image coordinate system corresponding to the infrared camera based on the feature points of the two lane lines.

[0094] The third determining unit 340 is configured to determine the target vanishing point coordinates based on the target gaze point coordinates and the intersection point coordinates.

[0095] As a method, the third determination unit 340 is specifically used to project the target gaze point coordinates into the image coordinate system corresponding to the infrared camera to obtain the gaze point mapping coordinates of the target gaze point in the image coordinate system corresponding to the infrared camera; perform Kalman filtering on the gaze point mapping coordinates and the intersection coordinates to determine the target vanishing point coordinates.

[0096] Furthermore, the third determination unit 340 is specifically used to obtain the actual installation positions of the in-vehicle camera and the infrared camera respectively; based on the actual installation positions of the in-vehicle camera and the infrared camera respectively, determine the transformation matrix between the in-vehicle camera and the infrared camera; based on the transformation matrix, project the target gaze point coordinates into the image coordinate system corresponding to the infrared camera to obtain the gaze point mapping coordinates of the target gaze point in the image coordinate system corresponding to the infrared camera.

[0097] The external parameter determination unit 350 is used to determine the external parameter angle of the infrared camera based on the target vanishing point coordinates.

[0098] As a method, the external parameter determination unit 350 is specifically used to substitute the target vanishing point coordinates into a preset calculation formula to determine the external parameter angle of the infrared camera; wherein the preset calculation formula includes: α=sin -1 R yz , Wherein, K is the camera intrinsic parameter matrix of the infrared camera, P ∞ =(u,v,1) T is the target vanishing point coordinate, (R xz ,R yz ,R zz ) is the rotation matrix of the infrared camera, α is the yaw angle of the infrared camera, and β is the pitch angle of the infrared camera.

[0099] It should be noted that the device embodiment in the present application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0100] The following will be combined Figure 4 A vehicle provided in the present application is described.

[0101] See also Figure 4 Based on the above-mentioned infrared camera external parameter calibration method and device, the embodiment of the present application also provides another vehicle 800 that can execute the above-mentioned infrared camera external parameter calibration method. The vehicle 800 includes one or more (only one is shown in the figure) processors 802, a memory 804, a network module 806, an in-vehicle camera 808, and an infrared camera 810 that are coupled to each other. Among them, the memory 804 stores a program that can execute the content of the above-mentioned embodiment, and the processor 802 can execute the program stored in the memory 804.

[0102] Among them, the processor 802 may include one or more processing cores. The processor 802 uses various interfaces and lines to connect various parts of the entire vehicle 800, and executes various functions and processes data of the vehicle 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 804, and calling data stored in the memory 804. Optionally, the processor 802 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 802 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 802, but may be implemented separately through a communication chip.

[0103] The memory 804 may include a random access memory (RAM) or a read-only memory (ROM). The memory 804 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 804 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the vehicle 800 during use (such as a phone book, audio and video data, chat record data), etc.

[0104] The network module 806 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices, such as communicating with a vehicle. The network module 806 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, etc. The network module 806 may communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices via wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. For example, the network module 806 may exchange information with a base station.

[0105] The in-car camera 808 may be a calibrated camera, and the in-car camera is a camera other than an infrared camera. In the embodiment of the present application, the in-car camera 808 may be used to collect multiple driver pupil images continuously or at a preset time interval.

[0106] The infrared camera 810 may be an infrared camera with camera internal parameters calibrated at the factory. In the embodiment of the present application, the infrared camera 810 may be used to collect the image of the road lane line in front of the vehicle.

[0107] Please refer to Figure 5 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 900 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0108] The computer-readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 910 can be compressed, for example, in an appropriate form.

[0109] The present application provides an infrared camera external parameter calibration method, device, vehicle and storage medium. When the vehicle is in the middle of two lane lines, a plurality of driver pupil images are obtained by an in-vehicle camera arranged on the vehicle, and a road lane line image is obtained by an infrared camera arranged on the vehicle, wherein the driver pupil image is an image of the driver's pupil captured by the in-vehicle camera when the driver visually looks at the infinite intersection of the two lane lines in front of the vehicle. Then, based on the plurality of driver pupil images, the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera are determined. Based on the road lane line image, the intersection coordinates of the two lane lines in the image coordinate system corresponding to the infrared camera are determined. Then, based on the target gaze point coordinates and the intersection coordinates, the target vanishing point coordinates are determined. Finally, based on the target vanishing point coordinates, the external parameter angle of the infrared camera is determined. Through the above method, the external parameter angle of the infrared camera is determined by combining the driver's pupil image and the road lane line image, thereby improving the calibration accuracy of the external parameter angle of the infrared camera.

[0110] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for calibrating external parameters of an infrared camera, characterized in that: The method comprises: When the vehicle is in the middle of two lane lines, a plurality of driver's pupil images are acquired by an in-vehicle camera disposed on the vehicle, and a road lane line image is acquired by an infrared camera disposed on the vehicle, wherein the driver's pupil image is an image of the driver's pupil acquired by the in-vehicle camera when the driver visually looks at the infinitely far intersection of the two lane lines in front of the vehicle; Based on the multiple driver pupil images, determining the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera; Based on the road lane line image, determining the coordinates of the intersection of the two lane lines in the image coordinate system corresponding to the infrared camera; Determining target vanishing point coordinates based on the target fixation point coordinates and the intersection point coordinates; Based on the target vanishing point coordinates, an external parameter angle of the infrared camera is determined.

2. The method according to claim 1, characterized in that The step of determining the target gaze point coordinates of the driver in the image coordinate system corresponding to the in-vehicle camera based on the multiple driver pupil images includes: Determine a plurality of pupil fixation points corresponding to each of the plurality of driver pupil images to obtain a plurality of groups of pupil fixation points, wherein each group of pupil fixation points includes a plurality of pupil fixation points corresponding to each of the driver pupil images; Determine a plurality of fixation point sets from the plurality of groups of pupil fixation points, and determine a plurality of fitting circles, wherein each of the fixation point sets is a set consisting of at least three pupil fixation points randomly selected from the plurality of pupil fixation points of each group of pupil fixation points; Determining, from each set of fixation points, respective sets of interior points corresponding to the plurality of fitted circles; Determining a best fitting circle from the plurality of fitting circles based on the sets of interior points corresponding to the plurality of fitting circles; The coordinates of the center of the best fitting circle are determined as the coordinates of the target gaze point of the driver in the image coordinate system corresponding to the in-vehicle camera.

3. The method according to claim 2, characterized in that The step of determining, from each set of fixation points, the respective sets of inner points corresponding to the plurality of fitted circles comprises: respectively determining fitting errors between pupil fixation points included in the plurality of fixation point sets and corresponding fitting circles; Based on the fitting error, inner point sets corresponding to each of the multiple fitting circles are determined from the multiple fixation point sets, wherein each of the inner point sets is a set consisting of pupil fixation points in the corresponding fixation point set whose corresponding fitting error is less than a preset threshold.

4. The method according to claim 1, characterized in that: The determining, based on the road lane line image, the coordinates of the intersection of two lane lines in the image coordinate system corresponding to the infrared camera comprises: Performing grayscale processing on the road lane line image to obtain a grayscale road lane line image; Acquire feature points of two lane lines from the grayscale road lane line image; Based on the feature points of the two lane lines, the coordinates of the intersection of the two lane lines in the image coordinate system corresponding to the infrared camera are determined.

5. The method according to claim 1, characterized in that The step of determining the target vanishing point coordinates based on the target gaze point coordinates and the intersection point coordinates includes: Projecting the target gaze point coordinates into the image coordinate system corresponding to the infrared camera to obtain the gaze point mapping coordinates of the target gaze point in the image coordinate system corresponding to the infrared camera; Kalman filtering is performed on the gaze point mapping coordinates and the intersection point coordinates to determine the target vanishing point coordinates.

6. The method according to claim 5, characterized in that The projecting the target gaze point coordinates into the image coordinate system corresponding to the infrared camera to obtain the gaze point mapping coordinates of the target gaze point coordinates in the image coordinate system corresponding to the infrared camera includes: Obtaining actual installation positions of the in-vehicle camera and the infrared camera respectively; Determine a conversion matrix between the in-vehicle camera and the infrared camera based on actual installation positions of the in-vehicle camera and the infrared camera respectively; Based on the transformation matrix, the target gaze point coordinates are projected into the image coordinate system corresponding to the infrared camera to obtain the gaze point mapping coordinates of the target gaze point coordinates in the image coordinate system corresponding to the infrared camera.

7. The method according to claim 1, characterized in that The step of determining the external parameter angle of the infrared camera based on the target vanishing point coordinates includes: Substituting the target vanishing point coordinates into a preset calculation formula to determine the external parameter angle of the infrared camera; Wherein, the preset calculation formula includes: Wherein, K is the camera intrinsic parameter matrix of the infrared camera, P ∞ =(u,v,1) T is the target vanishing point coordinate, (R xz ,R yz ,R zz ) is the rotation matrix of the infrared camera, α is the yaw angle of the infrared camera, and β is the pitch angle of the infrared camera.

8. An external parameter calibration device for an infrared camera, characterized in that: The device comprises: An image acquisition unit, used for acquiring a plurality of driver's pupil images through an in-vehicle camera disposed on the vehicle, and acquiring a road lane line image through an infrared camera disposed on the vehicle when the vehicle is in the middle of two lane lines, wherein the driver's pupil image is an image acquired by the in-vehicle camera of the driver's pupil when the driver visually looks at the infinitely far intersection of the two lane lines in front of the vehicle; A first determining unit, configured to determine the coordinates of a target gaze point of the driver in an image coordinate system corresponding to the in-vehicle camera based on the plurality of pupil images of the driver; A second determination unit, configured to determine the coordinates of an intersection of two lane lines in an image coordinate system corresponding to the infrared camera based on the road lane line image; A third determining unit, configured to determine a target vanishing point coordinate based on the target gaze point coordinate and the intersection point coordinate; The external parameter determination unit is used to determine the external parameter angle of the infrared camera based on the target vanishing point coordinates.

9. A vehicle, characterized in that: The method comprises an in-vehicle camera, an infrared camera, and one or more processors; one or more programs are stored in a memory and configured so that the one or more processors execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, wherein when the program codes are executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images