Vehicle-mounted camera external parameter calibration method and device, and related product

Abstract

The invention relates to a vehicle-mounted camera external parameter calibration method and device and a related product, and the method comprises the steps: obtaining an image of a vehicle-mounted camera during curve driving between two parallel lines, the image comprising two parallel lines, and the two parallel lines are respectively marked as a first parallel line and a second parallel line; acquiring transverse pixel coordinates of points on parallel lines at a preset height of the image; drawing a trend chart that the transverse pixel coordinates of the points on the parallel lines at the same preset height change along with time; acquiring an extreme point of the trend chart, and selecting an image at a moment corresponding to the extreme point as a calibration image; and calculating and generating external parameters of the vehicle-mounted camera according to the internal parameters of the vehicle-mounted camera, the world coordinates of the points on the two parallel lines in the calibration image and the corresponding pixel coordinates.

Description

Technical Field

[0001] The present invention relates to the field of assisted driving and automatic driving, and in particular to a method, device, program product, medium, computer device and mobile tool for calibrating external parameters of a vehicle-mounted camera. Background Art

[0002] Both assisted driving systems and autonomous driving systems include visual perception modules. In systems involving lateral control, the visual perception module is usually required to identify the ground lane lines in the image, and then convert them into lane lines in the vehicle coordinate system with a bird's-eye view through the camera's internal and external parameters. The vehicle control link performs lateral control based on the lane lines, such as lane keeping. Therefore, it is very important to calibrate the camera's internal and external parameters with high precision before the vehicle travels. Among them, the calibration of internal parameters is relatively mature, but the calibration accuracy of the rotation angle parameters of the external parameters is not high. For example, the rotation angle of the camera relative to the vehicle's driving direction often has calibration deviations, resulting in inaccurate relative relationship between the lane lines received by the control module and the vehicle, which ultimately causes the vehicle to deviate from the lane or even snake in a snake-like manner.

[0003] At present, the camera extrinsic calibration methods mainly include the vanishing point based method and the visual inertial odometry based method.

[0004] The vanishing point-based method requires that the vehicle and several parallel lines on the road remain parallel. It is difficult to guarantee this parallel requirement in actual operation, and it can only be calibrated statically. If a calibration room is used to guarantee the parallel requirement, the cost is high. This calibration method is to calibrate the camera extrinsic parameters to a point on the vehicle's center axis as considered by the operator. Since there are large subjective differences in the confirmation of the vehicle's center axis, and there may be an angular deviation between the vehicle's center axis and the vehicle's driving direction, even if the camera extrinsic parameters are accurately calibrated to the vehicle's center axis, it may still be inconsistent with the vehicle's driving direction, thereby affecting the final calibration accuracy.

[0005] The method based on visual inertial odometry calibrates the coordinate transformation relationship between the camera and the inertial sensor (Inertial Measurement Unit, IMU), while the coordinate transformation relationship between the IMU and the vehicle's driving direction still requires additional calibration. The angular relationship between the camera and the vehicle's driving direction is not directly calibrated, and the final calibration accuracy is uncontrollable. Summary of the invention

[0006] The purpose of the present invention is to provide a method, device, program product, medium, computer device and mobile tool for calibrating the external parameters of a vehicle-mounted camera in view of the defects of the prior art, so as to solve the problem of low calibration accuracy of the external parameters of a vehicle-mounted camera in the prior art.

[0007] To achieve the above object, the present invention provides a method for calibrating external parameters of a vehicle-mounted camera in a first aspect, comprising:

[0008] Acquire an image of the vehicle camera when the vehicle is traveling in a curve between two parallel lines, wherein the image includes the two parallel lines, which are respectively recorded as a first parallel line and a second parallel line;

[0009] Obtaining horizontal pixel coordinates of points on parallel lines at a preset height of the image;

[0010] Draw a trend graph of the horizontal pixel coordinates of points on parallel lines at the same preset height changing with time;

[0011] Obtaining the extreme point of the trend graph, and selecting the image at the time corresponding to the extreme point as the calibration image;

[0012] The vehicle camera external parameters are calculated and generated according to the vehicle camera internal parameters, the world coordinates of the points on the two parallel lines in the calibration image and the corresponding pixel coordinates.

[0013] Further, the acquiring of the horizontal pixel coordinates of points on parallel lines at a preset height of the image specifically includes:

[0014] Obtain pixel coordinates of two parallel lines in the image at each moment by using an image detection algorithm;

[0015] Acquire the horizontal pixel coordinates of points on the first parallel line and / or the second parallel line at a preset height of the image, and record the time corresponding to the horizontal pixel coordinates.

[0016] Further, the step of obtaining the extreme point of the trend graph and selecting the image at the time corresponding to the extreme point as the calibration image specifically includes:

[0017] If the trend graph includes a first trend curve and a second trend curve, compare the change trends of the first trend curve and the second trend curve; wherein the first trend curve is a trend line of changes in the horizontal pixel coordinates of points on the first parallel line at a preset height of the image over time, and the second trend curve is a trend line of changes in the horizontal pixel coordinates of points on the second parallel line at a preset height of the image over time;

[0018] If the changing trends are synchronized, the extreme points of the trend graph are obtained, and the image at the time corresponding to the extreme points is selected as the calibration image.

[0019] Further, the step of obtaining the extreme point of the trend graph and selecting the image at the time corresponding to the extreme point as the calibration image specifically includes:

[0020] If the preset height is not less than 2 preset heights, obtain the extreme value point of the trend graph at each preset height;

[0021] Calculate the mean time of the time corresponding to the synchronous extreme points at all preset heights;

[0022] The image at the mean time is selected as the calibration image.

[0023] Further, the vehicle camera extrinsic parameters are calculated and generated according to the vehicle camera intrinsic parameters, the world coordinates of the points on the two parallel lines in the calibration image, and the corresponding pixel coordinates, specifically including:

[0024] The ground is taken as the XY plane of the world coordinate system, and the first parallel line or the second parallel line is taken as the X axis;

[0025] Select two different calibration points on the first parallel line and the second parallel line in a frame of the calibration image, respectively, and record the pixel coordinates of the four calibration points;

[0026] Get the parallel line distance between the first parallel line and the second parallel line in the world coordinate system;

[0027] Generating the world coordinates of the four calibration points according to the parallel line distances;

[0028] The intrinsic parameters of the vehicle camera, the world coordinates and pixel coordinates of the four calibration points are substituted into the matrix equation that converts the pixel coordinates into the world coordinates through the homography matrix, and the Y-axis translation parameters and three rotation angle parameters of the extrinsic parameters of the vehicle camera are solved.

[0029] Furthermore, the two parallel lines are located on a flat and straight road surface, and the curve is an S-shaped curve.

[0030] Further, the calculation and generation of the vehicle camera external parameters according to the vehicle camera internal parameters, the world coordinates of the points on the two parallel lines in the calibration image and the corresponding pixel coordinates specifically includes:

[0031] The images at the time corresponding to multiple extreme points are selected as calibration images, a coordinate mapping equation group is established for each calibration image, and the final vehicle camera external parameters are generated according to the multiple coordinate mapping equation groups.

[0032] A second aspect of the present invention provides an extrinsic parameter calibration device for a vehicle-mounted camera, comprising:

[0033] A first acquisition module is used to acquire an image of the vehicle camera when the vehicle is traveling in a curve between two parallel lines, wherein the image includes the two parallel lines, which are respectively recorded as a first parallel line and a second parallel line;

[0034] A second acquisition module, used to acquire the horizontal pixel coordinates of points on parallel lines at a preset height of the image;

[0035] A drawing module, used for drawing a trend graph of the horizontal pixel coordinates of points on parallel lines at the same preset height changing with time;

[0036] A third acquisition module is used to obtain the extreme point of the trend graph, and select the image at the time corresponding to the extreme point as the calibration image;

[0037] The data processing module is used to calculate and generate the external parameters of the vehicle-mounted camera according to the internal parameters of the vehicle-mounted camera, the world coordinates of the points on the two parallel lines in the calibration image and the corresponding pixel coordinates.

[0038] A third aspect of the present invention provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the vehicle camera extrinsic parameter calibration method as described in any one of the first aspects.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0040] A fifth aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any method described in the first aspect.

[0041] The sixth aspect of the present invention provides a mobile tool, characterized in that it includes the computer device described in the fifth aspect.

[0042] The vehicle camera extrinsic parameter calibration method, device and related products provided by the embodiments of the present invention find the image at the moment when the vehicle's driving direction is parallel to the parallel lines according to the extreme points in the trend diagram when the vehicle camera is traveling between two parallel curves as the calibration image for calculating the vehicle camera extrinsic parameters, and then complete the vehicle camera extrinsic parameter calibration by taking calibration points on two parallel lines in the calibration image. The calibration method does not require the participation of systems such as precise positioning and inertial sensing, reduces the influence of system errors on the vehicle camera extrinsic parameter calibration results and dependence on other systems, thereby improving the accuracy and robustness of the calibration results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 One of the flow charts of the vehicle camera extrinsic parameter calibration method provided in the first embodiment of the present invention;

[0044] Figure 2 A bird's-eye view schematic diagram of a vehicle driving path provided in the first embodiment of the present invention;

[0045] Figure 3 A schematic diagram of selecting points on parallel lines at a preset height provided in the first embodiment of the present invention;

[0046] Figure 4 A trend diagram of horizontal pixel coordinates changing with time provided in the first embodiment of the present invention;

[0047] Figure 5 The second flowchart of the vehicle camera extrinsic parameter calibration method provided in the first embodiment of the present invention;

[0048] Figure 6 Flow chart 3 of the method for calibrating external parameters of a vehicle-mounted camera provided in the first embodiment of the present invention;

[0049] Figure 7 Flow chart 4 of the method for calibrating external parameters of a vehicle-mounted camera provided in the first embodiment of the present invention;

[0050] Figure 8 Flow chart 5 of the method for calibrating external parameters of a vehicle-mounted camera provided in the first embodiment of the present invention;

[0051] Fig. 9 A schematic diagram of the structure of an extrinsic parameter calibration device for a vehicle-mounted camera provided in Embodiment 2 of the present invention;

[0052] Fig.10 A schematic diagram of the structure of a computer program product provided in Embodiment 3 of the present invention;

[0053] Fig.11 This is a schematic diagram of the structure of a computer device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0056] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0057] In the present invention, "module", "device", "system" and the like refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, an element can be, but is not limited to, a process, a processor, an object, an executable element, an execution thread, a program and / or a computer running on a processor. In addition, an application or script program running on a server, a server can all be an element. One or more elements can be in an execution process and / or thread, and an element can be localized on a computer and / or distributed between two or more computers, and can be operated by various computer-readable media. An element can also communicate through local and / or remote processes according to a signal with one or more data packets, for example, a signal from a data that interacts with another element in a local system, a distributed system, and / or a network on the Internet through a signal to interact with other systems.

[0058] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0059] The vehicle-mounted camera external parameter calibration method, device and related products provided by the present invention find the image at the moment when the vehicle's driving direction is parallel to the parallel lines according to the extreme points in the trend diagram when the vehicle-mounted camera is traveling between two parallel curves as the calibration image for calculating the vehicle-mounted camera external parameters, and then complete the vehicle-mounted camera external parameter calibration by taking calibration points on the two parallel lines in the calibration image. The calibration method does not require the participation of systems such as precise positioning and inertial sensing, reduces the influence of system errors on the vehicle-mounted camera external parameter calibration results and the dependence on other systems, thereby improving the accuracy and robustness of the calibration results; by comparing the changing trends of the first trend curve and the second trend curve, the images obtained when the two parallel lines are not parallel are filtered out, thereby avoiding the calibration result error caused by the test condition error; by selecting multiple preset heights, the image corresponding to the mean moment is used as the calibration image, thereby avoiding the extreme point arrival time error caused by the horizontal pixel coordinate marking error; by calculating the average of the camera external parameters at multiple key moments, the final vehicle-mounted camera external parameters are generated, thereby further improving the accuracy of the calibration result.

[0060] Embodiment 1

[0061] The method of the present application is applicable to vehicles that can turn smoothly in any driving direction, such as Ackerman steering vehicles; the vehicle-mounted camera extrinsic parameter calibration method of the present application is applicable to the extrinsic parameter calibration of cameras set in the direction of vehicle movement, not limited to forward-facing vehicle-mounted cameras or rearward-facing vehicle-mounted cameras. For ease of explanation, the following embodiments are described using the forward-facing vehicle-mounted camera as an example.

[0062] Figure 1 This is a flow chart of a method for calibrating external parameters of a vehicle-mounted camera provided in Embodiment 1 of the present invention, as shown in FIG. Figure 1 As shown, the calibration method includes:

[0063] Step 110, obtaining an image of the vehicle camera when the vehicle is traveling in a curve between two parallel lines, wherein the image includes two parallel lines, which are respectively recorded as a first parallel line and a second parallel line.

[0064] Specifically, a camera is set on the vehicle, and the vehicle is controlled by manual operation or automatic control program to make a curved movement between two parallel lines. The two parallel lines can be two parallel lane lines or parallel lines drawn manually on the road surface. The curved path of the vehicle must meet the process of approaching and moving away from the two parallel lines. In a preferred embodiment, the two parallel lines are located on a flat and straight road surface, and the curve is an S-shaped curve.

[0065] In a specific example L1, Figure 2 A bird's-eye view schematic diagram of a vehicle driving path provided in the first embodiment of the present invention, such as Figure 2 As shown, it includes a vehicle 1, a driving curve 2, a first parallel line 3 and a second parallel line 4, wherein the distance between the first parallel line 3 and the second parallel line 4 is D0, and the vehicle 1 travels from point a to point e along the s-shaped driving curve 2. When the vehicle travels from a to b, the distance between the vehicle and the first parallel line 3 is decreasing. When the vehicle 1 travels to the turning point b, the vehicle 1 is parallel to the first parallel line 3. At this time, the distance between the vehicle and the first parallel line 3 no longer decreases, and the vehicle driving direction is parallel to the first parallel line direction. When the vehicle 1 continues to travel along the driving curve 2, the distance between the vehicle and the first parallel line 3 begins to increase. When the vehicle 1 travels to the turning point c, the distance between the vehicle 1 and the first parallel line 3 no longer increases, and the vehicle driving direction is parallel to the first parallel line direction. Similarly, when the vehicle reaches the turning point d, the vehicle driving direction is parallel to the first parallel line direction until it reaches the end point e.

[0066] Step 120, obtaining the horizontal pixel coordinates of points on parallel lines at a preset height of the image.

[0067] Specifically, firstly, the positions of parallel lines are detected by manual annotation or image detection algorithm to obtain the pixel coordinates of two parallel lines in the image at each moment.

[0068] Secondly, obtain the horizontal pixel coordinates of the points on the first parallel line and / or the second parallel line at a preset height of the image, and record the time corresponding to the horizontal pixel coordinates, wherein the value of the preset height must ensure that the horizontal pixel coordinates of the points on the first parallel line and / or the second parallel line can be obtained.

[0069] Combined with Example L1, in a specific example L2, Figure 3 A schematic diagram of selecting points on parallel lines at a preset height provided in the first embodiment of the present invention is shown in FIG. Figure 3 As shown, it includes an image 5, a first parallel line 3, a second parallel line 4, and a preset height line 6.

[0070] Among them, image 6 is a frame of image taken by the on-board camera at any time when the vehicle is traveling on a curve. The distance between the preset height line 6 and the bottom edge of the image is h. The origin of the pixel coordinate system of the image is in the upper left corner of image 5. The horizontal pixel coordinates of the intersection of the preset height line 6 and the first parallel line 3 and the second parallel line 4 are X1 and X2 respectively. If the first parallel line 3 and the second parallel line 4 are standard parallel lines, the difference between X2 and X1 remains unchanged.

[0071] Step 130 , draw a trend graph of the horizontal pixel coordinates of points on the parallel lines at the same preset height changing with time.

[0072] Specifically, the on-board camera captures N frames of images in real time at N moments when the vehicle is traveling in a curve between two parallel lines, each frame of the image corresponds to a time point, and the horizontal pixel coordinates of the points on the same parallel line at a preset height of the N frames of the image are obtained, and a trend graph of the horizontal pixel coordinates of the points on the same parallel line at the same preset height changing with time is drawn, wherein only a first trend curve of the horizontal pixel coordinates of the points on the first parallel line changing with time, or a second trend curve of the horizontal pixel coordinates of the points on the second parallel line changing with time can be drawn, or the first trend curve and the second trend curve can be drawn at the same time. In one possible implementation, if the horizontal pixel coordinates of the points on the parallel lines at N moments do not exceed the burr threshold, the discrete points are directly connected to generate the first trend curve or the second trend curve, and if the horizontal pixel coordinates of the points on the parallel lines at N moments exceed the burr threshold, the discrete points are curve-fitted by a curve fitting algorithm to generate the first trend curve or the second trend curve.

[0073] Combined with Example L2, in a specific Example L3, Figure 4 The trend diagram of the horizontal pixel coordinates changing with time provided in the first embodiment of the present invention is as follows: Figure 4 As shown, it includes a first trend curve 7 and a second trend curve 8.

[0074] The first trend curve 7 is a fitting curve of the horizontal pixel coordinates of the points on the first parallel line 3 at a height h at N moments changing with time; the second trend curve 8 is a fitting curve of the horizontal pixel coordinates of the points on the second parallel line 4 at a height h at N moments changing with time. The first trend curve 7 and the second trend curve 8 have the same changing trend. When the horizontal pixel coordinate X1 gradually increases from t1 to t3, the vehicle is Figure 2 The distance between the vehicle and the first parallel line 3 gradually approaches, and at time t3, the vehicle's driving direction is parallel to the first parallel line 3, and the vehicle is no longer close to the first parallel line 3, and starts to move away from the first parallel line 3.

[0075] Step 140, obtaining the extreme points of the trend graph, and selecting the image at the time corresponding to the extreme points as the calibration image.

[0076] Specifically, if the first trend curve and / or the second trend curve of the trend graph is generated by connecting discrete points, find the extreme point and the time corresponding to the extreme point from the discrete values; if the first trend curve and / or the second trend curve is a fitting curve, calculate the extreme point and the time corresponding to the extreme point according to the fitting equation of the curve; a frame of image at the time corresponding to the extreme point is used as a calibration image; at each extreme point, the vehicle's driving direction is parallel to the direction of the first parallel line and / or the second parallel line.

[0077] like Figure 4 As shown, in combination with Example L3, the first trend curve 7 and the second trend curve 8 are obtained by fitting the equation at t3, t6, t8, t 12 and t N-7 There are extreme points at all times.

[0078] In a possible implementation of step 140, Figure 5 The second flowchart of the vehicle camera external parameter calibration method provided in the first embodiment of the present invention is as follows: Figure 5 As shown, specifically including:

[0079] Step 1401: If the trend graph includes a first trend curve and a second trend curve, compare the change trends of the first trend curve and the second trend curve.

[0080] The first trend curve is a trend line of the horizontal pixel coordinates of points on the first parallel line at a preset height of the image changing with time, and the second trend curve is a trend line of the horizontal pixel coordinates of points on the second parallel line at a preset height of the image changing with time.

[0081] Step 1402, if the changing trends are synchronized, obtain the extreme points of the trend graph, and select the image at the time corresponding to the extreme point as the calibration image.

[0082] Specifically, when the changing trends are not synchronized, it means that the first parallel line and the second parallel line are not standard parallel lines, and the distance between the two parallel lines has changed. When the parallel lines are not calibrated accurately, new system errors will be introduced. Delete the image of the curve driving obtained when the parallel lines are not standard, reselect the lane lines or draw the parallel lines, and return to step 110.

[0083] When the changing trends are synchronized, it means that the first parallel line and the second parallel line are standard parallel lines. The image obtained when driving on the curve under the standard parallel lines will not introduce new system errors. The extreme points of the trend graph are obtained, and the image at the time corresponding to the extreme points is selected as the calibration image.

[0084] In another possible implementation of step 140, Figure 6 The third flowchart of the vehicle camera external parameter calibration method provided in the first embodiment of the present invention is as follows: Figure 6 As shown, specifically including:

[0085] Step 1403, if the preset height is not less than 2 preset heights, obtain the extreme value points of the trend graph at each preset height.

[0086] Step 1404, calculating the mean time of the time corresponding to the synchronous extreme value points at all preset heights.

[0087] Step 1405, selecting the image at the mean time as the calibration image.

[0088] In a specific example L4, assuming that there are a first preset height, a second preset height and a third preset height, at the first preset height, the first trend curve reaches the first extreme point at time W1, at the second preset height, the first trend curve reaches the first extreme point at time W2, and at the third preset height, the first trend curve reaches the first extreme point at time W3, and the image corresponding to the mean time (W1+W2+W3) / 3 of W1, W2 and W3 is selected as the calibration image.

[0089] Step 150, calculating and generating the extrinsic parameters of the vehicle-mounted camera according to the intrinsic parameters of the vehicle-mounted camera, the world coordinates of the points on two parallel lines in the calibration image, and the corresponding pixel coordinates.

[0090] Specifically, the calibration value of the vehicle camera's internal parameter can be selected from the factory parameters, or according to the existing technology, such as recalibration of the chessboard to obtain the calibration value of the vehicle camera's internal parameter. The external parameters of the vehicle camera include: three translation parameters of X, Y, and Z, and three rotation angle parameters along the X, Y, and Z axes. In this application, the vehicle camera's internal parameters are used as known parameters to participate in the solution of the vehicle camera's external parameters. In the calibration image, the actual distance between the two parallel lines is known. In the world coordinate system, the coordinates of the points on the two parallel lines along the direction perpendicular to the parallel lines are known, and all the coordinates along the direction perpendicular to the parallel lines on each parallel line are the same. When the vehicle is traveling in a curve between two parallel lines, the coordinates along the parallel lines and the coordinates along the perpendicular road surface in the world coordinate system cannot be determined. The pixel coordinates of any point on the two parallel lines in the image can be obtained in the pixel coordinate system. Since the coordinates along the parallel lines and the coordinates along the perpendicular road surface cannot be determined in the world coordinate system when the vehicle is traveling in a curve between the two parallel lines, only the coordinate offset parameters and three rotation parameters along the perpendicular direction of the parallel lines can be solved, with a total of 4 parameter variables. According to the conversion relationship between the pixel coordinates and the world coordinates, two points are randomly selected on the two parallel lines, and a set of coordinate mapping equations about the homography matrix with 4 constraints is established. Among them, the homography matrix includes the camera intrinsic parameters and extrinsic parameters. After substituting the known camera intrinsic parameters into the homography matrix, the homography matrix is ​​only related to the camera extrinsic parameters. In the world coordinate system, only the coordinates of the four selected points along the perpendicular direction of the parallel lines are known. The equation can solve the coordinate offset parameters and three rotation parameters along the perpendicular direction of the parallel lines.

[0091] In a possible implementation of step 150, Figure 7 This is a fourth flow chart of the method for calibrating the external parameters of a vehicle-mounted camera provided in the first embodiment of the present invention, as shown in FIG. Figure 7 As shown, specifically including:

[0092] Step 1501, taking the ground as the XY plane of the world coordinate system, and the first parallel line or the second parallel line as the X axis.

[0093] Step 1502: select two different calibration points on the first parallel line and the second parallel line in a frame of calibration image, respectively, and record the pixel coordinates of the four calibration points.

[0094] Step 1503, obtaining the parallel line distance between the first parallel line and the second parallel line in the world coordinate system.

[0095] Step 1504, generating world coordinates of four calibration points according to the parallel line distances.

[0096] Step 1505, substitute the vehicle camera internal parameters, the world coordinates and the pixel coordinates of the four calibration points into the matrix equation that converts the pixel coordinates into the world coordinates through the homography matrix, and solve the Y-axis translation parameter and three rotation angle parameters of the vehicle camera external parameters.

[0097] In a specific example, two different calibration points are taken on the first parallel line and the second parallel line in a calibration image, respectively, and are recorded as (u1, v1), (u2, v2), (u3, v3) and (u4, v4). The pixel coordinates of the four calibration points are all known, where (u1, v1) and (u2, v2) are points taken on the first parallel line, (u3, v3) and (u4, v4) are points taken on the second parallel line, and the distance between the first parallel line and the second parallel line in the world coordinate system is D0. With the first parallel line 3 as the X-axis, the coordinates of (u1, v1) and (u2, v2) in the world coordinate system are recorded as (x1, 0) and (x2, 0), and the coordinates of (u3, v3) and (u4, v4) in the world coordinate system are recorded as (x3, D0) and (x4, D0). Among them, x1, x2, x3 and x4 are unknown quantities.

[0098] The coordinate mapping equation group for transforming the world coordinates and pixel coordinates of points on parallel lines through the homography matrix is ​​(u,v,1)=H*(X,Y,1), where X and Y are the coordinates of the point in the world coordinate system, u and v are the coordinates of the point in the pixel coordinate system, and H is the homography matrix. When the intrinsic parameters are known, H is only related to the 6 variable parameters of the extrinsic parameters. The four pairs of coordinates are substituted into the coordinate mapping equation group (u,v,1)=H*(X,Y,1). Since the coordinates x1, x2, x3 and x4 in the world coordinate system are unknown, the coordinate mapping equation group with 4 constraints can be established through the 4 pairs of coordinates known in the Y-axis direction to obtain the Y-axis translation parameters and three rotation angle parameters of the vehicle-mounted camera extrinsic parameters.

[0099] In another possible implementation of step 150, Figure 8 The fifth flowchart of the vehicle camera external parameter calibration method provided in the first embodiment of the present invention is as follows: Figure 8 As shown, specifically including:

[0100] Step 1506, selecting images at times corresponding to multiple extreme points as calibration images, establishing a coordinate mapping equation group for each calibration image, and generating final vehicle camera external parameters based on the multiple coordinate mapping equation groups.

[0101] In a possible implementation of step 1506, first, the vehicle camera extrinsic parameters at the time corresponding to each extreme point are calculated according to each coordinate mapping equation group to generate the camera extrinsic parameters at the key moment; secondly, the mean values ​​of the camera extrinsic parameters at multiple key moments are calculated to generate the final vehicle camera extrinsic parameters.

[0102] In another possible implementation of step 1506, the optimal extrinsic parameters are solved by least square method by combining multiple coordinate mapping equations.

[0103] Embodiment 2

[0104] A second embodiment of the present invention provides an extrinsic parameter calibration device for a vehicle-mounted camera. Fig. 9 This is a schematic diagram of the structure of an extrinsic parameter calibration device for a vehicle-mounted camera provided in Embodiment 2 of the present invention. The extrinsic parameter calibration device 200 for a vehicle-mounted camera includes a first acquisition module 201, a second acquisition module 202, a drawing module 203, a third acquisition module 204 and a data processing module 205.

[0105] The first acquisition module 201 is used to acquire an image of a vehicle camera when the vehicle is traveling in a curve between two parallel lines. The image includes two parallel lines, which are respectively recorded as a first parallel line and a second parallel line.

[0106] The two parallel lines are located on a flat and straight road surface, and the curve is an S-shaped curve.

[0107] The second acquisition module 202 is used to acquire the horizontal pixel coordinates of points on parallel lines at a preset height of the image.

[0108] Among them, obtaining the horizontal pixel coordinates of points on parallel lines at a preset height of the image specifically includes: obtaining the pixel coordinates of two parallel lines in the image at each moment through an image detection algorithm; obtaining the horizontal pixel coordinates of points on the first parallel line and / or the second parallel line at the preset height of the image, and recording the moment corresponding to the horizontal pixel coordinates.

[0109] The drawing module 203 is used to draw a trend graph of the horizontal pixel coordinates of points on parallel lines at the same preset height changing with time.

[0110] The third acquisition module 204 is used to acquire the extreme point of the trend graph, and select the image at the time corresponding to the extreme point as the calibration image.

[0111] Among them, obtaining the extreme points of the trend graph, and selecting the image at the time corresponding to the extreme points as the calibration image, specifically includes: if the trend graph includes a first trend curve and a second trend curve, comparing the change trends of the first trend curve and the second trend curve; wherein the first trend curve is a trend line of the change of the horizontal pixel coordinates of the points on the first parallel line at a preset height of the image over time, and the second trend curve is a trend line of the change of the horizontal pixel coordinates of the points on the second parallel line at a preset height of the image over time; if the change trends are synchronized, obtaining the extreme points of the trend graph, and selecting the image at the time corresponding to the extreme points as the calibration image.

[0112] Among them, the extreme points of the trend graph are obtained, and the image at the time corresponding to the extreme points is selected as the calibration image, which specifically also includes: if the preset height is not less than 2 preset heights, the extreme points of the trend graph at each preset height are obtained; the mean time of the times corresponding to the synchronous extreme points at all preset heights is calculated; and the image at the mean time is selected as the calibration image.

[0113] The data processing module 205 is used to calculate and generate the extrinsic parameters of the vehicle-mounted camera according to the intrinsic parameters of the vehicle-mounted camera, the world coordinates of the points on two parallel lines in the calibration image, and the corresponding pixel coordinates.

[0114] Among them, the external parameters of the vehicle camera are calculated and generated according to the intrinsic parameters of the vehicle camera, the world coordinates of the points on two parallel lines in the calibration image, and the corresponding pixel coordinates, specifically including: taking the ground as the XY plane of the world coordinate system, and the first parallel line or the second parallel line as the X-axis; taking two different calibration points on the first parallel line and the second parallel line in a frame of calibration image, respectively, and recording the pixel coordinates of the four calibration points; obtaining the parallel line distance between the first parallel line and the second parallel line in the world coordinate system; generating the world coordinates of the four calibration points according to the parallel line distance; substituting the intrinsic parameters of the vehicle camera, the world coordinates and the pixel coordinates of the four calibration points into the matrix equation that converts the pixel coordinates into the world coordinates through the homography matrix, and solving the Y-axis translation parameter and three rotation angle parameters of the external parameters of the vehicle camera.

[0115] Among them, the external parameters of the vehicle camera are calculated based on the internal parameters of the vehicle camera, the world coordinates of the points on two parallel lines in the calibration image and the corresponding pixel coordinates. Specifically, it also includes: selecting the images at the time corresponding to multiple extreme points as calibration images, establishing a coordinate mapping equation group for each calibration image, and generating the final vehicle camera external parameters based on multiple coordinate mapping equation groups.

[0116] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the data processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0117] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0118] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0119] Embodiment 3

[0120] Embodiment 3 provides a computer program product including instructions, such as Fig.10 As shown, when the computer program product runs on a computer, the computer executes any one of the vehicle-mounted camera extrinsic parameter calibration methods provided in Example 1.

[0121] Embodiment 4

[0122] Embodiment 4 of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any one of the vehicle-mounted camera extrinsic parameter calibration methods provided in Embodiment 1.

[0123] Embodiment 5

[0124] A fifth embodiment of the present invention provides a computer device, Fig.11 This is a schematic diagram of the structure of a computer device provided in Embodiment 5 of the present invention. Fig.11 As shown, it includes a memory 300, a processor 400 and a computer program stored in the memory, and the processor executes the computer program to implement any one of the vehicle-mounted camera extrinsic parameter calibration methods provided in the above-mentioned embodiment 1.

[0125] Embodiment 6

[0126] A sixth embodiment of the present invention provides a mobile tool, including the computer device of the fifth embodiment described above.

[0127] The mobile tools referred to in this application may be those with tool functions, such as excavators, bulldozers, forklifts, rollers, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, sprinkler trucks, sweeping robot lawn mowers, etc.

[0128] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0129] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0130] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for calibrating external parameters of a vehicle-mounted camera, characterized in that: The method comprises: Acquire an image of the vehicle camera when the vehicle is traveling in a curve between two parallel lines, wherein the image includes the two parallel lines, which are respectively recorded as a first parallel line and a second parallel line; Obtaining horizontal pixel coordinates of points on parallel lines at a preset height of the image; Draw a trend graph of the horizontal pixel coordinates of points on parallel lines at the same preset height changing with time; Obtaining the extreme point of the trend graph, and selecting the image at the time corresponding to the extreme point as the calibration image; The vehicle camera external parameters are calculated and generated according to the vehicle camera internal parameters, the world coordinates of the points on the two parallel lines in the calibration image and the corresponding pixel coordinates.

2. The vehicle-mounted camera extrinsic parameter calibration method according to claim 1, characterized in that: The step of obtaining the horizontal pixel coordinates of points on a parallel line at a preset height of the image specifically includes: Obtain pixel coordinates of two parallel lines in the image at each moment by using an image detection algorithm; Acquire the horizontal pixel coordinates of points on the first parallel line and / or the second parallel line at a preset height of the image, and record the time corresponding to the horizontal pixel coordinates.

3. The vehicle-mounted camera extrinsic parameter calibration method according to claim 1, characterized in that: The step of obtaining the extreme point of the trend graph and selecting the image at the time corresponding to the extreme point as the calibration image specifically includes: If the trend graph includes a first trend curve and a second trend curve, compare the change trends of the first trend curve and the second trend curve; wherein the first trend curve is a trend line of changes in the horizontal pixel coordinates of points on the first parallel line at a preset height of the image over time, and the second trend curve is a trend line of changes in the horizontal pixel coordinates of points on the second parallel line at a preset height of the image over time; If the changing trends are synchronized, the extreme points of the trend graph are obtained, and the image at the time corresponding to the extreme points is selected as the calibration image.

4. The vehicle-mounted camera extrinsic parameter calibration method according to claim 1, characterized in that: The step of obtaining the extreme point of the trend graph and selecting the image at the time corresponding to the extreme point as the calibration image specifically includes: If the preset height is not less than 2 preset heights, obtain the extreme value point of the trend graph at each preset height; Calculate the mean time of the time corresponding to the synchronous extreme points at all preset heights; The image at the mean time is selected as the calibration image.

5. The vehicle-mounted camera extrinsic parameter calibration method according to claim 1, characterized in that: The vehicle camera extrinsic parameters are calculated and generated according to the vehicle camera intrinsic parameters, the world coordinates of the points on the two parallel lines in the calibration image, and the corresponding pixel coordinates, specifically including: The ground is taken as the XY plane of the world coordinate system, and the first parallel line or the second parallel line is taken as the X axis; Select two different calibration points on the first parallel line and the second parallel line in a frame of the calibration image, respectively, and record the pixel coordinates of the four calibration points; Get the parallel line distance between the first parallel line and the second parallel line in the world coordinate system; Generating the world coordinates of the four calibration points according to the parallel line distances; The intrinsic parameters of the vehicle camera, the world coordinates and pixel coordinates of the four calibration points are substituted into the matrix equation that converts the pixel coordinates into the world coordinates through the homography matrix, and the Y-axis translation parameters and three rotation angle parameters of the extrinsic parameters of the vehicle camera are solved.

6. The vehicle-mounted camera extrinsic parameter calibration method according to claim 1, characterized in that: The two parallel lines are located on a flat and straight road surface, and the curve is an S-shaped curve.

7. The vehicle-mounted camera extrinsic parameter calibration method according to claim 1, characterized in that: The step of calculating and generating the vehicle camera external parameters according to the vehicle camera internal parameters, the world coordinates of the points on the two parallel lines in the calibration image, and the corresponding pixel coordinates specifically includes: The images at the time corresponding to multiple extreme points are selected as calibration images, a coordinate mapping equation group is established for each calibration image, and the final vehicle camera external parameters are generated according to the multiple coordinate mapping equation groups.

8. A vehicle-mounted camera external parameter calibration device, characterized in that: The device comprises: A first acquisition module is used to acquire an image of the vehicle camera when the vehicle is traveling in a curve between two parallel lines, wherein the image includes the two parallel lines, which are respectively recorded as a first parallel line and a second parallel line; A second acquisition module, used to acquire the horizontal pixel coordinates of points on parallel lines at a preset height of the image; A drawing module, used for drawing a trend graph of the horizontal pixel coordinates of points on parallel lines at the same preset height changing with time; A third acquisition module is used to obtain the extreme point of the trend graph, and select the image at the time corresponding to the extreme point as the calibration image; The data processing module is used to calculate and generate the external parameters of the vehicle-mounted camera according to the internal parameters of the vehicle-mounted camera, the world coordinates of the points on the two parallel lines in the calibration image and the corresponding pixel coordinates.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claims 1-7.

11. A mobile tool, characterized in that: Comprising the computer device as claimed in claim 10.

Images