Calibration method and related devices for external parameters of vehicle-mounted cameras

Abstract

The present application discloses a calibration method and related device for the external parameters of a vehicle-mounted camera in the field of camera calibration technology. In the technical solution provided by the present application, the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera is determined according to the coordinates of the M first parallel lines in the acquired first image in the actual camera coordinate system of the vehicle-mounted camera and the constraints of the M second parallel lines in the bird's-eye view of the first image, where M is an integer greater than or equal to 3, and the M second parallel lines correspond one-to-one to the M first parallel lines; the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera and the transformation relationship between the virtual ideal camera coordinate system and the world coordinate system is determined according to the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system, thereby improving the flexibility of the external parameter calibration method of the vehicle-mounted camera.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to a calibration method and related device for external parameters of a vehicle-mounted camera. Background Art

[0002] As a sensor, the role of vehicle-mounted cameras in assisted driving and autonomous driving is becoming increasingly important. By associating the environment around the vehicle with the digital images taken by the vehicle-mounted camera, it can provide necessary information for safe driving. In the process of associating the environment around the vehicle with the digital images taken by the vehicle-mounted camera, the external parameters of the vehicle-mounted camera play an important role. Among them, the external parameters of the vehicle-mounted camera refer to the translation distance and rotation angle of the vehicle-mounted camera relative to the vehicle.

[0003] At present, the external parameters of the vehicle-mounted camera are usually calibrated with the help of parallel lines. The parallel lines used for the calibration of the external parameters of the vehicle-mounted camera can be lane lines drawn in a specific venue or lane lines on the road. During the calibration process, many existing technical solutions are set with specific constraints. For example, it is required that the distances between the three parallel lines used for the calibration of the external parameters of the vehicle-mounted camera be equal. Specifically, the pixel coordinate system coordinates of the three parallel lines are obtained, the pixel coordinate system coordinates of the three parallel lines are converted into camera coordinate system coordinates, and the equal distance between the three parallel lines is used as a constraint condition to obtain the external parameters of the vehicle-mounted camera. However, the use of equidistant parallel lines for the calibration of the vehicle-mounted camera makes the calibration method of the vehicle-mounted camera less flexible.

[0004] Therefore, how to improve the flexibility of the extrinsic calibration method of vehicle-mounted cameras has become an urgent problem to be solved. Summary of the invention

[0005] The present application provides a method and related device for calibrating the external parameters of a vehicle-mounted camera, which improves the flexibility of the external parameter calibration method of the vehicle-mounted camera.

[0006] In a first aspect, the present application provides a method for calibrating the external parameters of a vehicle-mounted camera, the method comprising: obtaining the coordinates of M first parallel lines in a first image in an actual camera coordinate system of the vehicle-mounted camera, the first image being an image taken by a vehicle to which the vehicle-mounted camera belongs during driving on a target road, the target road containing N parallel lane lines, the distance between any two of the N lane lines being known, the M first parallel lines corresponding one-to-one to the M first lane lines of the N lane lines, M being an integer greater than or equal to 3, and N being an integer greater than or equal to M; determining a transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and a virtual ideal camera coordinate system of the vehicle-mounted camera according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the constraints of the M second parallel lines in the bird's-eye view of the first image, the M second parallel lines corresponding one-to-one to the M first parallel lines, and the M second parallel lines corresponding one-to-one to the M first parallel lines. The constraints of the second parallel lines in the bird's-eye view include parallel constraints, vertical constraints, spacing ratio constraints and spacing constraints. The parallel constraint includes that the M second parallel lines are parallel, the vertical constraint includes that any one of the M second parallel lines is perpendicular to the x-axis direction of the virtual ideal camera coordinate system, the spacing ratio constraint includes that the distance ratio between any two of the M second parallel lines is the same as the distance ratio between the M lane lines corresponding to the any two second parallel lines, and the distance constraint includes that the difference between the distance between the any two second parallel lines and the distance between the M lane lines corresponding to the any two second parallel lines is minimum; according to the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera and the transformation relationship between the virtual ideal camera coordinate system and the world coordinate system, the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is determined.

[0007] In the method, first, according to the coordinates of the M first parallel lines in the acquired first image in the actual camera coordinate system of the vehicle-mounted camera and the constraints of the M second parallel lines in the bird's-eye view of the first image, the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera is determined, where M is an integer greater than or equal to 3, the M second parallel lines correspond one-to-one to the M first parallel lines, and the constraints of the M second parallel lines in the bird's-eye view include parallel constraints, vertical constraints, spacing ratio constraints, and spacing constraints; then, according to the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera and the transformation relationship between the virtual ideal camera coordinate system and the world coordinate system, the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is determined, where the first image The image is an image taken by the vehicle to which the vehicle camera belongs during driving on the target road. The target road contains N parallel lane lines, where N is an integer greater than or equal to M. By introducing the virtual ideal camera coordinate system of the vehicle-mounted camera as the transit coordinate system, it is only necessary to know the distance between any two adjacent lane lines among the N lane lines. The transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera can be determined according to the coordinates of the M first parallel lines in the first image in the actual camera coordinate system of the vehicle-mounted camera and the parallel constraints, vertical constraints, spacing ratio constraints and spacing constraints of the M second parallel lines in the bird's-eye view of the first image. There is no need to set other specific constraints for the N lane lines in the target road, thereby improving the flexibility of the external parameter calibration method of the vehicle-mounted camera.

[0008] In a possible implementation, the determining of the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the constraints of the M second parallel lines in the bird's-eye view of the first image includes: determining the x-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the parallel constraint; determining the y-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the vertical constraint; determining the z-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the spacing ratio constraint; and determining the z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the spacing constraint.

[0009] In this implementation, the x-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the parallel constraints; the y-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the vertical constraints; the z-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the spacing ratio constraints; the z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the spacing constraints, and the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera is obtained, thereby improving the accuracy of the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera.

[0010] In a possible implementation, the parallel constraint further includes: 0.5∑(θ i -θ i+1 ) 2 minimum, where θ i is the angle between the i-th second parallel line among the M second parallel lines and the x-axis direction of the virtual ideal camera coordinate system, θ i+1 is the angle between the i+1th second parallel line among the M second parallel lines and the x-axis direction of the virtual ideal camera coordinate system, i is an integer greater than or equal to 1 and less than M, θ i Less than or equal to 90 degrees, θ i+1 Less than or equal to 90 degrees.

[0011] In this implementation, 0.5∑(θ i -θ i+1 ) 2 The minimum x-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined as a parallel constraint, thereby improving the accuracy of the x-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system.

[0012] In a possible implementation, the vertical constraint further includes: 0.5∑(θ j -90) 2 Minimum, θ j is the angle between the jth second parallel line among the M second parallel lines and the x-axis direction of the virtual ideal camera coordinate system, j is an integer greater than or equal to 1 and less than or equal to M, θ j Less than or equal to 90 degrees.

[0013] In this implementation, 0.5∑(θ j -90)2 The minimum y-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined as a vertical constraint, thereby improving the accuracy of the y-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system.

[0014] In a possible implementation, the spacing ratio constraint further includes: minimum, where ω s，s+1 is the distance between the sth second parallel line and the s+1th second parallel line among the M second parallel lines, ω s+1，s+2 is the distance between the s+1th second parallel line and the s+2th second parallel line among the M second parallel lines, W s，s+1 is the actual distance between the lane line corresponding to the sth second parallel line and the lane line corresponding to the s+1th second parallel line among the M second parallel lines, W s+1，s+2 is the actual distance between the lane line corresponding to the s+1th second parallel line and the lane line corresponding to the s+2th second parallel line among the M second parallel lines, where s is an integer greater than or equal to 1 and less than M-1.

[0015] In this implementation, The z-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined as a minimum spacing ratio constraint, thereby improving the accuracy of the z-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system.

[0016] In a possible implementation, the spacing constraint further includes 0.5∑(ω i，i+1 -W i，i+1 ) 2 Minimum, ω i，i+1 is the distance between the i-th second parallel line and the i+1-th second parallel line among the M second parallel lines, W i，i+1 is the actual distance between the lane line corresponding to the i-th second parallel line and the lane line corresponding to the (i+1)-th second parallel line among the M second parallel lines, where i is an integer greater than or equal to 1 and less than M.

[0017] In this implementation, 0.5∑(ω i，i+1 -W i，i+1 ) 2 The minimum z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined as a spacing constraint, thereby improving the accuracy of the z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system.

[0018] In a possible implementation, the method also includes: obtaining the y-axis translation of the vehicle-mounted camera in the world coordinate system; obtaining the vehicle positioning information, the vehicle positioning information including the current position information of the vehicle and / or the acceleration of the vehicle and / or the wheel speed of the vehicle; establishing an observation model of the vehicle based on the vehicle positioning information and the y-axis translation of the vehicle-mounted camera in the world coordinate system; obtaining the heading angle of the vehicle at the current moment based on the observation model of the vehicle; compensating the heading angle of the vehicle at the current moment based on the heading angle in the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system to obtain a compensated heading angle; and updating the heading angle in the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system to the compensated heading angle.

[0019] In this implementation, the vehicle's current heading angle is obtained based on an observation model of the vehicle established by the vehicle's positioning information and the y-axis translation of the vehicle-mounted camera in the world coordinate system, and the vehicle's current heading angle is compensated using the heading angle in the external parameters of the vehicle-mounted camera in the world coordinate system to obtain the compensated heading angle, and the heading angle in the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is updated to the compensated heading angle, so that when the vehicle is not traveling completely parallel to the lane line, the influence of the vehicle's heading angle can be compensated, thereby improving the accuracy of the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system.

[0020] In a second aspect, the present application provides a device for calibrating external parameters of a vehicle-mounted camera, which may include modules for implementing the method in the first aspect, and these modules may be implemented in software and / or hardware.

[0021] In a third aspect, the present application provides a device for calibrating external parameters of a vehicle-mounted camera. The device may include a processor coupled to a memory. The memory is used to store program code, and the processor is used to execute the program code in the memory to implement the method in the first aspect or any one of the implementations thereof.

[0022] Optionally, the device may further include the memory.

[0023] In a fourth aspect, the present application provides a chip comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation method thereof.

[0024] In a fifth aspect, the present application provides a computer-readable medium storing a program code for execution by a device, wherein the program code includes a method for executing the method described in the first aspect or any possible implementation thereof.

[0025] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any possible implementation thereof.

[0026] In a seventh aspect, the present application provides a computing device comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, the communication interface communicates with a target system, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation method thereof.

[0027] In an eighth aspect, the present application provides a computing system comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, the communication interface communicates with a target system, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation method thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of a pixel coordinate system;

[0029] Figure 2 is a schematic diagram of a camera coordinate system;

[0030] Figure 3 A schematic diagram of an actual camera coordinate system and a virtual ideal camera coordinate system according to an embodiment of the present application;

[0031] Figure 4 A schematic diagram of a world coordinate system according to an embodiment of the present application;

[0032] Figure 5 A schematic diagram of an application scenario of an embodiment of the present application;

[0033] Figure 6 A schematic diagram of a flow chart of a method for calibrating external parameters of a vehicle-mounted camera according to an embodiment of the present application;

[0034] Figure 7 A schematic diagram of a first image according to an embodiment of the present application;

[0035] Figure 8 A schematic diagram of a bird's-eye view of an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of a heading angle compensation according to an embodiment of the present application;

[0037] Fig.10 A schematic diagram of a flow chart of another method for calibrating external parameters of a vehicle-mounted camera according to an embodiment of the present application;

[0038] Fig.11 A schematic structural diagram of a device for calibrating external parameters of a vehicle-mounted camera according to an embodiment of the present application;

[0039] Fig.12 This is a schematic structural diagram of a device for calibrating external parameters of a vehicle-mounted camera according to another embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] Figure 1 is a schematic diagram of a pixel coordinate system. Figure 1 As shown, the vertex of the upper left corner of the pixel coordinate system is the origin O p , the horizontal rightward direction is the u-axis, and the vertical downward direction is the v-axis.

[0042] Pixel coordinates refer to the position of a pixel in an image. In the pixel coordinate system, the coordinates of any pixel point can be expressed as (u i ,v i ). The pixel representation method cannot reflect the physical size of objects in the image.

[0043] Figure 2 is a schematic diagram of a camera coordinate system. Figure 2 As shown, the camera coordinate system uses the optical axis of the camera as the Z c Axis, the center of the light in the camera optical system is the origin O c , which is actually the center of the lens. The horizontal axis X of the camera coordinate system c With vertical axis Y c They are parallel to the u-axis and v-axis of the pixel coordinate system respectively.

[0044] Figure 3 FIG. 1 is a schematic diagram of an actual camera coordinate system and a virtual ideal camera coordinate system according to an embodiment of the present application. Figure 3 As shown in the figure, L1, L2 and L3 are three parallel lines located on the road surface where the car is traveling. Figure 2The origin, x-axis, y-axis and z-axis of the camera coordinate system are selected according to the selection rules and positional relationships. The center of the camera lens is selected as the origin of the actual camera coordinate system. r ; Select the camera's optical axis as the Z coordinate system of the actual camera r , Z r Parallel to the road surface, the front of the camera is Z r The positive direction of Z r The direction perpendicular to the road surface on which the car is traveling is the X direction of the actual camera coordinate system. r , select the direction from L2 to L3 as X r The positive direction is the direction perpendicular to the road surface on which the car is traveling as Y r (not shown in the figure), for example, the direction perpendicular to the road surface on which the car is traveling is selected as Y r positive direction.

[0045] The origin of the virtual ideal camera coordinate system and the origin of the actual camera coordinate system O r Coincident, the y-axis of the virtual ideal camera coordinate system and the Y r Overlap and have the same direction. r and X r Around Y r Rotate by angle β until X r When perpendicular to the three parallel lines on the road on which the car is traveling, the z-axis direction and the x-axis direction of the virtual ideal camera coordinate system are obtained.

[0046] Figure 4 FIG. 1 is a schematic diagram of a world coordinate system according to an embodiment of the present application. Figure 4 As shown in the figure, L1, L2 and L3 are three parallel lines located on the road surface where the car is driving, ZOX is the virtual ideal camera coordinate system of the camera, and any point on L2 is selected as the origin of the world coordinate system O w , the X of the world coordinate system w The direction is consistent with the Z-axis direction of the virtual ideal camera coordinate system of the camera, and the Y-axis direction of the world coordinate system w It is perpendicular to L2 and parallel to the road surface where the car is traveling. The direction from L2 to L1 is selected as Y. w positive direction.

[0047] Figure 5 A schematic diagram of an application scenario of an embodiment of the present application. Figure 5The scene shown is a scene in which the external parameters of the camera are calibrated using three parallel lines L1, L2 and L3 with known spacing on the road surface on which the vehicle is traveling. The external parameters of the camera are calibrated by driving the vehicle from one end of the three parallel lines to the other end. The three parallel lines L1, L2 and L3 with known spacing on the road surface can be lane lines drawn in a specific venue or lane lines on a normal road.

[0048] Understandably, Figure 5 The scenario shown is only an example, and the technical solution of the present application can also be applied to other scenarios as long as the scenario involves calibration of the external parameters of the camera. For example, the technical solution of the present application can also be applied to the calibration of the external parameters of the camera in the intelligent robot and other scenarios.

[0049] Figure 6 FIG. 1 is a flow chart of a method for calibrating the external parameters of a vehicle-mounted camera according to an embodiment of the present application. Figure 6 As shown, the method at least includes S601 to S603.

[0050] S601, obtaining coordinates of M first parallel lines in a first image in an actual camera coordinate system of the vehicle-mounted camera, where the first image is an image captured by a vehicle to which the vehicle-mounted camera belongs during its driving on a target road, where the target road includes N parallel lane lines, where a distance between any two of the N lane lines is known, and where the M first parallel lines correspond one-to-one to the M first lane lines of the N lane lines, where M is an integer greater than or equal to 3, and where N is an integer greater than or equal to M.

[0051] In a possible implementation, when a vehicle to which the on-board camera belongs is traveling on a target road, the on-board camera photographs M lane lines among N lane lines on the target road to obtain a first image. The first image includes M first parallel lines, which are obtained by photographing M lane lines among N lane lines on the target road by the on-board camera, and the M first parallel lines in the first image correspond one-to-one to M lane lines among N lane lines on the target road.

[0052] For example, Figure 7 FIG. 1 is a schematic diagram of a first image according to an embodiment of the present application. Figure 7 As shown, there are three first parallel lines L1, L2 and L3 in the first image, which respectively correspond to three lane lines among the N lane lines in the target road. At this time, M is equal to 3, and N is greater than or equal to 3.

[0053] Extract M first parallel lines from the first image, obtain the coordinates of the M first parallel lines in the pixel coordinate system, and convert the coordinates of the M parallel lines in the pixel coordinate system into the coordinates of the M parallel lines in the camera coordinate system of the vehicle-mounted camera. An example of the pixel coordinate system may be Figure 1 Another example of a camera coordinate system can be Figure 3 The camera coordinate system of the actual camera is shown.

[0054] As an example, the parallel line region and the contour in the first image are extracted by a segmentation algorithm, and then edge extraction is performed at a sub-pixel level to obtain M first parallel lines. For example, the segmentation algorithm includes a watershed algorithm and the like.

[0055] As another example, the straight lines in the first image are extracted by Hough transform, and then clustering and filtering methods are used to obtain M first parallel lines in the first image. Among them, Hough transform is a feature extraction technology in image processing, which can detect objects with specific shapes through voting algorithms; the clustering method clusters straight line segments with similar slopes and intercepts; the filtering method sets the area of ​​interest for screening by the camera reference installation angle and position.

[0056] S602, determining the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the constraints of the M second parallel lines in the bird's-eye view of the first image, the M second parallel lines corresponding one-to-one to the M first parallel lines, and the constraints of the M second parallel lines in the bird's-eye view include parallel constraints, vertical constraints, spacing ratio constraints and spacing constraints.

[0057] The transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera is the x-axis rotation angle, y-axis rotation angle, z-axis rotation angle and z-axis translation distance of the virtual ideal camera coordinate system of the vehicle-mounted camera relative to the actual camera coordinate system of the vehicle-mounted camera. Among them, an example of the actual camera coordinate system of the vehicle-mounted camera can be Figure 3 An example of the actual camera coordinate system shown in FIG. 1 and the virtual ideal camera coordinate system of the vehicle-mounted camera can be Figure 3 The virtual ideal camera coordinate system shown.

[0058] In a possible implementation, according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera, an inverse perspective transformation method is used to obtain a bird's-eye view of the M first parallel lines. There are M second parallel lines in the bird's-eye view, and the M second parallel lines correspond one-to-one to the M first parallel lines.

[0059] For example, Figure 8Schematic diagram of a bird's-eye view of an embodiment of the present application. Figure 8 As shown, there are three second parallel lines in the bird's-eye view, M is equal to 3, namely L1, L2 and L3, where θ1 is the angle between the second parallel line L1 and the x-axis direction of the virtual ideal coordinate system, θ2 is the angle between the second parallel line L2 and the x-axis direction of the virtual ideal coordinate system, θ3 is the angle between the second parallel line L3 and the x-axis direction of the virtual ideal coordinate system, ω 12 is the distance between the second parallel line L1 and the second parallel line L2, ω 23 is the distance between the second parallel line L2 and the second parallel line L3.

[0060] The parallel constraint includes that the M second parallel lines are parallel, the vertical constraint includes that any one of the M second parallel lines is perpendicular to the x-axis direction of the virtual ideal camera coordinate system, the spacing ratio constraint includes that the distance ratio between any two of the M second parallel lines is the same as the distance ratio between the M lane lines corresponding to any two second parallel lines, and the distance constraint includes that the difference between the distance between any two second parallel lines and the distance between the M lane lines corresponding to the any two second parallel lines is minimized.

[0061] In a possible implementation, the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera is optimized in sequence by optimizing the x-axis rotation angle, y-axis rotation angle, z-axis rotation angle and z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system.

[0062] According to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the parallel constraints, the x-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined by optimization or other methods.

[0063] Exemplarily, the parallel constraint includes making the loss function 0.5∑(θ i -θ i+1 ) 2 Minimum, θ i is the angle between the i-th second parallel line among the M second parallel lines and the x-axis direction of the virtual ideal camera coordinate system, θ i+1 is the angle between the i+1th second parallel line among the M second parallel lines and the x-axis direction of the virtual ideal camera coordinate system, i is an integer greater than or equal to 1 and less than M, θ i Less than or equal to 90 degrees, θ i+1 Less than or equal to 90 degrees.

[0064] According to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the vertical constraint, the y-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined by optimization or other methods.

[0065] Exemplarily, the vertical constraint includes making the loss function 0.5∑(θ j -90) 2 Minimum, θ j is the angle between the jth second parallel line among the M second parallel lines and the x-axis direction of the virtual ideal camera coordinate system, j is an integer greater than or equal to 1 and less than or equal to M, θ j Less than or equal to 90 degrees.

[0066] According to the coordinates and spacing ratio constraints of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera, the z-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined by optimization and other methods.

[0067] Exemplarily, the spacing ratio constraint includes making the loss function minimum, where ω s，s+1 is the distance between the sth second parallel line and the s+1th second parallel line among the M second parallel lines, ω s+1，s+2 is the distance between the s+1th second parallel line and the s+2th second parallel line among the M second parallel lines, W s，s+1 is the actual distance between the lane line corresponding to the sth second parallel line and the lane line corresponding to the s+1th second parallel line among the M second parallel lines, W s+1，s+2 is the actual distance between the lane line corresponding to the s+1th second parallel line and the lane line corresponding to the s+2th second parallel line among the M second parallel lines, where s is an integer greater than or equal to 1 and less than M-1.

[0068] According to the coordinates and spacing constraints of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera, the z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined by optimization and other methods.

[0069] Exemplarily, the spacing constraint also includes 0.5∑(ω i，i+1 -W i，i+1 ) 2 Minimum, ω i，i+1 is the distance between the i-th second parallel line and the i+1-th second parallel line among the M second parallel lines, W i，i+1 is the actual distance between the lane line corresponding to the i-th second parallel line and the lane line corresponding to the (i+1)-th second parallel line among the M second parallel lines, where i is an integer greater than or equal to 1 and less than M.

[0070] The optimized order of the x-axis rotation angle, y-axis rotation angle, z-axis rotation angle and z-axis translation distance of the virtual ideal camera coordinate system provided by this implementation relative to the actual camera coordinate system can avoid the error introduced by the coupling of Euler angles and improve the accuracy of the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera.

[0071] In another possible implementation, the constraints of the M second parallel lines in the bird's-eye view include making the loss function α∑(θ n -90) 2 +β∑(ω m,m+1 -W m,m+1 ) 2 minimum, where α and β are weight parameters, θ n is the angle between the nth second parallel line among the M second parallel lines and the x-axis direction of the virtual ideal camera coordinate system, ω m，m+1 is the distance between the mth second parallel line and the m+1th second parallel line among the M second parallel lines, W m，m+1 is the actual distance between the lane line corresponding to the m-th second parallel line and the lane line corresponding to the (m+1)-th second parallel line among the M second parallel lines, n is an integer greater than or equal to 1 and less than or equal to M, and m is an integer greater than or equal to 1 and less than M.

[0072] By adopting the constraint method provided by this implementation, the x-axis rotation angle, y-axis rotation angle, z-axis rotation angle and z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system can be obtained in combination with optimization and other methods. However, due to improper selection of initial values, only the local optimization value of the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera is obtained, rather than the global optimal value.

[0073] S603, determining a transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system according to a transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera and a transformation relationship between the virtual ideal camera coordinate system and the world coordinate system.

[0074] The transformation relationship between the virtual ideal camera coordinate system and the world coordinate system is the x-axis rotation angle, y-axis rotation angle, z-axis rotation angle and z-axis translation distance of the world coordinate system relative to the virtual ideal camera coordinate system of the vehicle-mounted camera. The transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is the external parameters of the vehicle-mounted camera, including the x-axis rotation angle, y-axis rotation angle, z-axis rotation angle and z-axis translation distance of the world coordinate system relative to the actual camera coordinate system of the vehicle-mounted camera, which can also be called roll angle (roll), pitch angle (pitch), yaw angle (yaw) and translation distance. Among them, an example of the actual camera coordinate system of the vehicle-mounted camera can be Figure 3 An example of the actual camera coordinate system of the vehicle-mounted camera shown in FIG. 1 and the virtual ideal camera coordinate system of the vehicle-mounted camera can be Figure 3 or Figure 4 The virtual ideal camera coordinate system of the vehicle-mounted camera shown, an example of the world coordinate system can be Figure 4 The world coordinate system shown.

[0075] In one possible implementation, by transforming the matrix The transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera is converted into the transformation relationship between the virtual ideal camera coordinate system and the world coordinate system, and the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is obtained, that is, the external parameters of the vehicle-mounted camera.

[0076] After obtaining the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system, the y-axis translation of the vehicle-mounted camera in the world coordinate system is obtained; and the positioning information of the vehicle is obtained from the sensor of the vehicle to which the vehicle-mounted camera belongs, wherein the sensor of the vehicle to which the vehicle-mounted camera belongs includes a wheel speed meter, an inertial measurement unit (IMU) and a global positioning system (GPS), etc. Correspondingly, the positioning information of the vehicle includes the current position information of the vehicle and / or the acceleration of the vehicle and / or the wheel speed of the vehicle, etc.

[0077] An observation model of the vehicle is established according to the positioning information of the vehicle and the y-axis translation of the vehicle-mounted camera in the world coordinate system; and the heading angle of the vehicle at the current moment is obtained according to the observation model of the vehicle; and the heading angle of the vehicle at the current moment is compensated according to the heading angle in the external parameters of the vehicle-mounted camera to obtain the compensated heading angle; and the heading angle in the external parameters of the vehicle-mounted camera is updated to the compensated heading angle.

[0078] For example, Fig. 9 A schematic diagram of a heading angle compensation in an embodiment of the present application is shown in FIG. Fig. 9As shown, the vehicle heading angle is the heading angle of the vehicle at the current moment obtained according to the observation model of the vehicle, and the camera heading angle is the heading angle in the external parameters of the determined on-board camera. The camera heading angle is subtracted from the vehicle heading angle to obtain the heading angle from the camera to the vehicle, which can also be called the compensated heading angle, and the determined heading angle in the on-board camera is updated to the compensated heading angle, thereby compensating for the influence of the vehicle heading angle obtained by vehicle positioning.

[0079] In another possible implementation, multiple external parameters of the vehicle-mounted camera after completing heading angle compensation are cached, the multiple external parameters of the vehicle-mounted camera after completing heading angle compensation are filtered to obtain the filtered external parameters of the vehicle-mounted camera, and the external parameters of the vehicle-mounted camera are updated to the filtered external parameters of the vehicle-mounted camera.

[0080] Exemplarily, multiple external parameters of the vehicle-mounted camera after the heading angle compensation are cached, and periodic optimal estimation is performed. If the cached external parameters of the vehicle-mounted camera after the heading angle compensation do not exceed the preset threshold when the vehicle drives out of the calibration site, the cached external parameters of the vehicle-mounted camera after the heading angle compensation are used for optimal estimation and calibration is completed; if the cached external parameters of the vehicle-mounted camera after the heading angle compensation exceed the preset threshold when the vehicle drives out of the calibration site, the external parameters of the vehicle-mounted camera after the heading angle compensation cached at intervals of a certain number (such as 10 frames) are selected for optimal estimation. The calibration site is the area where N lane lines in the target road for external parameter calibration of the vehicle-mounted camera are located.

[0081] Preferably, a set of data is cached for each type of external parameters of the external parameters of the vehicle-mounted camera after the heading angle compensation is completed, that is, a set of data is cached for the roll angle, pitch angle, heading angle and translation distance of the external parameters of the vehicle-mounted camera after the heading angle compensation is completed, and the kernel density estimation method is used to obtain the external parameter value with the highest probability in each set of data, which is the optimal estimate of this type of external parameter.

[0082] Specifically, the kernel density estimation method is to fit a Gaussian function to the statistical histogram of the data, and find a Gaussian function that best fits the histogram of the known data and can resist a certain amount of noise interference. For each type of external parameter, a set of external parameter values ​​corresponding to each type is used to estimate the fitted kernel function (preferably the Gaussian kernel function), find the peak value of the kernel function, and the corresponding horizontal axis value (where the probability of the Gaussian function is the maximum) is the optimal value of each type of external parameter.

[0083] Optionally, a four-dimensional kernel density estimation method can be used to simultaneously estimate the optimal value of each type of external parameter in four types of external parameters.

[0084] Optionally, a group of data is cached for each type of external parameter, and a median filtering method is used to obtain the median of each type of external parameter, which is the optimal value of each type of external parameter.

[0085] Optionally, a group of data is cached for each type of external parameter, and a mean filtering method is used to calculate the average value of each group of data. The mean value of each group of data obtained is the optimal value of each type of external parameter.

[0086] When the standard deviation of the optimal value of each type of external parameter is less than a specific threshold, the calibration result converges, the calibration is stopped, and the external parameters of the vehicle-mounted camera are obtained.

[0087] In the technical solution provided by the present application, according to the coordinates of M first parallel lines in the first image acquired by the vehicle-mounted camera in the actual camera coordinate system of the vehicle-mounted camera and the parallel constraints, vertical constraints, spacing ratio constraints and spacing constraints of M second parallel lines in the bird's-eye view of the first image, the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera is determined in an optimization order of the x-axis rotation angle, y-axis rotation angle, z-axis rotation angle and z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system; then, according to the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera and the transformation relationship between the virtual ideal camera coordinate system and the world coordinate system, the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is determined, thereby improving the flexibility of the external parameter calibration method of the vehicle-mounted camera and at the same time improving the accuracy of the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera (i.e., the external parameters of the vehicle-mounted camera).

[0088] Fig.10 FIG. 1 is a flow chart of another method for calibrating the external parameters of a vehicle-mounted camera according to an embodiment of the present application. Fig.10 As shown, the method at least includes S1001 to S1011.

[0089] S1001, obtaining coordinates of M first parallel lines in a first image in an actual camera coordinate system of a vehicle-mounted camera, where the first image is an image captured by a vehicle to which the vehicle-mounted camera belongs during its driving on a target road, where the target road includes N parallel lane lines, where a distance between any two of the N lane lines is known, and where the M first parallel lines correspond one-to-one to the M first lane lines of the N lane lines, where M is an integer greater than or equal to 3, and N is an integer greater than or equal to M.

[0090] S1002, determining the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the constraints of the M second parallel lines in the bird's-eye view of the first image, the M second parallel lines corresponding one-to-one to the M first parallel lines, and the constraints of the M second parallel lines in the bird's-eye view include parallel constraints, vertical constraints, spacing ratio constraints and spacing constraints.

[0091] It should be noted that S1001 to S1002 may refer to S601 to S602 and will not be described in detail here.

[0092] S1003, determining a first external parameter of the vehicle-mounted camera according to a transformation relationship between an actual camera coordinate system of the vehicle-mounted camera and a virtual ideal camera coordinate system of the vehicle-mounted camera and a transformation relationship between the virtual ideal camera coordinate system and a world coordinate system.

[0093] The transformation relationship between the virtual ideal camera coordinate system and the world coordinate system is the x-axis rotation angle, y-axis rotation angle, z-axis rotation angle and z-axis translation distance of the world coordinate system relative to the virtual ideal camera coordinate system of the vehicle-mounted camera. The transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is the external parameters of the vehicle-mounted camera, including the x-axis rotation angle, y-axis rotation angle, z-axis rotation angle and z-axis translation distance of the world coordinate system relative to the actual camera coordinate system of the vehicle-mounted camera, which can also be called roll angle (roll), pitch angle (pitch), yaw angle (yaw) and translation distance. Among them, an example of the actual camera coordinate system of the vehicle-mounted camera can be Figure 3 An example of the actual camera coordinate system of the vehicle-mounted camera shown in FIG. 1 and the virtual ideal camera coordinate system of the vehicle-mounted camera can be Figure 3 or Figure 4 The virtual ideal camera coordinate system of the vehicle-mounted camera shown, an example of the world coordinate system can be Figure 4 The world coordinate system shown.

[0094] In one possible implementation, by transforming the matrix The transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera is converted into a transformation relationship between the virtual ideal camera coordinate system and the world coordinate system, so as to obtain a first transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system, that is, a first external parameter of the vehicle-mounted camera.

[0095] S1004, obtaining the y-axis translation amount of the vehicle-mounted camera in the world coordinate system according to the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera and the transformation relationship between the virtual ideal camera coordinate system and the world coordinate system.

[0096] As an example, in Figure 4 In the world coordinate system shown, the vehicle-mounted camera is at the Y w The position on the axis is the y-axis translation of the vehicle-mounted camera in the world coordinate system.

[0097] S1005, establishing an observation model of the vehicle according to the positioning information of the vehicle to which the vehicle-mounted camera belongs and the y-axis translation of the vehicle-mounted camera in the world coordinate system, wherein the observation model includes the influence of the y-axis translation on the state quantity.

[0098] In one possible implementation, the vehicle's positioning information is obtained from sensors of the vehicle to which the vehicle-mounted camera belongs, wherein the sensors of the vehicle to which the vehicle-mounted camera belongs include a speedometer, an IMU, and a GPS, etc. Accordingly, the vehicle's positioning information includes the vehicle's current position information and / or the vehicle's acceleration and / or the vehicle's wheel speed, etc.

[0099] S1006, obtaining the current heading angle of the vehicle according to the observation model of the vehicle.

[0100] In a possible implementation, the observation model of the vehicle is filtered and updated, and the heading angle of the vehicle at the current moment is obtained.

[0101] S1007, compensating the current heading angle of the vehicle according to the heading angle in the first external parameter of the vehicle-mounted camera to obtain a compensated heading angle.

[0102] In one possible implementation, Fig. 9 The schematic diagram of heading angle compensation is shown, where the vehicle heading angle is the heading angle of the vehicle at the current moment obtained based on the vehicle's observation model, and the camera heading angle is the heading angle in the external parameters of the determined on-board camera. The camera heading angle is subtracted from the vehicle heading angle to obtain the heading angle from the camera to the vehicle, which can also be called the compensated heading angle, and the determined heading angle in the on-board camera is updated to the compensated heading angle, thereby compensating for the influence of the vehicle heading angle obtained by vehicle positioning.

[0103] S1008, updating the heading angle in the first external parameter of the vehicle-mounted camera to the compensated heading angle.

[0104] The first external parameters of the vehicle-mounted camera include a first roll angle, a first pitch angle, a first heading angle, and a first translation distance. The first heading angle in the first external parameters of the vehicle-mounted camera is updated to a compensated heading angle, and the first external parameters of the vehicle-mounted camera after updating include a first roll angle, a first pitch angle, a compensated heading angle, and a first translation distance.

[0105] S1009, cache multiple first external parameters of the vehicle-mounted camera.

[0106] In a possible implementation, during the calibration of the external parameters of the vehicle-mounted camera, the vehicle-mounted camera captures a plurality of first images from the time when the vehicle to which the vehicle-mounted camera belongs enters the calibration site to the time when the vehicle-mounted camera exits the calibration site, obtains a plurality of first external parameters of the vehicle-mounted camera based on the plurality of first images, and caches the plurality of first external parameters of the vehicle-mounted camera. The calibration site is an area where N lane lines in a target road for the external parameter calibration of the vehicle-mounted camera are located.

[0107] S1010, filtering a plurality of first external parameters of the cached vehicle-mounted camera to obtain a second external parameter of the vehicle-mounted camera.

[0108] In a possible implementation manner, a periodic optimal estimation is performed on a plurality of first external parameters of the cached vehicle-mounted camera.

[0109] Optionally, if the number of multiple first external parameters of the vehicle-mounted camera cached during the period from when the vehicle-mounted camera enters the calibration site to when it exits the calibration site does not exceed a preset threshold, then the multiple first external parameters of the cached vehicle-mounted camera are used for optimal estimation and the calibration is completed; if the number of multiple first external parameters of the vehicle-mounted camera cached during the period from when the vehicle-mounted camera enters the calibration site to when it exits the calibration site exceeds a preset threshold, then the first external parameters of the vehicle-mounted camera cached at intervals of a certain number (such as 10 frames) are selected for optimal estimation.

[0110] Preferably, a group of data is cached for each type of external parameter among the multiple first external parameters of the vehicle-mounted camera, that is, a group of data is cached for the roll angle, pitch angle, heading angle and translation distance among the multiple first external parameters of the vehicle-mounted camera, and the kernel density estimation method is used to obtain the external parameter value with the highest probability in each group of data, which is the optimal estimate of this type of external parameter.

[0111] Specifically, the kernel density estimation method is to fit a Gaussian function to the statistical histogram of the data, and find a Gaussian function that best fits the histogram of the known data and can resist a certain amount of noise interference. For each type of external parameter, a set of external parameter values ​​corresponding to each type is used to estimate the fitted kernel function (preferably the Gaussian kernel function), find the peak value of the kernel function, and the corresponding horizontal axis value (where the probability of the Gaussian function is the maximum) is the optimal value of each type of external parameter.

[0112] Optionally, a four-dimensional kernel density estimation method can be used to simultaneously estimate the optimal value of each type of external parameter in four types of external parameters.

[0113] Optionally, a group of data is cached for each type of external parameter, and a median filtering method is used to obtain the median of each type of external parameter, which is the optimal value of each type of external parameter.

[0114] Optionally, a group of data is cached for each type of external parameter, and a mean filtering method is used to calculate the average value of each group of data. The mean value of each group of data obtained is the optimal value of each type of external parameter.

[0115] When the standard deviation of the optimal value of each type of external parameter is less than a specific threshold, the calibration result converges, the calibration is stopped, and the second external parameter of the vehicle-mounted camera is obtained.

[0116] S1011, updating the external parameters of the vehicle-mounted camera to second external parameters.

[0117] In the technical solution provided by the present application, the heading angle of the vehicle at the current moment is compensated using the heading angle in the external parameters of the vehicle-mounted camera in the world coordinate system to obtain the compensated heading angle, and the heading angle in the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is updated to the compensated heading angle, so that when the vehicle is not traveling completely parallel to the lane line, the influence of the vehicle's heading angle can be compensated, thereby improving the accuracy of the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system.

[0118] Fig.11 This is a schematic structural diagram of a device for calibrating external parameters of a vehicle-mounted camera according to an embodiment of the present application. Fig.11 As shown, the apparatus 1100 may include an acquisition module 1101 and a processing module 1102. The apparatus 1100 may be used to implement the method shown in any one of the above embodiments.

[0119] In a possible implementation, the device 1100 may be used to implement the above Figure 6 For example, the acquisition module 1101 is used to implement S601, and the processing module 1102 is used to implement S602 and S603.

[0120] In another possible implementation, the apparatus 1100 further includes a compensation module, an update module, and a cache module. The apparatus 1100 in this implementation can be used to implement the above Fig.10 For example, the acquisition module 1101 is used to implement S1001, S1004 and S1006, the processing module 1102 is used to implement S1002, S1003, S1005 and S1010, the compensation module is used to implement S1007, the update module is used to implement S1008 and S1011, and the cache module is used to implement S1009.

[0121] Fig.12 This is a schematic structural diagram of a device for calibrating external parameters of a vehicle-mounted camera according to another embodiment of the present application. Fig.12 The device 1200 shown can be used to execute the method for calibrating the external parameters of the vehicle-mounted camera shown in any of the above embodiments.

[0122] like Fig.12 As shown, the device 1200 of this embodiment includes: a memory 1201, a processor 1202, a communication interface 1203 and a bus 1204. The memory 1201, the processor 1202 and the communication interface 1203 are connected to each other through the bus 1204.

[0123] The memory 1201 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1201 may store a program. When the program stored in the memory 1201 is executed by the processor 1202, the processor 1202 may be used to execute Figure 6 and Fig.10 The individual steps of the method are shown.

[0124] Processor 1202 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs to implement the calibration method of the external parameters of the vehicle-mounted camera of the method embodiment of the present application.

[0125] The processor 1202 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the method of each embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 1202 or an instruction in software form.

[0126] The processor 1202 may also be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0127] The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1201, and the processor 1202 reads the information in the memory 1201 and completes the functions required to be executed by each method in the embodiments of the present application in combination with its hardware. For example, it can execute Figure 6 and Fig.10 The various steps / functions of the illustrated embodiment.

[0128] The communication interface 1203 may use, but is not limited to, a transceiver or other transceiver device to implement communication between the apparatus 1200 and other devices or a communication network.

[0129] The bus 1204 may include a path for transmitting information between various components of the device 1200 (eg, the memory 1201 , the processor 1202 , and the communication interface 1203 ).

[0130] It should be understood that the device 1200 shown in the embodiment of the present application may be an electronic device, or may also be a chip configured in the electronic device.

[0131] It should be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0132] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0133] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0134] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0135] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0136] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0137] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for calibrating the external parameters of a vehicle-mounted camera, characterized in that: The method comprises: Obtaining coordinates of M first parallel lines in a first image in an actual camera coordinate system of the vehicle-mounted camera, wherein the first image is an image captured by a vehicle to which the vehicle-mounted camera belongs during driving on a target road, wherein the target road includes N parallel lane lines, wherein a distance between any two of the N lane lines is known, wherein the M first parallel lines correspond one-to-one to M first lane lines of the N lane lines, wherein M is an integer greater than or equal to 3, and N is an integer greater than or equal to M; According to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the constraints of the M second parallel lines in the bird's-eye view of the first image, determine the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera, the M second parallel lines correspond to the M first parallel lines one-to-one, and the constraints of the M second parallel lines in the bird's-eye view include parallel constraints, vertical constraints, spacing ratio constraints and spacing constraints. The parallel constraint includes that the M second parallel lines are parallel, the vertical constraint includes that any one of the M second parallel lines is perpendicular to the x-axis direction of the virtual ideal camera coordinate system, the spacing ratio constraint includes that the distance ratio between any two of the M second parallel lines is the same as the distance ratio between the M lane lines corresponding to the any two second parallel lines, and the spacing constraint includes that the difference between the distance between the any two second parallel lines and the distance between the M lane lines corresponding to the any two second parallel lines is minimum; The transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is determined according to the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera and the transformation relationship between the virtual ideal camera coordinate system and the world coordinate system.

2. The method according to claim 1, characterized in that The determining, according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the constraints of the M second parallel lines in the bird's-eye view of the first image, a transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera comprises: Determine an x-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the parallel constraint; Determine a y-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the vertical constraint; Determine a z-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the spacing ratio constraint; According to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the spacing constraint, a z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined.

3. The method according to claim 2, characterized in that The parallel constraints also include: The minimum, among which, is the first of the M second parallel lines The angle between the second parallel line and the x-axis direction of the virtual ideal camera coordinate system, is the first of the M second parallel lines The angle between the second parallel line and the x-axis direction of the virtual ideal camera coordinate system, is an integer greater than or equal to 1 and less than M, Less than or equal to 90 degrees, Less than or equal to 90 degrees.

4. The method according to any one of claims 1 to 3, characterized in that The vertical constraints also include: Minimum, is the first of the M second parallel lines The angle between the second parallel line and the x-axis direction of the virtual ideal camera coordinate system, is an integer greater than or equal to 1 and less than or equal to M, Less than or equal to 90 degrees.

5. The method according to claim 4, characterized in that The spacing ratio constraint also includes: The minimum, among which, is the first of the M second parallel lines The second parallel line The distance between the second parallel lines, is the first of the M second parallel lines The second parallel line The distance between the second parallel lines, is the first of the M second parallel lines The lane line corresponding to the second parallel line is The actual distance between the lane lines corresponding to the second parallel lines, is the first of the M second parallel lines The lane line corresponding to the second parallel line is The actual distance between the lane lines corresponding to the second parallel lines, is an integer greater than or equal to 1 and less than M-1.

6. The method according to any one of claims 1 to 3 and 5, characterized in that: The spacing constraints also include Minimum, is the first of the M second parallel lines The second parallel line The distance between the second parallel lines, is the first of the M second parallel lines The lane line corresponding to the second parallel line is The actual distance between the lane lines corresponding to the second parallel lines, is an integer greater than or equal to 1 and less than M.

7. The method according to any one of claims 1 to 3 and 5, characterized in that: The method further comprises: Obtaining the y-axis translation of the vehicle-mounted camera in the world coordinate system; Acquiring vehicle positioning information, wherein the vehicle positioning information includes current position information of the vehicle and / or acceleration of the vehicle and / or wheel speed of the vehicle; Establishing an observation model of the vehicle according to the positioning information of the vehicle and the y-axis translation of the vehicle-mounted camera in the world coordinate system; Acquiring the heading angle of the vehicle at a current moment according to the observation model of the vehicle; Compensating the heading angle of the vehicle at a current moment according to the heading angle in the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system to obtain a compensated heading angle; The heading angle in the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is updated to the compensated heading angle.

8. A device for calibrating the external parameters of a vehicle-mounted camera, characterized in that: The device comprises: an acquisition module, configured to acquire coordinates of M first parallel lines in a first image in an actual camera coordinate system of the vehicle-mounted camera, wherein the first image is an image captured by a vehicle to which the vehicle-mounted camera belongs during driving on a target road, wherein the target road includes N parallel lane lines, wherein a distance between any two of the N lane lines is known, wherein the M first parallel lines correspond one-to-one to the M first lane lines of the N lane lines, wherein M is an integer greater than or equal to 3, and N is an integer greater than or equal to M; a processing module, for determining a transformation relationship between an actual camera coordinate system of the vehicle-mounted camera and a virtual ideal camera coordinate system of the vehicle-mounted camera according to coordinates of the M first parallel lines in an actual camera coordinate system of the vehicle-mounted camera and constraints of the M second parallel lines in a bird's-eye view of the first image, wherein the M second parallel lines correspond one-to-one to the M first parallel lines, and the constraints of the M second parallel lines in the bird's-eye view include a parallel constraint, a vertical constraint, a spacing ratio constraint, and a spacing constraint, wherein the parallel constraint includes that the M second parallel lines are parallel, the vertical constraint includes that any one of the M second parallel lines is perpendicular to an x-axis direction of the virtual ideal camera coordinate system, the spacing ratio constraint includes that a distance ratio between any two of the M second parallel lines is the same as a distance ratio between M lane lines corresponding to the any two second parallel lines, and the spacing constraint includes that a difference between the distance between the any two second parallel lines and the distance between the M lane lines corresponding to the any two second parallel lines is minimum; The processing module is also used to determine the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system based on the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the virtual ideal camera coordinate system of the vehicle-mounted camera and the transformation relationship between the virtual ideal camera coordinate system and the world coordinate system.

9. The device according to claim 8, characterized in that The processing module is specifically used for: Determine an x-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the parallel constraint; Determine a y-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the vertical constraint; Determine a z-axis rotation angle of the virtual ideal camera coordinate system relative to the actual camera coordinate system according to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the spacing ratio constraint; According to the coordinates of the M first parallel lines in the actual camera coordinate system of the vehicle-mounted camera and the spacing constraint, a z-axis translation distance of the virtual ideal camera coordinate system relative to the actual camera coordinate system is determined.

10. The device according to claim 9, characterized in that The parallel constraints also include: The minimum, among which, is the first of the M second parallel lines The angle between the second parallel line and the x-axis direction of the virtual ideal camera coordinate system, is the first of the M second parallel lines The angle between the second parallel line and the x-axis direction of the virtual ideal camera coordinate system, is an integer greater than or equal to 1 and less than M, Less than or equal to 90 degrees, Less than or equal to 90 degrees.

11. The device according to any one of claims 8 to 10, characterized in that The vertical constraints also include: Minimum, is the first of the M second parallel lines The angle between the second parallel line and the x-axis direction of the virtual ideal camera coordinate system, is an integer greater than or equal to 1 and less than or equal to M, Less than or equal to 90 degrees.

12. The device according to claim 11, characterized in that The spacing ratio constraint also includes: The minimum, among which, is the first of the M second parallel lines The second parallel line The distance between the second parallel lines, is the first of the M second parallel lines The second parallel line The distance between the second parallel lines, is the first of the M second parallel lines The lane line corresponding to the second parallel line is The actual distance between the lane lines corresponding to the second parallel lines, is the first of the M second parallel lines The lane line corresponding to the second parallel line is The actual distance between the lane lines corresponding to the second parallel lines, is an integer greater than or equal to 1 and less than M-1.

13. The device according to any one of claims 8 to 10 and 12, characterized in that The spacing constraints also include Minimum, is the first of the M second parallel lines The second parallel line The distance between the second parallel lines, is the first of the M second parallel lines The lane line corresponding to the second parallel line is The actual distance between the lane lines corresponding to the second parallel lines, is an integer greater than or equal to 1 and less than M.

14. The device according to any one of claims 8 to 10 and 12, characterized in that The device further comprises a compensation module, wherein the compensation module is used for: Obtaining the y-axis translation of the vehicle-mounted camera in the world coordinate system; Acquiring vehicle positioning information, wherein the vehicle positioning information includes current position information of the vehicle and / or acceleration of the vehicle and / or wheel speed of the vehicle; Establishing an observation model of the vehicle according to the positioning information of the vehicle and the y-axis translation of the vehicle-mounted camera in the world coordinate system; Acquiring the heading angle of the vehicle at a current moment according to the observation model of the vehicle; Compensating the heading angle of the vehicle at a current moment according to the heading angle in the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system to obtain a compensated heading angle; The heading angle in the transformation relationship between the actual camera coordinate system of the vehicle-mounted camera and the world coordinate system is updated to the compensated heading angle.

15. A device for calibrating external parameters of a vehicle-mounted camera, characterized in that: include: Memory and processor; The memory is used to store program instructions; The processor is configured to call the program instructions in the memory to execute the method according to any one of claims 1 to 7.

16. A chip, characterized in that: The system comprises at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is used to run a computer program or instruction to execute the method according to any one of claims 1 to 7.

17. A computer readable medium, characterized in that The computer-readable medium stores a program code for computer execution, the program code including instructions for executing the method according to any one of claims 1 to 7.

Images