A joint calibration method and system for vehicle-mounted laser radar and camera

Abstract

The present invention provides a vehicle-mounted laser radar and camera joint calibration method, system, electronic device and storage medium. By acquiring point cloud data and camera image frame sequences under different motion states of the vehicle, the image coordinates and point cloud coordinates of the calibration object are obtained from the camera image and point cloud data. Calibration parameters are calculated for the above image coordinates and point cloud coordinates based on a pre-trained calibration parameter prediction model to obtain the calibration parameters of the vehicle camera. The entire calibration parameter calculation process does not require reliance on a calibration plate and is applicable to vehicle motion states and site environments under any circumstances, thereby greatly improving the adaptability of the method, saving calibration plate data acquisition and processing time, and improving annotation efficiency. By using the pre-trained model to calculate the calibration parameters from the image coordinates and point cloud coordinates, the reduction in calibration accuracy caused by the calibration method and human errors is reduced, and the accuracy of camera calibration is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of equipment calibration technology, and more specifically, to a method and system for jointly calibrating a vehicle-mounted laser radar and a camera. Background Art

[0002] In image measurement and machine vision applications, a geometric model of camera imaging must be established to determine the relationship between the three-dimensional geometric position of a point on a spatial object's surface and its corresponding point in the image. These geometric model parameters are known as camera parameters. Under most conditions, these parameters must be determined through experimentation and calculation. This process of determining these parameters is called camera calibration. Whether in image measurement or machine vision applications, camera parameter calibration is a critical step. The accuracy of the calibration results and the stability of the algorithm directly impact the accuracy of the camera's results. Therefore, successful camera calibration is a prerequisite for subsequent work, and improving calibration accuracy is a key focus of scientific research.

[0003] Traditional calibration methods mostly perform calibration when the vehicle is static and the road is flat. In real scenes, due to changes in the vehicle's motion state and the unevenness of the road, the actual parameters differ from the calibrated parameters. Therefore, how to further improve the accuracy of camera calibration in vehicle motion scenes is an urgent problem to be solved. Summary of the Invention

[0004] In response to the technical problems existing in the prior art, the present invention provides a vehicle-mounted lidar and camera joint calibration method, system, electronic device and storage medium to solve the problem of how to further improve the accuracy of camera calibration in vehicle motion scenarios.

[0005] A first aspect of the present invention provides a method for joint calibration of a vehicle-mounted laser radar and a camera, comprising:

[0006] Based on the motion state of the vehicle, at least one set of laser point cloud data and camera image frame sequence is obtained in each motion state;

[0007] Based on a preset key frame setting condition and a preset supplementary frame setting condition, obtaining a key frame in the camera picture frame sequence and two supplementary frames adjacent to the key frame;

[0008] Setting all objects in the key frame and the two supplementary frames as camera calibration objects;

[0009] In the laser point cloud data, obtaining a point cloud calibration object corresponding to the camera calibration object;

[0010] The image coordinates of the camera calibration object and the point cloud coordinates of the point cloud calibration object of each object are input into a pre-trained calibration parameter prediction model, and the camera of the vehicle is calibrated based on the obtained camera calibration parameters.

[0011] On the basis of the above technical solution, the present invention can also make the following improvements.

[0012] Preferably, before the step of obtaining the key frame in the camera image and the two supplementary frames adjacent to the key frame, the step includes:

[0013] Divide each frame of the camera image into nine regions at equal intervals along the horizontal and vertical directions, and retain at most one vehicle in each region;

[0014] Divide the camera image into three areas in the horizontal direction at equal intervals, retaining at most two horizontal straight objects in each area;

[0015] The camera image is divided into three regions along the vertical direction at equal intervals, and at most two vertical objects are retained in each region.

[0016] Preferably, the preset key frame setting condition means that the key frame image contains at least 2 vertical straight objects, at least 2 horizontal straight objects, the total number of vertical straight objects and horizontal straight objects is not less than 8, and at least 2 vehicles;

[0017] The preset supplementary frame setting condition is that the supplementary frame image contains at least one vehicle and two straight objects.

[0018] Preferably, the calibration parameter prediction model is:

[0019]

[0020] Among them, K is the calibration internal parameter, T is the calibration external parameter, P1 is the point cloud coordinate of the vehicle, X1 is the image coordinate of the vehicle, is the image coordinate of the i-th vehicle, P2 is the point cloud coordinate of the straight object, is the point cloud coordinate of the i-th straight object, X2 is the image coordinate of the straight object, f1 is the formula for finding the outer contour based on scattered points, f2 is the formula for finding the straight line based on scattered points, D1 is the formula for the minimum distance from a point to the outer contour, D2 is the formula for the distance from a point to a straight line, n1 is the number of vehicle image coordinate points, and n2 is the number of point cloud coordinate points of the straight target object.

[0021] Preferably, the image coordinates of the camera calibration object are: when the camera calibration object is a vehicle, the image coordinates are the values ​​of the outer contour of the camera calibration object in the UV coordinate system; when the camera calibration object is a straight object, the image coordinates are the values ​​of the symmetry center line of the camera calibration object in the UV coordinate system;

[0022] The point cloud coordinates of the point cloud calibration object are: the XYZ values ​​of all points of the point cloud calibration object.

[0023] Preferably, the motion state includes a stationary state, a low-speed state and a high-speed state. The vehicle speed in the stationary state is 0 km / h, the vehicle speed in the low-speed state is 0 km / h to 30 km / h, and the vehicle speed in the high-speed state is greater than 30 km / h.

[0024] Preferably, before the step of calibrating the vehicle camera based on the obtained calibration parameters, the step includes:

[0025] For each motion state, when there are multiple groups of laser point cloud data and camera image frame sequences, the obtained calibration parameters are used as the initial calibration parameter group. The initial calibration parameters that are less than the lower quartile and exceed the upper quartile in the initial calibration parameter group are removed using a box plot, and the variance of the remaining initial calibration parameters is calculated. When the variance is less than a preset variance threshold, any initial calibration parameter among the remaining initial calibration parameters is set as the calibration parameter of the vehicle in the motion state.

[0026] When the variance is greater than or equal to a preset variance threshold, the laser point cloud data and camera image of the vehicle are reacquired to calculate calibration parameters.

[0027] A second aspect of the present invention provides a vehicle-mounted laser radar and camera joint calibration system, comprising:

[0028] A data acquisition module, configured to acquire laser point cloud data and camera images of the vehicle based on the vehicle's motion state;

[0029] An image frame acquisition module, configured to acquire a key frame in the camera image and two supplementary frames adjacent to the key frame based on a preset key frame setting condition and a preset supplementary frame setting condition;

[0030] An image calibration module, configured to set all objects in the key frame and the two supplementary frames as camera calibration objects;

[0031] A point cloud calibration module, configured to obtain a point cloud calibration object corresponding to the camera calibration object in the laser point cloud data;

[0032] The parameter calculation module is used to input the image coordinates of the camera calibration object and the point cloud coordinates of the point cloud calibration object into a pre-trained calibration parameter prediction model, and calibrate the vehicle camera based on the obtained calibration parameters.

[0033] The third aspect of the present invention provides an electronic device comprising a memory and a processor, wherein the processor is configured to implement any step of the vehicle-mounted laser radar and camera joint calibration method in the first aspect when executing a computer management program stored in the memory.

[0034] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer management program is stored. When the computer management program is executed by a processor, the steps of the joint calibration method of any vehicle-mounted laser radar and camera in the above-mentioned first aspect are implemented.

[0035] The present invention provides a vehicle-mounted laser radar and camera joint calibration method, system, electronic device and storage medium. The method includes: based on the motion state of the vehicle, obtaining at least one set of laser point cloud data and camera picture frame sequence corresponding to the above motion state in each motion state; based on preset key frame setting conditions and preset supplementary frame setting conditions, obtaining a key frame in the above camera picture frame sequence and two supplementary frames adjacent to the above key frame; setting all objects in the above key frame and the two supplementary frames as camera calibration objects; obtaining point cloud calibration objects corresponding to the above camera calibration objects in the above laser point cloud data; inputting the image coordinates of the above camera calibration object and the point cloud coordinates of the above point cloud calibration object of each object into a pre-trained calibration parameter prediction model, and calibrating the above vehicle's camera based on the obtained camera calibration parameters. The present invention obtains point cloud data and camera images under different motion states of the vehicle, obtains the image coordinates and point cloud coordinates of the calibration object from the camera images and point cloud data, calculates calibration parameters for the above image coordinates and point cloud coordinates based on a pre-trained calibration parameter prediction model, and obtains the calibration parameters of the vehicle camera. The entire calibration parameter calculation process does not require reliance on a calibration plate and is applicable to vehicle motion states and site environments under any circumstances, thereby greatly improving the adaptability of the method, saving calibration plate data acquisition and processing time, and improving annotation efficiency. By using the pre-trained model to calculate the calibration parameters from the image coordinates and point cloud coordinates, the reduction in calibration accuracy caused by the calibration method and human errors is reduced, and the accuracy of camera calibration is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flow chart of a joint calibration method for a vehicle-mounted laser radar and camera provided by the present invention;

[0037] Figure 2 A schematic diagram of the structure of a vehicle-mounted laser radar and camera joint calibration system provided by the present invention;

[0038] Figure 3 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0039] Figure 4A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0040] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0041] Figure 1 This is a flow chart of a joint calibration method for a vehicle-mounted laser radar and camera provided by the present invention, such as Figure 1 As shown, the method includes:

[0042] Step S100: Based on the motion state of the vehicle, obtaining at least one set of laser point cloud data and camera image frame sequence in each motion state;

[0043] It should be noted that the execution subject of the method of this embodiment can be a computer terminal device with data processing, network communication, and program execution functions, such as an in-vehicle computer, a tablet computer, etc.; it can also be a server device with the same or similar functions, or a cloud server with similar functions, and this embodiment does not limit this. For ease of understanding, this embodiment and the following embodiments will be described using an in-vehicle computer as an example.

[0044] It is understood that the above motion states can be divided according to the vehicle's speed, including stationary state (0 km / h), low-speed state (0 km / h to 30 km / h), and high-speed state (>30 km / h). The laser point cloud data and camera image frame sequence corresponding to the above motion states are a set of data to be calculated. In order to further improve the accuracy of the calibration parameters, multiple sets of data can be selected for calculation in each motion state.

[0045] It should be understood that the above-mentioned laser point cloud data is obtained through the vehicle-mounted laser radar, and the above-mentioned camera images are obtained through the vehicle-mounted camera. After obtaining the above-mentioned laser point cloud data and camera images, it is also necessary to time-align the point cloud, images and vehicle motion status.

[0046] It can also be understood that the above-mentioned point cloud data and camera image frame sequence can be obtained by the vehicle when camera calibration is required. In order to achieve better data calibration effect, the data acquisition scene can be selected at a traffic intersection with more vehicles or a road with heavy traffic.

[0047] Step S200: acquiring a key frame in the camera image frame sequence and two supplementary frames adjacent to the key frame based on a preset key frame setting condition and a preset supplementary frame setting condition;

[0048] Furthermore, before the step of obtaining the key frame in the camera image and the two supplementary frames adjacent to the key frame, the step includes:

[0049] Step S001: Divide each frame of the camera image into nine areas in the horizontal and vertical directions at equal intervals, and retain at most one vehicle in each area;

[0050] Step S002: Divide the camera image into three areas in the horizontal direction at equal intervals, and retain at most two horizontal straight objects in each area;

[0051] Step S003: Divide the camera image into three areas along the vertical direction at equal intervals, and retain at most two vertical straight objects in each area.

[0052] Specifically, the image is divided into 9 areas horizontally and vertically at equal intervals, and at most one vehicle is retained in each area; the image is divided into 3 areas horizontally at equal intervals, and at most two horizontal vertical objects (such as traffic light poles and signboard poles) are retained in each area; the image is divided into 3 areas vertically at equal intervals, and at most two vertical vertical objects (such as street light poles and signboard poles) are retained in each area.

[0053] Furthermore, the preset key frame setting conditions are that the key frame image contains at least 2 vertical straight objects, at least 2 horizontal straight objects, the total number of vertical straight objects and horizontal straight objects is not less than 8, and at least 2 vehicles.

[0054] Furthermore, the preset supplementary frame setting condition is that the supplementary frame image contains at least one vehicle and two straight objects.

[0055] Step S300: setting all objects in the key frame and the two supplementary frames as camera calibration objects;

[0056] Step S400: obtaining a point cloud calibration object corresponding to the camera calibration object in the laser point cloud data;

[0057] Step S500: inputting the image coordinates of the camera calibration object and the point cloud coordinates of the point cloud calibration object of each object into a pre-trained calibration parameter prediction model, and calibrating the camera of the vehicle based on the obtained camera calibration parameters.

[0058] Furthermore, the image coordinates of the camera calibration object are: when the camera calibration object is a vehicle, the image coordinates are the values ​​of the outer contour of the camera calibration object in the UV coordinate system; when the camera calibration object is a straight object, the image coordinates are the values ​​of the symmetry center line of the camera calibration object in the UV coordinate system; the point cloud coordinates of the point cloud calibration object are: the XYZ values ​​of all points of the point cloud calibration object.

[0059] Specifically, the image coordinates of the camera calibration object are: when the camera calibration object is a vehicle, the image coordinates are the UV values ​​of the outer contour of the camera calibration object (the contour points are evenly selected along the outer contour); when the camera calibration object is a straight object, the image coordinates are the UV values ​​of the symmetry center line of the camera calibration object (the symmetry center points are evenly selected along the symmetry center line); the point cloud coordinates of the point cloud calibration object are: the XYZ values ​​of all points of the point cloud calibration object.

[0060] Furthermore, the calibration parameter prediction model is:

[0061]

[0062] Among them, K is the calibration internal parameter, T is the calibration external parameter, P1 is the point cloud coordinate of the vehicle, X1 is the image coordinate of the vehicle, is the image coordinate of the i-th vehicle, P2 is the point cloud coordinate of the straight object, is the point cloud coordinate of the i-th straight object, X2 is the image coordinate of the straight object, f1 is the formula for finding the outer contour based on scattered points, f2 is the formula for finding the straight line based on scattered points, D1 is the formula for the minimum distance from a point to the outer contour, D2 is the formula for the distance from a point to a straight line, n1 is the number of vehicle image coordinate points, and n2 is the number of point cloud coordinate points of the straight target object.

[0063] Furthermore, each of the above motion states may include at least 10 calibration groups, each calibration group performs model training separately, and 10 calibration parameter results are obtained for each motion state.

[0064] Furthermore, before the step of calibrating the vehicle camera based on the obtained calibration parameters, the following steps are included:

[0065] Step S401: For each motion state, when there are multiple groups of laser point cloud data and camera image frame sequences, the obtained calibration parameters are used as the initial calibration parameter group. The initial calibration parameters that are less than the lower quartile and exceed the upper quartile in the initial calibration parameter group are removed using a box plot, and the variance of the remaining initial calibration parameters is calculated. When the variance is less than a preset variance threshold, any initial calibration parameter among the remaining initial calibration parameters is set as the calibration parameter of the ego vehicle in the motion state; when the variance is greater than or equal to the preset variance threshold, the laser point cloud data and camera images of the ego vehicle are re-acquired to calculate the calibration parameters.

[0066] Specifically, a box plot is used to eliminate values ​​that are less than the lower quartile and exceed the upper quartile, and the variance of the remaining values ​​is calculated. If the variance is less than the set threshold, any set of parameters is taken as the final result. If the variance exceeds the set threshold, data is collected again and the model is trained.

[0067] Furthermore, each motion state will obtain a set of calibrated internal and external parameters. When in use, the vehicle speed state is determined according to the actual vehicle speed and the corresponding internal and external parameters are selected.

[0068] It is understandable that, based on the shortcomings of the background art, the embodiments of the present invention propose a joint calibration method for a vehicle-mounted laser radar and camera. The method includes: based on the motion state of the vehicle, obtaining at least one set of laser point cloud data and camera image frame sequences corresponding to the motion state for each motion state; based on preset key frame setting conditions and preset supplementary frame setting conditions, obtaining a key frame in the camera image frame sequence and two supplementary frames adjacent to the key frame; setting all objects in the key frame and the two supplementary frames as camera calibration objects; obtaining point cloud calibration objects corresponding to the camera calibration objects from the laser point cloud data; inputting the image coordinates of the camera calibration object and the point cloud coordinates of the point cloud calibration object for each object into a pre-trained calibration parameter prediction model, and calibrating the vehicle's camera based on the obtained camera calibration parameters. The present invention obtains point cloud data and camera images under different motion states of the vehicle, obtains the image coordinates and point cloud coordinates of the calibration object from the camera images and point cloud data, calculates calibration parameters for the above image coordinates and point cloud coordinates based on a pre-trained calibration parameter prediction model, and obtains the calibration parameters of the vehicle camera. The entire calibration parameter calculation process does not require reliance on a calibration plate and is applicable to vehicle motion states and site environments under any circumstances, thereby greatly improving the adaptability of the method, saving calibration plate data acquisition and processing time, and improving annotation efficiency. By using the pre-trained model to calculate the calibration parameters from the image coordinates and point cloud coordinates, the reduction in calibration accuracy caused by the calibration method and human errors is reduced, and the accuracy of camera calibration is greatly improved.

[0069] See also Figure 2 , Figure 2 A schematic diagram of a vehicle-mounted laser radar and camera joint calibration system structure provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, a vehicle-mounted laser radar and camera joint calibration system includes a data acquisition module 100, an image frame acquisition module 200, an image calibration module 300, a point cloud calibration module 400 and a parameter calculation module 500, wherein:

[0070] The data acquisition module 100 is used to obtain at least one set of laser point cloud data and camera image frame sequence in each motion state based on the motion state of the ego vehicle; the image frame acquisition module 200 is used to obtain the key frame in the camera image frame sequence and the two supplementary frames adjacent to the key frame based on preset key frame setting conditions and preset supplementary frame setting conditions; the image calibration module 300 is used to set all objects in the key frame and the two supplementary frames as camera calibration objects; the point cloud calibration module 400 is used to obtain the point cloud calibration objects corresponding to the camera calibration objects in the laser point cloud data; the parameter calculation module 500 is used to input the image coordinates of the camera calibration object and the point cloud coordinates of the point cloud calibration object of each object into a pre-trained calibration parameter prediction model, and calibrate the camera of the ego vehicle based on the obtained camera calibration parameters.

[0071] It can be understood that the vehicle-mounted laser radar and camera joint calibration system provided by the present invention corresponds to the vehicle-mounted laser radar and camera joint calibration method provided by the aforementioned embodiments. The relevant technical features of the vehicle-mounted laser radar and camera joint calibration system can refer to the relevant technical features of the vehicle-mounted laser radar and camera joint calibration method, which will not be repeated here.

[0072] See also Figure 3 , Figure 3 Schematic diagram of an embodiment of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device, including a memory 1310, a processor 1320, and a computer program 1311 stored in the memory 1310 and executable on the processor 1320. When the processor 1320 executes the computer program 1311, the following steps are implemented:

[0073] Based on the motion state of the ego vehicle, obtain at least one set of laser point cloud data and camera picture frame sequence corresponding to the above motion state in each motion state; based on preset key frame setting conditions and preset supplementary frame setting conditions, obtain the key frame in the above camera picture frame sequence and two supplementary frames adjacent to the above key frame; set all objects in the above key frame and the two supplementary frames as camera calibration objects; obtain the point cloud calibration objects corresponding to the above camera calibration objects in the above laser point cloud data; input the image coordinates of the above camera calibration object and the point cloud coordinates of the above point cloud calibration object of each object into the pre-trained calibration parameter prediction model, and calibrate the camera of the ego vehicle based on the obtained camera calibration parameters.

[0074] See also Figure 4 , Figure 4 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 4As shown, this embodiment provides a computer-readable storage medium 1400 on which a computer program 1411 is stored. When the computer program 1411 is executed by a processor, the following steps are implemented:

[0075] Based on the motion state of the ego vehicle, obtain at least one set of laser point cloud data and camera picture frame sequence corresponding to the above motion state in each motion state; based on preset key frame setting conditions and preset supplementary frame setting conditions, obtain the key frame in the above camera picture frame sequence and two supplementary frames adjacent to the above key frame; set all objects in the above key frame and the two supplementary frames as camera calibration objects; obtain the point cloud calibration objects corresponding to the above camera calibration objects in the above laser point cloud data; input the image coordinates of the above camera calibration object and the point cloud coordinates of the above point cloud calibration object of each object into the pre-trained calibration parameter prediction model, and calibrate the camera of the ego vehicle based on the obtained camera calibration parameters.

[0076] An embodiment of the present invention provides a vehicle-mounted laser radar and camera joint calibration method, system and storage medium, the method comprising: based on the motion state of the vehicle, obtaining at least one set of laser point cloud data and camera picture frame sequence corresponding to the above motion state in each motion state; based on preset key frame setting conditions and preset supplementary frame setting conditions, obtaining a key frame in the above camera picture frame sequence and two supplementary frames adjacent to the above key frame; setting all objects in the above key frame and the two supplementary frames as camera calibration objects; obtaining point cloud calibration objects corresponding to the above camera calibration objects in the above laser point cloud data; inputting the image coordinates of the above camera calibration object and the point cloud coordinates of the above point cloud calibration object of each object into a pre-trained calibration parameter prediction model, and calibrating the above vehicle's camera based on the obtained camera calibration parameters. The present invention obtains point cloud data and camera images under different motion states of the vehicle, obtains the image coordinates and point cloud coordinates of the calibration object from the camera images and point cloud data, calculates calibration parameters for the above image coordinates and point cloud coordinates based on a pre-trained calibration parameter prediction model, and obtains the calibration parameters of the vehicle camera. The entire calibration parameter calculation process does not require reliance on a calibration plate and is applicable to vehicle motion states and site environments under any circumstances, thereby greatly improving the adaptability of the method, saving calibration plate data acquisition and processing time, and improving annotation efficiency. By using the pre-trained model to calculate the calibration parameters from the image coordinates and point cloud coordinates, the reduction in calibration accuracy caused by the calibration method and human errors is reduced, and the accuracy of camera calibration is greatly improved.

[0077] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0078] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0082] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0083] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A joint calibration method for vehicle-mounted laser radar and camera, characterized in that: The method comprises: Based on the motion state of the vehicle, at least one set of laser point cloud data and camera image frame sequence is obtained in each motion state; Based on preset key frame setting conditions and preset supplementary frame setting conditions, a key frame in the camera image frame sequence and two supplementary frames adjacent to the key frame are obtained; wherein the preset key frame setting conditions are that the key frame image contains at least two vertical straight objects, at least two horizontal straight objects, the total number of vertical straight objects and horizontal straight objects is not less than eight, and at least two vehicles; and the preset supplementary frame setting conditions are that the supplementary frame image contains at least one vehicle and two straight objects; Setting all objects in the key frame and the two supplementary frames as camera calibration objects; In the laser point cloud data, obtaining a point cloud calibration object corresponding to the camera calibration object; The image coordinates of the camera calibration object and the point cloud coordinates of the point cloud calibration object of each object are input into a pre-trained calibration parameter prediction model, and the camera of the vehicle is calibrated based on the obtained camera calibration parameters.

2. The vehicle-mounted laser radar and camera joint calibration method according to claim 1, characterized in that: Before the step of obtaining the key frame in the camera image and the two supplementary frames adjacent to the key frame, the method includes: Divide each frame of the camera image into nine regions at equal intervals along the horizontal and vertical directions, and retain at most one vehicle in each region; Divide the camera image into three areas in the horizontal direction at equal intervals, retaining at most two horizontal straight objects in each area; The camera image is divided into three areas along the vertical direction at equal intervals, and at most two vertical straight objects are retained in each area.

3. The vehicle-mounted laser radar and camera joint calibration method according to claim 1, characterized in that: The calibration parameter prediction model is: in, K To calibrate the internal reference, T To calibrate the external parameters, P 1 is the point cloud coordinate of the vehicle, X 1 is the image coordinate of the vehicle, For the i The image coordinates of the vehicle, P 2 is the point cloud coordinate of the vertical object, For the i The point cloud coordinates of a straight object, X 2 is the image coordinate of the vertical object, f 1 is the formula for finding the outer contour based on scattered points, f 2 is the formula for finding a straight line based on scattered points. D 1 is the minimum distance formula from a point to the outer contour, D 2 is the distance formula from a point to a straight line, n 1 is the number of vehicle image coordinate points, n 2 is the number of vertical target point cloud coordinate points.

4. The vehicle-mounted laser radar and camera joint calibration method according to claim 1, characterized in that: The image coordinates of the camera calibration object are: when the camera calibration object is a vehicle, the image coordinates are the values ​​of the outer contour of the camera calibration object in the UV coordinate system; when the camera calibration object is a straight object, the image coordinates are the values ​​of the symmetry center line of the camera calibration object in the UV coordinate system; The point cloud coordinates of the point cloud calibration object are: the XYZ values ​​of all points of the point cloud calibration object.

5. The vehicle-mounted laser radar and camera joint calibration method according to claim 1, characterized in that: The motion state includes a stationary state, a low-speed state and a high-speed state. The vehicle speed in the stationary state is 0 km / h, the vehicle speed in the low-speed state is 0 km / h to 30 km / h, and the vehicle speed in the high-speed state is greater than 30 km / h.

6. The vehicle-mounted laser radar and camera joint calibration method according to claim 1, characterized in that: Before the step of calibrating the vehicle camera based on the obtained calibration parameters, the method includes: For each motion state, when there are multiple groups of laser point cloud data and camera image frame sequences, the obtained calibration parameters are used as the initial calibration parameter group. The initial calibration parameters that are less than the lower quartile and exceed the upper quartile in the initial calibration parameter group are removed using a box plot, and the variance of the remaining initial calibration parameters is calculated. When the variance is less than a preset variance threshold, any initial calibration parameter among the remaining initial calibration parameters is set as the calibration parameter of the vehicle in the motion state. When the variance is greater than or equal to a preset variance threshold, the laser point cloud data and camera image of the vehicle are reacquired to calculate calibration parameters.

7. A vehicle-mounted laser radar and camera joint calibration system, characterized in that: include: A data acquisition module is used to obtain at least one set of laser point cloud data and camera image frame sequence in each motion state based on the motion state of the vehicle; an image frame acquisition module, configured to acquire a key frame in the camera image frame sequence and two supplementary frames adjacent to the key frame based on preset key frame setting conditions and preset supplementary frame setting conditions; wherein the preset key frame setting conditions are that the key frame image contains at least two vertical straight objects, at least two horizontal straight objects, the total number of vertical straight objects and horizontal straight objects is not less than eight, and at least two vehicles; and the preset supplementary frame setting conditions are that the supplementary frame image contains at least one vehicle and two straight objects; An image calibration module, configured to set all objects in the key frame and the two supplementary frames as camera calibration objects; A point cloud calibration module, configured to obtain a point cloud calibration object corresponding to the camera calibration object in the laser point cloud data; A parameter calculation module is used to input the image coordinates of the camera calibration object and the point cloud coordinates of the point cloud calibration object of each object into a pre-trained calibration parameter prediction model, and calibrate the camera of the vehicle based on the obtained camera calibration parameters.

8. An electronic device, characterized in that: It includes a memory and a processor, and the processor is used to implement the steps of the vehicle-mounted laser radar and camera joint calibration method as described in any one of claims 1 to 6 when executing the computer management program stored in the memory.

9. A computer-readable storage medium, characterized in that A computer management program is stored thereon, and when the computer management program is executed by the processor, the steps of the vehicle-mounted laser radar and camera joint calibration method as described in any one of claims 1 to 6 are implemented.

Images