Simulation-based camera and radar calibration method, computer equipment and storage medium

Abstract

The present invention discloses a camera and radar calibration method based on simulation, a computer device and a storage medium, the method comprising: constructing a camera model and a radar model based on the actual positional relationship between the camera and the radar in a simulation environment; adding a feature model in the simulation environment; obtaining image information containing the feature model through the camera model, and obtaining radar data information containing the feature model through the radar model; identifying the three-dimensional positional information of the feature model in the camera coordinate system; generating a feature template based on the feature model and the three-dimensional positional information; converting the radar data information containing the feature model to the camera coordinate system to obtain point cloud data; matching the point cloud data with the feature template to obtain the second coordinate conversion relationship between the camera model and the radar model. The scheme is based on the actual positional relationship between the camera and the radar, and verifies the radar and camera calibration algorithms in a simulation environment, and the calibration accuracy is higher.

Description

Technical Field

[0001] The present invention relates to the technical field of camera radar calibration, and in particular to a simulation-based camera and radar calibration method, a computer device and a storage medium. Background Art

[0002] In scenarios such as autonomous driving or location recognition tasks, such as robotic arm grasping, RGB images and radar depth image fusion processing are often required, so the coordinate relationship between the camera and radar, that is, the external parameter, is crucial to the accuracy of the entire sensor system. If the camera and radar calibration accuracy is not enough, the accuracy of the entire system will decrease. In addition, since the cameras and radar equipment involved in the calibration are relatively expensive, and different models of cameras and radars need to be calibrated separately, the cost of calibration in on-site work will be high, and the work cycle will become extremely lengthy. Summary of the invention

[0003] The purpose of the present invention is to provide a camera and radar calibration method, computer equipment and storage medium based on simulation, which is based on the actual position relationship between the camera and the radar, and verifies the radar and camera calibration algorithms in a simulation environment, which greatly facilitates the development process and has higher calibration accuracy.

[0004] The technical solution provided by the present invention is as follows:

[0005] In a first aspect, the present invention provides a camera and radar calibration method based on simulation, comprising the steps of:

[0006] Constructing a camera model and a radar model based on the actual positional relationship between the camera and the radar in a simulation environment, and inputting simulation properties into the camera model and the radar model respectively;

[0007] Adding a feature model in the simulation environment, wherein the feature model is within the recognition range of the camera model and the radar model;

[0008] Acquire image information containing the feature model in an optical coordinate system through the camera model, and acquire radar data information containing the feature model in a radar coordinate system through the radar model;

[0009] Acquire a first coordinate transformation relationship between the optical coordinate system and the camera coordinate system of the camera model, and map the image information containing the feature model to the camera coordinate system based on the first coordinate transformation relationship, and identify the three-dimensional position information of the feature model in the camera coordinate system;

[0010] generating a feature template based on the feature model and the three-dimensional position information;

[0011] Convert the radar data information containing the feature model into the camera coordinate system to obtain point cloud data of the feature model;

[0012] The point cloud data is matched with the feature template to obtain a second coordinate transformation relationship between the camera model and the radar model.

[0013] This solution constructs a simulation environment, and constructs a camera model and a radar model based on the actual position relationship of the camera and the radar in the simulation environment. It can facilitate the adjustment of the camera model and the radar model according to the models and parameters of the camera and the radar, and realizes the calibration of the camera and the radar through simulation to verify the calibration algorithm of the camera and the radar, which is conducive to saving calibration costs. At the same time, by setting the feature model, the image information of the feature model in the optical coordinate system of the camera model is obtained, and the three-dimensional position information of the feature model in the camera coordinate system is identified based on the first coordinate transformation relationship between the optical coordinate system and the camera coordinate system and the corresponding feature template is generated, and then the radar data information containing the feature model is converted to the camera coordinate system to obtain the point cloud data of the feature model. By matching the point cloud data and the feature template, the coordinate transformation relationship between the camera model and the radar model can be obtained, thereby realizing accurate calibration of the camera and the radar in the simulation environment.

[0014] In some embodiments, the feature model is a two-dimensional code model, which is composed of a plurality of two-dimensional code sub-models, each of which corresponds to a two-dimensional code; and adding a feature model in the simulation environment specifically includes:

[0015] Constructing the two-dimensional code model by configuring the ID information, size information and offset information of each of the two-dimensional code sub-models;

[0016] The two-dimensional code model is added to the simulation environment in the form of a script or a plug-in.

[0017] By adopting the QR code model as the feature model, the feature model can be easily identified. Since information can be stored in the QR code, by using the QR code model as the feature model and storing the ID information, size information and offset information of the QR code model in the QR code, the three-dimensional information of the QR code model can be accurately identified after the QR code model is identified, thereby facilitating the accurate calibration of the camera model and the radar model. By setting the QR code model to be composed of several QR code sub-models, and the QR code styles of each QR code sub-model are different, the recognition accuracy of the camera model to the QR code model can be improved.

[0018] In some implementations, identifying the three-dimensional position information of the feature model in the camera coordinate system specifically includes:

[0019] Identify first coordinate information of the center or mark point of the two-dimensional code model in the optical coordinate system through the camera model;

[0020] Obtaining second coordinate information of the first coordinate information in the camera coordinate system based on the first coordinate transformation relationship;

[0021] Identify the ID information, size information and offset information of each of the two-dimensional code sub-models in the two-dimensional code model by using a two-dimensional code recognition algorithm;

[0022] The three-dimensional position information of the feature model in the camera coordinate system is obtained according to the second coordinate information and the ID information, size information and offset information of each of the two-dimensional code sub-models.

[0023] When identifying the three-dimensional position information of the feature model in the camera coordinate system, by identifying the first coordinate information of the center or preset mark point of the two-dimensional code model in the optical coordinate system, the second coordinate information in the camera coordinate system corresponding to the first coordinate information can be obtained based on the first coordinate conversion relationship between the optical coordinate system and the camera coordinate system. Based on the ID information, size information and offset information of the two-dimensional code sub-model identified by the two-dimensional code recognition algorithm, the three-dimensional position information of the two-dimensional code model in the camera coordinate system can be accurately calculated, thereby avoiding the errors that are easily generated when using conventional feature models for coordinate conversion.

[0024] In some implementations, converting the radar data information including the feature model into the camera coordinate system to obtain point cloud data of the feature model specifically includes:

[0025] The reference coordinates of the radar data information are uniformly changed to start from the camera coordinate system, and the point cloud data of the feature model remains unchanged.

[0026] In some implementations, generating a feature template based on the feature model and the three-dimensional position information specifically includes:

[0027] Acquire first position information corresponding to the center or marking point of the two-dimensional code model in the three-dimensional position information;

[0028] Taking the first position information as the center or marking point of the feature template, generating the feature template according to the ID information, size information and offset information of each of the two-dimensional code sub-models, so that the parameter information of the feature template is the same as that of the feature model;

[0029] The matching of the point cloud data with the feature template to obtain a second coordinate transformation relationship between the camera model and the radar model specifically includes:

[0030] In the simulation environment, adjusting the coordinate position of the radar data information converted to the camera coordinate system so that the point cloud data coincides with the feature template, and recording the adjustment parameters of the coordinate position of the radar data information;

[0031] The adjustment parameter is the second coordinate transformation relationship.

[0032] When calculating the coordinate transformation relationship between the radar model and the camera model, a feature template can be generated based on the parameter information of the feature model and its corresponding three-dimensional position information, thereby facilitating the matching of the three-dimensional position information and the point cloud data. By adjusting the coordinate position of the radar data information to make the point cloud data coincide with the feature template, the adjustment parameters of the radar data information can be determined, and the adjustment parameters can be regarded as the coordinate transformation relationship between the radar model and the camera model.

[0033] In some implementations, the simulation-based camera and radar calibration method further includes:

[0034] Information parameters of different sizes / different offset angles of the feature model are configured to recalibrate, and the second coordinate transformation relationship obtained by each calibration is compared to correct the second coordinate transformation relationship.

[0035] When the relative positions of the camera model and the radar model remain unchanged, only the size or angle information of the feature model is adjusted, which will not affect the calibration results of the radar model and the camera model. By configuring different size / different offset angle information parameters of the feature model for recalibration, it is possible to verify whether the calibration results of the radar model and the camera model are accurate, thereby correcting the second coordinate transformation relationship.

[0036] In some implementations, the simulation-based camera and radar calibration method further includes:

[0037] According to the actual positional relationship between the camera and the radar, a third coordinate transformation relationship between the camera model and the radar model is obtained, and the second coordinate transformation relationship and the third coordinate transformation relationship are compared to calibrate the calibration method.

[0038] After knowing the actual positional relationship between the camera and the radar, the coordinate transformation relationship between the radar model and the camera model can be directly obtained, and compared with the coordinate transformation relationship calculated by the above calibration algorithm in the simulation environment, so as to verify whether the calibration algorithm is accurate in the simulation environment.

[0039] In some implementations, before obtaining the first coordinate transformation relationship between the camera coordinate system and the optical coordinate system of the camera model, the method further includes:

[0040] Calibrate the internal parameters of the camera model by using a calibration algorithm to obtain corrected camera parameters of the camera model;

[0041] The simulation environment is restarted according to the corrected camera parameters to reduce errors in subsequent acquisition of the image information by the camera model.

[0042] Before calibrating the coordinate transformation relationship between the camera model and the radar model, that is, the external parameters, the internal parameters of the camera model are calibrated first. This can avoid the influence of the distortion of the subsequent image information obtained by the camera model on the external parameter calibration and improve the accuracy of the calibration.

[0043] In a second aspect, the present application provides a computer device comprising a memory, a processor, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to implement the steps of the camera and radar calibration method of the first aspect.

[0044] In a third aspect, the present application provides a computer storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the camera and radar calibration method of the first aspect.

[0045] According to a simulation-based camera and radar calibration method, computer equipment and storage medium provided by the present invention, by constructing a simulation environment, and constructing a camera model and a radar model in the simulation environment based on the actual position relationship of the camera and the radar, it is possible to adjust the camera model and the radar model in the simulation environment according to different models and parameters of the camera and the radar, so as to calibrate the camera and the radar through simulation, which is beneficial to saving calibration costs; at the same time, the coordinate transformation relationship between the camera model and the radar model can be accurately obtained, thereby realizing accurate calibration of the camera and the radar. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The preferred implementation scheme will be described below in a clear and understandable manner with reference to the accompanying drawings to further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of the present solution.

[0047] Figure 1 It is a schematic diagram of the overall process of an embodiment of the present invention;

[0048] Figure 2 is a flow chart of an embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of a two-dimensional code model of an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0051] In order to simplify the drawings, only the parts related to the present invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one of the parts with the same structure or function is schematically drawn or marked. In this article, "one" not only means "only one", but also means "more than one".

[0052] In scenarios such as autonomous driving or some location recognition tasks, such as robotic arm grasping, RGB images and radar depth image fusion processing are often required, so the coordinate relationship between the camera and the radar, that is, the external parameter, is crucial to the accuracy of the entire sensor system. If the accuracy of the camera and radar calibration is not enough, the accuracy of the entire system will be reduced. In addition, since the cameras and radar equipment involved in the calibration are relatively expensive, and different models of cameras and radars need to be calibrated separately, the cost of calibration will be high. The present application uses simulation to calibrate cameras and radars, which is convenient for adjusting the camera model and radar model in a simulation environment according to the model and parameters of the camera and radar, which is conducive to saving calibration costs. The technical solution of the present application will be described in detail below in conjunction with the accompanying drawings:

[0053] In one embodiment, the reference specification Figure 1 The present invention provides a camera and radar calibration method based on simulation, comprising the steps of:

[0054] S0. Build a simulation environment. Users can formulate the coordinate relationship of each component in the simulation environment to simulate the real environment and can be used for subsequent verification of the calibration results.

[0055] S1. Construct a camera model and a radar model based on the actual positional relationship between the camera and the radar in a simulation environment, and input simulation properties into the camera model and the radar model respectively to obtain image information under the camera model and radar data information under the radar model.

[0056] For example, add relevant position information parameters of the camera (for example, RGB camera) and radar (for example, lidar) to the urdf model file of the simulation system, build the camera model and radar model, and the relevant position information parameters are related to the actual position relationship of the camera and radar; and start the simulation environment, so that the information of the camera and radar can be observed in the visualization platform of the simulation system; at the same time, input simulation properties to the camera model and radar model to simulate the operation of the camera and radar.

[0057] S2. Add a feature model in the simulation environment, and the feature model is within the recognition range of the camera model and the radar model, so that the image information obtained by the camera model and the radar data information obtained by the radar model both contain data information of the feature model.

[0058] In order to realize the calibration of the camera and radar in the simulation environment, the present application adds feature models within the recognition range of the camera model and the radar model to be used for the recognition of the camera model and the radar model respectively. The present application does not limit the type of feature model, but the feature model should be an easily identifiable and specific or unique structure to improve the accuracy of recognition, such as a QR code model.

[0059] S3. Acquire image information containing the feature model in an optical coordinate system through a camera model, and acquire radar data information containing the feature model in a radar coordinate system through a radar model.

[0060] S4. Obtain a first coordinate transformation relationship between the optical coordinate system of the camera model and the camera coordinate system, and map the image information containing the feature model to the camera coordinate system based on the first coordinate transformation relationship, and identify the three-dimensional position information of the feature model in the camera coordinate system.

[0061] Typically, the optical coordinate system of a camera model is a right-handed coordinate system with the z-axis being perpendicular to the surface of the camera lens, and the x-axis and y-axis being parallel to the camera image plane. There is a fixed rotation transformation relationship between the optical coordinate system of the camera model and the camera coordinate system. After the present application obtains image information through a camera model in a simulation environment, the image information is based on the optical coordinate system of the camera model. It is necessary to first map the image information containing the feature model to the camera coordinate system according to the first coordinate transformation relationship between the optical coordinate system and the camera coordinate system.

[0062] Since the image information acquired by the camera model contains the feature model, by identifying the feature model, the three-dimensional position information of the feature model in the camera coordinate system can be further identified. The recognition of the feature model can be selected according to the specific type of the feature model. For example, when the feature model is a QR code model, the QR code model can be recognized according to the existing QR code recognition algorithm.

[0063] S5. Generate a feature template based on the feature model and the three-dimensional position information.

[0064] Feature templates can be created outside the simulation system and added to the simulation environment using scripts or plug-ins.

[0065] S6. Convert the radar data information containing the feature model into the camera coordinate system to obtain point cloud data of the feature model.

[0066] When image information containing a feature model within the recognition range is obtained through the camera model, radar data information containing the feature model can also be obtained through the radar model. The present application converts the radar data information containing the feature model into the camera coordinate system, and can obtain point cloud data corresponding to the feature model based on the camera coordinate system.

[0067] S7. Match the point cloud data with the feature template to obtain a second coordinate transformation relationship between the camera model and the radar model.

[0068] Since the reference coordinate system of the radar data information has been converted to the camera coordinate system as the starting point, the point cloud data about the feature model extracted from the radar data information is also based on the camera coordinate system; it can be seen from step S4 that the three-dimensional position information of the feature model also starts from the camera coordinate system. At this time, the point cloud data of the feature model and the feature template can be matched to obtain the coordinate conversion relationship between the camera model and the radar model.

[0069] However, the initial feature template and point cloud data will inevitably not overlap. This solution adjusts the coordinate position of the radar data information in the simulation environment to make the feature template and the point cloud data completely overlap. The second coordinate transformation relationship between the camera model and the radar model can be obtained from the adjustment parameters of the coordinate position of the radar data information, thereby realizing the calibration of the camera model and the radar model.

[0070] This solution constructs a simulation environment, and constructs a camera model and a radar model in the simulation environment based on the actual position relationship of the camera and the radar. This can facilitate the adjustment of the camera model and the radar model in the simulation environment according to the models and parameters of the camera and the radar, so as to realize the calibration of the camera and the radar through simulation, which is beneficial to saving the calibration cost. At the same time, by setting the feature model, the image information of the feature model in the optical coordinate system of the camera model is obtained, and based on the first coordinate transformation relationship between the optical coordinate system of the camera model and the camera coordinate system, the image information in the optical coordinate system of the camera model is mapped to the camera coordinate system, the three-dimensional position information of the feature model in the camera coordinate system is identified and the corresponding feature template is generated, and then the radar data information containing the feature model is converted to the camera coordinate system to obtain the point cloud data of the feature model, and by matching the point cloud data with the feature template, the coordinate transformation relationship between the camera model and the radar model can be obtained, thereby realizing the accurate calibration of the camera model and the radar model.

[0071] In one embodiment, based on the above-mentioned embodiment, the feature model of the present application uses a two-dimensional code model. By using the two-dimensional code model as the feature model, it is easy to identify the feature model, and then to calibrate the camera model and the radar model.

[0072] Preferably, the two-dimensional code model is composed of a plurality of two-dimensional code sub-models, and each two-dimensional code sub-model corresponds to a two-dimensional code.

[0073] Add feature models in the simulation environment, including:

[0074] The QR code model is constructed by configuring the ID information, size information and offset information of each QR code sub-model; and the QR code model is added to the simulation environment in the form of a script or a plug-in.

[0075] Since information can be stored in a QR code, by using a QR code model as a feature model and storing the model's ID information, size information, and offset information in the QR code, the three-dimensional information of the QR code model can be accurately identified after the QR code model is identified, thereby facilitating the accurate calibration of the camera model and the radar model. By setting the QR code model to be composed of a combination of several QR code sub-models, and each QR code sub-model has a different QR code style, the recognition accuracy of the QR code model can be improved. The present application does not limit the specific style and quantity of the QR code sub-models, and can be adjusted according to actual needs. For example, the present application binds four QR codes of different styles together to form a QR code model.

[0076] Reference Manual Attached Figure 3, add a two-dimensional code model 20 composed of several two-dimensional code sub-models to the simulation environment. The two-dimensional code model 20 is within the recognition range of the camera model 10. The two-dimensional code ID information, size information and offset in the two-dimensional code model 20 are configured through the two-dimensional code recognition algorithm, and the three-dimensional coordinate information of the two-dimensional code model can be observed in the visualization platform.

[0077] By adding the manufactured QR code model to the simulation environment, and pre-generating the coordinate transformation relationship between the camera optical coordinate system and the camera coordinate system, this coordinate transformation relationship is used as the first coordinate transformation relationship between the camera coordinate system and the optical coordinate system of the camera model in the simulation environment. Preferably, the three-dimensional position information of the feature model in the image information in the camera coordinate system is identified by the QR code recognition algorithm in the simulation environment. When the feature model uses the QR code model, the existing QR code recognition algorithm can be directly used to identify the QR code model without setting an additional model recognition algorithm, which facilitates calibration.

[0078] Furthermore, the ID information, size information and offset information of the produced QR code model can be configured in the QR code recognition algorithm, and then the QR code recognition algorithm can be started. The detection results of the QR code model under different configuration conditions can be observed in the visualization platform of the simulation environment. By comparing different detection results, the recognition accuracy of the feature model in the simulation environment can be further improved.

[0079] Furthermore, the three-dimensional position information of the feature model in the camera coordinate system is identified, including:

[0080] The first coordinate information of the center or mark point of the two-dimensional code model in the optical coordinate system is identified through the camera model; the second coordinate information of the first coordinate information in the camera coordinate system is obtained based on the first coordinate transformation relationship; the ID information, size information and offset information of each two-dimensional code sub-model in the two-dimensional code model are identified through the two-dimensional code recognition algorithm; the three-dimensional position information of the feature model in the camera coordinate system is obtained according to the second coordinate information and the ID information, size information and offset information of each two-dimensional code sub-model.

[0081] When identifying the three-dimensional position information of the feature model in the camera coordinate system, by identifying the first coordinate information of the center or preset mark point of the two-dimensional code model in the optical coordinate system, the second coordinate information in the camera coordinate system corresponding to the first coordinate information can be obtained based on the first coordinate conversion relationship between the optical coordinate system and the camera coordinate system. Based on the ID information, size information and offset information of the two-dimensional code sub-model identified by the two-dimensional code recognition algorithm, the three-dimensional position information of the two-dimensional code model in the camera coordinate system can be accurately calculated, thereby avoiding errors that are easily generated when using conventional feature models for coordinate conversion.

[0082] In one embodiment, based on the above-mentioned embodiment, the radar data information containing the feature model is converted into the camera coordinate system to obtain the point cloud data of the feature model, which specifically includes:

[0083] The reference coordinates of the radar data information are uniformly changed to start from the camera coordinate system; the point cloud data of the feature model remains unchanged.

[0084] In one embodiment, based on the above-mentioned embodiment, a feature template is generated based on the feature model and the three-dimensional position information, specifically including:

[0085] Obtain first position information corresponding to the center or marking point of the two-dimensional code model in the three-dimensional position information; take the first position information as the center or marking point, and generate a feature template according to the ID information, size information and offset information of each two-dimensional code sub-model, so that the parameter information of the feature template is the same as the feature model.

[0086] Reference Manual Attached Figure 2 , match the point cloud data and the feature template to obtain the second coordinate transformation relationship between the camera model and the radar model, specifically including:

[0087] S61. Generate a feature template based on the feature model and the three-dimensional position information, wherein the size parameters of the feature template are the same as those of the feature model; taking the feature model as a QR code model as an example, and taking the coordinate position of the QR code model as the center, generate a template to be matched with the same size parameters as the QR code model, i.e., the feature template. The feature template can be displayed on a visualization platform to facilitate subsequent matching.

[0088] S62. In the simulation environment, adjust the coordinate position of the radar data information converted to the camera coordinate system so that the point cloud data coincides with the feature template, and record the adjustment parameters of the coordinate position of the radar data information; display the point cloud data of the feature model through the visualization platform, and adjust the reference coordinate position of the radar data information until the point cloud data completely coincides with the feature template, and record the adjustment parameters of the coordinate position of the radar data information at this time. The adjustment parameters can be regarded as the second coordinate transformation relationship between the camera model and the radar model.

[0089] When calculating the coordinate transformation relationship between the radar model and the camera model, a feature template can be generated based on the parameter information of the feature model and its corresponding three-dimensional position information, thereby facilitating the matching of the three-dimensional position information and the point cloud data. By adjusting the coordinate position of the radar data information to make the point cloud data coincide with the feature template, the adjustment parameters of the radar data information can be determined, and the adjustment parameters can be regarded as the coordinate transformation relationship between the radar model and the camera model.

[0090] In one embodiment, the camera and radar calibration method based on simulation provided by the present invention further includes:

[0091] Information parameters of different sizes / different offset angles of the feature model are configured to recalibrate, and the second coordinate transformation relationship obtained by each calibration is compared to correct the second coordinate transformation relationship.

[0092] When the relative positions of the camera model and the radar model remain unchanged, only the size or angle information of the feature model is adjusted. By configuring the information parameters of different sizes / different offset angles of the feature model for recalibration, it is possible to verify whether the calibration results of the radar model and the camera model are accurate, thereby correcting the second coordinate transformation relationship.

[0093] In one embodiment, the simulation-based camera and radar calibration method provided by the present invention also includes: obtaining a third coordinate transformation relationship between the camera model and the radar model according to the actual position relationship between the camera and the radar, and comparing the second coordinate transformation relationship and the third coordinate transformation relationship to calibrate the calibration method.

[0094] After knowing the actual position relationship between the camera and the radar, the coordinate transformation relationship (i.e., the third coordinate transformation relationship) between the radar model and the camera model can be directly obtained based on the actual position relationship and the mapping between the simulation environment and the actual environment. By comparing the third coordinate transformation relationship with the second coordinate transformation relationship calculated by the above calibration algorithm, it is possible to verify whether the above calibration algorithm is accurate.

[0095] In one embodiment, the camera and radar calibration method based on simulation provided by the present invention further includes:

[0096] The three-dimensional position information and point cloud data are sent to the visualization platform, and the feature template is visualized.

[0097] The camera and radar calibration method based on simulation provided by the present invention also includes:

[0098] A parameter slider for adjusting the point cloud data of the radar model is encapsulated in the simulation environment, so that the relative posture of the camera model and the radar model can be adjusted by moving the parameter slider; the point cloud data is made to coincide with the feature template by moving the parameter slider. In order to facilitate the matching of three-dimensional position information and point cloud data, the present application can encapsulate a parameter slider for adjusting the reference coordinate system of the point cloud data. The parameter slider is displayed through a visualization platform, and can receive user operations to directly adjust the position of the reference coordinate system of the point cloud data. By moving the parameter slider, the point cloud data can be coincident with the feature template, and the posture change information of the radar model can be obtained, thereby obtaining the adjustment parameters of the reference coordinate system of the radar model.

[0099] In one embodiment, before obtaining the first coordinate conversion relationship between the image coordinate system and the optical coordinate system of the camera model, the method further includes:

[0100] Calibrate the internal parameters of the camera model through a calibration algorithm to obtain the corrected camera parameters of the camera model;

[0101] Restart the simulation environment with the corrected camera parameters.

[0102] Before calibrating the coordinate transformation relationship between the camera model and the radar model, that is, the external parameters, the internal parameters of the camera model are calibrated first. This can avoid the distortion of the image information subsequently obtained by the camera model from affecting the external parameter calibration and improve the accuracy of the calibration.

[0103] In one embodiment, based on the foregoing embodiment, the present application provides a specific calibration operation of a camera and a radar based on simulation, including:

[0104] 1. Model construction. This application uses an RGB camera and a livox_mid70 laser radar, and constructs a urdf model of the RGB camera and the livox_mid70 laser radar. The model construction information includes the relative position of the camera model and the radar model, visualized physical information, etc., and adds a plug-in for simulation. By adding simulation properties, the data of the camera and the laser radar can be simulated. This application also includes the construction of a QR code sdf model, with the QR code sdf model as a feature model. The present invention produces a QR code for simulation recognition. In view of the low accuracy of single QR code recognition, in order to improve the accuracy of camera model recognition of QR codes, the present invention binds four QR codes of different styles together as a complete QR code model.

[0105] 2. Calibration of the intrinsic parameters of the camera model. Since the camera has distortion, it is necessary to calibrate the intrinsic parameters of the camera model through a calibration algorithm before calibrating the external parameters of the radar model and the camera model, so as to correct the distortion of the camera and improve the accuracy of QR code recognition. This solution uses a black and white calibration plate with a checkerboard pattern commonly used in the prior art to calibrate the parameters of the camera model. During the calibration process, the position of the calibration plate is continuously moved so that a group of images are taken with the camera model in different postures. The image processing algorithm is used to detect the corner points of each checkerboard in this group of images respectively. The coordinates of these corner points can be used to calculate the intrinsic parameter matrix and distortion parameters of the camera model, so as to obtain the corrected camera parameters. Fill them into the startup simulation file of the camera model, restart the simulation environment, and the subsequent images collected by the camera model can be corrected camera images. The calibration algorithm is an existing intrinsic calibration algorithm, for example, Zhang's calibration algorithm, and the specific operation steps are: prepare calibration images: use the calibration plate to take a group of images at different positions and angles; extract corner point information: use the image processing algorithm to detect the corner points on the calibration plate, and extract their pixel coordinates; solve the intrinsic parameter matrix and the extrinsic parameter matrix: use the physical coordinates and image coordinates of the corner points on the calibration plate to solve the camera's intrinsic parameter matrix and extrinsic parameter matrix through mathematical methods; solve the distortion parameters: solve the camera's distortion parameters through the optimization algorithm; parameter optimization: use the Levenberg-Marquardt (LM) algorithm to iteratively optimize the parameters to improve the calibration accuracy. This application does not elaborate. It's just that the intrinsic parameter calibration process of the above-mentioned camera model is carried out under the simulation system.

[0106] 3. The image information containing the feature model obtained by the camera model is converted from the optical coordinate system to the camera coordinate system, and the image information containing the QR code model obtained by simulation is converted to the camera coordinate system, and the original information of the mapped image information and the camera parameter information are sent out. At the same time, the parameters of the QR code recognition algorithm are configured, the image information of the QR code model obtained by the RGB camera model is received and converted to the camera coordinate system, and the three-dimensional position information of the QR code model is recognized.

[0107] 4. Obtain the radar data information obtained by the lidar model, convert the radar data information containing the QR code model to the camera coordinate system starting from the camera model, obtain the point cloud data of the QR code model, and send the point cloud data in this camera coordinate system to the visualization platform.

[0108] 5. Construct a QR code template based on the QR code model and the three-dimensional position information in the simulation environment. The size parameters of the QR code template are the same as those of the QR code model.

[0109] 6. Encapsulate a parameter slider for adjusting the reference coordinate system of radar data information, which can move the point cloud data of the QR code model in the visualization platform.

[0110] 7. Adjust the parameter slider. When the point cloud data of the QR code template and the QR code model overlap, the adjustment parameters of the parameter slider are regarded as the coordinate transformation relationship between the radar model and the camera model.

[0111] The present application discloses a camera and radar calibration method based on simulation, and the calibration method is verified in a simulation system. The implementation of the method includes the construction of model files such as cameras and radars in the simulation system, and the construction of a two-dimensional code model. In the simulation system, the influence of camera model distortion on two-dimensional code recognition is eliminated by calibrating the internal parameters of the camera model. Subsequently, the optical coordinate system and camera coordinate system of the camera model are constructed, and the three-dimensional position information of the two-dimensional code model is obtained by using a two-dimensional code recognition algorithm. A two-dimensional code template is generated based on the two-dimensional code model and the three-dimensional position information, and then the point cloud data of the two-dimensional code model under the radar model is converted to the camera coordinate system of the camera model. Finally, by matching the point cloud data and the two-dimensional code template, the coordinate conversion relationship between the radar model and the camera model is obtained, thereby verifying the calibration results of the radar and camera in the real environment. The implementation and verification of the radar and camera calibration method in a simulation environment greatly facilitates the development process. In terms of expansion, in the simulation platform, with the help of simulation data, based on this solution, the calibration algorithms of various sensors can be tested, and the model positions of various sensors filled in by users in advance in the model files of various sensors can be compared with the coordinate transformation relationships of various sensors calibrated, so as to quickly implement the stability and reliability of the calibration algorithms of various sensors. I will not elaborate on this.

[0112] In one embodiment, the present application also provides a computer device, including a memory, a processor, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to implement the steps of the camera and radar calibration method in the above embodiment.

[0113] In one embodiment, the present application also provides a computer storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the camera and radar calibration method in the above embodiment.

[0114] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A camera and radar calibration method based on simulation, characterized in that: Includes steps: Constructing a camera model and a radar model based on the actual positional relationship between the camera and the radar in a simulation environment, and inputting simulation properties into the camera model and the radar model respectively; Adding a feature model in the simulation environment, wherein the feature model is a two-dimensional code model, and the feature model is within the recognition range of the camera model and the radar model; Acquire image information containing the feature model in an optical coordinate system through the camera model, and acquire radar data information containing the feature model in a radar coordinate system through the radar model; Acquire a first coordinate transformation relationship between the optical coordinate system and the camera coordinate system of the camera model, and map the image information containing the feature model to the camera coordinate system based on the first coordinate transformation relationship, identify the ID information, size information and offset information of each two-dimensional code sub-model in the two-dimensional code model through a two-dimensional code recognition algorithm, and identify the three-dimensional position information of the feature model in the camera coordinate system according to the ID information, size information and offset information of each two-dimensional code sub-model; generating a feature template based on the feature model and the three-dimensional position information; Converting the radar data information including the feature model to the camera coordinate system to obtain point cloud data of the feature model, specifically comprising: uniformly changing the reference coordinates of the radar data information to start from the camera coordinate system, and keeping the point cloud data corresponding to the feature model unchanged; Matching the point cloud data and the feature template to obtain a second coordinate transformation relationship between the camera model and the radar model specifically includes: adjusting the coordinate position of the radar data information converted to the camera coordinate system in the simulation environment so that the point cloud data coincides with the feature template, and recording adjustment parameters of the coordinate position of the radar data information, where the adjustment parameters are the second coordinate transformation relationship.

2. The camera and radar calibration method based on simulation according to claim 1, characterized in that: The two-dimensional code model is composed of a plurality of two-dimensional code sub-models, and each of the two-dimensional code sub-models corresponds to a two-dimensional code; The adding of feature models in the simulation environment specifically includes: Constructing the two-dimensional code model by configuring the ID information, the size information and the offset information of each of the two-dimensional code sub-models; The two-dimensional code model is added to the simulation environment in the form of a script or a plug-in.

3. The camera and radar calibration method based on simulation according to claim 2, characterized in that: The step of identifying the three-dimensional position information of the feature model in the camera coordinate system further includes: Identify first coordinate information of the center or mark point of the two-dimensional code model in the optical coordinate system through the camera model; Obtaining second coordinate information of the first coordinate information in the camera coordinate system based on the first coordinate transformation relationship; The three-dimensional position information of the two-dimensional code model in the camera coordinate system is obtained according to the second coordinate information and the ID information, the size information and the offset information of each of the two-dimensional code sub-models.

4. The camera and radar calibration method based on simulation according to claim 3, characterized in that: The generating of the feature template based on the feature model and the three-dimensional position information specifically includes: Acquire first position information corresponding to the center or marking point of the two-dimensional code model in the three-dimensional position information; The first position information is used as the center or marking point of the feature template, and the feature template is generated according to the ID information, the size information and the offset information of each of the two-dimensional code sub-models, so that the parameter information of the feature template is the same as that of the feature model.

5. The camera and radar calibration method based on simulation according to any one of claims 1 to 4, characterized in that: Also includes: Information parameters of different sizes / different offset angles of the feature model are configured to recalibrate, and the second coordinate transformation relationship obtained by each calibration is compared to correct the second coordinate transformation relationship.

6. The camera and radar calibration method based on simulation according to any one of claims 1 to 4, characterized in that: Also includes: According to the actual positional relationship between the camera and the radar, a third coordinate transformation relationship between the camera model and the radar model is obtained, and the second coordinate transformation relationship and the third coordinate transformation relationship are compared to calibrate the calibration method.

7. The camera and radar calibration method based on simulation according to any one of claims 1 to 4, characterized in that: Before obtaining the first coordinate transformation relationship between the camera coordinate system and the optical coordinate system of the camera model, the method further includes: Calibrate the internal parameters of the camera model by using a calibration algorithm to obtain corrected camera parameters of the camera model; The simulation environment is restarted according to the corrected camera parameters to reduce errors in subsequent acquisition of the image information by the camera model.

8. A computer device comprising a memory, a processor and a computer program / instructions stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the steps of the simulation-based camera and radar calibration method described in any one of claims 1-7.

9. A computer storage medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the simulation-based camera and radar calibration method described in any one of claims 1-7 are implemented.

Images