Sensing equipment calibration method and device and computer equipment

Abstract

The invention relates to a sensing equipment calibration method and device and computer equipment. The method comprises the steps of obtaining target point cloud data of a target area acquired by radar equipment on a target vehicle and target image data of the target area acquired by image acquisition equipment on the target vehicle, and adopting a fusion alignment model to perform fusion alignment on the target point cloud data and the target image data according to the target point cloud data and the target image data; the fusion alignment model is obtained by training an initial model by adopting a training data set, the training data set is obtained by processing a public automatic driving data set according to configuration information of a target vehicle, and a calibration result between the radar equipment and the image acquisition equipment is determined according to the configuration information of the target vehicle; the public automatic driving data set comprises calibrated standard point cloud data and standard image data corresponding to the standard point cloud data. By adopting the method, the radar equipment and the image acquisition equipment can be calibrated in real time, so that the calibration accuracy of the sensing equipment is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a sensor device calibration method, device and computer equipment, which can be applied to intelligent vehicles. Background Art

[0002] In order to ensure the reliability of vehicle autonomous driving, it is necessary to calibrate the sensors in the vehicle (for example, lidar and camera) so that the environmental data obtained by each sensor can be more matched.

[0003] In the existing sensor equipment calibration method, generally, each sensor is used to collect data from a pre-configured calibration board, and then each sensor is calibrated according to the data collection results of each sensor.

[0004] However, the existing sensor device calibration method uses a pre-configured calibration board to perform sensor calibration, which cannot perform real-time calibration during vehicle operation, thereby reducing the accuracy of sensor device calibration. Summary of the invention

[0005] Based on this, it is necessary to provide a sensor device calibration method, device and computer equipment that can improve the calibration accuracy of the sensor device in response to the above technical problems.

[0006] In a first aspect, the present application provides a sensor device calibration method, comprising:

[0007] Acquire target point cloud data of the target area collected by the radar device on the target vehicle, and target image data of the target area collected by the image acquisition device on the target vehicle;

[0008] Using the fusion alignment model, the calibration results between the radar device and the image acquisition device are determined based on the target point cloud data and the target image data;

[0009] Among them, the fusion alignment model is obtained by training the initial model with the training data set; the training data set is obtained by processing the public autonomous driving data set according to the configuration information of the target vehicle; the public autonomous driving data set contains calibrated standard point cloud data and standard image data corresponding to the standard point cloud data.

[0010] In one embodiment, the configuration information includes vehicle size information and field of view angle information of a radar device; and processing a public autonomous driving dataset according to the configuration information of the target vehicle includes:

[0011] The translation parameters are determined according to the vehicle size information; the rotation angle parameters are determined according to the field of view angle information; and the public autonomous driving data set is enhanced according to the translation parameters and the rotation angle parameters to obtain a training data set.

[0012] In one embodiment, data enhancement processing is performed on a public autonomous driving dataset according to a translation parameter and a rotation angle parameter to obtain a training dataset, including:

[0013] Determine the rotation-translation matrix according to the translation parameters and the rotation angle parameters; multiply each standard data in the public autonomous driving data set by the rotation-translation matrix to obtain the extended data corresponding to each standard data; use each standard data and the extended data corresponding to each standard data as the training data set.

[0014] In one embodiment, the training data set includes sample image data and sample point cloud data corresponding to the sample image data; the initial model includes an initial image branch, an initial point cloud branch, and an initial fusion alignment branch, and the fusion alignment model is trained in the following manner:

[0015] The sample image data is input into the initial image branch to obtain the sample image features corresponding to the sample image data; the image loss is determined according to the sample image data and the sample image features; the sample point cloud data is input into the initial point cloud branch to obtain the sample point cloud features corresponding to the sample point cloud data; the point cloud loss is determined according to the sample point cloud data and the sample point cloud features; the reprojection error and the feature matching loss are determined according to the sample image features and the sample point cloud features; the initial model is trained according to the image loss, the point cloud loss, the reprojection error and the feature matching loss to obtain the fusion alignment model.

[0016] In one embodiment, determining the reprojection error and the feature matching loss according to the sample image features and the sample point cloud features includes:

[0017] Each point cloud feature point in the sample point cloud feature is projected onto the image plane corresponding to the sample image feature to obtain the projection point corresponding to each point cloud feature point; the reprojection error is determined based on the position error between each image feature point in the sample image feature and the projection point corresponding to each point cloud feature point; the feature matching loss is determined based on the feature difference between the sample image feature and the sample point cloud feature.

[0018] In one embodiment, the fusion alignment model includes a target image branch, a target point cloud branch, and a target fusion alignment branch; the fusion alignment model is used to determine the calibration result between the radar device and the image acquisition device according to the target point cloud data and the target image data, including:

[0019] Based on the target image branch, feature extraction is performed on the target image data to obtain target image features; based on the target point cloud branch, feature extraction is performed on the target point cloud data to obtain target point cloud features; based on the target fusion alignment branch, the calibration result between the radar device and the image acquisition device is determined according to the target point cloud features and the target image features.

[0020] In one embodiment, the target image branch includes a convolutional neural network and a fully connected network; based on the target image branch, feature extraction is performed on the target image data to obtain target image features, including:

[0021] The target image data is input into the convolutional neural network for encoding processing to obtain the encoded image features; the encoded image features are input into the fully connected network for decoding processing to obtain the target image features.

[0022] In one embodiment, the target fusion alignment branch includes a feature fusion network, an encoder, and a multi-layer perceptron; based on the target fusion alignment branch, according to the target point cloud features and the target image features, the calibration result between the radar device and the image acquisition device is determined, including:

[0023] The target point cloud features and the target image features are input into the feature fusion network for fusion processing to obtain fused environment features; the fused environment features and the position coding information corresponding to the fused environment features are input into the encoder for encoding processing to obtain coded environment features; the coded environment features are input into the multi-layer perceptron for decoding processing to obtain the calibration results between the radar equipment and the image acquisition equipment.

[0024] In a second aspect, the present application also provides a sensor device calibration device, comprising:

[0025] A data acquisition module, used to acquire target point cloud data of the target area collected by the radar device on the target vehicle, and target image data of the target area collected by the image acquisition device on the target vehicle;

[0026] A result determination module is used to determine the calibration result between the radar device and the image acquisition device according to the target point cloud data and the target image data by using a fusion alignment model;

[0027] Among them, the fusion alignment model is obtained by training the initial model with the training data set; the training data set is obtained by processing the public autonomous driving data set according to the configuration information of the target vehicle; the public autonomous driving data set contains calibrated standard point cloud data and standard image data corresponding to the standard point cloud data.

[0028] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0029] Acquire target point cloud data of the target area collected by the radar device on the target vehicle, and target image data of the target area collected by the image acquisition device on the target vehicle;

[0030] Using the fusion alignment model, the calibration results between the radar device and the image acquisition device are determined based on the target point cloud data and the target image data;

[0031] Among them, the fusion alignment model is obtained by training the initial model with the training data set; the training data set is obtained by processing the public autonomous driving data set according to the configuration information of the target vehicle; the public autonomous driving data set contains calibrated standard point cloud data and standard image data corresponding to the standard point cloud data.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0033] Acquire target point cloud data of the target area collected by the radar device on the target vehicle, and target image data of the target area collected by the image acquisition device on the target vehicle;

[0034] Using the fusion alignment model, the calibration results between the radar device and the image acquisition device are determined based on the target point cloud data and the target image data;

[0035] Among them, the fusion alignment model is obtained by training the initial model with the training data set; the training data set is obtained by processing the public autonomous driving data set according to the configuration information of the target vehicle; the public autonomous driving data set contains calibrated standard point cloud data and standard image data corresponding to the standard point cloud data.

[0036] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0037] Acquire target point cloud data of the target area collected by the radar device on the target vehicle, and target image data of the target area collected by the image acquisition device on the target vehicle;

[0038] Using the fusion alignment model, the calibration results between the radar device and the image acquisition device are determined based on the target point cloud data and the target image data;

[0039] Among them, the fusion alignment model is obtained by training the initial model with the training data set; the training data set is obtained by processing the public autonomous driving data set according to the configuration information of the target vehicle; the public autonomous driving data set contains calibrated standard point cloud data and standard image data corresponding to the standard point cloud data.

[0040] The above-mentioned sensor equipment calibration method, device and computer equipment obtain the target point cloud data of the target area collected by the radar equipment on the target vehicle, and the target image data of the target area collected by the image acquisition equipment on the target vehicle; then, the calibration result between the radar equipment and the image acquisition equipment is determined according to the target point cloud data and the target image data by using the fusion alignment model obtained by training the initial model based on the training data set; wherein the training data set is obtained by processing the public automatic driving data set according to the configuration information of the target vehicle. Compared with the related art, in which each sensor is calibrated by collecting data from the configured calibration board, the above-mentioned method uses the training data set related to the target vehicle to train the fusion alignment model, and uses the fusion alignment model according to the target point cloud data and the target image data obtained in real time, so as to ensure the accuracy of the sensor equipment calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A diagram of an application environment of a sensor device calibration method in one embodiment;

[0043] Figure 2 A schematic diagram of a flow chart of a sensor device calibration method in one embodiment;

[0044] Figure 3 A schematic diagram of a process for determining a training data set in one embodiment;

[0045] Figure 4 A schematic diagram of a process of model training in one embodiment;

[0046] Figure 5A A schematic diagram of a process for determining a calibration result in one embodiment;

[0047] Figure 5B is a schematic diagram of a target image branch in one embodiment;

[0048] Figure 5C is a schematic diagram of a target point cloud branch in one embodiment;

[0049] Fig. 6A A schematic diagram of a flow chart for determining a calibration result in another embodiment;

[0050] Figure 6B is a schematic diagram of a target fusion alignment branch in one embodiment;

[0051] Figure 7 is a flow chart of a sensor device calibration method in another embodiment;

[0052] Figure 8 is a structural block diagram of a sensor device calibration device in one embodiment;

[0053] Fig. 9 It is a structural block diagram of a sensor device calibration device in another embodiment;

[0054] Fig.10 It is a structural block diagram of a sensor device calibration device in another embodiment;

[0055] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] In order to ensure the reliability of vehicle autonomous driving, it is necessary to calibrate the sensors in the vehicle (for example, lidar and camera) so that the environmental data obtained by each sensor can be more matched.

[0058] The existing sensor equipment calibration method generally uses each sensor to collect data on a pre-configured calibration board, and then calibrates each sensor based on the data collection results of each sensor. However, using the existing sensor equipment calibration method, the sensor calibration is performed using a pre-configured calibration board, which cannot be calibrated in real time during vehicle operation, reducing the accuracy of sensor equipment calibration.

[0059] Based on this, the sensor device calibration method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown in the figure, the radar device 102 and the image acquisition device 104 communicate with the vehicle terminal 106 through the network. For example, the vehicle terminal 106 obtains the target point cloud data of the target area acquired by the radar device 102 on the target vehicle, and the target image data of the target area acquired by the image acquisition device 104 on the target vehicle; then, the fusion alignment model is used to determine the calibration result between the radar device and the image acquisition device according to the target point cloud data and the target image data.

[0060] In an exemplary embodiment, Figure 2 As shown, a sensor device calibration method is provided, which is applied to Figure 1 The vehicle-mounted terminal in the example is used for explanation, and the specific steps include:

[0061] S201, acquiring target point cloud data of a target area collected by a radar device on a target vehicle, and target image data of a target area collected by an image acquisition device on the target vehicle.

[0062] Among them, the target vehicle is an intelligent vehicle with a sensor equipment calibration requirement; the target area is a real area used for sensor equipment calibration, for example, it can be the area passed by the target vehicle during the formation process; the radar equipment is an electronic device that uses electromagnetic waves to detect targets, which can be but is not limited to laser radar, etc.; the target point cloud data is the three-dimensional point cloud data corresponding to the target area; the image acquisition device is a device that can acquire images, which can be but is not limited to a camera and the like; the target image data is the image data corresponding to the target area.

[0063] Optionally, when a sensor device identification event is detected, a laser may be sent to the target area by a radar device deployed on the target vehicle to obtain target point cloud data of the target area; at the same time, target image data of the target area may be obtained by an image acquisition device deployed on the target vehicle. The sensor device identification event may be an automatically generated event set according to a preset calibration interval, or an event actively triggered by the driver.

[0064] It can be understood that in order to ensure the comprehensiveness of the acquisition of target image data, multiple image acquisition devices can be deployed on the target vehicle to simultaneously acquire target image data of multiple angles for the target area; for example, 6 cameras can be deployed around the target vehicle to acquire 6 target image data for the target area at the same time.

[0065] S202, using a fusion alignment model, and determining a calibration result between the radar device and the image acquisition device according to the target point cloud data and the target image data.

[0066] Among them, the fusion alignment model is obtained by training the initial model with the training data set; the training data set is obtained by processing the public autonomous driving data set according to the configuration information of the target vehicle; the public autonomous driving data set can be the data of various scenes that the vehicle passes through during autonomous driving, for example, the public autonomous driving data set can be the NuScenes data set.

[0067] The public autonomous driving dataset contains calibrated standard point cloud data and standard image data corresponding to the standard point cloud data; the calibration results are used to characterize the relative position and posture relationship between the radar device and the image acquisition device, which may include but are not limited to rotation and translation parameters.

[0068] Optionally, after acquiring the target point cloud data and the target image data, the target point cloud data and the target image data can be simultaneously input into a fusion alignment model, and the fusion alignment model extracts point cloud features from the target point cloud data and image features from the target image data; subsequently, the fusion alignment model can output the calibration results between the radar device and the image acquisition device based on the point cloud features, image features and model parameters.

[0069] Exemplarily, the point cloud data A collected by the lidar and the 6 image data B collected by the 6 cameras can be simultaneously input into the fusion alignment model, and the fusion alignment model extracts the point cloud feature A' from the point cloud data A, and extracts the 6 image features B' from the 6 image data B; then, the fusion alignment model can output the calibration result between the radar device and the image acquisition device based on the point cloud feature A', the 6 image features B' and the model parameters.

[0070] In the above sensor device calibration method, the target point cloud data of the target area collected by the radar device on the target vehicle and the target image data of the target area collected by the image acquisition device on the target vehicle are obtained; then, the calibration result between the radar device and the image acquisition device is determined according to the target point cloud data and the target image data by using the fusion alignment model obtained by training the initial model based on the training data set; wherein the training data set is obtained by processing the public autonomous driving data set according to the configuration information of the target vehicle. Compared with the related art, in which each sensor is calibrated by collecting data from the configured calibration board, the above method uses the training data set related to the target vehicle to train the fusion alignment model, and uses the fusion alignment model to obtain the target point cloud data and the target image data, so as to perform real-time calibration on the radar device and the image acquisition device, thereby ensuring the accuracy of the sensor device calibration.

[0071] It is understandable that the external parameters between the radar device and the image acquisition device in the public autonomous driving data set are calibrated and fixed. Although there is a lot of scene data, there is less training data for model training, that is, less true value data. Based on this, on the basis of the above embodiment, in this embodiment, the configuration information includes vehicle size information and field of view angle information of the radar device; further, an optional method for determining the training data set is provided, such as Figure 3 As shown, the specific steps include:

[0072] S301, determining translation parameters according to vehicle size information.

[0073] The vehicle dimension information is the dimension information of each structure in the vehicle, which may include but is not limited to the dimension information of the rigid loading device related to the radar equipment in the vehicle; the translation parameter is used to characterize the offset of the radar equipment in the radar coordinate system.

[0074] Optionally, in order to enrich the training data set based on the public autonomous driving data set, more external parameter training data can be constructed by applying a fixed transformation parameter to the coordinate system corresponding to the radar device. Furthermore, in order to ensure the rationality of the transformation parameter, the vehicle size information can be input into the trained first parameter determination model, and the first parameter determination model determines the translation parameters in the radar coordinate system based on the vehicle size information and the model parameters.

[0075] S302: Determine a rotation angle parameter according to the field of view angle information.

[0076] The field of view angle information is used to characterize the field of view of the radar device; and the rotation angle parameter is used to characterize the rotation angle of the radar device in the radar coordinate system.

[0077] Optionally, the field of view angle information may be input into a trained second parameter determination model, and the second parameter determination model determines the rotation angle parameters in the radar coordinate system according to the field of view angle information and model parameters.

[0078] S303: Perform data enhancement processing on the public autonomous driving data set according to the translation parameters and the rotation angle parameters to obtain a training data set.

[0079] Optionally, for each set of standard data in the public autonomous driving data set, data enhancement processing can be performed on the set of standard data using translation parameters and rotation angle parameters; then, the training data set is determined based on the data enhancement results of each set of standard data.

[0080] Exemplarily, the translation parameters and rotation angle parameters can be collected to translate and rotate the radar coordinate system to obtain a new radar coordinate system; then, for each set of standard data in the public autonomous driving data set, the enhanced data of the set of standard data in the new radar coordinate system can be determined, and the standard data and the enhanced data can be used as training data sets.

[0081] In an embodiment of the present application, data enhancement processing is performed on a public autonomous driving data set through vehicle size information and field of view angle information of a radar device to obtain a training data set, thereby ensuring the richness of the training data set.

[0082] In order to ensure the accuracy of the training data set, based on the above embodiment, in this embodiment, another optional method for determining the training data set is provided, specifically, determining the rotation translation matrix according to the translation parameters and the rotation angle parameters; multiplying each standard data in the public autonomous driving data set with the rotation translation matrix to obtain the extended data corresponding to each standard data; and using each standard data and the extended data corresponding to each standard data as the training data set.

[0083] Among them, standard data refers to the data in the public autonomous driving data set; the rotation and translation matrix is ​​the matrix used to process the standard data; and the extended data is the data expanded based on the standard data.

[0084] Optionally, a rotation and translation matrix may be constructed according to the parameters of each axis in the translation parameters and the parameters of each axis in the rotation angle parameters.

[0085] For example, the translation parameters include x, y and z, and the rotation angle parameters include , and In the case of , you can refer to the following formula (1), according to the translation parameter x and the rotation angle parameter , construct a first rotation matrix around the X axis. Among them, R x is the first rotation matrix.

[0086] (1)

[0087] Refer to the following formula (2), according to the translation parameter y and the rotation angle parameter , construct a second rotation matrix around the Y axis. Among them, R y is the second rotation matrix.

[0088] (2)

[0089] Refer to the following formula (3), according to the translation parameter z and the rotation angle parameter , construct a third rotation matrix around the Z axis. Among them, Rz is the third rotation matrix.

[0090] (3)

[0091] Furthermore, the final rotation matrix R can be constructed according to the first rotation matrix, the second rotation matrix and the third rotation matrix, that is, .

[0092] Accordingly, referring to the following formula (4), the translation matrix t can be constructed based on the translation parameters x, y and z; then, referring to the following formula (5), the rotation matrix R and the translation matrix t are used to construct the rotation translation matrix .

[0093] (4)

[0094] (5)

[0095] After determining the rotation and translation matrix, for the standard point cloud data in any standard data in the public autonomous driving data set, the standard point cloud data can be multiplied by the rotation and translation matrix to obtain the extended point cloud data corresponding to the standard point cloud data. For example, assuming that the standard point cloud data is , then the extended point cloud data is ; Subsequently, the extended point cloud data and the standard image data in the standard data can be used as extended data corresponding to the standard data; finally, each standard data and the extended data corresponding to each standard data can be directly used as a training data set.

[0096] In an embodiment of the present application, by multiplying each standard data in the public autonomous driving data set with the rotation and translation matrix respectively, the extended data corresponding to each standard data is obtained, thereby ensuring the accuracy of the training data set.

[0097] In order to ensure the reliability of the fusion alignment model, based on the above embodiment, in this embodiment, the training data set includes sample image data and sample point cloud data corresponding to the sample image data; the initial model includes an initial image branch, an initial point cloud branch and an initial fusion alignment branch; further, an optional method for model training is provided, such as Figure 4 As shown, the specific steps include:

[0098] S401, inputting sample image data into an initial image branch to obtain sample image features corresponding to the sample image data.

[0099] Among them, the sample image data is the image data in the training data set; the initial image branch is an untrained image feature extraction branch; and the sample image features are the image features extracted by the initial image branch.

[0100] It is understandable that in order to ensure the convenience of model training, the sample image data and sample point cloud data in the sample training set can be labeled before the sample training set is used to train the initial model. For example, the sample point cloud data and sample image data corresponding to the labeled objects in the training data set can be connected and aligned so that the identification information of the same object in the sample point cloud data is consistent with the identification information in the sample image data.

[0101] Optionally, for any group of data in the training data set, sample image data in the group of data may be input into the initial image branch, and the initial image branch performs feature extraction processing on the sample image data to obtain sample image features.

[0102] S402: Determine image loss according to sample image data and sample image features.

[0103] Among them, the image loss is the loss generated when the initial image branch extracts image features.

[0104] Optionally, the following formula (6) may be referred to, and the cross entropy loss function may be used to calculate the image loss according to the sample image data and the sample image features. Where, L I is the image loss; N is the number of samples; C is the number of pixel categories; y i，c is the data corresponding to the c-type pixel in the i-th sample image data; p i，c is the data corresponding to the c-type pixel in the i-th sample image feature.

[0105] (6)

[0106] S403, inputting the sample point cloud data into the initial point cloud branch to obtain sample point cloud features corresponding to the sample point cloud data.

[0107] Among them, the sample point cloud data is the point cloud data in the training data set; the initial point cloud branch is the untrained point cloud feature extraction branch; and the sample point cloud features are the point cloud features extracted by the initial point cloud branch.

[0108] Optionally, for any group of data in the training data set, the sample point cloud data in the group of data can be input into the initial point cloud branch, and the initial point cloud branch performs feature extraction processing on the sample point cloud data to obtain sample point cloud features.

[0109] S404: Determine the point cloud loss according to the sample point cloud data and the sample point cloud features.

[0110] Among them, the point cloud loss is the loss generated when the initial point cloud branch extracts point cloud features.

[0111] Optionally, referring to the above step S402, a cross entropy loss function may be used to calculate the point cloud loss L according to the sample point cloud data and the sample point cloud features. c ; Then, in order to prevent the model from overfitting, we can refer to the following formulas (7) and (8) and add the L2 regularization term to limit the scale of the network parameters, thereby obtaining the point cloud loss L C .in, is a hyperparameter that controls the regularization strength; W k is the weight parameter of the kth layer of the model; L reg is the L2 regularization term.

[0112] (7)

[0113] (8)

[0114] S405, determining a reprojection error and a feature matching loss according to the sample image features and the sample point cloud features.

[0115] Among them, the reprojection error refers to the error between the projection of a three-dimensional space point onto the image plane and the pixel point in the image; the feature matching loss is the difference between the sample image features and the sample point cloud features.

[0116] It can be understood that since there are differences in width and height between the sample image features and the sample point cloud features, in order to ensure the reliability of the difference determination, the larger value of the width and height of the two can be used as a reference. On the basis of aligning the elements in the features, the smaller features can be filled with 0 elements to ensure the consistency in size between the sample image features and the sample point cloud features.

[0117] Optionally, after the sample image features and the sample point cloud features are determined, the reprojection error and the feature matching loss may be determined according to the difference between the sample point cloud features and the sample point cloud features.

[0118] S406, training the initial model according to the image loss, point cloud loss, reprojection error and feature matching loss to obtain a fusion alignment model.

[0119] Optionally, the initial image branch in the initial model can be supervised and trained using image loss, the initial point cloud branch in the initial model can be supervised and trained using point cloud loss, and the initial fusion alignment branch in the initial model can be trained using reprojection error and feature matching loss to obtain a fusion alignment model.

[0120] Exemplarily, hyperparameters for controlling the reprojection error and the feature matching loss can be pre-set; then, the product of the hyperparameter corresponding to the reprojection error and the reprojection error is taken as the first error value, and the product of the hyperparameter corresponding to the feature matching loss and the feature matching loss is taken as the second error value; finally, the sum of the first error value and the second error value is taken as the fusion alignment loss L F .

[0121] Furthermore, the weight hyperparameter S corresponding to the image loss can be pre-set I , weight hyperparameter S corresponding to the point cloud loss C The weight hyperparameter S corresponding to the fusion alignment loss F , and then get the model loss function L=S I L I +S C L C +S F L F ; Then, based on the model loss function, the initial model is trained with the goal of minimizing the loss to obtain a fusion alignment model.

[0122] In an embodiment of the present application, the initial model is trained according to image loss, point cloud loss, reprojection error and feature matching loss to obtain a fusion alignment model, which can ensure the reliability of model training.

[0123] In order to ensure the accuracy of loss determination, based on the above embodiments, in this embodiment, an optional method of loss determination is provided, specifically, each point cloud feature point in the sample point cloud feature is projected onto the image plane corresponding to the sample image feature to obtain the projection point corresponding to each point cloud feature point; the reprojection error is determined according to the position error between each image feature point in the sample image feature and the projection point corresponding to each point cloud feature point; the feature matching loss is determined according to the feature difference between the sample image feature and the sample point cloud feature.

[0124] The point cloud feature points are feature points in the sample point cloud features; the projection points are points obtained by projecting the point cloud feature points onto the image plane.

[0125] Optionally, referring to the following formula (9), each point cloud feature point in the sample point cloud feature can be projected onto the image plane where the sample image feature is located to obtain the projection point corresponding to each point cloud feature point; then, the reprojection error L is determined based on the position error between each image feature point in the sample image feature and the projection point corresponding to each point cloud feature point. reproj .in, is the i-th point cloud feature point; is the i-th image feature point; π represents the operation of projecting the point cloud feature point onto the plane where the image is located; T is the calibration extrinsic parameter between the radar device and the image acquisition device.

[0126] (9)

[0127] Further, referring to the following formula (10), a deep point cloud feature extraction network and a deep image feature extraction network can be used to perform deep feature extraction on sample image features and sample point cloud features; then, based on the feature difference between the point cloud features and the image features after deep feature extraction, the feature matching loss L is determined: feat Among them, f lidar is the deep point cloud feature extraction network; f image It is a deep image feature extraction network.

[0128] (10)

[0129] In an embodiment of the present application, the reprojection error is determined by projecting onto an image plane corresponding to the sample image feature, and the feature matching loss is determined based on the feature difference between the sample image feature and the sample point cloud feature, thereby ensuring the accuracy of the loss determination.

[0130] In order to ensure the accuracy of the calibration results, based on the above embodiments, in this embodiment, the fusion alignment model includes a target image branch, a target point cloud branch and a target fusion alignment branch; further, an optional method for determining the calibration results is provided, such as Figure 5A As shown, the specific steps include:

[0131] S501, based on the target image branch, extract features of the target image data to obtain target image features.

[0132] Among them, the target image branch is the branch for extracting image features in the fusion alignment model; the target image features are the image features extracted from the target image data.

[0133] Optionally, the target image branch may be used to perform sampling processing on the target image data to obtain target image features.

[0134] Exemplarily, when the target image branch includes a convolutional neural network and a fully connected network, the target image data is input into the convolutional neural network for encoding processing to obtain encoded image features; the encoded image features are input into the fully connected network for decoding processing to obtain target image features.

[0135] Among them, the convolutional neural network can use EfficientNet, which is a neural network that increases the width, depth and resolution of the network at the same time, and can achieve high accuracy and low model complexity. The fully connected network (FCN) is a neural network commonly used in image segmentation tasks. It is based on the traditional convolutional network, removes the last fully connected layer, and uses deconvolution operations to upsample the feature map to restore it to the same size as the input image. FCN has higher flexibility and effectiveness in processing image data and is used for pixel-level image segmentation problems.

[0136] For example, refer to Figure 5B , the target image branch can be input into the multi-layer convolutional neural network EfficientNet to encode the image features and obtain the encoded image features; then, the encoded image features are input into the multi-layer FCN for decoding, and the image features output by the last layer of FCN are used as the target image features.

[0137] S502, based on the target point cloud branch, feature extraction is performed on the target point cloud data to obtain target point cloud features.

[0138] Among them, the target point cloud branch is a branch for extracting point cloud features in the fusion alignment model. For example, the target point cloud branch can be a point cloud feature extraction model pointnet++; the target point cloud feature is a point cloud feature extracted from the target point cloud data.

[0139] Optional, reference Figure 5C The target point cloud branch includes at least two cascaded feature extraction networks, each of which includes a sampling grouping layer and a point cloud feature extraction layer. For each feature extraction network, the sampling grouping layer of the feature extraction network is used to sample the input data of the feature extraction network to obtain each local point cloud group; the point cloud feature extraction layer of the feature extraction network is used to extract features from each local point cloud group to obtain the output data of the feature extraction network.

[0140] For example, the sampling grouping layer can use the Farthest Point Sampling (FPS) algorithm to sample the point cloud data. The algorithm constructs a sampling point set by iteratively selecting the point farthest from the selected point to ensure that the sampling points can evenly cover the entire point cloud; then, the ball query grouping algorithm is used to group the points around the sampling point. That is, given a center point and a radius, all points within the radius are divided into the same group to form a local point cloud group.

[0141] Furthermore, in each local point cloud group, the point cloud extraction network PointNet is used as a local feature learner for feature extraction, wherein PointNet is a multi-layer perceptron that can extract features from each point and aggregate the features of the point cloud feature points into features corresponding to the local point cloud group through the maximum pooling layer.

[0142] It can be understood that in the target point cloud branch, the input data of the first feature extraction network is the target point cloud data, the input data of any other feature extraction network is the output data of the previous feature extraction network, and the output data of the last feature extraction network is the target point cloud feature.

[0143] S503, based on the target fusion alignment branch, according to the target point cloud features and the target image features, determine the calibration result between the radar device and the image acquisition device.

[0144] Optionally, the target point cloud features and the target image features may be input into the target fusion alignment branch to obtain a calibration result between the radar device and the image acquisition device.

[0145] In an embodiment of the present application, the target point cloud data and the target image data are processed by adopting a target image branch, a target point cloud branch and a target fusion alignment branch to obtain a calibration result, which can ensure the accuracy of the calibration result.

[0146] In order to further ensure the accuracy of the calibration results, based on the above embodiments, in this embodiment, the target fusion alignment branch includes a feature fusion network, an encoder and a multi-layer perceptron; further, another optional method for determining the calibration results is provided, such as Fig. 6A As shown, the specific steps include:

[0147] S601, input the target point cloud features and the target image features into the feature fusion network for fusion processing to obtain fused environment features.

[0148] Among them, the feature fusion network is a network that fuses the target point cloud features and the target image features; the fused environment features are the features obtained after the fusion process.

[0149] Optional, reference Figure 6B , the target point cloud features and target image features can be input into the feature fusion network for fusion processing to obtain the fused environment features.

[0150] S602, inputting the fused environment feature and the position coding information corresponding to the fused environment feature into an encoder for coding processing to obtain the coded environment feature.

[0151] Among them, the encoded environment features are the environment features output by the encoder; the position encoding information is the position information of each feature in the fused feature.

[0152] Optional, reference Figure 6B First, the position encoding information corresponding to the fused environment feature is injected into the fused environment feature to obtain the fused environment feature with added position information; then, the fused environment feature with added position information is sent to the Transformer encoder for self-attention calculation to obtain the encoded environment feature.

[0153] The encoder layer of Transformer is composed of multiple identical layers stacked together, each of which contains a multi-head self-attention layer. The multi-head self-attention layer is the core part of Transformer, which allows the model to focus on information from different positions when processing the input sequence. That is, the multi-head self-attention mechanism divides the input sequence into multiple "heads", and each head independently learns different attention weights, thereby enhancing the model's ability to pay attention to different parts of the input sequence.

[0154] Furthermore, within each head, scaled dot product attention is used to calculate the attention score between the query and the key, and a weighted sum is performed based on these scores to obtain the encoded context features.

[0155] S603, inputting the encoded environmental features into a multi-layer perceptron for decoding processing, and obtaining a calibration result between the radar device and the image acquisition device.

[0156] Optional, reference Figure 6B , the encoded environmental features can be input into a multi-layer perceptron, which can then regress the relative posture (external parameters) between the radar device and the image acquisition device, and use the six components of the relative posture as the calibration results between the radar device and the image acquisition device.

[0157] In the embodiment of the present application, a feature fusion network, an encoder and a multi-layer perceptron are used to process the target point cloud features and the target image features to obtain a calibration result, which can ensure the accuracy of the calibration result.

[0158] Figure 7 FIG. 2 is a flow chart of a sensor device calibration method in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a sensor device calibration method. Figure 7 The specific implementation process is as follows:

[0159] S701, determining a translation parameter according to vehicle size information of the target vehicle.

[0160] S702, determining a rotation angle parameter according to the field of view angle information of the radar device in the target vehicle.

[0161] S703: Perform data enhancement processing on the public autonomous driving data set according to the translation parameters and the rotation angle parameters to obtain a training data set including sample image data and sample image data.

[0162] Optionally, a rotation-translation matrix is ​​determined based on the translation parameters and the rotation angle parameters; each standard data in the public autonomous driving data set is multiplied by the rotation-translation matrix to obtain extended data corresponding to each standard data; each standard data and the extended data corresponding to each standard data are used as training data sets.

[0163] S704, inputting the sample image data into the initial image branch to obtain sample image features corresponding to the sample image data.

[0164] S705: Determine image loss according to the sample image data and the sample image features.

[0165] S706, inputting the sample point cloud data into the initial point cloud branch to obtain sample point cloud features corresponding to the sample point cloud data.

[0166] S707, determining the point cloud loss according to the sample point cloud data and the sample point cloud features.

[0167] S708, determining a reprojection error and a feature matching loss according to the sample image features and the sample point cloud features.

[0168] Optionally, each point cloud feature point in the sample point cloud feature is projected onto the image plane corresponding to the sample image feature to obtain the projection point corresponding to each point cloud feature point; the reprojection error is determined based on the position error between each image feature point in the sample image feature and the projection point corresponding to each point cloud feature point; the feature matching loss is determined based on the feature difference between the sample image feature and the sample point cloud feature.

[0169] S709: Train the initial model according to the image loss, point cloud loss, reprojection error and feature matching loss to obtain a fusion alignment model.

[0170] S710, acquiring target point cloud data of the target area collected by a radar device on the target vehicle, and target image data of the target area collected by an image acquisition device on the target vehicle.

[0171] S711, based on the target image branch in the fusion alignment model, feature extraction is performed on the target image data to obtain target image features.

[0172] Optionally, the target image branch includes a convolutional neural network and a fully connected network; the target image data is input into the convolutional neural network for encoding processing to obtain encoded image features; the encoded image features are input into the fully connected network for decoding processing to obtain target image features.

[0173] S712, based on the target point cloud branch in the fusion alignment model, feature extraction is performed on the target point cloud data to obtain target point cloud features.

[0174] S713, based on the target fusion alignment branch in the fusion alignment model, determine the calibration result between the radar device and the image acquisition device according to the target point cloud features and the target image features.

[0175] Optionally, the target fusion alignment branch includes a feature fusion network, an encoder and a multi-layer perceptron; the target point cloud features and the target image features are input into the feature fusion network for fusion processing to obtain fused environment features; the fused environment features and the position coding information corresponding to the fused environment features are input into the encoder for coding processing to obtain coded environment features; the coded environment features are input into the multi-layer perceptron for decoding processing to obtain the calibration result between the radar device and the image acquisition device.

[0176] The specific process of the above S701-S713 can be found in the description of the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0177] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0178] Based on the same inventive concept, the embodiment of the present application also provides a sensor device calibration apparatus for implementing the sensor device calibration method involved above. The implementation solution provided by the apparatus is similar to the implementation solution recorded in the above method, so the specific limitations in one or more sensor device calibration apparatus embodiments provided below can refer to the limitations of the sensor device calibration method above, and will not be repeated here.

[0179] In an exemplary embodiment, Figure 8 As shown, a sensor device calibration device 1 is provided, comprising: a data acquisition module 10 and a result determination module 20, wherein:

[0180] The data acquisition module 10 is used to acquire target point cloud data of the target area acquired by the radar device on the target vehicle, and target image data of the target area acquired by the image acquisition device on the target vehicle;

[0181] A result determination module 20, for determining the calibration result between the radar device and the image acquisition device according to the target point cloud data and the target image data by using a fusion alignment model;

[0182] Among them, the fusion alignment model is obtained by training the initial model with the training data set; the training data set is obtained by processing the public autonomous driving data set according to the configuration information of the target vehicle; the public autonomous driving data set contains calibrated standard point cloud data and standard image data corresponding to the standard point cloud data.

[0183] In an exemplary embodiment, the configuration information includes vehicle size information and field of view angle information of the radar device; further, the sensor device calibration device 1 also includes a model training module 30, wherein, Fig. 9 As shown, the model training module 30 includes:

[0184] The data set determination unit 31 is used to determine the translation parameter according to the vehicle size information; determine the rotation angle parameter according to the field of view angle information; and perform data enhancement processing on the public autonomous driving data set according to the translation parameter and the rotation angle parameter to obtain a training data set.

[0185] In an exemplary embodiment, the data set determining unit 31 is further configured to:

[0186] Determine the rotation-translation matrix according to the translation parameters and the rotation angle parameters; multiply each standard data in the public autonomous driving data set by the rotation-translation matrix to obtain the extended data corresponding to each standard data; use each standard data and the extended data corresponding to each standard data as the training data set.

[0187] In an exemplary embodiment, the training data set includes sample image data and sample point cloud data corresponding to the sample image data; the initial model includes an initial image branch, an initial point cloud branch and an initial fusion alignment branch; further, as Fig. 9 As shown, the model training module 30 also includes:

[0188] The model training unit 32 is used to input the sample image data into the initial image branch to obtain the sample image features corresponding to the sample image data; determine the image loss based on the sample image data and the sample image features; input the sample point cloud data into the initial point cloud branch to obtain the sample point cloud features corresponding to the sample point cloud data; determine the point cloud loss based on the sample point cloud data and the sample point cloud features; determine the reprojection error and the feature matching loss based on the sample image features and the sample point cloud features; train the initial model based on the image loss, point cloud loss, reprojection error and feature matching loss to obtain the fusion alignment model.

[0189] In an exemplary embodiment, the model training unit 32 is further configured to:

[0190] Each point cloud feature point in the sample point cloud feature is projected onto the image plane corresponding to the sample image feature to obtain the projection point corresponding to each point cloud feature point; the reprojection error is determined based on the position error between each image feature point in the sample image feature and the projection point corresponding to each point cloud feature point; the feature matching loss is determined based on the feature difference between the sample image feature and the sample point cloud feature.

[0191] In an exemplary embodiment, the fusion alignment model includes a target image branch, a target point cloud branch, and a target fusion alignment branch; Fig.10 As shown, the result determination module 20 includes:

[0192] A first determining unit 21 is used to extract features from target image data based on the target image branch to obtain target image features;

[0193] A second determining unit 22 is used to extract features from the target point cloud data based on the target point cloud branch to obtain target point cloud features;

[0194] The third determination unit 23 is used to determine the calibration result between the radar device and the image acquisition device based on the target fusion alignment branch and according to the target point cloud features and the target image features.

[0195] In an exemplary embodiment, the target image branch includes a convolutional neural network and a fully connected network; further, the first determination unit 21 is specifically used to:

[0196] The target image data is input into the convolutional neural network for encoding processing to obtain the encoded image features; the encoded image features are input into the fully connected network for decoding processing to obtain the target image features.

[0197] In an exemplary embodiment, the target fusion alignment branch includes a feature fusion network, an encoder and a multi-layer perceptron; further, the third determination unit 23 is specifically used to:

[0198] The target point cloud features and the target image features are input into the feature fusion network for fusion processing to obtain fused environment features; the fused environment features and the position coding information corresponding to the fused environment features are input into the encoder for encoding processing to obtain coded environment features; the coded environment features are input into the multi-layer perceptron for decoding processing to obtain the calibration results between the radar equipment and the image acquisition equipment.

[0199] Each module in the above-mentioned sensor device calibration device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0200] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a sensor device calibration method is implemented.

[0201] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0202] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0203] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0204] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0205] It should be noted that the data involved in this application (including but not limited to sample training set data, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0206] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0207] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0208] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A sensor device calibration method, characterized in that: The method comprises: Acquire target point cloud data of a target area collected by a radar device on a target vehicle, and target image data of the target area collected by an image acquisition device on the target vehicle; Using a fusion alignment model, according to the target point cloud data and the target image data, determining a calibration result between the radar device and the image acquisition device; Among them, the fusion alignment model is obtained by training the initial model using a training data set; the training data set is obtained by processing a public autonomous driving data set according to the configuration information of the target vehicle; the public autonomous driving data set contains calibrated standard point cloud data and standard image data corresponding to the standard point cloud data.

2. The method according to claim 1, characterized in that The configuration information includes vehicle size information and field of view angle information of the radar device; The processing of the public autonomous driving data set according to the configuration information of the target vehicle includes: Determining translation parameters according to the vehicle size information; Determining a rotation angle parameter according to the field of view angle information; According to the translation parameter and the rotation angle parameter, data enhancement processing is performed on the public autonomous driving data set to obtain a training data set.

3. The method according to claim 2, characterized in that The step of performing data enhancement processing on the public autonomous driving data set according to the translation parameter and the rotation angle parameter to obtain a training data set includes: Determine a rotation-translation matrix according to the translation parameter and the rotation angle parameter; Multiply each standard data in the public autonomous driving data set by the rotation and translation matrix to obtain the extended data corresponding to each standard data; Each standard data and the extended data corresponding to each standard data are used as the training data set.

4. The method according to claim 1, characterized in that: The training data set includes sample image data and sample point cloud data corresponding to the sample image data; the initial model includes an initial image branch, an initial point cloud branch and an initial fusion alignment branch, and the fusion alignment model is trained in the following manner: Inputting the sample image data into the initial image branch to obtain sample image features corresponding to the sample image data; Determining image loss according to the sample image data and the sample image features; Inputting the sample point cloud data into the initial point cloud branch to obtain sample point cloud features corresponding to the sample point cloud data; Determine the point cloud loss according to the sample point cloud data and the sample point cloud features; Determining a reprojection error and a feature matching loss according to the sample image features and the sample point cloud features; The initial model is trained according to the image loss, the point cloud loss, the reprojection error and the feature matching loss to obtain the fusion alignment model.

5. The method according to claim 4, characterized in that The determining of the reprojection error and the feature matching loss according to the sample image feature and the sample point cloud feature includes: Projecting each point cloud feature point in the sample point cloud feature onto the image plane corresponding to the sample image feature to obtain a projection point corresponding to each point cloud feature point; Determine a reprojection error according to a position error between each image feature point in the sample image feature and a projection point corresponding to each point cloud feature point; A feature matching loss is determined according to a feature difference between the sample image feature and the sample point cloud feature.

6. The method according to claim 1, characterized in that The fusion alignment model includes a target image branch, a target point cloud branch and a target fusion alignment branch; The adopting of the fusion alignment model to determine the calibration result between the radar device and the image acquisition device according to the target point cloud data and the target image data includes: Based on the target image branch, feature extraction is performed on the target image data to obtain target image features; Based on the target point cloud branch, feature extraction is performed on the target point cloud data to obtain target point cloud features; Based on the target fusion alignment branch, a calibration result between the radar device and the image acquisition device is determined according to the target point cloud features and the target image features.

7. The method according to claim 6, characterized in that The target image branch includes a convolutional neural network and a fully connected network; The step of extracting features from the target image data based on the target image branch to obtain target image features includes: Inputting the target image data into the convolutional neural network for encoding processing to obtain encoded image features; The encoded image features are input into the fully connected network for decoding to obtain target image features.

8. The method according to claim 6, characterized in that The target fusion alignment branch includes a feature fusion network, an encoder and a multi-layer perceptron; The step of determining the calibration result between the radar device and the image acquisition device based on the target fusion alignment branch according to the target point cloud feature and the target image feature includes: Inputting the target point cloud features and the target image features into the feature fusion network for fusion processing to obtain fusion environment features; Inputting the fused environment feature and the position coding information corresponding to the fused environment feature into the encoder for coding processing to obtain the coded environment feature; The encoded environmental features are input into the multi-layer perceptron for decoding processing to obtain a calibration result between the radar device and the image acquisition device.

9. A sensor equipment calibration device, characterized in that: The device comprises: A data acquisition module, used to acquire target point cloud data of a target area acquired by a radar device on a target vehicle, and target image data of the target area acquired by an image acquisition device on the target vehicle; A result determination module, used to determine the calibration result between the radar device and the image acquisition device according to the target point cloud data and the target image data by using a fusion alignment model; Among them, the fusion alignment model is obtained by training the initial model using a training data set; the training data set is obtained by processing a public autonomous driving data set according to the configuration information of the target vehicle; the public autonomous driving data set contains calibrated standard point cloud data and standard image data corresponding to the standard point cloud data.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images