A training method and device of a calibration model, a storage medium and an electronic device

Abstract

The specification discloses a training method and device of a calibration model, a storage medium and an electronic device. A sample image and a sample point cloud are obtained, a sample calibration relationship is determined through a calibration model, the attributes of the sample point cloud are determined according to the sample calibration relationship, the attributes of a predicted point cloud corresponding to the sample image are determined through the sample image, and then the loss is determined through the gap between the attributes of the predicted point cloud and the attributes of the sample point cloud, so as to train the calibration model. Based on the calibration model in the specification, the accurate calibration relationship can be determined only by one frame of image and one frame of point cloud data without iteration for a large number of times, and the calibration efficiency is ensured.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a training method, apparatus, storage medium, and electronic device for a calibration model. Background Technology

[0002] Currently, with the development of autonomous driving technology, the driving safety of autonomous vehicles is becoming increasingly important. Obstacle detection and classification methods based on the fusion of image and radar data are widely used in obstacle detection and classification scenarios. However, the prerequisite for image and radar data fusion is the calibration of both radar and image sensors.

[0003] In existing technologies, edge detection is typically performed first on the acquired point cloud data and image data to obtain boundary points for each point cloud and each image. Point cloud boundary points are the boundary points of objects within the point cloud data, and image boundary points are the boundary points of objects within the image data. Then, based on an initial calibration relationship, the boundary points of each point cloud and each image located in the same coordinate system are determined, and these boundary points are matched to identify mutually matching point cloud and image boundary points. Finally, the calibration relationship is updated based on the mutually matching point cloud and image boundary points, and the process of identifying mutually matching point cloud and image boundary points continues based on the updated calibration relationship. This continues until the distance between the identified mutually matching point cloud and image boundary points is less than a distance threshold. The latest determined calibration relationship is then used as the calibration relationship between the radar sensor and the image sensor.

[0004] However, existing technologies require numerous iterations to determine accurate calibration parameters in the process of establishing the calibration relationship between radar sensors and image sensors, resulting in poor calibration efficiency. Summary of the Invention

[0005] This specification provides a training method, apparatus, storage medium, and electronic device for calibrating a model, in order to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides a method for training a calibration model, the method comprising:

[0008] The sample images acquired by the first device and the sample point clouds acquired by the second device are identified;

[0009] The sample image and the sample point cloud are input into the calibration model to be trained to obtain the sample calibration relationship output by the calibration model;

[0010] Based on the sample calibration relationship and the sample point cloud, when the calibration relationship between the first device and the second device is the sample calibration relationship, the attribute of the projection result of the sample point cloud in the first coordinate system corresponding to the first device is determined and used as the first attribute;

[0011] Based on the sample image, predict the attribute of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system when the calibration relationship between the first device and the second device is a standard calibration relationship, and use it as the second attribute;

[0012] The calibration model is trained based on the difference between the first attribute and the second attribute, and the calibration model is used to determine the calibration relationship between the image sensor and the radar sensor.

[0013] Optionally, the method further includes:

[0014] According to the preset initial calibration relationship, the sample point cloud data is mapped to the first coordinate system where the sample image is located, and the projection result of the sample point cloud in the first coordinate system is obtained.

[0015] The projection result of the sample point cloud in the first coordinate system and the sample image are combined to obtain the combined result;

[0016] The sample images and sample point clouds are input into the calibration model to be trained, specifically including:

[0017] The combined result is used as input to the calibration model to be trained.

[0018] Optionally, based on the sample calibration relationship and the sample point cloud, the attributes of the projection result of the sample point cloud in the first coordinate system corresponding to the first device are determined when the calibration relationship between the first device and the second device is the sample calibration relationship, and are used as the first attribute, specifically including:

[0019] From the various attributes of the sample point cloud, the target attribute is determined, wherein each attribute includes at least one of the point cloud intensity attribute, point cloud coordinate attribute, and point cloud depth attribute;

[0020] Based on the sample calibration relationship and the sample point cloud, determine the projection result of the sample point cloud in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is the sample calibration relationship;

[0021] The first attribute is obtained by extracting features from the projection result of the sample point cloud in the first coordinate system corresponding to the first device using the feature extraction method of the target attribute.

[0022] Optionally, based on the sample image, the attribute of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system is predicted when the calibration relationship between the first device and the second device is a standard calibration relationship, and is used as the second attribute. Specifically, this includes:

[0023] The sample image is input into a pre-trained prediction model to obtain the attributes of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system, which is used as the second attribute.

[0024] The prediction model is pre-trained based on image data and point cloud data collected by a first device and a second device with a standard calibration relationship.

[0025] Optionally, the calibration model is trained based on the difference between the first attribute and the second attribute, specifically including:

[0026] The first loss is determined based on the difference between the first attribute and the second attribute;

[0027] Determine the true calibration relationship between the second device that acquires the sample point cloud data and the first device that acquires the sample image;

[0028] Based on the true calibration relationship and the sample point cloud, when the calibration relationship between the first device and the second device is the true calibration relationship, the attribute of the projection result of the sample point cloud data in the first coordinate system corresponding to the first device is determined as the third attribute;

[0029] The second loss is determined based on the difference between the second attribute and the third attribute;

[0030] The calibration model is trained with the goal of minimizing the sum of the first loss and the second loss.

[0031] Optionally, based on the sample image, when the calibration relationship between the first device and the second device is a standard calibration relationship, the attribute of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system is predicted as the second attribute, specifically including:

[0032] The sample image is input into a pre-trained prediction model to obtain the attributes of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system, which is used as the second attribute.

[0033] The calibration model is trained based on the difference between the first attribute and the second attribute, specifically including:

[0034] The calibration model and the prediction model are trained based on the difference between the first attribute and the second attribute.

[0035] Optionally, the method further includes:

[0036] In response to a calibration request, determine the target point cloud acquired by the radar sensor and the target image acquired by the image sensor;

[0037] The target point cloud and the target image are used as inputs to the pre-trained calibration model to obtain the calibration relationship between the radar sensor that acquired the target point cloud and the image sensor that acquired the target image, as output by the calibration model.

[0038] This specification provides a training device for calibrating a model, the device comprising:

[0039] The sample determination module is used to determine the sample image acquired by the first device and the sample point cloud acquired by the second device;

[0040] The relationship determination module is used to input the sample image and the sample point cloud into the calibration model to be trained, and obtain the sample calibration relationship output by the calibration model;

[0041] The first determining module is used to determine, based on the sample calibration relationship and the sample point cloud, the attribute of the projection result of the sample point cloud in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is the sample calibration relationship, and use it as the first attribute;

[0042] The second determining module is used to predict, based on the sample image, the attribute of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system when the calibration relationship between the first device and the second device is a standard calibration relationship, as the second attribute;

[0043] The training module is used to train the calibration model based on the difference between the first attribute and the second attribute, the calibration model being used to determine the calibration relationship between the image sensor and the radar sensor.

[0044] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method for the calibration model described above.

[0045] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the training method of the above-described calibration model.

[0046] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0047] In the training method of the calibration model provided in this specification, sample images and sample point clouds are acquired. The calibration model is used to determine the sample calibration relationship. Based on the sample calibration relationship, the attributes of the sample point cloud are determined. The attributes of the predicted point cloud corresponding to the sample image are determined through the sample image. Then, the loss is determined by the difference between the attributes of the predicted point cloud and the attributes of the sample point cloud, so as to train the calibration model.

[0048] As can be seen from the above method, the training method of the calibration model in this specification does not require edge detection to determine the boundary points of the point cloud and the image, even when the edge detection error is large. Accurate calibration relationships can still be obtained based on the trained calibration model. Furthermore, based on this calibration model, a large number of iterations are not required; only one frame of image and one frame of point cloud data are needed to determine accurate calibration relationships, ensuring calibration efficiency. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0050] Figure 1 This is a flowchart illustrating the training method for the calibration model provided in this manual.

[0051] Figure 2 This is a flowchart illustrating the training method for the calibration model provided in this manual.

[0052] Figure 3 This is a schematic diagram of the training device for the calibration model provided in this manual;

[0053] Figure 4 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0055] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0056] It should be noted that all actions involving the acquisition of signals, information, or data described in this manual are performed in accordance with the applicable data protection laws and regulations in the locality and with authorization from the owner of the relevant device.

[0057] Unlike current methods that rely on separate edge detection of point cloud data and image attributes to obtain point cloud boundary points and image boundary points, and then determine matching point cloud boundary points and image boundary points through an initial calibration relationship, this method updates the initial calibration relationship based on the difference between the matching point cloud boundary points and image boundary points. This process is repeated until the distance between the determined matching point cloud boundary points and image boundary points is less than a distance threshold, at which point the newly determined calibration relationship is used as the calibration relationship between the radar sensor and the image sensor. However, this method requires numerous iterations to determine accurate calibration parameters, and even after multiple iterations, the determined calibration parameters remain inaccurate if the boundary points are not accurately identified.

[0058] This specification provides a new method for training a calibration model that does not require edge detection of point cloud data and image data, nor does it require determining matching point cloud boundary points and image boundary points based on edge detection results, and it can determine accurate calibration relationships without requiring a large number of iterations.

[0059] Figure 1 The flowchart for the training method of the calibration model provided in this manual is shown, and specifically includes the following steps:

[0060] S100: Determine the sample image acquired by the first device and the sample point cloud acquired by the second device.

[0061] This specification provides a method for training a calibration model. The training process of this calibration model can be executed by electronic devices such as servers used for sensor calibration. For ease of description, this specification uses the example of a server used for calibrating image sensors and point cloud sensors executing the training process of the calibration model.

[0062] Generally, model training methods typically include three stages: sample determination, sample processing, and training. The calibration model described in this specification is used to determine the calibration relationship between the image sensor and the point cloud sensor. Therefore, the server can determine the image data acquired by the image sensor and the point cloud data acquired by the point cloud sensor as training samples.

[0063] Specifically, the server can receive training requests, which carry the model identifier of the model to be trained and the sample identifier of the samples required by the model corresponding to the model identifier.

[0064] Then, the server can determine the calibration model to be trained from its stored models based on the model identifier.

[0065] Finally, based on the sample identifier, the server can determine the image data and point cloud data of the same area collected by the first device and the second device at the same time from the data stored in its own storage, and use them as the sample image collected by the first device and the sample point cloud collected by the second device.

[0066] The image data and point cloud data acquired at the same time can be image data and point cloud data acquired at two acquisition times with a difference of less than a preset time period threshold. Similarly, the same region can be the first region corresponding to the image data and the second region corresponding to the point cloud data with a difference of less than a preset region threshold. The first region and the second region are real regions.

[0067] Of course, the server can pre-determine the model results and initial model parameters of the calibration model to be trained, and then determine the sample images and sample point clouds used to train the calibration model from the mutually matched image data and point cloud data, and determine the acquisition device of the sample image as the first device and the acquisition device of the sample point cloud as the second device. The first device and the second device can be acquisition devices set on unmanned equipment.

[0068] S102: Input the sample image and the sample point cloud into the calibration model to be trained to obtain the sample calibration relationship output by the calibration model.

[0069] In one or more embodiments provided in this specification, as described above, the model training process includes a sample determination phase, a sample processing phase, and a training phase. Thus, after the samples are determined, the server can process them.

[0070] Specifically, the calibration model in this specification is used to determine the calibration relationship between the first device and the second device.

[0071] Therefore, the server can use the sample image and sample point cloud as input to the calibration model to obtain the sample calibration relationship output by the calibration model. This sample calibration relationship is the calibration relationship between the first device that acquired the sample image and the second device that acquired the sample point cloud.

[0072] The calibration relationship can be a transformation relationship between the first coordinate system corresponding to the first device and the second coordinate system corresponding to the second device (e.g., rotation matrix, translation matrix, quaternions and hexaternions used to determine the rotation and translation matrices), or it can be an external parameter between an image sensor that acquires image data and a radar sensor that acquires point cloud data. The specific type of this calibration relationship can be set as needed, and this specification does not limit it.

[0073] That is, the transformation matrix and translation matrix can also be the quaternion, hexaternion, etc. that determine the transformation matrix and translation matrix.

[0074] The specific form of the calibration relationship and how to perform conversion based on the calibration relationship can be set as needed, and this manual does not impose any restrictions on it.

[0075] S104: Based on the sample calibration relationship and the sample point cloud, determine the attribute of the projection result of the sample point cloud in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is the sample calibration relationship, and use it as the first attribute.

[0076] In one or more embodiments provided in this specification, the server can train the calibration model after processing the training samples. However, if the loss is determined directly based on the difference between the actual calibration relationship between the first and second devices and the determined sample calibration relationship, the determined loss contains less information, and it is difficult to train an accurate calibration model based on a loss with less information.

[0077] In this scenario, if the attributes of the point cloud corresponding to the sample image can be predicted based on the features of the image itself, and the loss is determined by the difference between the predicted attributes and the attributes of the point cloud determined according to the sample calibration relationship, then the determined loss can be used to characterize the reliability of the sample calibration relationship. Therefore, the server can determine the attributes of the sample point cloud through the sample calibration relationship.

[0078] Specifically, since the server needs to compare the attributes obtained through the sample calibration relationship with the attributes obtained based on the sample image, and the coordinate system of the attributes obtained based on the sample image is the first coordinate system corresponding to the sample image, the server needs to determine the attributes of the sample point cloud located in the first coordinate system.

[0079] Therefore, the server can, based on the sample calibration relationship, assuming the calibration relationship between the first device and the second device is the same as the sample calibration relationship, project the sample point cloud collected by the second device onto the first coordinate system according to the sample calibration relationship, and obtain the projection result. That is, by keeping the second device stationary, the first device with which the calibration relationship between the second device and the first device is determined as the designated device, and the sample point cloud is projected onto the designated coordinate system where the designated device is located. The projection result of the sample point cloud in the designated coordinate system is the attribute of the projection result of the sample point cloud in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is the same as the sample calibration relationship.

[0080] The server can then determine the attribute of the projection result of the point cloud data in the first coordinate system, which will be used as the first attribute. This attribute can be any attribute of the point cloud data, such as point cloud intensity attribute, point cloud coordinate attribute, point cloud depth attribute, etc.

[0081] S106: Based on the sample image, predict the attribute of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system when the calibration relationship between the first device and the second device is a standard calibration relationship, and use it as the second attribute.

[0082] In one or more embodiments provided in this specification, as described above, the server can predict the attributes of the point cloud corresponding to the sample image based on the features of the image itself, and then determine the loss by the difference between the predicted attributes and the attributes of the point cloud determined according to the sample calibration relationship. Thus, the server can predict the attributes of the predicted point cloud corresponding to the sample image using the sample image.

[0083] Specifically, since the server needs to compare the attributes obtained through the sample calibration relationship with the attributes obtained based on the sample image, and the coordinate system of the attributes obtained based on the sample image is the first coordinate system corresponding to the sample image, the server needs to predict the attributes of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system based on the sample image.

[0084] Therefore, based on the sample image, assuming the calibration relationship between the first device and the second device is a standard calibration relationship, the server predicts the predicted point cloud corresponding to the sample image, and predicts the attributes of the projection result of the predicted point cloud in the first coordinate system according to the standard calibration relationship. That is, the first device is kept stationary, and the second device, whose calibration relationship with the second device is a standard calibration relationship, is designated as the specific device. Based on the image data, the server determines the predicted point cloud corresponding to the image data and projects the predicted point cloud onto the first coordinate system where the first device is located. The server can then obtain the attributes of the projection result of the predicted point cloud in the first coordinate system.

[0085] The predicted point cloud can be: point cloud data corresponding to sample images of the same area collected by the first device at the same time from a specific device with a standard calibration relationship to the first device. The standard calibration relationship is used to characterize the calibration relationship between the specific device and the first device, and the standard calibration relationship can be preset.

[0086] The server can then determine the attributes of the projection result of the predicted point cloud in the first coordinate system, which will be used as the second attribute. This attribute can be any attribute of the point cloud data, such as point cloud intensity, point cloud coordinates, or point cloud depth.

[0087] S108: The calibration model is trained based on the difference between the first attribute and the second attribute. The calibration model is used to determine the calibration relationship between the image sensor and the radar sensor.

[0088] In one or more embodiments provided in this specification, as described above, after determining the attributes of the predicted point cloud corresponding to the sample image and the attributes of the sample point cloud determined based on the sample calibration relationship, the server can determine the loss based on the difference between the first attribute and the second attribute.

[0089] Specifically, taking the example where both the first attribute and the second attribute are point cloud intensity attributes, the first attribute can be the point cloud intensity of the projection result of the sample point cloud in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is a sample calibration relationship, and the second attribute can be the point cloud intensity of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system when the calibration relationship between the first device and the second device is a standard calibration relationship.

[0090] Therefore, the server can determine the difference between the point cloud intensity of the projection result of the sample point cloud and the point cloud intensity of the projection result of the predicted point cloud, and determine the loss based on the difference.

[0091] Finally, the server can adjust the model parameters of the calibration model based on the determined loss in order to train the calibration model.

[0092] Furthermore, during the operation of the autonomous driving device, it can acquire sensor data at a preset frequency. This sensor data is required for sensor data calibration and includes at least image data and point cloud data. The autonomous driving device is equipped with an image sensor and a radar sensor, namely, a first device and a second device. The first and second devices can be the same as or different from the first and second devices corresponding to the training samples in the calibration model training process.

[0093] Therefore, the autonomous driving device can receive calibration requests and, based on the received calibration requests, determine the target image acquired by the image sensor and the target point cloud acquired by the radar sensor. The target image and target point cloud can be image data and point cloud data of the same area acquired by the image sensor and radar sensor simultaneously.

[0094] The server can then use the determined target point cloud and target image as input to a pre-trained calibration model to obtain the calibration relationship between the radar sensor that acquired the target point cloud and the image sensor that acquired the target image, as output by the calibration model.

[0095] Furthermore, in this specification, to avoid inaccuracies in the calibration relationships determined based on single-frame image data and single-frame point cloud data, the server can also determine several data pairs. Each data pair contains a corresponding frame of target point cloud data and a frame of target image data. Then, each data pair is used as input to a pre-trained calibration model to obtain the calibration relationships corresponding to each data pair output by the calibration model. Finally, based on each calibration relationship, the target calibration relationship between the image sensor and the radar sensor is determined. Specifically, the server can determine the error corresponding to the calibration relationship for each data pair, based on the calibration relationship and the determined data pairs, and finally determine the target calibration relationship based on the errors corresponding to each calibration relationship.

[0096] Furthermore, after determining the calibration relationship, the server can determine the point cloud data and its projection in the first reference frame based on the acquired sensor data and the calibration relationship. Then, it fuses this projection with the image data from the sensor data to determine the fusion result, and performs obstacle detection on the fusion result to determine the location of obstacles. How the image data and point cloud data are fused based on the calibration relationship after it is determined can be configured as needed, and this specification does not impose any restrictions on this.

[0097] based on Figure 1 The provided calibration model training method acquires sample images and sample point clouds. Through the calibration model, it determines the sample calibration relationship and, based on this relationship, determines the attributes of the sample point cloud. It also determines the attributes of the predicted point cloud corresponding to the sample image. Finally, it determines the loss based on the difference between the attributes of the predicted point cloud and the attributes of the sample point cloud, thus training the calibration model. This calibration model training method does not require edge detection to determine point cloud and image boundary points in the sample images and point clouds. Even with large edge detection errors, accurate calibration relationships can be obtained based on the trained calibration model. Furthermore, this calibration model requires only one frame of image and one frame of point cloud data to determine accurate calibration relationships, ensuring calibration efficiency.

[0098] Furthermore, typically, this calibration model can determine the calibration relationship between the second device that acquired the sample point cloud and the first device that acquired the sample image based on the difference between the sample point cloud and the sample image in the same coordinate system. Therefore, when inputting the sample point cloud and sample image into the calibration model, the sample image and sample point cloud in the same coordinate system can be determined as the input data of the calibration model.

[0099] Specifically, the server stores the initial calibration relationship. Therefore, the server can project the sample image and sample point cloud into the same coordinate system according to the pre-stored initial calibration relationship, and then combine the sample point cloud and sample image in the same coordinate system, using the combined result as the input data for the calibration model.

[0100] Furthermore, since this specification requires training the model based on the attributes of the predicted point cloud and the sample point cloud in the first coordinate system, the model input can be determined based on the sample point cloud and sample image in the first coordinate system during the model input stage.

[0101] Specifically, the server can project the sample point cloud onto the first coordinate system where the image data is located according to the preset initial calibration relationship, and obtain the projection result of the sample point cloud in the image coordinate system.

[0102] Then, the server can combine the projection result of the sample point cloud in the image coordinate system with the sample image to obtain the combined result.

[0103] Finally, the server can use the combined result as input to the sample point cloud.

[0104] Furthermore, as mentioned earlier, the attributes of a point cloud can include point cloud intensity data, point cloud coordinate attributes, and point cloud depth attributes. Therefore, when determining the first attribute and the second attribute, at least one attribute of the point cloud can be determined as either the first attribute or the second attribute.

[0105] Specifically, the server can determine the target attribute from the various attributes of the sample point cloud, where each attribute includes at least one of the point cloud intensity attribute, point cloud coordinate attribute, and point cloud depth attribute.

[0106] Therefore, the server can determine the projection result of the sample point cloud in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is a sample calibration relationship, based on the sample calibration relationship and the sample point cloud.

[0107] Finally, the service allows for feature extraction of the projection result using the target attribute feature extraction method to obtain the first attribute.

[0108] Furthermore, in this specification, the process of determining the attributes of the predicted point cloud based on the sample image can also be achieved through a prediction model.

[0109] Specifically, the server is equipped with a pre-trained prediction model. This prediction model is pre-trained based on image data and point cloud data collected by a first device and a second device with a standard calibration relationship.

[0110] The server can then input the sample image into the prediction model to obtain the attributes of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system, which is used as the second attribute.

[0111] The prediction model was trained in the following manner:

[0112] Image data and point cloud data collected by the first and second devices with a standard calibration relationship are acquired. The image data is used as training samples. Based on the standard calibration relationship and the point cloud data, the attributes of the point cloud data are determined and used as annotations for the training samples. The annotation model is then trained based on the training samples and their annotations.

[0113] Furthermore, during model training, the server can determine the loss not only based on the difference between the first and second attributes, but also based on the difference between the second attribute and the third attribute obtained from the true calibration relationship.

[0114] Specifically, the server can determine the difference between the first attribute and the second attribute, and determine the first loss. The difference between the first attribute and the second attribute can be determined using either a cross-entropy loss function or a comparison function.

[0115] Secondly, the server can determine the true calibration relationship between the second device that acquired the sample point cloud data and the first device that acquired the sample image.

[0116] Then, the service determines, based on the true calibration relationship and the sample point cloud, the attribute of the projection result of the sample point cloud data in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is the true calibration relationship, and uses this attribute as the third attribute. The steps for determining the third attribute are similar to those described above for determining the first attribute, and will not be repeated here.

[0117] The service then determines a second loss based on the difference between the second and third attributes. Similarly, the difference between the second and third attributes can be determined using either a cross-entropy loss function or a comparison function. The specific determination of the first and second losses can be configured as needed, and this specification does not impose any restrictions on this.

[0118] Finally, the server can use minimizing the sum of the first loss and the second loss as the optimization objective to adjust the model parameters of the calibration model in order to train the calibration model.

[0119] In addition, the calibration model and the prediction model can be trained jointly:

[0120] First, the server can determine the sample image and sample point cloud, and input the sample image and sample point cloud into the calibration model to be trained to obtain the sample calibration relationship output by the calibration model.

[0121] Meanwhile, the server can input the sample image into the prediction model to be trained to obtain the attributes of the predicted point cloud output by the prediction model, which will serve as the second attribute.

[0122] Then, the server can determine the first attribute based on the sample labeling relationship, and determine the third attribute based on the true labeling relationship.

[0123] Finally, based on the first and second attributes, a first loss is determined; based on the second and third attributes, a second loss is determined; and the prediction model and calibration model are trained with the optimization objective of minimizing the sum of the first and second losses. Figure 2 As shown.

[0124] Figure 2This is a flowchart illustrating the training method for the calibration model provided in this specification. The server determines sample point clouds and sample images, and inputs them into the calibration model to obtain the sample calibration relationship. Then, the server determines a first attribute based on the sample point cloud and the sample calibration relationship, and a third attribute based on the sample point cloud and the true calibration relationship. Next, the server determines a second attribute based on the sample image using a prediction model. Finally, the server determines a first loss based on the difference between the first and second attributes, and a second loss based on the difference between the second and third attributes, and then trains the calibration model and the prediction model using the first and second losses.

[0125] In scenarios where the prediction model and the calibration model are jointly trained, the aforementioned standard calibration relationship obviously cannot be pre-defined manually or specifically given. Therefore, this standard calibration relationship can be obtained by adjusting the prediction model based on the loss, and the attributes of the predicted point cloud corresponding to the sample image can be determined based on the adjusted calibration relationship. How this standard calibration relationship is determined and whether it can be represented by functions, formulas, etc., can be set as needed; this specification does not impose any restrictions on this.

[0126] The above describes training methods for calibration models provided by one or more embodiments of this specification. Based on the same idea, this specification also provides corresponding training devices for calibration models, such as... Figure 3 As shown.

[0127] Figure 3 The training apparatus for the calibration model provided in this specification includes:

[0128] The sample determination module 200 is used to determine the sample image acquired by the first device and the sample point cloud acquired by the second device.

[0129] The relationship determination module 202 is used to input the sample image and the sample point cloud into the calibration model to be trained, and obtain the sample calibration relationship output by the calibration model.

[0130] The first determining module 204 is used to determine, based on the sample calibration relationship and the sample point cloud, the attribute of the projection result of the sample point cloud in the first coordinate system corresponding to the first device, when the calibration relationship between the first device and the second device is the sample calibration relationship, as the first attribute.

[0131] The second determining module 206 is used to predict, based on the sample image, the attributes of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system when the calibration relationship between the first device and the second device is a standard calibration relationship, as the second attribute.

[0132] The training module 208 is used to train the calibration model based on the difference between the first attribute and the second attribute, the calibration model being used to determine the calibration relationship between the image sensor and the radar sensor.

[0133] Optionally, the sample determination module 200 is used to project the sample point cloud data into the first coordinate system where the sample image is located according to a preset initial calibration relationship, to obtain the projection result of the sample point cloud in the first coordinate system, to combine the projection result of the sample point cloud in the first coordinate system and the sample image to obtain a combined result, and to input the combined result as input into the calibration model to be trained.

[0134] Optionally, the first determining module 204 is used to determine a target attribute from the attributes of the sample point cloud, wherein the attributes include at least one of point cloud intensity attribute, point cloud coordinate attribute, and point cloud depth attribute. Based on the sample calibration relationship and the sample point cloud, it determines the projection result of the sample point cloud in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is the sample calibration relationship. The first attribute is obtained by extracting features from the projection result of the sample point cloud in the first coordinate system corresponding to the first device through the feature extraction method of the target attribute.

[0135] Optionally, the second determining module 206 is used to input the sample image into a pre-trained prediction model to obtain the attribute of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system, as the second attribute, wherein the prediction model is pre-trained based on the image data and point cloud data collected by the first and second devices with a standard calibration relationship.

[0136] Optionally, the training module 208 is configured to determine a first loss based on the difference between the first attribute and the second attribute, determine the true calibration relationship between the second device that collects the sample point cloud data and the first device that collects the sample image, determine the attribute of the projection result of the sample point cloud data in the first coordinate system corresponding to the first device as a third attribute when the calibration relationship between the first device and the second device is the true calibration relationship, determine a second loss based on the difference between the second attribute and the third attribute, and train the calibration model with the minimum sum of the first loss and the second loss as the optimization objective.

[0137] Optionally, the second determining module 206 is used to input the sample image into a pre-trained prediction model to obtain the attribute of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system, which is output by the prediction model, as the second attribute. The training module 208 is used to train the calibration model and the prediction model according to the difference between the first attribute and the second attribute.

[0138] Optionally, the training module 208 is configured to, in response to a calibration request, determine the target point cloud acquired by the radar sensor and the target image acquired by the image sensor, and input the target point cloud and the target image as inputs into the pre-trained calibration model to obtain the calibration relationship between the radar sensor that acquired the target point cloud and the image sensor that acquired the target image, as output by the calibration model.

[0139] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The training method for the provided calibration model.

[0140] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The training method for the provided calibration model.

[0141] This instruction manual also provides Figure 4 The diagram shows a schematic structural representation of the electronic device. Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The training method for the calibration model is described above. Of course, besides software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0142] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0143] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0144] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0145] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0146] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0156] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0157] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A training method for a calibration model, characterized in that, The method includes: The sample images acquired by the first device and the sample point clouds acquired by the second device are identified; The sample image and the sample point cloud are input into the calibration model to be trained to obtain the sample calibration relationship output by the calibration model; Based on the sample calibration relationship and the sample point cloud, when the calibration relationship between the first device and the second device is the sample calibration relationship, the attribute of the projection result of the sample point cloud in the first coordinate system corresponding to the first device is determined and used as the first attribute; Based on the sample image, predict the attribute of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system when the calibration relationship between the first device and the second device is a standard calibration relationship, and use it as the second attribute; The calibration model is trained based on the difference between the first attribute and the second attribute, and the calibration model is used to determine the calibration relationship between the image sensor and the radar sensor. The calibration model is trained based on the difference between the first attribute and the second attribute, specifically including: The first loss is determined based on the difference between the first attribute and the second attribute; Determine the true calibration relationship between the second device that acquires the sample point cloud data and the first device that acquires the sample image; Based on the true calibration relationship and the sample point cloud, when the calibration relationship between the first device and the second device is the true calibration relationship, the attribute of the projection result of the sample point cloud data in the first coordinate system corresponding to the first device is determined as the third attribute; The second loss is determined based on the difference between the second attribute and the third attribute; The calibration model is trained with the goal of minimizing the sum of the first loss and the second loss.

2. The method as described in claim 1, characterized in that, The method further includes: According to the preset initial calibration relationship, the sample point cloud data is projected onto the first coordinate system where the sample image is located, so as to obtain the projection result of the sample point cloud in the first coordinate system; The projection result of the sample point cloud in the first coordinate system and the sample image are combined to obtain the combined result; The sample images and sample point clouds are input into the calibration model to be trained, specifically including: The combined result is used as input to the calibration model to be trained.

3. The method as described in claim 1, characterized in that, Based on the sample calibration relationship and the sample point cloud, when the calibration relationship between the first device and the second device is the sample calibration relationship, the attribute of the projection result of the sample point cloud in the first coordinate system corresponding to the first device is determined as the first attribute, specifically including: From the various attributes of the sample point cloud, the target attribute is determined, wherein each attribute includes at least one of the point cloud intensity attribute, point cloud coordinate attribute, and point cloud depth attribute; Based on the sample calibration relationship and the sample point cloud, determine the projection result of the sample point cloud in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is the sample calibration relationship; The first attribute is obtained by extracting features from the projection result of the sample point cloud in the first coordinate system corresponding to the first device using the feature extraction method of the target attribute.

4. The method as described in claim 1, characterized in that, Based on the sample image, predict the attributes of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system when the calibration relationship between the first device and the second device is a standard calibration relationship. These attributes, as the second attribute, specifically include: The sample image is input into a pre-trained prediction model to obtain the attributes of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system, which is used as the second attribute. The prediction model is pre-trained based on image data and point cloud data collected by a first device and a second device with a standard calibration relationship.

5. The method as described in claim 1, characterized in that, Based on the sample image, predict the attributes of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system when the calibration relationship between the first device and the second device is a standard calibration relationship. These attributes, as the second attribute, specifically include: The sample image is input into a pre-trained prediction model to obtain the attributes of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system, which is used as the second attribute. The calibration model is trained based on the difference between the first attribute and the second attribute, specifically including: The calibration model and the prediction model are jointly trained based on the difference between the first attribute and the second attribute.

6. The method as described in claim 1, characterized in that, The method further includes: In response to a calibration request, determine the target point cloud acquired by the radar sensor and the target image acquired by the image sensor; The target point cloud and the target image are used as inputs to the pre-trained calibration model to obtain the calibration relationship between the radar sensor that acquired the target point cloud and the image sensor that acquired the target image, as output by the calibration model.

7. A training device for calibrating a model, characterized in that, The device includes: The sample determination module is used to determine the sample image acquired by the first device and the sample point cloud acquired by the second device; The relationship determination module is used to input the sample image and the sample point cloud into the calibration model to be trained, and obtain the sample calibration relationship output by the calibration model; The first determining module is used to determine, based on the sample calibration relationship and the sample point cloud, the attribute of the projection result of the sample point cloud in the first coordinate system corresponding to the first device when the calibration relationship between the first device and the second device is the sample calibration relationship, and use it as the first attribute; The second determining module is used to predict, based on the sample image, the attribute of the projection result of the predicted point cloud corresponding to the sample image in the first coordinate system when the calibration relationship between the first device and the second device is a standard calibration relationship, as the second attribute; The training module is used to train the calibration model based on the difference between the first attribute and the second attribute, wherein the calibration model is used to determine the calibration relationship between the image sensor and the radar sensor. The calibration model is trained based on the difference between the first attribute and the second attribute, specifically including: The first loss is determined based on the difference between the first attribute and the second attribute; Determine the true calibration relationship between the second device that acquires the sample point cloud data and the first device that acquires the sample image; Based on the true calibration relationship and the sample point cloud, when the calibration relationship between the first device and the second device is the true calibration relationship, the attribute of the projection result of the sample point cloud data in the first coordinate system corresponding to the first device is determined as the third attribute; The second loss is determined based on the difference between the second attribute and the third attribute; The calibration model is trained with the goal of minimizing the sum of the first loss and the second loss.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 6.

Images