Training method, device, computer equipment and storage medium for three-dimensional detection model

Abstract

The present application relates to a training method, device, computer equipment and storage medium for a three-dimensional detection model. It includes: obtaining a sample driving area image and a sample depth point cloud image; determining the sample point cloud features corresponding to the sample point cloud in the sample depth point cloud image; performing random perturbation processing on the sample depth point cloud image, and determining the sample image pixels located in the sample driving area image corresponding to each sample point cloud in the randomly perturbed sample depth point cloud image; for each sample point cloud in the sample depth point cloud image, the sample point cloud features corresponding to the current sample point cloud are fused with the sample pixel features of the corresponding sample image pixels to obtain the sample target point cloud features; according to the difference between the sample target point cloud features and the point cloud feature labels, a trained three-dimensional detection model is obtained. The use of this method can improve the training accuracy of the three-dimensional detection model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, apparatus, computer equipment and storage medium for a three-dimensional detection model. Background Art

[0002] With the development of computer technology, sensor information fusion technology plays an important role in fields such as autonomous driving. LiDAR and cameras are commonly used sensors. Cameras have the ability to obtain high-resolution information such as color, while LiDAR has accurate three-dimensional measurement capabilities for environmental space. The fusion of the two can more accurately identify moving targets.

[0003] At present, the existing image data and laser point cloud data fusion training algorithm needs to establish a mapping model between two-dimensional plane images and three-dimensional point clouds, and when calculating the mapping model, it is necessary to find the point cloud corresponding to each calibration point on the two-dimensional plane image in the three-dimensional point cloud.

[0004] However, the laser radar and camera are easily affected by the external environment, causing the position to shake and offset, or when the shooting delay occurs, the laser radar and camera timestamps are difficult to match accurately. Therefore, the time and space asynchrony between the laser radar and the camera causes poor accuracy in mapping model training. It can be concluded that how to accurately fuse image data and laser point cloud data is a problem that needs to be solved in this disclosure. Summary of the invention

[0005] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for a three-dimensional detection model that can improve the accuracy of fusion training in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a method for training a three-dimensional detection model. The method comprises:

[0007] Obtaining a sample driving area image, a sample depth point cloud image, and a point cloud feature label corresponding to a sample point cloud in the sample depth point cloud image;

[0008] Determine a sample point cloud feature corresponding to a sample point cloud in the sample depth point cloud image;

[0009] Performing random disturbance processing on the sample depth point cloud image, and determining sample image pixels located in the sample driving area image corresponding to each sample point cloud in the randomly disturbed sample depth point cloud image;

[0010] For each sample point cloud in the sample depth point cloud image, a sample point cloud feature corresponding to the current sample point cloud is fused with a sample pixel feature of a corresponding sample image pixel to obtain a sample target point cloud feature;

[0011] According to the difference between the sample target point cloud features and the point cloud feature labels, the three-dimensional detection model is trained until the training stop condition is reached to obtain a trained three-dimensional detection model; the three-dimensional detection model is used to identify the target object in the driving area.

[0012] In one of the embodiments, the random perturbation processing of the sample depth point cloud image includes: obtaining a calibration matrix corresponding to the sample depth point cloud image; determining a perturbation rotation angle and a perturbation offset corresponding to the calibration matrix, and determining a target calibration matrix corresponding to the calibration matrix based on the perturbation rotation angle and the perturbation offset; and performing random perturbation processing on the sample depth point cloud image through the target calibration matrix.

[0013] In one of the embodiments, the random perturbation processing of the sample depth point cloud image through the target calibration matrix includes: obtaining the random perturbation probability corresponding to the sample depth point cloud image; based on the random perturbation probability, determining the target sample point cloud in the sample depth point cloud image; and randomly perturbating the target sample point cloud through the target calibration matrix.

[0014] In one of the embodiments, the random perturbation processing of the target sample point cloud through the target calibration matrix includes: determining the point cloud coordinates of the target sample point cloud; converting the point cloud coordinates through the target calibration matrix to obtain target coordinates; determining the sample image pixels located in the sample driving area image corresponding to each sample point cloud in the sample depth point cloud image after random perturbation, includes: taking the sample image pixels in the sample driving area image with the target coordinates as the pixels corresponding to the target sample point cloud.

[0015] In one embodiment, the three-dimensional detection model includes a convolution layer, a multi-layer perceptron layer and a normalization layer, and the sample point cloud features corresponding to the current sample point cloud are fused with the sample pixel features of the corresponding sample image pixels to obtain sample target point cloud features, including: performing convolution processing on the sample pixel features of the sample image pixels corresponding to the current sample point cloud through the convolution layer in the three-dimensional detection model to obtain candidate sample pixel features; performing linear processing on the sample point cloud features corresponding to the current sample point cloud through the multi-layer perceptron layer in the three-dimensional detection model to obtain candidate sample point cloud features; performing normalization processing on the candidate sample pixel features and the candidate sample point cloud features through the normalization layer in the three-dimensional detection model; and performing linear combination processing on the normalized candidate sample pixel features and the normalized candidate sample point cloud features to obtain sample target point cloud features.

[0016] In one embodiment, the training method of the three-dimensional detection model also includes training a two-dimensional supervisory structure, and the training steps of the two-dimensional supervisory structure include: obtaining sample pixels in the sample driving area image and pixel feature labels corresponding to the sample pixels; determining the initial sample pixels in the sample driving area image, and the initial sample pixel features corresponding to each of the initial sample pixels, and obtaining the first-round two-dimensional supervisory substructure based on the initial sample pixel features and the pixel feature labels corresponding to the initial sample pixels; determining the round sample pixels in the sample driving area image in the current round starting from the second round after the first round, and determining the round sample pixels corresponding to each of the round sample pixels through the two-dimensional supervisory substructure of the historical rounds. Features; the historical round is at least one round before the current round; according to the two-dimensional supervisory substructure of the historical round, the round sample pixel features, and the pixel feature labels corresponding to the round sample pixels, the two-dimensional supervisory substructure of the current round is obtained; the next round is taken as the current round, and the current round starting from the second round after the first round is returned to determine the round sample pixels in the sample driving area image, and through the two-dimensional supervisory substructure of the historical round, the steps of determining the round sample pixel features corresponding to each of the round sample pixels are continued until the two-dimensional supervisory substructure of the current round is obtained; the two-dimensional supervisory substructure corresponding to each round is integrated to obtain the two-dimensional supervisory structure corresponding to the sample driving area image.

[0017] In one embodiment, the three-dimensional detection model includes a three-dimensional supervision structure and a two-dimensional supervision structure; the three-dimensional supervision structure is used to supervise the accuracy of the sample target point cloud features extracted by the point cloud feature extraction structure; the two-dimensional supervision structure is used to supervise the accuracy of the sample pixel features extracted by the image feature extraction structure; the point cloud feature extraction structure is used to extract sample point cloud features of the sample point cloud; and the image feature extraction structure is used to extract sample pixel features of the sample image pixels.

[0018] In one of the embodiments, the three-dimensional detection model is trained according to the difference between the sample target point cloud features and the point cloud feature labels until a training stop condition is reached to obtain a trained three-dimensional detection model, including: determining a first loss function corresponding to the three-dimensional supervision structure according to the difference between the sample target point cloud features and the point cloud feature labels; determining a second loss function corresponding to the two-dimensional supervision structure according to the difference between the sample pixel features and the pixel feature labels; combining the first loss function and the second loss function to obtain a combined loss function; updating the model parameters in the three-dimensional detection model through the combined loss function and based on the back propagation algorithm until a training stop condition is reached to obtain a trained three-dimensional detection model.

[0019] In a second aspect, the present application also provides a training device for a three-dimensional detection model. The device comprises:

[0020] An image acquisition module, used to acquire a sample driving area image, a sample depth point cloud image, and a point cloud feature label corresponding to a sample point cloud in the sample depth point cloud image;

[0021] A feature recognition module is used to determine the sample point cloud features corresponding to the sample point cloud in the sample depth point cloud image; perform random perturbation processing on the sample depth point cloud image, and determine the sample image pixels located in the sample driving area image corresponding to each sample point cloud in the sample depth point cloud image after random perturbation;

[0022] The model training module is used to fuse the sample point cloud features corresponding to the current sample point cloud with the sample pixel features of the corresponding sample image pixels for each sample point cloud in the sample depth point cloud image to obtain the sample target point cloud features; according to the difference between the sample target point cloud features and the point cloud feature labels, the three-dimensional detection model is trained until the training stop condition is reached to obtain a trained three-dimensional detection model; the three-dimensional detection model is used to identify the target objects in the driving area.

[0023] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0024] Obtaining a sample driving area image, a sample depth point cloud image, and a point cloud feature label corresponding to a sample point cloud in the sample depth point cloud image;

[0025] Determine a sample point cloud feature corresponding to a sample point cloud in the sample depth point cloud image;

[0026] Performing random disturbance processing on the sample depth point cloud image, and determining sample image pixels located in the sample driving area image corresponding to each sample point cloud in the randomly disturbed sample depth point cloud image;

[0027] For each sample point cloud in the sample depth point cloud image, a sample point cloud feature corresponding to the current sample point cloud is fused with a sample pixel feature of a corresponding sample image pixel to obtain a sample target point cloud feature;

[0028] According to the difference between the sample target point cloud features and the point cloud feature labels, the three-dimensional detection model is trained until the training stop condition is reached to obtain a trained three-dimensional detection model; the three-dimensional detection model is used to identify the target object in the driving area.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0030] Obtaining a sample driving area image, a sample depth point cloud image, and a point cloud feature label corresponding to a sample point cloud in the sample depth point cloud image;

[0031] Determine a sample point cloud feature corresponding to a sample point cloud in the sample depth point cloud image;

[0032] Performing random disturbance processing on the sample depth point cloud image, and determining sample image pixels located in the sample driving area image corresponding to each sample point cloud in the randomly disturbed sample depth point cloud image;

[0033] For each sample point cloud in the sample depth point cloud image, a sample point cloud feature corresponding to the current sample point cloud is fused with a sample pixel feature of a corresponding sample image pixel to obtain a sample target point cloud feature;

[0034] According to the difference between the sample target point cloud features and the point cloud feature labels, the three-dimensional detection model is trained until the training stop condition is reached to obtain a trained three-dimensional detection model; the three-dimensional detection model is used to identify the target object in the driving area.

[0035] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0036] Obtaining a sample driving area image, a sample depth point cloud image, and a point cloud feature label corresponding to a sample point cloud in the sample depth point cloud image;

[0037] Determine a sample point cloud feature corresponding to a sample point cloud in the sample depth point cloud image;

[0038] Performing random disturbance processing on the sample depth point cloud image, and determining sample image pixels located in the sample driving area image corresponding to each sample point cloud in the randomly disturbed sample depth point cloud image;

[0039] For each sample point cloud in the sample depth point cloud image, a sample point cloud feature corresponding to the current sample point cloud is fused with a sample pixel feature of a corresponding sample image pixel to obtain a sample target point cloud feature;

[0040] According to the difference between the sample target point cloud features and the point cloud feature labels, the three-dimensional detection model is trained until the training stop condition is reached to obtain a trained three-dimensional detection model; the three-dimensional detection model is used to identify the target object in the driving area.

[0041] The training method, device, computer equipment, storage medium and computer program product of the above-mentioned three-dimensional detection model obtains the sample driving area image and the sample depth point cloud image, and performs random perturbation processing on the sample depth point cloud image, so that each sample point cloud in the randomly perturbed sample depth point cloud image can be determined to correspond to the sample image pixel in the sample driving area image; and then the sample point cloud feature corresponding to each sample point cloud is fused with the sample pixel feature of the corresponding sample image pixel to obtain the sample target point cloud feature, so that the three-dimensional detection model is trained by the sample target point cloud feature to obtain a trained three-dimensional detection model. Since the present application performs random perturbation processing on the sample depth point cloud image, and then fuses the sample point cloud feature with the corresponding sample pixel feature, compared with the traditional algorithm that directly performs fusion training through the mapping model, the present application directly uses the fused sample target point cloud feature as the training object, which greatly improves the training efficiency of the three-dimensional detection model. At the same time, since the sample depth point cloud image and the sample driving area image after random perturbation processing are used as the fusion object, the influence of external environmental interference on the fusion process is overcome, and the anti-disturbance ability of the three-dimensional detection model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A diagram of an application environment of a training method for a three-dimensional detection model in one embodiment;

[0043] Figure 2 A schematic diagram of a flow chart of a method for training a three-dimensional detection model in one embodiment;

[0044] Figure 3 A schematic diagram of a process for training a two-dimensional supervision structure in one embodiment;

[0045] Figure 4 A schematic diagram of the principle of a training method for a three-dimensional detection model in an embodiment;

[0046] Figure 5 is a structural block diagram of a training device for a three-dimensional detection model in one embodiment;

[0047] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The training method of the three-dimensional detection model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. Among them, the terminal 102 is used to send the acquired sample driving area image and sample depth point cloud image to the server 104. The server 104 is used to fuse and train the received sample driving area image and sample depth point cloud image, and return the trained three-dimensional detection model to the terminal 102. The terminal 102 can be used to receive the trained three-dimensional detection model so that the target object in the driving area can be identified by the three-dimensional detection model later. Among them, the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0050] In one embodiment, Figure 2 As shown, a training method for a three-dimensional detection model is provided, and the method is applied to Figure 1 The computer device in the example is used to illustrate that the computer device can be Figure 1 In the terminal or server, the training method of the three-dimensional detection model includes the following steps:

[0051] Step 202: Obtain a sample driving area image, a sample depth point cloud image, and a point cloud feature label corresponding to a sample point cloud in the sample depth point cloud image.

[0052] Among them, the sample driving area image is a two-dimensional image, which can be collected by cameras, camera heads and other devices; the sample depth point cloud image is a three-dimensional image, which can be collected by laser radar and other devices; the point cloud feature label is the sample point cloud feature corresponding to the sample point cloud in the sample depth point cloud image, such as a 3D detection box.

[0053] Specifically, the user can take multiple photos of the area where the vehicle is located through a camera to obtain multiple sample driving area images of the vehicle, and scan the area where the vehicle is located through a lidar multiple times to obtain multiple sample depth point cloud images of the vehicle, and input the collected depth point cloud images and driving area images into a computer device.

[0054] In one of the embodiments, sample driving area images and sample depth point cloud images can be obtained through data augmentation. Data augmentation can make data samples more diversified, thereby making the trained three-dimensional detection model have stronger generalization ability.

[0055] In one embodiment, a target recognition interface is displayed in the computer device, and the target recognition interface displays a laser radar control button and a camera control button. The computer device responds to the user's triggering operation on the camera control button, that is, by adjusting the camera shooting angle through the control button, determines the driving area to be recognized, and collects sample driving area images of the driving area; after determining the driving area to be collected, the computer device responds to the user's triggering operation on the laser radar control button to collect sample depth point cloud images of the driving area.

[0056] Step 204 : determining the sample point cloud features corresponding to the sample point cloud in the sample depth point cloud image.

[0057] Among them, the point cloud refers to a set of point data on the surface of an object obtained by a measuring instrument. Sample point cloud features are classified according to geometric features and may include point-by-point features, voxel features, front view projection features, etc.

[0058] Specifically, the computer device performs a first feature extraction on the sample driving area image through an image feature extraction network to obtain corresponding sample pixel features; the computer device performs a second feature extraction on the sample depth point cloud image through a point cloud feature extraction network to obtain sample point cloud features corresponding to each sample point cloud. The image feature extraction network may be ResNet50 (Residual Network), and the point cloud feature extraction network may be SECOND (Sparsely Embedded Convolutional Detection).

[0059] In one embodiment, the computer device can determine sample pixel features of different granularity information through an image feature extraction network, and combine the sample pixel features of different granularity information to obtain multi-scale sample pixel features. The pixel value in the sample pixel feature can be a gray value or a color value, which can provide color information of the sample driving area image.

[0060] Step 206 , performing random disturbance processing on the sample depth point cloud image, and determining the sample image pixels located in the sample driving area image corresponding to each sample point cloud in the randomly disturbed sample depth point cloud image.

[0061] Specifically, the computer device obtains a calibration matrix corresponding to the sample depth point cloud image, wherein the calibration matrix is ​​a mapping matrix corresponding to when the point cloud coordinates of the three-dimensional image are projected onto the pixel coordinates of the two-dimensional image. The computer device performs random perturbation processing on the calibration matrix, such as adding a perturbation matrix including a perturbation rotation angle and a perturbation offset, and multiplies the calibration matrix by a homogeneous matrix of the perturbation matrix, thereby obtaining a target calibration matrix corresponding to the calibration matrix after the random perturbation processing of the calibration matrix.

[0062] Furthermore, the computer device determines the point cloud coordinates of each sample point cloud in the sample depth point cloud image, and projects the point cloud coordinates of each sample point cloud onto the sample driving area image through the target calibration matrix, and determines the sample image pixels in the sample driving area image that correspond to the point cloud coordinates of each sample point cloud, that is, projects the three-dimensional point cloud coordinates into two-dimensional pixel coordinates.

[0063] In one of the embodiments, the depth point cloud image is randomly perturbed using a target calibration matrix, including: obtaining a random perturbation probability corresponding to the sample depth point cloud image; determining a target sample point cloud in the sample depth point cloud image based on the random perturbation probability; and randomly perturbating the target sample point cloud using a target calibration matrix.

[0064] Among them, the value of the random disturbance probability needs to be determined through practical professional experience in the field of three-dimensional detection.

[0065] The computer device obtains the random disturbance probability corresponding to the sample depth point cloud image, and distinguishes the target sample point cloud from the non-target sample point cloud from all the sample point clouds in the sample depth point cloud image according to the random disturbance probability, and then projects the point cloud coordinates of the target sample point cloud onto the sample driving area image through the target calibration matrix. For example, if the random disturbance probability obtained is 50%, the total number of sample point clouds in the sample depth point cloud image is 100, therefore, 50 of the 100 sample point clouds are randomly selected as target sample point clouds, and the remaining 50 are selected as non-target sample point clouds, and then the point cloud coordinates of the 50 target sample point clouds are projected onto the sample driving area image. Since only the target sample point cloud is projected through the target calibration matrix, the amount of data processed for the sample point cloud is reduced.

[0066] Step 208 : For each sample point cloud in the sample depth point cloud image, the sample point cloud feature corresponding to the current sample point cloud is fused with the sample pixel feature of the corresponding sample image pixel to obtain the sample target point cloud feature.

[0067] Specifically, the computer device determines the point cloud coordinates and sample point cloud features of each sample point cloud, the sample image pixels corresponding to each sample point cloud coordinate, and determines the sample pixel features of the sample image pixels corresponding to the sample point cloud features of each sample point cloud according to the sample image pixels corresponding to each sample point cloud coordinate. The sample pixel features are obtained by extracting features of the sample image pixels using a two-dimensional supervisory structure. The computer device fuses the sample point cloud features corresponding to each sample point cloud with the sample pixel features of the corresponding sample image pixels to obtain the target sample point cloud features corresponding to each sample point cloud.

[0068] In one embodiment, the computer device may fuse the sample point cloud features corresponding to the sample point cloud with the sample pixel features of the corresponding sample image pixels through the following formula:

[0069]

[0070] in, represents the target sample point cloud features obtained after feature fusion, n represents the number of sample point clouds, and f n Represents the sample point cloud features corresponding to the nth sample point cloud, Represents the sample pixel features of the sample image pixel corresponding to the nth sample point cloud, Conv 1×1 Indicates one-dimensional convolution processing of sample pixel features, MLP indicates linearization processing of sample point cloud features, and LN indicates normalization processing of sample pixel features after one-dimensional convolution processing and sample point cloud features after linearization processing. It can represent a way to linearly combine the normalized features.

[0071] In one of the embodiments, a computer device may train a two-dimensional supervisory structure through sample pixel features of sample image pixels, and the trained two-dimensional supervisory structure is used for two-dimensional semantic segmentation, two-dimensional target tracking, etc.

[0072] Step 210, training the three-dimensional detection model according to the difference between the sample target point cloud features and the point cloud feature labels until the training stop condition is reached, thereby obtaining a trained three-dimensional detection model.

[0073] For example, the 3D detection model can be a 3D detection head. When the training sample is a sample target point cloud feature, the training label is a 3D detection box. When the loss function of the 3D detection head converges, it is considered that the training stop condition has been reached, and a trained 3D detection head can be obtained. The trained 3D detection model can be used to perform target object recognition, 3D semantic segmentation, 3D target tracking, etc. on target objects in the driving area.

[0074] In one embodiment, the three-dimensional detection model includes a three-dimensional supervision structure and a two-dimensional supervision structure; the three-dimensional supervision structure is used to supervise the accuracy of the sample target point cloud features extracted by the point cloud feature extraction structure; the two-dimensional supervision structure is used to supervise the accuracy of the sample pixel features extracted by the image feature extraction structure; the point cloud feature extraction structure is used to extract sample point cloud features of the sample point cloud; and the image feature extraction structure is used to extract sample pixel features of the sample image pixels.

[0075] In one of the embodiments, the computer device can realize two-dimensional semantic segmentation, two-dimensional target tracking, etc. through a three-dimensional detection model.

[0076] In one embodiment, the computer device can determine the object features of the target object in the driving area through the three-dimensional detection model, wherein the object features include the center point coordinates of the target object, the side length of the three-dimensional detection box, and the rotation angle of the three-dimensional detection box. The computer device completes the recognition of the target object in the driving area based on the analyzed object features.

[0077] In one of the embodiments, a target recognition interface is displayed in a computer device, and the target object and the object features of the target object are displayed through the target recognition interface, and the identified target object is selected through a three-dimensional detection frame.

[0078] In the training method of the above-mentioned three-dimensional detection model, by obtaining a sample driving area image and a sample depth point cloud image, and performing random perturbation processing on the sample depth point cloud image, it is possible to determine the sample image pixels located in the sample driving area image corresponding to each sample point cloud in the randomly perturbed sample depth point cloud image; and then the sample point cloud features corresponding to each sample point cloud are fused with the sample pixel features of the corresponding sample image pixels to obtain the sample target point cloud features, so that the three-dimensional detection model is trained through the sample target point cloud features to obtain a trained three-dimensional detection model. Since the present application performs random perturbation processing on the sample depth point cloud image and then fuses the sample point cloud features with the corresponding sample pixel features, compared with the traditional algorithm that directly performs fusion training through a mapping model, the present application directly uses the fused sample target point cloud features as the training object, which greatly improves the training efficiency of the three-dimensional detection model. At the same time, since the sample depth point cloud image and the sample driving area image after random perturbation processing are used as fusion objects, the influence of external environmental interference on the fusion process is overcome, and the anti-disturbance ability of the three-dimensional detection model is improved.

[0079] In one of the embodiments, the sample depth point cloud image is subjected to random perturbation processing, including: obtaining a calibration matrix corresponding to the sample depth point cloud image; determining a perturbation rotation angle and a perturbation offset corresponding to the calibration matrix, and determining a target calibration matrix corresponding to the calibration matrix based on the perturbation rotation angle and the perturbation offset; and performing random perturbation processing on the sample depth point cloud image through the target calibration matrix.

[0080] Among them, the disturbance rotation angle is a three-dimensional rotation angle, which is a deflection angle that can be formed by disturbance rotation along the X-axis, Y-axis and Z-axis respectively, and the disturbance offset is a distance formed by disturbance offset along the X-axis, Y-axis and Z-axis respectively.

[0081] Specifically, the computer device obtains the calibration matrix corresponding to the sample depth point cloud image, wherein the calibration matrix is ​​a fourth-order matrix composed of a calibration rotation angle and a calibration offset, and the corresponding perturbation rotation angle and perturbation offset are determined by performing a certain degree of random perturbation on the calibration matrix, and then a fourth-order perturbation matrix is ​​constructed by the perturbation rotation angle and the perturbation offset, and the perturbation matrix is ​​homogeneously transformed to form a perturbation homogeneous matrix. The computer device can obtain a target calibration matrix corresponding to the calibration matrix by multiplying the calibration matrix with the perturbation homogeneous matrix, and then the sample depth point cloud image can be randomly perturbed by the target calibration matrix.

[0082] For example, when the calibration matrix of the sample depth point cloud image is , where R and t are the rotation matrix and offset vector in the calibration matrix respectively.

[0083] By performing a certain degree of random perturbation on the calibration matrix, the perturbation offset is obtained as vector t1 and the perturbation rotation angle is ω = (ω1, ω2, ω3) T , where ω1, ω2, and ω3 are the deflection angles formed by the perturbation rotation along the X-axis, Y-axis, and Z-axis, respectively.

[0084] The rotation matrix corresponding to the perturbation rotation angle is obtained as follows: Where I is the identity matrix.

[0085] According to the disturbance offset and the disturbance rotation angle, the obtained disturbance matrix is:

[0086] According to the calibration matrix and the perturbation matrix, the target calibration matrix can be obtained as: Among them, T1′ is a homogeneous transformation of the perturbation matrix to form a perturbation homogeneous matrix.

[0087] In this embodiment, by performing random perturbation processing on the calibration matrix, the subsequent fusion process can adapt to target calibration matrices of different qualities, thereby enhancing the accuracy of converting point cloud coordinates through the target calibration matrix.

[0088] In one of the embodiments, a target sample point cloud is randomly perturbed through a target calibration matrix, including: determining point cloud coordinates of the target sample point cloud; transforming the point cloud coordinates through the target calibration matrix to obtain target coordinates; determining sample image pixels in a sample driving area image corresponding to each sample point cloud in the sample depth point cloud image after random perturbation, including: using sample image pixels in the sample driving area image with target coordinates as pixels corresponding to the target sample point cloud.

[0089] Among them, the target sample point cloud is a point cloud selected from the sample depth point cloud image according to the random perturbation probability.

[0090] Specifically, the computer device determines the point cloud coordinates of the target sample point cloud, and transforms the sample point cloud coordinates through the target calibration matrix to obtain the target coordinates corresponding to the sample point cloud coordinates, that is, projecting the point cloud coordinates of the target sample point cloud onto the sample driving area image. The target coordinates are the coordinates corresponding to the sample image pixels in the sample driving area image. The computer device determines the sample image pixels in the sample driving area image corresponding to the target sample point cloud through the target coordinates.

[0091] For example, through the coordinate transformation formula:

[0092] Among them, π is the intrinsic parameter matrix of the sample depth point cloud image, R2 and t2 are the rotation matrix and offset vector in the target calibration matrix respectively. When the point cloud coordinates of the target point cloud are (X, Y, Z), the target coordinates (x, y) projected onto the sample driving area image can be obtained through the target calibration matrix.

[0093] In this embodiment, since only the coordinate transformation is performed on the target sample point cloud in the randomly disturbed sample depth point cloud image, the efficiency of the subsequent feature fusion process is improved.

[0094] In one of the embodiments, the computer device transforms the point cloud coordinates of the non-target sample point cloud through a calibration matrix, that is, projects the point cloud coordinates of the non-target sample point cloud onto the sample driving area image.

[0095] In one embodiment, the three-dimensional detection model includes a convolution layer, a multi-layer perceptron layer and a normalization layer, and the sample point cloud features corresponding to the current sample point cloud are merged with the sample pixel features of the corresponding sample image pixels to obtain the sample target point cloud features, including: performing convolution processing on the sample pixel features of the sample image pixels corresponding to the current sample point cloud through the convolution layer in the three-dimensional detection model to obtain candidate sample pixel features; performing linear processing on the sample point cloud features corresponding to the current sample point cloud through the multi-layer perceptron layer in the three-dimensional detection model to obtain candidate sample point cloud features; performing normalization processing on the candidate sample pixel features and the candidate sample point cloud features through the normalization layer in the three-dimensional detection model; performing linear combination processing on the normalized candidate sample pixel features and the normalized candidate sample point cloud features to obtain sample target point cloud features.

[0096] Specifically, the computer device determines the sample point cloud features corresponding to the current sample point cloud for each target sample point cloud in the sample depth point cloud image, and inputs the sample point cloud features corresponding to the current sample point cloud into the multi-layer perceptron layer in the three-dimensional detection model, and linearly processes the point cloud features of the current sample point cloud through the multi-layer perceptron layer to obtain processed candidate sample point cloud features. At the same time, the computer device determines the sample pixel features corresponding to each sample image pixel through the sample image pixels corresponding to each target sample point cloud, and inputs the sample pixel features corresponding to the current sample point cloud into the convolution layer in the three-dimensional detection model, and performs one-dimensional convolution processing on the sample pixel features of the current sample point cloud through the convolution layer to obtain processed candidate sample pixel features, wherein the candidate sample pixel features may be a pixel feature including information of different granularities.

[0097] Furthermore, the computer device inputs the candidate sample pixel features and candidate sample point cloud features corresponding to each sample point cloud into the normalization layer in the three-dimensional detection model, and normalizes the candidate sample pixel features and candidate sample point cloud features corresponding to each sample point cloud respectively through the normalization layer, and linearly combines the normalized candidate sample pixel features and the normalized candidate sample point cloud features to obtain the target sample point cloud features. The normalized candidate sample pixel features and the normalized candidate sample point cloud features can be linearly combined by the concat feature fusion algorithm.

[0098] In this embodiment, by directly fusing the sample pixel features and the sample point cloud features, the fused target sample point cloud features are obtained, and then the three-dimensional detection model can be quickly trained through the fused target sample point cloud features.

[0099] In one of the embodiments, the computer device inputs the fused target sample point cloud features into a three-dimensional detection head, and performs three-dimensional supervision on the target sample point cloud features through a pre-acquired three-dimensional detection frame.

[0100] In this embodiment, the training efficiency of the three-dimensional detection model is improved by adopting a method of fusing the sample point cloud features corresponding to the sample point cloud with the sample pixel features of the corresponding sample image pixels for training.

[0101] In one of the embodiments, a three-dimensional detection model is trained according to the difference between the sample target point cloud features and the point cloud feature labels until a training stop condition is reached to obtain a trained three-dimensional detection model, including: determining a first loss function corresponding to the three-dimensional supervision structure according to the difference between the sample target point cloud features and the point cloud feature labels; determining a second loss function corresponding to the two-dimensional supervision structure according to the sample pixel features and the pixel feature labels; combining the first loss function and the second loss function to obtain a combined loss function; updating the model parameters in the three-dimensional detection model through the combined loss function and based on the back propagation algorithm until a training stop condition is reached to obtain a trained three-dimensional detection model.

[0102] Among them, the three-dimensional supervision structure is a structure for supervising the sample target point cloud features extracted by the point cloud feature extraction structure; the two-dimensional supervision structure is a structure for supervising the sample pixel features extracted by the image feature extraction structure; the first loss function and the second loss function can both be a hinge loss function, an exponential loss function, etc.; the back propagation algorithm is an algorithm for adjusting the weight parameters of the network model according to the error between the true value and the estimated value. For example, the back propagation algorithm can be a gradient descent algorithm, in which the sum of squares of errors can be used as the objective function, and the partial derivatives of the objective function to the weights of each neuron are obtained layer by layer to form the gradient of the objective function to the weight parameters, thereby modifying the model parameters according to the direction of the gradient descent, and the training of the network model is completed in the process of modifying the model parameters.

[0103] Specifically, the computer device determines the first loss function corresponding to the three-dimensional supervision structure according to the difference between the sample target point cloud features and the point cloud feature labels, and determines the second loss function corresponding to the two-dimensional supervision structure according to the sample pixel features and the pixel feature labels. The computer device combines the first loss function with the second loss function to obtain a joint loss function, and the joint loss function is used to update the three-dimensional detection model. The computer device can obtain the estimated sample target point cloud features and estimated sample pixel features corresponding to the sample point cloud through the three-dimensional detection model trained in the previous round, and based on the point cloud feature labels and the estimated sample target point cloud features, the pixel feature labels and the estimated sample pixel features, and according to the joint loss function, the point cloud feature loss value can be obtained.

[0104] Furthermore, the computer device determines the gradient of the point cloud feature loss value of the joint loss function through the back-propagation algorithm, and updates the model parameters in the three-dimensional detection model along the gradient direction of the point cloud feature loss value, that is, the point cloud feature loss value is minimized as much as possible by using the attenuation of the model parameters, so that the joint loss function converges and a trained three-dimensional detection model is obtained.

[0105] In this embodiment, the update method of the joint weight parameter is determined by combining the first loss function and the second loss function, and then the three-dimensional detection model is updated and trained, thereby improving the performance of the three-dimensional detection model as a whole, and realizing the joint training of the two-dimensional supervision structure and the three-dimensional supervision structure.

[0106] In one embodiment, the training method of the three-dimensional detection model further includes training the two-dimensional supervision structure, such as Figure 3 As shown, the steps of training a two-dimensional supervision structure include:

[0107] Step 301: Obtain sample pixels in a sample driving area image and pixel feature labels corresponding to the sample pixels.

[0108] The training of sample pixels in the sample driving area image is mainly for training a model that can accurately extract features from sample image pixels in the three-dimensional detection model. The pixel feature label can be a two-dimensional label, such as a two-dimensional detection box, a two-dimensional semantic label, and the like.

[0109] Step 302, determine the initial sample pixels in the sample driving area image and the initial sample pixel features corresponding to each initial sample pixel, and obtain the first round of two-dimensional supervision substructure according to the initial sample pixel features and the pixel feature labels corresponding to the initial sample pixels.

[0110] The initial sample pixel may be at least one sample pixel randomly selected by the user from the sample pixels before coordinate projection of the target sample point cloud, or may be a projection pixel of the non-target sample point cloud in the sample driving area image determined according to the random disturbance probability.

[0111] The computer device trains the two-dimensional supervision structure according to the difference between the initial sample pixel features and the pixel feature labels until the training stop condition is reached, and obtains the first round of trained two-dimensional supervision substructure. For example, if sample pixel No. 1 and sample pixel No. 2 are the initial sample pixels, the first round of two-dimensional supervision substructure can be obtained through training.

[0112] Step 303, from the current round starting from the second round after the first round, determine the round sample pixels in the sample driving area image, and determine the round sample pixel features corresponding to each round sample pixel through the two-dimensional supervision substructure of the historical rounds; the historical round is at least one round before the current round.

[0113] The round sample pixel can be at least one sample pixel randomly selected by the user from the sample pixels after the coordinate projection of the target sample point cloud, or it can be the projection pixel of the target sample point cloud in the sample driving area image determined according to the random perturbation probability. The number of round sample pixels can be specified by the user or determined by the number of sample target point clouds in the 3D detection model. The execution of the current round can be specified by the user or determined by the process of fusing the sample point cloud features in the 3D detection model with the sample pixel features.

[0114] For example, referring to the above example, when it is determined that the sample image pixel corresponding to the sample point cloud of sample target No. 3 is sample pixel No. 3, that is, sample pixel No. 3 is a round sample pixel, the execution process of the second round after the first round can be started. Based on the two-dimensional supervision substructure of the first round, the round sample pixel features of sample pixel No. 3 can be determined, and then the round sample pixel features and the pixel feature labels corresponding to the round sample pixels are added to the first round two-dimensional supervision substructure for training to obtain the trained two-dimensional supervision substructure of the current round. At the same time, in the three-dimensional detection model, it can be determined to fuse the sample point cloud features of sample point cloud No. 3 with the corresponding round sample pixel features of sample pixel No. 3 to obtain the process of sample target point cloud features No. 3. Among them, sample target point cloud No. 3 can be at least one sample target point cloud, that is, sample pixel No. 3 can be at least one sample pixel.

[0115] Step 304 , obtaining a trained two-dimensional supervision substructure of the current round according to the two-dimensional supervision substructure of the historical round, the round sample pixel features, and the pixel feature labels corresponding to the round sample pixels.

[0116] The computer device trains the two-dimensional supervision substructure based on the historical rounds and in combination with the round sample pixel features and the difference between the pixel feature labels corresponding to the round sample pixels until the training stop condition is reached, thereby obtaining the trained two-dimensional supervision substructure of the current round.

[0117] Step 305, taking the next round as the current round, and returning to the current round starting from the second round after the first round, determining the round sample pixels in the sample driving area image, and through the two-dimensional supervisory substructure of the historical rounds, determining the steps of the round sample pixel features corresponding to each round sample pixel and continuing to execute until the trained two-dimensional supervisory substructure of the current round is obtained.

[0118] Step 306 , synthesizing the two-dimensional supervision substructures corresponding to each round to obtain a two-dimensional supervision structure corresponding to the sample driving area image.

[0119] In this embodiment, the direct supervision of the driving area image by the two-dimensional supervision structure can obtain better sample pixel features, which helps to improve the effect of the fusion process of the sample point cloud features and sample pixel features in the three-dimensional detection model. At the same time, since the two-dimensional supervision structure is only an auxiliary supervision model, the sample pixel features are only calculated during the training process, and do not participate in the calculation of the process of identifying the target object through the three-dimensional detection model, which improves the convenience of using the three-dimensional detection model.

[0120] In one embodiment, Figure 4 As shown, Figure 4 The schematic diagram of the principle of the training method of the three-dimensional detection model in another embodiment. The computer device obtains the sample driving area image, the sample depth point cloud image, and the calibration matrix corresponding to the sample depth point cloud image, and inputs the sample driving area image into the multi-level image feature extraction network to extract the multi-scale sample pixel features of the sample driving area image, and at the same time inputs the sample depth point cloud image into the multi-stage feature extraction network to extract the sample point cloud features of the sample depth point cloud image. The computer device adds a random disturbance including a random disturbance probability, a disturbance rotation angle, and a disturbance offset to the calibration matrix to obtain a disturbed target calibration matrix, for example, adding a random disturbance with a random disturbance probability of 50%, a disturbance rotation angle of 0.2 degrees along the X-axis, and a disturbance offset that conforms to a 20cm Gaussian distribution. Based on the disturbed target calibration matrix, the computer device projects the coordinates corresponding to the sample point cloud features of the sample depth point cloud image onto the sample driving area image, and determines the projection points on the sample driving area image and the multi-scale sample pixel features corresponding to the projection points.

[0121] Furthermore, the computer device fuses the sample point cloud features with the sample pixel features to form a fused target sample point cloud feature, and inputs the target sample point cloud feature into the 3D detection head, and performs 3D supervision on the target sample point cloud feature through a preset 3D detection frame. At the same time, the computer device inputs the multi-level sample pixel features into the 2D detection head, and performs 2D supervision on the sample pixel features through a preset 2D detection frame. Among them, 2D supervision is used to assist 3D supervision.

[0122] In this embodiment, by performing 3D supervision on the fused target point cloud features, a trained 3D detection model can be accurately obtained, and at the same time, 2D supervision is added as auxiliary supervision, so that the sample point cloud features and the sample pixel features have a better fusion effect.

[0123] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0124] Based on the same inventive concept, the embodiment of the present application also provides a training device for a three-dimensional detection model for implementing the training method for a three-dimensional detection model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more three-dimensional detection models provided below can refer to the limitations of the training method for a three-dimensional detection model above, and will not be repeated here.

[0125] In one embodiment, Figure 5 As shown, a three-dimensional detection model training device 500 is provided, including: an image acquisition module 502, a feature recognition module 504 and a model training module 506, wherein:

[0126] Image acquisition module 502, used to acquire sample driving area images, sample depth point cloud images and point cloud feature labels corresponding to sample point clouds in the sample depth point cloud images

[0127] The feature recognition module 504 is used to determine the sample point cloud features corresponding to the sample point cloud in the sample depth point cloud image; perform random perturbation processing on the sample depth point cloud image, and determine the sample image pixels located in the sample driving area image corresponding to each sample point cloud in the randomly perturbed sample depth point cloud image;

[0128] The model training module 506 is used to fuse the sample point cloud features corresponding to the current sample point cloud with the sample pixel features of the corresponding sample image pixels for each sample point cloud in the sample depth point cloud image, so as to obtain the sample target point cloud features; and train the three-dimensional detection model according to the difference between the sample target point cloud features and the point cloud feature labels until the training stop condition is reached, so as to obtain a trained three-dimensional detection model.

[0129] In one embodiment, the feature recognition module 504 also includes a disturbance processing module 5041, which is used to obtain a calibration matrix corresponding to the sample depth point cloud image; determine the disturbance rotation angle and the disturbance offset corresponding to the calibration matrix, and determine the target calibration matrix corresponding to the calibration matrix based on the disturbance rotation angle and the disturbance offset; and perform random disturbance processing on the sample depth point cloud image through the target calibration matrix.

[0130] In one of the embodiments, the disturbance processing module 5041 is also used to obtain the random disturbance probability corresponding to the sample depth point cloud image; based on the random disturbance probability, determine the target sample point cloud in the sample depth point cloud image; and perform random disturbance processing on the target sample point cloud through the target calibration matrix.

[0131] In one embodiment, the feature recognition module 504 also includes a coordinate conversion module 5042, which is used to determine the point cloud coordinates of the target sample point cloud; convert the point cloud coordinates through the target calibration matrix to obtain the target coordinates; determine the sample image pixels located in the sample driving area image corresponding to each sample point cloud in the sample depth point cloud image after random disturbance, including: taking the sample image pixels with target coordinates in the sample driving area image as pixels corresponding to the target sample point cloud.

[0132] In one embodiment, the model training module 506 includes a three-dimensional model training module 5061, which is used for a three-dimensional detection model including a convolution layer, a multi-layer perceptron layer and a normalization layer, and the sample point cloud features corresponding to the current sample point cloud are merged with the sample pixel features of the corresponding sample image pixels to obtain sample target point cloud features, including: through the convolution layer in the three-dimensional detection model, the sample pixel features of the sample image pixels corresponding to the current sample point cloud are convolved to obtain candidate sample pixel features; through the multi-layer perceptron layer in the three-dimensional detection model, the sample point cloud features corresponding to the current sample point cloud are linearly processed to obtain candidate sample point cloud features; through the normalization layer in the three-dimensional detection model, the candidate sample pixel features and the candidate sample point cloud features are normalized; the normalized candidate sample pixel features and the normalized candidate sample point cloud features are linearly combined to obtain sample target point cloud features.

[0133] In one embodiment, the model training module 506 includes a supervision structure training module 5062, which is used to obtain sample pixels in the sample driving area image and pixel feature labels corresponding to the sample pixels; determine the initial sample pixels in the sample driving area image, and the initial sample pixel features corresponding to each initial sample pixel, and obtain the first round of two-dimensional supervision substructure according to the initial sample pixel features and the pixel feature labels corresponding to the initial sample pixels; determine the round sample pixels in the sample driving area image in the current round starting from the second round after the first round, and determine the round sample pixel features corresponding to each round sample pixel through the two-dimensional supervision substructure of the historical rounds; the historical rounds. The round is at least one round before the current round. According to the two-dimensional supervision substructure of the historical round, the round sample pixel features, and the pixel feature labels corresponding to the round sample pixels, the two-dimensional supervision substructure of the current round is obtained; the next round is taken as the current round, and the current round starting from the second round after the first round is returned to determine the round sample pixels in the sample driving area image, and the steps of determining the round sample pixel features corresponding to each round sample pixel through the two-dimensional supervision substructure of the historical round are continued until the two-dimensional supervision substructure of the current round is obtained; the two-dimensional supervision substructure corresponding to each round is integrated to obtain the two-dimensional supervision structure corresponding to the sample driving area image.

[0134] In one embodiment, the training device 500 of the three-dimensional detection model includes a three-dimensional supervision structure and a two-dimensional supervision structure; the three-dimensional supervision structure is used to supervise the accuracy of the sample target point cloud features extracted by the point cloud feature extraction structure; the two-dimensional supervision structure is used to supervise the accuracy of the sample pixel features extracted by the image feature extraction structure; the point cloud feature extraction structure is used to extract sample point cloud features of the sample point cloud; and the image feature extraction structure is used to extract sample pixel features of the sample image pixels.

[0135] In one embodiment, the model training module 506 includes a model joint update module 5063, which is used to determine a first loss function corresponding to the three-dimensional supervision structure based on the difference between the sample target point cloud features and the point cloud feature labels; determine a second loss function corresponding to the two-dimensional supervision structure based on the sample pixel features and the pixel feature labels; combine the first loss function and the second loss function to obtain a joint loss function; update the model parameters in the three-dimensional detection model through the joint loss function and based on the back propagation algorithm until the training stop condition is reached, thereby obtaining a trained three-dimensional detection model.

[0136] Each module in the training of the above three-dimensional detection model can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0137] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample point cloud data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a three-dimensional detection model is implemented.

[0138] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0140] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0141] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0142] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0143] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a three-dimensional detection model, characterized in that: The method comprises: Obtaining a sample driving area image, a sample depth point cloud image, and a point cloud feature label corresponding to a sample point cloud in the sample depth point cloud image; Determine a sample point cloud feature corresponding to a sample point cloud in the sample depth point cloud image; Performing random disturbance processing on the sample depth point cloud image, and determining sample image pixels located in the sample driving area image corresponding to each sample point cloud in the randomly disturbed sample depth point cloud image; For each sample point cloud in the sample depth point cloud image, a sample point cloud feature corresponding to the current sample point cloud is fused with a sample pixel feature of a corresponding sample image pixel to obtain a sample target point cloud feature; According to the difference between the sample target point cloud features and the point cloud feature labels, the three-dimensional detection model is trained until a training stop condition is reached, thereby obtaining a trained three-dimensional detection model; the three-dimensional detection model is used to identify the target object in the driving area; The step of fusing the sample point cloud features corresponding to the current sample point cloud with the sample pixel features of the corresponding sample image pixels to obtain the sample target point cloud features includes: The sample pixel features of the sample image pixels corresponding to the current sample point cloud are convolved through the convolution layer in the three-dimensional detection model to obtain candidate sample pixel features; the sample point cloud features corresponding to the current sample point cloud are linearly processed through the multi-layer perceptron layer in the three-dimensional detection model to obtain candidate sample point cloud features; the candidate sample pixel features and the candidate sample point cloud features are normalized through the normalization layer in the three-dimensional detection model; the normalized candidate sample pixel features and the normalized candidate sample point cloud features are linearly combined to obtain sample target point cloud features.

2. The method according to claim 1, characterized in that The randomly perturbing the sample depth point cloud image comprises: Obtaining a calibration matrix corresponding to the sample depth point cloud image; Determine a disturbance rotation angle and a disturbance offset corresponding to the calibration matrix, and determine a target calibration matrix corresponding to the calibration matrix according to the disturbance rotation angle and the disturbance offset; The sample depth point cloud image is randomly perturbed using the target calibration matrix.

3. The method according to claim 2, characterized in that The randomly perturbing the sample depth point cloud image by using the target calibration matrix includes: Obtaining a random disturbance probability corresponding to the sample depth point cloud image; Based on the random disturbance probability, determining a target sample point cloud in the sample depth point cloud image; The target sample point cloud is randomly disturbed by the target calibration matrix.

4. The method according to claim 3, characterized in that: The performing random perturbation processing on the target sample point cloud by using the target calibration matrix includes: Determining the point cloud coordinates of the target sample point cloud; The point cloud coordinates are transformed by the target calibration matrix to obtain target coordinates; The determining of the sample image pixels located in the sample driving area image corresponding to each sample point cloud in the sample depth point cloud image after random disturbance comprises: The sample image pixels having the target coordinates in the sample driving area image are used as pixels corresponding to the target sample point cloud.

5. The method according to claim 1, characterized in that The training method of the three-dimensional detection model also includes training a two-dimensional supervisory structure; the two-dimensional supervisory structure is used to determine sample pixel features of sample image pixels corresponding to the sample point cloud; The training steps of the two-dimensional supervision structure include: Obtaining sample pixels in the sample driving area image and pixel feature labels corresponding to the sample pixels; Determine the initial sample pixels in the sample driving area image and the initial sample pixel features corresponding to each of the initial sample pixels, and obtain the first round of two-dimensional supervision substructure according to the initial sample pixel features and the pixel feature labels corresponding to the initial sample pixels; In a current round starting from the second round after the first round, round sample pixels in the sample driving area image are determined, and round sample pixel features corresponding to each of the round sample pixels are determined through the two-dimensional supervision substructure of the historical rounds; the historical rounds are at least one round before the current round; Obtaining a two-dimensional supervision substructure of a current round according to the two-dimensional supervision substructure of a historical round, the round sample pixel features, and the pixel feature labels corresponding to the round sample pixels; Taking the next round as the current round, and returning to the current round from the second round after the first round, determining the round sample pixels in the sample driving area image, and determining the round sample pixel features corresponding to each of the round sample pixels through the two-dimensional supervision substructure of the historical rounds, and continuing to perform the steps until the two-dimensional supervision substructure of the current round is obtained; The two-dimensional supervision substructures corresponding to each round are integrated to obtain the two-dimensional supervision structure corresponding to the sample driving area image.

6. The method according to claim 1, characterized in that The three-dimensional detection model includes a three-dimensional supervision structure and a two-dimensional supervision structure; the three-dimensional supervision structure is used to supervise the accuracy of the sample target point cloud features extracted by the point cloud feature extraction structure; the two-dimensional supervision structure is used to supervise the accuracy of the sample pixel features extracted by the image feature extraction structure; the point cloud feature extraction structure is used to extract the sample point cloud features of the sample point cloud; the image feature extraction structure is used to extract the sample pixel features of the sample image pixels.

7. The method according to claim 6, characterized in that The three-dimensional detection model is trained according to the difference between the sample target point cloud feature and the point cloud feature label until a training stop condition is reached to obtain a trained three-dimensional detection model, including: Determining a first loss function corresponding to the three-dimensional supervision structure according to a difference between the sample target point cloud feature and the point cloud feature label; Determining a second loss function corresponding to the two-dimensional supervision structure according to the difference between the sample pixel feature and the pixel feature label; Combining the first loss function and the second loss function to obtain a combined loss function; The model parameters in the three-dimensional detection model are updated through the joint loss function and based on the back propagation algorithm until the training stop condition is reached, thereby obtaining a trained three-dimensional detection model.

8. A training device for a three-dimensional detection model, characterized in that: The device comprises: An image acquisition module, used to acquire a sample driving area image, a sample depth point cloud image, and a point cloud feature label corresponding to a sample point cloud in the sample depth point cloud image; A feature recognition module is used to determine the sample point cloud features corresponding to the sample point cloud in the sample depth point cloud image; perform random perturbation processing on the sample depth point cloud image, and determine the sample image pixels located in the sample driving area image corresponding to each sample point cloud in the sample depth point cloud image after random perturbation; The model training module is used to fuse the sample point cloud features corresponding to the current sample point cloud with the sample pixel features of the corresponding sample image pixels for each sample point cloud in the sample depth point cloud image to obtain the sample target point cloud features; according to the difference between the sample target point cloud features and the point cloud feature labels, the three-dimensional detection model is trained until the training stop condition is reached to obtain a trained three-dimensional detection model; the three-dimensional detection model is used to identify the target object in the driving area; The step of fusing the sample point cloud features corresponding to the current sample point cloud with the sample pixel features of the corresponding sample image pixels to obtain the sample target point cloud features includes: The sample pixel features of the sample image pixels corresponding to the current sample point cloud are convolved through the convolution layer in the three-dimensional detection model to obtain candidate sample pixel features; the sample point cloud features corresponding to the current sample point cloud are linearly processed through the multi-layer perceptron layer in the three-dimensional detection model to obtain candidate sample point cloud features; the candidate sample pixel features and the candidate sample point cloud features are normalized through the normalization layer in the three-dimensional detection model; the normalized candidate sample pixel features and the normalized candidate sample point cloud features are linearly combined to obtain sample target point cloud features.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images