Training method and device of point cloud perception model

By voxelizing and self-supervising training of vehicle-mounted radar point cloud data, combined with the semantic information of RGB images, the problem of insufficient semantic information extraction capability of point cloud perception model is solved, and more efficient information extraction and training effects are achieved.

CN118887656APending Publication Date: 2024-11-01NEUSOFT REACH AUTOMOBILE TECH (SHENYANG) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410917207.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

The prior art is difficult to effectively improve the semantic information extraction capability of point cloud perception models, resulting in insufficient information extraction in applications such as autonomous driving perception systems.

Method used

By acquiring point cloud data of the vehicle-mounted radar for voxelization, the self-supervised training model combines the semantic information of RGB images to perform random occlusion voxel feature input and RGB prediction value training, combining feature extraction network and feature map transformation module and 2D transposed convolutional layer to realize the training of point cloud-aware model.

Benefits of technology

The semantic information extraction capability of the point cloud perception model is improved, the need for labeled data is reduced, the training speed and sufficient level of the model is improved, and the semantic information extraction capability of the point cloud data is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887656A_ABST
    Figure CN118887656A_ABST
Patent Text Reader

Abstract

The invention discloses a training method of a point cloud perception model, which is applied to the field of deep learning, and comprises the following steps: carrying out voxelization processing on obtained point cloud data to obtain voxel features of a point cloud, each voxel feature comprising a three-dimensional coordinate corresponding to the voxel feature, then inputting the voxel features subjected to random shielding into a self-supervised training model, and carrying out self-supervised training on the self-supervised training model; the method comprises the following steps: obtaining an RGB predicted value of an occluded voxel, projecting a three-dimensional coordinate of each voxel feature to an RGB image of a vehicle-mounted camera, determining a pixel point corresponding to each voxel feature, determining an RGB color value corresponding to the pixel point as an RGB true value of the voxel, taking the RGB true value as a label of model training, and supervising the RGB predicted value. And training the point cloud sensing model. Rich semantic information in the RGB image is used as guidance, so that the semantic information in the point cloud data can be better extracted by the point cloud sensing model, and the semantic information extraction capability of the point cloud sensing model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of deep learning, and in particular to a training method and device for a point cloud perception model. Background Art

[0002] The point cloud perception model is a model used to process and analyze point cloud data acquired by devices such as lidar in three-dimensional space. By processing and analyzing these point cloud data, it can extract useful information and realize various applications such as target recognition, positioning, and three-dimensional reconstruction. The point cloud perception model is currently widely used in the field of autonomous driving perception systems.

[0003] In some cases, in order to improve the semantic information extraction capability of the point cloud perception model, the point cloud perception model needs to be trained.

[0004] Therefore, how to train the point cloud perception model to improve the semantic information extraction capability of the point cloud perception model is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The present application provides a training method for a point cloud perception model to improve the semantic information extraction capability of the point cloud perception model. The present application also provides a training device for a point cloud perception model.

[0006] In a first aspect, the present application provides a method for training a point cloud perception model, comprising:

[0007] Get the point cloud data of the vehicle-mounted radar;

[0008] Performing voxel processing on the point cloud data to obtain voxel features of the point cloud, each voxel feature including the three-dimensional coordinates of the corresponding voxel feature;

[0009] Input the voxel features after random occlusion into the self-supervised training model to obtain the RGB prediction value of the occluded voxel, wherein the self-supervised training model includes a feature extraction network that is the same as the backbone module of the point cloud perception model;

[0010] Projecting the three-dimensional coordinates of each voxel feature onto the RGB image of the vehicle-mounted camera, determining the pixel point corresponding to each voxel feature, and determining the RGB color value corresponding to the pixel point as the RGB true value of the voxel;

[0011] The RGB true value is used as a label for model training, and the RGB predicted value is supervised, so as to train the point cloud perception model.

[0012] Optionally, the using the RGB true value as a label for model training, using the RGB true value as a label for model training, supervising the RGB predicted value, and thus training the point cloud perception model includes:

[0013] Using the true RGB value as a label for model training, supervising the RGB prediction value, thereby training the feature extraction network;

[0014] The weight parameters trained by the feature extraction network are loaded into the point cloud perception model.

[0015] Optionally, the self-supervised training model includes a feature extraction network, a feature map transformation module, and a 2D transposed convolution layer; the step of inputting the voxel features after random occlusion into the self-supervised training model to obtain the RGB prediction value of the occluded voxel includes:

[0016] Extracting features from the voxel features using the feature extraction network to obtain a feature map;

[0017] Using the feature map transformation module to perform feature transformation on the feature map to convert the 3D features of the feature map into 2D features;

[0018] The feature map after feature transformation is input into the 2D transposed convolutional layer for decoding and segmentation to output the RGB prediction value of the occluded voxel.

[0019] Optionally, the step of inputting the feature map after feature transformation into a 2D transposed convolutional layer for decoding and segmentation includes:

[0020] The feature map after feature transformation is input into the 2D transposed convolution layer for decoding to obtain the decoded feature map;

[0021] The 0th dimension of the decoded feature map is segmented.

[0022] Optionally, the method further includes:

[0023] Determining a true value of a voxel occupancy of the point cloud data;

[0024] The voxel features after random occlusion are input into the self-supervised training model to obtain the predicted value of voxel occupancy;

[0025] The true value of the voxel occupancy is used as a label for model training, and the RGB prediction value is supervised, so as to train the point cloud perception model.

[0026] In a second aspect, the present application provides a training device for a point cloud perception model, comprising:

[0027] An acquisition unit, used for acquiring point cloud data of the vehicle-mounted radar;

[0028] A first obtaining unit is used to perform voxel processing on the point cloud data to obtain voxel features of the point cloud, each voxel feature including a three-dimensional coordinate of the corresponding voxel feature;

[0029] A second obtaining unit is used to input the voxel features after random occlusion into the self-supervised training model to obtain the RGB prediction value of the occluded voxel, wherein the self-supervised training model includes a feature extraction network that is the same as the backbone network module of the point cloud perception model;

[0030] A first determining unit is used to project the three-dimensional coordinates of each voxel feature onto the RGB image of the vehicle-mounted camera, determine the pixel point corresponding to each voxel feature, and determine the RGB color value corresponding to the pixel point as the RGB true value of the voxel;

[0031] The first training unit is used to use the RGB true value as a label for model training and supervise the RGB predicted value, thereby training the point cloud perception model.

[0032] Optionally, the first determining unit is specifically configured to:

[0033] The RGB true value is used as the label for model training.

[0034] Supervising the RGB prediction values ​​to train the feature extraction network;

[0035] The weight parameters trained by the feature extraction network are loaded into the point cloud perception model.

[0036] Optionally, the self-supervised training model includes a feature extraction network, a feature map transformation module and a 2D transposed convolution layer; the first acquisition unit is specifically used to:

[0037] Using the feature extraction network to extract features from the voxel features to obtain a feature map;

[0038] Using the feature map transformation module to perform feature transformation on the feature map to convert the 3D features of the feature map into 2D features;

[0039] The feature map after feature transformation is input into the 2D transposed convolutional layer for decoding and segmentation to output the RGB prediction value of the occluded voxel.

[0040] Optionally, the first obtaining unit is specifically configured to:

[0041] The feature map after feature transformation is input into the 2D transposed convolution layer for decoding to obtain the decoded feature map;

[0042] The 0th dimension of the decoded feature map is segmented.

[0043] Optionally, the device further comprises:

[0044] A second determining unit, used to determine a true value of a voxel occupancy condition of the point cloud data;

[0045] A third obtaining unit is used to input the voxel features after random occlusion into the self-supervised training model to obtain a predicted value of the voxel occupancy;

[0046] The second training unit is used to use the true value of the voxel occupancy as a label for model training, supervise the RGB prediction value, and thus train the point cloud perception model.

[0047] In a third aspect, an embodiment of the present application provides a device, comprising a memory and a processor, wherein the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes any method described in the first aspect.

[0048] In a fourth aspect, an embodiment of the present application provides a computer storage medium, in which a code is stored. When the code is executed, a device executing the code implements the method described in any one of the first aspects above.

[0049] Compared with the prior art, this application has the following beneficial effects:

[0050] In this application, the point cloud data of the vehicle-mounted radar is obtained, and the point cloud data is voxelized to obtain the voxel features of the point cloud. Each voxel feature includes the three-dimensional coordinates of the corresponding voxel feature, and then the voxel features after random occlusion are input into the self-supervised training model to obtain the RGB prediction value of the occluded voxel, wherein the self-supervised training model includes a feature extraction network that is the same as the backbone module of the point cloud perception model. Then, the three-dimensional coordinates of each voxel feature are projected onto the RGB image of the vehicle-mounted camera, and the pixel points corresponding to each voxel feature are determined. The RGB color value corresponding to the pixel point is determined as the RGB true value of the voxel. Finally, the RGB true value is used as the label for model training, and the RGB prediction value is supervised, thereby training the point cloud perception model. In this application, the point cloud perception model is trained with the rich semantic information in the RGB image as a guide, so that the point cloud perception model can better extract the semantic information in the point cloud data, thereby improving the semantic information extraction capability of the point cloud perception model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flowchart of a method for training a point cloud perception model provided in an embodiment of the present application;

[0053] Figure 2 A structural schematic diagram of a specific implementation of the point cloud perception model training device provided in this application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] It should be noted that the training method and device of a point cloud perception model provided in this application are used in the field of deep learning. It can also be used in any other field except the field of deep learning. The above is only an example and does not limit the application field of the method and device provided in this application.

[0056] The point cloud perception model is a model used to process and analyze point cloud data acquired by devices such as lidar in three-dimensional space. By processing and analyzing these point cloud data, it can extract useful information and realize various applications such as target recognition, positioning, and three-dimensional reconstruction. The point cloud perception model is currently widely used in the field of autonomous driving perception systems.

[0057] In some cases, in order to improve the semantic information extraction capability of the point cloud perception model, the point cloud perception model needs to be trained.

[0058] Therefore, how to train the point cloud perception model to improve the semantic information extraction capability of the point cloud perception model is a technical problem that technical personnel in this field urgently need to solve.

[0059] After research, the inventor proposed the technical solution of this application. In this application, the point cloud data of the vehicle-mounted radar is obtained, the point cloud data is voxelized, and the voxel features of the point cloud are obtained. Each voxel feature includes the three-dimensional coordinates of the corresponding voxel feature, and then the voxel features after random occlusion are input into the self-supervised training model to obtain the RGB prediction value of the occluded voxel, wherein the self-supervised training model includes the same feature extraction network as the backbone network backbone module of the point cloud perception model. Then the three-dimensional coordinates of each voxel feature are projected onto the RGB image of the vehicle-mounted camera, the pixel point corresponding to each voxel feature is determined, and the RGB color value corresponding to the pixel point is determined as the RGB true value of the voxel. Finally, the RGB true value is used as the label for model training, and the RGB prediction value is supervised, thereby training the point cloud perception model. In this application, the point cloud perception model is trained with the rich semantic information in the RGB image as a guide, so that the point cloud perception model can better extract the semantic information in the point cloud data, thereby improving the semantic information extraction capability of the point cloud perception model.

[0060] The method provided in the embodiment of the present application can be executed by a terminal device or a server. The terminal device can be, for example, a mobile phone, a tablet computer, a computer, or other devices.

[0061] In order to make the technical personnel in this field better understand the present application, the present application is further described in detail below in combination with the drawings and specific implementation methods. The method provided in the present application embodiment is described below by taking the first device as an example.

[0062] Figure 1 A flowchart of a method for training a point cloud perception model provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0063] S101: Obtaining point cloud data of the vehicle-mounted radar.

[0064] The first device obtains the point cloud data of the vehicle-mounted radar, wherein the vehicle-mounted radar may be a vehicle-mounted laser radar, and the point cloud data of the vehicle-mounted radar refers to the vehicle-mounted laser radar emitting laser signals to the surroundings and collecting the reflected signals. These reflected signals are converted into points in three-dimensional space, thereby forming point cloud data. Each point has its position information in three-dimensional space, such as x, y, and z coordinates, and some also include attributes such as reflection intensity and color. These data accurately represent the three-dimensional shape and characteristics of the surrounding environment.

[0065] In some possible implementations, the point cloud data collected by the vehicle-mounted radar can also be preprocessed to remove noise, extract features, or perform other forms of optimization. For example, based on a point cloud fitting algorithm or a feature extraction algorithm, the noise of the point cloud data can be removed.

[0066] S102: voxelizing the point cloud data to obtain voxel features of the point cloud.

[0067] The first device performs voxel processing on the obtained point cloud data to obtain voxel features of the point cloud, wherein each voxel feature may include four dimensions (X, Y, Z, I), X, Y, Z respectively correspond to the three-dimensional coordinates of the voxel feature, and I is the reflection intensity information, which is usually an important attribute of the point cloud data, and it represents the reflection intensity of each point when receiving the laser signal. It is directly related to the properties of the object surface and the intensity of the incident light.

[0068] S103: Input the voxel features after random occlusion into the self-supervised training model to obtain the RGB prediction value of the occluded voxel.

[0069] The first device randomly occludes the voxel features (voxels), and inputs the voxel features after random occlusion into the self-supervised training model to obtain the RGB prediction value of the occluded voxels. It should be noted that since direct training of the point cloud perception model is relatively complex and difficult to implement, the feature extraction network in the self-supervised model that is the same as the backbone network backbone module of the point cloud perception model can be trained, and the weight parameters of the point cloud perception model can be replaced with the weight parameters trained by the feature extraction network to complete the training of the point cloud perception model. In addition, the present application adopts self-supervised training, which requires relatively less labeled data than the training method of the prior art, thereby reducing the labeling cost of point cloud data.

[0070] Specifically, the self-supervised training model includes a feature extraction network, a feature map transformation module, and a 2D transposed convolution layer, wherein the feature extraction network is the same as the backbone network backbone module of the point cloud perception model.

[0071] The first device can input the voxel features after random occlusion into the feature extraction network, use the feature extraction network to extract the voxel features, and obtain a feature map (C, H, W, D), where C is the dimension of the feature map, and H, W, and D represent the length, width, and height of the feature map, respectively.

[0072] The feature map is then input into a feature map transformation module, which is used to perform feature transformation on the feature map, flattening the two dimensions D and C to obtain (CD, H, W) to convert the 3D features of the feature map into 2D features.

[0073] Finally, the feature map after feature transformation is input into the 2D transposed convolutional layer for decoding and segmentation to output the RGB prediction value of the occluded voxel.

[0074] Furthermore, the feature map after feature transformation is input into the 2D transposed convolution layer for decoding to obtain the decoded feature map (4Dv , H v , W v ), and then segment the 0th dimension of the decoded feature map to obtain (4, H v , W v , D v ) where D v , H v , W v Respectively represent the resolution of the point cloud after voxelization, 4 represents the predicted result of the (OCC, R, G, B) values ​​of all voxels in the point cloud, that is, R, G, B represent the RGB predicted values ​​of the occluded voxels, and OCC is the predicted value of the voxel occupancy. Here, the 3D feature decoding problem is converted to a 2D feature decoding problem, the height and feature dimensions are fused, and a lightweight feature decoder, that is, 2D transposed convolution, is used to increase the resolution layer by layer, and finally the RGB value of each voxel is predicted, thereby solving the problem that the feature decoding part in the prior art is too complicated (using Transformer or 3D transposed convolution), resulting in slow training speed, and the feature decoder is too powerful, which leads to insufficient training of the feature extraction network, thereby improving the training speed of the model and the degree of sufficient training of the model.

[0075] S104: Projecting the three-dimensional coordinates of each voxel feature onto the RGB image of the vehicle-mounted camera, determining the pixel point corresponding to each voxel feature, and determining the RGB color value corresponding to the pixel point as the RGB true value of the voxel.

[0076] The first device can project the three-dimensional coordinates (X, Y, Z) of each voxel feature onto the RGB image of the vehicle-mounted camera through the coordinate change matrix from the laser radar to the vehicle-mounted camera and the camera internal parameter matrix.

[0077] Specifically, first, the relative position and posture between the lidar and the camera need to be determined. This is usually done through external parameter calibration, using a specific calibration target that is captured by the lidar and camera simultaneously in multiple different postures. By comparing the observations of the two, the coordinate transformation matrix between them can be calculated, including the rotation matrix and the translation vector. Once the coordinate transformation matrix is ​​obtained, the lidar point cloud data can be transformed from the lidar coordinate system to the camera coordinate system. This usually involves applying rotation and translation operations to each point in the point cloud to obtain their new positions in the camera coordinate system.

[0078] The camera's intrinsic parameter matrix can be used to project the point cloud data in the camera coordinate system onto the RGB image plane. This involves the process of converting points in three-dimensional space into two-dimensional image coordinates so that the point cloud data can correspond to the camera image. Among them, the camera intrinsic parameter matrix describes the internal geometry and optical properties of the camera, such as focal length, principal point coordinates, and lens distortion coefficients. These parameters are crucial for projecting point cloud data onto the camera image plane.

[0079] Then, the pixel area corresponding to the voxel feature in the RGB image is determined according to the three-dimensional coordinates of each voxel feature, and then the pixel point corresponding to each voxel feature in the pixel area is determined, and finally the RGB color value corresponding to the pixel point is determined as the RGB true value of the voxel.

[0080] In some possible implementations, since the lidar and the on-board camera are installed at different locations on the vehicle and have different viewing angles, the voxel features outside the viewing angle of the on-board camera can be cleared to ensure that each remaining voxel feature has an RGB true value.

[0081] S105: Using the RGB true value as a label for model training, supervising the RGB prediction value, thereby training the point cloud perception model.

[0082] The first device can use the RGB true value as a label for model training to supervise the RGB prediction value, thereby training the point cloud perception model.

[0083] Specifically, the RGB true value can be used as a label for model training to supervise the RGB predicted value, thereby training the feature extraction network. Since the backbone network of the feature extraction network and the point cloud perception model are the same, the weight parameters trained by the feature extraction network can be loaded into the point cloud perception model, thereby completing the training of the point cloud perception model.

[0084] In some possible implementations, the first device can also determine the true value of the voxel occupancy of the point cloud data, input the voxel features after random occupancy into the self-supervised training model, and obtain the predicted value of the voxel occupancy. The specific steps are as described in S103 and are not described in detail here.

[0085] Finally, the true value of the voxel occupancy is used as the label for model training, and the RGB prediction value is supervised to train the point cloud perception model. The training principle is similar to the principle of using the RGB true value as the label for model training, supervising the RGB prediction value, and training the point cloud perception model as described above, and no unnecessary description is given here.

[0086] The present application can also load the trained feature extraction network weight parameters into the point cloud perception model, and then use a small amount of labeled data to train the point cloud perception model, so that the point cloud perception model can obtain the same performance as that of training with a large amount of labeled data. This can greatly reduce the use of labeled data and reduce the time, manpower and financial resources consumed in data labeling.

[0087] In this application, the point cloud data of the vehicle-mounted radar is obtained, and the point cloud data is voxelized to obtain the voxel features of the point cloud. Each voxel feature includes the three-dimensional coordinates of the corresponding voxel feature, and then the voxel features after random occlusion are input into the self-supervised training model to obtain the RGB prediction value of the occluded voxel, wherein the self-supervised training model includes a feature extraction network that is the same as the backbone module of the point cloud perception model. Then, the three-dimensional coordinates of each voxel feature are projected onto the RGB image of the vehicle-mounted camera, and the pixel points corresponding to each voxel feature are determined. The RGB color value corresponding to the pixel point is determined as the RGB true value of the voxel. Finally, the RGB true value is used as the label for model training, and the RGB prediction value is supervised, thereby training the point cloud perception model. In this application, the point cloud perception model is trained with the rich semantic information in the RGB image as a guide, so that the point cloud perception model can better extract the semantic information in the point cloud data, thereby improving the semantic information extraction capability of the point cloud perception model.

[0088] The above are some specific implementation methods of the training method of the point cloud perception model provided in the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided in the embodiment of the present application will be introduced from the perspective of functional modularization. The device and the training method of the point cloud perception model described above can be referenced to each other.

[0089] Figure 2 The structural block diagram of the training device of the point cloud perception model provided by the embodiment of the present invention is referred to as the specific implementation mode 2, and the reference Figure 2 The device may include:

[0090] An acquisition unit 210 is used to acquire point cloud data of a vehicle-mounted radar;

[0091] A first obtaining unit 220 is used to perform voxel processing on the point cloud data to obtain voxel features of the point cloud, each voxel feature including three-dimensional coordinate information of the corresponding voxel feature;

[0092] A second obtaining unit 230 is used to input the voxel features after random occlusion into the self-supervised training model to obtain the RGB prediction value of the occluded voxel, wherein the self-supervised training model includes a feature extraction network that is the same as the backbone network module of the point cloud perception model;

[0093] A first determining unit 240 is used to project the three-dimensional coordinates of each voxel feature onto the RGB image of the vehicle-mounted camera, determine the pixel point corresponding to each voxel feature, and determine the RGB color value corresponding to the pixel point as the RGB true value of the voxel;

[0094] The first training unit 250 is used to use the RGB true value as a label for model training and supervise the RGB prediction value, thereby training the point cloud perception model.

[0095] Optionally, the first determining unit is specifically configured to:

[0096] The RGB true value is used as the label for model training.

[0097] Supervising the RGB prediction values ​​to train the feature extraction network;

[0098] The weight parameters trained by the feature extraction network are loaded into the point cloud perception model.

[0099] Optionally, the self-supervised training model includes a feature extraction network, a feature map transformation module and a 2D transposed convolution layer; the first acquisition unit is specifically used to:

[0100] Using the feature extraction network to extract features from the voxel features to obtain a feature map;

[0101] Using the feature map transformation module to perform feature transformation on the feature map to convert the 3D features of the feature map into 2D features;

[0102] The feature map after feature transformation is input into the 2D transposed convolutional layer for decoding and segmentation to output the RGB prediction value of the occluded voxel.

[0103] Optionally, the first obtaining unit is specifically configured to:

[0104] The feature map after feature transformation is input into the 2D transposed convolution layer for decoding to obtain the decoded feature map;

[0105] The 0th dimension of the decoded feature map is segmented.

[0106] Optionally, the device further comprises:

[0107] A second determining unit, used to determine a true value of a voxel occupancy condition of the point cloud data;

[0108] A third obtaining unit is used to input the voxel features after random occlusion into the self-supervised training model to obtain a predicted value of the voxel occupancy;

[0109] The second training unit is used to use the true value of the voxel occupancy as a label for model training, supervise the RGB prediction value, and thus train the point cloud perception model.

[0110] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0111] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the method described in any embodiment of the present application.

[0112] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the method described in any embodiment of the present application.

[0113] The "first" and "second" in the names such as "first" and "second" (if any) mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0114] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0115] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0116] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for training a point cloud perception model, characterized in that: include: Obtain point cloud data from vehicle-mounted radar; Performing voxel processing on the point cloud data to obtain voxel features of the point cloud, each voxel feature including the three-dimensional coordinates of the corresponding voxel feature; Inputting the voxel features after random occlusion into the self-supervised training model to obtain the RGB prediction value of the occluded voxel, wherein the self-supervised training model includes a feature extraction network that is the same as the backbone network module of the point cloud perception model; Projecting the three-dimensional coordinates of each voxel feature onto the RGB image of the vehicle-mounted camera, determining the pixel point corresponding to each voxel feature, and determining the RGB color value corresponding to the pixel point as the RGB true value of the voxel; The RGB true value is used as a label for model training, and the RGB predicted value is supervised, so as to train the point cloud perception model.

2. The method according to claim 1, characterized in that The using the RGB true value as a label for model training and supervising the RGB predicted value, thereby training the point cloud perception model, includes: Using the true RGB value as a label for model training, supervising the RGB prediction value, thereby training the feature extraction network; The weight parameters trained by the feature extraction network are loaded into the point cloud perception model.

3. The method according to claim 1, characterized in that The self-supervised training model includes a feature extraction network, a feature map transformation module and a 2D transposed convolution layer; The step of inputting the voxel features after random occlusion into the self-supervised training model to obtain the RGB prediction value of the occluded voxel includes: Extracting features from the voxel features using the feature extraction network to obtain a feature map; Using the feature map transformation module to perform feature transformation on the feature map to convert the 3D features of the feature map into 2D features; The feature map after feature transformation is input into the 2D transposed convolutional layer for decoding and segmentation to output the RGB prediction value of the occluded voxel.

4. The method according to claim 3, characterized in that: The step of inputting the feature map after feature transformation into the 2D transposed convolution layer for decoding and segmentation includes: The feature map after feature transformation is input into the 2D transposed convolution layer for decoding to obtain the decoded feature map; The 0th dimension of the decoded feature map is segmented.

5. The method according to claim 1, characterized in that The method further comprises: Determining a true value of a voxel occupancy of the point cloud data; The voxel features after random occlusion are input into the self-supervised training model to obtain the predicted value of voxel occupancy; The true value of the voxel occupancy is used as the label for model training. The RGB prediction values ​​are supervised to train the point cloud perception model.

6. A training device for a point cloud perception model, characterized in that: include: An acquisition unit, used for acquiring point cloud data of the vehicle-mounted radar; A first obtaining unit is used to perform voxel processing on the point cloud data to obtain voxel features of the point cloud, each voxel feature including a three-dimensional coordinate of the corresponding voxel feature; A second obtaining unit is used to input the voxel features after random occlusion into the self-supervised training model to obtain the RGB prediction value of the occluded voxel, wherein the self-supervised training model includes a feature extraction network that is the same as the backbone network module of the point cloud perception model; A first determining unit is used to project the three-dimensional coordinates of each voxel feature onto the RGB image of the vehicle-mounted camera, determine the pixel point corresponding to each voxel feature, and determine the RGB color value corresponding to the pixel point as the RGB true value of the voxel; The first training unit is used to use the RGB true value as a label for model training and supervise the RGB predicted value, thereby training the point cloud perception model.

7. The device according to claim 6, characterized in that The first determining unit is specifically configured to: The RGB true value is used as the label for model training. Supervising the RGB prediction values ​​to train the feature extraction network; The weight parameters trained by the feature extraction network are loaded into the point cloud perception model parameters.

8. The device according to claim 6, characterized in that The self-supervised training model includes a feature extraction network, a feature map transformation module and a 2D transposed convolution layer; the first acquisition unit is specifically used to: Extracting features from the voxel features using the feature extraction network to obtain a feature map; Using the feature map transformation module to perform feature transformation on the feature map to convert the 3D features of the feature map into 2D features; The feature map after feature transformation is input into the 2D transposed convolutional layer for decoding and segmentation to output the RGB prediction value of the occluded voxel.

9. The device according to claim 8, characterized in that The first obtaining unit is specifically used for: The feature map after feature transformation is input into the 2D transposed convolution layer for decoding to obtain the decoded feature map; The 0th dimension of the decoded feature map is segmented.

10. The device according to claim 6, characterized in that The device also includes: A second determining unit, used to determine a true value of a voxel occupancy condition of the point cloud data; A third obtaining unit is used to input the voxel features after random occlusion into the self-supervised training model to obtain a predicted value of the voxel occupancy; The second training unit is used to use the true value of the voxel occupancy as a label for model training, supervise the RGB prediction value, and thus train the point cloud perception model.

Citation Information

Cited By

  • Model training method, object detection method, electronic device, medium, and product

    WO2026118802A1