Model training method and device, electronic equipment and storage medium

Abstract

The invention provides a model training method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring initial point cloud data; determining one or more target point cloud enhancement algorithms from a point cloud enhancement algorithm set, wherein the point cloud enhancement algorithm set comprises a plurality of candidate point cloud enhancement algorithms; enhancing the initial point cloud data based on a target point cloud enhancement algorithm to obtain enhanced point cloud data; and training the initial deep learning model based on the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model. Therefore, according to the scheme, the model is trained based on the enhanced point cloud data, the diversity of data in the model is improved, then the robustness and noise immunity of the model are improved, and the model training effect is promoted to be improved. The target deep learning model is applied to automatic driving, so that the obstacle recognition accuracy of the vehicle can be improved, the vehicle is helped to avoid obstacles, and the safety of automatic driving is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a model training method, device, electronic device and storage medium. Background Art

[0002] In autonomous driving, four-dimensional (4D) millimeter wave, as a new perception sensor, has more point clouds, more information and richer point cloud feature information.

[0003] Currently, most point cloud data enhancement methods are based on laser point cloud data enhancement. However, due to the richer dimensional information of 4D millimeter wave data, laser point cloud data enhancement methods are not applicable to 4D millimeter wave point clouds. This lack of data enhancement methods has hindered the development of 4D millimeter wave algorithms. Summary of the Invention

[0004] The present application aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the first purpose of this application is to propose a model training method to improve the diversity of data in the model, thereby improving the robustness and noise resistance of the model, reducing the overfitting of the model training, and promoting the improvement of model training effects and providing an important foundation for training better models.

[0006] The second purpose of this application is to provide a model training device.

[0007] The third objective of this application is to provide an electronic device.

[0008] The fourth object of this application is to provide a computer-readable storage medium.

[0009] A fifth object of this application is to provide a computer program product.

[0010] To achieve the above-mentioned purpose, the first aspect embodiment of the present application proposes a model training method, including: obtaining initial point cloud data; determining one or more target point cloud enhancement algorithms from a point cloud enhancement algorithm set, wherein the point cloud enhancement algorithm set includes multiple candidate point cloud enhancement algorithms; enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data; training an initial deep learning model based on the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model.

[0011] To achieve the above-mentioned purpose, the second aspect embodiment of the present application proposes a model training device, including: an acquisition module for acquiring initial point cloud data; a determination module for determining one or more target point cloud enhancement algorithms from a point cloud enhancement algorithm set, wherein the point cloud enhancement algorithm set includes multiple candidate point cloud enhancement algorithms; an enhancement module for enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data; a training module for training an initial deep learning model based on the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model.

[0012] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor; and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor can execute the model training method described in the first aspect embodiment above.

[0013] To achieve the above-mentioned purpose, the fourth embodiment of the present application proposes a computer-readable storage medium on which a computer program is stored, and the computer instructions are used to enable the computer to execute the model training method described in the above-mentioned embodiment.

[0014] To achieve the above-mentioned purpose, the fifth embodiment of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the model training method described in the first embodiment above.

[0015] The training method, device, electronic device and storage medium of the model provided in the present application obtain initial point cloud data and determine the target point cloud enhancement algorithm. The initial point cloud data is enhanced based on the target point cloud enhancement algorithm to obtain enhanced point cloud data, so as to improve the diversity and robustness of the point cloud data. Furthermore, the initial deep learning model is trained using the initial point cloud data and the enhanced point cloud data to obtain the target deep learning model. Training the model based on the enhanced point cloud data can improve the diversity of the data in the model, thereby improving the robustness and noise resistance of the model, reducing the overfitting of the model training, and promoting the improvement of the model training effect, providing an important foundation for training better models. Applying the target deep learning model in autonomous driving can improve the accuracy of the vehicle in identifying obstacles, help the vehicle avoid obstacles, and improve the safety of autonomous driving.

[0016] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0018] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of another model training method provided in an embodiment of the present application;

[0020] Figure 3 A flowchart of another model training method provided in an embodiment of the present application;

[0021] Figure 4 A flowchart illustrating a process for calculating enhanced point cloud data in a model training method provided in an embodiment of the present application;

[0022] Figure 5 A flowchart illustrating a process for calculating enhanced point cloud data in a model training method provided in an embodiment of the present application;

[0023] Figure 6 A flowchart illustrating a process for calculating enhanced point cloud data in another model training method provided in an embodiment of the present application;

[0024] Figure 7 A flowchart illustrating a process for calculating enhanced point cloud data in another model training method provided in an embodiment of the present application;

[0025] Figure 8 A flowchart illustrating a process for calculating enhanced point cloud data in another model training method provided in an embodiment of the present application;

[0026] Figure 9 A flowchart illustrating a process for calculating enhanced point cloud data in another model training method provided in an embodiment of the present application;

[0027] Figure 10 A schematic diagram of the process of model training based on point cloud data provided in an embodiment of the present application;

[0028] Figure 11 A schematic structural diagram of a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] Figure 1 is a flow chart of a model training method according to an exemplary embodiment. Figure 1 As shown, the training method of the model in the embodiment of the present application includes but is not limited to the following steps:

[0031] S101, obtaining initial point cloud data.

[0032] It should be noted that the execution subject of the model training method in the embodiment of the present application is an electronic device, which can be an intelligent platform, server, etc. with data information processing capabilities. The model training method in the embodiment of the present application can be executed by the model training device in the embodiment of the present application, and the model training device in the embodiment of the present application can be configured in any electronic device to execute the model training method in the embodiment of the present application. In the embodiment of the present application, the model training method can be explained by taking the autonomous driving of a vehicle as an example.

[0033] In some implementations, the initial point cloud data can be collected using the vehicle's own sensors or sensors independent of the vehicle. Alternatively, the initial point cloud data can be collected using one or more millimeter wave sensors of the vehicle. The embodiments of the present application are not limited to the millimeter wave sensors of the vehicle. For example, a four-dimensional (4D) millimeter wave radar can be used to collect the initial point cloud data.

[0034] S102: Determine one or more target point cloud enhancement algorithms from a point cloud enhancement algorithm set, where the point cloud enhancement algorithm set includes a plurality of candidate point cloud enhancement algorithms.

[0035] In some implementations, one or more point cloud enhancement algorithms may be randomly selected from the point cloud enhancement algorithm set as target point cloud enhancement algorithms based on the probabilities of multiple candidate point cloud enhancement algorithms in the point cloud enhancement algorithm set.

[0036] Optionally, multiple candidate point cloud enhancement algorithms include but are not limited to: 4D millimeter wave multidimensional random flipping algorithm, 4D millimeter wave multidimensional random rotation angle data enhancement algorithm, 4D millimeter wave multidimensional random copy-paste algorithm, 4D millimeter wave multidimensional point cloud random sampling algorithm, random mask data enhancement method, point cloud noise enhancement algorithm, 4D millimeter wave multidimensional point cloud random translation, proportional scaling, etc.

[0037] In some implementations, you can set the probabilities of multiple candidate point cloud enhancement algorithms and determine the target point cloud enhancement algorithm based on the probabilities. For example, if the probability of the 4D millimeter wave multi-dimensional random flip algorithm is 20%, then when performing 10 point cloud data enhancements, the 4D millimeter wave multi-dimensional random flip algorithm will be selected as the target point cloud enhancement algorithm twice.

[0038] Optionally, in order to avoid repeated invalid data enhancement superposition algorithms, a target point cloud enhancement algorithm can be determined from the point cloud enhancement algorithm set based on the mutual exclusion between multiple candidate point cloud enhancement algorithms and the probability of each point cloud enhancement algorithm.

[0039] S103: Enhance the initial point cloud data based on a target point cloud enhancement algorithm to obtain enhanced point cloud data.

[0040] In some implementations, the initial point cloud data is enhanced based on a target point cloud enhancement algorithm to obtain more accurate and complete enhanced point cloud data. Optionally, when a target point cloud enhancement algorithm is determined to enhance the initial point cloud data, the algorithm is directly used for enhancement to obtain enhanced point cloud data.

[0041] Optionally, when multiple target point cloud enhancement algorithms are determined to enhance the initial point cloud data, the initial point cloud data can be divided into multiple areas, different target point cloud enhancement algorithms are selected for different areas, the initial point cloud data are enhanced, and the enhanced point cloud data of multiple areas are fused to obtain enhanced point cloud data of the initial point cloud data.

[0042] In some implementations, the initial point cloud data may be enhanced by performing random flipping, random rotation, noise enhancement, point cloud addition, and the like on the initial point cloud data.

[0043] S104: Training the initial deep learning model based on the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model.

[0044] In some implementations, the initial point cloud data and the enhanced point cloud data can be used as the training set and validation set of the initial deep learning model, respectively, and the initial deep learning model can be trained based on the loss function until the training is completed to obtain the target deep learning model.

[0045] Optionally, the initial point cloud data and enhanced point cloud data can be input into the initial deep learning model, and the loss function can be calculated based on the output point cloud data and enhanced point cloud data, and the model can be corrected and returned to continue training until the training is completed to obtain the target deep learning model.

[0046] Optionally, the training end condition may be that the number of training times of the deep learning model reaches a set value. The training end condition may also be that the accuracy of the output data of the deep learning model reaches a set value.

[0047] Alternatively, the target deep learning model can be applied to technical fields such as autonomous driving, target recognition and tracking, 3D reconstruction and modeling. In the field of autonomous driving technology, the target deep learning model can perform target detection, obstacle segmentation, trajectory prediction, etc.

[0048] In the training method of the model provided in the embodiment of the present application, initial point cloud data is obtained and a target point cloud enhancement algorithm is determined. The initial point cloud data is enhanced based on the target point cloud enhancement algorithm to obtain enhanced point cloud data, so as to improve the diversity and robustness of the point cloud data. Furthermore, the initial deep learning model is trained using the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model. Training the model based on the enhanced point cloud data can improve the diversity of the data in the model, thereby improving the robustness and noise resistance of the model, reducing the overfitting of the model training, and promoting the improvement of the model training effect, providing an important foundation for training better models. Applying the target deep learning model in autonomous driving can improve the accuracy of the vehicle in identifying obstacles, help the vehicle avoid obstacles, and improve the safety of autonomous driving.

[0049] Figure 2 is a flow chart of a model training method according to an exemplary embodiment. Figure 2 As shown, the training method of the model in the embodiment of the present application includes but is not limited to the following steps:

[0050] S201, obtaining initial point cloud data.

[0051] In the embodiment of the present application, the implementation method of step S201 can be implemented by any method in the various embodiments of the present application, which is not limited here and will not be repeated.

[0052] S202, for the i-th round of training, determine one or more candidate point cloud enhancement algorithms from the point cloud enhancement algorithm set based on random probability as the target point cloud enhancement algorithm used in the i-th round of training, where i is a positive integer greater than or equal to 1.

[0053] In some implementations, the initial deep learning model can be trained multiple times to improve model training results. For the i-th round of training, a target point cloud enhancement algorithm for the i-th round of training can be determined from a set of point cloud enhancement algorithms based on the random probabilities corresponding to multiple candidate point cloud enhancement algorithms. One or more target point cloud enhancement algorithms can be used in one training round.

[0054] In some implementations, when determining any candidate point cloud enhancement algorithm as the target point cloud enhancement algorithm, the determination can be based on random probability. When determining multiple candidate point cloud enhancement algorithms as the target point cloud enhancement algorithm, in order to avoid repeated and invalid data enhancement superposition algorithms, multiple target point cloud enhancement algorithms can be determined based on the mutually exclusive relationship between the algorithms.

[0055] Optionally, a target point cloud enhancement algorithm is determined from the multiple candidate point cloud enhancement algorithms based on the mutually exclusive relationship by determining the mutually exclusive relationship between the multiple candidate point cloud enhancement algorithms.

[0056] For example, the 4D millimeter-wave multidimensional random copy-paste algorithm and the random mask data enhancement method are mutually exclusive. When the random mask data enhancement method is selected, the random probability of selecting the 4D millimeter-wave multidimensional random copy-paste algorithm is reduced to ensure the effectiveness of the data enhancement algorithm.

[0057] S203: Determine the initial point cloud data i used for the i-th round of training.

[0058] S204 , based on the target point cloud enhancement algorithm used in the i-th round of training, performing enhancement processing on the point cloud data i used in the i-th round of training to obtain the target point cloud data i used in the i-th round of training.

[0059] In some implementations, the point cloud data output by the deep learning model in the i-1th round can be used as the initial point cloud data i used in the i-th round of training, and the initial point cloud data i can be enhanced based on the target point cloud enhancement algorithm determined for use in the i-th round to obtain the target point cloud data i.

[0060] The initial point cloud data i and the target point cloud data i are used to input into the deep learning model for training in the i-th round of training. The target point cloud data i can be used as the enhanced point cloud data i used in the i-th round and input into the deep learning model for training together with the initial point cloud data i.

[0061] S205: Train the deep learning model based on the initial point cloud data i and the enhanced point cloud data i to obtain a target deep learning model.

[0062] In the embodiment of the present application, the implementation method of step S205 can be implemented by any method in the various embodiments of the present application, which is not limited here and will not be repeated.

[0063] In the training method of the model provided in the embodiment of the present application, by obtaining the initial point cloud data and determining the target point cloud enhancement algorithm for the current i-th round of training when multiple rounds of model training are performed, the initial point cloud data i of the i-th round can be enhanced based on the target point cloud enhancement algorithm to obtain enhanced point cloud data i, so as to improve the diversity and robustness of the point cloud data. Furthermore, the target deep learning model can be obtained by training the deep learning model using the initial point cloud data i and the enhanced point cloud data i. Training the model based on the enhanced point cloud data can improve the diversity of the data in the model, thereby improving the robustness and noise resistance of the model, reducing the overfitting of the model training, and promoting the improvement of the model training effect, providing an important foundation for training better models. Applying the target deep learning model in autonomous driving can improve the accuracy of the vehicle in identifying obstacles, help the vehicle avoid obstacles, and improve the safety of autonomous driving.

[0064] Figure 3 is a flow chart of a model training method according to an exemplary embodiment. Figure 3 As shown, the training method of the model in the embodiment of the present application includes but is not limited to the following steps:

[0065] S301, obtaining initial point cloud data.

[0066] In the embodiment of the present application, the implementation method of step S301 can be implemented by any method in the various embodiments of the present application, which is not limited here and will not be repeated.

[0067] S302: Determine a plurality of perception areas included in the initial point cloud data.

[0068] In some implementations, the initial point cloud data can be divided into multiple perception areas, and the target point cloud enhancement algorithms corresponding to different perception areas can be determined. The target point cloud enhancement algorithms are then used to enhance the point cloud data of different perception areas to obtain enhanced point cloud data.

[0069] Optionally, the point cloud data includes geometric positions, and the boundary and range of the perception area can be determined based on the initial point cloud data, and then the multiple perception areas included in the initial point cloud data can be determined.

[0070] S303 , for the perception area j, determine a target point cloud enhancement algorithm for the perception area j from a set of point cloud enhancement algorithms, where j is a positive integer, 1≤j≤N, and N is the number of perception areas.

[0071] In some implementations, for each of the N perception regions, a corresponding target point cloud enhancement algorithm can be determined and used to obtain enhanced point cloud data. Optionally, for perception region j, a point cloud enhancement algorithm is determined from a set of point cloud enhancement algorithms based on the random probability and mutual exclusion relationship of multiple candidate point cloud enhancement algorithms as the target point cloud enhancement algorithm for perception region j.

[0072] S304 , based on the target point cloud enhancement algorithm of the perception area j, the initial point cloud data in the perception area j is enhanced to obtain enhanced point cloud data of the perception area j.

[0073] In some implementations, the initial point cloud data within perception region j is enhanced using a target point cloud enhancement algorithm to generate enhanced point cloud data for perception region j. Furthermore, the enhanced point cloud data for N perception regions are fused to generate enhanced point cloud data for the initial point cloud data.

[0074] S305: Training the initial deep learning model based on the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model.

[0075] In the embodiment of the present application, the implementation method of step S305 can be implemented by any method in the various embodiments of the present application, which is not limited here and will not be repeated.

[0076] In the training method of the model provided in the embodiment of the present application, initial point cloud data is obtained and the initial point cloud data is divided into regions to obtain multiple perception areas. Then, the target point cloud enhancement algorithm for different perception areas can be determined. For perception area j, the initial point cloud data is enhanced based on the target point cloud enhancement algorithm to obtain enhanced point cloud data of perception area j, so as to improve the diversity and robustness of the point cloud data. Further, the initial deep learning model is trained using the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model. Training the model based on the enhanced point cloud data can improve the diversity of data in the model, thereby improving the robustness and noise resistance of the model, reducing the overfitting of the model training, and promoting the improvement of the model training effect, providing an important foundation for training better models. Applying the target deep learning model in autonomous driving can improve the accuracy of the vehicle in identifying obstacles, help the vehicle avoid obstacles, and improve the safety of autonomous driving.

[0077] Based on the above embodiments, the embodiments of the present application can explain the 4D millimeter wave multi-dimensional random flipping algorithm, such as Figure 4 As shown, the process of calculating enhanced point cloud data using the algorithm includes but is not limited to the following steps:

[0078] S401, obtaining the vehicle's driving status information and the Doppler velocity of the millimeter-wave radar.

[0079] In some implementations, vehicle driving state information can be obtained based on the vehicle's onboard sensors. The vehicle driving state information includes, but is not limited to, vehicle speed and vehicle attitude information, such as pitch angle, heading angle, and roll angle. Alternatively, the vehicle driving state information can be obtained based on the vehicle's speed sensor, steering sensor, and the like.

[0080] In some implementations, the Doppler velocity of a millimeter-wave radar can be calculated based on the Doppler effect. The Doppler effect refers to the frequency shift caused by an object moving toward a radar transmitter or receiver. The Doppler velocity can be extracted from the initial point cloud data collected by the millimeter-wave radar.

[0081] S402 : Based on the driving state information and the Doppler velocity, the initial point cloud data is randomly flipped to obtain enhanced point cloud data.

[0082] In some implementations, enhanced point cloud data can be obtained by determining an axis for random flipping and randomly flipping the initial point cloud data based on driving state information and Doppler velocity. Alternatively, the initial point cloud data can be randomly flipped based on a random flipping algorithm.

[0083] Alternatively, a random flip algorithm uses the vertical plane of the millimeter-wave radar as the axis, driving state information and Doppler velocity as input, and randomly flips the initial point cloud data to output enhanced point cloud data. The enhanced point cloud data simultaneously enhances Doppler velocity, radar cross-section (RCS), and signal-to-interference plus noise ratio (SNR).

[0084] Among them, the Doppler velocity needs to be recalculated based on the position and velocity information after the flip, and the RCS and SNR can be considered to be unchanged with the flip.

[0085] Optionally, the Doppler velocity of the enhanced point cloud data should satisfy the Doppler characteristics while its noise should also satisfy the noise distribution of the Doppler velocity of the millimeter wave radar.

[0086] Optionally, random flipping can enhance the initial point cloud data by flipping the coordinate axes of the initial point cloud data. For example, it can be flipped along the X axis, along the Y axis, or along the Z axis. Random flipping can change the spatial position information of the initial point cloud data without changing the shape of the initial point cloud data. For example, object A in a frame of point cloud remains object A after being flipped.

[0087] In the implementation of this application, the initial point cloud data is randomly flipped to increase the diversity and richness of the point cloud data, thereby obtaining enhanced point cloud data to improve the generalization ability of the model.

[0088] Based on the above embodiments, the embodiments of the present application can explain the 4D millimeter wave multi-dimensional random rotation angle data enhancement algorithm, such as Figure 5 As shown, the process of calculating enhanced point cloud data using the algorithm includes but is not limited to the following steps:

[0089] S501, obtaining vehicle driving status information and signal feature information of initial point cloud data.

[0090] In the embodiment of the present application, the method of obtaining the driving status information of the vehicle in step S501 can be implemented by any of the methods in the various embodiments of the present application. This is not limited here and will not be repeated.

[0091] In some implementations, feature extraction can be performed on the initial point cloud data to obtain signal feature information of the initial point cloud data, wherein the signal feature information of the initial point cloud data includes but is not limited to: the position, Doppler velocity, RCS, SNR, etc. of the initial point cloud data.

[0092] S502 : Based on the driving state information and the initial signal feature information, the initial point cloud data is randomly rotated to obtain enhanced point cloud data.

[0093] In some implementations, a point cloud rotation model can be established based on a random rotation algorithm, and based on the model, driving state information, and initial signal feature information, the initial point cloud data can be randomly rotated to obtain enhanced point cloud data. The enhanced point cloud data satisfies Doppler and radar characteristics.

[0094] Optionally, a random rotation angle range may be predetermined, and the initial point cloud data may be randomly rotated according to the rotation angle range. For example, the initial point cloud data may be rotated along a certain coordinate axis by a set angle, or along to obtain enhanced point cloud data.

[0095] Optionally, a rotation matrix may be constructed based on the driving state information, and the signal feature information of the initial point cloud data may be rotated based on the rotation matrix to obtain enhanced point cloud data.

[0096] It is understandable that random rotation can change the spatial position information of the initial point cloud data, but will not change the shape of the initial point cloud data. For example, object B in a frame of point cloud remains object B after rotation.

[0097] In an embodiment of the present application, by randomly rotating the initial point cloud data, the diversity of the point cloud data can be increased, thereby obtaining enhanced point cloud data with high robustness to improve the generalization ability of the model.

[0098] Based on the above embodiments, the embodiments of the present application can explain the 4D millimeter wave multi-dimensional random copy-paste algorithm, such as Figure 6 As shown, the process of calculating enhanced point cloud data using the algorithm includes but is not limited to the following steps:

[0099] S601: Determine a filling area in the initial point cloud data.

[0100] In some implementations, by segmenting the initial point cloud data, dividing the initial point cloud data into multiple regions, and performing target detection on the initial point cloud data, the regions where targets exist in the initial point cloud data can be determined, and the regions where no targets exist can be used as fill regions.

[0101] S602: Acquire point cloud data of at least one existing target from a point cloud database.

[0102] In some implementations, the point cloud data of an existing object may be obtained by querying a point cloud database. Alternatively, the point cloud database may be queried based on features of the initial point cloud data, and point cloud data of at least one existing object may be obtained from the point cloud database.

[0103] S603: Add the existing point cloud data of the target to the filling area to obtain enhanced point cloud data.

[0104] In some implementations, enhanced point cloud data can be obtained by deleting the point cloud data in the filled area and adding the point cloud data of the existing target to the filled area, wherein the enhanced point cloud data satisfies the Doppler characteristic and the radar characteristic.

[0105] Optionally, the point cloud data of the existing target may be subjected to a coordinate system transformation to transform it into the coordinate system of the initial point cloud data, thereby adding the point cloud data of the existing target to the filling area.

[0106] In some implementations, using the point cloud enhancement algorithm of the embodiment of the present application, it is possible to randomly place the point cloud data of existing targets in different scenarios into new point cloud data frames, thereby achieving point cloud data enhancement of random data.

[0107] In the embodiment of the present application, by performing data filling on the initial point cloud data, the diversity and richness of the point cloud data can be increased, thereby obtaining enhanced point cloud data to improve the generalization ability of the model.

[0108] Based on the above embodiments, the embodiments of the present application can explain the 4D millimeter wave multi-dimensional point cloud random sampling algorithm, such as Figure 7 As shown, the process of calculating enhanced point cloud data using the algorithm includes but is not limited to the following steps:

[0109] S701: Determine a weight value of each point cloud based on signal feature information of each point cloud in the initial point cloud data.

[0110] In some implementations, the signal characteristic information of a point cloud may represent the signal strength of the point cloud. A greater signal strength indicates greater stability and reliability, and thus a greater corresponding weight. Alternatively, the weight of each point cloud may be determined by obtaining the signal characteristic information of each point cloud in the initial point cloud data and performing a weighted calculation on each point cloud based on the signal characteristic information.

[0111] Optionally, the signal characteristic information of the point cloud includes but is not limited to: Doppler velocity, RCS, SNR, etc. of the point cloud.

[0112] S702: Perform random sampling of point clouds according to the weight value of each point cloud to obtain enhanced point cloud data.

[0113] In some implementations, random sampling of point clouds can be performed based on the weight of each point cloud. A larger weight indicates a higher probability of sampling. This random sampling of point clouds can generate enhanced point cloud data. The enhanced point cloud data conforms to the characteristic distribution of 4D millimeter-wave point clouds.

[0114] In an embodiment of the present application, by determining the weight value of each point cloud in the initial point cloud data and performing random sampling of the point cloud based on the weight value, the size of the data can be reduced, the complexity of the calculation can be reduced, and enhanced point cloud data can be obtained to improve the generalization ability of the model.

[0115] Based on the above embodiments, the embodiments of the present application can explain the random mask data enhancement method, such as Figure 8 As shown, the process of calculating enhanced point cloud data using the algorithm includes but is not limited to the following steps:

[0116] S801: Randomly mask the initial point cloud data, and delete the point cloud data in the masked area.

[0117] In some implementations, a random mask may be generated based on a mask function, and a masking operation may be performed on the initial point cloud data to mask the point cloud data within the masked area, and then the point cloud data within the masked area may be deleted.

[0118] Optionally, different methods may be used to delete the point cloud data of the occluded area according to the definition of the occluded area, such as a rectangular area, a circular area, etc., so that no point cloud data exists in the occluded area.

[0119] S802: Randomly add point cloud data to the occluded area to obtain enhanced point cloud data.

[0120] In some implementations, a model can be pre-built based on the initial point cloud data to obtain a scene model of the point cloud data. Based on the scene model, random point clouds can be added to the occluded areas to obtain enhanced point cloud data. The randomly added point clouds should meet the Doppler and radar characteristics of 4D millimeter waves.

[0121] Optionally, the number of randomly increased point clouds may be generated, and point cloud data may be generated based on the number of point clouds, thereby achieving random point cloud addition in the occluded area.

[0122] In an embodiment of the present application, the initial point cloud data is masked and the data in the masked area is deleted, and then random point clouds are added to the masked area to increase the diversity and richness of the point cloud data, thereby obtaining enhanced point cloud data to improve the generalization ability of the model.

[0123] Based on the above embodiments, the present application embodiment can explain the point cloud noise enhancement algorithm, such as Figure 9 As shown, the process of calculating enhanced point cloud data using the algorithm includes but is not limited to the following steps:

[0124] S901: Perform noise statistics on the initial point cloud data to obtain noise characteristics of the initial point cloud data.

[0125] In some implementations, statistical analysis of the initial point cloud data can be performed, combined with experimental data from real-world scenarios, to determine the noise characteristics of the initial point cloud data. Alternatively, statistical analysis of the initial point cloud data can be used to determine the noise characteristics of the point cloud data. For example, this can include calculating the local density, curvature, and distance distribution of the point cloud.

[0126] Furthermore, the statistical features are compared with the real features, and by comparing and analyzing their differences, the noise characteristics of the initial point cloud data can be determined.

[0127] S902: Perform noise enhancement on the initial point cloud data based on the noise characteristics to obtain enhanced point cloud data.

[0128] In some implementations, the initial point cloud data can be enhanced using Gaussian noise based on noise characteristics. This is achieved by adding random noise that follows a Gaussian distribution to the initial point cloud data and adjusting the random noise based on the noise characteristics, ultimately generating enhanced point cloud data. The added noise should conform to actual statistical noise characteristics and distribution.

[0129] In the implementation of this application, by obtaining the noise characteristics of the initial point cloud data and performing random noise enhancement based on the noise characteristics, the point cloud data is made more consistent with the actual situation and the robustness of the point cloud data is improved. In turn, enhanced point cloud data can be obtained to improve the generalization ability of the model.

[0130] like Figure 10 The figure shows a flow chart of model training based on point cloud data. Initial point cloud data is obtained and one or more target point cloud enhancement algorithms are determined from a set of point cloud enhancement algorithms. The target point cloud enhancement algorithms can be determined based on random probability. When determining multiple target point cloud enhancement algorithms, the mutually exclusive relationships between the point cloud enhancement algorithms must be considered.

[0131] Figure 10 The point cloud enhancement algorithm set includes: 1.4D millimeter wave multidimensional random flip algorithm, 2.4D millimeter wave multidimensional random rotation angle data enhancement algorithm, 3.4D millimeter wave multidimensional random copy-paste algorithm, 4.4D millimeter wave multidimensional point cloud random sampling algorithm, 5.4D millimeter wave random mask data enhancement method, 6.4D millimeter wave point cloud noise enhancement algorithm, 7.4D millimeter wave point cloud random data enhancement combination strategy.

[0132] Among them, the enhancement algorithms of Enhancement Algorithm 2, Enhancement Algorithm 3, and Enhancement Algorithm 5 are mutually exclusive. When selecting the target enhancement algorithm for the same frame of initial point cloud data, Enhancement Algorithm 2, Enhancement Algorithm 3, and Enhancement Algorithm 5 cannot be applied to the same frame of initial point cloud data. In this case, one of Enhancement Algorithm 2, Enhancement Algorithm 3, and Enhancement Algorithm 5 can be used for enhancement. The initial point cloud data is enhanced using the target point cloud enhancement algorithm to obtain enhanced point cloud data. The initial deep learning model is then trained multiple times based on the initial and enhanced point cloud data to ultimately obtain the target deep learning model.

[0133] In order to implement the above embodiment, the present application also proposes a model training device.

[0134] Figure 11 A schematic structural diagram of a model training device provided in an embodiment of the present application.

[0135] like Figure 11 As shown, the training device 1100 of the model includes:

[0136] An acquisition module 1101 is used to acquire initial point cloud data;

[0137] A determination module 1102 is configured to determine one or more target point cloud enhancement algorithms from a point cloud enhancement algorithm set, wherein the point cloud enhancement algorithm set includes a plurality of candidate point cloud enhancement algorithms;

[0138] An enhancement module 1103 is configured to enhance the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data;

[0139] The training module 1104 is used to train the initial deep learning model based on the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model.

[0140] In a possible implementation of an embodiment of the present application, the determination module 1102 is further used to: for the i-th round of training, determine one or more of the candidate point cloud enhancement algorithms from the point cloud enhancement algorithm set based on random probability as the target point cloud enhancement algorithm used in the i-th round of training, where i is a positive integer greater than or equal to 1.

[0141] In a possible implementation of an embodiment of the present application, the enhancement module 1103 is also used to: determine the initial point cloud data i used in the i-th round of training; based on the target point cloud enhancement algorithm used in the i-th round of training, enhance the point cloud data i used in the i-th round of training to obtain the target point cloud data i used in the i-th round of training.

[0142] In a possible implementation of an embodiment of the present application, the determination module 1102 is further used to: determine the mutually exclusive relationship between the multiple candidate point cloud enhancement algorithms; and determine the target point cloud enhancement algorithm from the multiple candidate point cloud enhancement algorithms based on the mutually exclusive relationship.

[0143] In a possible implementation of an embodiment of the present application, the determination module 1102 is further used to: determine multiple perception areas included in the initial point cloud data; for perception area j, determine the target point cloud enhancement algorithm for the perception area j from the point cloud enhancement algorithm set, where j is a positive integer, 1≤j≤N, and N is the number of perception areas; based on the target point cloud enhancement algorithm for the perception area j, enhance the initial point cloud data within the perception area j to obtain enhanced point cloud data of the perception area j.

[0144] In a possible implementation of an embodiment of the present application, the enhancement module 1103 is further used to: obtain the vehicle's driving status information and the Doppler speed of the millimeter-wave radar; and randomly flip the initial point cloud data based on the driving status information and the Doppler speed to obtain the enhanced point cloud data.

[0145] In a possible implementation of an embodiment of the present application, the enhancement module 1103 is also used to: obtain the vehicle's driving status information and the signal characteristic information of the initial point cloud data; and based on the driving status information and the initial signal characteristic information, randomly rotate the initial point cloud data to obtain the enhanced point cloud data.

[0146] In a possible implementation of an embodiment of the present application, the enhancement module 1103 is also used to: determine the filling area in the initial point cloud data; obtain point cloud data of at least one existing target from the point cloud database; and add the point cloud data of the existing target to the filling area to obtain the enhanced point cloud data.

[0147] In a possible implementation of an embodiment of the present application, the enhancement module 1103 is also used to: determine the weight value of each point cloud based on the signal feature information of each point cloud in the initial point cloud data; and perform random sampling of the point cloud according to the weight value of each point cloud to obtain the enhanced point cloud data.

[0148] In a possible implementation of an embodiment of the present application, the enhancement module 1103 is also used to: randomly mask the initial point cloud data and delete the point cloud data in the occluded area blocked by the mask; and randomly add point clouds to the occluded area to obtain the enhanced point cloud data.

[0149] In a possible implementation of an embodiment of the present application, the enhancement module 1103 is further used to: perform noise statistics on the initial point cloud data to obtain the noise characteristics of the initial point cloud data; and perform noise enhancement on the initial point cloud data based on the noise characteristics to obtain the enhanced point cloud data.

[0150] In the training device of the model provided in the embodiment of the present application, initial point cloud data is obtained and a target point cloud enhancement algorithm is determined. The initial point cloud data is enhanced based on the target point cloud enhancement algorithm to obtain enhanced point cloud data, thereby improving the diversity and robustness of the point cloud data. Furthermore, the initial deep learning model is trained using the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model. Training the model based on the enhanced point cloud data can improve the diversity of the data in the model, thereby improving the robustness and noise resistance of the model, reducing the overfitting of the model training, and promoting the improvement of the model training effect, providing an important foundation for training better models.

[0151] It should be noted that the above explanation of the embodiment of the model training method is also applicable to the model training device of this embodiment, and will not be repeated here.

[0152] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0153] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0154] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0155] The collection, storage, use, processing, transmission, provision and application of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0156] It is important to note that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold beyond these legitimate uses. Furthermore, such collection / sharing should be conducted only after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes the relevant user information before using the feature. Furthermore, any necessary steps must be taken to safeguard and secure access to such personal information and ensure that others with access to personal information comply with its privacy policy and procedures.

[0157] This application contemplates providing implementation options for users to selectively block the use or access of personal information data. Specifically, this application contemplates providing hardware and / or software to prevent or block access to such personal information data. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Furthermore, where applicable, such personal information can be de-identified to protect user privacy.

[0158] In the descriptions of the foregoing embodiments, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are mutually inconsistent.

[0159] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0160] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0161] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0162] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0163] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0164] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0165] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Get initial point cloud data; Determining one or more target point cloud enhancement algorithms from a point cloud enhancement algorithm set, wherein the point cloud enhancement algorithm set includes a plurality of candidate point cloud enhancement algorithms; enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data; An initial deep learning model is trained based on the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model.

2. The method according to claim 1, characterized in that The determining one or more target point cloud enhancement algorithms from the point cloud enhancement algorithm set includes: For the i-th round of training, one or more candidate point cloud enhancement algorithms are determined from the point cloud enhancement algorithm set based on random probability as the target point cloud enhancement algorithm used in the i-th round of training, where i is a positive integer greater than or equal to 1.

3. The method according to claim 2, characterized in that The step of enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data includes: Determine the initial point cloud data i used in the i-th round of training; Based on the target point cloud enhancement algorithm used in the i-th round of training, the point cloud data i used in the i-th round of training is enhanced to obtain the target point cloud data i used in the i-th round of training.

4. The method according to claim 3, characterized in that The step of determining one or more candidate point cloud enhancement algorithms from the point cloud enhancement algorithm set based on random probability as target point cloud enhancement algorithms used in the i-th round of training includes: Determining a mutually exclusive relationship between the plurality of candidate point cloud enhancement algorithms; Based on the mutually exclusive relationship, the target point cloud enhancement algorithm is determined from the multiple candidate point cloud enhancement algorithms.

5. The method according to any one of claims 1 to 4, characterized in that The determining one or more target point cloud enhancement algorithms from the point cloud enhancement algorithm set includes: Determining a plurality of perception areas included in the initial point cloud data; For perception area j, determine a target point cloud enhancement algorithm for the perception area j from the set of point cloud enhancement algorithms, where j is a positive integer, 1≤j≤N, and N is the number of perception areas; Based on the target point cloud enhancement algorithm of the perception area j, the initial point cloud data in the perception area j is enhanced to obtain enhanced point cloud data of the perception area j.

6. The method according to any one of claims 1 to 3, characterized in that The step of enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data includes: Obtain vehicle driving status information and Doppler velocity of millimeter-wave radar; Based on the driving state information and the Doppler velocity, the initial point cloud data is randomly flipped to obtain the enhanced point cloud data.

7. The method according to any one of claims 1 to 3, characterized in that The step of enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data includes: Acquiring driving state information of the vehicle and signal characteristic information of the initial point cloud data; Based on the driving state information and the initial signal feature information, the initial point cloud data is randomly rotated to obtain the enhanced point cloud data.

8. The method according to any one of claims 1 to 3, characterized in that The step of enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data includes: Determining a fill area in the initial point cloud data; Acquire point cloud data of at least one existing target from a point cloud database; The point cloud data of the existing target is added to the filling area to obtain the enhanced point cloud data.

9. The method according to any one of claims 1 to 3, characterized in that The step of enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data includes: Determining a weight value of each point cloud based on signal feature information of each point cloud in the initial point cloud data; The point cloud is randomly sampled according to the weight value of each point cloud to obtain the enhanced point cloud data.

10. The method according to any one of claims 1 to 3, characterized in that The step of enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data includes: Performing random masking on the initial point cloud data, and deleting point cloud data within an occlusion area covered by the mask; Random point cloud addition is performed on the occluded area to obtain the enhanced point cloud data.

11. The method according to any one of claims 1 to 3, characterized in that The step of enhancing the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data includes: Performing noise statistics on the initial point cloud data to obtain noise characteristics of the initial point cloud data; Noise enhancement is performed on the initial point cloud data based on the noise characteristics to obtain the enhanced point cloud data.

12. A model training device, characterized in that: The device comprises: Acquisition module, used to obtain initial point cloud data; a determination module, configured to determine one or more target point cloud enhancement algorithms from a point cloud enhancement algorithm set, wherein the point cloud enhancement algorithm set includes a plurality of candidate point cloud enhancement algorithms; An enhancement module, configured to enhance the initial point cloud data based on the target point cloud enhancement algorithm to obtain enhanced point cloud data; A training module is used to train an initial deep learning model based on the initial point cloud data and the enhanced point cloud data to obtain a target deep learning model.

13. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 11 when executed by a processor.

15. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 11 when being executed by a processor.

Images