Training method and training device of perception model, and electronic device

Abstract

The application discloses a training method and device of a perception model and an electronic device, and the training method comprises the following steps: acquiring actual point cloud data continuously collected by a laser radar; performing target modeling according to the actual point cloud data to obtain a target model; calibrating and measuring line bundle information of the laser radar to obtain the distribution of laser line bundles; obtaining simulation point cloud data according to reflected laser points of the distribution of the laser line bundles on the target model; obtaining fusion training data according to the actual point cloud data and the simulation point cloud data, and training the perception model by using the fusion training data. The above scheme can take into account the generation efficiency and diversity of the training data for training the perception model, and improves the training effect of the perception model.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of point cloud data processing technology, and in particular to training methods, training devices and electronic devices for perception models. Background Technology

[0002] LiDAR is a widely used sensor in the field of autonomous driving, providing accurate distance measurement and 3D geometric information. However, the sparsity of LiDAR point clouds poses challenges to target detection. To ensure the performance of detection algorithms, especially for long-range, occluded, and special target scene detection tasks, a large amount of training and testing data is required. To meet the demand for large amounts of training data for LiDAR perception models, data augmentation is typically performed.

[0003] Currently, there are two commonly used data augmentation methods for laser point clouds:

[0004] One approach is to use real point cloud data for data augmentation, achieved through sampling, rotation, mirroring, partial deletion, copying and pasting, etc. However, on the one hand, since the distribution of point clouds changes with distance and direction, using actual point cloud data is greatly limited in order to ensure data consistency and authenticity; on the other hand, different LiDARs and different sensor deployment schemes will produce different point clouds for the same scene, so when the sensor is changed, the actual point cloud data previously collected by the LiDAR cannot be used as training data.

[0005] Another approach is to use simulation data for data augmentation. This involves constructing a target model using modeling software, placing the target model in a simulation environment, and using simulated LiDAR beams to generate reflected laser points. The advantage of this method is that the generated training data is not limited by the target's distance, orientation, or pose, and it can simulate arbitrarily distributed LiDAR beams, making it more flexible. However, modeling detailed targets is very time-consuming (e.g., complex surface modeling is required for vehicles), thus preventing batch modeling. Furthermore, if existing models from virtual scenes such as games are used, the available models (e.g., models for different vehicle models) are very limited, making it difficult to meet the diverse training data requirements of detection algorithms. Summary of the Invention

[0006] In view of this, embodiments of this specification provide a training method, training device, and electronic device for a perception model, which can balance the generation efficiency and diversity of training data used for training the perception model, thereby improving the training effect of the perception model.

[0007] This specification discloses a training method for a perception model, comprising:

[0008] Acquire actual point cloud data continuously collected by lidar;

[0009] Target modeling is performed based on the actual point cloud data to obtain the target model;

[0010] The laser beam information of the lidar is calibrated and measured to obtain the distribution of the laser beam;

[0011] Based on the distribution of the laser beams on the reflected laser points on the target model, simulated point cloud data is obtained;

[0012] The fused training data is obtained based on the actual point cloud data and the simulated point cloud data, and the perception model is trained using the fused training data.

[0013] Optionally, the step of performing target modeling based on the actual point cloud data to obtain the target model includes:

[0014] The target point cloud is obtained by extracting the actual point cloud data;

[0015] Register multiple frames of target point clouds to obtain dense point clouds;

[0016] Based on the dense point cloud, the target model is obtained through point cloud surface reconstruction.

[0017] Optionally, before performing point cloud surface reconstruction, the following steps are also included:

[0018] The dense point cloud is subjected to denoising, ground removal, and / or smoothing processes to obtain a smooth dense point cloud for point cloud surface reconstruction.

[0019] Optionally, the registration of multiple frames of target point clouds to obtain a dense point cloud includes:

[0020] The first frame is selected as the base frame from the multi-frame target point cloud according to the acquisition order, and the subsequent frames in the multi-frame target point cloud are matched and superimposed on the superimposed result of the previous frame in the acquisition order to obtain the dense point cloud.

[0021] Optionally, the step of sequentially matching and superimposing subsequent frames from the multi-frame target point cloud onto the superimposed result of the previous frame according to the acquisition order to obtain the dense point cloud includes:

[0022] The subsequent frames in the multi-frame target point cloud are first globally matched with the superposition result of the corresponding previous frame, and then locally matched with decreasing step size until the matching degree reaches a preset threshold. Then the matching and superposition of the next frame with the current superposition result is performed until the matching and superposition of the last frame is completed.

[0023] Optionally, the registration of multiple frames of target point clouds to obtain a dense point cloud includes:

[0024] The multi-frame target point cloud is divided into multiple sub-sequences according to the acquisition order;

[0025] The first frame of each of the multiple subsequences is used as the base frame of the corresponding subsequence. For any subsequence, the subsequent frames are matched and superimposed on the superimposed result of the previous frame in the order of acquisition until the matching and superimposition of the last frame in the subsequence is completed.

[0026] The results of the superposition of the multiple sub-sequences are matched and superimposed sequentially according to the acquisition order to obtain the dense point cloud.

[0027] Optionally, obtaining the simulated point cloud data based on the distribution of the laser beam on the reflected laser points on the target model includes:

[0028] The simulated point cloud data is obtained based on the reflected laser points distributed on the target model by the laser beam in different scenarios;

[0029] And / or,

[0030] Based on the reflected laser points distributed on the target model under different lidar and sensor settings, the simulated point cloud data is obtained.

[0031] And / or,

[0032] The simulated point cloud data is obtained by distributing the reflected laser points on the target model under different poses of the laser beam.

[0033] Optionally, obtaining fused training data based on the actual point cloud data and the simulated point cloud data includes:

[0034] Based on the required data volume of the perception model, determine the amount of actual point cloud data and the fusion ratio of the simulated point cloud data to be used, and obtain the fused training data.

[0035] This specification also provides an embodiment of a training apparatus for a perception model, comprising:

[0036] The actual point cloud acquisition unit is suitable for acquiring actual point cloud data continuously collected by lidar.

[0037] The modeling unit is adapted to perform target modeling based on the actual point cloud data to obtain the target model;

[0038] The calibration measurement unit is adapted to calibrate and measure the beam information of the lidar to obtain the distribution of the laser beam;

[0039] The simulation point cloud acquisition unit is adapted to obtain simulation point cloud data based on the distribution of the laser beam on the reflected laser points on the target model;

[0040] The fusion training unit is adapted to obtain fusion training data based on the actual point cloud data and the simulated point cloud data, and to use the fusion training data to train the perception model.

[0041] This specification also provides an electronic device, including a memory and a processor, wherein the memory stores computer instructions executable on the processor, wherein the processor executes the steps of the method described in any of the foregoing embodiments when executing the computer instructions.

[0042] The training method for the perception model using the embodiments of this specification firstly, since the target model is constructed based on the actual point cloud data continuously collected by the lidar, a large number of detailed target models can be constructed rapidly and accurately on a large scale. Secondly, by calibrating and measuring the lidar beam information, the distribution of the laser beams is obtained. Based on the reflected laser points on the target model according to the distribution of the laser beams, the resulting simulated point cloud data is not limited by target distance, direction, or pose, and can simulate arbitrarily distributed laser beams of the lidar. Therefore, a large amount of simulated point cloud data can be generated in batches, efficiently meeting the requirements for the quantity and diversity of training data during the lidar perception model training process. Therefore, by fusing the actual point cloud data and the simulated point cloud data, and then using the resulting fused training data to train the perception model, both the diversity and accuracy of the training data can be considered, improving the training effect of the perception model.

[0043] Furthermore, by extracting the actual point cloud data, a target point cloud is obtained. Then, multiple frames of target point clouds are registered to obtain a dense point cloud. Based on the dense point cloud, the target model is obtained through point cloud surface reconstruction, which can achieve accurate, fast, and large-scale target modeling.

[0044] Furthermore, before reconstructing the point cloud surface, the dense point cloud is denoised, grounded, and / or smoothed to obtain a smooth dense point cloud for point cloud surface reconstruction. This reduces data noise and further improves the accuracy of the reconstructed target model.

[0045] Furthermore, the multi-frame target point cloud is divided into multiple sub-sequences according to the acquisition order. The first frame of each sub-sequence is used as the base frame for that sub-sequence. For any sub-sequence, subsequent frames are sequentially matched and superimposed onto the superimposed result of the previous frame according to the acquisition order, until the matching and superimposition of the last frame in the sub-sequence is completed. Then, the superimposed results of the multiple sub-sequences are sequentially matched and superimposed according to the acquisition order to obtain the dense point cloud. Using the above segmented superimposition method can reduce superimposition matching errors and improve registration accuracy.

[0046] Furthermore, the simulated point cloud data can be obtained based on the reflected laser points distributed on the target model by the laser beam in different scenarios; and / or, based on the reflected laser points distributed on the target model by the laser beam under different lidar and sensor setting parameters; and / or, based on the reflected laser points distributed on the target model by the laser beam in different poses of the target model. Accordingly, the perception performance of the perception model for different scenarios can be improved, and the generated simulated point cloud data can be made unrestricted by target distance, direction, pose, etc., thereby enhancing the perception model's ability to solve long-tail problems.

[0047] Furthermore, based on the required data volume of the perception model, the quantity and fusion ratio of the actual point cloud data and the simulated point cloud data are determined to obtain the fused training data. Simulated point cloud data and actual point cloud data can be selected controllably according to requirements to fully leverage the data augmentation effect of simulated point cloud data, improve the target model's ability to solve long-tail problems, and enhance the perception performance of the perception model in different scenarios (including low-probability scenarios). Attached Figure Description

[0048] Figure 1 A flowchart of a training method for a perception model in an embodiment of this specification is shown;

[0049] Figure 2 A flowchart of a method for obtaining a target model in an embodiment of this specification is shown;

[0050] Figure 3 This shows a dense point cloud map obtained by registration in a specific scenario of Embodiment 1 of this specification;

[0051] Figure 4 The diagram illustrates a smooth, dense point cloud obtained in a specific scenario of Embodiment 1 of this specification.

[0052] Figure 5 This document shows a diagram of the reconstructed target model in a specific scenario of Embodiment 1 of this specification.

[0053] Figure 6 The diagram shows a simulated point cloud of a specific scene obtained in an embodiment of this specification.

[0054] Figure 7 A schematic diagram of the structure of a training device for a perception model in an embodiment of this specification is shown;

[0055] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of this specification is shown. Detailed Implementation

[0056] A point cloud is a massive collection of points that represents the spatial distribution and surface effects of a target within the same spatial reference frame. It is the set of points obtained after acquiring the spatial coordinates of each sampled point on the object's surface. Point clouds obtained through laser measurement include information such as three-dimensional coordinates (XYZ) and laser reflection intensity. The laser reflection intensity information is related to the target's surface material, roughness, incident angle, and the laser radar's emission capability and laser wavelength.

[0057] As described in the background section, the sparsity of LiDAR point clouds poses challenges to target detection. To ensure the performance of detection algorithms, especially for tasks involving long-range, occluded, or special target scenarios, a large amount of training and testing data is required. Currently, there are two commonly used data augmentation methods for LiDAR point clouds:

[0058] One approach is to use actual point cloud data for data augmentation, achieved through sampling, rotation, mirroring, partial deletion, copying and pasting, etc. However, on the one hand, since the distribution of point clouds changes with distance and direction, using actual point cloud data is greatly limited in order to ensure data consistency and authenticity; on the other hand, different radars and different sensor deployment schemes will produce different point clouds for the same scene, so when the sensor is changed, the previously collected actual point cloud data cannot be used as training data.

[0059] Another approach is to use simulation data for data augmentation. This involves constructing a target model using modeling software, placing the target model in a simulation environment, and using simulated laser beams to generate reflected laser points. The advantage of this method is that the generated training data is not limited by the target's distance, orientation, or pose, and it can simulate arbitrarily distributed laser beams, making it more flexible. However, modeling detailed targets is very time-consuming (e.g., complex surface modeling is required for vehicles), thus preventing batch modeling. Furthermore, if existing models from virtual scenes such as games are used, the available models (e.g., models for different vehicle models) are very limited, making it difficult to meet the diverse training data requirements of detection algorithms.

[0060] To address the aforementioned issues, this specification provides a corresponding training scheme for the perception model. First, a target model is constructed based on continuously acquired actual point cloud data from a lidar system. This allows for the large-scale, rapid, and accurate generation of numerous detailed target models. Next, by calibrating and measuring the lidar's beam information, the distribution of the laser beams is obtained. Based on the reflected laser points on the target model according to this distribution, the resulting simulated point cloud data is not limited by target distance, direction, or pose, and can simulate arbitrarily distributed laser beams from the lidar. Therefore, a large amount of simulated point cloud data can be generated in batches, efficiently meeting the requirements for quantity and diversity of training data during lidar perception model training. Thus, by fusing the actual point cloud data and the simulated point cloud data, and then using the resulting fused training data to train the perception model, both the diversity and accuracy of the training data can be considered, improving the training effect of the perception model.

[0061] To enable those skilled in the art to better understand the technical concept, technical principle and advantages of the embodiments of this specification, and to better implement the solutions in the embodiments of this specification, the training scheme of the perception model used in the embodiments of this specification will be described in detail below with reference to the accompanying drawings, through specific scenarios and application examples.

[0062] Reference Figure 1 The flowchart of the training method for the perception model is shown in the embodiments of this specification. The perception model can be trained using the following steps.

[0063] S11, acquire the actual point cloud data continuously collected by the lidar.

[0064] The actual point cloud data is derived from real data actually collected by the lidar. In the embodiments of this specification, the type and arrangement of lidar that collects the actual point cloud data are not limited.

[0065] In practical implementation, the actual point cloud data may include multiple frames of point cloud information and annotation information. To improve the utilization efficiency of the actual point cloud data, it may include multiple frames of continuously acquired point cloud information and annotation information. The annotation information may vary depending on the target object and application requirements. For vehicles, according to the needs of autonomous driving, the annotation information may include 7-dimensional coordinate data of the point cloud, including the vehicle center point coordinates (XYZ), length, width, height information, and heading angle.

[0066] In practice, existing open-source datasets can be used as the actual point cloud data.

[0067] S12, Target modeling is performed based on the actual point cloud data to obtain the target model.

[0068] In practical implementation, a corresponding target model can be created based on the target object appearing in the scene. In the embodiments of this specification, target modeling can be performed on rigid objects (such as vehicles) that do not deform during operation. During the observation process of the lidar, each frame of point cloud data is an accurate measurement of the object at different angles.

[0069] S13, calibrate the laser radar beam information to obtain the distribution of the laser beam.

[0070] In practice, the distribution of the laser beams of the lidar in the simulation environment can be obtained by calibrating and measuring the beam information of the real lidar.

[0071] As an optional example, calibration measurements can be performed on lidars with different layout schemes (e.g., different numbers, different distribution locations, etc.) to obtain the distribution of the laser beams of the corresponding lidars.

[0072] S14. Based on the distribution of the laser beams on the reflected laser points on the target model, simulated point cloud data is obtained.

[0073] In practical implementation, based on the distribution of the laser beams among the reflected laser points on the target model, simulated point cloud data can be obtained in various ways. Some examples are given below:

[0074] 1) Based on the reflected laser points distributed on the target model under different scenarios, the simulated point cloud data is obtained;

[0075] 2) Based on the reflected laser points distributed on the target model under different lidar and sensor settings, the simulated point cloud data is obtained;

[0076] 3) Based on the reflected laser points distributed on the target model under different poses of the laser beam, the simulated point cloud data is obtained.

[0077] In specific implementation, the simulated point cloud data can be obtained by any one or more of the above methods, thereby obtaining rich simulated point cloud data as training samples for the lidar perception model.

[0078] In practical applications, the actual point cloud data collected by LiDAR lacks sufficient coverage for low-probability scenarios such as long distances, occlusion, and special targets. Consequently, training the LiDAR perception model using actual point cloud data results in a limited ability to identify targets in low-probability scenarios. Furthermore, during training, data from low-probability scenarios may be ignored as noise, making it difficult for the perception model to identify targets in these situations. Therefore, in practical implementations, for different scenarios, especially low-probability scenarios, simulated point cloud data for corresponding target models can be generated controllably according to requirements by changing at least one of the following methods: adjusting the LiDAR sensor settings and target placement pose. This adapts to different LiDAR sensor solutions, offering high flexibility and improving the LiDAR perception model's recognition ability in different scenarios, especially low-probability scenarios, thus comprehensively enhancing the detection performance of the LiDAR perception model.

[0079] S15, obtain fused training data based on the actual point cloud data and the simulated point cloud data, and use the fused training data to train the perception model.

[0080] In practical implementation, the amount of actual point cloud data and the fusion ratio of simulated point cloud data can be determined according to the required data volume of the perception model to obtain the fused training data, thereby taking into account both the accuracy and diversity of the training data and obtaining better training results.

[0081] When training the perception model, a reasonable training strategy can be formulated as needed. The sampling methods and loss function weights of simulated point cloud data and actual point cloud data can be selected according to the task requirements to give full play to the data augmentation effect of simulated point cloud data, improve the perception model's ability to solve problems in medium and long tail regions, and improve perception performance in low-probability scenarios.

[0082] The trained perception model is tested using a real point cloud dataset until it meets the testing requirements, thus obtaining a target perception model that can be used for target detection.

[0083] The actual point cloud dataset used for testing may include point cloud information and annotation information, wherein the annotation information may include target pose information. In a specific implementation, the point cloud information of the actual point cloud dataset is input into the trained perception model for testing to determine whether the perception model meets the performance requirements. If the perception model does not meet the performance requirements, the training method described in the foregoing embodiments of this specification is used to train it until it passes the test.

[0084] More specifically, during the training process of the perception model, it is determined whether the target in the point cloud of the corresponding frame is consistent with the corresponding annotation information. Based on the test results of the test dataset, it is determined whether the perception model meets preset test requirements, such as overall accuracy and accuracy in various specific scenarios. If the test requirements are not met, training data is re-inputted into the perception model. Here, based on the previous test performance, the ratio and amount of simulated point cloud data and actual point cloud data in the training data can be adjusted, or the amount of training data in certain specific scenarios can be increased.

[0085] Using the embodiments of this specification, firstly, a target model is constructed based on actual point cloud data, enabling the large-scale, rapid, and accurate construction of numerous refined target models. Then, by calibrating and measuring the laser beam information of the lidar, the distribution of the laser beams is obtained. Based on the distribution of the laser beams and the reflected laser points on the target model, simulated point cloud data is obtained. This allows for the simulation of laser beams from lidars with arbitrary distributions in a simulation environment, without limitations on target distance, direction, or pose. Therefore, a large amount of simulated point cloud data can be generated in batches, efficiently meeting the requirements for the quantity and diversity of training data during lidar perception model training. Thus, by combining the actual point cloud data and the simulated point cloud data to obtain fused point cloud data, which jointly participates in the training of the perception model, the training effect of the perception model can be comprehensively improved, thereby comprehensively enhancing the detection performance of the perception model.

[0086] In practice, the training method of the above-mentioned perception model can be further optimized and extended as needed. The following is a detailed description through specific examples and application scenarios.

[0087] Taking a vehicle as an example, the target modeling principle of the embodiments in this specification is explained as follows: Assuming the vehicle is a rigid object that does not deform during operation, each frame of the laser point cloud during lidar observation represents an accurate measurement of the vehicle from different angles. By superimposing multiple frames of the target vehicle's observation point clouds at specific relative poses, dense target surface measurements from multiple angles can be obtained. Based on this principle, the embodiments in this specification utilize the superposition of measurements from different observation angles to achieve denser geometric measurements. This overcomes the difficulty caused by the sparsity of the laser point cloud itself in modeling the target object, achieving accurate and efficient target modeling.

[0088] Reference Figure 2 The flowchart of the method for obtaining the target model is shown below. For step S12, the specific steps are shown to illustrate a specific method for obtaining the target model.

[0089] S121, the actual point cloud data is extracted to obtain the target point cloud.

[0090] In practice, for input multi-frame actual point cloud data, the target point cloud can be obtained by cropping. Specifically, based on the annotation information corresponding to the actual point cloud data, such as the annotation bounding box, the cropping range can be determined. Cropping the observed point cloud within the annotation bounding box yields the target point cloud.

[0091] It should also be noted that, in specific implementation, if the actual laser point cloud is acquired by multiple lidars, the laser point clouds acquired by the multiple lidars can be fused to obtain a single observation point cloud with a wider field of view. The target point cloud can then be obtained by cropping the fused observation point cloud.

[0092] S122, register multiple frames of target point clouds to obtain dense point clouds.

[0093] In practice, point cloud registration algorithms can be used to obtain dense point clouds. Users can choose the appropriate point cloud registration algorithm to implement it as needed.

[0094] In some embodiments of this specification, a first frame is selected from the multi-frame target point cloud according to the acquisition order as the base frame, and subsequent frames in the multi-frame target point cloud are sequentially matched and superimposed onto the superimposed result of the previous frame according to the acquisition order to obtain the target point cloud. More specifically, for a multi-frame target point cloud, the second frame can be superimposed with the first frame as the base frame to obtain the superimposed result of the second frame and the first frame. Then, the third frame of the target point cloud is matched and superimposed with the superimposed result, and so on, until every frame in the multi-frame target point cloud has been matched and superimposed.

[0095] In particular, for any two frames of target point clouds, a superposition process can be performed, such as superimposing the second frame target point cloud with the first frame target point cloud in a multi-frame target point cloud, or superimposing the i-th frame with the (i-1)-th superposition result, where i is a natural number and i≥2. In specific implementation, global matching can be used first, followed by local matching for matching and superposition.

[0096] Specifically, subsequent frames in the multi-frame target point cloud can be globally matched with the superposition result of their corresponding previous frames, and then locally matched with decreasing step sizes until the matching degree reaches a preset threshold. Then, the next frame is matched and superimposed with the current superposition result, until the matching and superposition of the last frame is completed. During the local matching process, the matching step size can be decreased. That is, a coarse matching can be performed with a larger step size initially. If the matching degree reaches the corresponding threshold, the matching is completed; otherwise, the step size is continuously reduced for a more refined matching until the matching degree reaches the corresponding threshold.

[0097] During global matching, if it is determined to be a mirror image overlay, coordinate matching is performed first. During local matching, local features of the two point clouds can be extracted and matched to obtain a coordinate transformation matrix, which can then be used for registration.

[0098] In the aforementioned local matching process, a relatively accurate position can be quickly achieved through coarse matching with a larger step size. Then, a more refined matching with a smaller step size can improve the matching accuracy. Therefore, both registration accuracy and registration efficiency can be balanced.

[0099] In specific implementation, in order to reduce superposition errors during the registration process, the multi-frame target point cloud can be divided into multiple sub-sequences according to the acquisition order. Then, the first frame of each of the multiple sub-sequences is used as the base frame of the corresponding sub-sequence. For any sub-sequence, the subsequent frames are matched and superimposed onto the superposition result of the previous frame according to the acquisition order until the matching and superposition of the last frame in the sub-sequence is completed. Then, the superposition results of the multiple sub-sequences can be matched and superimposed sequentially according to the acquisition order to obtain the dense point cloud.

[0100] The matching process for two target point clouds in each subsequence can be referred to as the superposition process of any two target point clouds mentioned above, or a method of first global matching and then local matching can be adopted, as described in detail in the aforementioned embodiments.

[0101] In addition, after the matching and superposition of each subsequence is completed, in order to fill in any missing observation angles, since vehicles are generally symmetrical, the superposition result can be mirrored again. Therefore, the mirrored superposition result can be superimposed again with the original superposition result to form the dense point cloud.

[0102] In an example scenario of this specification, a schematic diagram of the point cloud effect obtained by overlaying the second frame target point cloud with the first frame target point cloud in a multi-frame target point cloud is shown below. Figure 3 As shown, it can be seen that the superposition and registration of point clouds of different targets can yield more refined surface measurements.

[0103] S123, the dense point cloud is subjected to denoising, ground removal and / or smoothing to obtain a smooth dense point cloud for point cloud surface reconstruction.

[0104] The dense point cloud obtained from registration may contain noisy point clouds. Therefore, optional step S123 can be used for filtering.

[0105] Specifically, overlaying multiple frames of target point clouds may introduce noisy point clouds and ground point clouds. These can be identified using height information and local relative distances, allowing for the removal of ground and noisy point clouds to obtain a smooth, dense point cloud. Figure 4The diagram shows a smooth, dense point cloud effect.

[0106] S124, Based on the dense point cloud, the target model is obtained by reconstructing the point cloud surface.

[0107] Based on the dense point cloud obtained in step S122 or the smooth dense point cloud obtained in step S123, triangular facets can be obtained through surface reconstruction to complete the simulation reconstruction based on the actual point cloud data and obtain the target model. In specific implementations, the dense point cloud data can be input into modeling and simulation software, and the target model can be obtained through point cloud generation. For example... Figure 4 The smooth point cloud shown, after being reconstructed from the point cloud surface, yields the target model as follows: Figure 5 As shown. Figure 6 A simulated point cloud data obtained using the method described in the embodiments of this specification is presented.

[0108] In the above embodiments, simulated point cloud data generated based on actual point cloud data is used for LiDAR perception model training. Since the simulated point cloud data can be generated in batches and has diversity, after training the perception model with the actual point cloud data and simulated point cloud data, the perception model can be used for target detection, which can meet the detection algorithm's requirements for the diversity and accuracy of point cloud data, including low-probability scenarios, thus improving detection performance.

[0109] In practical implementation, the trained perception model can be applied to an autonomous vehicle, which can be equipped with a LiDAR and a target detection system. The target detection system can include the perception model trained using the training method described in the foregoing embodiments of this specification. By continuously acquiring real point cloud data through the LiDAR and inputting the real point cloud data into the perception model for target recognition, a target detection result can be obtained.

[0110] In practical implementation, the target detection result can be based on the identified target object, outputting the detection information contained therein, such as the target object and its pose information. As a specific application example, the data can be further processed according to specific business needs, or the detection result can be further input into downstream applications for decision-making in specific business operations.

[0111] For example, the detection results can be displayed as an image or overlaid on a map. Alternatively, the detection results can be output to an autonomous driving system to determine appropriate driving strategies, including driving speed and whether to avoid obstacles.

[0112] Before training the LiDAR perception model, a target model is first obtained based on actual point cloud data. The distribution of the laser beams is obtained by calibrating and measuring the beam information of the LiDAR. Based on the reflected laser points on the target model according to the distribution of the laser beams, simulated point cloud data is obtained. The simulated point cloud data obtained above and the actual point cloud data are fused for training the perception model, which can greatly expand the scale and diversity of the data used for training the perception model. Therefore, the obtained perception model has better robustness. Thus, using the LiDAR perception model for target detection can improve target detection performance, including the ability to identify targets in low-probability scenarios.

[0113] This specification also provides product embodiments corresponding to the training method of the above-described perception model, which will be described below with reference to the accompanying drawings.

[0114] Reference Figure 7 The schematic diagram of the training device for the perception model shown is illustrated in the embodiments of this specification. The training device 70 for the perception model may include: an actual point cloud acquisition unit 71, a modeling unit 72, a calibration and measurement unit 73, a simulated point cloud acquisition unit 74, and a fusion training unit 75, wherein:

[0115] The actual point cloud acquisition unit 71 is adapted to acquire actual point cloud data continuously collected by the lidar.

[0116] The modeling unit 72 is adapted to perform target modeling based on the actual point cloud data to obtain a target model;

[0117] The calibration measurement unit 73 is adapted to calibrate and measure the beam information of the lidar to obtain the distribution of the laser beam;

[0118] The simulation point cloud acquisition unit 74 is adapted to obtain simulation point cloud data based on the distribution of the laser beam on the reflected laser points on the target model.

[0119] The fusion training unit 75 is adapted to obtain fusion training data based on the actual point cloud data and the simulated point cloud data, and to train the perception model using the fusion training data.

[0120] In a specific implementation, the modeling unit 72 is adapted to extract the actual point cloud data to obtain the target point cloud, register multiple frames of the target point cloud to obtain a dense point cloud, and obtain the target model based on the dense point cloud by reconstructing the point cloud surface.

[0121] As an optional example, the modeling unit 72 may also perform denoising, ground removal and / or smoothing on the dense point cloud before performing point cloud surface reconstruction to obtain a smooth dense point cloud for point cloud surface reconstruction.

[0122] The method by which the modeling unit 72 registers multiple frames of target point clouds to obtain dense point clouds can be found in the aforementioned embodiments, and will not be described in detail here.

[0123] In specific implementation, the simulation point cloud acquisition unit 74 obtains simulation point cloud data in various ways based on the distribution of the laser beam on the reflected laser points on the target model. For example, the simulation point cloud data can be obtained through at least one of the following methods:

[0124] 1) Based on the reflected laser points distributed on the target model under different scenarios, the simulated point cloud data is obtained;

[0125] 2) Based on the reflected laser points distributed on the target model under different lidar and sensor settings, the simulated point cloud data is obtained;

[0126] 3) Based on the reflected laser points distributed on the target model under different poses of the laser beam, the simulated point cloud data is obtained.

[0127] In a specific implementation, the fusion training unit 75 is adapted to determine the amount of actual point cloud data and the fusion ratio of simulated point cloud data based on the required data volume of the perception model, so as to obtain the fusion training data.

[0128] In specific implementations, the training method of the perception model described in the foregoing embodiments can run on electronic devices such as computer terminals, including vehicle terminals, personal computers, tablet computers, etc., or it can run on servers, in the cloud, or through computer clusters. The training device of the perception model described in the foregoing embodiments can also be installed, stored, and run on electronic devices.

[0129] Reference Figure 8 The schematic diagram of the electronic device shown in this specification illustrates an electronic device provided in this embodiment, such as... Figure 8 As shown, the electronic device 80 includes a memory 81 and a processor 82. The memory 81 stores computer instructions that can be executed on the processor 82, wherein the processor 82 executes the steps of the method described in any of the foregoing embodiments when executing the computer instructions.

[0130] In a specific implementation, the electronic device may also include a display 83, which is suitable for displaying the running results and can also output and display the intermediate execution process.

[0131] In specific implementation, the real point cloud data or training data can be obtained through the input interface 84, or the actual point cloud data or training data can be obtained through the communication interface 85.

[0132] In a specific implementation, the memory 81, processor 82, display 83, input interface 84 and communication interface 85 can communicate with each other via bus 86.

[0133] This specification also provides a computer-readable storage medium storing computer instructions thereon, wherein the computer instructions, when executed, can perform the steps of the methods described in any of the foregoing embodiments.

[0134] In practice, the computer-readable storage medium can be any suitable readable storage medium such as optical disc, hard disk drive, solid-state drive, etc.

[0135] As described in the previous embodiments, the aforementioned perception model can be applied to autonomous vehicles. The autonomous vehicle can be equipped with corresponding algorithm modules for autonomous driving decision-making. Of course, these algorithm modules will differ depending on the type of autonomous vehicle. For example, different algorithm modules will be involved for logistics vehicles, public service vehicles, medical service vehicles, and terminal service vehicles. The following provides examples of the algorithm modules for these four types of autonomous vehicles:

[0136] Logistics vehicles refer to vehicles used in logistics scenarios, such as logistics vehicles with automatic sorting functions, logistics vehicles with refrigeration and insulation functions, and logistics vehicles with measurement functions. These logistics vehicles involve different algorithm modules.

[0137] For example, logistics vehicles can be equipped with automated sorting devices that can automatically retrieve, transport, sort, and store goods after the vehicle arrives at its destination. This involves an algorithm module for goods sorting, which mainly implements the logical control of goods retrieval, handling, sorting, and storage.

[0138] For example, in cold chain logistics scenarios, logistics vehicles can also be equipped with refrigeration and insulation devices. These devices can refrigerate or keep warm transported fruits, vegetables, aquatic products, frozen foods, and other perishable foods, maintaining them in a suitable temperature environment and solving the problem of long-distance transportation of perishable foods. This involves an algorithm module for refrigeration and insulation control. This module is mainly used to dynamically and adaptively calculate the appropriate temperature for refrigeration or insulation based on information such as the nature of the food (or item), its perishability, transportation time, current season, and climate. Based on this appropriate temperature, the refrigeration and insulation device is automatically adjusted. This eliminates the need for manual temperature adjustments by transport personnel when transporting different foods or items, freeing them from tedious temperature control and improving the efficiency of refrigerated and insulated transportation.

[0139] For example, in most logistics scenarios, charges are based on package volume and / or weight. However, the number of logistics packages is enormous, and relying solely on couriers to measure package volume and / or weight is highly inefficient and labor-intensive. Therefore, some logistics vehicles are equipped with measuring devices that automatically measure the volume and / or weight of logistics packages and calculate the cost. This involves an algorithm module for logistics package measurement. This module primarily identifies the type of logistics package, determines the measurement method (e.g., volume measurement, weight measurement, or a combination of both), and completes the volume and / or weight measurement based on the determined method, as well as calculating the cost based on the measurement results.

[0140] Public service vehicles refer to vehicles that provide certain public services, such as fire trucks, de-icing trucks, water trucks, snowplows, garbage trucks, and traffic control vehicles. These public service vehicles involve different algorithm modules.

[0141] For example, the main task of an autonomous fire truck is to carry out reasonable fire extinguishing tasks at the fire scene. This involves an algorithm module for fire extinguishing tasks. This algorithm module needs to realize at least the logic of fire condition identification, fire extinguishing plan planning, and automatic control of fire extinguishing devices.

[0142] For example, the main task of a de-icing truck is to remove ice and snow from the road surface. This involves a de-icing algorithm module, which at least needs to identify the ice and snow conditions on the road surface, formulate a de-icing plan based on the ice and snow conditions, such as which road sections need de-icing, which road sections do not need de-icing, whether to use salting, the amount of salt to be applied, and the logic for automatic control of the de-icing device after determining the de-icing plan.

[0143] Among them, medical service vehicles refer to autonomous vehicles capable of providing one or more medical services. These vehicles can provide medical services such as disinfection, temperature measurement, medication dispensing, and isolation. This involves algorithm modules that provide various self-service medical services. These algorithm modules mainly realize the identification of disinfection needs and the control of disinfection devices so that the disinfection devices can disinfect patients; or the identification of patient positions and the control of temperature measuring devices to automatically place them close to the patient's forehead or other positions to measure the patient's temperature; or the identification of symptoms, the provision of prescriptions based on the diagnosis results, and the identification of medicines / medicine containers, as well as the control of the medication dispensing robot to make it grab medicines for patients according to the prescription, etc.

[0144] Among them, terminal service vehicles refer to self-service autonomous vehicles that can replace some terminal equipment to provide users with certain convenient services. For example, these vehicles can provide users with services such as printing, attendance, scanning, unlocking, payment, and retail.

[0145] For example, in some application scenarios, users often need to go to a specific location to print or scan documents, which is time-consuming and laborious. Therefore, a terminal service vehicle has emerged that can provide printing / scanning services to users. These service vehicles can interconnect with user terminal devices. Users issue print commands through their terminal devices, and the service vehicle responds by automatically printing the required documents and delivering them to the user's location. Users do not need to queue at the printer, greatly improving printing efficiency. Alternatively, it can respond to scanning commands issued by users through their terminal devices, move to the user's location, and the user places the document to be scanned on the service vehicle's scanning tool to complete the scan, eliminating the need to queue at the printer / scanner and saving time and effort. This involves an algorithm module that provides printing / scanning services. This algorithm module needs to at least recognize the interconnection with the user's terminal device, the response to print / scan commands, the user's location, and movement control.

[0146] For example, with the development of new retail businesses, more and more e-commerce companies are using vending machines to deliver goods to office buildings and public areas. However, these vending machines are placed in fixed locations and cannot be moved. Users need to go to the vending machine to purchase the goods they need, which is still inconvenient. Therefore, self-driving vehicles that provide retail services have emerged. These vehicles can carry goods automatically and provide corresponding self-service shopping applications (APPs) or shopping portals. Users can place orders with the self-driving vehicles using their mobile phones or other terminals through the APP or shopping portal. The order includes the name and quantity of the goods to be purchased and the user's location. After receiving the order request, the vehicle can determine whether there are any remaining goods and whether the quantity is sufficient. If it determines that there are goods and the quantity is sufficient, it can automatically move these goods to the user's location and provide them to the user, further improving the convenience of shopping, saving users time, and allowing them to focus their time on more important things. This involves the algorithm modules that provide retail services. These modules mainly implement logic such as responding to user order requests, order processing, product information maintenance, user location positioning, and payment management.

[0147] While the embodiments disclosed in this specification are as described above, the present invention is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A training method for a perceptual model, wherein, include: Acquire actual point cloud data continuously collected by lidar; Target modeling is performed based on the actual point cloud data to obtain the target model; The laser beam information of the lidar is calibrated and measured to obtain the distribution of the laser beam; Based on the distribution of the laser beams on the reflected laser points on the target model, simulated point cloud data is obtained, and the simulated point cloud data is used to achieve data augmentation. The fused training data is obtained based on the actual point cloud data and the simulated point cloud data, and the perception model is trained using the fused training data. The step of obtaining simulated point cloud data based on the distribution of the laser beams on the reflected laser points of the target model includes: The simulated point cloud data is obtained based on the reflected laser points distributed on the target model by the laser beam in different scenarios; And / or, Based on the reflected laser points distributed on the target model under different lidar and sensor settings, the simulated point cloud data is obtained. And / or, The simulated point cloud data is obtained by distributing the reflected laser points on the target model under different poses of the laser beam.

2. The method according to claim 1, wherein, The step of modeling the target based on the actual point cloud data to obtain the target model includes: The target point cloud is obtained by extracting the actual point cloud data; Register multiple frames of target point clouds to obtain dense point clouds; Based on the dense point cloud, the target model is obtained through point cloud surface reconstruction.

3. The method according to claim 2, wherein, Before performing point cloud surface reconstruction, the following is also included: The dense point cloud is subjected to denoising, ground removal, and / or smoothing processes to obtain a smooth dense point cloud for point cloud surface reconstruction.

4. The method according to claim 2, wherein, The process of registering multiple frames of target point clouds to obtain a dense point cloud includes: The first frame is selected as the base frame from the multi-frame target point cloud according to the acquisition order, and the subsequent frames in the multi-frame target point cloud are matched and superimposed on the superimposed result of the previous frame in the acquisition order to obtain the dense point cloud.

5. The method according to claim 4, wherein, The step of sequentially matching and superimposing subsequent frames from the multi-frame target point cloud onto the superimposed result of the previous frame according to the acquisition order to obtain the dense point cloud includes: The subsequent frames in the multi-frame target point cloud are first globally matched with the superposition result of the corresponding previous frame, and then locally matched with decreasing step size until the matching degree reaches a preset threshold. Then the matching and superposition of the next frame with the current superposition result is performed until the matching and superposition of the last frame is completed.

6. The method according to claim 2, wherein, The process of registering multiple frames of target point clouds to obtain a dense point cloud includes: The multi-frame target point cloud is divided into multiple sub-sequences according to the acquisition order; The first frame of each of the multiple subsequences is used as the base frame of the corresponding subsequence. For any subsequence, the subsequent frames are matched and superimposed on the superimposed result of the previous frame in the order of acquisition until the matching and superimposition of the last frame in the subsequence is completed. The results of the superposition of the multiple sub-sequences are matched and superimposed sequentially according to the acquisition order to obtain the dense point cloud.

7. The method according to claim 1, wherein, The process of obtaining fused training data based on the actual point cloud data and the simulated point cloud data includes: Based on the required data volume of the perception model, determine the amount of actual point cloud data and the fusion ratio of the simulated point cloud data to be used, and obtain the fused training data.

8. A training device for a perceptual model, wherein, include: The actual point cloud acquisition unit is suitable for acquiring actual point cloud data continuously collected by lidar. The modeling unit is adapted to perform target modeling based on the actual point cloud data to obtain the target model; The calibration measurement unit is adapted to calibrate and measure the beam information of the lidar to obtain the distribution of the laser beam; A simulation point cloud acquisition unit is adapted to obtain simulation point cloud data based on the distribution of reflected laser points of the laser beam on the target model; wherein, the simulation point cloud data is used for data augmentation, and the simulation point cloud acquisition unit is adapted to obtain simulation point cloud data based on the distribution of reflected laser points of the laser beam on the target model, including: The simulated point cloud data is obtained based on the reflected laser points distributed on the target model by the laser beam in different scenarios; And / or, Based on the reflected laser points distributed on the target model under different lidar and sensor settings, the simulated point cloud data is obtained. And / or, The simulated point cloud data is obtained by distributing the reflected laser points on the target model under different poses of the laser beam. The fusion training unit is adapted to obtain fusion training data based on the actual point cloud data and the simulated point cloud data, and to train the perception model using the fusion training data.

9. An electronic device comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, wherein, When the processor executes the computer instructions, it performs the steps of the method according to any one of claims 1 to 7.

Images