Point cloud simulation method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a point cloud simulation method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of automobile simulation, and the method comprises the steps: obtaining target sensing data; the target sensing data is processed through a target point cloud simulation model to obtain simulation point cloud data, the target point cloud simulation model is obtained by training a basic point cloud simulation model based on multiple pieces of sample sensing data, and the basic point cloud simulation model is constructed based on first actual sensing data; the sample sensing data and the first actual sensing data are obtained through detection of an actual sensor. According to the technical scheme, the target point cloud simulation model can be obtained through the actual sensing data collected by the actual sensor, so that more real simulation point cloud data can be obtained through the target point cloud simulation model, the simulation authenticity is improved, and the cost can be reduced. The simulation test of the auxiliary driving of the vehicle can be carried out based on the simulation point cloud data obtained by the large model.

Description

Technical Field

[0001] The present disclosure relates to the field of automobile simulation technology, and in particular to a point cloud simulation method, device, electronic device, storage medium, and program product. Background Art

[0002] The rapid development of assisted driving technology has placed higher demands on testing and verification. While real-world vehicle testing can provide realistic environmental data, it suffers from high costs, low efficiency, and difficulty ensuring safety. Therefore, simulation testing, as an efficient, safe, and highly repeatable testing method, has gradually become an essential component of assisted driving system development. Summary of the Invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a point cloud simulation method, device, electronic device, storage medium and program product, which can improve the authenticity of the point cloud data obtained by simulation.

[0004] According to a first aspect of an embodiment of the present disclosure, a point cloud simulation method is provided, comprising: Acquire target sensor data; The target sensor data is processed by a target point cloud simulation model to obtain simulated point cloud data. The target point cloud simulation model is obtained by training a basic point cloud simulation model based on multiple sample sensor data. The basic point cloud simulation model is constructed based on first actual sensor data. The sample sensor data and the first actual sensor data are both collected by actual sensors.

[0005] By constructing a basic point cloud simulation model through the actual sensor data collected by the actual sensor, and training the basic point cloud simulation model through the actual sensor data, a target point cloud simulation model that is closer to the function of the real sensor can be obtained, so that more realistic simulation point cloud data can be obtained through the target point cloud simulation model, thereby improving the authenticity of the simulation.

[0006] In some possible implementations, the simulated point cloud data is used to construct a scene database for assisted driving.

[0007] In this embodiment, the simulation point cloud data obtained by the above method can be used to construct a scene database for assisted driving of the vehicle. The data in the scene database can be used to perform simulation tests on the assisted driving of the vehicle as a whole or the vehicle controller to test the assisted driving capability of the vehicle. Alternatively, the data in the scene database can be used to train the assisted driving of the vehicle to improve the assisted driving capability of the vehicle.

[0008] In some possible implementations, acquiring target sensor data includes: Acquire simulated sensor data in the target scenario; The simulated sensor data is preprocessed to obtain the target sensor data; wherein the preprocessing includes cleaning and feature extraction of the simulated sensor data.

[0009] By cleaning the simulated sensor data, some data irrelevant to the point cloud simulation can be eliminated, thereby obtaining more accurate simulated point cloud data. By performing feature extraction on the candidate sensor data, some point cloud-related features can be extracted, which helps the model simulate the point cloud and obtain more realistic simulated point cloud data.

[0010] In some possible implementations, the method further includes: Processing the first actual sensing data to obtain point cloud distribution data and environmental impact parameters; constructing a first function for the number of point clouds and a second function for the reflection intensity based on the point cloud distribution data and the environmental impact parameter; The basic point cloud simulation model is constructed according to the first function and the second function.

[0011] By processing the actual sensor data, we can obtain point cloud distribution data and environmental impact parameters related to the number of point clouds and the reflection intensity of the point clouds. Then, by analyzing the point cloud distribution data and environmental impact parameters, we can more comprehensively analyze the factors affecting the point cloud data, fit the first function for the number of point clouds and the second function for the reflection intensity, and further construct a basic point cloud simulation model, which can make the obtained basic point cloud simulation model closer to the real sensor.

[0012] In some possible implementations, the first actual sensing data includes characteristic parameters and environmental parameters; The processing of the first actual sensing data to obtain point cloud distribution data and environmental impact parameters includes: Inputting the characteristic parameters into a target regression model to obtain the point cloud distribution data; The environmental parameters are processed to obtain the environmental impact parameters.

[0013] By processing the feature parameters through the target regression model, the corresponding point cloud data can be obtained. Combining these feature parameters, the overall point cloud distribution data can be obtained to facilitate point cloud data analysis. By obtaining the environmental impact parameters through the environmental parameters, the factors affecting the point cloud data can be made more comprehensive.

[0014] In some possible implementations, the method further includes: Periodically acquiring second actual sensing data; The target point cloud simulation model is updated according to the second actual sensing data.

[0015] By periodically acquiring the second actual sensing data so as to periodically update the target point cloud simulation model, the target point cloud simulation model can adapt to changes in the real environment and the point cloud data output by the target point cloud simulation model can be more realistic.

[0016] In some possible implementations, the second actual sensing data includes first sub-data and second sub-data; The periodically acquiring the second actual sensing data includes: periodically acquiring the first sub-data according to a first period duration; periodically acquiring the second sub-data according to a second cycle duration, wherein the first cycle duration is shorter than the second cycle duration; The updating of the target point cloud simulation model according to the second actual sensing data includes: The target point cloud simulation model is iteratively updated according to the first sub-data and the second sub-data respectively.

[0017] Short-term and long-term updates of the target point cloud simulation model are achieved through different cycle lengths. Short-term updates can quickly respond to short-term changes in the actual environment so as to improve short-term prediction accuracy, while long-term updates can ensure the consistency of long-term trends and improve the generalization ability of the target point cloud simulation model.

[0018] According to a second aspect of an embodiment of the present disclosure, there is provided a point cloud simulation device, comprising: A first acquisition module is configured to acquire target sensor data; The simulation module is configured to process the target sensor data through a target point cloud simulation model to obtain simulated point cloud data. The target point cloud simulation model is obtained by training a basic point cloud simulation model based on multiple sample sensor data. The basic point cloud simulation model is constructed based on first actual sensor data. The sample sensor data and the first actual sensor data are both collected by actual sensors.

[0019] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including: processor; a memory for storing processor-executable instructions; The processor is configured to implement the steps of the point cloud simulation method provided in the first aspect of the embodiment of the present disclosure when executing.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the steps of the point cloud simulation method provided in the first aspect of the embodiment of the present disclosure are implemented.

[0021] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the point cloud simulation method provided in the first aspect of the embodiment of the present disclosure.

[0022] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects: By using actual sensor data to build a basic point cloud simulation model and using actual sensor data to train the basic point cloud simulation model, a target point cloud simulation model that is closer to the function of a real sensor can be obtained, thereby obtaining more realistic simulation point cloud data through the target point cloud simulation model and improving the authenticity of the simulation.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] Figure 1 It is a schematic diagram of an actual scene corresponding to a point cloud simulation method according to an exemplary embodiment.

[0026] Figure 2 The figure is a flow chart of a point cloud simulation method according to an exemplary embodiment.

[0027] Figure 3 The present invention is a flow chart of a system architecture for obtaining a target point cloud simulation model according to an exemplary embodiment.

[0028] Figure 4 The figure is a block diagram of a point cloud simulation device according to an exemplary embodiment.

[0029] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements.

[0031] The embodiments described in the following examples of the present disclosure do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the corresponding data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0033] The rapid development of assisted driving technology has placed higher demands on testing and verification. While real-world vehicle testing can provide realistic environmental data, it suffers from high costs, low efficiency, and safety challenges. Therefore, simulation testing, as an efficient, safe, and highly repeatable testing method, has gradually become a crucial component of assisted driving system development. Related technologies, based on the physical model of electromagnetic wave propagation, can achieve point cloud simulation through physical modeling in a simulated environment. However, this is costly and lacks realism.

[0034] In response to the above technical problems, the embodiments of the present disclosure provide a point cloud simulation method, device, electronic device, storage medium and program product, which can construct a basic point cloud simulation model through actual sensor data collected by actual sensors. The basic point cloud simulation model is a neural network model, and the basic point cloud simulation model is trained by actual sensor data to obtain a target point cloud simulation model that is closer to the function of the real sensor, so that more realistic simulation point cloud data can be obtained through the target point cloud simulation model, thereby improving the authenticity of the simulation and reducing costs.

[0035] Figure 1 is a schematic diagram of an actual scene corresponding to a point cloud simulation method according to an exemplary embodiment. Figure 1 As shown, in actual scenarios, actual sensors, such as millimeter-wave radars, cameras, and lidars installed on actual vehicles, can be used for detection to obtain sensor data, which can then be processed to obtain point cloud data. The image data collected by the camera can be used to obtain the position of objects in the image through depth recognition, thereby generating corresponding point cloud data. The point cloud simulation method provided in the present disclosure can output simulated point cloud data based on the acquired sensor data by simulating the sensor, so as to perform simulation tests on vehicle-related functions based on the simulated point cloud data. For example, it can be used for simulation tests of the following scenarios: Assisted driving algorithm development and verification: used for training and testing of assisted driving perception algorithms (target detection / tracking / classification), supporting multi-sensor fusion algorithm development (radar + vision + Lidar).

[0036] HIL (Hardware-in-the-Loop) testing: Integrated with the HIL test bench, it provides real-time 4D millimeter-wave radar signal input and supports closed-loop verification of ECU (Electronic Control Unit) algorithms.

[0037] Assisted driving system safety certification: Meet the virtual verification requirements of ISO 21448 intended functional safety and build a test scenario library that complies with NCAP (New Car Assessment Program) standards.

[0038] Vehicle-road cooperative system testing: Simulate road testing of 4D millimeter-wave radar data, verify V2X (Vehicle-to-Everything) communication protocols, and build a large-scale traffic flow simulation environment.

[0039] Production sensor calibration: This replaces on-vehicle calibration and allows for virtual calibration of radar installation positions and angles, optimizing sensor FOV (Field of View) overlap parameters.

[0040] Assisted driving teaching and scientific research: Provides a customizable radar simulation teaching platform for universities / research institutions, supporting the generation of data sets for algorithm research.

[0041] Figure 2 is a flow chart of a point cloud simulation method according to an exemplary embodiment. Figure 2 As shown, the following steps may be included.

[0042] In step S201 , target sensing data is acquired.

[0043] In this embodiment, the target sensor data obtained may be sensor data for a target scenario. Different scenarios can be obtained based on different vehicle driving environments. For example, different road types can be divided into urban road scenarios, highway scenarios, and rural road scenarios. Different weather types can also be divided into rainy and snowy scenarios, sunny scenarios, and extreme weather scenarios. Based on the simulated driving environment, the corresponding target scenario can be determined, and the target sensor data can be obtained. The target sensor data can be simulated sensor data or real sensor data collected by actual sensors.

[0044] In step S202, the target sensor data is processed by the target point cloud simulation model to obtain simulated point cloud data. The target point cloud simulation model is obtained by training a basic point cloud simulation model based on multiple sample sensor data. The basic point cloud simulation model is constructed based on the first actual sensor data. Both the sample sensor data and the first actual sensor data are collected by actual sensors.

[0045] In this embodiment, both the first actual sensor data and the second actual sensor data can be obtained from data collected by actual sensors. For example, actual vehicle sensor data can be collected and preprocessed based on an onboard data acquisition unit deployed on the vehicle, such as a 4D millimeter-wave radar, a camera, a lidar, a global navigation satellite system, and an inertial measurement unit, to obtain the first actual sensor data or the second actual sensor data. A basic point cloud simulation model that is more consistent with actual conditions can be pre-constructed using the first actual sensor data. This basic point cloud simulation model is a neural network model with optimizable parameters. This basic point cloud simulation model takes sensor data as input and outputs corresponding point cloud data.

[0046] After constructing the basic point cloud simulation model, the basic point cloud simulation model can be trained using multiple sample sensor data. The sample sensor data may include third actual sensor data and actual point cloud data corresponding to the third actual sensor data. The basic point cloud simulation model is trained using the third actual sensor data under actual conditions and supervised learning is performed using the actual point cloud data, so that a target point cloud simulation model with more realistic and accurate output results can be obtained. Furthermore, during application, the target point cloud simulation model can be used to process the target sensor data to obtain corresponding simulated point cloud data. The simulated point cloud data may include the number of point clouds and the reflection intensity of the point clouds. Preferably, different scenes may correspond to different point cloud simulation models. Based on the target scene corresponding to the target sensor data, the target point cloud simulation model corresponding to the target scene may be determined to obtain more accurate and realistic simulated point cloud data.

[0047] In this embodiment, a basic point cloud simulation model is constructed by using actual sensor data collected by actual sensors, and the basic point cloud simulation model is trained by using actual sensor data, so that a target point cloud simulation model that is closer to the function of the real sensor can be obtained, thereby obtaining more realistic simulation point cloud data through the target point cloud simulation model, thereby improving the authenticity of the simulation.

[0048] In some possible implementations, the simulated point cloud data is used to construct a scene database for assisted driving.

[0049] In this embodiment, the simulation point cloud data obtained by the above method can be used to construct a scene database for assisted driving of the vehicle. The data in the scene database can be used to perform simulation tests on the assisted driving of the vehicle as a whole or the vehicle controller to test the assisted driving capability of the vehicle. Alternatively, the data in the scene database can be used to train the assisted driving of the vehicle to improve the assisted driving capability of the vehicle.

[0050] In some possible implementations, obtaining target sensor data may include: Acquire simulated sensor data in a target scene; preprocess the simulated sensor data to obtain target sensor data; wherein the preprocessing includes cleaning and feature extraction of the simulated sensor data.

[0051] In this embodiment, the target sensor data can be obtained by simulation. The simulated sensor data of the target scene can be obtained and then preprocessed to eliminate some data irrelevant to the point cloud simulation, thereby simulating more accurate simulated point cloud data.

[0052] Preprocessing methods may include cleaning and feature extraction. The simulated sensor data can be cleaned to obtain candidate sensor data. For example, a sliding window can be used for anomaly detection, and dynamic background filtering can be used to eliminate unnecessary signals, such as the vehicle's internal command transmission signals, as well as some abnormal signals. The sliding window is used to detect and eliminate abnormal signals in point cloud data. A fixed-size window is slid across the data, and statistics within the window (such as the mean, standard deviation, etc.) are calculated. These statistics are used to determine whether the current data point is abnormal. If a data point is abnormal, such as a noise point, an isolated point, or a spike signal, the abnormal data point is eliminated. Dynamic background filtering can be used to remove background signals and separate them from the target signal by analyzing the dynamic characteristics of the background.

[0053] Then, feature extraction is performed on the candidate sensor data to extract some point cloud-related features, such as spatial features, reflection features, environmental parameters, etc. The spatial features may include the volume calculation of the target bounding box, and the reflection features may include the radar reflection cross-sectional area and signal-to-noise ratio, so as to help the model simulate the point cloud and obtain more realistic simulated point cloud data.

[0054] In some possible implementations, the target point cloud simulation model may be obtained by the following steps: Based on the first actual sensor data, a basic point cloud simulation model is constructed; and the basic point cloud simulation model is trained through a plurality of sample sensor data to obtain a target point cloud simulation model.

[0055] In this embodiment, the first actual sensor data can be obtained by preprocessing the raw data collected by the actual sensor, and the second actual sensor data can also be obtained by preprocessing the raw data collected by the actual sensor. The preprocessing may include cleaning the raw data and extracting features. The specific method can refer to the preprocessing method of the simulated sensor data, which will not be repeated here. Before processing the raw data, the raw data can also be synchronized. Preferably, the raw data can be synchronized through PTP (Precision Time Protocol), including time alignment of the raw data and spatial coordinate system one, to facilitate subsequent processing.

[0056] A basic point cloud simulation model can be constructed in advance based on first actual sensor data acquired in an actual scenario. This basic point cloud simulation model is a neural network model with adjustable parameters related to the point cloud data. The basic point cloud simulation model is then trained using a plurality of labeled sample sensor data. This sample sensor data may include third actual sensor data, which may be data acquired in an actual scenario. Each piece of third actual sensor data is labeled with corresponding actual point cloud data, and the actual point cloud data is the accurate point cloud data corresponding to the third actual sensor data. The basic point cloud simulation model is trained using the third actual sensor data in an actual scenario and supervised learning is performed using the actual point cloud data, so that a target point cloud simulation model with more realistic and accurate output results can be obtained.

[0057] In some possible implementations, the method for constructing a basic point cloud simulation model may be: The first actual sensing data is processed to obtain point cloud distribution data and environmental impact parameters; based on the point cloud distribution data and the environmental impact parameters, a first function for the number of point clouds and a second function for the reflection intensity are constructed; and a basic point cloud simulation model is constructed based on the first function and the second function.

[0058] In this embodiment, the first actual sensor data can be processed to obtain point cloud distribution data and environmental impact parameters. The first actual sensor data may include spatial characteristics, reflection characteristics, and environmental parameters. The point cloud distribution data may include target object volume, detection distance, number of point clouds, and reflection intensity. Environmental impact parameters may include weather correction factors and basic noise terms. Alternatively, the point cloud distribution data can be obtained based on the first actual sensor data in conjunction with a target regression model. Alternatively, the point cloud distribution data can be obtained based on the first actual sensor data in conjunction with actual sensors. Environmental parameters may include visibility and road conditions.

[0059] After obtaining the point cloud distribution data and environmental impact parameters, the impact of each parameter on the number of point clouds and reflection intensity can be analyzed to construct a first function for the number of point clouds and a second function for the reflection intensity. Alternatively, a single variable control method can be used to compare the number of point clouds and reflection intensity at different distances for point clouds of the same volume, as well as the number of point clouds and reflection intensity at the same distance for point clouds of different volumes. By combining the impact of environmental impact parameters on the number of point clouds and adding some variable parameters, an empirical formula for the number of point clouds can be constructed, i.e., the first function for determining the number of point clouds.

[0060] Preferably, the expression of the first function may be:

[0061] in, is the number of point clouds, is the volume of the target object, is the measured calibration value, To detect the distance, is the distance attenuation coefficient, is the weather correction factor, is the basic noise term, and It is a variable parameter that can be adjusted and determined through model training.

[0062] And based on the reflection cross-sectional area, signal-to-noise ratio and reflection intensity values of different types of point clouds at different distances, and adding some variable parameters, an empirical formula for the reflection intensity can be constructed, that is, a second function for determining the reflection intensity.

[0063] Preferably, the expression of the second function may be:

[0064] in, is the reflection intensity, is the reflection cross-sectional area, is the target category benchmark value, is the shape parameter, and is a variable parameter that can be adjusted and determined through model training. The target categories may include vehicles and pedestrians. The value of can be calibrated by maximum likelihood estimation.

[0065] Furthermore, a neural network model including variable parameters can be constructed through the first function and the second function, so as to obtain a basic point cloud simulation model with sensor data as input and point cloud quantity and reflection intensity as output.

[0066] By processing the actual sensor data, we can obtain point cloud distribution data and environmental impact parameters related to the number of point clouds and the reflection intensity of the point clouds. Then, by analyzing the point cloud distribution data and environmental impact parameters, we can more comprehensively analyze the factors affecting the point cloud data, fit the first function for the number of point clouds and the second function for the reflection intensity, and further construct a basic point cloud simulation model, which can make the obtained basic point cloud simulation model closer to the real sensor.

[0067] In some possible implementations, the first actual sensing data includes characteristic parameters and environmental parameters; The first actual sensing data is processed to obtain point cloud distribution data and environmental impact parameters, including: The characteristic parameters are input into the target regression model to obtain point cloud distribution data; the environmental parameters are processed to obtain environmental impact parameters.

[0068] In this embodiment, point cloud distribution data can be derived through a target regression model. Preferably, the target regression model can be a hybrid regression model. Feature parameters can be input into the target regression model, which can then output the corresponding number of point clouds and reflection intensity. The point cloud distribution data is then derived by combining the feature parameters, number of point clouds, reflection intensity, and the sensor's detection range. Feature parameters can include spatial and reflection characteristics. Environmental parameters can include visibility and road conditions. These environmental parameters can be processed to analyze their impact on the number of point clouds and reflection intensity. For example, different weather conditions can correspond to different visibilities, and weather correction factors corresponding to these weather conditions can be determined. The weather correction factor can be positively correlated with visibility.

[0069] By processing the feature parameters through the target regression model, the corresponding point cloud data can be obtained. Combining these feature parameters, the overall point cloud distribution data can be obtained to facilitate point cloud data analysis. By obtaining the environmental impact parameters through the environmental parameters, the factors affecting the point cloud data can be made more comprehensive.

[0070] In some possible implementations, the plurality of sample sensor data include a plurality of first sample sensor data corresponding to a general scenario and a plurality of second sample sensor data corresponding to a target scenario; The basic point cloud simulation model is trained through multiple sample sensor data to obtain the target point cloud simulation model, including: The basic point cloud simulation model is trained using multiple first sample sensor data to obtain a general point cloud simulation model; a scene adaptation layer is added to the general point cloud simulation model to obtain a basic adaptation model; and the basic adaptation model is trained using multiple second sample sensor data to obtain a target point cloud simulation model corresponding to the target scene.

[0071] In this embodiment, a base point cloud simulation model can be pre-trained based on multiple first sample data from a common scenario so that the base point cloud simulation model can learn a wider range of features and patterns, thereby improving its generalization ability on unseen data or new tasks. Based on the trained common point cloud simulation model, a transfer learning method can be used, using the original structure of the common point cloud simulation model as a shared layer. A scene adaptation layer can be added to the common point cloud simulation model to obtain a base adaptation model. Preferably, the proportion of the shared layer in the base adaptation model can be between 80% and 90%, and the proportion of the scene adaptation layer can be between 10% and 20%. The base adaptation model can then be further trained using multiple second sample sensor data from a target scenario so that the base adaptation model learns the features of the point cloud data from the target scenario, thereby obtaining a target point cloud simulation model that better conforms to the target scenario. Preferably, the second sample sensor data can be 100 frames of data from the target scenario. Through the above-mentioned transfer learning method, the amount of sample data collected in the target scene can be reduced, and the model convergence can be achieved using less sample data in the target scene, which significantly reduces the time and computing resources required for training the new model and quickly obtains the target point cloud simulation model that meets the target scene.

[0072] In some possible implementations, training a basic point cloud simulation model using a plurality of first sample sensor data to obtain a universal point cloud simulation model includes: The basic point cloud simulation model is iteratively trained for multiple rounds using a plurality of first sample sensor data; after each round of training, the predicted point cloud data output by the current round of training is obtained; the prediction loss corresponding to the current round of training is determined based on the predicted point cloud data and the actual point cloud data corresponding to the current round of training; the basic point cloud simulation model is optimized based on the prediction loss corresponding to the current round of training; when the basic point cloud simulation model meets the preset stopping conditions, the training is stopped to obtain a universal point cloud simulation model.

[0073] In this embodiment, the first sample sensor data includes third actual sensor data and actual point cloud data corresponding to the third actual sensor data. Multiple rounds of iterative training can be performed on a basic point cloud simulation model using multiple first sample sensor data from common scenarios to obtain a common point cloud simulation model. For each round of iterative training, the third actual sensor data can be input into the basic point cloud simulation model to obtain corresponding predicted point cloud data output by the basic point cloud simulation model, thereby obtaining the predicted point cloud data corresponding to the current round of training. Furthermore, based on a target loss function, the prediction loss corresponding to the current round of training can be determined by comparing the difference between the predicted point cloud data and the actual point cloud data corresponding to the current round of training.

[0074] Preferably, the target loss function may be a cross-entropy loss function. Furthermore, based on the predicted point cloud data corresponding to the current round of training, the gradient descent algorithm may be used to calculate the optimized values corresponding to the variable parameters in the model, and the basic point cloud simulation model may be optimized based on the optimized values. The preset stopping condition may include model convergence. When the basic point cloud simulation model converges, training is stopped to obtain a universal point cloud simulation model. Alternatively, the preset stopping condition may include a preset number of rounds. When the number of training rounds reaches the preset number, training may be stopped to obtain a universal point cloud simulation model. The process of training the basic adaptation model using the second sample sensor data may refer to the above-mentioned training method and will not be described in detail here.

[0075] The basic point cloud simulation model is iteratively trained through multiple first sample sensor data, and the prediction loss is determined based on the gap between the output preset point cloud data and the actual point cloud data, so that the basic point cloud simulation model can be continuously optimized based on the prediction loss, so that the basic point cloud simulation model can continuously learn, and the output predicted point cloud data is closer to the actual point cloud data, thereby improving the authenticity of the point cloud data output by the obtained universal point cloud simulation model.

[0076] In some possible implementations, the method further includes: Periodically acquiring second actual sensing data; and updating the target point cloud simulation model according to the second actual sensing data.

[0077] In this embodiment, since the actual situation may be constantly changing, in order to ensure the accuracy of the target point cloud simulation model and make the target point cloud simulation model more consistent with the current actual situation, second actual sensor data can be periodically acquired. This allows the current target point cloud simulation model to be iteratively updated based on the second actual sensor data, allowing the target point cloud simulation model to adapt to changes in the real environment and making the point cloud data output by the target point cloud simulation model more realistic. The second actual sensor data may include spatial features, reflection features, and environmental parameters.

[0078] Optionally, the second actual sensing data may include multiple data in a general scenario, so that the shared layer of the target point cloud simulation model is updated through the second actual sensing data; or, the second actual sensing data may include multiple data in a target scenario, so that the scene adaptation layer of the target point cloud simulation model is updated through the second actual sensing data; or, the second actual sensing data may include multiple data in a general scenario and multiple data in a target scenario, so that the shared layer and scene adaptation layer of the target point cloud simulation model are updated through the second actual sensing data. This enables the target point cloud simulation model to adapt to changes in the real environment, making the point cloud data output by the target point cloud simulation model more realistic.

[0079] In some possible implementations, the second actual sensing data includes first sub-data and second sub-data.

[0080] The second actual sensing data is periodically acquired, including: periodically acquiring first sub-data according to a first cycle duration; and periodically acquiring second sub-data according to a second cycle duration, wherein the first cycle duration is shorter than the second cycle duration.

[0081] Updating the target point cloud simulation model according to the second actual sensing data includes: iteratively updating the target point cloud simulation model according to the first sub-data and the second sub-data respectively.

[0082] In this embodiment, the target point cloud simulation model can be updated periodically based on different cycle durations. Preferably, short-term updates and long-term updates can be performed, where the short-term updates correspond to a first cycle duration, and the long-term updates correspond to a second cycle duration. Preferably, the first cycle duration can be one hour, and the second cycle duration can be one week.

[0083] For short-term updates, first sub-data can be periodically acquired based on the first period duration. The first sub-data may include spatial features, reflection features, and environmental parameters, etc. The first sub-data may include multiple data in a general scenario, so that the shared layer of the target point cloud simulation model can be updated through the first sub-data; or the first sub-data may include multiple data in a target scenario, so that the scene adaptation layer of the target point cloud simulation model can be updated through the first sub-data; or the first sub-data may include multiple data in a general scenario and multiple data in a target scenario, so that the shared layer and the scene adaptation layer of the target point cloud simulation model can be updated through the first sub-data. For example, every hour, the sensor data collected by the actual sensor within this hour is acquired, and then the sensor data within this hour is used as a training sample to train the target point cloud simulation model to achieve the update of the target point cloud simulation model. The specific training method can refer to the training method in the above embodiment and will not be repeated here. Short-term updates can quickly respond to short-term changes in the actual environment so as to improve the prediction accuracy in the short term.

[0084] For long-term updates, second sub-data can be periodically acquired based on the second period duration. The second sub-data may include spatial features, reflection features, and environmental parameters, etc. The second sub-data may include multiple data in a general scenario, so that the shared layer of the target point cloud simulation model can be updated through the second sub-data; or, the second sub-data may include multiple data in a target scenario, so that the scene adaptation layer of the target point cloud simulation model can be updated through the second sub-data; or, the second sub-data may include multiple data in a general scenario and multiple data in a target scenario, so that the shared layer and the scene adaptation layer of the target point cloud simulation model can be updated through the second sub-data. For example, every week, the sensor data collected by the actual sensor within this week is acquired, and then the sensor data within this week is used as a training sample to train the target point cloud simulation model, so as to achieve the update of the target point cloud simulation model. The specific training method can refer to the training method in the above embodiment and will not be repeated here. Long-term updates can ensure the consistency of long-term trends and improve the generalization ability of the target point cloud simulation model.

[0085] After obtaining the target point cloud simulation model, it can be tested using test data, based on static and dynamic indicators. The target point cloud simulation model outputs test point cloud data, such as the number of output point clouds and reflection intensity, which can be compared and verified with the actual number of point clouds and reflection intensity in the test data. The point cloud density error is less than 15%. The target point cloud simulation model also outputs target classification, with an accuracy rate exceeding 92%. The simulated point cloud data output by the target point cloud simulation model is verified using multi-radar cross-validation, showing a target spatial overlap greater than 85%, confirming that the target point cloud simulation model has high accuracy and generalization capabilities.

[0086] Figure 3 This is a flow chart of a system architecture for obtaining a target point cloud simulation model according to an exemplary embodiment. Figure 3As shown, sensor data can be collected in advance based on a variety of actual sensors installed on the actual vehicle. These sensors may include 4D millimeter-wave radar, cameras, lidar, global navigation satellite systems, and inertial measurement units. The collected raw data can then be synchronized via PTP through a synchronization control module, including time alignment and spatial coordinate system alignment for subsequent processing. The collected raw data is then preprocessed, including data cleaning, such as point cloud denoising, outlier filtering, and format standardization. The target feature extraction engine further performs feature extraction, calculating the volume of the target bounding box, radar cross-sectional area, and signal-to-noise ratio to obtain spatial features, reflection features, and environmental parameters. The environmental parameter calculation module then processes the environmental parameters to obtain environmental impact parameters, which may include weather correction factors and basic noise terms. A basic point cloud simulation model can be constructed using a hybrid regression model trainer and trained using a parameter optimization engine to obtain a target point cloud simulation model, namely a 4D point cloud generator, which can then be used to simulate and generate simulated point cloud data. After obtaining the target point cloud simulation model, it can be verified. The multi-radar collaborative control module can be used for multi-radar cross-validation, including time synchronization and spatial consistency verification. The generated simulated point cloud data can be processed by the data post-processing module, including format conversion, accuracy calibration, and logging. The verification module can then be used to perform simulation tests based on the simulated point cloud data.

[0087] Figure 4 FIG. 1 is a block diagram of a point cloud simulation device according to an exemplary embodiment. Figure 4 The point cloud simulation device 400 may include a first acquisition module 401 and a simulation module 402.

[0088] A first acquisition module 401 is configured to acquire target sensor data; The simulation module 402 is configured to process the target sensor data through a target point cloud simulation model to obtain simulated point cloud data. The target point cloud simulation model is obtained by training a basic point cloud simulation model based on multiple sample sensor data. The basic point cloud simulation model is constructed based on the first actual sensor data. The sample sensor data and the first actual sensor data are both collected by actual sensors.

[0089] In some possible implementations, the simulated point cloud data is used to construct a scene database for assisted driving.

[0090] In some possible implementations, the first acquisition module 401 includes: A first acquisition submodule is configured to acquire simulated sensor data in a target scene; The preprocessing submodule is configured to preprocess the simulated sensor data to obtain the target sensor data; wherein the preprocessing includes cleaning and feature extraction of the simulated sensor data.

[0091] In some possible implementations, the point cloud simulation device 400 further includes: an acquisition submodule, configured to process the first actual sensing data to obtain point cloud distribution data and environmental impact parameters; a construction submodule configured to construct a first function for the number of point clouds and a second function for the reflection intensity based on the point cloud distribution data and the environmental impact parameter; The construction submodule is further configured to construct the basic point cloud simulation model according to the first function and the second function.

[0092] In some possible implementations, the first actual sensing data includes characteristic parameters and environmental parameters; The obtaining submodule includes: a first obtaining unit configured to input the feature parameters into a target regression model to obtain the point cloud distribution data; The first obtaining unit is further configured to process the environmental parameters to obtain the environmental impact parameters.

[0093] In some possible implementations, the point cloud simulation device 400 further includes: a second acquisition module, configured to periodically acquire second actual sensing data; An updating module is configured to update the target point cloud simulation model according to the second actual sensing data.

[0094] In some possible implementations, the second actual sensing data includes first sub-data and second sub-data; The second acquisition module includes: A second acquisition submodule is configured to periodically acquire the first sub-data according to a first period duration; The second acquisition submodule is further configured to periodically acquire the second sub-data according to a second cycle duration, where the first cycle duration is shorter than the second cycle duration; The update module includes: The updating submodule is configured to iteratively update the target point cloud simulation model according to the first sub-data and the second sub-data respectively.

[0095] Regarding the point cloud simulation device 400 in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0096] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which implement the steps of the point cloud simulation method provided by the present disclosure when the program instructions are executed by a processor.

[0097] Figure 5 FIG. 5 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 500 may be a simulation device or a server.

[0098] Reference Figure 5 , the electronic device 500 may include one or more of the following components: a first processing component 502 , a first memory 504 , a first power supply component 506 , a multimedia component 508 , an audio component 510 , a first input / output interface 512 , a sensor component 514 , and a communication component 516 .

[0099] The first processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The first processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-described point cloud simulation method. In addition, the first processing component 502 may include one or more modules to facilitate interaction between the first processing component 502 and other components. For example, the first processing component 502 may include a multimedia module to facilitate interaction between the multimedia component 508 and the first processing component 502.

[0100] The first memory 504 is configured to store various types of data to support operations on the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, videos, etc. The first memory 504 can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0101] The first power supply assembly 506 provides power to the various components of the electronic device 500. The first power supply assembly 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0102] The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, it may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When the electronic device 500 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have a variable focal length and optical zoom capability.

[0103] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in the first memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0104] The first input / output interface 512 provides an interface between the first processing component 502 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0105] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500. The sensor assembly 514 can also detect changes in the position of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and temperature changes of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0106] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0107] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned point cloud simulation method.

[0108] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is further provided, such as a first memory 504 including instructions. The instructions are executable by the processor 520 of the electronic device 500 to perform the above-described point cloud simulation method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0109] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for performing the above-mentioned point cloud simulation method when executed by the programmable device.

[0110] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented through electronic hardware, computer software, or a combination of both. Whether such functions are implemented through hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0111] In the above detailed description, terms such as "center," "upper," "lower," "left," and "right" indicate directions or positional relationships. Since the components of the described devices can be positioned in a variety of different orientations, the directional terms are used for illustrative purposes and are not intended to be limiting. It should be understood that other aspects may be utilized and structural or logical changes may be made without departing from the concepts of the present disclosure. Therefore, the following detailed description should not be considered in a limiting sense.

[0112] It will be understood that the features of the various embodiments of the present disclosure described herein may be combined with each other unless specifically stated otherwise.

[0113] Although terms such as "first", "second" and "third" may be used herein to describe various components, parts, regions, layers or sections, these components, parts, regions, layers or sections are not limited to these terms. On the contrary, these terms are only used to distinguish one component, part, region, layer or section from another component, part, region, layer or section. Therefore, without departing from the teachings of each example, the first component, part, region, layer or section mentioned in the examples described herein may also be referred to as the second component, part, region, layer or section. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one such feature. In the description herein, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise clearly and specifically defined.

[0114] Furthermore, the word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word exemplary is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies to A or B" is intended to mean any of the natural inclusive permutations. That is, if X applies to A; X applies to B; or X applies to both A and B, then "X applies to A or B" satisfies any of the aforementioned instances. Furthermore, the articles "a" and "an," as used in this application and the appended claims, are generally understood to mean "one or more," unless otherwise specified or clear from the context to refer to the singular form.

[0115] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. With particular regard to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if not structurally equivalent to the disclosed structure. In addition, although particular features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "include," "have," "have," "have," or variations thereof are used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term "comprising."

[0116] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0117] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A point cloud simulation method, characterized in that: include: Acquire target sensor data; The target sensor data is processed by a target point cloud simulation model to obtain simulated point cloud data. The target point cloud simulation model is obtained by training a basic point cloud simulation model based on multiple sample sensor data. The basic point cloud simulation model is constructed based on first actual sensor data. The sample sensor data and the first actual sensor data are both collected by actual sensors.

2. The point cloud simulation method according to claim 1, characterized in that: The simulated point cloud data is used to construct a scene database for assisted driving.

3. The point cloud simulation method according to claim 1, characterized in that: The acquiring target sensor data includes: Acquire simulated sensor data in the target scenario; The simulated sensor data is preprocessed to obtain the target sensor data; wherein the preprocessing includes cleaning and feature extraction of the simulated sensor data.

4. The point cloud simulation method according to claim 1, characterized in that: The method further comprises: Processing the first actual sensing data to obtain point cloud distribution data and environmental impact parameters; constructing a first function for the number of point clouds and a second function for the reflection intensity based on the point cloud distribution data and the environmental impact parameter; The basic point cloud simulation model is constructed according to the first function and the second function.

5. The point cloud simulation method according to claim 4, characterized in that: The first actual sensing data includes characteristic parameters and environmental parameters; The processing of the first actual sensing data to obtain point cloud distribution data and environmental impact parameters includes: Inputting the characteristic parameters into a target regression model to obtain the point cloud distribution data; The environmental parameters are processed to obtain the environmental impact parameters.

6. The point cloud simulation method according to any one of claims 1 to 5, characterized in that: The method further comprises: Periodically acquiring second actual sensing data; The target point cloud simulation model is updated according to the second actual sensing data.

7. The point cloud simulation method according to claim 6, characterized in that: The second actual sensing data includes first sub-data and second sub-data; The periodically acquiring the second actual sensing data includes: periodically acquiring the first sub-data according to a first period duration; periodically acquiring the second sub-data according to a second cycle duration, wherein the first cycle duration is shorter than the second cycle duration; The updating of the target point cloud simulation model according to the second actual sensing data includes: The target point cloud simulation model is iteratively updated according to the first sub-data and the second sub-data respectively.

8. A point cloud simulation device, characterized in that: include: A first acquisition module is configured to acquire target sensor data; The simulation module is configured to process the target sensor data through a target point cloud simulation model to obtain simulated point cloud data. The target point cloud simulation model is obtained by training a basic point cloud simulation model based on multiple sample sensor data. The basic point cloud simulation model is constructed based on first actual sensor data. The sample sensor data and the first actual sensor data are both collected by actual sensors.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the steps of the point cloud simulation method described in any one of claims 1 to 7 when executed.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the point cloud simulation method described in any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The method comprises a computer program which, when executed by a processor, implements the steps of the point cloud simulation method according to any one of claims 1 to 7.

Images