Laser radar point cloud data simulation method, system, electronic device, and storage medium

Abstract

A method, system, electronic device, and storage medium for simulating laser radar point cloud data, the method comprising: providing a first input set to a generating network, the generating network comprising at least an intermediate feature layer, the intermediate feature layer being calculated based on a previous second input set; comparing the similarity of each first input in the first input set with the corresponding second input in the second input set; for a first input whose similarity exceeds a predetermined threshold, obtaining a second parameter corresponding to the second input in the intermediate feature layer as a first parameter corresponding to the first input in the intermediate feature layer; for a first input whose similarity does not exceed the predetermined threshold, calculating a third parameter corresponding to the first input in the intermediate feature layer based on the first input; the generating network outputs a first simulated point cloud. The present invention reduces the amount of calculation of the generating network for partially similar inputs and improves the synthesis efficiency of the generating network.

Description

Technical Field

[0001] The present invention relates to the field of point cloud simulation technology, and in particular to a laser radar point cloud data simulation method, system, electronic device, and storage medium. Background Art

[0002] At present, autonomous vehicles are developing rapidly. A safe autonomous driving system requires a massive test of 1 billion kilometers. The existing autonomous driving verification adopts two methods: real vehicle verification or simulation environment verification. It is more practical to test and evaluate through simulation system. Current simulation software mostly simulates the traffic environment by using game engines or high-fidelity computer graphics. The dynamic forms and dynamic responses of traffic participants rendered by simulation are relatively rigid and have low realism. At the same time, it is often impossible to achieve real-time simulation due to the large consumption of rendering resources for complex scenes. Therefore, the traffic scenes are not rich enough, and the generalization ability of the algorithm cannot be effectively verified.

[0003] There are limitations in using real vehicle sensor data as a data source to verify the functions of the perception module. It is only suitable for the verification of a single perception scene. When the verification object includes perception planning control, the real vehicle sensor data cannot match the scene changes caused by the changed system actions and cannot be used for fast closed-loop verification.

[0004] The sensor data simulated by the simulator is established completely in accordance with the ideal sensor physical principles, which deviates from the real data. Directly using it as sensor data input cannot fully verify the performance of the perception module in the real environment. In addition, due to the computing power limitations of the simulation system, the simulation point cloud generation in the existing technology is not timely enough and cannot achieve real-time simulation. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a laser radar point cloud data simulation method, system, electronic device, and storage medium.

[0006] The present invention provides a laser radar point cloud data simulation method, the method comprising:

[0007] Providing a first input set to a generative network, the generative network comprising at least an intermediate feature layer, the intermediate feature layer being calculated based on a previous second input set;

[0008] comparing the similarity of each first input in the first input set with the corresponding second input in the second input set;

[0009] For a first input whose similarity exceeds a predetermined threshold, obtaining a second parameter corresponding to the second input in the intermediate feature layer as a first parameter corresponding to the first input in the intermediate feature layer; for a first input whose similarity does not exceed the predetermined threshold, calculating a third parameter corresponding to the first input in the intermediate feature layer based on the first input;

[0010] Based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer, the generation network outputs a first simulation point cloud.

[0011] According to a laser radar point cloud data simulation method provided by the present invention, the first input set and the second input set are respectively used to generate two adjacent frames of laser radar point cloud data.

[0012] According to a laser radar point cloud data simulation method provided by the present invention, based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer, the generation network outputs a first simulation point cloud, and then includes:

[0013] Based on the first simulated point cloud, the noise collected by the real laser radar is superimposed to generate a second simulated point cloud.

[0014] According to a laser radar point cloud data simulation method provided by the present invention, the noise includes at least one of the following:

[0015] Weather noise, light intensity noise, and noise from dynamic traffic participants.

[0016] According to a laser radar point cloud data simulation method provided by the present invention, the training method of the generation network includes:

[0017] Get real LiDAR point cloud;

[0018] Input the real laser radar point cloud and the second simulated point cloud into a discriminant network to determine true or false;

[0019] The discriminative network and the generative network are optimized alternately.

[0020] According to a laser radar point cloud data simulation method provided by the present invention, the laser radar point cloud data is point cloud data acquired based on the detection of the external traffic environment by a vehicle-mounted laser radar in an autonomous driving scenario.

[0021] The present invention also provides a laser radar point cloud data simulation system, the system comprising:

[0022] Providing a module for providing a first input set to a generative network, the generative network comprising at least an intermediate feature layer, the intermediate feature layer being calculated based on a previous second input set;

[0023] a comparison module, configured to compare the similarity of each first input in the first input set with the corresponding second input in the second input set;

[0024] a sharing module, configured to obtain, for a first input whose similarity exceeds a predetermined threshold, a second parameter corresponding to the second input in the intermediate feature layer as a first parameter corresponding to the first input in the intermediate feature layer; and, for a first input whose similarity does not exceed the predetermined threshold, calculate, based on the first input, a third parameter corresponding to the first input in the intermediate feature layer;

[0025] The output module is used for generating the network to output a first simulation point cloud based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer.

[0026] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the laser radar point cloud data simulation method as described in any one of the above items are implemented.

[0027] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the laser radar point cloud data simulation method as described in any one of the above items are implemented.

[0028] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the laser radar point cloud data simulation method as described in any one of the above items.

[0029] The laser radar point cloud data simulation method, system, electronic device, and storage medium provided by the present invention reduce the amount of calculation of the generation network for some similar input parameters, improve the synthesis efficiency of the generation network, realize real-time simulation, and are applied to closed-loop simulation of multiple modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 A schematic diagram of a flow chart of a laser radar point cloud data simulation method provided by the present invention;

[0032] Figure 2A schematic diagram of the structure of a discriminant network and a generative network provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of the structure of a discriminant network provided by an embodiment of the present invention;

[0034] Figure 4 A schematic diagram of the structure of a laser radar point cloud data simulation system provided by the present invention;

[0035] Figure 5 A schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The following is a detailed description of the laser radar point cloud data simulation method provided in the embodiment of the present application through specific embodiments and their application scenarios in combination with the accompanying drawings.

[0038] Figure 1 A schematic diagram of a flow chart of a laser radar point cloud data simulation method provided by the present invention, such as Figure 1 As shown, the present invention provides a laser radar point cloud data simulation method, which may include the following steps.

[0039] S100: providing a first input set to a generation network, where the generation network includes at least an intermediate feature layer, and the intermediate feature layer is calculated based on a previous second input set.

[0040] S200 : Compare the similarity between each first input in the first input set and the corresponding second input in the second input set.

[0041] S300. For a first input whose similarity exceeds a predetermined threshold, obtain a second parameter corresponding to the second input in the intermediate feature layer as the first parameter corresponding to the first input in the intermediate feature layer; for a first input whose similarity does not exceed a predetermined threshold, calculate a third parameter corresponding to the first input in the intermediate feature layer based on the first input.

[0042] S400: Generate a network output first simulation point cloud based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer.

[0043] Optionally, the generation network adopts a deep convolutional neural network modeling method to output laser echo distance and light intensity information.

[0044] Optionally, the latent variable is used as an input of the generative network to generate the simulated laser point cloud light intensity and spot distribution under clear and normal light intensity. Further, the latent variable is random N-dimensional data ranging from -1 to 1.

[0045] Optionally, the generation network includes processes such as noise input, transposed convolution, upsampling, and generation of feature point cloud. Noise input refers to generating a random noise matrix such as 16*16*1 based on latent variables, using transposed convolution to complete upsampling, and outputting a matrix such as 32*32*1. Based on the transposed convolution, a secondary transposed convolution is performed to output simulated point cloud data such as 128*128*1. It should be noted that the aforementioned matrix dimensions are only for exemplary reference and are not intended to be limiting.

[0046] This embodiment reduces the amount of calculation of the generation network for some similar inputs, improves the synthesis efficiency of the generation network, realizes real-time simulation, and is applied to closed-loop simulation of multiple modules.

[0047] Further, based on the aforementioned embodiment, in another embodiment, this embodiment provides a laser radar point cloud data simulation method, and the first input set and the second input set are respectively used to generate two adjacent frames of laser radar point cloud data.

[0048] Optionally, in two adjacent frames of simulated point clouds, the traffic environment changes only slightly, and there are some similar input and convolution operations, so there are also a large number of similar values ​​in the intermediate feature layers, and some intermediate feature layer parameters are shared without affecting the performance of the generated network.

[0049] This embodiment shares some intermediate layer parameters in two adjacent frames of simulation point clouds, reduces input parameters, reduces the amount of calculation, and speeds up the generation time of the laser point cloud. The generation time trecon of the laser point cloud is less than the sampling time tsample of the laser point cloud (100ms, laser 10Hz sampling), thereby realizing real-time simulation of the lidar point cloud.

[0050] Further, based on the foregoing embodiment, in another embodiment, this embodiment provides a laser radar point cloud data simulation method, which generates a network output first simulation point cloud based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer, and then includes:

[0051] Based on the first simulated point cloud, the noise collected by the real laser radar is superimposed to generate a second simulated point cloud.

[0052] Optionally, the noise includes at least one of the following:

[0053] Weather noise, light intensity noise, and noise from dynamic traffic participants.

[0054] Optionally, a real road scene is selected to collect laser point cloud data, that is, distance and light intensity data of the laser point cloud.

[0055] Optionally, refer to the product data sheet of the simulated laser radar model to obtain the laser point cloud detection range and point cloud distribution.

[0056] Optionally, the laser point cloud of the laser radar under different weather conditions, such as cloudy days, rainy days, foggy days, snowy days, etc., is obtained, and the noise distribution of the laser point cloud under different weather conditions is obtained by calculating the error compared with sunny days.

[0057] Optionally, obtain the laser radar in different light intensity environments, such as daytime, night, strong light, and backlight environments, and obtain the noise distribution of the laser point cloud under different lighting conditions by calculating the error compared with normal lighting conditions.

[0058] Optionally, the laser point cloud reflection distribution of different traffic participants, such as cars, vans, vans, trucks, container trucks, flatbed trucks, etc., at different incident angles is obtained to form point cloud models of different traffic participants.

[0059] In this embodiment, noise is superimposed on the first simulation point cloud, so that the fidelity of the second simulation point cloud is greatly improved.

[0060] Further, based on the above-mentioned embodiment, in another embodiment, this embodiment provides a laser radar point cloud data simulation method, and the training method of generating a network includes:

[0061] Get real LiDAR point cloud;

[0062] The real laser radar point cloud and the second simulated point cloud are input into the discriminant network to determine whether they are true or false;

[0063] Alternately optimize the discriminative network and the generative network.

[0064] Optionally, Figure 2 A schematic diagram of the structure of a discriminant network and a generator network provided by an embodiment of the present invention, such as Figure 2 As shown, the adversarial network model formed by the discriminant network and the generative network includes a generative network, a discriminant network and a loss function. The first simulated point cloud output by the generative network is superimposed with the weather noise, light intensity noise and dynamic traffic participant noise collected by the real lidar to form a second simulated point cloud. The second simulated point cloud serves as the input of the adversarial network model.

[0065] Optionally, the generation model of dynamic traffic participant noise synthesizes different laser light intensities and point cloud distributions according to the real-time positions and postures of traffic participants.

[0066] Optionally, the real lidar point cloud and the second simulated point cloud are sampled and input into the discriminant network. The goal of the discriminant network is to output the discrimination result of the second simulated point cloud: is it real point cloud data or simulated point cloud data? During the training process of the generative network, the discriminant network is deceived as much as possible, while the discriminant network makes correct judgments on the point cloud data as much as possible. If the judgment result is false, the weight of the generative network is updated to optimize the generative network to increase the classification error of the discriminant network until it judges to be true. If the judgment result is true, the weight of the discriminant network is updated to optimize the discriminant network to reduce the classification error of the discriminant network until it judges to be false. Alternately optimize the discriminant network and the generative network to optimize the loss function. Furthermore, keep the probability of the discriminant network judging the authenticity of any sample at about 50%.

[0067] Optionally, Figure 3 A schematic diagram of a discriminant network structure provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown in the figure, the discriminant network includes the processes of real laser point cloud input, convolution, pooled downsampling and feature extraction. The discriminant network is constructed using a convolutional neural network. The input dimension is consistent with the output dimension of the generating network, such as 128*128*1. The output is the judgment result of the authenticity of the simulated point cloud, expressed as a probability between 0 and 1. The discriminant network completes the dimensionality reduction of the point cloud and the fusion of the feature dimension through convolution, pooled downsampling and convolution, and finally completes the feature generation output of the classification result through the fully connected layer.

[0068] This embodiment discloses that a discriminant network is relied upon to optimize a generative network, and through the confrontation between the two networks of a zero-sum game, a high-fidelity simulation point cloud output of the generative network is finally achieved.

[0069] Furthermore, based on the foregoing embodiment, in another embodiment, this embodiment provides a method for simulating laser radar point cloud data, wherein the laser radar point cloud data is point cloud data acquired based on the detection of the external traffic environment by a vehicle-mounted laser radar in an autonomous driving scenario.

[0070] This embodiment can make the data of the simulated sensor close to the real sensor data, thereby improving the accuracy of simulation verification; it can enrich the laser point cloud test scenarios, realize the functions of laser point cloud generation and editing, and achieve the same effect as real data, which can greatly save annotation costs and improve model training efficiency; it can be applied to real-time hardware-in-the-loop simulation systems to improve the effect of autonomous driving system verification and improve the efficiency of autonomous driving system verification.

[0071] The laser radar point cloud data simulation system provided by the present invention is described below. The laser radar point cloud data simulation system described below and the laser radar point cloud data simulation method described above can be referenced to each other.

[0072] Figure 4 A schematic diagram of the structure of a laser radar point cloud data simulation system provided by the present invention is shown in FIG. Figure 4 As shown, the present invention also provides a laser radar point cloud data simulation system, the system comprising:

[0073] Providing a module for providing a first input set to a generative network, the generative network comprising at least an intermediate feature layer, the intermediate feature layer being calculated based on a previous second input set;

[0074] A comparison module, used to compare the similarity of each first input in the first input set with the corresponding second input in the second input set;

[0075] A sharing module is used to obtain, for a first input whose similarity exceeds a predetermined threshold, a second parameter corresponding to the second input in the intermediate feature layer as a first parameter corresponding to the first input in the intermediate feature layer; for a first input whose similarity does not exceed the predetermined threshold, calculate a third parameter corresponding to the first input in the intermediate feature layer based on the first input;

[0076] The output module is used to generate a network output first simulation point cloud based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer.

[0077] This embodiment reduces the amount of calculation of the generated network for some similar input parameters, improves the synthesis efficiency of the generated network, realizes real-time simulation, and is applied to closed-loop simulation of multiple modules.

[0078] Figure 5 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the laser radar point cloud data simulation method, and the method includes:

[0079] Providing a first input set to a generative network, the generative network comprising at least an intermediate feature layer, the intermediate feature layer being calculated based on a previous second input set;

[0080] comparing the similarity of each first input in the first input set with the corresponding second input in the second input set;

[0081] For a first input whose similarity exceeds a predetermined threshold, obtaining a second parameter corresponding to the second input in the intermediate feature layer as a first parameter corresponding to the first input in the intermediate feature layer; for a first input whose similarity does not exceed the predetermined threshold, calculating a third parameter corresponding to the first input in the intermediate feature layer based on the first input;

[0082] Based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer, the generation network outputs a first simulation point cloud.

[0083] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0084] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute the laser radar point cloud data simulation method provided by the above methods, the method comprising:

[0085] Providing a first input set to a generative network, the generative network comprising at least an intermediate feature layer, the intermediate feature layer being calculated based on a previous second input set;

[0086] comparing the similarity of each first input in the first input set with the corresponding second input in the second input set;

[0087] For a first input whose similarity exceeds a predetermined threshold, obtaining a second parameter corresponding to the second input in the intermediate feature layer as a first parameter corresponding to the first input in the intermediate feature layer; for a first input whose similarity does not exceed the predetermined threshold, calculating a third parameter corresponding to the first input in the intermediate feature layer based on the first input;

[0088] Based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer, the generation network outputs a first simulation point cloud.

[0089] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned laser radar point cloud data simulation method, the method comprising:

[0090] Providing a first input set to a generative network, the generative network comprising at least an intermediate feature layer, the intermediate feature layer being calculated based on a previous second input set;

[0091] comparing the similarity of each first input in the first input set with the corresponding second input in the second input set;

[0092] For a first input whose similarity exceeds a predetermined threshold, obtaining a second parameter corresponding to the second input in the intermediate feature layer as a first parameter corresponding to the first input in the intermediate feature layer; for a first input whose similarity does not exceed the predetermined threshold, calculating a third parameter corresponding to the first input in the intermediate feature layer based on the first input;

[0093] Based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer, the generation network outputs a first simulation point cloud.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A laser radar point cloud data simulation method, characterized in that: The method comprises: Providing a first input set to a generative network, the generative network comprising at least an intermediate feature layer, the intermediate feature layer being calculated based on a previous second input set; comparing the similarity of each first input in the first input set with the corresponding second input in the second input set; For a first input whose similarity exceeds a predetermined threshold, obtaining a second parameter corresponding to the second input in the intermediate feature layer as a first parameter corresponding to the first input in the intermediate feature layer; for a first input whose similarity does not exceed the predetermined threshold, calculating a third parameter corresponding to the first input in the intermediate feature layer based on the first input; Based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer, the generation network outputs a first simulation point cloud.

2. The laser radar point cloud data simulation method according to claim 1, characterized in that: The first input set and the second input set are respectively used to generate two adjacent frames of lidar point cloud data.

3. The laser radar point cloud data simulation method according to claim 1, characterized in that: Based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer, the generating network outputs a first simulation point cloud, and then comprises: Based on the first simulated point cloud, the noise collected by the real laser radar is superimposed to generate a second simulated point cloud.

4. The laser radar point cloud data simulation method according to claim 3, characterized in that: The noise includes at least one of the following: Weather noise, light intensity noise, and noise from dynamic traffic participants.

5. The laser radar point cloud data simulation method according to claim 3, characterized in that: The training method of the generation network includes: Get real LiDAR point cloud; Input the real laser radar point cloud and the second simulated point cloud into a discriminant network to determine true or false; The discriminative network and the generative network are optimized alternately.

6. The laser radar point cloud data simulation method according to any one of claims 1 to 5, characterized in that: The laser radar point cloud data is point cloud data obtained based on the detection of the external traffic environment by the vehicle-mounted laser radar in the autonomous driving scenario.

7. A laser radar point cloud data simulation system, characterized in that: The system comprises: Providing a module for providing a first input set to a generative network, the generative network comprising at least an intermediate feature layer, the intermediate feature layer being calculated based on a previous second input set; a comparison module, configured to compare the similarity of each first input in the first input set with the corresponding second input in the second input set; a sharing module, configured to obtain, for a first input whose similarity exceeds a predetermined threshold, a second parameter corresponding to the second input in the intermediate feature layer as a first parameter corresponding to the first input in the intermediate feature layer; and, for a first input whose similarity does not exceed the predetermined threshold, calculate, based on the first input, a third parameter corresponding to the first input in the intermediate feature layer; The output module is used for generating the network to output a first simulation point cloud based on the first parameter of the intermediate feature layer and the third parameter of the intermediate feature layer.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the laser radar point cloud data simulation method as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the laser radar point cloud data simulation method as described in any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the laser radar point cloud data simulation method as described in any one of claims 1 to 6 are implemented.

Images