Laser radar simulation method and device, equipment and storage medium

Abstract

The invention provides a laser radar simulation method, device and equipment and a storage medium, and relates to the technical field of computers, in particular to the technical field of automatic driving, automatic driving simulation and laser radar. The method comprises the following steps: acquiring a scene image, and position information and object labeling information of a simulation point cloud corresponding to the scene image; based on the scene image, performing first prediction on the reflection intensity and the light falling phenomenon of each point in the simulation point cloud; furthermore, on the basis of the position information and the object labeling information, second prediction is performed on the reflection intensity of each point, corresponding to the first object, in the simulation point cloud; and determining the intensity information and the light falling phenomenon of the simulation point cloud according to the first prediction result and the second prediction result. According to the invention, the authenticity of laser radar simulation can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of autonomous driving, autonomous driving simulation, and laser radar technology. Background Art

[0002] Autonomous driving simulation is crucial in the development of autonomous driving technology. It allows engineers to test and verify autonomous driving systems in a safe and controllable virtual environment, thereby identifying and correcting potential problems, optimizing algorithms, and significantly reducing the risks and costs of actual road testing, accelerating the process of technology development, and improving the safety and reliability of autonomous driving systems. Among them, LiDAR simulation is an important part of the autonomous driving simulation system. Currently, most LiDAR simulation algorithms cannot simulate the reflection intensity and raydrop phenomenon of real radars, which greatly affects the authenticity of LiDAR simulation. Summary of the invention

[0003] The present disclosure provides a laser radar simulation method, device, equipment and storage medium.

[0004] According to one aspect of the present disclosure, a laser radar simulation method is provided, comprising:

[0005] Obtaining a scene image, and location information and object annotation information of a simulation point cloud corresponding to the scene image;

[0006] Based on the scene image, a first prediction is made of the reflection intensity and light fallback phenomenon of each point in the simulated point cloud; and based on the position information and the object annotation information, a second prediction is made of the reflection intensity of each point in the simulated point cloud corresponding to the first object;

[0007] According to the result of the first prediction and the result of the second prediction, the intensity information and the light fallback phenomenon of the simulated point cloud are determined.

[0008] According to another aspect of the present disclosure, a laser radar simulation method is provided, comprising:

[0009] The scene image is input into a pre-trained prediction model, and the prediction model outputs the prediction results of the reflection intensity of each point in the simulation point cloud and the prediction results of the light fallback phenomenon;

[0010] The prediction result of the light fall-off phenomenon includes whether the light fall-off phenomenon exists at a point or whether the light fall-off phenomenon does not exist at a point.

[0011] According to another aspect of the present disclosure, a prediction model training method is provided, comprising:

[0012] Determine scene image samples and point cloud samples;

[0013] The scene image sample is input into the prediction model to be trained, and the prediction model to be trained outputs the prediction result of the reflection intensity of each point in the simulation point cloud and the prediction result of the light fallback phenomenon; wherein the prediction result of the light fallback phenomenon includes whether the point has the light fallback phenomenon or does not have the light fallback phenomenon;

[0014] The loss function is calculated using the predicted results of the reflection intensity of each point, the predicted results of the light fallback phenomenon, and the point cloud samples;

[0015] The parameters of the prediction model to be trained are adjusted based on the loss function.

[0016] According to another aspect of the present disclosure, a laser radar simulation method is provided, comprising:

[0017] Acquire location information and object annotation information of the simulation point cloud; the object annotation information includes the object and object material corresponding to the point in the simulation point cloud; the object corresponds to at least one object material;

[0018] Based on the object annotation information, determine a point in the simulation point cloud corresponding to the first object, and determine an object material of the first object corresponding to the point;

[0019] Find the expression corresponding to the first object and the object material;

[0020] Based on the position information of the point corresponding to the first object and the expression, the reflection intensity of the point corresponding to the first object is predicted.

[0021] According to another aspect of the present disclosure, a laser radar simulation device is provided, comprising:

[0022] A first acquisition module is used to acquire a scene image, and location information and object annotation information of a simulation point cloud corresponding to the scene image;

[0023] A first prediction module, used for making a first prediction of the reflection intensity and light fallback phenomenon of each point in the simulation point cloud based on the scene image;

[0024] A second prediction module, configured to perform a second prediction of the reflection intensity of each point corresponding to the first object in the simulation point cloud based on the position information and the object labeling information;

[0025] The first determination module is used to determine the intensity information and the light fallback phenomenon of the simulation point cloud according to the result of the first prediction and the result of the second prediction.

[0026] According to another aspect of the present disclosure, a laser radar simulation device is provided, comprising:

[0027] A first input module is used to input a scene image into a pre-trained prediction model, and the prediction model outputs a prediction result of the reflection intensity of each point in the simulation point cloud and a prediction result of the light fallback phenomenon;

[0028] The prediction result of the light fall-off phenomenon includes whether the light fall-off phenomenon exists at a point or whether the light fall-off phenomenon does not exist at a point.

[0029] According to another aspect of the present disclosure, a prediction model training device is provided, comprising:

[0030] A third determination module is used to determine scene image samples and point cloud samples;

[0031] The second input module is used to input the scene image sample into the prediction model to be trained, and the prediction model to be trained outputs the prediction result of the reflection intensity of each point in the simulation point cloud and the prediction result of the light fallback phenomenon; wherein the prediction result of the light fallback phenomenon includes whether the point has the light fallback phenomenon or does not have the light fallback phenomenon;

[0032] The adjustment module is used to calculate the loss function using the prediction results of the reflection intensity of each point and the prediction results of the light fallback phenomenon, as well as the point cloud samples; and adjust the parameters of the prediction model to be trained based on the loss function.

[0033] According to another aspect of the present disclosure, a laser radar simulation device is provided, comprising:

[0034] The second acquisition module is used to acquire the position information and object annotation information of the simulation point cloud; the object annotation information includes the object and the object material corresponding to the point in the simulation point cloud; the object corresponds to at least one object material;

[0035] A fourth determination module, used to determine a point in the simulation point cloud corresponding to the first object based on the object annotation information, and determine an object material of the first object corresponding to the point;

[0036] The third prediction module is used to find the expression corresponding to the first object and the object material; and predict the reflection intensity of the point corresponding to the first object according to the position information and the expression of the point corresponding to the first object.

[0037] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0038] at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0041] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0042] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0043] The laser radar simulation method proposed in the present disclosure can use scene data and corresponding simulation point clouds to simulate and simulate the reflection intensity and light fallback phenomenon of the simulation point clouds, thereby improving the realism of the simulation.

[0044] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0046] Figure 1 is a flow chart of an implementation of a laser radar simulation method 100 according to an embodiment of the present disclosure;

[0047] Figure 2 is a general framework diagram of a laser radar simulation method according to an embodiment of the present disclosure;

[0048] Figure 3 is a schematic diagram of predicting laser radar reflection intensity and light fallback based on a pre-trained model according to an embodiment of the present disclosure;

[0049] Figure 4 This is a flow chart of laser radar reflection intensity simulation based on material sub-classification according to an embodiment of the present disclosure;

[0050] Figure 5 is a flowchart of a laser radar simulation method 500 according to an embodiment of the present disclosure;

[0051] Figure 6 is a flowchart of a training method 600 for a prediction model according to an embodiment of the present disclosure;

[0052] Figure 7 is a flowchart of an implementation of a laser radar simulation method 700 according to an embodiment of the present disclosure;

[0053] Figure 8 is a schematic structural diagram of a laser radar simulation device 800 according to an embodiment of the present disclosure;

[0054] Fig. 9 is a schematic structural diagram of a laser radar simulation device 900 according to an embodiment of the present disclosure;

[0055] Fig.10 is a structural schematic diagram of a laser radar simulation device 1000 according to an embodiment of the present disclosure;

[0056] Fig.11 is a structural schematic diagram of a prediction model training device 1100 according to an embodiment of the present disclosure;

[0057] Fig.12 is a schematic structural diagram of a laser radar simulation device 1200 according to an embodiment of the present disclosure;

[0058] Fig.13 is a schematic structural diagram of a laser radar simulation device 1300 according to an embodiment of the present disclosure;

[0059] Fig.14 A schematic block diagram of an example electronic device 1400 is shown that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0060] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0061] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of types. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0062] Autonomous driving simulation is crucial in the development of autonomous driving technology. It can test and verify the autonomous driving system in a virtual environment, thereby identifying and correcting potential problems, optimizing algorithms, and significantly reducing the risk and cost of actual road testing, accelerating the process of technology research and development, and improving the safety and reliability of autonomous driving systems. Among them, LiDAR simulation is an important component of the autonomous driving simulation system. At present, most autonomous driving simulation systems use LiDAR simulation algorithms based on computer graphics (CG).

[0063] Computer graphics-based laser radar simulation generally uses the principle of simulated ray tracing to measure the target. Based on the parameters of the real laser radar, the laser radar in the simulation will emit simulated rays within the detection range at the same frequency; when the ray hits the object, the corresponding coordinate position is recorded and the obtained data is output according to the standard to obtain the simulated point cloud information of the target.

[0064] The LiDAR simulation implementation based on computer graphics has the following two obvious disadvantages:

[0065] 1. This LiDAR simulation solution cannot obtain the reflection intensity information of the point cloud like the actual LiDAR. The lack of this reflection intensity information will greatly affect the authenticity of the LiDAR simulation.

[0066] 2. When collecting point clouds, real LiDAR will experience ray fallback, that is, some of the emitted LiDAR rays will not have any return values. However, the LiDAR point cloud obtained by the implementation solution based on computer graphics is perfect and cannot simulate the ray fallback phenomenon.

[0067] It can be seen that the existing lidar simulation method cannot simulate the real point cloud, including the intensity information and light fallback phenomenon of the point cloud.

[0068] In view of the above situation, the embodiment of the present disclosure proposes a laser radar simulation method, which is data-driven and can be well combined with existing computer graphics solutions to simulate and emulate the reflection intensity and light fallback phenomenon, thereby improving the realism of the simulation.

[0069] The following is an explanation of the related technologies of the present disclosure, and the following content is not intended to limit the technical solutions of the present disclosure.

[0070] 1. LiDAR:

[0071] Laser Radar is a radar system that emits laser beams to detect the position, speed and other characteristic quantities of a target. Its working principle is to emit a detection signal (laser beam) to the target, and then compare the received signal reflected from the target (target echo) with the transmitted signal. After appropriate processing, relevant information about the target can be obtained, such as target distance, direction, height, speed, attitude, and even shape parameters, so as to detect, track and identify the target.

[0072] 2. Point Cloud:

[0073] Point cloud data is a multi-dimensional complex data set. It is generally obtained by 3D scanning devices such as lidar to obtain information on several points in space. This information includes location information and reflection intensity.

[0074] The point cloud includes multiple points, each of which has position information and intensity information. The position information may include three-dimensional coordinates (X, Y, Z), representing the spatial coordinates of the point; the intensity information represents the reflection intensity of the laser corresponding to the point.

[0075] The reflection intensity is related to the surface reflection characteristics of the object. Different surface materials, colors and angles will result in different reflection intensity values.

[0076] 3. Raydrop

[0077] During the processing and rendering of LiDAR point cloud data, due to various reasons (such as scattering, sensor performance limitations, etc.), part of the laser beam may not be able to successfully return to the receiver, resulting in missing data. This phenomenon can be called light fallback.

[0078] The disclosed embodiment proposes a laser radar simulation method, which can be applied to an autonomous driving simulation system to simulate the perception system of an autonomous driving vehicle. The method can simulate and simulate the intensity information and light fallback phenomenon of a point cloud, thereby improving the authenticity of the simulated point cloud.

[0079] Figure 1 1 is a flowchart of an implementation of a laser radar simulation method 100 according to an embodiment of the present disclosure, including:

[0080] S110, acquiring a scene image, and location information and object annotation information of a simulation point cloud corresponding to the scene image;

[0081] S120, based on the scene image, performing a first prediction of the reflection intensity and light fallback phenomenon of each point in the simulation point cloud;

[0082] S130, performing a second prediction of the reflection intensity of each point corresponding to the first object in the simulation point cloud based on the position information and the object labeling information;

[0083] S140: Determine intensity information and light fallback phenomenon of the simulation point cloud according to the result of the first prediction and the result of the second prediction.

[0084] There is no restriction on the execution order of the above-mentioned step S120 and step S130; either step can be executed first or they can be executed simultaneously.

[0085] In some examples, the scene image may include a virtual image generated by a simulation engine such as Unreal Engine (UE), which can simulate the scene image taken from the perspective of an autonomous driving vehicle in a simulation scene; and the simulation point cloud information of the simulation scene (that is, the simulation point cloud information corresponding to the scene image) is generated by a simulation engine such as UE, including the location information and object annotation information of the simulation point cloud. The object annotation information may include the type of obstacles in the simulation scene, such as pedestrians, cars, etc., and may also include multiple materials of various types of objects; for example, the material of a person includes skin, clothes, etc.; for another example, the material of a car includes metal, glass, etc.

[0086] The disclosed embodiment is based on an existing computer graphics simulation scheme, and performs a first prediction of the reflection intensity and light fallback phenomenon of a point cloud, and a second prediction of the reflection intensity of a point cloud, and merges the two prediction results to obtain the intensity information of the simulated point cloud and the prediction results of the light fallback phenomenon, thereby being able to simulate and emulate the intensity information and light fallback phenomenon of the point cloud and improve the realism of the simulated point cloud.

[0087] Figure 2 FIG. 1 is a general framework diagram of a laser radar simulation method according to an embodiment of the present disclosure. Figure 2 As shown, the laser radar simulation proposed in the embodiment of the present disclosure is mainly divided into the following four stages:

[0088] In the first stage, UE and other simulation engines are used to generate scene images, location information of simulation point clouds and object annotation information;

[0089] In the second stage, the scene image is used to perform lidar simulation using a pre-trained prediction model, that is, the first prediction mentioned above is performed. In this stage, the reflection intensity and light fallback phenomenon of each point in the simulation point cloud can be estimated.

[0090] In the third stage, using the position information and object annotation information in the simulated point cloud, for objects with detailed annotations in the point cloud, such as cars, pedestrians, bicycles, etc., a laser radar simulation method based on material segmentation is used to estimate the reflection intensity, that is, to perform the above-mentioned second prediction. In the embodiment of the present disclosure, the "detailedly annotated object" is referred to as the first object, which can be considered as the object of focus in the simulation.

[0091] In the fourth stage, based on the position information of the point cloud, the results of the second and third stages are fused to obtain the final lidar simulation result, which includes the intensity information of the simulated point cloud and the light fallback phenomenon.

[0092] It is easy to understand that there is no restriction on the execution order of the second stage and the third stage, and the two can be executed simultaneously or any stage can be executed first.

[0093] In one example, for each point in the simulated point cloud, at least one of the following methods is used to determine the reflection intensity and light fallback phenomenon of the point:

[0094] When the result of the first prediction indicates that the point has a light fallback phenomenon, determining the point as not existing;

[0095] When the result of the first prediction indicates that there is no light fallback phenomenon at the point and the second prediction result includes the reflection intensity of the point, assigning the reflection intensity of the point to the reflection intensity in the second prediction result;

[0096] When the result of the first prediction indicates that there is no light fallback phenomenon at the point and the second prediction result does not include the reflection intensity of the point, the reflection intensity of the point is assigned to the reflection intensity in the first prediction result.

[0097] In one example, the following formula (1) is used for fusion:

[0098]

[0099] The above formula (1) uses a single point in the point cloud as the granularity to illustrate the fusion method; where C final Indicates the reflection intensity of a point in the point cloud in the fusion result;

[0100] C LINet represents the prediction result of the reflection intensity of the point in the second prediction stage;

[0101] C v represents the prediction result of the reflection intensity of the point in the third prediction stage;

[0102] mask indicates whether the point corresponds to the first object; mask = 0, indicating that the point corresponds to the first object; mask = 1, indicating that the point does not correspond to the first object;

[0103] Ray drop indicates whether there is a ray drop phenomenon at the point in the prediction result of the second prediction stage; ray drop = 1 indicates that there is a ray drop phenomenon; ray drop = 0 indicates that there is no ray drop phenomenon.

[0104] In the above formula (1), the meanings of the various values ​​of mask and ray drop are only examples. The embodiments of the present disclosure may also use other representations, which are not limited here.

[0105] According to the above formula (1), for any point in the point cloud, if the pre-trained prediction model is used to predict that the point has a light fallback phenomenon, then in the fused result, the reflection intensity of the point is "empty", indicating that the point has a light fallback phenomenon; if the pre-trained prediction model is used to predict that the point does not have a light fallback phenomenon and the point corresponds to the first object, the reflection intensity predicted (or estimated) by the laser radar simulation method based on material segmentation is used as the reflection intensity of the point in the fusion result; if the pre-trained prediction model is used to predict that the point does not have a light fallback phenomenon and the point does not correspond to the first object, the reflection intensity predicted (or estimated) by the laser radar simulation method based on the pre-trained model is used as the reflection intensity of the point in the fusion result. This fusion method can simulate the light fallback phenomenon of the laser radar point cloud; and when the point in the point cloud corresponds to the object of focus (i.e., the first object mentioned above), the reflection intensity estimated by the simulation method dedicated to the object of focus is selected as the reflection intensity after fusion, thereby improving the accuracy of the reflection intensity estimation.

[0106] The above describes the overall process of the laser radar simulation method proposed in the embodiment of the present disclosure. The following describes the first prediction stage and the second prediction stage in detail.

[0107] 1. The first prediction stage:

[0108] In this stage, based on the scene image, a first prediction is made of the reflection intensity and light fallback phenomenon of each point in the simulated point cloud. In some examples, the prediction process includes:

[0109] The scene image is input into a pre-trained prediction model, and the prediction model outputs the prediction results of the reflection intensity of each point in the simulation point cloud and the prediction results of the light fallback phenomenon;

[0110] The prediction result of the light fall-back phenomenon includes whether the light fall-back phenomenon exists at the point or whether the light fall-back phenomenon does not exist.

[0111] By using a pre-trained prediction model, the reflection intensity and light falloff phenomenon of each point corresponding to the scene image can be predicted based on the scene image; since the scene image is necessary information in the autonomous driving simulation, there is no need to generate or use additional information for simulation, which can simplify the simulation process; and since the model is used for simulation, the accuracy of the simulation can be guaranteed.

[0112] Figure 3 FIG. 1 is a schematic diagram of an embodiment of the present disclosure for predicting laser radar reflection intensity and light fallback based on a pre-trained model. Figure 3 As shown, the scene image generated by the simulation engine is input into the pre-trained prediction model, and the scene image may include an RGB (Read Green Blue) image. The output of the prediction model includes two channels, one of which represents the prediction of the laser radar reflection intensity (Intensity), and the other channel represents the prediction of the light fallback (raydrop); the prediction results of the above two channels can be represented in the form of an image, and a back-projection operation is performed based on the prediction results of the above two channels, that is, the final point cloud can be obtained. The back-projection operation is a common operation in laser radar technology, which is used to realize the conversion between image information and point cloud data, and will not be introduced in detail here.

[0113] like Figure 3 As shown, in some examples, the pre-trained prediction model proposed in the embodiments of the present disclosure includes a plurality of sequentially connected layers, each layer including an encoding module, a gating module, a conversion module and a decoding module; wherein,

[0114] The encoding module in each layer is used to extract feature information from the scene image;

[0115] The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer;

[0116] The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

[0117] exist Figure 3 In the example shown, the prediction model includes 5 layers, and the first to fifth layers are connected in sequence. In this example, the first layer (L1) is called the highest layer, and the fifth layer (L5) is called the lowest layer. Figure 3In the figure, the encoding module, gating module, conversion module and decoding module are represented by Ei, Gi, Ti and Di respectively, where i represents the level of the module. For example, the first layer (L1) includes the encoding module (E1), the gating module (G1), the conversion module (T1) and the decoding module (D1). The encoding modules of each layer are connected one by one from high to low, wherein the output data of E1 is used as the input data of E2, the output data of E2 is used as the input data of E3, the output data of E3 is used as the input data of E4, and the output data of E4 is used as the input data of E5; the decoding modules of each layer are connected one by one from high to low, wherein the output data of D5 is used as the input data of D4, the output data of D4 is used as the input data of D3, the output data of D3 is used as the input data of D2, and the output data of D2 is used as the input data of D1. The output data of D1 is used as the output data of the entire prediction model, including the prediction results of the reflection intensity and the prediction results of the light fallback phenomenon.

[0118] In some examples, the five encoding modules can be implemented using the last five layers of the Dinov2 network. The Dinov2 network is an unlabeled knowledge distillation method that can be used as a feature extractor for an image. The last five layers of the Dinov2 network are referred to as E1 to E5. E1 to E5 can perform a downsampling process on data features. For example, assuming that E1 extracts features from the input scene image and obtains feature data of size 384*384, then E2 downsamples the feature data output by E1 to obtain feature data of size 192*192, E3 downsamples the feature data output by E2 to obtain feature data of size 96*96, E4 downsamples the feature data output by E3 to obtain feature data of size 48*48, and E5 downsamples the feature data output by E4 to obtain feature data of size 24*24. In this example, the output data of E1 to E5 are recorded as E1. i , where i∈{1,2,3,4,5}, i represents the level.

[0119] In this example, the output data from D1 to D5 is recorded as D i , where i∈{1,2,3,4,5}, i represents the level. In the initial state, it is D i to set the initial value.

[0120] In some embodiments, in each layer:

[0121] The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain the gating value;

[0122] The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value;

[0123] The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

[0124] In one example, the operation process of the gate control module can be expressed by the following formula (2):

[0125] G i =P(S(Conv(Cat(E i ,D i )))) (2)

[0126] Among them, G i Represents the output data of the gating module (i.e. Gi) at the i-th layer;

[0127] E i Represents the output data of the encoding module (i.e., Ei) at the i-th layer;

[0128] D i Represents the output data of the decoding module (i.e. Di) of the i-th layer;

[0129] Cat means splicing by channel;

[0130] Conv(.) represents the convolution operation;

[0131] S(.) represents the Sigmoid function operation;

[0132] P(.) represents global average pooling.

[0133] The gating module controls the multi-scale information flow between the encoding module and the decoding module. As shown in equation (2), in a layer, the output of the encoding module and the output of the decoding module are input to the gating module, which concatenates the two data by channel, performs convolution operation on the concatenated result, performs sigmoid function operation on the result after convolution operation, and performs global average pooling on the result after operation. The final output result can be a value between 0 and 1.

[0134] In one example, the operation process of the decoding module can be expressed by the following formula (3):

[0135]

[0136] Among them, D i Represents the output data of the decoding module (i.e. Di) of the i-th layer;

[0137] D i+, Represents the output data of the decoding module of the i+1th layer;

[0138] Up(.) indicates upsampling operation;

[0139] G i Represents the output data of the gating module (i.e. Gi) at the i-th layer;

[0140] T i Represents the output data of the conversion module (i.e. Ti) at the i-th layer;

[0141] The output data of the encoding model of the i-th layer is used as the input data of Ti;

[0142] Conv(.) represents the convolution operation.

[0143] As can be seen from equation (3), in the 1st, 2nd, 3rd or 4th layer, the input data of the decoding module includes the output data of the gating module of the layer, the output data of the conversion module, and the output data of the decoding module of the next layer of the layer. The decoding module multiplies the output data of the gating module and the output data of the conversion module, and upsamples the output data of the decoding module of the next layer, adds the result of the multiplication and the result of the upsampling, and then performs a convolution operation on the added data. In the 5th layer, the input data of the decoding module includes the output data of the gating module and the output data of the conversion module of the layer, and the decoding module of the 5th layer multiplies the output data of the gating module and the output data of the conversion module, and performs a convolution operation on the multiplied result.

[0144] Depend on Figure 3 It can be seen that the highest level decoding module (such as Figure 3 The output data of D1 in the above example is used as the output data of the entire prediction model. The output data of D1 includes two channels, one of which represents the prediction of raydrop and the other represents the prediction of the laser radar reflection intensity. The predicted result can be expressed as a matrix with a size equal to the size of the scene image. The laser radar point cloud of the scene image can be obtained by back-projecting the data of the two predicted channels, which is a simulated point cloud.

[0145] The disclosed embodiment adopts the above-mentioned prediction model including a multi-layer "encoding-decoding" structure, which can extract feature data of the corresponding scene from the scene image, and the feature data includes multiple feature data of different sizes, and then generate the prediction results of the reflection intensity of each point in the simulated point cloud and the prediction results of the light fallback phenomenon based on the extracted feature data, thereby realizing the estimation of the point cloud reflection intensity and the light fallback phenomenon.

[0146] The above describes an implementation method of using a pre-trained prediction model for laser radar simulation. The present disclosure also provides a prediction model training method that can train the above prediction model. In some examples, the prediction model training method includes:

[0147] Determine scene image samples and point cloud samples;

[0148] Inputting scene image samples into the prediction model to be trained, the prediction model to be trained outputs the prediction results of the reflection intensity of each point in the simulation point cloud and the prediction results of the light fallback phenomenon; wherein the prediction results of the light fallback phenomenon include: the light fallback phenomenon exists at the point or does not exist;

[0149] The loss function is calculated using the predicted results of the reflection intensity of each point and the predicted results of the light fallback phenomenon, as well as the point cloud sample;

[0150] The parameters of the prediction model to be trained are adjusted based on the loss function.

[0151] Among them, the model structure of the prediction model to be trained has been introduced in the above content and will not be described in detail here.

[0152] In one example, the scene image samples and point cloud samples can be obtained based on real collected data. For example, a road-collecting vehicle is used to capture scene images, and the laser radar of the road-collecting vehicle is used to generate laser radar point cloud data; the scene image obtained by the road-collecting vehicle is used as a scene image sample, and the laser radar point cloud data obtained by the road-collecting vehicle is used to generate corresponding point cloud samples. The collected multiple groups of samples are combined into a training set and a validation set.

[0153] Since each point in the real point cloud represents a laser radar ray with a return value, there will be no corresponding point in the point cloud for a laser radar ray without a return value (that is, a laser radar ray with light fallback); therefore, the laser radar point cloud data obtained by the road acquisition vehicle does not contain light fallback information. In order to judge the prediction results of the model for the light fallback phenomenon during the training process, it is necessary to additionally add the estimated results of the light fallback phenomenon to the collected point cloud data. For example, based on the reflection intensity of each point in the point cloud data, estimate which points have the light fallback phenomenon. That is, the point cloud samples used in the training process of the disclosed embodiment include the actual results of the reflection intensity of each point in the point cloud and the estimated results of the light fallback phenomenon.

[0154] In some examples, the reflection intensity of the real collected point cloud data can be interpolated by window and binarized by threshold to form mask information, so that the model can learn the phenomenon of light fallback of the real lidar from the mask.

[0155] In one example, a method of determining an estimation result of a light fallback phenomenon at each point in a point cloud sample includes:

[0156] Using the reflection intensities of multiple points in the point cloud sample, an interpolation algorithm is used to calculate the reflection intensity estimates of the adjacent positions of the multiple points;

[0157] Based on the reflection intensity estimation value and a preset threshold, it is determined whether there is a point with light fallback phenomenon in the vicinity of the multiple points.

[0158] For example, in a point cloud sample, there are points corresponding to the body of the vehicle, but there are no points corresponding to the window of the vehicle. Then, based on the reflection intensity of multiple points corresponding to the body, the difference algorithm can be used to calculate the reflection intensity estimation value of the points at the adjacent positions of these points (such as the points at the corresponding window positions); compare the reflection intensity estimation value with the preset threshold value. If it is greater than or equal to the preset threshold value, it indicates that the reflection intensity estimation value of the adjacent position is larger, but the point at the adjacent position does not exist in the point cloud, which shows that the light fallback phenomenon occurs at the adjacent position; if it is less than the preset threshold value, it indicates that the reflection intensity estimation value of the adjacent position is smaller, so it can be judged that the absence of the corresponding point at the adjacent position is not caused by the light fallback phenomenon, but by the smaller reflection intensity, that is, the light fallback phenomenon does not exist at the adjacent position.

[0159] The above describes a method for determining a sample point cloud. Using the predicted results of the reflection intensity of each point and the predicted results of the light fallback phenomenon, as well as the point cloud sample, the loss function can be calculated.

[0160] For example, the loss function is expressed as follows:

[0161] L=L1+L2; (4)

[0162] Where L represents the loss function;

[0163] L1 represents the error between the predicted result of reflection intensity and the actual result of reflection intensity in the point cloud sample, and the mean square error between the two can be calculated;

[0164] L2 represents the error between the predicted result of the light fallback phenomenon and the estimated result of the light fallback phenomenon in the point cloud sample, and the mean square error of the two can be calculated. Since the light fallback phenomenon in the point cloud sample is estimated in the above way, it is called "estimated result" here. The "estimated result" is approximately equal to the "true result" and plays a supervisory role equivalent to the "true result" when calculating the loss function.

[0165] (II) The second prediction stage:

[0166] This stage can serve as an effective supplement to the first prediction stage. It uses a statistically-based material classification lidar reflection intensity simulation method to solve the problem of unstable reflection intensity prediction of point clouds of key focus categories, and predicts the reflection intensity of point clouds of some key focus objects (referred to as the first object in this embodiment).

[0167] In some implementations, for a simulation point cloud generated by a simulation engine, based on position information of points in the point cloud and object annotation information, a second prediction is made on the reflection intensity of each point in the simulation point cloud corresponding to the first object, including:

[0168] Based on the object annotation information, determine a point in the simulation point cloud corresponding to the first object, and determine the object material of the first object corresponding to the point; wherein the object annotation information of the simulation point cloud includes the object and the object material corresponding to each point in the simulation point cloud, and each object corresponds to at least one object material;

[0169] Find the expression corresponding to the first object and the material of the object;

[0170] Based on the position information of the point corresponding to the first object and the expression, the reflection intensity of the point corresponding to the first object is predicted.

[0171] Through the above process, the embodiment of the present disclosure can utilize the expression obtained based on statistics to simulate the reflection intensity of the point cloud of the focused object (ie, the first object), thereby solving the problem of unstable prediction of the reflection intensity of the point cloud of the focused category.

[0172] For example, according to the object annotation information of the simulation point cloud, it is determined that the simulation point cloud contains multiple points corresponding to a person, where the "person" is the "first object", and multiple points corresponding to the person's skin and multiple points corresponding to the person's clothes are divided from the multiple points corresponding to the person. Here, the "skin" and "clothes" of the person are two types of object materials of the person.

[0173] Next, an expression corresponding to the combination of "human (i.e., an object)" and "skin (i.e., an object material)" is searched. The expression may be a polynomial, which is used to represent the functional relationship between the position information of the point corresponding to "human skin" and the reflection intensity. Based on the position information of the point corresponding to "human skin" in the simulation point cloud and the expression, the reflection intensity of the point can be calculated.

[0174] Furthermore, an expression corresponding to the combination of "person (i.e., an object)" and "clothes (i.e., an object material)" is searched, and the expression may be a polynomial, which is used to represent the functional relationship between the position information of the point corresponding to "person's clothes" and the reflection intensity. According to the position information of the point corresponding to "person's clothes" in the simulation point cloud and the expression, the reflection intensity of the point can be calculated.

[0175] In order to implement the above process, multiple expressions may be predetermined and saved, each expression corresponding to a first object and an object material of the first object; the expression represents a functional relationship between the position information of a point in the point cloud and the reflection intensity for the first object and the object material, wherein the position information may include the depth of the point.

[0176] In some implementations, the process of determining the expression includes:

[0177] Acquire multiple annotation data of a first object, each annotation data including a point cloud of the first object;

[0178] The plurality of annotated data are grouped according to the position information to obtain a plurality of data groups; each data group includes a plurality of annotated data, and each data group corresponds to a piece of position information;

[0179] For each data group, clustering the point cloud contained in the labeled data in the data group to obtain a plurality of clusters, each cluster containing a plurality of points, and each cluster corresponding to an object material of the first object;

[0180] Based on the multiple clusters, the above expression is determined.

[0181] In some examples, based on the multiple clusters, determining the above expression includes:

[0182] Determine the reflection intensity of the center point of each cluster;

[0183] Fitting is performed using the reflection intensity of the center point of each cluster of each data group and the position information corresponding to each data group to obtain N polynomials of the first object, each polynomial corresponding to the first object and an object material of the first object; wherein N is the number of object materials corresponding to the first object.

[0184] Through the above process, statistical and data fitting methods can be used to predetermine polynomials corresponding to different objects and object materials; thus, in the simulation process, by searching the polynomial, the reflection intensity of the point cloud corresponding to different objects and object materials can be predicted.

[0185] Figure 4 FIG. 1 is a flowchart of a laser radar reflection intensity simulation based on material sub-classification according to an embodiment of the present disclosure. Figure 4 As shown, the process can be divided into three stages, including data labeling, data annotation, distribution estimation, and deployment.

[0186] (1) Data annotation stage: At least two parts of annotated data can be used: the first part includes the existing data of public data sets and self-collected data, and the second part includes supplementary data annotated by large visual models such as Grounded-SAM. Through these two parts of data, the bounding boxes of key obstacles that the autonomous driving perception system pays attention to, such as cars, pedestrians, and bicycles, can be obtained, laying the foundation for the subsequent point cloud intensity analysis of these categories.

[0187] (2) Distribution estimation stage: The disclosed embodiment may use a clustering algorithm such as kmeans++ to cluster the point cloud intensities of the same category of objects at different distances, and classify the number of preset object materials (such as skin, clothes) (represented by k) to obtain k groups of center points at the distance. Afterwards, a random sample consensus (RANSAC) algorithm and other algorithms are used to perform polynomial fitting on the cluster center points of the point clouds of objects of the same type and material at different distances to obtain polynomials corresponding to multiple specific object types and object materials, each polynomial representing the functional relationship between the distance and the point cloud intensity of the point cloud corresponding to the object type and object material. Specifically, it can be expressed by the following formula (5):

[0188] k_clusters=KMeans++(k,points)

[0189] poly = RANSAC(k_clusters, degree) (5)

[0190] Among them, points represents a point cloud of a certain type within a fixed distance (or depth);

[0191] k represents the number of object materials corresponding to this type of object;

[0192] KMeans++(.) means KMeans++ clustering;

[0193] k_clusters represents the reflection intensity of the center point after clustering the point cloud;

[0194] Degree indicates the degree of the polynomial used;

[0195] RANSAC(.) means using the RANSAC algorithm for polynomial fitting;

[0196] Poly represents the final k polynomials, each of which corresponds to an object and object material.

[0197] For example, multiple groups of point cloud data are analyzed, each group of point cloud data contains points corresponding to people, and these points are divided according to a fixed distance (or depth) range. For example, the point cloud with a distance of 5 meters is divided into one group, and the point cloud with a distance of 10 meters is divided into one group, and so on. In this way, it is divided into 20 groups, and each group of point cloud corresponds to a distance.

[0198] Assume that objects such as "people" are divided into two materials, including clothes and skin. For each group of point clouds, cluster the group of point clouds, the number of categories is 2, and then two clusters are obtained after clustering. Each cluster contains multiple point clouds, and the reflection intensity of the center point of each cluster is determined. In this way, for 20 groups of point clouds, a total of 40 center points are determined; 20 center points correspond to the "clothes" material, and the other 20 center points correspond to the "skin" material.

[0199] The information of each center point is taken as a set of data, which includes the distance corresponding to the center point and the reflection intensity of the center point. The above 40 data are used for data fitting. For example, the 20 center points corresponding to the "clothing" material are fitted to obtain a polynomial, which contains the corresponding relationship between distance and reflection intensity; the 20 center points corresponding to the "skin" material are fitted to obtain a polynomial, which contains the corresponding relationship between distance and reflection intensity. Figure 4 In the distribution estimation stage, images corresponding to multiple polynomials are displayed. Each point in the image represents a center point. The horizontal coordinate of the center point represents the distance of the point, and the vertical coordinate represents the reflection intensity of the point. The lines in the image represent the polynomials fitted using multiple center points.

[0200] (3) Deployment phase: The deployment phase, i.e., the laser radar simulation phase proposed in the embodiments of the present disclosure, first separates the materials of different objects in digital modeling, for example, dividing the materials of pedestrians into skin and clothing, and then assigns the point cloud reflection intensity information at different distances in the laser radar simulation according to the polynomial fitting functions of different materials.

[0201] For example, when it is determined that the object corresponding to a point cloud is a "person" and the corresponding object material is "clothes", the polynomial corresponding to the person "person" and "clothes" is found, and then the distance between each point in the point cloud and the polynomial is used to calculate the reflection intensity of the point.

[0202] The above introduces the laser radar simulation method proposed in the embodiment of the present disclosure. The method combines two prediction methods and can simulate the intensity information and light fallback phenomenon of the point cloud. In the above process, two prediction methods are introduced; these two prediction methods can be used as complete solutions to realize the estimation of the intensity information and light fallback phenomenon of the point cloud, or to estimate the intensity information of the point cloud.

[0203] Specifically, the embodiment of the present disclosure also proposes a laser radar simulation method. Figure 5 : is a flowchart of a laser radar simulation method 500 according to an embodiment of the present disclosure, including:

[0204] S510, inputting the scene image into a pre-trained prediction model, and the prediction model outputs the prediction results of the reflection intensity of each point in the simulation point cloud and the prediction results of the light fallback phenomenon;

[0205] The prediction result of the light fall-off phenomenon includes whether the light fall-off phenomenon exists at a point or whether the light fall-off phenomenon does not exist at a point.

[0206] In some embodiments, the pre-trained prediction model includes a plurality of sequentially connected layers, each layer including an encoding module, a gating module, a conversion module, and a decoding module; wherein,

[0207] The encoding module in each layer is used to extract feature information from the scene image;

[0208] The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer;

[0209] The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

[0210] In some embodiments, in a single layer of a pre-trained prediction model:

[0211] The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain the gating value;

[0212] The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value;

[0213] The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

[0214] For other details, please refer to the relevant content of the first prediction stage mentioned above, which will not be repeated here.

[0215] The disclosed embodiment also provides a prediction model training method. Figure 6 6 is a flowchart of a prediction model training method 600 according to an embodiment of the present disclosure, including:

[0216] S610, determining scene image samples and point cloud samples;

[0217] S620, inputting the scene image sample into the prediction model to be trained, and the prediction model to be trained outputs the prediction result of the reflection intensity of each point in the simulation point cloud and the prediction result of the light fallback phenomenon; wherein the prediction result of the light fallback phenomenon includes whether the point has the light fallback phenomenon or does not have the light fallback phenomenon;

[0218] S630, calculating a loss function using the prediction results of the reflection intensity of each point and the prediction results of the light fallback phenomenon, as well as the point cloud samples;

[0219] S640: Adjust the parameters of the prediction model to be trained based on the loss function.

[0220] In some implementations, the point cloud sample includes actual results of the reflection intensity of each point in the point cloud and an estimated result of the light fallback phenomenon.

[0221] In some embodiments, the method of determining the estimated result of the light fallback phenomenon at each point in the point cloud sample includes:

[0222] Using the reflection intensities of multiple points in the point cloud sample, an interpolation algorithm is used to calculate reflection intensity estimates of adjacent positions of the multiple points;

[0223] Based on the reflection intensity estimation value and a preset threshold, it is determined whether there is a point with light fallback phenomenon at adjacent positions of the plurality of points.

[0224] In some embodiments, the prediction model to be trained includes a plurality of sequentially connected layers, each layer including an encoding module, a gating module, a conversion module, and a decoding module; wherein,

[0225] The encoding module in each layer is used to extract feature information from the scene image;

[0226] The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer;

[0227] The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

[0228] In some embodiments, in a single layer of a prediction model to be trained:

[0229] The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain the gating value;

[0230] The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value;

[0231] The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

[0232] For other details, please refer to the relevant content of the first prediction stage mentioned above, which will not be repeated here.

[0233] The disclosed embodiment also provides a laser radar simulation method. Figure 7 700 is a flowchart of a laser radar simulation method according to an embodiment of the present disclosure, including:

[0234] S710, obtaining position information and object annotation information of the simulation point cloud; the object annotation information includes the object and object material corresponding to the point in the simulation point cloud; the object corresponds to at least one object material;

[0235] S720, determining a point in the simulation point cloud corresponding to the first object based on the object annotation information, and determining an object material of the first object corresponding to the point;

[0236] S730, searching for an expression corresponding to the first object and the object material;

[0237] S740. Predict the reflection intensity of the point corresponding to the first object according to the position information and expression of the point corresponding to the first object.

[0238] In some embodiments, it also includes predetermining and saving multiple expressions, each expression corresponds to a first object and an object material of the first object; the expression represents the functional relationship between the position information of a point in the point cloud and the reflection intensity.

[0239] In some embodiments, determining the expression includes:

[0240] Acquire multiple annotated data of the first object, each annotated data including a point cloud of the first object;

[0241] According to the position information, the multiple labeled data are grouped to obtain multiple data groups; each data group contains multiple labeled data, and each data group corresponds to one position information;

[0242] For each data group, clustering the point cloud contained in the labeled data in the data group to obtain a plurality of clusters, each cluster containing a plurality of points, and each cluster corresponding to an object material of the first object;

[0243] Based on the multiple clusters, the above expression is determined.

[0244] In some examples, based on the multiple clusters, determining the above expression includes:

[0245] Determine the reflection intensity of the center point of each cluster;

[0246] Fitting is performed using the reflection intensity of the center point of each cluster of each data group and the position information corresponding to each data group to obtain N polynomials of the first object, each polynomial corresponding to the first object and an object material of the first object; wherein N is the number of object materials corresponding to the first object.

[0247] For other details, please refer to the relevant content of the second prediction stage mentioned above, which will not be repeated here.

[0248] The disclosed embodiment also provides a laser radar simulation device, Figure 8 8 is a schematic diagram of a structure of a laser radar simulation device 800 according to an embodiment of the present disclosure, including:

[0249] A first acquisition module 810 is used to acquire a scene image, and location information and object annotation information of a simulation point cloud corresponding to the scene image;

[0250] A first prediction module 820, configured to make a first prediction of the reflection intensity and light fallback phenomenon of each point in the simulation point cloud based on the scene image;

[0251] A second prediction module 830, configured to perform a second prediction of the reflection intensity of each point corresponding to the first object in the simulation point cloud based on the position information and the object labeling information;

[0252] The first determination module 840 is used to determine the intensity information and light fallback phenomenon of the simulated point cloud according to the results of the first prediction and the results of the second prediction.

[0253] In some embodiments, the first prediction module 820 is used to input the scene image into a pre-trained prediction model, and the prediction model outputs the prediction results of the reflection intensity of each point in the simulation point cloud and the prediction results of the light fallback phenomenon;

[0254] The prediction result of the light fall-off phenomenon includes whether the light fall-off phenomenon exists at a point or whether the light fall-off phenomenon does not exist at a point.

[0255] In some embodiments, the pre-trained prediction model includes a plurality of sequentially connected layers, each layer including an encoding module, a gating module, a conversion module, and a decoding module;

[0256] The encoding module in each layer is used to extract feature information from the scene image;

[0257] The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer;

[0258] The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

[0259] In some embodiments, in a single layer of a pre-trained prediction model:

[0260] The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain the gating value;

[0261] The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value;

[0262] The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

[0263] In some embodiments, the object annotation information includes an object and an object material corresponding to a point in the simulation point cloud; the object corresponds to at least one object material;

[0264] The second prediction module 830 is used to:

[0265] Based on the object annotation information, determine a point in the simulation point cloud corresponding to the first object, and determine an object material of the first object corresponding to the point;

[0266] Find the expression corresponding to the first object and the object material;

[0267] Based on the position information of the point corresponding to the first object and the expression, the reflection intensity of the point corresponding to the first object is predicted.

[0268] The disclosed embodiment also provides a laser radar simulation device, Fig. 9 is a schematic diagram of the structure of a laser radar simulation device 900 according to an embodiment of the present disclosure, including:

[0269] The second determination module 950 is used to predetermine and save multiple expressions, each expression corresponds to a first object and an object material of the first object; the expression represents the functional relationship between the position information of a point in the point cloud and the reflection intensity.

[0270] In some implementations, the second determination module 950 is configured to:

[0271] Acquire multiple annotation data of a first object, each annotation data including a point cloud of the first object;

[0272] According to the position information, the multiple labeled data are grouped to obtain multiple data groups; each data group contains multiple labeled data, and each data group corresponds to one position information;

[0273] For each data group, clustering the point cloud contained in the labeled data in the data group to obtain a plurality of clusters, each cluster containing a plurality of points, and each cluster corresponding to an object material of the first object;

[0274] Based on the multiple clusters, the above expression is determined.

[0275] In some examples, the second determination module 950 is used to:

[0276] Determine the reflection intensity of the center point of each cluster;

[0277] Fitting is performed using the reflection intensity of the center point of each cluster of each data group and the position information corresponding to each data group to obtain N polynomials of the first object, each polynomial corresponding to the first object and an object material of the first object; wherein N is the number of object materials corresponding to the first object.

[0278] In some implementations, the first determination module 840 is configured to:

[0279] For each point in the simulated point cloud, the reflection intensity of the point is determined by at least one of the following methods:

[0280] In the case where the result of the first prediction indicates that the point has a light fallback phenomenon, determining the point as not existing;

[0281] When the result of the first prediction indicates that there is no light fallback phenomenon at the point and the second prediction result includes the reflection intensity of the point, assigning the reflection intensity of the point to the reflection intensity in the second prediction result;

[0282] When the result of the first prediction indicates that there is no light fallback phenomenon at the point and the second prediction result does not include the reflection intensity of the point, the reflection intensity of the point is assigned to the reflection intensity in the first prediction result.

[0283] The disclosed embodiment also provides a laser radar simulation device, Fig.10 1 is a schematic diagram of the structure of a laser radar simulation device 1000 according to an embodiment of the present disclosure, comprising:

[0284] A first input module 1010 is used to input a scene image into a pre-trained prediction model, and the prediction model outputs a prediction result of the reflection intensity of each point in the simulation point cloud and a prediction result of the light fallback phenomenon;

[0285] The prediction result of the light fall-off phenomenon includes whether the light fall-off phenomenon exists at a point or whether the light fall-off phenomenon does not exist at a point.

[0286] In some embodiments, the pre-trained prediction model includes a plurality of sequentially connected layers, each layer including an encoding module, a gating module, a conversion module, and a decoding module; wherein,

[0287] The encoding module in each layer is used to extract feature information from the scene image;

[0288] The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer;

[0289] The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

[0290] In some embodiments, in a single layer of a pre-trained prediction model:

[0291] The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain the gating value;

[0292] The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value;

[0293] The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

[0294] The disclosed embodiment also provides a prediction model training device, Fig.11 1 is a schematic diagram of a training device 1100 for a prediction model according to an embodiment of the present disclosure, comprising:

[0295] A third determination module 1110, configured to determine scene image samples and point cloud samples;

[0296] The second input module 1120 is used to input the scene image sample into the prediction model to be trained, and the prediction model to be trained outputs the prediction result of the reflection intensity of each point in the simulation point cloud and the prediction result of the light fallback phenomenon; wherein the prediction result of the light fallback phenomenon includes whether the point has the light fallback phenomenon or does not have the light fallback phenomenon;

[0297] The adjustment module 1130 is used to calculate the loss function using the prediction results of the reflection intensity of each point and the prediction results of the light fallback phenomenon, as well as the point cloud samples; and adjust the parameters of the prediction model to be trained based on the loss function.

[0298] In some implementations, the point cloud sample includes actual results of the reflection intensity of each point in the point cloud and an estimated result of the light fallback phenomenon.

[0299] In some embodiments, the method of determining the estimated result of the light fallback phenomenon at each point in the point cloud sample includes:

[0300] Using the reflection intensity of multiple points in the point cloud sample, an interpolation algorithm is used to calculate the reflection intensity estimation value of the adjacent positions of the multiple points;

[0301] Based on the reflection intensity estimation value and a preset threshold, it is determined whether there is a point with light fallback phenomenon in the vicinity of the plurality of points.

[0302] In some embodiments, the prediction model to be trained includes a plurality of sequentially connected layers, each layer including an encoding module, a gating module, a conversion module, and a decoding module; wherein,

[0303] The encoding module in each layer is used to extract feature information from the scene image;

[0304] The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer;

[0305] The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

[0306] In some embodiments, in a single layer of a prediction model to be trained:

[0307] The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain the gating value;

[0308] The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value;

[0309] The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

[0310] The disclosed embodiment also provides a laser radar simulation device, Fig.12 1 is a schematic diagram of the structure of a laser radar simulation device 1200 according to an embodiment of the present disclosure, including:

[0311] The second acquisition module 1210 is used to acquire the position information and object annotation information of the simulation point cloud; the object annotation information includes the object and the object material corresponding to the point in the simulation point cloud; the object corresponds to at least one object material;

[0312] The fourth determination module 1220 is used to determine a point in the simulation point cloud corresponding to the first object based on the object annotation information, and determine the object material of the first object corresponding to the point;

[0313] The third prediction module 1230 is used to find the expression corresponding to the first object and the object material; and predict the reflection intensity of the point corresponding to the first object according to the position information and the expression of the point corresponding to the first object.

[0314] The disclosed embodiment also provides a laser radar simulation device, Fig.131 is a schematic diagram of the structure of a laser radar simulation device 1300 according to an embodiment of the present disclosure, including:

[0315] The fifth determination module 1340 is used to predetermine and save multiple expressions, each expression corresponds to a first object and an object material of the first object; the expression represents the functional relationship between the position information of a point in the point cloud and the reflection intensity.

[0316] In some implementations, the fifth determination module 1340 is configured to:

[0317] Acquire multiple annotated data of the first object, each annotated data including a point cloud of the first object;

[0318] According to the position information, the multiple labeled data are grouped to obtain multiple data groups; each data group contains multiple labeled data, and each data group corresponds to one position information;

[0319] For each data group, clustering the point cloud contained in the labeled data in the data group to obtain a plurality of clusters, each cluster containing a plurality of points, and each cluster corresponding to an object material of the first object;

[0320] Based on the multiple clusters, the above expression is determined.

[0321] In some examples, the fifth determination module 1340 is used to:

[0322] Determine the reflection intensity of the center point of each cluster;

[0323] Fitting is performed using the reflection intensity of the center point of each cluster of each data group and the position information corresponding to each data group to obtain N polynomials of the first object, each polynomial corresponding to the first object and an object material of the first object; wherein N is the number of object materials corresponding to the first object.

[0324] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0325] In the technical solution disclosed in the present invention, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0326] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0327] Fig.14A schematic block diagram of an example electronic device 1400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0328] like Fig.14 As shown, the device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the device 1400 can also be stored. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0329] A number of components in the device 1400 are connected to the I / O interface 1405, including: an input unit 1406, such as a keyboard, a mouse, etc.; an output unit 1407, such as various types of displays, speakers, etc.; a storage unit 1408, such as a disk, an optical disk, etc.; and a communication unit 1409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1409 allows the device 1400 to exchange data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0330] The computing unit 1401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1401 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0331] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0332] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0333] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0334] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0335] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0336] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0337] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0338] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A laser radar simulation method, comprising: Acquire a scene image, and location information and object annotation information of a simulation point cloud corresponding to the scene image; Based on the scene image, a first prediction is made on the reflection intensity and light fallback phenomenon of each point in the simulation point cloud; Furthermore, based on the position information and the object labeling information, a second prediction is made on the reflection intensity of each point corresponding to the first object in the simulation point cloud; According to the result of the first prediction and the result of the second prediction, intensity information and light fallback phenomenon of the simulation point cloud are determined.

2. The method according to claim 1, wherein: The first prediction of the reflection intensity and light fallback phenomenon of each point in the simulation point cloud based on the scene image includes: Inputting the scene image into a pre-trained prediction model, and having the prediction model output a prediction result of the reflection intensity of each point in the simulation point cloud and a prediction result of the light fallback phenomenon; The prediction result of the light fall-back phenomenon includes whether the light fall-back phenomenon exists at the point or not.

3. The method according to claim 2, wherein: The pre-trained prediction model includes a plurality of sequentially connected layers, each of which includes an encoding module, a gating module, a conversion module and a decoding module; wherein, The encoding module in each layer is used to extract feature information in the scene image; The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer; The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

4. The method according to claim 3, wherein: In a single layer of the pre-trained prediction model: The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain a gating value; The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value; The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

5. The method according to claim 1, wherein: The object annotation information includes the object and the object material corresponding to the point in the simulation point cloud; the object corresponds to at least one object material; The performing a second prediction of the reflection intensity of each point corresponding to the first object in the simulation point cloud based on the position information and the object labeling information includes: Based on the object annotation information, determining a point in the simulation point cloud corresponding to the first object, and determining an object material of the first object corresponding to the point; Finding the expression corresponding to the first object and the material of the object; The reflection intensity of the point corresponding to the first object is predicted according to the position information of the point corresponding to the first object and the expression.

6. The method according to claim 5, further comprising: Predetermining and saving a plurality of expressions, each of the expressions corresponding to a first object and an object material of the first object; The expression represents the functional relationship between the position information of a point in the point cloud and the reflection intensity.

7. The method according to claim 6, wherein: Ways to determine the expression include: Acquire a plurality of annotated data of a first object, each of the annotated data comprising a point cloud of the first object; According to the position information, the plurality of annotated data are grouped to obtain a plurality of data groups; each data group includes a plurality of annotated data, and each data group corresponds to one piece of the position information; For each data group, clustering the point cloud contained in the labeled data in the data group to obtain a plurality of clusters, each cluster containing a plurality of points, and each cluster corresponding to an object material of the first object; Based on the plurality of clusters, the expression is determined.

8. The method according to claim 7, wherein: The step of determining the expression based on the plurality of clusters comprises: Determine the reflection intensity of the center point of each of the clusters; Fitting is performed using the reflection intensity of the center point of each cluster of each data group and the position information corresponding to each data group to obtain N polynomials of the first object, each polynomial corresponding to the first object and an object material of the first object; wherein N is the number of object materials corresponding to the first object.

9. The method according to any one of claims 1 to 8, wherein: Determining the intensity information and the light fallback phenomenon of the simulation point cloud according to the result of the first prediction and the result of the second prediction includes: For each point in the simulated point cloud, the reflection intensity and light fallback phenomenon of the point are determined by at least one of the following methods: When the result of the first prediction indicates that the point has a light fallback phenomenon, determining that the point does not exist; When the result of the first prediction indicates that there is no light fallback phenomenon at the point, and the second prediction result includes the reflection intensity of the point, assigning the reflection intensity of the point to the reflection intensity in the second prediction result; When the result of the first prediction indicates that there is no light fallback phenomenon at the point and the second prediction result does not include the reflection intensity of the point, the reflection intensity of the point is assigned to the reflection intensity in the first prediction result.

10. A laser radar simulation method, comprising: Inputting the scene image into a pre-trained prediction model, the prediction model outputs the prediction results of the reflection intensity of each point in the simulation point cloud and the prediction results of the light fallback phenomenon; The prediction result of the light fall-back phenomenon includes whether the light fall-back phenomenon exists at the point or not.

11. The method according to claim 10, wherein: The pre-trained prediction model includes a plurality of sequentially connected layers, each of which includes an encoding module, a gating module, a conversion module and a decoding module; wherein, The encoding module in each layer is used to extract feature information in the scene image; The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer; The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

12. The method according to claim 11, wherein: In a single layer of the pre-trained prediction model: The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain a gating value; The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value; The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

13. A method for training a prediction model, comprising: Determine scene image samples and point cloud samples; The scene image sample is input into the prediction model to be trained, and the prediction model to be trained outputs the prediction result of the reflection intensity of each point in the simulation point cloud and the prediction result of the light fallback phenomenon; wherein the prediction result of the light fallback phenomenon includes whether the light fallback phenomenon exists at the point or not; Calculating a loss function using the predicted results of the reflection intensity of each point and the predicted results of the light fallback phenomenon, as well as the point cloud samples; The parameters of the prediction model to be trained are adjusted based on the loss function.

14. The method according to claim 13, wherein: The point cloud sample includes the actual result of the reflection intensity of each point in the point cloud and the estimated result of the light fallback phenomenon.

15. The method according to claim 14, wherein: The method of determining the estimation result of the light fallback phenomenon of each point in the point cloud sample includes: Using the reflection intensities of the plurality of points in the point cloud sample, an interpolation algorithm is used to calculate the reflection intensity estimation values ​​of the adjacent positions of the plurality of points; Based on the reflection intensity estimation value and a preset threshold, it is determined whether there is a point with light fallback phenomenon in the vicinity of the plurality of points.

16. The method according to any one of claims 13 to 15, wherein: The prediction model to be trained includes a plurality of sequentially connected layers, each of which includes an encoding module, a gating module, a conversion module and a decoding module; wherein, The encoding module in each layer is used to extract feature information in the scene image; The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer; The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

17. The method according to claim 16, wherein: In a single layer of the prediction model to be trained: The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain a gating value; The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value; The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

18. A laser radar simulation method, comprising: Obtain the location information and object annotation information of the simulation point cloud; The object annotation information includes the object and the object material corresponding to the point in the simulation point cloud; The object corresponds to at least one object material; Based on the object annotation information, determining a point in the simulation point cloud corresponding to the first object, and determining an object material of the first object corresponding to the point; Finding the expression corresponding to the first object and the material of the object; The reflection intensity of the point corresponding to the first object is predicted according to the position information of the point corresponding to the first object and the expression.

19. The method according to claim 18, further comprising: Predetermining and saving a plurality of expressions, each of the expressions corresponding to a first object and an object material of the first object; The expression represents the functional relationship between the position information of a point in the point cloud and the reflection intensity.

20. The method according to claim 19, wherein: Determining the expression includes: Acquire a plurality of annotated data of a first object, each of the annotated data comprising a point cloud of the first object; According to the position information, the plurality of annotated data are grouped to obtain a plurality of data groups; each data group includes a plurality of annotated data, and each data group corresponds to one piece of the position information; For each data group, clustering the point cloud contained in the labeled data in the data group to obtain a plurality of clusters, each cluster containing a plurality of points, and each cluster corresponding to an object material of the first object; Based on the plurality of clusters, the expression is determined.

21. The method according to claim 20, wherein: The step of determining the expression based on the plurality of clusters comprises: Determine the reflection intensity of the center point of each of the clusters; Fitting is performed using the reflection intensity of the center point of each cluster of each data group and the position information corresponding to each data group to obtain N polynomials of the first object, each polynomial corresponding to the first object and an object material of the first object; wherein N is the number of object materials corresponding to the first object.

22. A laser radar simulation device, comprising: A first acquisition module, used to acquire a scene image, and location information and object annotation information of a simulation point cloud corresponding to the scene image; A first prediction module, configured to make a first prediction of the reflection intensity and light fallback phenomenon of each point in the simulation point cloud based on the scene image; A second prediction module, configured to perform a second prediction on the reflection intensity of each point corresponding to the first object in the simulation point cloud based on the position information and the object labeling information; The first determination module is used to determine the intensity information and the light fallback phenomenon of the simulation point cloud according to the result of the first prediction and the result of the second prediction.

23. The device according to claim 22, wherein: The first prediction module is used to input the scene image into a pre-trained prediction model, and the prediction model outputs the prediction results of the reflection intensity of each point in the simulation point cloud and the prediction results of the light fallback phenomenon; The prediction result of the light fall-back phenomenon includes whether the light fall-back phenomenon exists at the point or not.

24. The device according to claim 23, wherein: The pre-trained prediction model includes a plurality of sequentially connected layers, each of which includes an encoding module, a gating module, a conversion module, and a decoding module; The encoding module in each layer is used to extract feature information in the scene image; The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer; The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

25. The device according to claim 24, wherein: In a single layer of the pre-trained prediction model: The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain a gating value; The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value; The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

26. The device according to claim 22, wherein: The object annotation information includes the object and the object material corresponding to the point in the simulation point cloud; the object corresponds to at least one object material; The second prediction module is used for: Based on the object annotation information, determining a point in the simulation point cloud corresponding to the first object, and determining an object material of the first object corresponding to the point; Finding the expression corresponding to the first object and the material of the object; The reflection intensity of the point corresponding to the first object is predicted according to the position information of the point corresponding to the first object and the expression.

27. The apparatus according to claim 26, further comprising: A second determination module is used to predetermine and save a plurality of expressions, each of which corresponds to a first object and an object material of the first object; The expression represents the functional relationship between the position information of a point in the point cloud and the reflection intensity.

28. The device according to claim 27, wherein The second determining module is used for: Acquire a plurality of annotated data of a first object, each of the annotated data comprising a point cloud of the first object; According to the position information, the plurality of annotated data are grouped to obtain a plurality of data groups; each data group includes a plurality of annotated data, and each data group corresponds to one piece of the position information; For each data group, clustering the point cloud contained in the labeled data in the data group to obtain a plurality of clusters, each cluster containing a plurality of points, and each cluster corresponding to an object material of the first object; Based on the plurality of clusters, the expression is determined.

29. The device according to claim 28, wherein The second determining module is used for: Determine the reflection intensity of the center point of each of the clusters; Fitting is performed using the reflection intensity of the center point of each cluster of each data group and the position information corresponding to each data group to obtain N polynomials of the first object, each polynomial corresponding to the first object and an object material of the first object; wherein N is the number of object materials corresponding to the first object.

30. The device according to any one of claims 22 to 29, wherein: The first determining module is used for: For each point in the simulated point cloud, the reflection intensity of the point is determined by at least one of the following methods: When the result of the first prediction indicates that the point has a light fallback phenomenon, determining that the point does not exist; When the result of the first prediction indicates that there is no light fallback phenomenon at the point, and the second prediction result includes the reflection intensity of the point, assigning the reflection intensity of the point to the reflection intensity in the second prediction result; When the result of the first prediction indicates that there is no light fallback phenomenon at the point and the second prediction result does not include the reflection intensity of the point, the reflection intensity of the point is assigned to the reflection intensity in the first prediction result.

31. A laser radar simulation device, comprising: A first input module is used to input a scene image into a pre-trained prediction model, and the prediction model outputs a prediction result of the reflection intensity of each point in the simulation point cloud and a prediction result of the light fallback phenomenon; The prediction result of the light fall-back phenomenon includes whether the light fall-back phenomenon exists at the point or not.

32. The device according to claim 31, wherein The pre-trained prediction model includes a plurality of sequentially connected layers, each of which includes an encoding module, a gating module, a conversion module and a decoding module; wherein, The encoding module in each layer is used to extract feature information in the scene image; The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer; The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

33. The device according to claim 32, wherein: In a single layer of the pre-trained prediction model: The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain a gating value; The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value; The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

34. A training device for a prediction model, comprising: A third determination module is used to determine scene image samples and point cloud samples; A second input module is used to input the scene image sample into the prediction model to be trained, and the prediction model to be trained outputs the prediction result of the reflection intensity of each point in the simulation point cloud and the prediction result of the light fallback phenomenon; wherein the prediction result of the light fallback phenomenon includes whether the light fallback phenomenon exists at the point or not; An adjustment module is used to calculate a loss function using the prediction results of the reflection intensity of each point and the prediction results of the light fallback phenomenon, as well as the point cloud samples; and to adjust the parameters of the prediction model to be trained based on the loss function.

35. The device according to claim 34, wherein The point cloud sample includes the actual result of the reflection intensity of each point in the point cloud and the estimated result of the light fallback phenomenon.

36. The device according to claim 35, wherein The manner in which the third determination module determines the estimation result of the light fallback phenomenon at each point in the point cloud sample includes: Using the reflection intensities of the plurality of points in the point cloud sample, an interpolation algorithm is used to calculate the reflection intensity estimation values ​​of the adjacent positions of the plurality of points; Based on the reflection intensity estimation value and a preset threshold, it is determined whether there is a point with light fallback phenomenon in the vicinity of the plurality of points.

37. The device according to any one of claims 34 to 36, wherein: The prediction model to be trained includes a plurality of sequentially connected layers, each of which includes an encoding module, a gating module, a conversion module and a decoding module; wherein, The encoding module in each layer is used to extract feature information in the scene image; The output data of the encoding module of the high-level layer is used as the input data of the encoding model of the adjacent low-level layer, and the output data of the decoding module of the low-level layer is used as the input data of the decoding model of the adjacent high-level layer; The output data of the highest-level decoding module is used as the output data of the pre-trained prediction model.

38. The device according to claim 37, wherein In a single layer of the prediction model to be trained: The gating module is used to calculate the output result of the encoding module and the output result of the decoding module in the layer to obtain a gating value; The conversion module is used to calculate the output result of the encoding module in the layer to obtain a conversion value; The decoding module is used to calculate the gate value and the conversion value in the layer, and the decoded data output by the decoding module of the adjacent lower layer of the layer, so as to obtain the decoded data of the layer.

39. A laser radar simulation device, comprising: The second acquisition module is used to obtain the location information and object marking information of the simulation point cloud; The object annotation information includes the object and the object material corresponding to the point in the simulation point cloud; The object corresponds to at least one object material; A fourth determination module, configured to determine, based on the object annotation information, a point in the simulation point cloud corresponding to the first object, and determine an object material of the first object corresponding to the point; The third prediction module is used to find the expression corresponding to the first object and the material of the object; and predict the reflection intensity of the point corresponding to the first object according to the position information of the point corresponding to the first object and the expression.

40. The apparatus of claim 39, further comprising: A fifth determination module, used to predetermine and save a plurality of expressions, each of which corresponds to a first object and an object material of the first object; The expression represents the functional relationship between the position information of a point in the point cloud and the reflection intensity.

41. The device according to claim 40, wherein The fifth determining module is used for: Acquire a plurality of annotated data of a first object, each of the annotated data comprising a point cloud of the first object; According to the position information, the plurality of annotated data are grouped to obtain a plurality of data groups; each data group includes a plurality of annotated data, and each data group corresponds to one piece of the position information; For each data group, clustering the point cloud contained in the labeled data in the data group to obtain a plurality of clusters, each cluster containing a plurality of points, and each cluster corresponding to an object material of the first object; Based on the plurality of clusters, the expression is determined.

42. The device according to claim 41, wherein The fifth determining module is used for: Determine the reflection intensity of the center point of each of the clusters; Fitting is performed using the reflection intensity of the center point of each cluster of each data group and the position information corresponding to each data group to obtain N polynomials of the first object, each polynomial corresponding to the first object and an object material of the first object; wherein N is the number of object materials corresponding to the first object.

43. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 21.

44. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-21.

45. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 21.

Images