Data generation method and device

Abstract

The embodiment of the invention provides a data generation method and device. The method comprises the following steps: constructing an extreme road condition scene at least comprising a laser radar, a static object and a dynamic object in a simulation environment; according to the object information of the static object and the dynamic object, scanning parameters of the laser radar are adjusted; controlling the laser radar to collect point cloud data of the static object and the dynamic object in the extreme road condition scene according to the adjusted scanning parameters; and generating a three-dimensional occupation true value in the extreme road condition scene according to the collected point cloud data. An extreme road condition scene including various static objects and dynamic objects is simulated in a simulation environment, and scanning parameters of a laser radar are dynamically adjusted by using object information of the static objects and the dynamic objects to optimize point cloud data distribution, so that the laser radar can acquire dense and smooth point cloud data in the extreme road condition scene, and the point cloud data can be accurately acquired. And the data quality of the three-dimensional occupancy truth value is improved.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to a data generation method and apparatus. Background Technology

[0002] 3D occupancy (3D OCC) ground truth is a type of environmental perception data commonly used in cutting-edge fields such as autonomous driving and robot navigation. It is used to represent obstacles and free space in the environment. For example, a 3D occupancy ground truth of 0 can usually represent free space, while a 3D occupancy ground truth of 1 can usually represent obstacles.

[0003] The true value of 3D occupancy serves as the basis for training and validating 3D occupancy prediction algorithms, directly affecting the performance and robustness of the algorithms.

[0004] Traditional methods for generating 3D occupancy ground truth mainly rely on vehicles equipped with LiDAR (such as cars) to collect point cloud data from the real environment in real time, and to determine the semantic information and 3D bounding boxes (bboxes) of the point cloud through manual annotation; then, 3D occupancy ground truth is generated based on the collected point cloud data.

[0005] However, some extreme road conditions are not always encountered in the real world, making it difficult to collect point cloud data under extreme road conditions. This results in sparse point cloud data collected from the simulation environment (lacking point cloud data under extreme road conditions), leading to low data quality of the 3D occupancy ground truth, which in turn affects the robustness and generalization ability of the 3D occupancy prediction algorithm. Summary of the Invention

[0006] In view of the above, one or more embodiments of this specification provide a data generation method and apparatus, electronic device, and storage medium to solve the problems existing in the related art.

[0007] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0008] According to a first aspect of the embodiments of this specification, a data generation method is provided, the method comprising:

[0009] Construct extreme road condition scenarios in a simulation environment that include at least lidar, static objects, and dynamic objects;

[0010] The scanning parameters of the lidar are adjusted based on the object information of the static and dynamic objects.

[0011] The lidar is controlled to collect point cloud data of static and dynamic objects in the extreme road condition scenario according to the adjusted scanning parameters;

[0012] Based on the collected point cloud data, the true 3D occupancy value for the extreme road condition scenario is generated.

[0013] Optionally, the scanning parameters include at least one of the following: number of scanning beams, acquisition resolution, and scanning range.

[0014] Optionally, the object information includes the distance between a static object and the lidar, and the distance between a dynamic object and the lidar;

[0015] Adjusting the scanning parameters of the lidar based on the object information of the static and dynamic objects includes at least one of the following:

[0016] For distant areas where the distance to the lidar is greater than a preset distance, the density of static and / or dynamic objects within the distant area is calculated; if the density is greater than a preset density, the number of scanning beams of the lidar is adjusted according to the density, and the acquisition resolution is adjusted according to the adjusted number of scanning beams.

[0017] For close-range areas where the distance to the lidar is no greater than a preset distance, the scanning range of the lidar is increased.

[0018] Optionally, generating the 3D occupancy ground truth for the extreme road condition scenario based on the collected point cloud data includes:

[0019] The collected point cloud data is divided into a first point cloud subset of static objects and a second point cloud subset of dynamic objects.

[0020] Based on the first point cloud subset and the second point cloud subset, generate the three-dimensional occupancy ground truth value for the extreme road condition scenario.

[0021] Optionally, before generating the 3D occupancy ground truth for the extreme road condition scenario based on the first point cloud subset and the second point cloud subset, the method further includes:

[0022] Denoising processing is performed on the first point cloud subset and / or the second point cloud subset respectively;

[0023] The step of generating the 3D occupancy ground truth value for the extreme road condition scenario based on the first point cloud subset and the second point cloud subset includes:

[0024] Based on the first and second point cloud subsets after denoising, the true occupancy value for the extreme road condition scenario is generated.

[0025] Optionally, the denoising process performed on the first point cloud subset and / or the second point cloud subset includes:

[0026] Perform k-nearest neighbor denoising on the first point cloud subset; and / or,

[0027] Point cloud filtering is performed on the second point cloud subset to filter out point cloud data that do not belong to the current dynamic object.

[0028] Optionally, before segmenting the collected point cloud data into a first point cloud subset of static objects and a second point cloud subset of dynamic objects, the method further includes:

[0029] The collected point cloud data is downsampled to obtain a key point cloud set composed of key points in the point cloud data;

[0030] The step of dividing the collected point cloud data into a first point cloud subset of static objects and a second point cloud subset of dynamic objects includes:

[0031] The key point cloud set is divided into a first point cloud subset of static objects and a second point cloud subset of dynamic objects.

[0032] Optionally, the step of downsampling the collected point cloud data to obtain a key point cloud set composed of key points in the point cloud data includes:

[0033] For each point in the collected point cloud data, the curvature of each point is calculated based on the point cloud normal vector;

[0034] Key points with curvature greater than a threshold are selected to obtain a set of key points.

[0035] Optionally, the step of downsampling the collected point cloud data to obtain a key point cloud set composed of key points in the point cloud data includes:

[0036] The collected point cloud data is divided into front and back parts;

[0037] For the first part, the coordinates of each point are rounded to the first unit of precision, and points with unique coordinates are retained through a uniqueness operation;

[0038] For the latter part, the coordinates of each point are rounded to the second unit of precision, and points with unique coordinates are retained through a uniqueness operation; where the second unit of precision is higher than the first unit of precision.

[0039] The points processed in the two parts are stacked vertically and merged into a set of key points.

[0040] Optionally, generating the 3D occupancy ground truth for the extreme road condition scenario based on the denoised first and second point cloud subsets includes:

[0041] Spatial range filtering is performed on the point clouds in the first and second point cloud subsets after denoising to generate point clouds of the region of interest.

[0042] The point cloud of the region of interest is mapped onto a 3D volumetric pixel mesh to generate a 3D occupancy mesh without semantics.

[0043] By utilizing the target point cloud which inherently possesses semantic meaning, semantic labels are assigned to the 3D occupancy grid through nearest neighbor search, generating semantically meaningful 3D occupancy ground truth values.

[0044] According to a second aspect of the embodiments of this specification, a data generation apparatus is provided, the apparatus comprising:

[0045] The building unit constructs extreme road condition scenarios in a simulation environment, including at least lidar, static objects, and dynamic objects.

[0046] The adjustment unit adjusts the scanning parameters of the lidar based on the object information of the static and dynamic objects.

[0047] The acquisition unit controls the lidar to acquire point cloud data of static and dynamic objects in the extreme road condition scenario according to the adjusted scanning parameters;

[0048] The generation unit generates the true 3D occupancy value under the extreme road condition scenario based on the collected point cloud data.

[0049] According to a third aspect of the embodiments of this specification, an electronic device is provided, including a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus;

[0050] The memory stores machine-readable instructions, and the processor executes the above method by invoking the machine-readable instructions.

[0051] According to a fourth aspect of the embodiments of this specification, a machine-readable storage medium is provided, the machine-readable storage medium storing machine-readable instructions, which, when invoked and executed by a processor, implement the above-described method.

[0052] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0053] By simulating extreme road conditions involving various static and dynamic objects in a simulation environment, and then using the object information of these static and dynamic objects, the scanning parameters of the LiDAR are dynamically adjusted to optimize the point cloud data distribution (making the point cloud data distribution both dense and smooth). This allows the LiDAR to acquire high-quality (dense and smooth) point cloud data under extreme road conditions. In this way, it can cover point cloud data in extreme road conditions that are difficult to acquire using traditional methods, improving the data quality of 3D occupancy ground truth. This, in turn, helps the 3D occupancy prediction algorithm better learn the object features and spatial relationships under extreme road conditions, enhancing the algorithm's robustness and generalization ability. Attached Figure Description

[0054] Figure 1 A flowchart illustrating a data generation method provided as an exemplary embodiment of this specification;

[0055] Figure 2 A schematic diagram of a static object and a dynamic object provided for an exemplary embodiment of this specification;

[0056] Figure 3 A schematic diagram of the scanning area of ​​a lidar provided for an exemplary embodiment of this specification;

[0057] Figure 4 A schematic diagram of the structure of an electronic device in which a data generation apparatus is located, provided as an exemplary embodiment of this specification;

[0058] Figure 5 This is a block diagram of a data generation apparatus provided for an exemplary embodiment of this specification. Detailed Implementation

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0060] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0061] It should be noted that the OCC truth value in this application refers to the three-dimensional occupancy truth value, which is the occupancy truth value in three-dimensional space. It can be understood that three-dimensional can also be referred to as 3D.

[0062] Traditional methods for generating 3D occupancy ground truth primarily rely on vehicles equipped with LiDAR (such as cars) to collect point cloud data from the real-world environment in real time, and then manually annotate the point clouds to determine their semantic information and 3D bounding boxes; finally, 3D occupancy ground truth is generated based on the collected point cloud data. However, this method has the following problems:

[0063] 1. The data quality is low. Low data quality can be divided into two aspects:

[0064] Firstly, real-time point cloud data is limited by sensor hardware performance and environmental conditions, and is easily affected by factors such as sensor noise, changes in illumination, occlusion, and interference from dynamic objects, leading to a decrease in the accuracy of the OCC true value. For example, in multi-frame point cloud stitching, dynamic objects (such as pedestrians and vehicles) may introduce point clouds of non-target objects, causing errors or noise in the OCC representation and affecting the reliability of subsequent algorithms.

[0065] Secondly, in real-world environments, data collection in extreme scenarios (such as heavy rain, dense fog, nighttime traffic congestion, or rare combinations of obstacles) is costly and difficult, and may even be impossible due to safety restrictions. This results in insufficient coverage of OCC ground truth values ​​in these scenarios, limiting the robustness and generalization ability of 3D occupancy prediction algorithms.

[0066] 2. High annotation costs. Semantic and 3D annotation of point clouds requires professional personnel to analyze and annotate frame by frame, a complex and time-consuming process. This is especially true when dealing with large-scale datasets or frequently acquired data, where labor costs increase dramatically. Furthermore, the consistency and accuracy of annotations are difficult to guarantee, increasing the uncertainty in generating OCC ground truth values.

[0067] 3. Sparsity problem: LiDAR point cloud data is usually sparse, especially in long-distance, complex terrain or low-resolution scenarios. The point cloud representation of the ground and obstacles is not accurate enough, resulting in the loss of details in the OCC ground truth, which makes it difficult to meet the needs of high-precision environmental perception.

[0068] In view of this, the embodiments of this specification provide a data generation scheme. On the one hand, by simulating and constructing high-quality extreme road condition scenarios through physical modeling in a simulation environment and Unreal Engine rendering technology, it can cover extreme road condition scenarios that are difficult to collect using traditional methods, improve the data quality of 3D occupancy ground truth, and thus help the 3D occupancy prediction algorithm better learn the object features and spatial relationships in extreme road condition scenarios, improving the algorithm's robustness and generalization ability. This solves the problem of low data quality mentioned above.

[0069] On the other hand, by utilizing the "God's-eye view" in a simulation environment, semantic annotation and 3D labeling can be automatically performed on the point cloud data collected by the lidar, thereby solving the problem of high labeling costs.

[0070] On the other hand, by dynamically adjusting the scanning parameters of the lidar, the point cloud distribution can be adaptively optimized in different scenarios to ensure that the collected point cloud data is both dense and smooth, thereby solving the above-mentioned sparsity problem.

[0071] Please refer to the following. Figure 1 , Figure 1 A flowchart of a data generation method provided for an exemplary embodiment of this specification, the method may include the following steps:

[0072] Step 110: Construct an extreme road condition scenario in the simulation environment that includes at least lidar, static objects, and dynamic objects.

[0073] High-fidelity scenes are built in the simulation environment, including both static and dynamic objects, supporting extreme road conditions. Physically modeled LiDAR and camera sensors simulate the physical properties of light beam propagation, reflection, and scattering, generating multimodal data (such as RGB images and LiDAR data). Unreal Engine (UE) rendering technology ensures realistic visuals and point cloud data of the scene through accurate lighting models, material reflection properties, and dynamic object behavior simulation.

[0074] In this specification, the static object may include, but is not limited to, stationary objects such as roads, buildings, and traffic signs.

[0075] The dynamic objects may include, but are not limited to, moving objects such as motor vehicles, non-motor vehicles, and pedestrians.

[0076] like Figure 2 As shown, the extreme road condition scenario constructed in the simulation environment can include dynamic pedestrians 1, dynamic vehicles 1-4, and static roadside trees 5.

[0077] The extreme road conditions mentioned may include, but are not limited to, roads under severe weather conditions such as heavy rain, dense fog, and heavy snow, roads under complex traffic conditions, and roads under special obstacles (such as road construction or traffic accidents).

[0078] Step 120: Adjust the scanning parameters of the lidar based on the object information of the static and dynamic objects.

[0079] To address the issue of point cloud sparsity, this step involves dynamically adjusting the scanning parameters of the lidar to achieve denser point cloud distribution.

[0080] The scanning parameters include at least one of the following: number of scanning beams, acquisition resolution, and scanning range.

[0081] In implementation, a balanced acquisition resolution (e.g., 256*256 or 512*512) and an initial number of beams (e.g., 128 beams) can be set based on the acquisition range of the 3D occupancy ground truth. However, the scanning parameters of the lidar are adjusted according to the object information of the static and dynamic objects to ensure that the density of the acquired point cloud data meets the requirements.

[0082] In practical applications, such as Figure 3 As shown, since the default scanning parameters of the lidar 31 need to take into account both the distant region 33 and the close region 32 (the distance from the lidar 31 outside X can be considered as distant, and the distance within X can be considered as close), it cannot capture the point cloud data of objects in the distant region well, and it also leads to the problem of sparsity of the point cloud data in the close region.

[0083] Therefore, for close-range areas where the distance to the lidar is no greater than a preset distance, the scanning range of the lidar is increased. This increases the scanning range to collect denser point cloud data.

[0084] For distant areas where the distance to the lidar is greater than a preset distance, although it is possible to simply increase the beam size to 256 or higher 512 and use a high resolution of 2k for acquisition, this will result in slow acquisition efficiency.

[0085] Therefore, in order to balance efficiency and results, this embodiment can first calculate the density of static and / or dynamic objects in the distant area where the distance from the lidar is greater than a preset distance. If the density is greater than the preset density, the number of scanning beams of the lidar is adjusted according to the density, and the acquisition resolution is adjusted according to the adjusted number of scanning beams.

[0086] In this way, it can cover the point cloud of objects in a distant area, while avoiding the jagged effect caused by low acquisition resolution.

[0087] In this embodiment, the object information may include the distance between a static object and the lidar, and the distance between a dynamic object and the lidar; the target static object may refer to a static object whose distance from the lidar is greater than a preset distance, and the target dynamic object may refer to a dynamic object whose distance from the lidar is greater than a preset distance.

[0088] The density can refer to the ratio between the number of static objects in the distant area and the total number of objects in the extreme road condition scenario (the sum of the number of static objects and the number of dynamic objects);

[0089] The density can also refer to the ratio between the number of target dynamic objects in the long-distance area and the total number of objects in the extreme road condition scenario;

[0090] The density can also refer to the ratio between the total number of dynamic and static target objects in the long-distance area and the total number of objects in the extreme road condition scenario.

[0091] If the density is greater than the preset density, it means that the scanning parameters of the lidar need to be adjusted.

[0092] Specifically, the process involves adjusting the number of scanning beams of the lidar based on the density, and adjusting the acquisition resolution based on the adjusted number of scanning beams. The scanning beam data can be adjusted using the following formula:

[0093] N = N_0 + (D - D_threshold) / K

[0094] Where N is the adjusted number of scan beams, N_0 is the initial (or before adjustment) number of scan beams, D is the density, D_threshold is the preset density, and K is the adjustment coefficient used to control the increase in the number of beams.

[0095] It is understandable that the aforementioned D_threshold and K can be calculated based on experience or experiments to adapt to different application scenarios and needs.

[0096] In addition, the above adjustment of the acquisition resolution based on the adjusted number of scan lines can be implemented as follows:

[0097] Substitute the adjusted number of scan lines (i.e., the aforementioned N value) into the acquisition resolution calculation formula to calculate the adjusted acquisition resolution. This acquisition resolution calculation formula can be a preset formula, such as an empirical formula summarized from historical data. In an exemplary embodiment, the acquisition resolution calculation formula can be 2N*2N; for example, if the adjusted number of scan lines is 256, then the acquisition resolution is 512*512.

[0098] Step 130: Control the lidar to collect point cloud data of static and dynamic objects in the extreme road condition scenario according to the adjusted scanning parameters.

[0099] By dynamically adjusting the scanning parameters of the LiDAR as described above, the point cloud distribution can be adaptively optimized in different scenarios, ensuring that the collected point cloud data is both dense and smooth, thereby improving the generation quality of the OCC ground truth.

[0100] Additionally, it should be noted that since the simulation environment has relevant information about all objects (such as size, coordinates, category, etc.), the "God's-eye view" of the simulation environment can be used to automatically generate semantic annotations (such as object category), instance IDs (such as object ID), and 3D bounding box annotations for each point in the point cloud data, which can reduce costs compared to traditional manual annotation.

[0101] To adapt to subsequent data processing needs, the point cloud data output from the simulation environment can be converted into a standardized format (such as PCD or LAS format). The standardization process may include unifying the coordinate system, aligning timestamps, and optimizing the data structure to ensure consistency between the point cloud data and the semantic / instance annotations.

[0102] Step 140: Generate the true 3D occupancy value under the extreme road condition scenario based on the collected point cloud data.

[0103] In practical applications, it has been found that lidar is easily affected by environmental interference factors (such as background objects, temporary obstructions, etc.), resulting in the collection of point cloud data containing point clouds of non-static or non-dynamic objects.

[0104] Based on this, in an exemplary embodiment, the above-mentioned generation of the 3D occupancy ground truth under the extreme road condition scenario based on the collected point cloud data may further include:

[0105] The collected point cloud data is divided into a first point cloud subset of static objects and a second point cloud subset of dynamic objects.

[0106] Based on the first point cloud subset and the second point cloud subset, generate the three-dimensional occupancy ground truth value for the extreme road condition scenario.

[0107] In this embodiment, the above segmentation can be based on the instance ID (used to uniquely represent the object to which it belongs) and the 3D bounding box (used to locate the three-dimensional spatial range in which the instance ID represents the object) provided by the simulation environment to extract the first point cloud subset of static objects and the second point cloud subset of dynamic objects from the point cloud data collected by the LiDAR.

[0108] Furthermore, based on the ego pose of the vehicle where the lidar is located and the extrinsic parameters of the camera sensor, all point clouds can be transformed from the sensor coordinate system to a unified global coordinate system in order to prepare for subsequent point cloud stitching.

[0109] After segmentation, the second point cloud subset of dynamic objects and the first point cloud subset of static objects can be denoised as follows:

[0110] That is, before generating the 3D occupancy ground truth under the extreme road condition scenario based on the first point cloud subset and the second point cloud subset, it may further include:

[0111] Denoising processing is performed on the first point cloud subset and / or the second point cloud subset respectively;

[0112] The step of generating the 3D occupancy ground truth value for the extreme road condition scenario based on the first point cloud subset and the second point cloud subset includes:

[0113] Based on the first and second point cloud subsets after denoising, the true occupancy value for the extreme road condition scenario is generated.

[0114] In one exemplary embodiment, denoising the second point cloud subset of a dynamic object can be performed by point cloud filtering on the second point cloud subset, filtering out point cloud data in the second point cloud subset that do not belong to the current dynamic object.

[0115] In implementation, the `mmcv.points_in_boxes_cpu` function can be used to check whether the points in the second point cloud subset are within the 3D bounding box of the current dynamic object. This `mmcv.points_in_boxes_cpu` function can efficiently calculate the spatial inclusion relationship between points and 3D bounding boxes using CPU-efficient computation and returns the index of the 3D bounding box to which each point belongs.

[0116] Then, for the target point located within the 3D bounding box of the current dynamic object, it is further verified whether the instance ID of the target point is consistent with the instance ID of the 3D bounding box. If they are consistent, it means that the target point is a point of the current dynamic object; if they are inconsistent, it means that the target point is an interference point that overlaps with the current dynamic object (i.e., it is not actually a point of the current dynamic object).

[0117] Point cloud filtering can ensure the purity of point clouds of dynamic objects, reduce noise interference in the generation of 3D occupancy ground truth, and provide data quality for generating 3D occupancy ground truth.

[0118] In one exemplary embodiment, denoising a first subset of point clouds of a static object can be performed by k-nearest neighbor denoising on the first subset of point clouds.

[0119] To address semantic labeling biases caused by incorrect material configurations or rendering issues in simulation environments (such as ground points being mislabeled as "obstacles"), k-nearest neighbor denoising can be based on the assumption of spatial continuity of point clouds, namely, the principle that points on the same object's surface cluster in space and their semantic labels should be consistent, thus correcting semantic labeling biases.

[0120] In implementation, the three-dimensional coordinates (x, y, z) of the first point cloud sub-center of the static object and the semantic labels (such as semantic categories such as ground, road, building, etc.) can be extracted.

[0121] Next, a KD tree is constructed to efficiently find the k nearest neighbors of each point (k can be set to 10 by default, and the value of k can be adjusted according to the point cloud density, for example, set to 5 in sparse regions and 15 in dense regions).

[0122] For each point, the semantic categories of its k neighbors are counted, and a majority voting mechanism is used to determine the dominant semantic category. For example, if 8 out of 10 neighbors are labeled "ground," then the semantic label of that point is updated to "ground." The majority voting can employ a weighted strategy, where the weight of neighboring points can be decayed according to distance (e.g., 1 / distance) to enhance the influence of nearest neighbors. The updated semantic label is then reassigned to the point cloud to ensure semantic consistency of static objects.

[0123] To avoid over-smoothing, a minimum voting threshold can be set (e.g., at least 60% of neighbors agree) to prevent miscorrection of the correct labels for sparse regions.

[0124] The above methods can effectively correct isolated or incorrectly labeled semantic tags and improve the reliability of point clouds in the first point cloud sub-set of static objects.

[0125] In an exemplary embodiment, before segmenting the acquired point cloud data into a first point cloud subset of static objects and a second point cloud subset of dynamic objects, the following may be included:

[0126] The collected point cloud data is downsampled to obtain a key point cloud set composed of key points in the point cloud data;

[0127] Accordingly, the above-mentioned segmentation of the collected point cloud data into a first point cloud subset of static objects and a second point cloud subset of dynamic objects includes:

[0128] The key point cloud set is divided into a first point cloud subset of static objects and a second point cloud subset of dynamic objects.

[0129] In practical applications, the point cloud data acquired by LiDAR typically has a high data volume. In particular, this embodiment uses dynamic adjustment of scanning parameters to enable LiDAR to acquire denser point cloud data. The larger the data volume, the higher the resource requirements for the above segmentation and denoising (low computational efficiency). At the same time, considering that too little data will result in insufficient information (leading to low quality of 3D ground truth), a downsampling operation is proposed to balance efficiency and quality. This operation prioritizes retaining key points in the point cloud data (such as obstacle edge points) and removes redundant points (such as duplicate points in flat areas of the ground).

[0130] The downsampling operation can be implemented in the following ways.

[0131] In one implementation:

[0132] For each point in the collected point cloud data, the curvature of each point is calculated based on the point cloud normal vector;

[0133] Key points with curvature greater than a threshold are selected to obtain a key point cloud set composed of these key points.

[0134] In this embodiment, the formula for calculating the curvature of the point cloud normal vector is as follows:

[0135]

[0136] Where Curvature is the curvature, n p Let n be the normal vector of the current point p. i Let k be the normal vector of the i-th neighbor of the current point p, and k be the number of neighbors.

[0137] In another implementation:

[0138] For high-density areas (such as the ground), the number of point clouds can be reduced by random sampling or distance-based uniform sampling (such as nearest neighbor distance greater than 0.05m), and the point set in the area can be controlled within the target density (such as 100 points per cubic meter).

[0139] In yet another implementation:

[0140] The collected point cloud data is divided into front and back parts;

[0141] For the first part, the coordinates of each point are rounded to the first unit of precision, and points with unique coordinates are retained through a uniqueness operation;

[0142] For the latter part, the coordinates of each point are rounded to the second unit of precision, and points with unique coordinates are retained through a uniqueness operation; where the second unit of precision is higher than the first unit of precision.

[0143] The points processed in the two parts are stacked vertically and merged into a set of key points.

[0144] In this embodiment, the input point cloud data (containing coordinates x, y, z and other attributes such as intensity, semantic label, and instance ID) can be divided into two parts: the first half (the first 50% of points) and the second half (the last 50% of points). For the first half of the point cloud, the coordinates (x, y, z) are rounded to 0.1 units of precision (i.e., retaining one decimal place), and then a uniqueness operation is performed to retain points with unique coordinates. Specifically, the rounded coordinates are compared, points with duplicate coordinates are removed, and only the first occurrence of the point cloud and all its attributes (including intensity, label, etc.) are retained. For the second half of the point cloud, the coordinates are rounded to 0.02 units of precision (i.e., retaining two decimal places), and a uniqueness operation is also performed to retain points with unique coordinates. The higher precision rounding ensures that the second half of the point cloud retains more detail, making it suitable for capturing complex geometric structures.

[0145] Then, the two processed point clouds (the unique points in the first half and the unique points in the second half) are vertically stacked and merged into a set of downsampled key points for output.

[0146] This approach balances the sparsity of the point cloud and the preservation of geometric information through partitioning and rounding with varying precision. The first half uses lower precision (0.1 units) to reduce redundant points, suitable for flat areas (such as the ground); the second half uses higher precision (0.02 units) to preserve details, suitable for complex areas (such as obstacle edges). Furthermore, compared to the two methods mentioned above that require a fixed 3D mesh size, this approach is more adaptable and can dynamically adjust the point cloud density in different scenes.

[0147] Downsampling using any of the above methods typically reduces point cloud data volume by more than 90% while preserving key geometric and semantic information, providing efficient input for subsequent segmentation, denoising, and stitching processes.

[0148] In an exemplary embodiment, the aforementioned generation of the 3D occupancy ground truth under the extreme road condition scenario based on the first and second point cloud subsets after denoising processing may include:

[0149] Spatial range filtering is performed on the point clouds in the first and second point cloud subsets after denoising to generate point clouds of the region of interest.

[0150] The point cloud of the region of interest is mapped onto a 3D volumetric mesh to generate a 3D occupancy mesh without semantics.

[0151] By utilizing the target point cloud which inherently possesses semantic meaning, semantic labels are assigned to the 3D occupancy grid through nearest neighbor search, generating semantically meaningful 3D occupancy ground truth values.

[0152] In this embodiment, spatial range filtering can refer to applying spatial range constraints to point clouds (including coordinates and semantic labels). The constraints can be achieved by defining the range of the region of interest, for example, retaining point clouds whose absolute values ​​of the x-axis and y-axis are less than preset values ​​(such as 5 meters) and whose z-axis height is within a preset range (such as -2 meters to 2 meters), thereby focusing on the region of interest and reducing interference from irrelevant point clouds.

[0153] Generate a semantically undefined 3D occupancy mesh. Specifically, map the point cloud of the region of interest to a 3D volumetric mesh (resolution can be set to 0.1m). Determine the occupancy state (occupied, free, or unknown) of each volumetric mesh based on the point cloud distribution. The occupancy state is determined by the presence or absence of points in the point cloud within the volumetric mesh, generating an initial 3D occupancy mesh. At this point, the 3D occupancy mesh is not assigned any semantic meaning; therefore, the generated 3D occupancy mesh is semantically undefined.

[0154] To generate semantically meaningful 3D occupancy ground truth, a semantic point cloud (containing object category labels) can be used. A nearest neighbor search is then employed to assign a semantic label to the center point of each 3D occupancy grid. This method ensures spatial consistency of semantic information by calculating the distance between the 3D occupancy grid and the semantic point cloud and selecting the semantic label of the nearest point.

[0155] After completing the above steps, post-processing such as noise suppression and boundary optimization can be performed on the semantic 3D occupancy ground truth to improve the quality of the 3D occupancy mesh representation.

[0156] In practical applications, the generated semantically meaningful 3D occupancy truth can be verified for data quality through downstream algorithms.

[0157] For example, the generated semantically meaningful 3D occupancy ground truth is applied to the downstream task of obstacle detection to verify its effectiveness in both normal and extreme road condition scenarios. Experimental comparisons show that the 3D occupancy ground truth generated by this embodiment outperforms traditional methods in terms of accuracy, robustness, and computational efficiency, evaluating its support capability for 3D occupancy prediction algorithms.

[0158] Validation can include algorithm cold start testing (initializing the model using general data), edge sample supplementation (improving the model's generalization ability), and performance evaluation in complex scenarios.

[0159] Experimental results show that the three-dimensional occupancy ground truth data generated in this embodiment can be used for algorithm cold start, and in special verification, it improves the performance of the algorithm in extreme scenarios, such as improving the obstacle detection accuracy of special obstacles by more than 70%.

[0160] In an exemplary embodiment of this specification, an apparatus capable of implementing the above-described method is also provided.

[0161] Figure 4 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment. Please refer to... Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, such as the processor reading the corresponding computer program from non-volatile memory into memory and then running it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0162] Please refer to Figure 5 In one software implementation, a data generation apparatus is provided, the apparatus comprising:

[0163] Construction unit 510 constructs an extreme road condition scenario in a simulation environment, which includes at least lidar, static objects, and dynamic objects;

[0164] The adjustment unit 520 adjusts the scanning parameters of the lidar based on the object information of the static and dynamic objects.

[0165] The acquisition unit 530 controls the lidar to acquire point cloud data of static and dynamic objects in the extreme road condition scenario according to the adjusted scanning parameters;

[0166] The generation unit 540 generates the true three-dimensional occupancy value under the extreme road condition scenario based on the collected point cloud data.

[0167] Optionally, the scanning parameters include at least one of the following: number of scanning beams, acquisition resolution, and scanning range.

[0168] Optionally, the object information includes the distance between a static object and the lidar, and the distance between a dynamic object and the lidar;

[0169] The adjustment unit 520 includes at least one of the following:

[0170] The first adjustment subunit calculates the density of static and / or dynamic objects in a distant region where the distance to the lidar is greater than a preset distance. If the density is greater than a preset density, the number of scanning beams of the lidar is adjusted according to the density, and the acquisition resolution is adjusted according to the adjusted number of scanning beams.

[0171] The second adjustment subunit increases the scanning range of the lidar for close-range areas where the distance to the lidar is no greater than a preset distance.

[0172] Optionally, the generation unit 540 includes:

[0173] The segmentation subunit divides the collected point cloud data into a first point cloud subset of static objects and a second point cloud subset of dynamic objects.

[0174] A sub-unit is generated to generate the true 3D occupancy value under the extreme road condition scenario based on the first point cloud subset and the second point cloud subset.

[0175] Optionally, the process may further include the following before generating the subunit:

[0176] The denoising subunit performs denoising processing on the first point cloud subset and the second point cloud subset respectively;

[0177] The generation subunit is further used to generate the three-dimensional occupancy ground truth value under the extreme road condition scenario based on the first point cloud subset and the second point cloud subset after the denoising process.

[0178] Optionally, the denoising subunit is further configured to perform k-nearest neighbor denoising on the first point cloud subset; and / or to perform point cloud filtering on the second point cloud subset, filtering out point cloud data in the second point cloud subset that do not belong to the current dynamic object.

[0179] Optionally, before the segmented subunit, the following may also be included:

[0180] The sampling subunit performs a downsampling operation on the collected point cloud data to obtain a key point cloud set composed of key points in the point cloud data;

[0181] Accordingly, the segmentation subunit is further configured to segment the key point cloud set into a first point cloud subset of static objects and a second point cloud subset of dynamic objects.

[0182] Optionally, the sampling subunit includes:

[0183] The curvature filtering subunit calculates the curvature of each point in the collected point cloud data based on the point cloud normal vector; it filters out key points with curvature greater than a threshold, thus obtaining a set of key points composed of the key points.

[0184] Optionally, the sampling subunit includes:

[0185] The partitioned sampling subunit divides the collected point cloud data into front and back parts. For the front part, the coordinates of each point are rounded to the first unit of precision, and points with unique coordinates are retained through a uniqueness operation. For the back part, the coordinates of each point are rounded to the second unit of precision, and points with unique coordinates are retained through a uniqueness operation. The second unit of precision is higher than the first unit of precision. The points processed by the front and back parts are vertically stacked and merged into a key point set.

[0186] Optionally, the generation subunit is further configured to perform spatial range filtering on the point clouds in the first and second point cloud subsets after denoising to generate point clouds of region of interest; map the point clouds of region of interest onto a three-dimensional voxel grid to generate a three-dimensional occupancy grid without semantics; and use the target point cloud which has semantics to assign semantic labels to the three-dimensional occupancy grid through nearest neighbor search to generate a three-dimensional occupancy ground truth with semantics.

[0187] For details on the implementation process of the functions and roles of each module in the above-mentioned device, please refer to the implementation process of the corresponding steps in the above-mentioned data generation method. For relevant details, please refer to the description of the method implementation method. It will not be repeated here.

[0188] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the units or modules can be selected to achieve the purpose of the solution described in this specification, depending on actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0189] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0190] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0191] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0192] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0193] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0194] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0195] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0196] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0197] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A data generation method, characterized in that, The method includes: Construct extreme road condition scenarios in a simulation environment that include at least lidar, static objects, and dynamic objects; The scanning parameters of the lidar are adjusted based on the object information of the static and dynamic objects. The lidar is controlled to collect point cloud data of static and dynamic objects in the extreme road condition scenario according to the adjusted scanning parameters; Based on the collected point cloud data, the true 3D occupancy value for the extreme road condition scenario is generated.

2. The method according to claim 1, characterized in that, The scanning parameters include at least one of the following: number of scanning beams, acquisition resolution, and scanning range.

3. The method according to claim 2, characterized in that, The object information includes the distance between a static object and the lidar, and the distance between a dynamic object and the lidar; Adjusting the scanning parameters of the lidar based on the object information of the static and dynamic objects includes at least one of the following: For distant areas where the distance to the lidar is greater than a preset distance, the density of static and / or dynamic objects within the distant area is calculated; if the density is greater than a preset density, the number of scanning beams of the lidar is adjusted according to the density, and the acquisition resolution is adjusted according to the adjusted number of scanning beams. For close-range areas where the distance to the lidar is no greater than a preset distance, the scanning range of the lidar is increased.

4. The method according to claim 1, characterized in that, The step of generating the 3D ground truth value of the extreme road condition scenario based on the collected point cloud data includes: The collected point cloud data is divided into a first point cloud subset of static objects and a second point cloud subset of dynamic objects. Based on the first point cloud subset and the second point cloud subset, generate the three-dimensional occupancy ground truth value for the extreme road condition scenario.

5. The method according to claim 4, characterized in that, Before generating the 3D occupancy ground truth for the extreme road condition scenario based on the first point cloud subset and the second point cloud subset, the method further includes: Denoising processing is performed on the first point cloud subset and / or the second point cloud subset respectively; The step of generating the 3D occupancy ground truth value for the extreme road condition scenario based on the first point cloud subset and the second point cloud subset includes: Based on the first and second point cloud subsets after denoising, the true occupancy value for the extreme road condition scenario is generated.

6. The method according to claim 5, characterized in that, The denoising process performed on the first point cloud subset and / or the second point cloud subset includes: Perform k-nearest neighbor denoising on the first point cloud subset; and / or, Point cloud filtering is performed on the second point cloud subset to filter out point cloud data that do not belong to the current dynamic object.

7. The method according to claim 4, characterized in that, Before segmenting the collected point cloud data into a first point cloud subset of static objects and a second point cloud subset of dynamic objects, the method further includes: The collected point cloud data is downsampled to obtain a key point cloud set composed of key points in the point cloud data; The step of dividing the collected point cloud data into a first point cloud subset of static objects and a second point cloud subset of dynamic objects includes: The key point cloud set is divided into a first point cloud subset of static objects and a second point cloud subset of dynamic objects.

8. The method according to claim 7, characterized in that, The downsampling operation on the collected point cloud data to obtain a key point cloud set composed of key points in the point cloud data includes: For each point in the collected point cloud data, the curvature of each point is calculated based on the point cloud normal vector; Key points with curvature greater than a threshold are selected to obtain a set of key points.

9. The method according to claim 7, characterized in that, The downsampling operation on the collected point cloud data to obtain a key point cloud set composed of key points in the point cloud data includes: The collected point cloud data is divided into front and back parts; For the first part, the coordinates of each point are rounded to the first unit of precision, and points with unique coordinates are retained through a uniqueness operation; For the latter part, the coordinates of each point are rounded to the second unit of precision, and points with unique coordinates are retained through a uniqueness operation; where the second unit of precision is higher than the first unit of precision. The points processed in the two parts are stacked vertically and merged into a set of key points.

10. The method according to claim 5, characterized in that, The step of generating the 3D ground truth value of the extreme road condition scenario based on the first and second point cloud subsets after denoising includes: Spatial range filtering is performed on the point clouds in the first and second point cloud subsets after denoising to generate point clouds of the region of interest. The point cloud of the region of interest is mapped onto a 3D volumetric pixel mesh to generate a 3D occupancy mesh without semantics. By utilizing the target point cloud which inherently possesses semantic meaning, semantic labels are assigned to the 3D occupancy grid through nearest neighbor search, generating semantically meaningful 3D occupancy ground truth values.

11. A data generation apparatus, characterized in that, The device includes: The building unit constructs extreme road condition scenarios in a simulation environment, including at least lidar, static objects, and dynamic objects. The adjustment unit adjusts the scanning parameters of the lidar based on the object information of the static and dynamic objects. The acquisition unit controls the lidar to acquire point cloud data of static and dynamic objects in the extreme road condition scenario according to the adjusted scanning parameters; The generation unit generates the true 3D occupancy value under the extreme road condition scenario based on the collected point cloud data.

12. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-10 by executing the executable instructions.

13. A machine-readable storage medium having stored thereon machine-readable instructions that, when executed by a processor, implement the method as claimed in any one of claims 1-10.

Images