Label generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation

Abstract

The application discloses a label generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation, which generates a depth map and a depth confidence map of a center frame through a depth estimation method, and performs system initialization; generates 3D point clouds according to the depth map and the depth confidence map, and performs point cloud accumulation before and after the center frame to obtain accumulated point clouds; separates static point clouds and dynamic point clouds from the accumulated point clouds, and generates a static region bird's-eye view label and a dynamic target instance bird's-eye view label; and generates a bird's-eye view label based on the static region bird's-eye view label and the dynamic target instance bird's-eye view label. The application utilizes multi-frame information and GPU acceleration to automatically generate high-quality labels for automatic driving bird's-eye view panoramic segmentation model training, realizes low-cost acquisition of high-precision bird's-eye view semantic segmentation labels, and lays a foundation for subsequent automatic driving work of bird's-eye view perspective perception.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a label generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation. Background Technology

[0002] The rapid development of autonomous driving technology has placed extremely high demands on the accuracy and reliability of perception systems. Deep learning-based perception models, in particular, rely on large-scale, high-quality training data. Bird's-eye view panoramic segmentation, as a key autonomous driving perception technology, can simultaneously provide semantic understanding of the scene (such as roads and buildings) and the differentiation of dynamic object instances (such as different vehicles and pedestrians), offering rich and intuitive environmental representations for downstream decision-making and planning modules. Therefore, acquiring high-quality labeled data for training bird's-eye view panoramic segmentation models is a crucial step in improving the perception capabilities of autonomous driving systems.

[0003] However, current methods for generating panoramic segmentation training labels for bird's-eye views still have many limitations, making it difficult to efficiently and accurately meet the needs of large-scale model training. Traditional manual annotation methods, especially pixel-level annotation of dense semantic and instance information from a bird's-eye view perspective, are not only costly and time-consuming, but also prone to introducing human error and inconsistencies. Annotators need to perform complex correspondences and interpretations between multiple sensor data (such as camera and LiDAR) to accurately delineate the boundaries of different categories and instances in the bird's-eye view, which is almost infeasible for building large-scale datasets.

[0004] The main drawbacks of the above methods are: inverse perspective transformation is highly sensitive to road surface flatness and camera calibration accuracy, and road labels will be deformed when the road surface is uneven or the calibration is incorrect; the sparsity of LiDAR point clouds, especially for distant or small targets, makes the dynamic target bird's-eye view mask generated by projection not fine enough, which is difficult to meet the requirements of high-quality segmentation training; at the same time, the method still relies on the initial annotation of the image and point cloud (road area and 3D bounding box) and is not fully automated; and the sensor synchronization and calibration accuracy requirements are extremely high, and any deviation will affect the label quality. Summary of the Invention

[0005] The label generation method, device, and storage medium based on multi-frame depth information fusion and bird's-eye view semantic segmentation proposed in this invention can at least solve one of the technical problems in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A label generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation includes the following steps: S100. Generate the depth map and depth confidence map of the center frame using the depth estimation method, and perform system initialization; S200: Generate a 3D point cloud based on the depth map and depth confidence map, and accumulate the point cloud before and after the center frame to obtain the accumulated point cloud; S300: Separate static point cloud and dynamic point cloud from cumulative point cloud, and generate static area bird's-eye view labels and dynamic target instance bird's-eye view labels; S400: Generate bird's-eye view labels based on static area bird's-eye view labels and dynamic target instance bird's-eye view labels.

[0007] Furthermore, the system initialization in step S100 of the present invention includes: environmental check and bird's-eye view grid parameter definition; The bird's-eye view grid parameters are defined as the grid parameters used to create the bird's-eye view canvas based on the user-defined X-axis range, Y-axis range, Z-axis filtering range, and resolution parameters.

[0008] Furthermore, the environment check described in this invention includes: checking the availability and class definition loading of the CUDA environment, CuPy library, and cuML library; The category definition loading process involves loading semantic category information from a predefined category label module, including: Front view semantic segmentation data: a list of attributes for each category; Semantic segmentation mapping data: a mapping from training IDs to label names; Ignore label: Defines the number of the ignore label; Dynamic object IDs: A set of training IDs for dynamic object categories; Static object IDs: A set of training IDs for static object categories; All category IDs: The set of training IDs for all valid categories; Obstacle identifier: A dictionary that defines whether each training number is an "obstacle".

[0009] Furthermore, the method for obtaining the accumulated point cloud in step S200 of the present invention includes: S210, Perform multi-frame data acquisition and point cloud accumulation; S211. Determine the time window based on multiple frames of data; Select the original image of the center frame, and based on the current center frame, determine the frame range for forward and backward data acquisition according to the user-defined time window size; S212, Data loading and verification; A specific frame within the time window; Load the depth map, depth confidence map, pre-annotated panoramic segmentation mask, panoramic segmentation annotation information, and geometric information file for this frame; If any file is missing or the data format is incorrect, skip that frame. S213, Pixel-level 3D Reconstruction and Coordinate Transformation; For each pixel (u,v) in the k-th frame image, read its depth value d and confidence value c; If a pixel's semantic category is valid, its depth value is greater than the minimum threshold, and its confidence level c is greater than or equal to the user-defined minimum confidence threshold, then the pixel is considered valid. The effective pixels (u, v) and their depth d are back-projected onto the 3D points in the camera coordinate system of the k-th frame using the camera intrinsic parameters. The projected point cloud is relatively sparse and almost invisible in a single frame image. Using the camera-to-vehicle extrinsic parameters, the 3D points in the camera coordinate system of the kth frame are transformed to the 3D points in the vehicle coordinate system of the kth frame; Using the inverse extrinsic transformation of the vehicle body to the world coordinate system in the center frame and the pose of the vehicle body to the world coordinate system in the current frame k, calculate the transformation matrix from the vehicle body coordinate system in the current frame k to the vehicle body coordinate system in the center frame. Transform the 3D points in the vehicle coordinate system of the kth frame to the 3D points in the vehicle coordinate system of the center frame using a transformation matrix; Store the 3D points in the vehicle body coordinate system of the center frame together with the original training number, confidence score, original segmentation number and the current frame index k; S214, Point Cloud Accumulation: The effective 3D points obtained from processing all frames within the time window are aggregated to form a cumulative point cloud.

[0010] Furthermore, the static area bird's-eye view label generation method in step S300 of the present invention includes: S311. Extract static point cloud: From the accumulated point cloud, select the points whose training numbers belong to the static object category to obtain the static point cloud and its corresponding training numbers. S312. Project the static point cloud onto the bird's-eye view to obtain the initial sparse projection image; The static point cloud is projected onto a predefined bird's-eye view grid to obtain the coordinates of each point on the bird's-eye view canvas. At the same time, points whose height exceeds the range are filtered out according to the Z-axis range of the bird's-eye view to obtain the initial sparse projection image. S313. Perform morphological processing and class numbering on the initial sparse projection image; Initialize a temporary static category view with the same size as the viewpoint canvas, and fill it with the numbers of the defined ignored labels; Static categories such as structures, natural features, sidewalks, and drivable areas are processed in the following order according to preset priority: Extract the bird's-eye view projection points for each of the above categories; Adjust the size of the dilation kernel based on the number of point clouds in each category to create an elliptical dilation kernel; For structural categories, morphological closing operations are first performed to fill the internal small holes; for other static categories, morphological expansion operations are performed directly. The processed category mask is then filled with the training number of the corresponding category in the area of ​​the temporary static category bird's-eye view that corresponds to the ignored label. S314. Perform connectivity analysis and instance allocation on the merged static category bird's-eye view; For each static category, use 8-connectivity to find connected components; For each found connected region, calculate the area of ​​the connected region; If the area is smaller than the minimum area threshold set by the user, then ignore the area and get the area after removing the instance; Assign a globally unique semantic number to each valid statically connected region.

[0011] Furthermore, the method for generating dynamic target instance bird's-eye view labels in step S300 of the present invention includes: S321. Extracting dynamic category examples from the center frame: Based on the original panoramic segmentation annotation information of the center frame, obtain the set of semantic numbers of all training numbers that belong to dynamic instances; S322. Process the dynamic category instances of the center frame one by one, and extract the filtered 3D point cloud; Iterate through each dynamic instance of the center frame, extract the 3D point cloud belonging to the current instance from the accumulated point cloud, and skip the instance if the number of point clouds is too small. Multi-stage point cloud filtering is performed on the extracted 3D point cloud: First, outliers are statistically removed from the instance point cloud. Then, points within a certain range from the camera's optical center are retained based on the instance category. Finally, the remaining point cloud is clustered using DBSCAN from the cuML library, with the parameters eps and min_samples adaptively adjusted according to the instance category and point cloud density. The largest cluster is selected as the valid point cloud. If the number of points in the largest cluster is too small, the result of distance filtering is used as a fallback. S323. Based on the filtered 3D point cloud, generate the outline and draw the bird's-eye view mask; If the number of filtered point clouds meets the requirements, the filtered 3D point cloud will be projected onto the XY plane of the bird's-eye view. Calculate projection onto bird's-eye view Figure X The convex hull of two-dimensional points in the Y plane is used to connect the calculated vertices of the convex hull and draw a filled polygon on the bird's-eye view canvas, which serves as the bird's-eye view mask for this instance. S324. Record instance information for the bird's-eye view mask: Assign a new globally unique semantic number to each successfully generated dynamic instance bird's-eye view mask. In the panoramic bird's-eye view semantic segmentation map, the pixel values ​​of the dynamic instance bird's-eye view mask region are assigned the corresponding semantic numbers.

[0012] As can be seen from the above technical solution, this invention utilizes multi-frame information and GPU acceleration to automatically generate high-quality labels for training panoramic segmentation models of bird's-eye views for autonomous driving. It achieves high-precision semantic segmentation labels for bird's-eye views at low cost, laying the foundation for subsequent obstacle-sensing autonomous driving work. To address the shortcomings of current bird's-eye view label generation, which relies on high-cost LiDAR and whose accuracy is easily affected by uneven road surfaces, this invention employs refined point cloud processing, customized contour generation strategies for different object categories, and strict parameter configuration to minimize annotation costs while ensuring relatively reliable accuracy. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the single-center frame panoramic segmentation tag generation process of the present invention; Figure 2 Original image; Figure 3 This is the depth map of the Kth frame of the original image; Figure 4 This is the depth confidence map of the original image; Figure 5 A pre-annotated panoramic segmentation mask; Figure 6 This is a schematic diagram of a single-frame point cloud aerial view; Figure 7 A schematic diagram of point cloud accumulation for a single frame; Figure 8 This is a schematic diagram of the initial sparse projection of the point cloud. Figure 9 This is a comparison image showing the effects before and after morphological manipulation; Figure 10 This is a schematic diagram of the aerial view labels for the static area. Figure 11 A diagram illustrating a bird's-eye view label for a dynamic object; Figure 12 This is a diagram illustrating the labeling of an aerial view. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0015] The tag generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation described in this embodiment includes the following steps: S100. Generate the depth map and depth confidence map of the center frame using the depth estimation method, and perform system initialization; S200: Generate a 3D point cloud based on the depth map and depth confidence map, and accumulate the point cloud before and after the center frame to obtain the accumulated point cloud; S300: Separate static point cloud and dynamic point cloud from cumulative point cloud, and generate static area bird's-eye view labels and dynamic target instance bird's-eye view labels; S400: Generate bird's-eye view labels based on static area bird's-eye view labels and dynamic target instance bird's-eye view labels.

[0016] The following provides a detailed explanation of each step: S100. Generate the depth map and depth confidence map of the center frame using the depth estimation method, and perform system initialization; Before starting the label generation process, it is necessary to ensure that an appropriate depth estimation method is used to generate the depth map and depth confidence map of the front view image, and to perform instance-level annotation on the front view image.

[0017] The system performs the following initialization and parameter configuration: 1. Environment check: Check the availability of the CUDA environment, CuPy library, and cuML library.

[0018] Category definition loading: Loads semantic category information from the predefined category label module, including: Front view semantic segmentation data: a list containing attributes for each category (number, name, color, etc.).

[0019] Semantic segmentation mapping data: a mapping from training IDs to label names.

[0020] Ignore label: Defines the number of the ignore label.

[0021] Dynamic object IDs: A set of training IDs for dynamic object categories, for example: {3, 6, 7, 8, 9} (corresponding to vertical obstacles, people, two-wheeled vehicles, cars, and large vehicles).

[0022] Static object IDs: A set of training IDs for static object categories, for example: {0, 1, 2, 4} (corresponding to drivable areas, sidewalks, structures, and natural objects).

[0023] All category numbers: The set of training numbers for all valid categories.

[0024] Obstacle identifier: A dictionary that defines whether each training number is an "obstacle" (i.e., an instance).

[0025] 2. Bird's-eye view grid parameter definition: Creates the grid parameters for the bird's-eye view canvas based on user-defined parameters, including: X-axis range: for example, [-25.0, 25.0] meters.

[0026] Y-axis range: for example, [0, 50.0] meters (the front of the vehicle is the positive direction).

[0027] Z-axis filtering range: for example, [-2.0, 3.0] meters, used to filter point cloud height.

[0028] Bird's-eye view resolution: for example, 0.05 meters per pixel.

[0029] This allows us to calculate the width and height of the bird's-eye view grid.

[0030] In computationally intensive steps such as 3D point cloud reconstruction, coordinate transformation, point cloud filtering (e.g., K-nearest neighbors, DBSCAN clustering), and morphological processing, the graphics processing unit (GPU) is fully utilized to perform parallel acceleration computation through libraries such as CUDA, CuPy, and cuML to improve label generation efficiency.

[0031] S200: Generate a 3D point cloud based on the depth map and depth confidence map, and accumulate the point cloud before and after the center frame to obtain the accumulated point cloud; like Figure 1 As shown, for a given center frame (specified by sequence number and frame number), the process of generating its bird's-eye view panoramic segmentation label mainly includes the following steps: Methods for accumulating point clouds include: S210, multi-frame data acquisition and point cloud accumulation; S211. Determine the time window; like Figure 2 As shown, the original image selected as the center frame is used as a reference. Based on the current center frame, the frame range for forward and backward data acquisition is determined according to the user-defined time window size (e.g., 5 frames).

[0032] S212, Data Loading and Verification: A specific frame within the time window (denoted as frame k): like Figure 3 , Figure 4 , Figure 5 As shown, the depth map, depth confidence map, pre-annotated panoramic segmentation mask map, panoramic segmentation annotation information (including the category number corresponding to each instance number, etc.) and geometric information file (including camera intrinsic parameters, camera to vehicle coordinate system extrinsic parameters, and vehicle to world coordinate system pose) of the frame are loaded.

[0033] If any file is missing or the data format is incorrect, skip that frame.

[0034] S213, Pixel-level 3D Reconstruction and Coordinate Transformation (GPU Accelerated): For each pixel (u,v) in the k-th frame image, read its depth value d and confidence value c.

[0035] A pixel is considered valid if its semantic category is valid (not an ignored label), its depth value is greater than the minimum threshold (e.g., 0.1 meters), and its confidence level c is greater than or equal to the user-defined minimum confidence threshold (e.g., 0.5).

[0036] The effective pixels (u, v) and their depth d are back-projected onto 3D points in the camera coordinate system of the k-th frame using camera intrinsic parameters. The projected point cloud is relatively sparse and almost invisible in a single frame image, such as... Figure 6 As shown.

[0037] By using the camera-to-vehicle extrinsic parameters, the 3D points in the camera coordinate system of the k-th frame are transformed to the 3D points in the vehicle coordinate system of the k-th frame.

[0038] Using the inverse extrinsic transformation of the vehicle body from the center frame to the world coordinate system and the pose of the vehicle body from the current frame k to the world coordinate system, calculate the transformation matrix from the vehicle body coordinate system of the current frame k to the vehicle body coordinate system of the center frame.

[0039] Transform the 3D points in the vehicle body coordinate system of the k-th frame to the 3D points in the vehicle body coordinate system of the center frame using a transformation matrix.

[0040] Store the 3D points in the vehicle body coordinate system of the center frame together with the original training number, confidence level, original segmentation number, and the current frame index k.

[0041] S214, Point Cloud Accumulation: The effective 3D points (transformed to the vehicle body coordinate system of the center frame) obtained from processing all frames within the time window are aggregated to form a cumulative point cloud. The accumulated image is as follows: Figure 7 As shown.

[0042] S300: Separate static point cloud and dynamic point cloud from cumulative point cloud, and generate static area bird's-eye view labels and dynamic target instance bird's-eye view labels; The method for generating labels for static area bird's-eye view images is as follows: S311. Extracting Static Point Cloud: Filter out the points in the cumulative point cloud whose training numbers belong to the static object category to obtain the static point cloud and its corresponding training numbers.

[0043] S312. Project the static point cloud onto the bird's-eye view to obtain the initial sparse projection image (GPU accelerated). like Figure 8As shown, the static point cloud is projected onto a predefined bird's-eye view grid to obtain the (u_bird's-eye view, v_bird's-eye view) coordinates of each point on the bird's-eye view canvas. At the same time, points whose heights exceed the range are filtered out according to the Z-axis range of the bird's-eye view to obtain the initial sparse projection image.

[0044] S313. Perform morphological processing and class number assignment on the initial sparse projection image (GPU accelerated). Initialize a temporary static category view with the same size as the viewpoint canvas, and fill it with the numbers of the defined ignored labels; Static categories of structures, natural features, sidewalks, and drivable areas are processed in the following order according to preset priorities (e.g., structures (2) > natural features (4) > sidewalks (1) > drivable areas (0)): Extract the bird's-eye view projection points for each of the above categories.

[0045] The size of the dilation kernel is adjusted based on the number of point clouds in each category (the base kernel size is 19 pixels, an odd number; increase the kernel size if there are fewer points, and decrease it if there are more points).

[0046] like Figure 9 As shown in (a), a binary bird's-eye view is created for this category, with the projection point position set to 1; Create an elliptical expansion kernel; For structural categories, morphological closing operations are first performed to fill the internal small holes; for other static categories, morphological dilation operations are performed directly. All of the above operations are executed on the GPU using CuPy. like Figure 9 As shown in (b), the processed category mask is filled with the training number of the category only in the area corresponding to the ignored label in the temporary static category bird's-eye view.

[0047] S314. Perform connectivity analysis and instance allocation on the merged static category bird's-eye view; like Figure 10 As shown in (a), for the merged static category bird's-eye view generated in the previous step, 8-connectivity is used to find connected regions for each static category (by priority); For each connected region found, calculate its area; If the area is smaller than the user-defined minimum area threshold (e.g., 50 pixels, which can be adjusted according to the category; for example, roads can be smaller, while structures need to be larger), then the area is ignored, and the final area after removing instances is as follows: Figure 10 As shown in (b); Assign a globally unique semantic number (incrementing from 1) to each valid statically connected region.

[0048] In the final panoramic bird's-eye view semantic segmentation map (initially all 0s), the pixel values ​​of the connected region are assigned the corresponding semantic numbers.

[0049] Record the information of the semantic ID into the semantic segmentation list, including the category ID, whether it is an obstacle, the bounding box, and the area.

[0050] The method for generating labels for dynamic target instance bird's-eye view images is as follows: S321. Extracting dynamic category examples from the center frame: Based on the original panoramic segmentation annotation information of the center frame, obtain the set of semantic IDs for all training IDs that belong to dynamic instances.

[0051] S322. Process the dynamic category instances of the center frame one by one, and extract the filtered 3D point cloud; like Figure 11 As shown, the 3D point cloud belonging to the current instance (matching the semantic number and with the frame index being the center frame) is extracted from the accumulated point cloud.

[0052] If the number of points is too low (for example, fewer than 3 points for a utility pole and fewer than 5 points for others), the instance is skipped.

[0053] Special category handling: Telegraph pole: Find the point closest to the camera, and extend it outward by a fixed distance (e.g., 0.15 meters) along the line connecting that point and the origin (camera). Project this extended point onto the bird's-eye view, and use this point as the center to draw a filled circle with a fixed radius (e.g., 3 pixels) as the bird's-eye view mask for this telephone pole instance.

[0054] Pedestrian: Find the point closest to the camera (front end point), and extend it outward by a fixed distance (e.g., 0.2 meters) along the line connecting this point and the origin (camera) to obtain the rear end point. Project these two points onto the bird's-eye view, and use the line segment formed by these two points as the center line of the major axis to draw a filled rotating rectangle with a fixed width (e.g., 8 pixels) as the bird's-eye view mask for this pedestrian instance.

[0055] General dynamic target processing (e.g., vehicles): Multi-stage point cloud filtering (GPU accelerated) is performed on the extracted 3D point cloud: First, outlier removal is performed on the instance point cloud using parameters such as nb_neighbors=10 and std_ratio=2.5.

[0056] Then, based on the instance category (such as two-wheeled vehicle, car, large vehicle), points within a certain range from the camera's optical center are retained (for example, for cars, points within a range of the nearest point distance +5.0 meters are retained).

[0057] Finally, the remaining point cloud is clustered using DBSCAN from the cuML library. The parameters eps and min_samples are adaptively adjusted based on the instance category and point cloud density (e.g., for cars, eps=2.8). The largest cluster is selected as the valid point cloud. If the number of points in the largest cluster is too small (e.g., less than 15% of the instance points or less than 5 points), the result of distance filtering is used as a fallback.

[0058] Dynamic instances identified from the center frame are first refined in multiple stages using the cumulative point cloud of their associated multi-frames, including statistical outlier removal, nearest-neighbor point retention based on instance category, and DBSCAN clustering. Then, customized contour generation strategies are adopted for different types of dynamic targets. For example, for general targets, the convex hull of their refined point cloud on the bird's-eye view plane is calculated as a mask, while for slender targets (such as utility poles and pedestrians), a simplified geometric model (such as a circle or a rotated rectangle) based on nearest-neighbor extension is used to generate a mask.

[0059] A separate processing strategy is adopted for static regions (such as roads and sidewalks) and dynamic instances (such as vehicles and pedestrians) in the scene. For the static part in the accumulated point cloud, after projecting it onto the bird's-eye view (bird's-eye view) plane, morphological operations (such as dilation and closing operations) based on category priority and adaptive kernel size are used to densify and optimize the boundaries. Instance identifiers are assigned to each independent static region block in combination with connected component analysis.

[0060] S323. Based on the filtered 3D point cloud, generate the outline and draw the bird's-eye view mask; If the number of filtered point clouds meets the requirements (e.g., greater than or equal to 5 points), the filtered 3D point cloud is projected onto the XY plane of the bird's-eye view.

[0061] Calculate projection onto bird's-eye view Figure X Convex hull of two-dimensional points in the Y-plane; Connect the vertices of the calculated convex hull (polygon) and draw the filled polygon on the bird's-eye view canvas as the bird's-eye view mask for this instance.

[0062] S324. Record instance information for the bird's-eye view mask: Assign a new globally unique semantic number to each successfully generated dynamic instance bird's-eye view mask.

[0063] In the panoramic bird's-eye view semantic segmentation map, the pixel values ​​of the masked region are assigned the corresponding semantic numbers.

[0064] Record the information of the semantic ID into the semantic segmentation list, including the category ID, whether it is an obstacle, the bounding box, and the area.

[0065] S400: Generate bird's-eye view labels based on static area bird's-eye view labels and dynamic target instance bird's-eye view labels; Step S4: Merge label output 1. Save NPY file: Save the generated panoramic bird's-eye view semantic segmentation map (a two-dimensional array containing instance numbers) as a .npy format file.

[0066] 2. Save JSON file: Save the semantic segmentation list and related metadata (sequence number, frame number, processing parameters, etc.) as a .json format annotation file.

[0067] 3. Save the visualization as a PNG file: Based on the category and instance number in the semantic segmentation list, using predefined or random colors, render the panoramic bird's-eye view semantic segmentation map into a color visualization image and save it as a .png file. The final result is as follows: Figure 12 As shown.

[0068] In summary, this embodiment, through the steps described above, effectively utilizes multi-frame information and GPU acceleration to automatically generate high-quality labels for training a panoramic segmentation model of bird's-eye view for autonomous driving. It achieves high-precision semantic segmentation labels for bird's-eye view images at low cost, laying the foundation for subsequent obstacle-sensing autonomous driving work. To address the shortcomings of current bird's-eye view label generation, which relies on high-cost LiDAR and is easily affected by uneven road surfaces, this embodiment employs refined point cloud processing, customized contour generation strategies for different object categories, and strict parameter configuration to minimize annotation costs while ensuring relatively reliable accuracy.

[0069] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the tag generation methods based on multi-frame depth information fusion and bird's-eye view semantic segmentation in the above embodiments.

[0070] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0071] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive, such as a Solid State Disk (SSD)).

[0072] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0073] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A label generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation, characterized in that, Includes the following steps: S100. Generate the depth map and depth confidence map of the center frame using the depth estimation method, and perform system initialization; S200: Generate a 3D point cloud based on the depth map and depth confidence map, and accumulate the point cloud before and after the center frame to obtain the accumulated point cloud; S300: Separate static point cloud and dynamic point cloud from cumulative point cloud, and generate static area bird's-eye view labels and dynamic target instance bird's-eye view labels; S400: Generate bird's-eye view labels based on static area bird's-eye view labels and dynamic target instance bird's-eye view labels; The method for obtaining the cumulative point cloud in step S200 includes: S210, Perform multi-frame data acquisition and point cloud accumulation; S211. Determine the time window based on multiple frames of data; Select the original image of the center frame, and based on the current center frame, determine the frame range for forward and backward data acquisition according to the user-defined time window size; S212, Data loading and verification; Within a certain frame of the time window Load the depth map, depth confidence map, pre-annotated panoramic segmentation mask, panoramic segmentation annotation information, and geometric information file for this frame; If any file is missing or the data format is incorrect, skip that frame. S213, Pixel-level 3D Reconstruction and Coordinate Transformation; For each pixel (u,v) in the k-th frame image, read its depth value d and confidence value c; If a pixel's semantic category is valid, its depth value is greater than the minimum threshold, and its confidence level c is greater than or equal to the user-defined minimum confidence threshold, then the pixel is considered valid. The effective pixels (u, v) and their depth d are back-projected onto the 3D points in the camera coordinate system of the k-th frame using the camera intrinsic parameters. The projected point cloud is relatively sparse and almost invisible in a single frame image. Using the camera-to-vehicle extrinsic parameters, the 3D points in the camera coordinate system of the kth frame are transformed to the 3D points in the vehicle coordinate system of the kth frame; Using the inverse extrinsic transformation of the vehicle body to the world coordinate system in the center frame and the pose of the vehicle body to the world coordinate system in the current frame k, calculate the transformation matrix from the vehicle body coordinate system in the current frame k to the vehicle body coordinate system in the center frame. Transform the 3D points in the vehicle coordinate system of the kth frame to the 3D points in the vehicle coordinate system of the center frame using a transformation matrix; Store the 3D points in the vehicle body coordinate system of the center frame together with the original training number, confidence score, original segmentation number and the current frame index k; S214, Point Cloud Accumulation: The effective 3D points obtained from processing all frames within the time window are aggregated to form a cumulative point cloud.

2. The tag generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation according to claim 1, characterized in that, System initialization in step S100 includes: environmental checks and bird's-eye view grid parameter definition; The bird's-eye view grid parameters are defined as the grid parameters used to create the bird's-eye view canvas based on the user-defined X-axis range, Y-axis range, Z-axis filtering range, and resolution parameters.

3. The tag generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation according to claim 2, characterized in that, The environment check includes checking the availability and class definition loading of the CUDA environment, CuPy library, and cuML library; The category definition loading process involves loading semantic category information from a predefined category label module, including: Front view semantic segmentation data: a list of attributes for each category; Semantic segmentation mapping data: a mapping from training IDs to label names; Ignore label: Defines the number of the ignore label; Dynamic object IDs: A set of training IDs for dynamic object categories; Static object IDs: A set of training IDs for static object categories; All category IDs: The set of training IDs for all valid categories; Obstacle identifier: A dictionary that defines whether each training number is an "obstacle".

4. The tag generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation according to claim 1, characterized in that, The method for generating static area bird's-eye view labels in step S300 includes: S311. Extract static point cloud: From the accumulated point cloud, select the points whose training numbers belong to the static object category to obtain the static point cloud and its corresponding training numbers. S312. Project the static point cloud onto the bird's-eye view to obtain the initial sparse projection image; The static point cloud is projected onto a predefined bird's-eye view grid to obtain the coordinates of each point on the bird's-eye view canvas. At the same time, points whose height exceeds the range are filtered out according to the Z-axis range of the bird's-eye view to obtain the initial sparse projection image. S313. Perform morphological processing and class numbering on the initial sparse projection image; Initialize a temporary static category view with the same size as the viewpoint canvas, and fill it with the numbers of the defined ignored labels; Static categories such as structures, natural features, sidewalks, and drivable areas are processed in the following order according to preset priority: Extract the bird's-eye view projection points for each of the above categories; Adjust the size of the dilation kernel based on the number of point clouds in each category to create an elliptical dilation kernel; For structural categories, morphological closing operations are first performed to fill the internal small holes; for other static categories, morphological expansion operations are performed directly. The processed category mask is then filled with the training number of the corresponding category in the area of ​​the temporary static category bird's-eye view that corresponds to the ignored label. S314. Perform connectivity analysis and instance allocation on the merged static category bird's-eye view; For each static category, use 8-connectivity to find connected components; For each found connected region, calculate the area of ​​the connected region; If the area is smaller than the minimum area threshold set by the user, then ignore the area and get the area after removing the instance; Assign a globally unique semantic number to each valid statically connected region.

5. The tag generation method based on multi-frame depth information fusion and bird's-eye view semantic segmentation according to claim 1, characterized in that, The method for generating dynamic target instance bird's-eye view labels in step S300 includes: S321. Extracting dynamic category examples from the center frame: Based on the original panoramic segmentation annotation information of the center frame, obtain the set of semantic numbers of all training numbers that belong to dynamic instances; S322. Process the dynamic category instances of the center frame one by one, and extract the filtered 3D point cloud; Iterate through each dynamic instance of the center frame, extract the 3D point cloud belonging to the current instance from the accumulated point cloud, and skip the instance if the number of point clouds is too small. Multi-stage point cloud filtering is performed on the extracted 3D point cloud: First, outliers are statistically removed from the instance point cloud. Then, points within a certain range from the camera's optical center are retained based on the instance category. Finally, the remaining point cloud is clustered using DBSCAN from the cuML library, with the parameters eps and min_samples adaptively adjusted according to the instance category and point cloud density. The largest cluster is selected as the valid point cloud. If the number of points in the largest cluster is too small, the result of distance filtering is used as a fallback. S323. Based on the filtered 3D point cloud, generate the outline and draw the bird's-eye view mask; If the number of filtered point clouds meets the requirements, the filtered 3D point cloud will be projected onto the XY plane of the bird's-eye view. Calculate the convex hull of the two-dimensional points projected onto the XY plane of the bird's-eye view, connect the vertices of the calculated convex hull, and draw a filled polygon on the bird's-eye view canvas as the bird's-eye view mask for this instance; S324. Record instance information for the bird's-eye view mask: Assign a new globally unique semantic number to each successfully generated dynamic instance bird's-eye view mask. In the panoramic bird's-eye view semantic segmentation map, the pixel values ​​of the dynamic instance bird's-eye view mask region are assigned the corresponding semantic numbers.

Images