3D point cloud segmentation method and device, electronic equipment and storage medium

Abstract

The invention provides a 3D point cloud segmentation method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining 3D point cloud data and image data corresponding to the same sampling moment and in the same scene; converting the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data; the pixel coordinates of each 2D mask are converted into a 3D point cloud coordinate system, so that the corresponding view cone space of each 2D mask in the 3D point cloud coordinate system is determined; and for each 2D mask, segmenting 3D point cloud data in a view cone space corresponding to the mask from the 3D point cloud data to determine a point cloud block where the object is located. Therefore, 3D point cloud data does not need to be manually labeled, point cloud segmentation can be guided only by using the mask generated by the image, and the manual labeling cost is reduced; the method does not depend on specific sensor parameters, adapts to different hardware configurations, reduces the adaptation cost of different platforms, and enhances the generalization ability.

Description

Technical Field

[0001] One or more embodiments of the present disclosure relate to the field of point cloud segmentation technology, and in particular, to a 3D point cloud segmentation method, device, electronic device, and storage medium. Background Art

[0002] In related technologies, 3D point cloud segmentation is the process of identifying and separating different objects (such as vehicles and pedestrians) from disordered and sparse 3D point cloud data. Through high-precision 3D point cloud segmentation, the system can perceive the surrounding environment in real time and classify and track various dynamic and static objects, providing reliable data support for decision-making and control.

[0003] In some embodiments, 3D point cloud segmentation technology relies on large-scale, manually annotated 3D datasets for model training to achieve object recognition and separation. Furthermore, to adapt to specific sensor configurations (such as the number of LiDAR lines and the camera's internal and external parameters), the model must be customized for specific hardware parameters to ensure segmentation accuracy meets actual requirements.

[0004] However, on the one hand, this method has extremely high costs for manual labeling, and it is difficult for labeling to cover a variety of scenarios and object types, resulting in limitations in model training; on the other hand, the model is too dependent on sensor configuration. When the vehicle platform is changed (different vehicles have different sensor models and installation locations), the model must be retrained, increasing the adaptation cost. Summary of the Invention

[0005] In view of this, the present disclosure provides a 3D point cloud segmentation method, the method comprising: Obtain 3D point cloud data and image data corresponding to the same sampling time and the same scene; Converting the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data; Convert the pixel coordinates of each 2D mask into a 3D point cloud coordinate system, thereby determining the corresponding viewing cone space of each 2D mask in the 3D point cloud coordinate system; For each 2D mask, 3D point cloud data located in the viewing cone space corresponding to the mask is segmented from the 3D point cloud data to determine the point cloud block where the object is located.

[0006] Optionally, converting the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data, includes: Conditioning the visual base model to mask generation mode; Inputting the image data into a visual base model to obtain a set of 2D masks output by the visual base model; Each 2D mask corresponds to a binary image, and the binary image identifies a contour area of ​​an object contained in the image data.

[0007] Optionally, converting the pixel coordinates of each 2D mask into a 3D point cloud coordinate system, thereby determining a viewing cone space corresponding to each 2D mask in the 3D point cloud coordinate system, includes: For each 2D mask, extract its boundary pixel coordinates; Converting the boundary pixel coordinate set to a ray direction vector in an image sensor coordinate system according to an intrinsic parameter matrix of the image sensor; According to the extrinsic parameter matrix from the image sensor to the point cloud sensor, the ray direction vector is converted to a 3D point cloud coordinate system, and the converted ray direction vector is connected with the optical center of the image sensor as the vertex to obtain the corresponding viewing cone space of each 2D mask in the 3D point cloud coordinate system.

[0008] Optionally, the method is applied to a system, wherein the system maintains a voxel database, and the method further comprises: According to the voxel coordinates of the point cloud block where each segmented object is located, the point cloud block where the historical object with which it overlaps is searched in the voxel database; For each segmented object, the correlation between the point cloud block where the object is located and the point cloud block where each historical object with which the object overlaps is calculated one by one, and then the point cloud block where the object is located is updated to the voxel database.

[0009] Optionally, for each segmented object, calculating the correlation between the point cloud block where the object is located and the point cloud block where each historical object having an overlapping portion is located, and then adding the point cloud block where the object is located to the voxel database, includes: For each segmented object, calculate the correlation between the point cloud block where the object is located and the point cloud block where each historical object that overlaps with it is located. If the correlation between the point cloud block where the object is located and the point cloud block where a target historical object with which the object is located overlaps satisfies a preset condition, the point cloud block where the object is located and the point cloud block where the target historical object is located are merged in the voxel database; If the correlation between the point cloud block where the object is located and the point cloud block where each target historical object having an overlapping portion thereof is located does not meet a preset condition, the point cloud block where the object is located is newly added to the voxel database.

[0010] Optionally, each 2D mask has a predicted intersection-over-union score when generated, and the method further comprises: The historical object is regarded as an active object in the voxel database if the correlation between the point cloud block where any segmented object is located and the point cloud block where the historical object is located meets a preset condition; The confidence level of the 3D point cloud segmentation result is obtained by summing the intersection-over-union (IoU) scores of the 2D masks of the segmented objects corresponding to each active object and dividing the summed IoU score by the total number of active objects.

[0011] Optionally, the system further maintains a 3D segmentation model, wherein the 3D segmentation model is used to segment the point cloud block where the object is located in the 3D point cloud data according to the 2D mask. The method further includes: If the confidence level of the 3D point cloud segmentation result is lower than a first preset threshold, the active object associated with the point cloud block where the object corresponding to the 2D mask whose intersection-over-union score exceeds a second preset threshold is located, and the point cloud block where the active object is located are used as pseudo labels; Based on the pseudo labels, a lightweight trainable submodule of the 3D segmentation model is fine-tuned online.

[0012] The present disclosure also provides a 3D point cloud segmentation device, the device comprising: An acquisition unit, configured to acquire 3D point cloud data and image data corresponding to the same sampling moment and the same scene; a first conversion unit, configured to convert the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data; a second conversion unit, configured to convert the pixel coordinates of each 2D mask into a 3D point cloud coordinate system, thereby determining a viewing cone space corresponding to each 2D mask in the 3D point cloud coordinate system; The segmentation unit is used to segment the 3D point cloud data located in the viewing cone space corresponding to each 2D mask in the 3D point cloud data to determine the point cloud block where the object is located.

[0013] The present disclosure further provides an electronic device, comprising a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0014] The present disclosure also provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the above method is implemented.

[0015] As can be seen, the disclosed technical solution acquires 3D point cloud data and image data corresponding to the same sampling moment and scene, and uses the mapping relationship between the 2D mask derived from the image data and the 3D viewing cone space to filter the object point cloud using the constraints of the corresponding viewing cone space. The core of this method is to use the contour information (2D mask) of the object in the image to constrain the segmentation range of the 3D point cloud, avoiding the reliance on large-scale annotated data and sensor configuration required for simple point cloud segmentation.

[0016] Through the above methods, the present disclosure realizes that there is no need to manually label 3D point cloud data, and only the mask generated by the image can be used to guide the point cloud segmentation, thereby reducing the cost of manual labeling; moreover, this method does not rely on specific sensor parameters and is adaptable to different hardware configurations, reducing the adaptation cost of the 3D point cloud segmentation model on different platforms and enhancing the generalization ability of the technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 is a flowchart of a 3D point cloud segmentation method shown in an exemplary embodiment; Figure 2 is a flowchart of another 3D point cloud segmentation method shown in an exemplary embodiment; Figure 3 is a schematic diagram of online adjustment of a 3D segmentation model according to an exemplary embodiment; Figure 4 is a hardware structure diagram of an electronic device shown in an exemplary embodiment; Figure 5 It is a block diagram of a 3D point cloud segmentation device shown in an exemplary embodiment. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0020] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this disclosure. In some other embodiments, the method may include more or fewer steps than those described in this disclosure. In addition, a single step described in this disclosure may be broken down into multiple steps for description in other embodiments; and multiple steps described in this disclosure may be combined into a single step for description in other embodiments.

[0021] In related technologies, 3D point cloud segmentation is the process of identifying and separating different objects (such as vehicles and pedestrians) from disordered and sparse 3D point cloud data. Through high-precision 3D point cloud segmentation, the system can perceive the surrounding environment in real time and classify and track various dynamic and static objects, providing reliable data support for decision-making and control.

[0022] In some embodiments, 3D point cloud segmentation technology relies on large-scale, manually annotated 3D datasets for model training, enabling the identification and separation of different objects. Furthermore, to adapt to specific sensor configurations (such as the number of LiDAR lines and the camera's internal and external parameters), the model requires customized training based on specific hardware parameters to ensure segmentation accuracy meets actual requirements.

[0023] However, on the one hand, this method has extremely high costs for manual labeling, and it is difficult for labeling to cover a variety of scenarios and object types, resulting in limitations in model training; on the other hand, the model is too dependent on sensor configuration. When the vehicle platform is changed (different vehicles have different sensor models and installation locations), the model must be retrained, increasing the adaptation cost.

[0024] In view of this, the present disclosure aims to propose a technical solution for converting the 2D mask generated by the image into a 3D point cloud cone space, and then segmenting the corresponding point cloud blocks in the cone space, thereby realizing 3D point cloud segmentation without the need for manual annotation of training data.

[0025] This technical solution first obtains 3D point cloud data and image data corresponding to the same sampling moment and the same scene; then, the image data is converted into a set of 2D masks, each 2D mask identifying the contour area of ​​an object contained in the image data; further, the pixel coordinates of each 2D mask are converted into a 3D point cloud coordinate system, thereby determining the corresponding viewing cone space of each 2D mask in the 3D point cloud coordinate system; finally, for each 2D mask, the 3D point cloud data located in the viewing cone space corresponding to the mask is segmented from the 3D point cloud data to determine the point cloud block where the object is located.

[0026] For example, consider a self-driving vehicle traveling along a road filled with pedestrians and various vehicles. To ensure safety and make accurate driving decisions, the vehicle needs to accurately identify all objects in its surroundings and their locations. The vehicle's onboard LiDAR and forward-facing camera simultaneously collect data from the road ahead. The LiDAR outputs 3D point cloud data containing the 3D coordinates (x, y, z) of objects on the road, such as vehicles, pedestrians, and guardrails. The camera outputs image data representing the 2D visual outlines of these objects. The vehicle system (hereafter referred to as the system) then feeds the camera image into a pre-trained basic vision model, such as the Segment Anything Model (SAM). This generates three 2D masks, one for each vehicle, one for each pedestrian, and one for each guardrail. Each mask represents the outline of an object in the form of a binary image (pixels within the mask are 1, and background pixels are 0). Furthermore, the system uses the camera's intrinsic and extrinsic parameters to convert the three 2D masks from pixel coordinates to a 3D point cloud coordinate system, obtaining the corresponding viewing cone space for each mask. Finally, the system segments the 3D point cloud data corresponding to the car's 2D mask within the viewing cone space that encloses the car (such as the point cloud within the range of 5-15 meters for x, -2-2 meters for y, and 0-1.5 meters for z) from the 3D point cloud data to obtain the point cloud block corresponding to the car. Similarly, the system segmented the corresponding pedestrian point cloud blocks and guardrail point cloud blocks through the 2D masks and viewing cone space of pedestrians and guardrails, thereby completing the 3D point cloud segmentation in this scene.

[0027] As can be seen, the disclosed technical solution acquires 3D point cloud data and image data corresponding to the same sampling moment and scene, and uses the mapping relationship between the 2D mask derived from the image data and the 3D viewing cone space to filter the object point cloud using the constraints of the corresponding viewing cone space. The core of this method is to use the contour information (2D mask) of the object in the image to constrain the segmentation range of the 3D point cloud, avoiding the reliance on large-scale annotated data and sensor configuration required for simple point cloud segmentation.

[0028] Through the above methods, the present disclosure realizes that there is no need to manually label 3D point cloud data, and only the mask generated by the image can be used to guide the point cloud segmentation, thereby reducing the cost of manual labeling; moreover, this method does not rely on specific sensor parameters and is adaptable to different hardware configurations, reducing the adaptation cost of the 3D point cloud segmentation model on different platforms and enhancing the generalization ability of the technology.

[0029] The present disclosure is described below through specific embodiments in combination with specific application scenarios.

[0030] See Figure 1 , Figure 1 FIG. 1 is a flow chart of a 3D point cloud segmentation method according to an exemplary embodiment. The method may perform the following steps: Step 102: Acquire 3D point cloud data and image data corresponding to the same sampling moment and the same scene.

[0031] For example, the lidar and front camera on a vehicle simultaneously collect data on the road ahead at a certain moment. The 3D point cloud data output by the lidar contains the 3D coordinates (x, y, z) of objects such as vehicles, pedestrians, and guardrails on the road, and the image data output by the camera presents the 2D visual outlines of these objects.

[0032] The 3D point cloud data and image data corresponding to the same sampling time can be collected at the same time, or collected at different times but converted to the same time. The core feature of a 3D point cloud is its 3D spatial distribution, which is used to describe the three-dimensional structure of the real world. Each point contains 3D spatial coordinates (x, y, z), and some points may also contain additional information such as color and reflection intensity. These points are collected by 3D sensors such as lidar and depth cameras to reflect the 3D form of objects in the scene. Image data is 2D visual information collected by the camera, including the color, texture, and 2D outline of the object.

[0033] Step 104: Convert the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data.

[0034] For example, the system inputs camera images into a pre-trained visual basic model, such as Segment AnythingModel (SAM), to generate three 2D masks, one for a car, one for a pedestrian, and one for a guardrail. Each mask marks the outline of the object in the form of a binary image (the pixel value in the mask is 1 and the background is 0).

[0035] The number of objects in the image is equal to the number of 2D masks, and each 2D mask is a binary image used to identify the outline of a single object in the image. The object (1) is distinguished from the background (0) by pixel values. The image data can be converted into a 2D mask by using a pre-trained visual basic model, which supports automatic recognition of all objects in the image and generates a 2D mask without manual annotation, and is adaptable to a variety of scenarios. In addition to using the visual basic model to generate a 2D mask, it is also possible to identify edges in the image by applying an edge detection algorithm, and then use contour tracing technology to determine the boundary of the object, or use methods such as superpixel segmentation, which are not limited by the present disclosure.

[0036] Step 106: Convert the pixel coordinates of each 2D mask into a 3D point cloud coordinate system, thereby determining the viewing cone space corresponding to each 2D mask in the 3D point cloud coordinate system.

[0037] For example, the system uses the camera's internal parameters (including focal length, principal point position, etc.) and external parameters (describing the camera's position and orientation relative to other sensors or the world coordinate system) to convert three 2D masks corresponding to a car, a pedestrian, and a guardrail from the pixel coordinate system to the 3D point cloud coordinate system, and obtain the viewing cone space corresponding to each mask.

[0038] Among them, the viewing cone space refers to a three-dimensional area defined by the camera position, mask boundary points and depth information. It represents the spatial range that the object corresponding to the 2D mask may occupy in the 3D space. In order to accurately define this area, in addition to relying on the internal and external parameters of the camera, depth estimation technology can also be combined. For example, learning-based methods such as convolutional neural networks can be used to predict the depth value of each pixel in the image, or hardware devices such as structured light and time of flight can be used to directly measure the depth. Once the depth information of each pixel in the 2D mask is obtained, the 2D mask can be converted into a point set in 3D space through the perspective projection inverse transformation formula, and the 3D point cloud data located in the viewing cone space can be further filtered out.

[0039] Step 108: For each 2D mask, segment the 3D point cloud data located in the viewing cone space corresponding to the mask from the 3D point cloud data to determine the point cloud block where the object is located.

[0040] For example, based on the 2D mask of a car, the system defines a viewing cone encompassing the car, with an x-axis range of 5 to 15 meters (representing distance along the road), a y-axis range of -2 to 2 meters (representing lateral offset of the vehicle), and a z-axis range of 0 to 1.5 meters (representing height). The system then segments the 3D point cloud data within this viewing cone, classifying all 3D point cloud data within this range as belonging to the car. Similarly, using the 2D masks and viewing cone of pedestrians and guardrails, the system can segment the corresponding pedestrian point cloud blocks and guardrail point cloud blocks, thus completing the 3D point cloud segmentation for this scene.

[0041] By filtering out 3D point cloud data that falls within the viewing cone, the system can accurately extract the 3D point cloud blocks corresponding to the 2D mask. Furthermore, efficient indexing structures, such as KD trees or voxel hashing, can be used to accelerate the point cloud query process. These methods can quickly find points within a specific area within large amounts of point cloud data, significantly improving processing efficiency.

[0042] In one embodiment shown, the image data is converted into a set of 2D masks, each 2D mask identifying a contour area of ​​an object contained in the image data, including: adjusting the visual basis model to a mask generation mode; inputting the image data into the visual basis model to obtain a set of 2D masks output by the visual basis model; wherein each 2D mask corresponds to a binary image, and the binary image identifies a contour area of ​​an object contained in the image data.

[0043] For example, a vehicle's front-facing camera captures road data ahead. The camera's output image data represents 2D visual outlines of objects on the road, such as vehicles, pedestrians, and guardrails. This image is fed into a pre-trained vision-based model (such as the Segment Anything Model (SAM)) set to mask generation mode. The SAM model analyzes the input image through its internal mechanisms, automatically identifying and segmenting different objects, and generating a set of 2D masks. Each 2D mask is a binary image that precisely marks the outline of a specific object in the image. For example, for a car, the SAM model generates a corresponding 2D mask. This mask is a binary image where pixel values ​​of 1 represent the car's location, while pixel values ​​of 0 represent background or other non-target objects. The same process is applied to other objects, such as pedestrians and guardrails, resulting in a separate 2D mask for each detected object.

[0044] The Visual Base Model (SAM) is a deep learning model pre-trained on large, diverse datasets. It possesses strong generalization capabilities and can be used for a variety of visual tasks without extensive additional training. The SAM model incorporates a built-in mask encoder and a hint decoder. The former extracts image features, while the latter generates corresponding segmentation masks based on user-provided hints (such as points and boxes). This design enables the SAM model to accurately segment various objects in images even without explicit annotations.

[0045] In one embodiment shown, the pixel coordinates of each 2D mask are converted into a 3D point cloud coordinate system to determine the viewing cone space corresponding to each 2D mask in the 3D point cloud coordinate system, including: for each 2D mask, extracting its boundary pixel coordinate set; according to the intrinsic parameter matrix of the image sensor, converting the boundary pixel coordinate set into a ray direction vector in the image sensor coordinate system; according to the extrinsic parameter matrix from the image sensor to the point cloud sensor, converting the ray direction vector to the 3D point cloud coordinate system, and connecting the converted ray direction vectors with the optical center of the image sensor as the vertex to obtain the viewing cone space corresponding to each 2D mask in the 3D point cloud coordinate system.

[0046] For example, the system extracts the boundary pixels of the car's 2D mask (e.g., the coordinates of the upper left corner (100, 200) and the lower right corner (300, 400) of the car's silhouette). These pixel coordinates are then converted into ray direction vectors in the camera coordinate system using the camera's intrinsic parameter matrix. These rays represent lines originating from the camera's optical center and pointing toward the car's outline boundaries. Specifically, each pixel corresponds to a ray originating from the camera's optical center, with the direction of this ray calculated using the intrinsic parameter matrix. Next, the system uses the extrinsic parameter matrix between the lidar and camera (rotation and translation parameters describing their relative position and orientation) to transform these ray direction vectors into the lidar's 3D point cloud coordinate system. This process takes into account the position and orientation of the camera (image sensor) relative to the lidar (point cloud sensor), ensuring that the ray direction vectors are accurately located in the point cloud coordinate system. Finally, all converted ray direction vectors are connected, with the camera's optical center as the vertex, to form a viewing cone encompassing the car. This viewing cone defines the region in the 3D point cloud data corresponding to the 2D mask.

[0047] Among them, image sensors include but are not limited to cameras and depth cameras, and point cloud sensors include but are not limited to lidars, 3D scanners, etc. The intrinsic parameter matrix describes the internal geometric characteristics of the camera, mainly including the focal length and principal point coordinates, which are used to convert pixel points on the 2D image into direction vectors in 3D space. Through the intrinsic parameter matrix, the system can calculate the direction of each ray, that is, the direction that is emitted from the optical center of the camera and passes through a specific pixel point on the image plane. The extrinsic parameter matrix describes the position and posture of the image sensor relative to the point cloud sensor. Through the extrinsic parameter matrix, the system can accurately map the objects in the image sensor to the corresponding positions in the 3D point cloud coordinate system.

[0048] In one embodiment shown, the method is applied to a system that maintains a voxel database. The method also includes: searching the voxel database for point cloud blocks containing historical objects that overlap with the point cloud blocks of each segmented object based on the voxel coordinates of the point cloud blocks where each segmented object is located; for each segmented object, calculating the degree of association between the point cloud block where the object is located and the point cloud blocks where each historical object that overlaps with the point cloud block is located, and then updating the point cloud block where the object is located to the voxel database.

[0049] For example, the system uses a 3D segmentation model to identify and segment multiple objects in the current scene, such as cars, pedestrians, and guardrails, and determines the corresponding point cloud blocks for each object. The system then searches a pre-maintained voxel database for historical objects that overlap with these objects based on the voxel coordinates of these point cloud blocks. Specifically, if a car is detected in the current frame, the system searches the voxel database for objects in previous frames that overlap with the car's position. For each historical object found, the system calculates the correlation between the current car's point cloud block and the historical object's point cloud block. This correlation is assessed using a variety of factors, including spatial overlap, semantic feature similarity, and geometric feature similarity. Finally, based on the high correlation between the current car and a historical object, the system determines whether the car represents observations of the same historical object at different times. The system then updates the information about the current car's point cloud block to the corresponding historical object record in the voxel database. This process ensures that the system continuously tracks and updates the status of each object in the environment.

[0050] Among them, the voxel database is a data structure used to store object information in 3D space. It divides the space into several small cubes (i.e., voxels) and manages and queries the data within the voxels through an indexing mechanism. The voxel database can be implemented based on a hash table or an octree to optimize storage and retrieval efficiency. The spatial overlap rate refers to the degree of overlap between two point cloud blocks in the voxel space, usually measured by the intersection-over-union ratio. Semantic feature similarity is usually achieved by extracting the feature vectors of two point cloud blocks and measuring the proximity of the two feature vectors through cosine similarity, which helps to distinguish different types of objects (even if there is spatial overlap). Geometric feature similarity evaluates the similarity of point cloud blocks by analyzing their local geometric structure. A common method is: for the two point cloud blocks to be compared, the FPFH (Fast Point Feature Histograms) descriptor of each point inside is calculated respectively; then, the FPFH descriptors of all points in each point cloud block are aggregated (for example, the average value of all descriptor vectors is calculated to obtain a global feature vector representing the point cloud block); finally, the geometric similarity of the two point cloud blocks is measured by comparing the similarity between the two global feature vectors.

[0051] In order to facilitate those skilled in the art to better understand this solution, the following Figure 2 This paper introduces the 3D point cloud segmentation and object management method disclosed in this paper. Figure 2 , Figure 2 FIG. 1 is a flow chart of another 3D point cloud segmentation method shown in an exemplary embodiment. Figure 2As shown, the system first receives 3D point cloud data and associated 2D image data from sensors (such as lidar and cameras). Next, the system processes the 2D image data using a semantic segmentation network or a vision-based model to generate a 2D mask for each object in the image, representing its location and boundaries. Based on the generated 2D mask, the system then calculates the position and extent of each object in 3D space, known as the view frustum. This process is achieved by back-projecting 2D pixel coordinates into 3D space, providing spatial constraints for subsequent 3D point cloud segmentation. Furthermore, the system uses the view frustum to segment objects in the 3D point cloud, generating point cloud blocks for each object. Based on these point cloud blocks, the system then performs historical object matching in a voxel database and dynamically updates the database to maintain a maintained voxel database. The voxel database stores and manages historical state information for all known objects in the scene. By continuously updating the voxel database, the system can maintain real-time tracking of object states in the scene. Finally, the system outputs the segmentation results for each object point cloud block in the current frame, which can be used for subsequent high-level tasks such as path planning and behavior prediction.

[0052] In this embodiment, the introduction of a voxel database enables continuous maintenance and efficient querying of object states in 3D scenes, improving the consistency of multi-frame data tracking and effectively alleviating the problems of target switching and loss caused by occlusion, temporary disappearance, or perspective changes. By calculating correlation, the accuracy and robustness of object matching are enhanced, enabling the system to more reliably distinguish between newly appeared objects and historical objects, and achieving accurate data updates in the voxel database.

[0053] In one embodiment shown, for each segmented object, the correlation between the point cloud block where the object is located and the point cloud block where each historical object with which it has an overlapping portion is located is calculated one by one, and then the point cloud block where the object is located is added to the voxel database, including: for each segmented object, the correlation between the point cloud block where the object is located and the point cloud block where each historical object with which it has an overlapping portion is located is calculated one by one; if the correlation between the point cloud block where the object is located and the point cloud block where a target historical object with which it has an overlapping portion is located meets a preset condition, the point cloud block where the object is located and the point cloud block where the target historical object is located are merged in the voxel database; if the correlation between the point cloud block where the object is located and the point cloud block where each target historical object with which it has an overlapping portion is located does not meet the preset condition, the point cloud block where the object is located is added to the voxel database.

[0054] For example, the system segments a point cloud block of a car in the current frame and, based on its voxel coordinates, locates a point cloud block of a historical car in the voxel database, with the two partially overlapping in space. The system calculates the correlation between the car and the historical cars, which involves spatial overlap, semantic feature similarity, and geometric feature similarity. Assuming the calculated spatial overlap exceeds 0.5 and the geometric feature similarity exceeds 0.8, meeting the preset conditions, the system determines that the car is the result of an observation of the historical car in the new frame. The point cloud block of the car is then merged with the historical car record, updating information such as the car object's location, point cloud density, and timestamp. Conversely, if the system detects a completely new pedestrian, and the correlation between the pedestrian's point cloud block and all historical objects in the database is less than 0.8, the pedestrian is determined to be a newly appeared object, and new processing is performed, adding a new entry for "pedestrian" to the voxel database.

[0055] Each entry typically represents an observed object in the voxel database and includes, but is not limited to, an entry ID, current / historical point cloud data, a timestamp, motion status (position, velocity), and semantic labels. The voxel database can use a hash table or KD tree as its underlying data structure, enabling fast search of specific spatial regions (i.e., voxel coordinates), effectively finding historical objects that overlap with the current point cloud block. In this embodiment, the purpose of the new processing is to initialize a newly detected object. The system creates a new entry in the voxel database, assigns it a unique ID, and stores information such as the current frame's point cloud block, initial timestamp, and semantic label. This newly detected object will be used as a historical object for association and matching in subsequent frames. The purpose of the merging process is to update and enrich long-term observations of the same object. A simple approach is to directly stitch the point clouds together; advanced approaches can perform voxel filtering (removing duplicate points), weighted averaging (smoothing points in overlapping areas), or surface reconstruction to improve point cloud quality. The object's latest position, timestamp, and other parameters are also updated.

[0056] In one embodiment shown, each 2D mask has a predicted intersection-over-union score when it is generated, and the method further includes: treating any historical object whose point cloud block where the segmented object is located and the point cloud block where the historical object is located as an active object in the voxel database if the correlation between the point cloud block where the segmented object is located meets a preset condition; summing the intersection-over-union scores of the 2D masks of the segmented objects corresponding to each active object, and dividing the summed intersection-over-union score by the total number of active objects to obtain the confidence level of the 3D point cloud segmentation result.

[0057] For example, a self-driving vehicle is navigating a city intersection. Its lidar and camera work synchronously, collecting 3D point cloud data and image data, respectively. In the current frame, the visual base model identifies five objects from the image data: two pedestrians, a bicycle, and two cars. It generates a 2D mask for each object and a corresponding predicted Intersection-over-Union (IoU) score of 0.85, 0.90, 0.75, 0.80, and 0.92. Next, the system back-projects the pixel coordinates of these 2D masks into the 3D point cloud coordinate system, determines the viewing frustum space of each 2D mask, and then selects the point cloud blocks within the corresponding viewing frustum, obtaining the point cloud blocks corresponding to each of the five objects. Assuming that the voxel database has accumulated information about these objects over the past few frames, the system can use this historical data to determine whether the point cloud blocks of potential objects in the current frame are associated with objects in the historical voxel database. Historical objects whose association meets a preset condition are marked as active objects. Assume there are six historical objects in the voxel database, five of which are active objects (two pedestrians, a bicycle, and two cars). Finally, the system calculates the sum of the intersection-over-union scores of the 2D masks of the five objects identified in the image data corresponding to the five active objects: 0.85 + 0.90 + 0.75 + 0.80 + 0.92 = 4.22. This is then divided by the number of active objects, 5, resulting in a confidence score of 0.844 for the 3D point cloud segmentation result. This indicates that the 3D point cloud segmentation of the active objects in this image scene is generally good.

[0058] The predicted IoU score for a 2D mask is the predicted IoU score output by the visual base model when generating each 2D mask. It measures the degree of overlap between the mask and the true object outline (range 0-1). A higher score indicates higher mask accuracy. Active objects refer to objects in the current image data frame that have been successfully matched to historical objects in the voxel database. The confidence level of the 3D point cloud segmentation result is a metric used to assess the reliability of the 3D point cloud segmentation results based on the current image data frame.

[0059] In one embodiment shown, the system also maintains a 3D segmentation model, which is used to segment the point cloud block where the object is located in the 3D point cloud data based on the 2D mask. The method also includes: if the segmentation quality score is lower than a first preset threshold, the active object associated with the point cloud block where the object corresponding to the 2D mask whose intersection-over-union score exceeds a second threshold is located, and the point cloud block where the associated active object is located is used as a pseudo-label; based on the pseudo-label, the lightweight trainable sub-module of the 3D segmentation model is fine-tuned online.

[0060] For example, in an autonomous driving scenario, assume the system sets a first preset threshold for the confidence level of 3D point cloud segmentation results to 0.75 and a second threshold for the intersection-over-union (IoU) score of 2D masks to 0.9. At a certain moment, as the vehicle passes through a tunnel, image quality degrades due to a sudden change in illumination. The system calculates the confidence level of the current 3D point cloud segmentation result to 0.72 (lower than 0.75), triggering the online adjustment mechanism. At this point, the IoU scores of the five 2D masks generated by the visual base model are 0.88, 0.93, 0.79, 0.91, and 0.85, respectively. The scores exceeding the second threshold (0.9) are 0.93 (for the car ahead) and 0.91 (for the pedestrian on the right). The system determines that the object point cloud blocks corresponding to the two 2D masks are associated with the active objects "car A" and "pedestrian B" in the voxel database, so "car A" and its point cloud block, "pedestrian B" and its point cloud block are used as pseudo labels, and the pseudo labels are used to train the lightweight PointNet++ submodule (including trainable parameters) responsible for point cloud feature refinement in the 3D segmentation model. ) performed online fine-tuning: using the loss function to calculate the loss, setting the learning rate to 5e-5, and completing the parameter update after three iterations. After fine-tuning, the segmentation quality score of the next frame increased to 0.81, significantly improving the segmentation accuracy of cars and pedestrians.

[0061]

[0062] in, are the updated and current trainable parameters. is the learning rate for online fine-tuning, . is the loss function calculated based on pseudo labels, is the gradient of the loss function calculated based on the pseudo-label, is the 3D object segmentation result of the current image data frame, which is the set of all active objects in the voxel database. is the 3D point cloud collected at the current moment, is a pseudo label.

[0063] In this embodiment, the loss function can actually be Dice Loss or Lovasz-SoftmaxLoss. The core logic of online fine-tuning of the 3D segmentation model is to update the trainable parameters so that the output of the 3D segmentation model is closer to the pseudo-label. Fine-tuning lightweight trainable sub-modules can help the 3D segmentation model quickly adapt to changing environmental conditions without retraining the entire model. On the one hand, this reduces the demand for computing resources and time consumption, which is particularly important for application scenarios that require immediate response, such as autonomous driving; on the other hand, it enables the 3D segmentation model to run efficiently on devices with limited resources and strong adaptability.

[0064] In order to facilitate those skilled in the art to better understand this solution, the following Figure 3 This paper introduces the online adjustment process of the 3D segmentation model disclosed in this paper. Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing an exemplary embodiment of online adjustment of a 3D segmentation model. Figure 3 As shown, after the 3D segmentation model in the vehicle system completes the 3D point cloud segmentation, the confidence of the 3D point cloud segmentation result of the current image frame is evaluated. Next, the system will determine whether the confidence is lower than the first threshold. If the confidence is not lower than the first threshold, the current 3D segmentation model is used to continue processing subsequent frames; conversely, if the confidence is lower than the first threshold, the 3D segmentation model needs to be fine-tuned online to optimize its performance. After fine-tuning, an updated 3D segmentation model is obtained, and the updated 3D segmentation model is used to continue processing the data of subsequent frames. This process ensures that the system can make timely adjustments when faced with low-confidence segmentation results of the 3D segmentation model, thereby improving the adaptive ability and segmentation accuracy of the system operation.

[0065] Corresponding to the above-mentioned embodiment of the 3D point cloud segmentation method, the present disclosure also provides an embodiment of a 3D point cloud segmentation device.

[0066] See Figure 4 , Figure 4 This is a hardware structure diagram of an electronic device shown in an exemplary embodiment. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of the present disclosure can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to software implementation, one or more embodiments of the present disclosure do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0067] See Figure 5 , Figure 5 FIG. 5 is a block diagram of a 3D point cloud segmentation device according to an exemplary embodiment. The 3D point cloud segmentation device 500 can be applied to Figure 4 The electronic device shown in the figure is used to implement the technical solution of the present disclosure. The device includes: An acquisition unit 502 is configured to acquire 3D point cloud data and image data corresponding to the same sampling moment and the same scene; A first conversion unit 504 is configured to convert the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data; A second conversion unit 506 is configured to convert the pixel coordinates of each 2D mask into a 3D point cloud coordinate system, thereby determining a corresponding viewing cone space of each 2D mask in the 3D point cloud coordinate system; The segmentation unit 508 is configured to segment, for each 2D mask, the 3D point cloud data located in the viewing cone space corresponding to the mask in the 3D point cloud data, so as to determine the point cloud block where the object is located.

[0068] In some embodiments, converting the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data, comprises: Conditioning the visual base model to mask generation mode; Inputting the image data into a visual base model to obtain a set of 2D masks output by the visual base model; Each 2D mask corresponds to a binary image, and the binary image identifies a contour area of ​​an object contained in the image data.

[0069] In some embodiments, converting the pixel coordinates of each 2D mask into a 3D point cloud coordinate system, thereby determining a viewing cone space corresponding to each 2D mask in the 3D point cloud coordinate system, includes: For each 2D mask, extract its boundary pixel coordinates; Converting the boundary pixel coordinate set to a ray direction vector in an image sensor coordinate system according to an intrinsic parameter matrix of the image sensor; According to the extrinsic parameter matrix from the image sensor to the point cloud sensor, the ray direction vector is converted to a 3D point cloud coordinate system, and the converted ray direction vector is connected with the optical center of the image sensor as the vertex to obtain the corresponding viewing cone space of each 2D mask in the 3D point cloud coordinate system.

[0070] In some embodiments, the method is applied to a system, wherein the system maintains a voxel database, the apparatus further comprising: A search unit 510 is configured to search, in a voxel database, for point cloud blocks of historical objects that overlap with the point cloud blocks of each segmented object based on the voxel coordinates of the point cloud blocks where the historical objects are located. The first calculation unit 512 is used to calculate the correlation between the point cloud block where each segmented object is located and the point cloud block where each historical object with which the object has an overlapping portion is located, and then update the point cloud block where the object is located to the voxel database.

[0071] In some embodiments, for each segmented object, calculating the correlation between the point cloud block where the object is located and the point cloud block where each historical object that overlaps with the object is located, and then adding the point cloud block where the object is located to the voxel database, includes: For each segmented object, calculate the correlation between the point cloud block where the object is located and the point cloud block where each historical object that overlaps with it is located. If the correlation between the point cloud block where the object is located and the point cloud block where a target historical object with which the object is located overlaps satisfies a preset condition, the point cloud block where the object is located and the point cloud block where the target historical object is located are merged in the voxel database; If the correlation between the point cloud block where the object is located and the point cloud block where each target historical object having an overlapping portion thereof is located does not meet a preset condition, the point cloud block where the object is located is newly added to the voxel database.

[0072] In some embodiments, each of the 2D masks has a predicted intersection-over-union score when generated, and the apparatus further comprises: A determining unit 514 is configured to consider any segmented object where the point cloud block contains the historical object and the historical object where the correlation degree between the segmented object and the point cloud block contains the historical object satisfies a preset condition as an active object in the voxel database; The second calculation unit 516 is configured to sum the IoU scores of the 2D masks of the segmented objects corresponding to the active objects, and divide the summed IoU score by the total number of active objects to obtain the confidence level of the 3D point cloud segmentation result.

[0073] In some embodiments, the system further maintains a 3D segmentation model, wherein the 3D segmentation model is used to segment the point cloud block where the object is located in the 3D point cloud data according to the 2D mask, and the apparatus further includes: a labeling unit 518 configured to, if the confidence level of the 3D point cloud segmentation result is lower than a first preset threshold, use the active object associated with the point cloud block where the object corresponding to the 2D mask with an IoU score exceeding a second preset threshold is located, and the point cloud block where the active object is located as a pseudo label; The fine-tuning unit 520 is configured to perform online fine-tuning on the lightweight trainable sub-module of the 3D segmentation model based on the pseudo-label.

[0074] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0075] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0076] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0077] In a typical configuration, a computer includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0078] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0079] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0080] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0081] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0082] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] The terms used in one or more embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present disclosure. The singular forms "a," "the," and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0084] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0085] The above description is merely a preferred embodiment of one or more embodiments of the present disclosure and is not intended to limit one or more embodiments of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present disclosure shall be included in the scope of protection of one or more embodiments of the present disclosure.

Claims

1. A 3D point cloud segmentation method, characterized in that: The method comprises: Obtain 3D point cloud data and image data corresponding to the same sampling time and the same scene; Converting the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data; Convert the pixel coordinates of each 2D mask into a 3D point cloud coordinate system, thereby determining the corresponding viewing cone space of each 2D mask in the 3D point cloud coordinate system; For each 2D mask, 3D point cloud data located in the viewing cone space corresponding to the mask is segmented from the 3D point cloud data to determine the point cloud block where the object is located.

2. The method according to claim 1, characterized in that The converting the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data, comprises: Conditioning the visual base model to mask generation mode; Inputting the image data into a visual base model to obtain a set of 2D masks output by the visual base model; Each 2D mask corresponds to a binary image, and the binary image identifies a contour area of ​​an object contained in the image data.

3. The method according to claim 1, characterized in that The pixel coordinates of each 2D mask are converted into a 3D point cloud coordinate system, thereby determining a corresponding viewing cone space of each 2D mask in the 3D point cloud coordinate system, including: For each 2D mask, extract its boundary pixel coordinates; Converting the boundary pixel coordinate set to a ray direction vector in an image sensor coordinate system according to an intrinsic parameter matrix of the image sensor; According to the extrinsic parameter matrix from the image sensor to the point cloud sensor, the ray direction vector is converted to a 3D point cloud coordinate system, and the converted ray direction vector is connected with the optical center of the image sensor as the vertex to obtain the corresponding viewing cone space of each 2D mask in the 3D point cloud coordinate system.

4. The method according to claim 1, wherein The method is applied to a system, wherein the system maintains a voxel database, and the method further comprises: According to the voxel coordinates of the point cloud block where each segmented object is located, the point cloud block where the historical object with which it overlaps is searched in the voxel database; For each segmented object, the correlation between the point cloud block where the object is located and the point cloud block where each historical object with which the object overlaps is calculated one by one, and then the point cloud block where the object is located is updated to the voxel database.

5. The method according to claim 4, characterized in that The step of calculating, for each segmented object, the correlation between the point cloud block where the object is located and the point cloud block where each historical object that overlaps with the object is located, and then adding the point cloud block where the object is located to the voxel database includes: For each segmented object, calculate the correlation between the point cloud block where the object is located and the point cloud block where each historical object that overlaps with it is located. If the correlation between the point cloud block where the object is located and the point cloud block where a target historical object with which the object is located overlaps satisfies a preset condition, the point cloud block where the object is located and the point cloud block where the target historical object is located are merged in the voxel database; If the correlation between the point cloud block where the object is located and the point cloud block where each target historical object having an overlapping portion thereof is located does not meet a preset condition, the point cloud block where the object is located is newly added to the voxel database.

6. The method according to claim 4, characterized in that Each of the 2D masks has a predicted intersection-over-union score when generated, and the method further comprises: The historical object is regarded as an active object in the voxel database if the correlation between the point cloud block where any segmented object is located and the point cloud block where the historical object is located meets a preset condition; The confidence level of the 3D point cloud segmentation result is obtained by summing the intersection-over-union (IoU) scores of the 2D masks of the segmented objects corresponding to each active object and dividing the summed IoU score by the total number of active objects.

7. The method according to claim 6, characterized in that The system further maintains a 3D segmentation model, wherein the 3D segmentation model is used to segment the point cloud block where the object is located in the 3D point cloud data according to the 2D mask. The method further includes: If the confidence level of the 3D point cloud segmentation result is lower than a first preset threshold, the active object associated with the point cloud block where the object corresponding to the 2D mask whose intersection-over-union score exceeds a second preset threshold is located, and the point cloud block where the active object is located are used as pseudo labels; Based on the pseudo labels, a lightweight trainable submodule of the 3D segmentation model is fine-tuned online.

8. A 3D point cloud segmentation device, characterized in that: The device comprises: An acquisition unit, configured to acquire 3D point cloud data and image data corresponding to the same sampling moment and the same scene; a first conversion unit, configured to convert the image data into a set of 2D masks, each 2D mask identifying a contour region of an object contained in the image data; a second conversion unit, configured to convert the pixel coordinates of each 2D mask into a 3D point cloud coordinate system, thereby determining a viewing cone space corresponding to each 2D mask in the 3D point cloud coordinate system; The segmentation unit is used to segment the 3D point cloud data located in the viewing cone space corresponding to each 2D mask in the 3D point cloud data to determine the point cloud block where the object is located.

9. An electronic device, characterized in that: It includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 7 by calling the machine-readable instructions.

10. A machine-readable storage medium, characterized in that The machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images