Online point cloud semantic segmentation method and device, storage medium and electronic equipment

Abstract

The present disclosure relates to the field of artificial intelligence, in particular to an online point cloud semantic segmentation method, an online point cloud semantic segmentation device, a storage medium and an electronic device. The online point cloud semantic segmentation method comprises: collecting a color depth image in real time, and converting pixel points in the color depth image into a current point cloud set based on depth data of the pixel points; performing vertex fusion on the current point cloud set and a historical point cloud set through point cloud registration to obtain geometric features and time sequence semantic states corresponding to the pixel points; performing feature fusion on the geometric features and color data of the pixel points in the color depth image to obtain semantic features of the color depth image; and performing semantic segmentation according to the semantic features and the time sequence semantic states to obtain a semantic segmentation result of the pixel points. The online point cloud semantic segmentation method aims to output accurate and stable point cloud semantic segmentation results online.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to an online point cloud semantic segmentation method, an online point cloud semantic segmentation device, a storage medium, and an electronic device. Background Technology

[0002] In recent years, the development of artificial intelligence (AI) technology has improved computers' ability to analyze and understand 3D data. Mainstream research has chosen to learn models from large-scale collected data to achieve semantic perception and understanding of 3D scenes, which has promoted the research and application of autonomous driving and intelligent robots.

[0003] Real-time semantic information is crucial for robot scene understanding. Currently, the massive data scale and complex geometric structure analysis process of scene point clouds make it difficult for traditional point cloud segmentation algorithms to be implemented online. Furthermore, processing video stream data frame-by-frame through image semantic segmentation cannot effectively establish connections between multi-view data or acquire spatial geometric features, leading to chaotic semantic segmentation results in reconstructed point clouds. Additionally, convolutional neural networks often exhibit uncertainty when processing 2D images, causing online semantic segmentation to easily generate numerous ambiguities during viewpoint movement, resulting in unstable semantic segmentation. Therefore, online point cloud understanding in 3D scenes requires further research.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an online point cloud semantic segmentation method, an online point cloud semantic segmentation device, a storage medium, and an electronic device, which aim to output accurate and stable point cloud semantic segmentation results online.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to one aspect of the present disclosure, an online point cloud semantic segmentation method is provided, comprising: acquiring a color depth image in real time, and converting the pixels into a point cloud based on the depth data of the pixels in the color depth image to obtain a current point cloud set; performing vertex fusion of the current point cloud set and a historical point cloud set through point cloud registration to obtain the geometric features and temporal semantic state corresponding to the pixels; performing feature fusion of the geometric features and the color data of the pixels in the color depth image to obtain the semantic features of the color depth image; and performing semantic segmentation based on the semantic features and the temporal semantic state to obtain the semantic segmentation result of the pixels.

[0008] According to some embodiments of this disclosure, based on the foregoing scheme, the step of converting the pixels into a point cloud based on the depth data of the pixels in the color depth image to obtain the current point cloud set includes: acquiring camera parameters for acquiring the color depth image; for a pixel, converting the pixel into a voxel point in a three-dimensional coordinate system according to the depth data of the pixel and the camera parameters; traversing each pixel in the color depth image to obtain the voxel points corresponding to each pixel to form the current point cloud set.

[0009] According to some embodiments of this disclosure, based on the foregoing scheme, the historical point cloud includes registered historical registration points. The step of fusing the current point cloud with the historical point cloud through point cloud registration includes: converting the coordinate data of voxels in the current point cloud to register current registration points; determining whether there are current registration points and historical registration points to be fused; and when there are current registration points and historical registration points to be fused, fusing the current registration points and the historical registration points according to preset fusion rules to update the historical point cloud.

[0010] According to some embodiments of this disclosure, based on the foregoing scheme, determining whether there are current registration points and historical registration points to be merged includes: for the current point cloud - current registration point, determining the first position of the current registration point on the first coordinate axis, the second position on the second coordinate axis, and the third position on the third coordinate axis; searching from the historical point cloud for historical registration points that match the first position to form a first historical point cloud; when the first historical point cloud is a non-empty set, searching from the first historical point cloud for historical registration points that match the second position to form a second historical point cloud; when the second historical point cloud is a non-empty set, searching from the second historical point cloud for historical registration points that match the second position to determine whether there are current registration points and historical registration points to be merged.

[0011] According to some embodiments of this disclosure, based on the foregoing scheme, after converting the coordinate data of the voxel points in the current point cloud set to register and obtain the current registered point, the method further includes: obtaining the registration timestamp, geometric features, temporal semantic state, and point cloud semantic information of the current registered point for use in the vertex fusion; wherein, the geometric features include one or more of the following: the point cloud boundary distance value of the current registered point relative to the current point cloud set, the point cloud minimum point distance value, the normal vector, and the curvature.

[0012] According to some embodiments of this disclosure, based on the foregoing scheme, when there are current registration points and historical registration points to be merged, the merging of the current registration point and the historical registration point according to a preset merging rule includes: when the registration timestamp of the current registration point and the registration timestamp of the historical registration point meet a preset condition, replacing the coordinate data and registration timestamp of the current registration point with the historical registration point; when the registration timestamp of the current registration point and the registration timestamp of the historical registration point do not meet the preset condition, replacing the coordinate data, registration timestamp, geometric features, temporal semantic state, and point cloud semantic information of the current registration point with the historical registration point.

[0013] According to some embodiments of this disclosure, based on the foregoing scheme, the step of fusing the geometric features and the color data of pixels in the color depth image to obtain the semantic features of the color depth image includes: encoding the geometric features using a first method to obtain first encoded data, and encoding the color data using a second method to obtain second encoded data; upsampling the first encoded data and the second encoded data respectively to obtain a fusion result; and decoding the fusion result to obtain the semantic features.

[0014] According to some embodiments of this disclosure, based on the foregoing scheme, the step of obtaining the semantic segmentation result of the pixel by performing semantic segmentation based on the semantic features and the temporal semantic state includes: for a pixel, extracting temporal features using convolutional layers in a recurrent neural network based on the semantic features and the temporal semantic state of the current registration point corresponding to the pixel; wherein the recurrent neural network includes two long short-term memory networks; calculating a semantic score using fully connected layers in the recurrent neural network based on the temporal features; predicting the semantic segmentation result of the pixel based on the semantic score; and traversing each pixel in the color depth image to obtain the semantic segmentation result corresponding to each pixel.

[0015] According to a second aspect of the present disclosure, an online point cloud semantic segmentation apparatus is provided, comprising: an acquisition module, configured to acquire a color depth image in real time, and convert the pixels into a point cloud based on the depth data of the pixels in the color depth image to obtain a current point cloud set; a fusion module, configured to perform vertex fusion of the current point cloud set and a historical point cloud set through point cloud registration to obtain the geometric features and temporal semantic state corresponding to the pixel; a feature module, configured to perform feature fusion on the geometric features and the color data of the pixels in the color depth image to obtain the semantic features of the color depth image; and a segmentation module, configured to perform semantic segmentation based on the semantic features and the temporal semantic state to obtain the semantic segmentation result of the pixel.

[0016] According to a third aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the online point cloud semantic segmentation method as described in the above embodiments.

[0017] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the online point cloud semantic segmentation method as described in the above embodiments.

[0018] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:

[0019] In some embodiments of this disclosure, color depth images are acquired in real time, and vertex fusion is performed between these images and historical point cloud sets through point cloud transformation and registration. Based on the vertex fusion result, the geometric features and temporal semantic states corresponding to each pixel in the image are obtained. Then, feature fusion is performed on the geometric features and color data to obtain semantic features. Finally, semantic segmentation is performed based on the semantic features and the temporal semantic states to obtain pixel-level semantic segmentation results. Based on this online point cloud semantic segmentation method, on the one hand, it can acquire color depth images in real time and perform vertex fusion of point cloud sets to output the semantic segmentation results of each pixel online, making it suitable for online scene semantic understanding scenarios. On the other hand, vertex fusion can maintain the temporal semantic features of each vertex in the 3D point cloud, establish inter-frame links from different perspectives to optimize semantic recognition results, improve the accuracy and consistency of semantic segmentation, and avoid the situation where isolated perspectives between multiple frames lead to chaotic semantic segmentation results.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0022] Figure 1 The illustration shows a flowchart of an online point cloud semantic segmentation method according to an exemplary embodiment of the present disclosure;

[0023] Figure 2 The illustration schematically shows a flowchart of a vertex fusion method according to an exemplary embodiment of the present disclosure;

[0024] Figure 3 The illustration shows a flowchart of a method for obtaining semantic features in an exemplary embodiment of the present disclosure;

[0025] Figure 4 This schematic diagram illustrates the structure of a multimodal semantic feature network according to an exemplary embodiment of the present disclosure.

[0026] Figure 5 This schematic diagram illustrates the composition of an online point cloud semantic segmentation apparatus according to an exemplary embodiment of the present disclosure;

[0027] Figure 6 This schematic diagram illustrates a computer-readable storage medium according to an exemplary embodiment of the present disclosure;

[0028] Figure 7 The schematic diagram illustrates the structure of a computer system of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0029] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0030] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] Although deep learning research in the image domain is quite mature, image content understanding is still affected by various objective factors, such as shape loss from the camera's perspective, occlusion in physical space, excessive distance between objects, complex texture details, and changing lighting conditions. These factors require networks with very strong parameter learning capabilities and rely on sufficiently large datasets for construction. Even so, it is impossible to completely avoid the uncertainty of single-frame semantic segmentation and the frequent semantic fluctuations that occur during viewpoint movement. Therefore, it is necessary to shift from a two-dimensional to a three-dimensional perspective.

[0034] In existing technologies, semantic segmentation algorithms for point clouds often consume significant computational resources due to their complex geometric processing and the massive scale of point clouds. Therefore, most deep learning methods for 3D point clouds are offline, require extensive preprocessing, and are difficult to apply to online scenarios, despite the substantial online demand for scene semantic understanding applications. While processing video stream data frame-by-frame through image semantic segmentation can obtain real-time semantic information, this isolated approach fails to effectively establish connections between multi-view data and capture spatial geometric features, leading to chaotic semantic segmentation results in reconstructed point clouds. Furthermore, convolutional neural networks frequently exhibit uncertainty when processing 2D images, causing online semantic segmentation to become prone to significant ambiguity during viewpoint shifts, resulting in unstable semantic segmentation.

[0035] Therefore, in view of the problems existing in the prior art, this disclosure provides an online point cloud semantic segmentation method. The semantic prediction of this disclosure is carried out simultaneously with the 3D point cloud reconstruction, aiming to output accurate and stable point cloud semantic information.

[0036] The implementation details of the technical solutions of the embodiments of this disclosure are described in detail below.

[0037] Figure 1This illustration schematically shows a flowchart of an online point cloud semantic segmentation method according to an exemplary embodiment of this disclosure. Figure 1 As shown, the online point cloud semantic segmentation method includes steps S101 to S104:

[0038] Step S101: Real-time acquisition of color depth images, and point cloud conversion of the pixels based on the depth data of the pixels in the color depth images to obtain the current point cloud set;

[0039] Step S102: The current point cloud set and the historical point cloud set are fused by point cloud registration to obtain the geometric features and temporal semantic state corresponding to the pixel.

[0040] Step S103: Perform feature fusion on the geometric features and the color data of the pixels in the color depth image to obtain the semantic features of the color depth image;

[0041] Step S104: Perform semantic segmentation based on the semantic features and the temporal semantic state to obtain the semantic segmentation result of the pixel.

[0042] In some embodiments of this disclosure, color depth images are acquired in real time, and vertex fusion is performed between these images and historical point cloud sets through point cloud transformation and registration. Based on the vertex fusion result, the geometric features and temporal semantic states corresponding to each pixel in the image are obtained. Then, feature fusion is performed on the geometric features and color data to obtain semantic features. Finally, semantic segmentation is performed based on the semantic features and the temporal semantic states to obtain pixel-level semantic segmentation results. Based on this online point cloud semantic segmentation method, on the one hand, it can acquire color depth images in real time and perform vertex fusion of point cloud sets to output the semantic segmentation results of each pixel online, making it suitable for online scene semantic understanding scenarios. On the other hand, vertex fusion can maintain the temporal semantic features of each vertex in the 3D point cloud, establish inter-frame links from different perspectives to optimize semantic recognition results, improve the accuracy and consistency of semantic segmentation, and avoid the situation where isolated perspectives between multiple frames lead to chaotic semantic segmentation results.

[0043] The following will describe in more detail each step of the online point cloud semantic segmentation method in this exemplary embodiment, with reference to the accompanying drawings and embodiments.

[0044] In step S101, a color depth image is acquired in real time, and the pixel points are converted into a point cloud based on the depth data of the pixel points in the color depth image to obtain the current point cloud set.

[0045] In one embodiment of this disclosure, a portable sensor device can be used to acquire color depth images in real time, so that the definition of object boundaries is no longer limited to color or lighting conditions. The color depth image is an RGB-D image, which contains a data format of aligned color information channel C and depth information channel D. For example, in the logistics and distribution field, an RGB-D depth camera can be equipped directly in front of an AGV (Automated Guided Vehicle) to acquire RGB-D images.

[0046] Reliable depth information is used to represent the spatial relationships of object distribution, converting two-dimensional images into three-dimensional point cloud data to facilitate semantic prediction from a geometric perspective. Since color depth images contain multiple pixels, each of which can be projected into three-dimensional space, further point cloud transformation is needed to assemble the current point cloud set from the pixels in each frame of the color depth image.

[0047] Furthermore, the current point cloud includes voxels corresponding to each pixel in the color depth image. The step of converting the point cloud based on the depth data in the color depth image to obtain the current point cloud includes: acquiring camera parameters for acquiring the color depth image; for a pixel, converting the pixel into a voxel in a three-dimensional coordinate system according to the depth data of the pixel and the camera parameters; and traversing each pixel in the color depth image to obtain the voxels corresponding to each pixel to form the current point cloud.

[0048] Specifically, the camera's intrinsic parameter matrix The image domain of the acquired color depth image is denoted as I, and the pixel coordinates of pixel i in the image are... Color gamut information (i.e., color data) is Depth domain information (i.e., depth data) is

[0049] A mapping relationship between two-dimensional pixel coordinates in the image and three-dimensional voxel points in the three-dimensional camera coordinate system is established using intrinsic parameter matrices and depth data. K : According to the mapping relationship f K The pixel i in the current frame's color depth image (denoted as the current frame) can be converted into a voxel to obtain a point cloud. The specific point cloud transformation formula is defined as shown in formula (1):

[0050] P(i) T =f K (i)·(u i v i ,1) T (1)

[0051] In the formula, f u and f v The focal length of the camera is represented by (u0, v0), which are the pixel coordinates of the camera's principal point.

[0052] In another embodiment of this disclosure, to facilitate subsequent information retrieval, the mapping relationship f can also be used. K The color data of each pixel is projected onto each voxel to obtain the color point cloud data, i.e., voxels corresponding to the pixels are defined.

[0053] In step S102, the current point cloud set and the historical point cloud set are fused by point cloud registration to obtain the geometric features and temporal semantic state corresponding to the pixel.

[0054] In one embodiment of this disclosure, since the method of this disclosure is applied to online scenarios, color depth images of multiple consecutive frames are acquired in real time, and the multi-frame data is processed. At that time, the method in step S101 can be used to obtain the point cloud set corresponding to each frame image. However, the space occupied by each frame of data is independent, making it impossible to directly manipulate the point cloud set. Union (i.e., vertex fusion). Here, SLAM (Simultaneous Localization and Mapping) technology can be used to register multiple frames of depth images to estimate relative pose relationships, so as to fuse vertices from different perspectives and reconstruct a large-scale scene point cloud for semantic segmentation.

[0055] Furthermore, Figure 2 The illustration schematically shows a flowchart of a vertex fusion method according to an exemplary embodiment of this disclosure. Figure 2 As shown, the vertex fusion method includes steps S201 to S203:

[0056] Step S201: Convert the coordinate data of the voxels in the current point cloud to obtain the current registered point;

[0057] Step S202: Determine whether there are any current registration points and historical registration points to be merged;

[0058] Step S203: When there are current registration points and historical registration points to be merged, merge the current registration points and the historical registration points according to the preset fusion rules to update the historical point cloud.

[0059] Specifically, in step S201, the coordinate data of the voxels in the current point cloud are converted to obtain the current registered point.

[0060] First, the current color depth image is scanned using the device to perform coordinate transformation on its corresponding voxel points to complete the registration. The specific transformation relationship is shown in formula (2):

[0061]

[0062] In the formula, T t Let be the global pose matrix at time t. The coordinates of the voxel points before registration are obtained from point cloud data; x i y i , z i The coordinates of the voxel point after registration (i.e., the current registration point). For ease of description, both the current registration point and the historical registration points can be referred to as vertices.

[0063] In one embodiment of this disclosure, vertex fusion also requires obtaining the registration timestamp, geometric features, temporal semantic state, and point cloud semantic information of the current registered points. The registration timestamp scanned by the device is denoted as tsp, the geometric features as feat, the temporal semantic state stored in the point cloud as status, and the memorized point cloud semantic information as label. Therefore, the current registered points of the current point cloud set after registration can be obtained. The current point of convergence is

[0064] In order to avoid problems such as insufficient description of the shape of an object from a single viewpoint, changes in object size with distance, instability of recognition results caused by occlusion of objects, and reduced color recognition of shape due to complex textures and changing ambient lighting, it is necessary to design geometric features. Geometric features have the geometric properties of being independent of viewpoint and satisfying rotation invariance, which are crucial for semantic segmentation.

[0065] In this disclosure, incremental point cloud registration is used to design geometric features. The geometric features include one or more of the following: the distance of the currently registered point relative to the point cloud boundary of the current point cloud set, the distance to the lowest point of the point cloud, the normal vector, and the curvature.

[0066] Specifically, for the current point cloud collection after registration The geometric features of each current registration point P(i) can be defined as: feat i =(distance) i height i (normal, curvature). Where, distance i This refers to the distance value of the currently registered point relative to the point cloud boundary of the current point cloud set, i.e. For example, the distance from the wall. (height)i It refers to the height difference between the current registered point and the lowest point in the point cloud of the current point set. i =z i -min{z} is typically the distance from the ground. `normal` and `curvature` refer to the normal and PCA curvature of P(i), respectively, which can be estimated using principal component analysis.

[0067] It is important to note that PCA curvature is typically estimated using a set of K-nearest neighbors (which can be maintained and efficiently retrieved using a KD-tree) as the neighborhood. Since KD-tree data structures are difficult to implement online, and the K-nearest neighbor algorithm is also challenging to implement online, this method dynamically maintains the neighborhood. Each point records its K neighboring points, where K is a preset value set according to requirements. During point cloud registration, these K points are dynamically updated based on the image's two-dimensional neighborhood structure. This disclosure only recalculates the normal vector and curvature for newly registered vertices or vertices with significant neighborhood changes to save computational resources.

[0068] It should be noted that the point cloud data registered here does not need to store color information, because color, as an input feature of the image domain, has already been condensed into the status. Furthermore, the color of the vertices will vary depending on the viewpoint or lighting conditions; it is not a particularly fixed value.

[0069] Furthermore, the global pose matrix can be solved using SLAM (Simultaneous Localization and Mapping) or reconstruction algorithms, and most datasets also provide standard pose information. During 3D reconstruction, the camera pose can be solved using methods such as Iterative Closest Point (ICP) and loop closure optimization. Pose solving algorithms can be implemented using existing technologies, and will not be elaborated upon here.

[0070] The formation process of historical point clouds is also similar to that described above, involving incremental point cloud registration of multiple consecutive frames of color depth images. Historical point clouds comprise historical registration points from acquired historical color depth images after registration and fusion. Each historical registration point also includes point cloud data such as coordinate data, registration timestamp, geometric features, temporal semantic state, and point cloud semantic information.

[0071] As the number of color depth image frames in 3D reconstruction increases, vertex fusion is required to reduce the size of the point cloud. This means that vertex fusion needs to be performed based on the point cloud data of historical registered point cloud sets and the point cloud data of currently registered points.

[0072] In step S202, it is determined whether there are current registration points and historical registration points to be merged.

[0073] Specifically, the historical point cloud before time t is denoted as Among them historical registration points Let the current point cloud set after registration in the current frame be denoted as The current registration point Define the fusion rules If the following conditions are met, as shown in formula (3), then vertex V will be... a and vertex V b To integrate.

[0074] Grid(x a y a , z a ) = Grid(x b y b , z b (3)

[0075] In the formula, Where ∈ represents the size of a voxel unit, which is a preset value, and Grid represents the voxel grid point.

[0076] In one embodiment of this disclosure, frequent vertex access is required during point cloud fusion to determine whether there are currently registered points and historical registered points to be fused.

[0077] In the merged point cloud set, the Grid(x, y, z) corresponding to each vertex V is unique. This is because merging is necessary when Grid(x, y, z) are identical. Therefore, it can be... The search is performed using the key value of vertex V.

[0078] Typically, point cloud adjacency queries require the use of KD-trees or octrees, employing traditional hash algorithms for vertex access. However, these data structures generally do not support dynamic online construction, while the application environment described in this paper is online, necessitating a fast memory access structure adapted for online processing. Furthermore, as image data increases, the point cloud size N can reach millions or even tens of millions, an extremely large number. This massive scale leads to frequent hash collisions, causing a degraded hash algorithm performance and reducing query efficiency. Therefore, a reasonable access strategy needs to be designed to reduce time complexity.

[0079] Taking the z-axis as an example, it is generally used to describe the height of a point in the scene (assuming the maximum value is H). The key used for memory access only follows the z-axis. If they are related, then the key values ​​have the following relationship, as shown in formula (4):

[0080]

[0081] If (h, ∈) = (5m, 2cm), then There are no more than 250 possible values. To address this characteristic, this disclosure establishes a triple-nested hash structure for memory access, starting from the three independent dimensions of the point cloud.

[0082] Furthermore, in one embodiment of this disclosure, determining whether there are current registration points and historical registration points to be merged includes: for the current point cloud - current registration point, determining the current registration point at a first position on a first coordinate axis, a second position on a second coordinate axis, and a third position on a third coordinate axis; searching from the historical point cloud for historical registration points matching the first position to form a first historical point cloud; when the first historical point cloud is a non-empty set, searching from the first historical point cloud for historical registration points matching the second position to form a second historical point cloud; when the second historical point cloud is a non-empty set, searching from the second historical point cloud for historical registration points matching the second position to determine whether there are current registration points and historical registration points to be merged.

[0083] For example, the search is performed in the order of x-axis, y-axis to z-axis, and the position information of each coordinate axis is as follows: The specific memory access process is as follows:

[0084] Search(Tree,(x,y,z))=Search(SubTree(key x ), (y, z))

[0085] Search(SubTree(key x ), (y, z))=Search(SubTree(key x key y ), z)

[0086] Search(SubTree(key x key y z) = Node(key) x key y key z )

[0087] Each level of query can be maintained using a red-black tree structure, with the upper limit of time complexity for each query, modification, and addition operation being O(log H). Since H << N, nested binary trees can significantly reduce hash collisions, improve algorithm efficiency, and this structure supports parallel operations, which is highly beneficial for point cloud registration processes that involve numerous node changes.

[0088] In step S203, when there are current registration points and historical registration points to be merged, the current registration points and the historical registration points are merged according to preset fusion rules to update the historical point cloud.

[0089] In one embodiment of this disclosure, when the registration timestamp of the current registration point and the registration timestamp of the historical registration point meet a preset condition, the coordinate data and registration timestamp of the current registration point are replaced with the historical registration point; when the registration timestamp of the current registration point and the registration timestamp of the historical registration point do not meet the preset condition, the coordinate data, registration timestamp, geometric features, temporal semantic state, and point cloud semantic information of the current registration point are replaced with the historical registration point.

[0090] Specifically, when vertex V a and vertex V b The timestamp satisfies (tsp) a <tsp b )∩(tsp b -tsp a When the time is <10s, the new coordinate data and registration timestamp can be retained, while the original geometric features, temporal semantic state, and point cloud semantic information can be reused. That is, the fused vertex is V. c =(x b y b , z b ,tsp b feat a status a label a ) is used to replace V a If the condition is not met, the point is treated as a completely new point, meaning its coordinate data, registration timestamp, geometric features, temporal semantic state, and point cloud semantic information are all replaced. This not only saves on the computation of geometric features.

[0091] At this point, vertex registration and fusion of multiple consecutive frames of color depth images are completed, so as to build a mapping relationship and communication mechanism between the image and the point cloud. That is, each pixel in the image can be mapped to the registration point in the historical point cloud set, and the corresponding feature data can be uploaded or downloaded accordingly.

[0092] In step S103, the geometric features and the color data of the pixels in the color depth image are fused to obtain the semantic features of the color depth image.

[0093] In one embodiment of this disclosure, it is necessary to perform feature fusion on image color gamut features (view-dependent) and geometric features downloaded from point cloud (view-independent).

[0094] Furthermore, Figure 3 This illustration schematically depicts a flowchart of a method for obtaining semantic features according to an exemplary embodiment of this disclosure. Figure 3 As shown, the method for obtaining semantic features includes steps S301 to S303:

[0095] Step S301: Encode the geometric features using a first method to obtain first encoded data, and encode the color data using a second method to obtain second encoded data;

[0096] Step S302: Upsample the first encoded data and the second encoded data respectively to obtain the fusion result;

[0097] Step S303: Decode the fusion result to obtain the semantic features.

[0098] Specifically, since the rotation-invariant geometric features on the point cloud do not have the same data distribution pattern as the color features, they cannot be pre-trained. Therefore, this disclosure designs two different encoders for color data and geometric features respectively.

[0099] When designing an encoder, a pre-trained convolutional neural network model can be used, followed by appropriate optimization, which can save training time and improve network performance. For example, a geometric encoder can be trained using the ResNet101 framework to encode geometric features; and a color encoder can be fine-tuned using a pre-trained ResNet152 to encode color data. Then, the encoded features are upsampled, aligned, and fused; finally, they are jointly decoded to generate the semantic features of the image domain corresponding to the color depth image, denoted as feat. img .

[0100] It should be noted that the depth channel and color channel do not have to be aligned (due to inconsistent resolution). Aligning the two data directly through interpolation is feasible, but it cannot optimize the use of the pre-trained color feature codec.

[0101] In step S104, semantic segmentation is performed based on the semantic features and the temporal semantic state to obtain the semantic segmentation result of the pixel.

[0102] In one embodiment of this disclosure, a multimodal semantic feature network can be constructed based on a recurrent neural network to achieve semantic segmentation. Further, the step of obtaining the semantic segmentation result of the pixel based on the semantic features and the temporal semantic state includes: for a pixel, extracting temporal features using convolutional layers in a recurrent neural network based on the semantic features and the temporal semantic state of the current registration point corresponding to that pixel; wherein the recurrent neural network includes two long short-term memory networks; calculating a semantic score using fully connected layers in the recurrent neural network based on the temporal features; predicting the semantic segmentation result of the pixel based on the semantic score; and traversing each pixel in the color depth image to obtain the semantic segmentation result corresponding to each pixel.

[0103] Specifically, for the pixel points of the acquired current frame color depth image, I(i) = (u i v i The corresponding point cloud voxels are registered as V(i), and the semantic features in the image domain are feat. img The timing status value is status. i Therefore, the temporal update of semantic features is as follows:

[0104] (feat used status i =RNN(feat) img status i )

[0105] Score fused =FC(feat) used )

[0106] label i =argmax{Score fused}

[0107] Wherein, RNN stands for Recurrent Neural Network, and FC stands for Temporal Feature Architecture. used The fully connected layer between the semantic score and the predicted label i As the semantic segmentation result of pixel I(i).

[0108] Figure 4 The diagram illustrates the structure of a multimodal semantic feature network according to an exemplary embodiment of this disclosure. Using continuous RGB-D images and depth camera poses as input, the multimodal semantic feature network outputs pixel-level semantic information and semantic segmentation results of 3D point clouds online.

[0109] like Figure 4As shown, this multimodal semantic feature network first acquires depth and color data from RGB-D images, and uses the depth data for registration and fusion to construct a 3D point cloud to establish a link between the image and the point cloud. Then, the geometric features on the point cloud are downloaded to the current 2D image through the link relationship. Multimodal semantic feature learning is performed by combining the image color domain features (viewpoint-dependent) and the geometric features downloaded from the point cloud (viewpoint-independent). The geometric encoder 40 and color encoder 41 are used to encode the geometric features and color data respectively, and the encoded features are upsampled for data alignment and fusion to jointly decode and generate semantic features. Finally, the link relationship is used again to maintain the temporal semantic state of the point cloud during the registration process through a recurrent neural network, and the final point cloud semantics are optimized through an upload and download mechanism.

[0110] Among them, blocks 401, 403, 404, 405, 410, 411, 412, 413, 421, 422, 424, and 425 are convolutional layers, consisting of a cascade of Conv+BN+ReLU; blocks 402 and 423 are pooling layers; block 406 is an Atrous Spatial Pyramid Pooling (ASPP) layer; block 428 is a Recurrent Neural Network (RNN), which consists of two Long Short-Term Memory (LSTM) networks; blocks 427 and 429 are used to record the temporal semantic state of the point cloud, which changes as the vertex is fused; blocks 426 and 430 are semantic features; and block 431 is the semantic segmentation result.

[0111] Based on the above method, this disclosure designs a recurrent neural network to remember temporal state features for semantic feature changes in the temporal domain of point clouds. Point cloud fusion optimizes the computational cost of geometric feature extraction and can maintain the temporal feature sequence of each vertex under different perspectives to establish the link relationship between frames under different perspectives to optimize the semantic recognition results and improve the accuracy and consistency of semantic segmentation.

[0112] Figure 5 This schematic diagram illustrates the composition of an online point cloud semantic segmentation apparatus according to an exemplary embodiment of the present disclosure, such as... Figure 5 As shown, the online point cloud semantic segmentation device 500 may include an acquisition module 501, a fusion module 502, and a feature module 503. Wherein:

[0113] The acquisition module 501 is used to acquire color depth images in real time, and convert the pixels into point clouds based on the depth data of the pixels in the color depth images to obtain the current point cloud set.

[0114] The fusion module 502 is used to perform vertex fusion between the current point cloud set and the historical point cloud set through point cloud registration, so as to obtain the geometric features and temporal semantic state corresponding to the pixel.

[0115] The feature module 503 is used to perform feature fusion on the geometric features and the color data of the pixels in the color depth image to obtain the semantic features of the color depth image;

[0116] The segmentation module 504 is used to perform semantic segmentation based on the semantic features and the temporal semantic state to obtain the semantic segmentation result of the pixel.

[0117] According to an exemplary embodiment of this disclosure, the acquisition module 501 is further configured to acquire camera parameters for acquiring the color depth image; for a pixel, convert the pixel into a voxel point in a three-dimensional coordinate system according to the depth data of the pixel and the camera parameters; and traverse each pixel in the color depth image to obtain the current point cloud composed of the voxel points corresponding to each pixel.

[0118] According to an exemplary embodiment of this disclosure, the historical point cloud includes registered historical registration points, and the fusion module 502 further includes a registration unit, a judgment unit, and a fusion unit. The registration unit is used to convert the coordinate data of voxels in the current point cloud to register a current registration point. The judgment unit is used to determine whether there are current registration points and historical registration points to be fused. The fusion unit is used to merge the current registration points and the historical registration points according to a preset fusion rule to update the historical point cloud when there are current registration points and historical registration points to be fused.

[0119] According to an exemplary embodiment of this disclosure, the determining unit is configured to, for the current point cloud - current registered point, determine the first position of the current registered point on the first coordinate axis, the second position on the second coordinate axis, and the third position on the third coordinate axis; search for historical registered points matching the first position from the historical point cloud to form a first historical point cloud; when the first historical point cloud is a non-empty set, search for historical registered points matching the second position from the first historical point cloud to form a second historical point cloud; when the second historical point cloud is a non-empty set, search for historical registered points matching the second position from the second historical point cloud to determine whether there are current registered points and historical registered points to be merged.

[0120] According to an exemplary embodiment of this disclosure, the fusion module 502 further includes an acquisition unit, configured to acquire the registration timestamp, geometric features, temporal semantic state, and point cloud semantic information of the current registration point for use in the vertex fusion; wherein the geometric features include one or more of the following: the point cloud boundary distance value of the current registration point relative to the current point cloud set, the point cloud minimum point distance value, the normal vector, and the curvature.

[0121] According to an exemplary embodiment of this disclosure, the fusion unit is configured to replace the coordinate data and registration timestamp of the current registration point with the registration timestamp of the historical registration point when the registration timestamp of the current registration point and the registration timestamp of the historical registration point meet a preset condition; and to replace the coordinate data, registration timestamp, geometric features, temporal semantic state, and point cloud semantic information of the current registration point with the registration timestamp of the historical registration point when the registration timestamp of the current registration point and the registration timestamp of the historical registration point do not meet the preset condition.

[0122] According to an exemplary embodiment of this disclosure, the feature module 503 is configured to encode the geometric features using a first method to obtain first encoded data, and to encode the color data using a second method to obtain second encoded data; to upsample the first encoded data and the second encoded data respectively to obtain a fusion result; and to decode the fusion result to obtain the semantic features.

[0123] According to an exemplary embodiment of this disclosure, the segmentation module 504 is configured to extract temporal features for a pixel using convolutional layers in a recurrent neural network based on the semantic features and the temporal semantic state of the current registration point corresponding to the pixel; wherein the recurrent neural network includes two long short-term memory networks; calculate a semantic score using fully connected layers in the recurrent neural network based on the temporal features; predict the semantic segmentation result of the pixel based on the semantic score; and traverse each pixel in the color depth image to obtain the semantic segmentation result corresponding to each pixel.

[0124] The specific details of each module in the aforementioned online point cloud semantic segmentation device 500 have been described in detail in the corresponding online point cloud semantic segmentation method, so they will not be repeated here.

[0125] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0126] In an exemplary embodiment of this disclosure, a storage medium capable of implementing the above-described method is also provided. Figure 6 This schematic diagram illustrates a computer-readable storage medium according to an exemplary embodiment of the present disclosure, such as... Figure 6 As shown, a program product 600 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a mobile phone. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0127] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided. Figure 7 The schematic diagram illustrates the structure of a computer system of an electronic device according to an exemplary embodiment of the present disclosure.

[0128] It should be noted that, Figure 7 The computer system 700 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0129] like Figure 7 As shown, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 702 or programs loaded from storage section 708 into Random Access Memory (RAM) 703. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An Input / Output (I / O) interface 705 is also connected to the bus 704.

[0130] The following components are connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 710 as needed so that computer programs read from it can be installed into storage section 708 as needed.

[0131] In particular, according to embodiments of this disclosure, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs various functions defined in the system of this disclosure.

[0132] It should be noted that the computer-readable medium shown in the embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0134] The units described in the embodiments of this disclosure can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the unit itself.

[0135] In another aspect, this disclosure also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0136] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0137] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this disclosure.

[0138] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0139] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An online point cloud semantic segmentation method, characterized in that, include: Real-time acquisition of color depth images, and point cloud transformation of the pixels based on the depth data of the pixels in the color depth images to obtain the current point cloud set; By registering the point cloud, the current point cloud set and the historical point cloud set are calculated according to the preset fusion rules and vertex fusion is performed to obtain the geometric features and temporal semantic state corresponding to the pixel. The semantic features of the color depth image are obtained by feature fusion of the geometric features and the color data of the pixels in the color depth image; The semantic segmentation result of the pixel is obtained by performing semantic segmentation using a recurrent neural network based on the semantic features and the temporal semantic state; wherein, the convolutional layer of the recurrent neural network extracts temporal features based on the semantic features and the temporal semantic state, the fully connected layer of the recurrent neural network calculates a semantic score based on the temporal features, and predicts the semantic segmentation result based on the semantic score.

2. The online point cloud semantic segmentation method according to claim 1, characterized in that, The step of converting the pixels into a point cloud based on the depth data of the pixels in the color depth image to obtain the current point cloud set includes: Obtain the camera parameters for capturing the color depth image; For a given pixel, the pixel is converted into a voxel in a three-dimensional coordinate system based on the pixel's depth data and the camera parameters. Traverse each pixel in the color depth image to obtain the voxel points corresponding to each pixel, which form the current point cloud.

3. The online point cloud semantic segmentation method according to claim 1, characterized in that, The historical point cloud includes registered historical points. The step of merging the current point cloud with the historical point cloud according to a preset fusion rule and performing vertex fusion includes: The coordinate data of the voxels in the current point cloud are transformed to obtain the current registered point; Determine if there are any current or historical registration points to be merged; When there are current registration points and historical registration points to be merged, the current registration points and the historical registration points are merged according to the preset merging rules to update the historical point cluster.

4. The online point cloud semantic segmentation method according to claim 3, characterized in that, The determination of whether there are current registration points and historical registration points to be merged includes: For the current point cluster - current registration point, determine the first position of the current registration point on the first coordinate axis, the second position on the second coordinate axis, and the third position on the third coordinate axis; A first historical point set is formed by searching the historical point cloud set for historical registration points that match the first location. When the first historical point cloud is a non-empty set, search for historical registration points that match the second location from the first historical point cloud to form a second historical point cloud; When the second historical point cloud is a non-empty set, search for historical registration points that match the second position from the second historical point cloud to determine whether there are current registration points and historical registration points to be merged.

5. The online point cloud semantic segmentation method according to claim 3, characterized in that, After converting the coordinate data of the voxel points in the current point cloud to obtain the current registered point, the method further includes: The registration timestamp, geometric features, temporal semantic state, and point cloud semantic information of the current registered point are obtained for use in the vertex fusion; wherein, the geometric features include one or more of the following: the point cloud boundary distance value of the current registered point relative to the current point cloud set, the point cloud minimum point distance value, the normal vector, and the curvature.

6. The online point cloud semantic segmentation method according to claim 5, characterized in that, When there are current registration points and historical registration points to be merged, the step of merging the current registration points and the historical registration points according to preset merging rules includes: When the registration timestamp of the current registration point and the registration timestamp of the historical registration point meet the preset conditions, the coordinate data and registration timestamp of the current registration point are replaced with those of the historical registration point. When the registration timestamp of the current registration point does not meet the preset condition with the registration timestamp of the historical registration point, the coordinate data, registration timestamp, geometric features, temporal semantic state, and point cloud semantic information of the current registration point are replaced with those of the historical registration point.

7. The online point cloud semantic segmentation method according to claim 1, characterized in that, The step of fusing the geometric features and the color data of pixels in the color depth image to obtain the semantic features of the color depth image includes: The geometric features are encoded using a first method to obtain first encoded data, and the color data is encoded using a second method to obtain second encoded data; The first encoded data and the second encoded data are upsampled respectively to obtain the fusion result; The semantic features are obtained by decoding the fusion result.

8. The online point cloud semantic segmentation method according to claim 1, characterized in that, The step of obtaining the semantic segmentation result of the pixel by performing semantic segmentation using a recurrent neural network based on the semantic features and the temporal semantic state includes: For a single pixel, temporal features are extracted using convolutional layers in the recurrent neural network based on the semantic features and the temporal semantic state of the current registration point corresponding to that pixel; wherein, the recurrent neural network includes two long short-term memory networks; The semantic score is calculated using the fully connected layer in the recurrent neural network based on the temporal features. The semantic segmentation result of the pixel is obtained by predicting based on the semantic score. The semantic segmentation results corresponding to each pixel in the color depth image are obtained by traversing the image.

9. An online point cloud semantic segmentation device, characterized in that, include: The acquisition module is used to acquire color depth images in real time, and convert the pixels into point clouds based on the depth data of the pixels in the color depth images to obtain the current point cloud set. The fusion module is used to register the current point cloud set and the historical point cloud set according to a preset fusion rule and perform vertex fusion to obtain the geometric features and temporal semantic state corresponding to the pixel. The feature module is used to perform feature fusion on the geometric features and the color data of the pixels in the color depth image to obtain the semantic features of the color depth image; The segmentation module is used to perform semantic segmentation of the pixel points using a recurrent neural network based on the semantic features and the temporal semantic state; wherein, the convolutional layer of the recurrent neural network extracts temporal features based on the semantic features and the temporal semantic state, the fully connected layer of the recurrent neural network calculates a semantic score based on the temporal features, and predicts the semantic segmentation result based on the semantic score.

10. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the online point cloud semantic segmentation method as described in any one of claims 1 to 9.

11. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the online point cloud semantic segmentation method as described in any one of claims 1 to 9.

Images