Robot small sample point cloud semantic segmentation method and device

Abstract

The application relates to the technical field of robots, and provides a robot small-sample point cloud semantic segmentation method and device, which comprises the following steps: performing mean value operation on support point cloud sample masks and support point cloud sample features to obtain category prototypes and category clue features; and calculating a mapping matrix between query scene point cloud features and the category clue features, so as to obtain category-specific prototypes based on the category prototypes and the mapping matrix; performing dynamic weighted fusion on global geometric prototypes and the category-specific prototypes to obtain category-aware geometric prototypes; and matching the category-aware geometric prototypes with the query scene point cloud features to output a semantic segmentation result of a new-class object. The method provided by the application fully mines and utilizes foreground information under a small-sample condition, avoids foreground clues from being submerged in background information, realizes effective migration of fine-grained geometric information, and thus significantly improves the accuracy and comprehensive performance of new-class object point cloud semantic segmentation in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method and apparatus for semantic segmentation of small sample point clouds for robots. Background Technology

[0002] Few-shot point cloud semantic segmentation is a crucial aspect of robot 3D scene perception, playing a vital role in robot navigation and manipulation. "Few-shot" refers to the ability to segment the point cloud semantics of novel object classes using a limited set of N-way-K-shot point cloud samples, where N and K represent the number of novel object classes and the number of point cloud samples supported by each class, respectively. Examples include 1-way-1-shot and 2-way-5-shot. 2-way-5-shot refers to a scenario containing two novel object classes, with each class having 5 point cloud samples. The goal of robot few-shot point cloud semantic segmentation is to train a robot few-shot point cloud semantic segmentation network. During inference, this network segments points belonging to a novel object class from the query scene point cloud based on a limited set of labeled support point cloud samples. The robot few-shot point cloud semantic segmentation network is trained in a contextualized manner, incorporating a large set of labeled base class object support point cloud samples. Existing methods mostly extract point cloud features from supporting point cloud samples and construct category prototypes, and then perform segmentation by calculating the geometric similarity between the point cloud features corresponding to the query scene point cloud and the category prototypes.

[0003] However, in reality, the foreground object point cloud sometimes constitutes a very small proportion of the background point cloud in the query scene point cloud. Existing methods mostly treat foreground and background point clouds equally, resulting in foreground cues being submerged in background information. This leads to insufficient mining and utilization of foreground information, reducing the performance of robot semantic segmentation of small sample point clouds. Summary of the Invention

[0004] This invention provides a robot few-sample point cloud semantic segmentation method and apparatus to solve the defect in the existing robot few-sample point cloud semantic segmentation method that, by processing the foreground object point cloud and the background point cloud equally, the mining and utilization of foreground information is insufficient, thus reducing the performance of robot few-sample point cloud semantic segmentation.

[0005] This invention provides a robot few-sample point cloud semantic segmentation method, comprising: Obtain the query scene point cloud of the scene to be detected where the robot is located, as well as the supporting point cloud samples and supporting point cloud sample masks of the new type of objects in the scene to be detected; Based on the query scenario point cloud features, the support point cloud sample features, and the support point cloud sample mask, a global geometric prototype is generated. The supporting point cloud sample mask and the supporting point cloud sample features are averaged to obtain category prototypes and category clue features. The mapping matrix between the query scene point cloud features and the category clue features is calculated to obtain a category-specific prototype based on the category prototype and the mapping matrix. The global geometric prototype and the category-specific prototype are dynamically weighted and fused to obtain a category-aware geometric prototype. The category-aware geometric prototype is matched with the point cloud features of the query scene to output the semantic segmentation result of the new object class.

[0006] According to the present invention, a robot few-sample point cloud semantic segmentation method is provided, wherein calculating the mapping matrix between the query scene point cloud features and the category cue features includes: The point cloud features of the query scene are subjected to pooling downsampling processing to obtain downsampled query features; Calculate the product between the downsampled query features and the category cue features to obtain the category cue feature association matrix; The category clue feature association matrix is ​​transposed and subjected to multi-level linear mapping to obtain the mapping matrix.

[0007] According to the present invention, a robot few-sample point cloud semantic segmentation method is provided, wherein the step of calculating a category-specific prototype based on the category prototype and the mapping matrix includes: The mapping matrix is ​​normalized, and the product of the normalized mapping matrix and the category prototype is calculated to obtain the category-specific prototype.

[0008] According to the present invention, a robot few-sample point cloud semantic segmentation method is provided, wherein the dynamic weighted fusion of the global geometric prototype and the category-specific prototype to obtain a category-aware geometric prototype includes: The global geometric prototype and the category-specific prototype are linearly mapped to obtain the mapped global geometric prototype and the mapped category-specific prototype, respectively. The category-aware geometric prototype is obtained by adding the global geometric prototype of the mapping and the category-specific prototype of the mapping.

[0009] According to the present invention, a robot few-sample point cloud semantic segmentation method is provided, wherein matching the category-aware geometric prototype with the point cloud features of the query scene and outputting the semantic segmentation result of the new object class includes: Calculate the matching score matrix between the category-aware geometric prototype and the point cloud features of the query scene; Perform a maximum value index extraction operation on the last dimension of the matching score matrix to obtain the semantic segmentation result of the new type of object.

[0010] According to a robot few-sample point cloud semantic segmentation method provided by the present invention, the step of obtaining the query scene point cloud includes: Acquire color and depth images of the scene to be detected; Extract the pixel coordinates of each pixel in the color image, and the depth value corresponding to each pixel in the depth image; Based on the pixel coordinates and depth values ​​corresponding to each pixel, as well as the camera intrinsic data, the three-dimensional point coordinates corresponding to each pixel are calculated. The coordinates of all three-dimensional points corresponding to the scene to be detected are collected to obtain the point cloud of the query scene.

[0011] According to the present invention, a robot few-sample point cloud semantic segmentation method is provided, wherein acquiring the color image and depth image of the scene to be detected includes: The color image and the depth image are acquired using a visual sensor; The visual sensor is mounted on the robot.

[0012] According to the present invention, a robot few-sample point cloud semantic segmentation method is provided, wherein the method for acquiring point cloud samples includes: Obtain the initial object point cloud corresponding to the new type of object; Determine a preset number of target point clouds in the initial object point cloud; The target point cloud is labeled one by one to obtain the supporting point cloud sample.

[0013] According to the present invention, a robot few-sample point cloud semantic segmentation method is provided, wherein generating a global geometric prototype based on the query scene point cloud features, the support point cloud sample features, and the support point cloud sample mask includes: Based on graph convolutional networks, feature extraction is performed on the query scene point cloud and the support point cloud samples respectively to obtain the query scene point cloud features and the support point cloud sample features; The query scene point cloud features, the supporting point cloud sample features, and the supporting point cloud sample mask are input into the general point cloud semantic segmentation network to obtain the global geometric prototype generated by the general point cloud semantic segmentation network.

[0014] The present invention also provides a robot few-sample point cloud semantic segmentation device, comprising: The acquisition unit acquires the query scene point cloud of the scene to be detected where the robot is located, as well as the supporting point cloud samples and supporting point cloud sample masks of the new type of objects in the scene to be detected. The global prototype generation unit generates a global geometric prototype based on the query scene point cloud features, the support point cloud sample features, and the support point cloud sample mask. The geometric transfer unit performs a mean-averaging operation on the support point cloud sample mask and the support point cloud sample features to obtain category prototypes and category clue features. It also calculates a mapping matrix between the query scene point cloud features and the category clue features to calculate a category-specific prototype based on the category prototype and the mapping matrix. The prototype fusion unit performs dynamic weighted fusion of the global geometric prototype and the category-specific prototype to obtain a category-aware geometric prototype. The semantic segmentation unit matches the category-aware geometric prototype with the point cloud features of the query scene and outputs the semantic segmentation result of the new object class.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot few-sample point cloud semantic segmentation method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot few-sample point cloud semantic segmentation method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot few-sample point cloud semantic segmentation method as described above.

[0018] The robot few-sample point cloud semantic segmentation method and apparatus provided by this invention adds a category-specific geometric transfer process on the basis of global geometric transfer. It extracts category cue features using support point cloud sample masks and support point cloud sample features, and generates category-specific prototypes. Then, it performs adaptive dynamic weighted fusion with global geometric prototypes to obtain semantic segmentation results of new object classes. It fully explores and utilizes foreground information under few-sample conditions, avoids foreground cues being submerged in background information, and realizes effective transfer of fine-grained geometric information, thereby significantly improving the accuracy and overall performance of semantic segmentation of new object point clouds in complex scenes. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the robot few-sample point cloud semantic segmentation method provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the robot small sample point cloud semantic segmentation device provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] It should be understood that, in the description of the embodiments of the present invention, "multiple" means two or more, unless otherwise explicitly specified. Furthermore, all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the applicable data protection laws and regulations of the locality and with permission granted by the owner of the respective device.

[0023] Existing robot few-sample point cloud semantic segmentation methods suffer from insufficient foreground information mining and utilization, leading to reduced segmentation performance, because they treat foreground and background point clouds equally. This invention provides a robot few-sample point cloud semantic segmentation method to improve the accuracy and overall performance of point cloud semantic segmentation. Figure 1 This is a flowchart illustrating the robot few-sample point cloud semantic segmentation method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes: Step 110: Obtain the query scene point cloud of the scene to be detected where the robot is located, as well as the support point cloud samples and support point cloud sample masks of the new type of objects in the scene to be detected.

[0024] In practical 3D scene perception applications, the scene to be detected typically refers to the 3D physical space that needs to be analyzed and perceived, such as an indoor room environment. The query scene point cloud is the set of 3D points representing the scene to be detected, containing all the geometric structures and spatial information within the scene. Here, a new type of object refers to a target category that was not seen during the initial large-scale base class training phase of the network but needs to be identified in the current application phase. Such tasks are usually set under the condition of a small number of samples (N-way-K-shot, where N is the number of new types of objects and K is the number of support samples for each new type of object). Therefore, the support point cloud samples here refer to a small number of point cloud examples with labeled information provided for these new types of objects, while the support point cloud sample mask is the real label data used to accurately indicate the region (i.e., the foreground region) where the new type of object is located in the support point cloud samples.

[0025] Specifically, robots equipped with visual sensors and other devices can scan and transform the scene to obtain a point cloud of the query scene that reflects its true physical dimensions. Simultaneously, for each newly defined object category, a very small number of point cloud samples are acquired and labeled point by point to construct supporting point cloud samples and their corresponding mask information.

[0026] Step 120: Generate a global geometric prototype based on the query scene point cloud features, the supporting point cloud sample features, and the supporting point cloud sample mask.

[0027] Here, the query scene point cloud features and supporting point cloud sample features are deep feature representations obtained by extracting high-dimensional semantic features from the original point cloud coordinate set. They contain the local geometric structure of the point cloud and global contextual information. In addition, the global geometric prototype here is a feature reference template constructed to represent the geometric distribution law of the overall scene after transferring the prior knowledge in the supporting point cloud samples to the query scene point cloud feature space.

[0028] Specifically, firstly, a pre-constructed point cloud feature extraction network can be used to extract the corresponding high-dimensional feature matrices from the original query scene point cloud and support point cloud samples, respectively, thus obtaining the query scene point cloud features and support point cloud sample features. Subsequently, the query scene point cloud features, support point cloud sample features, and support point cloud sample masks used to distinguish foreground and background can be input into the currently trained general point cloud segmentation network. The general point cloud segmentation network performs joint processing and information interaction to output a global geometric prototype.

[0029] By incorporating masking information, the global geometric transfer process can learn the macroscopic distribution in supporting samples, thereby generating a global geometric prototype with universal guiding significance for the entire scene. Understandably, by extracting the global geometric prototype, widely existing background and overall structural information in the scene is effectively captured, providing a fundamental global geometric reference for subsequent semantic segmentation.

[0030] Step 130: Perform mean averaging on the support point cloud sample mask and the support point cloud sample features to obtain category prototypes and category clue features; and calculate the mapping matrix between the query scene point cloud features and the category clue features to calculate a category-specific prototype based on the category prototype and the mapping matrix.

[0031] Here, the category prototype represents the basic category-centric semantics of the new object class in the feature space. The category cue features can be considered as further condensing the key cue information of the new object class under the difference between foreground and background. Additionally, the mapping matrix here reflects the degree of association or similarity between each point in the query scene and the known category cues of the new class. Here, the category-specific prototype is a refined feature template specifically tailored for the target new object class in the current query scene.

[0032] Specifically, the foreground portion of the support point cloud sample features can be extracted using a support point cloud sample mask. By performing averaging operations of different dimensions, such as average pooling, we can output both a macroscopically instructive category prototype and category cue features containing sufficient foreground information. It should be noted that the averaging operation here aims to extract the most representative essential information from the complex feature matrix.

[0033] Next, matrix operations are performed using the query scene point cloud features and category clue features, such as calculating products or similarities, to establish the correlation between the two in the feature space, thereby calculating a mapping matrix. Finally, this mapping matrix is ​​applied to the previously extracted category prototypes to generate category-specific prototypes.

[0034] It should be noted that this process fully explores and utilizes extremely limited foreground object information, effectively preventing key foreground cues from being overwhelmed by a large amount of redundant background information, and achieving effective transfer of refined features for specific categories.

[0035] Step 140: Perform dynamic weighted fusion of the global geometric prototype and the category-specific prototype to obtain the category-aware geometric prototype.

[0036] Here, the category-aware geometric prototype is the final geometric reference system that integrates the global view of the scene with the local details of the foreground.

[0037] Specifically, through linear mapping or other attention mechanisms, the information contained in the global geometric prototype and the category-specific prototype is adaptively adjusted and fused, for example, by aligning the feature dimensions and adding them together, so as to obtain a category-aware geometric prototype that has both global geometric perception and fine-grained category focusing capabilities. This allows for an adaptive balance between global background knowledge and specific foreground details, ensuring that the prototype features have high robustness and high discriminativeness when dealing with complex scenarios.

[0038] Step 150: Match the category-aware geometric prototype with the point cloud features of the query scene, and output the semantic segmentation result of the new object class.

[0039] Here, semantic segmentation results refer to predicting the category label of each 3D point in the query scene point cloud, which belongs to one of N new object categories or the background category.

[0040] Specifically, after prototype fusion, the category-aware geometric prototype, which contains rich prior knowledge, is geometrically matched with the point cloud features of the query scene to be parsed. For example, the inner product matching score between the two is calculated to measure the similarity between each point in the query scene and the category-aware geometric prototype. By performing an index extraction operation on the matching score, the category to which each point belongs can be determined, thus finally outputting the point cloud semantic segmentation result of the query scene.

[0041] The method provided in this invention adds a category-specific geometric transfer process to the global geometric transfer process. It extracts category cue features using support point cloud sample masks and support point cloud sample features, and generates a category-specific prototype. Then, it adaptively and dynamically weights and fuses the category-specific prototype with the global geometric prototype to obtain the semantic segmentation result of the new object class. This fully explores and utilizes the foreground information under small sample conditions, avoids foreground cues being submerged in background information, and achieves effective transfer of fine-grained geometric information. As a result, it significantly improves the accuracy and overall performance of semantic segmentation of point clouds of new objects in complex scenes.

[0042] Based on any of the above embodiments, step 130, calculating the mapping matrix between the query scene point cloud features and the category clue features, includes: The point cloud features of the query scene are subjected to pooling downsampling processing to obtain downsampled query features; Calculate the product between the downsampled query features and the category cue features to obtain the category cue feature association matrix; The category clue feature association matrix is ​​transposed and subjected to multi-level linear mapping to obtain the mapping matrix.

[0043] Specifically, firstly, the point cloud features of the query scene are downsampled using pooling to obtain downsampled query features. Here, the downsampled query features are the core semantic information of the query scene retained after dimensionality compression. In practice, this can be achieved by performing adaptive average pooling and other operations on the high-dimensional point cloud features of the query scene, reducing the original massive number of points to a specified dimension, thereby outputting the downsampled query features. This process significantly reduces the data dimensionality while preserving the key semantics of the query scene, reducing computational overhead for subsequent matrix calculations and aligning the feature scale.

[0044] Next, the product between the downsampled query features and the category clue features is calculated to obtain the category clue feature association matrix. Here, the category clue feature association matrix is ​​a numerical matrix reflecting the similarity or response intensity between each local region in the query scene and the new object clue. In specific implementation, the downsampled query features obtained from the dimensionality reduction above can be multiplied with the previously extracted category clue features to calculate the category clue feature association matrix that represents the interaction relationship between the two features. Through this calculation, the degree of correlation between the query scene point cloud and the known category clues in the feature space can be explicitly captured, providing a basis for accurate mapping of the prototype.

[0045] Furthermore, the category cue feature association matrix is ​​sequentially transposed and subjected to multi-level linear mapping to obtain a mapping matrix. Here, the mapping matrix can be considered as the final output weight allocation benchmark used to guide the category prototypes in scene adaptive migration. In specific implementation, the result of the previously calculated category cue feature association matrix is ​​first transposed, and then the transposed result is sequentially passed through the first and second linear layers for multi-level linear dimension mapping, and specific dimensions are exchanged to finally obtain the required mapping matrix.

[0046] The method provided in this invention performs downsampling on the point cloud features of the query scene, calculates the product of the feature and the category clue feature, and then performs transpose and multi-level linear mapping to obtain a mapping matrix. This achieves efficient fusion and alignment of query scene features and specific category clues, thereby generating a mapping relationship that accurately reflects the relationship between the scene and the category, improving the accuracy and computational efficiency of feature mapping.

[0047] Based on any of the above embodiments, step 130, calculating a category-specific prototype based on the category prototype and the mapping matrix, includes: The mapping matrix is ​​normalized, and the product of the normalized mapping matrix and the category prototype is calculated to obtain the category-specific prototype.

[0048] Specifically, a normalization function, such as the Softmax function, can be used to normalize the previously calculated mapping matrix, transforming the associated values ​​in the matrix into a probabilistic weight distribution. Next, the product of the normalized mapping matrix and the extracted category prototype is calculated to obtain the category-specific prototype. It should be noted that this weighted multiplication method allows for a smooth and precise allocation of the prior knowledge of new object classes contained in the category prototype according to the actual distribution patterns of the query scene, thereby generating a category-specific prototype highly focused on the specific foreground details of the current scene.

[0049] The method provided in this invention normalizes the mapping matrix and calculates its product with the category prototype to obtain a category-specific prototype. This effectively transforms the original mapping association into a smooth probability distribution weight, achieving precise weighted guidance of the category prototype in the query scenario. As a result, a customized category-specific prototype that is highly adapted to the current scenario is obtained, significantly enhancing the fine-grained recognition capability of robot small sample point cloud semantic segmentation.

[0050] In one embodiment, this can be achieved by constructing a category-specific geometry transfer branch, which supports point cloud sample features. Supports point cloud sample masks Query scene point cloud features Input to the category-specific geometry transfer branch, and obtain the category-specific prototype output by the category-specific geometry transfer branch. .

[0051] First, the network structure of the category-specific geometry transition branch is explained. Here, the category-specific geometry transition branch includes the prototype pooling operation MeanPool_1 ( ,1) and the category cue prototype transfer unit. The category cue prototype transfer unit consists of MeanPool_2, which supports point cloud sample pooling operations. ,64), Calculation of the correlation matrix of category clue features, first linear layer Second linear layer The prototype association matrix is ​​constructed by cascading calculations. MeanPool_1 ( ,1) 1 and 1 are the input and output dimensions of the first prototype pooling operation, respectively. In this embodiment of the invention... The preferred value is 2048; MeanPool_2( ,64) 64 and 64 are the input and output dimensions, respectively, supporting point cloud sample pooling operations. In this embodiment of the invention... The preferred value is 2048; of and These are the input and output dimensions of the first linear layer, respectively, in this embodiment of the invention. The preferred value is 64. The preferred value is 128; of and These are the input and output dimensions of the second linear layer, respectively, in this embodiment of the invention. The preferred value is 2. Additionally, the first prototype pooling operation MeanPool_1 ( ,1) Implemented by the nn.Avgmeanpool function in the PyTorch library, supporting point cloud sample pooling operations MeanPool_2( ,64) Implemented by the nn.AdaptiveAvgPool1d function in the PyTorch library, the first linear layer Second linear layer All of these are implemented using the nn.Linear function in the PyTorch library.

[0052] Secondly, in the category-specific geometry transfer branch, the data processing flow may include: first, supporting the masking of point cloud samples. and support point cloud sample features After prototype pooling operation MeanPool_1( , Process and output category prototypes. Its size is N ×( N +1)×128. Additionally, it supports masking of point cloud samples. and support point cloud sample features After supporting point cloud sample pooling operation MeanPool_2( ,64) Process and output category cue features Its size is N ×64×128.

[0053] Then, query the scene point cloud features. After supporting point cloud sample pooling operation MeanPool_2( Process 64) and output downsampling query features. Its size is N ×64×128. Next, downsampling query features. Category cue features Calculate the category cue feature association matrix to obtain the category cue feature association matrix. Its size N ×64×64, the calculation method is as follows: , here This refers to the torch.matmul function in the PyTorch library.

[0054] Furthermore, it can be achieved through the first linear layer. Association matrix of features for category clues The processed result is transposed and then passed through a second linear layer. Processing yields the mapping feature correlation matrix. Its size is N ×128×( N+ 1). Next, the feature correlation matrix is ​​mapped. The mapping matrix is ​​obtained by swapping the second and third dimensions. Its size is N ×( N+ 1)×128. Then, through the mapping matrix... and category prototype Perform prototype association matrix calculation to obtain category-specific prototypes. Its size is N ×( N+ 1) × 128, the calculation method is as follows: ,in It refers to To perform the softmax operation, use the nn.Softmax function in the PyTorch library.

[0055] Based on any of the above embodiments, step 140 includes: The global geometric prototype and the category-specific prototype are linearly mapped to obtain the mapped global geometric prototype and the mapped category-specific prototype, respectively. The category-aware geometric prototype is obtained by adding the global geometric prototype of the mapping and the category-specific prototype of the mapping.

[0056] Specifically, firstly, linear mapping is performed on the global geometric prototype and the category-specific prototype respectively to obtain the mapped global geometric prototype and the mapped category-specific prototype. In practice, the aforementioned global geometric prototype and category-specific prototype can be input into different linear transformation layers for processing, thereby outputting the mapped global geometric prototype and the mapped category-specific prototype with consistent dimensions and semantic space alignment.

[0057] It should be noted that this process, while preserving the advantages of each feature, projects macroscopic global geometric information and microscopic category-specific details into a unified feature space, achieving dimensional unification for subsequent feature fusion. Therefore, the mapped global geometric prototype and mapped category-specific prototype here are feature representations aligned in both feature dimension and semantic scale after the aforementioned feature space transformation.

[0058] Then, the global geometric prototype and the category-specific prototype are added together to obtain the category-aware geometric prototype. Here, the category-aware geometric prototype is the final comprehensive feature that integrates the overall scene structure and the details of the foreground object. In specific implementation, the obtained global geometric prototype and the category-specific prototype are directly matrix-added to perform element-level numerical summation to generate the category-aware geometric prototype.

[0059] The method provided in this invention aligns the feature space by performing linear mapping on the global geometric prototype and the category-specific prototype respectively, and then effectively combines the two by matrix addition. This achieves dynamic adaptive fusion of global macro-geometric information and local category-specific details, thereby generating a category-aware geometric prototype with strong expressiveness and scene adaptability.

[0060] Based on any of the above embodiments, step 150 includes: Calculate the matching score matrix between the category-aware geometric prototype and the point cloud features of the query scene; Perform a maximum value index extraction operation on the last dimension of the matching score matrix to obtain the semantic segmentation result of the new type of object.

[0061] Specifically, firstly, a matching score matrix is ​​calculated between the category-aware geometric prototype and the point cloud features of the query scene. This matching score matrix is ​​a multi-dimensional numerical matrix used to quantify the geometric and semantic similarity between each 3D point in the query scene and the fused target prototype. In practice, matrix multiplication can be performed between the fused category-aware geometric prototype and the initially extracted query scene point cloud features to calculate the matching score matrix representing the degree of spatial matching between the two, thus transforming the high-dimensional feature vector into an intuitive similarity score.

[0062] Next, a maximum index extraction operation is performed on the last dimension of the matching score matrix to obtain the semantic segmentation result of the new object class. Here, the last dimension typically corresponds to the category dimension, containing the classification of background and new object classes. The maximum index extraction operation is used to find the category index with the highest score in the category dimension, which serves as the final predicted label. In specific implementation, the index position with the largest value is found on the last dimension of the smoothed matching score matrix, and the category label corresponding to that index is assigned to the corresponding 3D point, thereby outputting the final semantic segmentation result of the point cloud of the scene to be detected.

[0063] The method provided in this invention calculates the matching score matrix between the category-aware geometric prototype and the point cloud features of the query scene, and then performs a maximum value index extraction operation to transform the high-dimensional feature matching score into a clear point-by-point category label, thereby achieving robust output of the prediction results and significantly improving the point cloud semantic segmentation accuracy of new object classes in complex 3D scenes.

[0064] In one embodiment, a globally geometric prototype can be dynamically weighted using an adaptive prototype fusion module. and category-specific prototypes This process then outputs the semantic segmentation result of the query scene point cloud. Here, the adaptive prototype fusion module includes a prototype weighting unit and a geometric matching unit. The prototype weighting unit outputs a category-aware geometric prototype, and the geometric matching unit matches the query scene point cloud with the category-aware geometric prototype to output the semantic segmentation result of the query scene point cloud.

[0065] In the prototype weighted unit, the global geometric prototype is received. and category-specific prototypes Output category-aware geometric prototype Here, the prototype weighted unit consists of the third linear layer. Fourth linear layer Composition. Among them, Both the input dimension and the output dimension are , Both the input dimension and the output dimension are Third linear layer and the fourth linear layer All are implemented using the `nn.Linear` function from the PyTorch library. Global geometric prototype. and category-specific prototypes Passing through the third linear layer respectively and the fourth linear layer Processing yields the mapped global geometric prototype. and mapping category-specific prototypes , and All dimensions are N ×( N+ 1)×128. Then and Addition to form a category-aware geometric prototype Its size is N ×( N+ 1)×128.

[0066] In the geometric matching unit, a category-aware geometric prototype is received. Point cloud features in the query scenario Output geometric matching score Its size is N ×2048×( N+ 1) The calculation method is as follows: And in geometric matching score The argmax operation is performed on the last dimension to obtain the category prediction result for each point, thereby completing the semantic segmentation of the point cloud in the query scenario. The argmax operation is implemented using the torch.Argmax function in the PyTorch library.

[0067] Based on any of the above embodiments, step 110, the step of obtaining the scene point cloud, includes: Acquire color and depth images of the scene to be detected; Extract the pixel coordinates of each pixel in the color image, and the depth value corresponding to each pixel in the depth image; Based on the pixel coordinates and depth values ​​corresponding to each pixel, as well as the camera intrinsic data, the three-dimensional point coordinates corresponding to each pixel are calculated. The coordinates of all three-dimensional points corresponding to the scene to be detected are collected to obtain the point cloud of the query scene.

[0068] Specifically, firstly, color and depth images of the scene to be detected are acquired. In practice, this can be done by using appropriate image acquisition devices to capture images of the unknown 3D space, simultaneously acquiring color and depth images corresponding to the viewpoint.

[0069] Next, the pixel coordinates of each pixel in the color image and the depth value corresponding to each pixel in the depth image are extracted. In specific implementation, the acquired color image is traversed to extract the horizontal and vertical pixel coordinates of each pixel, and the depth value at the corresponding position is read from the depth image using the alignment relationship between the color image and the depth image.

[0070] Furthermore, based on the pixel coordinates and depth values ​​corresponding to each pixel, as well as the camera intrinsic data, the 3D point coordinates corresponding to each pixel are calculated. The camera intrinsic data here includes optical attribute parameters such as the focal length and principal point of the image acquisition device. In specific implementation, based on the principle of a pinhole camera model, the horizontal and vertical focal lengths and principal point coordinates from the camera intrinsic data, combined with the extracted pixel coordinates and depth values, are used to perform spatial coordinate system transformation calculations. This allows the 2D image pixels to be back-projected into 3D space, thereby obtaining accurate 3D point coordinates, accurately restoring the 2D multimodal image information into the 3D structure of the real physical space.

[0071] Here, the 3D coordinates of any pixel can be calculated using the following formula, as shown below: ; In the formula, Represents pixels in an RGB image The pixel coordinates are the pixel points. The corresponding depth value in the depth image. These are the coordinates of the principal point in the pixel coordinate system within the intrinsic parameters of the RGB-D camera. These represent the camera's intrinsic parameters. shaft and Focal length along the axial direction, Pixels in an RGB image The corresponding three-dimensional point coordinates.

[0072] Finally, the coordinates of all 3D points corresponding to the scene to be detected are aggregated to obtain the query scene point cloud. In specific implementation, all the 3D point coordinates calculated above are aggregated into an unordered set of 3D points, which constitutes the query scene point cloud used for subsequent semantic segmentation network parsing.

[0073] Based on any of the above embodiments, acquiring a color image and a depth image of the scene to be detected includes: The color image and the depth image are acquired using a visual sensor; The visual sensor is mounted on the robot.

[0074] The visual sensor used here is typically a Red Green Blue-Depth Camera (RGB-D Camera), which can simultaneously capture color and depth information. The Kinectv2 is the preferred RGB-D camera. In practice, the visual sensor is mounted on a platform such as the robot's mobile chassis or robotic arm. When the robot moves or performs tasks in a 3D scene, it uses its onboard visual sensor to scan the surrounding environment in real time, thereby acquiring corresponding color and depth images.

[0075] The method provided in this invention enables the robot to autonomously perceive 3D scene data in unknown environments by mounting a vision sensor on the robot to acquire color and depth images in real time, thus obtaining basic data for downstream tasks such as robot visual navigation and grasping operations.

[0076] Based on any of the above embodiments, in step 110, the method for acquiring point cloud samples includes: Obtain the initial object point cloud corresponding to the new type of object; Determine a preset number of target point clouds in the initial object point cloud; The target point cloud is labeled one by one to obtain the supporting point cloud sample.

[0077] Specifically, firstly, the initial object point cloud corresponding to the new object type is obtained. Here, the initial object point cloud refers to the original 3D point set collected by sensors, containing the new object itself and a small amount of cluttered background around it. In practice, a specific new object type can be scanned to obtain its original 3D structure data as the initial object point cloud. It should be noted that the method for obtaining the initial object point cloud can be the same as the method described above for querying scene point clouds, and will not be elaborated further here.

[0078] Next, a predetermined number of target point clouds are determined from the initial object point cloud. This predetermined number corresponds to the sample number requirement in few-shot learning tasks, i.e., the K value in K-shot tasks, such as 1 or 5. In practice, high-quality point clouds that meet the required number are selected from the collected initial object point clouds as target point clouds to control subsequent annotation costs.

[0079] Furthermore, the target point cloud is labeled point by point to obtain supporting point cloud samples. In practice, the selected target point cloud is manually or semi-automatically labeled point by point to clearly distinguish the foreground target points and background points belonging to the new object class, and to generate supporting point cloud samples with mask labels.

[0080] The method provided in this invention obtains an initial object point cloud and selects a preset number of target point clouds for annotation one by one, thereby constructing an accurate small sample support set with minimal annotation cost, which meets the small sample data requirements of new object recognition tasks.

[0081] Based on any of the above embodiments, step 120 includes: Based on graph convolutional networks, feature extraction is performed on the query scene point cloud and the support point cloud samples respectively to obtain the query scene point cloud features and the support point cloud sample features; The query scene point cloud features, the supporting point cloud sample features, and the supporting point cloud sample mask are input into the general point cloud semantic segmentation network to obtain the global geometric prototype generated by the general point cloud semantic segmentation network.

[0082] Specifically, firstly, based on a graph convolutional network, features are extracted from both the query scene point cloud and the support point cloud samples to obtain the query scene point cloud features and the support point cloud sample features. Here, the graph convolutional network is preferably a Dynamic Graph Convolutional Neural Network (DGCNN). In the specific implementation, the query scene point cloud and the support point cloud samples are input into the graph convolutional network. By establishing a dynamic adjacency graph locally in the point cloud and aggregating information, a high-dimensional feature matrix containing rich local geometric structure and contextual information is extracted, thus obtaining the query scene point cloud features and the support point cloud sample features.

[0083] In one embodiment, the query scene point cloud can be... The data is fed into a point cloud feature extraction module constructed from a graph convolutional network to obtain the point cloud features of the query scene. The size is N ×2048 (number of points) ×128 (dimensions); Supports point cloud samples The data is fed into the point cloud feature extraction module to obtain the primary features of the supported point cloud samples. The size is N × K ×2048×128, then... The second dimension is averaged to obtain the supporting point cloud sample features. The size is N ×2048×128 。

[0084] Then, the query scene point cloud features, supporting point cloud sample features, and supporting point cloud sample masks are input into a general point cloud semantic segmentation network to obtain a global geometric prototype generated by the general point cloud semantic segmentation network. This general point cloud semantic segmentation network is a parameter-free network module used to achieve cross-category geometric information transfer and prototype construction, such as a parameterized few-sample 3D scene segmentation network (Seg-PN). In specific implementation, the extracted high-dimensional features, along with the supporting point cloud sample masks used to distinguish foreground and background, are directly input into this general network module for interactive computation. The general point cloud semantic segmentation network then directly outputs a global geometric prototype with a global scene view. In one embodiment, the query scene point cloud features can be used as the basis for this process. Supports point cloud sample features And masks that support point cloud samples The input is fed into the global geometry transfer branch constructed by the general point cloud semantic segmentation network, and processed by the general point cloud semantic segmentation network. , and Processing is performed to generate a global geometric prototype. The size is N ×( N +1)×128 (dimensions), where ( N The "1" in "+1") refers to the background.

[0085] Based on any of the above embodiments, the robot few-sample point cloud semantic segmentation method specifically includes: During the training phase, the training set from the ScanNet dataset can be used to train the robot's few-shot point cloud semantic segmentation network. The training process of the robot's few-shot point cloud semantic segmentation network employs the AdamW optimizer, with the loss function... Point cloud segmentation loss The point cloud segmentation loss can be expressed by the following formula, as shown in the following equation:

[0086] In the formula, The total number of object categories. The One-Hot vector of the true label of the point cloud. The semantic segmentation network prediction result for the robot's small sample point cloud belongs to the first... i The probability of a class.

[0087] In the application phase, the trained robot few-shot point cloud semantic segmentation network can be applied to the robot few-shot point cloud semantic segmentation method of this invention. The method specifically includes: First, the robot uses its own RGB-D camera to acquire RGB images and depth images of the scene to be detected, and combines these with camera intrinsic parameters to obtain the point cloud of the scene to be detected, using this point cloud as the query scene point cloud. Then, for the N new object categories and K supporting point cloud samples for each category corresponding to the N-way-K-shot segmentation task, the robot uses its own RGB-D camera to acquire K RGB images and depth images corresponding to each new object category, and combines these with camera intrinsic parameters to obtain the K point clouds corresponding to each new object category. Finally, the K point clouds corresponding to each new object category are labeled point by point to constitute the supporting point cloud samples for each new object category.

[0088] Finally, the query scene point cloud and support point cloud samples of various new object classes are input into the trained robot few-shot point cloud semantic segmentation network, which outputs the semantic segmentation result of the point cloud of the scene to be detected. Here, the robot few-shot point cloud semantic segmentation network includes a point cloud feature extraction module, a geometry transfer module, and an adaptive prototype fusion module. The geometry transfer module is used to transfer support category prototypes to the query scene point cloud feature space. The geometry transfer module includes a global geometry transfer branch and a category-specific geometry transfer branch. The global geometry transfer branch outputs a global geometric prototype, and the category-specific geometry transfer branch outputs a category-specific prototype.

[0089] The method provided in this invention adds a category-specific geometry transfer branch to the global geometry transfer branch to extract refined features of specific foreground categories by combining support point cloud sample masks. Furthermore, it incorporates adaptive prototype fusion, adaptively weighting the category-specific prototype output from the category-specific geometry transfer branch with the global geometry prototype output from the global geometry transfer branch to achieve effective transfer of fine-grained geometric information, ultimately improving the performance of robot semantic segmentation of small-sample point clouds.

[0090] Based on any of the above embodiments Figure 2 This is a schematic diagram of the structure of the robot small sample point cloud semantic segmentation device provided by the present invention, as shown below. Figure 2 As shown, the device includes: The acquisition unit 210 acquires the query scene point cloud of the scene to be detected where the robot is located, as well as the support point cloud samples and support point cloud sample masks of the new type of objects in the scene to be detected. The global prototype generation unit 220 generates a global geometric prototype based on the query scene point cloud features, the support point cloud sample features, and the support point cloud sample mask. The geometric transfer unit 230 performs a mean-averaging operation on the support point cloud sample mask and the support point cloud sample features to obtain category prototypes and category clue features, and calculates a mapping matrix between the query scene point cloud features and the category clue features, so as to calculate a category-specific prototype based on the category prototype and the mapping matrix. The prototype fusion unit 240 performs dynamic weighted fusion of the global geometric prototype and the category-specific prototype to obtain a category-aware geometric prototype. The semantic segmentation unit 250 matches the category-aware geometric prototype with the point cloud features of the query scene and outputs the semantic segmentation result of the new type of object.

[0091] The apparatus provided in this invention adds a category-specific geometric transfer process to the global geometric transfer process. It extracts category cue features using support point cloud sample masks and support point cloud sample features, and generates a category-specific prototype. Then, it adaptively and dynamically weights and fuses the category-specific prototype with the global geometric prototype to obtain the semantic segmentation result of the new object class. It fully explores and utilizes the foreground information under small sample conditions, avoids the foreground cues from being submerged in the background information, and realizes the effective transfer of fine-grained geometric information. This significantly improves the accuracy and overall performance of semantic segmentation of point clouds of new objects in complex scenes.

[0092] Based on any of the above embodiments, the geometric migration unit is specifically used for: The point cloud features of the query scene are subjected to pooling downsampling processing to obtain downsampled query features; Calculate the product between the downsampled query features and the category cue features to obtain the category cue feature association matrix; The category clue feature association matrix is ​​transposed and subjected to multi-level linear mapping to obtain the mapping matrix.

[0093] Based on any of the above embodiments, the geometric migration unit is further specifically used for: The mapping matrix is ​​normalized, and the product of the normalized mapping matrix and the category prototype is calculated to obtain the category-specific prototype.

[0094] Based on any of the above embodiments, the prototype fusion unit is specifically used for: The global geometric prototype and the category-specific prototype are linearly mapped to obtain the mapped global geometric prototype and the mapped category-specific prototype, respectively. The category-aware geometric prototype is obtained by adding the global geometric prototype of the mapping and the category-specific prototype of the mapping.

[0095] Based on any of the above embodiments, the semantic segmentation unit is specifically used for: Calculate the matching score matrix between the category-aware geometric prototype and the point cloud features of the query scene; Perform a maximum value index extraction operation on the last dimension of the matching score matrix to obtain the semantic segmentation result of the new type of object.

[0096] Based on any of the above embodiments, the acquisition unit is specifically used for: Acquire color and depth images of the scene to be detected; Extract the pixel coordinates of each pixel in the color image, and the depth value corresponding to each pixel in the depth image; Based on the pixel coordinates and depth values ​​corresponding to each pixel, as well as the camera intrinsic data, the three-dimensional point coordinates corresponding to each pixel are calculated. The coordinates of all three-dimensional points corresponding to the scene to be detected are collected to obtain the point cloud of the query scene.

[0097] Based on any of the above embodiments, the acquisition unit is further specifically used for: The color image and the depth image are acquired using a visual sensor; The visual sensor is mounted on the robot.

[0098] Based on any of the above embodiments, the acquisition unit is further specifically used for: Obtain the initial object point cloud corresponding to the new type of object; Determine a preset number of target point clouds in the initial object point cloud; The target point cloud is labeled one by one to obtain the supporting point cloud sample.

[0099] Based on any of the above embodiments, the global prototype generation unit is specifically used for: Based on graph convolutional networks, feature extraction is performed on the query scene point cloud and the support point cloud samples respectively to obtain the query scene point cloud features and the support point cloud sample features; The query scene point cloud features, the supporting point cloud sample features, and the supporting point cloud sample mask are input into the general point cloud semantic segmentation network to obtain the global geometric prototype generated by the general point cloud semantic segmentation network.

[0100] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a robot few-sample point cloud semantic segmentation method. This method includes: acquiring a query scene point cloud in the scene to be detected where the robot is located, as well as supporting point cloud samples and a supporting point cloud sample mask for a new type of object in the scene to be detected; generating a global geometric prototype based on the query scene point cloud features, the supporting point cloud sample features, and the supporting point cloud sample mask; performing a mean operation on the supporting point cloud sample mask and the supporting point cloud sample features to obtain a category prototype and category cue features; calculating a mapping matrix between the query scene point cloud features and the category cue features to calculate a category-specific prototype based on the category prototype and the mapping matrix; performing dynamic weighted fusion on the global geometric prototype and the category-specific prototype to obtain a category-aware geometric prototype; matching the category-aware geometric prototype with the query scene point cloud features to output the semantic segmentation result of the new type of object.

[0101] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot small sample point cloud semantic segmentation method provided by the above methods. The method includes: acquiring a query scene point cloud in the scene to be detected where the robot is located, and supporting point cloud samples and supporting point cloud sample masks of new types of objects in the scene to be detected; generating a global geometric prototype based on the query scene point cloud features, the supporting point cloud sample features, and the supporting point cloud sample mask; performing a mean operation on the supporting point cloud sample mask and the supporting point cloud sample features to obtain a category prototype and category cue features; calculating a mapping matrix between the query scene point cloud features and the category cue features to calculate a category-specific prototype based on the category prototype and the mapping matrix; performing dynamic weighted fusion on the global geometric prototype and the category-specific prototype to obtain a category-aware geometric prototype; matching the category-aware geometric prototype with the query scene point cloud features to output the semantic segmentation result of the new type of object.

[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a robot small-sample point cloud semantic segmentation method provided by the above methods. The method includes: acquiring a query scene point cloud in a scene to be detected where the robot is located, and supporting point cloud samples and supporting point cloud sample masks for new types of objects in the scene to be detected; generating a global geometric prototype based on the query scene point cloud features, the supporting point cloud sample features, and the supporting point cloud sample mask; performing a mean operation on the supporting point cloud sample mask and the supporting point cloud sample features to obtain a category prototype and category cue features; calculating a mapping matrix between the query scene point cloud features and the category cue features to calculate a category-specific prototype based on the category prototype and the mapping matrix; performing dynamic weighted fusion on the global geometric prototype and the category-specific prototype to obtain a category-aware geometric prototype; matching the category-aware geometric prototype with the query scene point cloud features to output the semantic segmentation result of the new type of object.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for semantic segmentation of point clouds in robots with few samples, characterized in that, include: Obtain the query scene point cloud of the scene to be detected where the robot is located, as well as the supporting point cloud samples and supporting point cloud sample masks of the new type of objects in the scene to be detected; Based on the query scene point cloud features, the support point cloud sample features, and the support point cloud sample mask, a global geometric prototype is generated. The supporting point cloud sample mask and the supporting point cloud sample features are averaged to obtain category prototypes and category clue features. The mapping matrix between the query scene point cloud features and the category clue features is calculated to obtain a category-specific prototype based on the category prototype and the mapping matrix. The global geometric prototype and the category-specific prototype are dynamically weighted and fused to obtain a category-aware geometric prototype. The category-aware geometric prototype is matched with the point cloud features of the query scene to output the semantic segmentation result of the new object class. Based on the query scene point cloud features, the supporting point cloud sample features, and the supporting point cloud sample mask, a global geometric prototype is generated, including: Based on graph convolutional networks, feature extraction is performed on the query scene point cloud and the support point cloud samples respectively to obtain the query scene point cloud features and the support point cloud sample features; The query scene point cloud features, the supporting point cloud sample features, and the supporting point cloud sample mask are input into a general point cloud semantic segmentation network to obtain a global geometric prototype generated by the general point cloud semantic segmentation network. The global geometric prototype is a feature reference template constructed to represent the geometric distribution pattern of the overall scene after transferring the prior knowledge in the supporting point cloud samples to the query scene point cloud feature space. The calculation of the category-specific prototype based on the category prototype and the mapping matrix includes: The mapping matrix is ​​normalized, and the product of the normalized mapping matrix and the category prototype is calculated to obtain the category-specific prototype; the category-specific prototype is a refined feature template customized for the target new class of objects in the current query scenario.

2. The robot few-sample point cloud semantic segmentation method according to claim 1, characterized in that, The calculation of the mapping matrix between the query scene point cloud features and the category clue features includes: The point cloud features of the query scene are subjected to pooling downsampling processing to obtain downsampled query features; Calculate the product between the downsampled query features and the category cue features to obtain the category cue feature association matrix; The category clue feature association matrix is ​​transposed and subjected to multi-level linear mapping to obtain the mapping matrix.

3. The robot small-sample point cloud semantic segmentation method according to any one of claims 1 to 2, characterized in that, The dynamic weighted fusion of the global geometric prototype and the category-specific prototype to obtain the category-aware geometric prototype includes: The global geometric prototype and the category-specific prototype are linearly mapped to obtain the mapped global geometric prototype and the mapped category-specific prototype, respectively. The category-aware geometric prototype is obtained by adding the global geometric prototype of the mapping and the category-specific prototype of the mapping.

4. The robot small-sample point cloud semantic segmentation method according to any one of claims 1 to 2, characterized in that, The step of matching the category-aware geometric prototype with the point cloud features of the query scene and outputting the semantic segmentation result of the new object class includes: Calculate the matching score matrix between the category-aware geometric prototype and the point cloud features of the query scene; Perform a maximum value index extraction operation on the last dimension of the matching score matrix to obtain the semantic segmentation result of the new type of object.

5. The robot small-sample point cloud semantic segmentation method according to any one of claims 1 to 2, characterized in that, The steps for obtaining the point cloud of the query scenario include: Acquire color and depth images of the scene to be detected; Extract the pixel coordinates of each pixel in the color image, and the depth value corresponding to each pixel in the depth image; Based on the pixel coordinates and depth values ​​corresponding to each pixel, as well as the camera intrinsic data, the three-dimensional point coordinates corresponding to each pixel are calculated. The coordinates of all three-dimensional points corresponding to the scene to be detected are collected to obtain the point cloud of the query scene.

6. The robot few-sample point cloud semantic segmentation method according to claim 5, characterized in that, The acquisition of the color image and depth image of the scene to be detected includes: The color image and the depth image are acquired using a visual sensor; The visual sensor is mounted on the robot.

7. The robot small-sample point cloud semantic segmentation method according to any one of claims 1 to 2, characterized in that, The method for obtaining point cloud samples includes: Obtain the initial object point cloud corresponding to the new type of object; Determine a preset number of target point clouds in the initial object point cloud; The target point cloud is labeled one by one to obtain the supporting point cloud sample.

8. A robot few-shot point cloud semantic segmentation apparatus, used to perform the robot few-shot point cloud semantic segmentation method as described in any one of claims 1 to 7, characterized in that, include: The acquisition unit acquires the query scene point cloud of the scene to be detected where the robot is located, as well as the supporting point cloud samples and supporting point cloud sample masks of the new type of object in the scene to be detected. The global prototype generation unit generates a global geometric prototype based on the query scene point cloud features, the support point cloud sample features, and the support point cloud sample mask. The geometric transfer unit performs a mean-averaging operation on the support point cloud sample mask and the support point cloud sample features to obtain category prototypes and category clue features. It also calculates a mapping matrix between the query scene point cloud features and the category clue features to calculate a category-specific prototype based on the category prototype and the mapping matrix. The prototype fusion unit performs dynamic weighted fusion of the global geometric prototype and the category-specific prototype to obtain a category-aware geometric prototype. The semantic segmentation unit matches the category-aware geometric prototype with the point cloud features of the query scene and outputs the semantic segmentation result of the new object class.

Images