A multi-task three-dimensional point cloud segmentation method and device

Abstract

The application relates to the technical field of three-dimensional point cloud segmentation, and provides a multi-task three-dimensional point cloud segmentation method and device.The method comprises the following steps: dividing input point clouds into super points, performing feature pooling on the point clouds in the super points, and taking the pooled features as super point features; for an interactive segmentation task, visual prompt features for coding visual prompt information are obtained from the super point features according to user input visual prompt information; for a reference segmentation task, a text encoder is used to extract text prompt features of user input text prompt information; the super point features, the visual prompt features and the text prompt features are input into a mask decoder to obtain multi-task masks, and multi-task three-dimensional point cloud segmentation is realized.The application understands user input visual prompts and text prompts by utilizing edge texture information of three-dimensional point clouds, so that multi-task three-dimensional point cloud segmentation is realized, and the occupation of computing resources and storage resources in the multi-task case is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional point cloud segmentation, in particular to a multi-task three-dimensional point cloud segmentation method and device. BACKGROUND

[0002] Three-dimensional point cloud segmentation divides point cloud scenes according to the category semantics and shape texture of point clouds. The technical field of three-dimensional point cloud segmentation involves numerous sub-tasks, including but not limited to three-dimensional point cloud instance segmentation, semantic segmentation, panoramic segmentation, interactive segmentation, reference segmentation, and open vocabulary semantic segmentation tasks. Among them, three-dimensional instance segmentation, semantic segmentation, and panoramic segmentation aim to segment the target categories and target individuals labeled in the training set. Three-dimensional point cloud interactive segmentation aims to segment the specified three-dimensional point cloud individual according to the visual cues input by the user. Three-dimensional point cloud reference segmentation aims to segment the described three-dimensional point cloud individual according to the text description input by the user. Three-dimensional point cloud segmentation is a basic problem in the field of three-dimensional scene understanding, and multi-task three-dimensional point cloud segmentation technology has a wide and practical application prospect.

[0003] Existing three-dimensional point cloud segmentation methods usually divide three-dimensional point clouds into voxels for subsequent processing, but voxels divide three-dimensional point clouds into regularly arranged three-dimensional grids with a fixed voxel size preset by humans, ignoring the edge texture of instance individuals in three-dimensional point clouds. In addition, existing three-dimensional point cloud segmentation methods are usually designed for a single three-dimensional point cloud segmentation task, such as QueryFormer, which aims to achieve excellent three-dimensional point cloud instance segmentation performance, and OctFormer, which aims to achieve excellent three-dimensional point cloud semantic segmentation performance. When applied to multi-task scenarios such as smart homes, three-dimensional point cloud segmentation methods designed for a single task need to run single-task point cloud segmentation methods for multiple tasks, which poses a large demand for computing resources and storage resources. Therefore, designing a multi-task three-dimensional point cloud segmentation method and researching the use of edge texture information of three-dimensional point clouds have very important application value.

[0004] Therefore, overcoming the defects of the prior art is a problem to be solved in the technical field. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a multi-task three-dimensional point cloud segmentation method.

[0006] The present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a multi-task three-dimensional point cloud segmentation method, comprising:

[0008] extracting three-dimensional point cloud features; dividing the input point cloud into super points, and performing feature pooling on the point clouds in the super points, taking the pooled features as super point features;

[0009] For the interactive segmentation task, the visual cue feature encoding the visual cue information is sampled from the superpoint feature according to the visual cue information input by the user; for the reference segmentation task, the text cue feature of the text cue information input by the user is extracted by using a text encoder;

[0010] The superpoint feature, the visual cue feature and the text cue feature are input into a mask decoder to obtain a multi-task mask, so as to realize multi-task three-dimensional point cloud segmentation;

[0011] The knowledge distillation technology is used to constrain the training process of the instance segmentation, the semantic segmentation and the reference segmentation task by using the output information of the interactive segmentation task;

[0012] The contrast learning technology is used to constrain the mutual alignment of the visual cue feature and the text cue feature.

[0013] Preferably, the three-dimensional point cloud feature extraction specifically comprises:

[0014] Reading the three-dimensional point cloud data of the input scene Wherein N is the number of three-dimensional points contained in the three-dimensional point cloud, each three-dimensional point is described by three-dimensional coordinates x, y, z and three-channel color r, g, b; the three-dimensional point cloud backbone network is used to extract the three-dimensional point cloud feature Wherein d in is the feature depth of the three-dimensional point cloud feature.

[0015] Preferably, the input point cloud is divided into superpoints, the feature of the point cloud in the superpoint is pooled, and the pooled feature is taken as the superpoint feature, specifically comprising:

[0016] The three-dimensional point cloud is divided into M superpoints according to the texture information of the point cloud, the three-dimensional point feature of the three-dimensional point cloud contained in each superpoint is pooled to obtain the superpoint feature Wherein

[0017] Preferably, for the interactive segmentation task, the visual cue feature encoding the visual cue information is sampled from the superpoint feature according to the visual cue information input by the user, specifically comprising:

[0018] The superpoint feature of the superpoint containing the three-dimensional coordinates corresponding to the visual cue information is sampled as the visual cue feature

[0019] For the reference segmentation task, the text cue feature of the text cue information input by the user is extracted by using a text encoder, specifically comprising:

[0020] The text cue information input by the user is extracted, and the text cue information is input into tokenizer to obtain the token corresponding to the text, represented as where l denotes the length of the input text, c denotes the dimension of the token; the text token is sent into the text encoder to obtain the text feature The text feature is input into the MLP to map the feature depth of the text feature to , thereby obtaining the text feature F with the feature depth of T .

[0021] Preferably, the super point feature, the visual cue feature and the text cue feature are input into the mask decoder to obtain a multi-task mask, and multi-task three-dimensional point cloud segmentation is realized, specifically including:

[0022] The query is used to uniformly represent the super point feature, the visual cue feature and the text cue feature, specifically:

[0023] m super point features are randomly sampled from the super point feature F s to form a super point query, denoted as In the training stage, m < M is set to reduce the calculation consumption, and in the inference stage, m = M is set to perform category prediction for each super point; the visual cue feature and the text cue feature are directly used as the visual cue query and the text cue query, respectively denoted as where K p is the number of visual cues input by the user, and K t is the number of text cues input by the user;

[0024] In order to guide the model to distinguish the super point query, the visual cue query and the text cue query, the task

[0025] is designed.

[0026] The embedding of the query is denoted as e u , e p and e t ; where q u = q u + e u , q p = q p + e p , q t = q t + e t , where are the super point query embedding, the visual cue query embedding and the text cue query embedding, respectively; when the query and the embedding are added, the feature shapes of the super point query embedding, the visual cue query embedding and the text cue query embedding are broadcasted to

[0027] The visual Transformer is selected as the mask decoder, and the super point query, the visual prompt query and the text prompt query with the superimposed task embedding are input into the mask decoder for three-dimensional point cloud segmentation prediction.

[0028] Preferably, the visual Transformer is selected as the mask decoder, and the super point query, the visual prompt query and the text prompt query with the superimposed task embedding are input into the mask decoder for three-dimensional point cloud segmentation prediction, specifically comprising:

[0029] The output feature f of the mask decoder is calculated out =MaskDecoder(q=Concat(q u ,q p ,q t );k=F s ;v=F s ), wherein is the output feature of the mask decoder, d out is the feature depth; wherein the visual Transformer is composed of a cross-attention layer and a self-attention layer, the self-attention layer performs feature interaction on the input query, and the cross-attention layer does not perform feature interaction on the input query; the visual prompt query and the text prompt query are only sent into the cross-attention layer in the visual Transformer, and the super point query is sent into the cross-attention layer and the self-attention layer of the visual Transformer;

[0030] The category information is encoded as a category embedding, and the model predicts the category embedding to realize category prediction; wherein the extraction of the category embedding is obtained by sequentially sending the category name text into the text encoder and the MLP; for the category embedding, the output feature depth of the MLP is d out ; the category embedding is represented as K v is the number of category names, and the model outputs the mask pred and the category prediction result cls pred is obtained by the following formula:

[0031]

[0032] wherein the MLP projects the feature depth to d out , and respectively represent the i-th prediction mask and the i-th class prediction result; the prediction mask and the class prediction result are superpoint segmentation results, to obtain a three-dimensional point cloud segmentation result, the superpoint is inversely projected to the three-dimensional point cloud, and the segmentation result of the three-dimensional point contained in the superpoint is consistent with the superpoint.

[0033] Preferably, the output information of the interactive segmentation task is used to constrain the training process of the instance segmentation, the semantic segmentation and the reference segmentation task by using the knowledge distillation technology, and specifically includes:

[0034] The knowledge distillation technology is used to optimize the three-dimensional point cloud segmentation accuracy, and specifically:

[0035] The output result of the superpoint query is represented as The one-to-one matching algorithm is used to establish Pred u The label of the visual prompt segmentation task is matched with The output result matched is divided into positive samples, represented as

[0036] The mask prediction result of the visual prompt segmentation can be represented as The pixel coordinates of the k% pixel points with the highest confidence in the prediction mask are selected as the learning area

[0037]

[0038] The BCE loss function is applied The mask prediction result of the superpoint query is constrained by the mask prediction result of the visual prompt segmentation task:

[0039]

[0040] Wherein mask pos (R) and mask p (R) are the prediction values of the positive sample prediction mask and the visual prompt segmentation task prediction mask in the learning area R respectively.

[0041] The class prediction results of the visual prompt query and the text prompt query referring to the same instance individual are respectively represented as cls p and cls t The BCE loss function is applied The class prediction result of the reference segmentation task is constrained by the class prediction result of the visual prompt segmentation task, and specifically:

[0042]

[0043] The BCE loss function is applied and The three-dimensional point cloud segmentation accuracy is optimized.

[0044] Preferably, the contrast learning technique is used to constrain the visual prompt feature and the text prompt feature to be aligned with each other, specifically including:

[0045] The contrast learning technique is used to optimize the three-dimensional point cloud segmentation accuracy, specifically:

[0046] The visual prompt information and the text prompt information referring to the same instance individual are defined as a set of visual-text prompt pairs, and the prompt features of the B set of visual-text prompt pairs are projected into visual embeddings and text embeddings, respectively represented as and Among them, the B set of visual-text prompt pairs is the visual-text prompt pair of the current training batch;

[0047] The contrast learning loss function and The formula is: Where σ is a learnable parameter used to control the size of the contrast learning loss;

[0048] The vector dot product of the i-th visual embedding and the j-th text embedding is used as the similarity measurement result of the visual embedding and the text embedding, denoted as s i,j ; The sorting rule s i,i > s i,j is designed, and the sorting loss function The formula is:

[0049]

[0050] The contrast learning loss function and the sorting loss function are used to optimize the three-dimensional point cloud segmentation accuracy.

[0051] In a second aspect, the present application further provides a multi-task three-dimensional point cloud segmentation device for implementing the multi-task three-dimensional point cloud segmentation method of the first aspect, the device comprising:

[0052] At least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the multi-task three-dimensional point cloud segmentation method of the first aspect.

[0053] In a third aspect, the present application also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method in the first aspect.

[0054] In a fourth aspect, a chip is provided, which comprises a processor and an interface, and is used to call and run a computer program stored in a memory to execute the method in the first aspect.

[0055] In a fifth aspect, a computer program product comprising instructions which, when executed on a computer or processor, cause the computer or processor to carry out the method of the first aspect.

[0056] The present application understands the visual cues and text cues of the user input by utilizing the edge texture information of the three-dimensional point cloud, thereby realizing multi-task three-dimensional point cloud segmentation, such as one or more of instance segmentation, semantic segmentation, panoramic segmentation, cross-type segmentation, reference segmentation and open vocabulary semantic segmentation, reducing the occupation of computing resources and storage resources in the multi-task case, and expanding the application prospect of the three-dimensional point cloud segmentation algorithm in the multi-task scene such as smart home. The present application can realize various point cloud segmentation tasks based on a single model, and establish the relevance between tasks through knowledge distillation and contrast learning technology to improve the segmentation accuracy of the model on multiple tasks. The present application can also utilize the edge texture information of the three-dimensional point cloud, since the three-dimensional point cloud has disorder and the edge distribution of the instance individual is irregular, and the utilization of the edge texture information of the three-dimensional point cloud is helpful for the three-dimensional point cloud segmentation task; and the multi-task point cloud segmentation can be realized based on a single model, effectively expanding the application value of the three-dimensional point cloud segmentation method in the multi-task scene. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0058] Figure 1 is a flowchart of a first multi-task three-dimensional point cloud segmentation method provided by the embodiments of the present application;

[0059] Figure 2 is a flowchart of a second multi-task three-dimensional point cloud segmentation method provided by the embodiments of the present application;

[0060] Figure 3 is a flowchart of a third multi-task three-dimensional point cloud segmentation method provided by the embodiments of the present application;

[0061] Figure 4 is a flowchart of a fourth multi-task three-dimensional point cloud segmentation method provided by an embodiment of the present application;

[0062] Figure 5 is a schematic diagram of a multi-task three-dimensional point cloud segmentation method provided by an embodiment of the present application;

[0063] Figure 6 is a schematic diagram of another multi-task three-dimensional point cloud segmentation method provided by an embodiment of the present application;

[0064] Figure 7 is a schematic diagram of a multi-task three-dimensional point cloud segmentation device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0066] Unless otherwise required by the context, the term "comprising" is to be interpreted as open and inclusive, i.e. "including but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" are intended to mean that a particular feature, structure, material or characteristic included in at least one embodiment or example of the present disclosure. The illustrative representation of the above terms does not necessarily mean the same embodiment or example. In addition, the specific features, structures, materials or characteristics described can be included in any one or more embodiments or examples in any appropriate manner, i.e. although they are carried in the embodiments or examples of the above terms due to the order of appearance and location, they are not limited to the combination of one embodiment or example.

[0067] In the description of the present application, the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included in one or more features. In the description of the embodiments of the present disclosure, unless otherwise stated, the meaning of "multiple" is two or more. In addition, for example, in the description, the same type of nouns can also be described as two independent individuals by adding "A", "B" at the end, in which case the features limited by "A", "B" are only used for the purpose of distinguishing the same type of individual description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0068] In the description of the present application, the expression "A and / or B" (wherein A and B represent specific features in the form) includes the following three combinations: only A, only B, and the combination of A and B.

[0069] The "about", "approximately" or "approximately" used in the present application includes the stated value and the average value within the acceptable deviation range of the specific value, wherein the acceptable deviation range is determined by the person skilled in the art considering the measurement being discussed and the error related to the measurement of the specific quantity (i.e. the limitation of the measurement system).

[0070] In order to make the technical solutions of the present application more clear, some terms used in the present application are explained here:

[0071] tokenizer is a class for vectorizing text, converting text into a sequence. When processing language, computers cannot understand the meaning of the text, and usually convert a word (Chinese single word or word) into a positive integer, and a text becomes a sequence, and then the sequence is vectorized, and the vectorized data is sent to the model for processing.

[0072] Token is the result converted by tokenizer, that is, sequence.

[0073] Transformer is a kind of deep neural network based on self-attention mechanism.

[0074] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as there is no conflict between them.

[0075] Embodiment 1:

[0076] Embodiment 1 of the present application provides a multi-task three-dimensional point cloud segmentation method, as shown in Figure 1 The method comprises the following steps:

[0077] In step 201, the three-dimensional point cloud features are extracted; the input point cloud is divided into super points, and the point cloud in the super points is feature pooled, and the pooled features are used as super point features.

[0078] In step 202, for the interactive segmentation task, the visual prompt features of the coding visual prompt information are sampled from the super point features according to the visual prompt information input by the user; for the reference segmentation task, the text prompt features of the text prompt information input by the user are extracted by using the text encoder.

[0079] In step 203, the superpoint feature, the visual cue feature and the text cue feature are input into a mask decoder to obtain a multi-task mask, and multi-task three-dimensional point cloud segmentation is realized.

[0080] In step 204, the knowledge distillation technology is used to constrain the training process of the instance segmentation, semantic segmentation and reference segmentation tasks by the output information of the interactive segmentation task.

[0081] In step 205, the contrast learning technology is used to constrain the mutual alignment of the visual cue feature and the text cue feature.

[0082] In an embodiment of the present application, the three-dimensional point cloud feature extraction specifically comprises:

[0083] Reading three-dimensional point cloud data of an input scene Wherein N is the number of three-dimensional points contained in the three-dimensional point cloud, each three-dimensional point is described by three-dimensional coordinates x, y, z and three-channel color r, g, b; and a three-dimensional point cloud backbone network is used to extract three-dimensional point cloud features Wherein d in is the feature depth of the three-dimensional point cloud feature.

[0084] The input point cloud is divided into superpoints, the point cloud in the superpoint is subjected to feature pooling, and the pooled feature is taken as the superpoint feature, specifically comprising:

[0085] According to the texture information of the point cloud, the three-dimensional point cloud is divided into M superpoints, the three-dimensional point features of the three-dimensional point cloud contained in each superpoint are subjected to feature pooling, and the superpoint feature is obtained Wherein

[0086] M is obtained by experience analysis of those skilled in the art.

[0087] In an optional embodiment, for the interactive segmentation task, the visual cue feature encoding the visual cue information is sampled from the superpoint feature according to the visual cue information input by the user, specifically comprising:

[0088] The superpoint feature of the superpoint containing the three-dimensional coordinates corresponding to the visual cue information is sampled as the visual cue feature That is, the visual cue information is represented by three-dimensional coordinates.

[0089] For the reference segmentation task, a text encoder is used to extract the text cue feature of the text cue information input by the user, specifically comprising:

[0090] The text cue information input by the user is extracted, and the text cue information is input into a tokenizer to obtain tokens corresponding to the text, represented as wherein I represents the length of the input text, and c represents the dimension of the token; the text token (i.e., the token corresponding to the above text) is sent into the text encoder to obtain text features The Multilayer Perceptron (MLP) is applied to map the feature depth of the text features to . Thus, the text features F T .

[0091] In a specific application scenario, the super point features, the visual cue features and the text cue features are input into the mask decoder to obtain a multi-task mask, so as to realize multi-task three-dimensional point cloud segmentation, as shown in Figure 2 , specifically comprising:

[0092] In step 301, the query is used to uniformly represent the super point features, the visual cue features and the text cue features, specifically:

[0093] m super point features are randomly sampled from the super point features F s to form a super point query, denoted as In the training stage, m < M is set to reduce the calculation consumption, and in the inference stage, m = M is set to make a class prediction for each super point; the visual cue features and the text cue features are directly used as visual cue queries and text cue queries, respectively, denoted as wherein K p is the number of visual cues input by the user, and K t is the number of text cues input by the user.

[0094] In step 302, the embedding is designed to guide the model to distinguish the super point query, the visual cue query and the text cue query, denoted as e u , e p , e t ; wherein,

[0095] ″′

[0096] q u = q u + e u , q p = q p + e p , q t = q t + e t , wherein superpoint query embedding, visual cue query embedding and text cue query embedding, respectively; when the query and the embedding are added, the feature shapes of the superpoint query embedding, the visual cue query embedding and the text cue query embedding are broadcasted as

[0097] In step 303, a visual Transformer is selected as a mask decoder, and the superpoint query, the visual cue query and the text cue query superimposed with the task embedding are input into the mask decoder for three-dimensional point cloud segmentation prediction.

[0098] The superpoint query, the visual cue query and the text cue query superimposed with the task embedding are input into the mask decoder for three-dimensional point cloud segmentation prediction, as shown in Figure 3 The specific steps include:

[0099] In step 401, the output feature f out of the mask decoder is calculated u . p t ); k = F s ; v = F s , where is the output feature of the mask decoder, d out is the feature depth; the superpoint query is used for three-dimensional instance segmentation, three-dimensional semantic segmentation, three-dimensional panoramic segmentation and three-dimensional open vocabulary semantic segmentation prediction, and such three-dimensional point cloud segmentation tasks cannot perceive the cue information from the user in the inference process, therefore, the superpoint query needs to avoid feature interaction with the visual cue query and the text cue query in the training process, so as to ensure the consistency of the training and the inference process. Therefore, the visual Transformer is composed of a cross-attention layer and a self-attention layer, the self-attention layer performs feature interaction on the input query, and the cross-attention layer does not perform feature interaction on the input query; the visual cue query and the text cue query are only sent into the cross-attention layer in the visual Transformer, and the superpoint query is sent into the cross-attention layer and the self-attention layer of the visual Transformer.

[0100] ​In step 402, the category information is encoded into a category embedding, and the model predicts the category embedding to realize category prediction; wherein the category embedding is extracted by sequentially feeding the category name text into a text encoder and an MLP; that is, the extraction step of the category embedding is the same as the extraction step of the text prompt feature, the difference is that for the category embedding, the output feature depth of the MLP is d out ; the category embedding is represented as K v is the number of category names, and the model outputs a mask mask pred and a category prediction result cls pred , which are obtained by the following formula:

[0101]

[0102] wherein the MLP projects the feature depth to d out , and represent the i-th predicted mask and the i-th category prediction result, respectively; both the predicted mask and the category prediction result are the segmentation results of the superpoint, in order to obtain the three-dimensional point cloud segmentation result, the superpoint is inversely projected to the three-dimensional point cloud, and the segmentation result of the three-dimensional point contained in the superpoint is consistent with the superpoint.

[0103] In actual application scenarios, the training process of the instance segmentation, semantic segmentation, and reference segmentation tasks is constrained by the output information of the interactive segmentation task using the knowledge distillation technology, specifically including:

[0104] The knowledge distillation technology is used to optimize the three-dimensional point cloud segmentation accuracy, specifically:

[0105] The output result of the superpoint query is represented as A one-to-one matching algorithm is used to establish a one-to-one matching between Pred u and the label of the visual prompt segmentation task , and the output results that match are divided into positive samples, represented as

[0106] The mask prediction result of the visual prompt segmentation can be represented as The pixel coordinates of the k% of the pixel points with the highest confidence in the predicted mask are selected as the learning area k is obtained by demand analysis by those skilled in the art.

[0107] The binary cross entropy loss (BCE loss) function The mask prediction result of the visual prompt segmentation task is used to constrain the mask prediction result of the super point query:

[0108]

[0109] Wherein, mask pos (R) and mask p (R) are respectively the prediction values of the positive sample prediction mask and the visual prompt segmentation task prediction mask in the learning area R.

[0110] The class prediction results of the visual prompt query and the text prompt query referring to the same instance individual are respectively denoted as cls p and cls t , and the BCE loss function The class prediction result of the visual prompt segmentation task is used to constrain the class prediction result of the reference segmentation task, specifically:

[0111]

[0112] The BCE loss function and are used to optimize the three-dimensional point cloud segmentation accuracy.

[0113] In some embodiments, the contrast learning technology is used to constrain the alignment of the visual prompt feature and the text prompt feature, specifically including:

[0114] The contrast learning technology is used to optimize the three-dimensional point cloud segmentation accuracy, specifically:

[0115] The visual prompt information and the text prompt information referring to the same instance individual are defined as a group of visual-text prompt pairs, and the prompt features of the B group of visual-text prompt pairs are projected into visual embedding and text embedding, respectively denoted as and Wherein, the B group of visual-text prompt pairs are the visual-text prompt pairs in the current training batch; that is, it is assumed that there are B group of visual-text prompt pairs in the current training batch.

[0116] The contrast learning loss function and is expressed by formula as: Wherein, σ is a learnable parameter used to control the size of the contrast learning loss.

[0117] The vector dot product of the i-th visual embedding and the j-th text embedding is used as the similarity measurement result of the visual embedding and the text embedding, denoted as s i,j; design ordering rules s i,i i,j , get the ordering loss function Expressed in formula as:

[0118]

[0119] Apply the contrast learning loss function And the ordering loss function Optimize the three-dimensional point cloud segmentation accuracy.

[0120] The embodiment understands the visual prompt and the text prompt of the user input by using the edge texture information of the three-dimensional point cloud, thereby realizing multi-task three-dimensional point cloud segmentation, such as one or more of instance segmentation, semantic segmentation, panoramic segmentation, cross-type segmentation, reference segmentation and open vocabulary semantic segmentation, reducing the occupation of algorithm resources and storage resources in the multi-task case, and expanding the application prospect of the three-dimensional point cloud segmentation algorithm in the multi-task scene such as smart home. The present application can realize various point cloud segmentation tasks based on a single model, and establish the correlation between tasks by knowledge distillation and contrast learning technology to improve the segmentation accuracy of the model in multiple tasks. The embodiment can also utilize the edge texture information of the three-dimensional point cloud. Since the three-dimensional point cloud has disorder, the edge distribution of the instance individual is irregular, and the edge texture information of the three-dimensional point cloud is helpful for the three-dimensional point cloud segmentation task; and multi-task point cloud segmentation can be realized based on a single model, effectively expanding the application value of the three-dimensional point cloud segmentation method in the multi-task scene.

[0121] Embodiment 2:

[0122] The present application is based on the method described in embodiment 1, combined with specific application scenarios, and described by related technical expressions in the scene to illustrate the implementation process in the characteristic scene of the present application.

[0123] The method of the embodiment divides the three-dimensional point cloud into super points by color texture in the three-dimensional point cloud scene, and realizes three-dimensional point cloud segmentation by processing super point features for mask prediction and class prediction. The embodiment can understand the visual prompt information and the text prompt information input by the user, encode the user information into the input query of the mask decoder through super point feature sampling and text information encoding, and realize multi-task three-dimensional point cloud segmentation by parallel processing of the mask decoder from multiple tasks. Figure 4 As shown in the formula (1), the method comprises the following steps:

[0124] In step 501, the three-dimensional point cloud features are extracted, specifically including:

[0125] ​In the embodiment, the ScanNet20 dataset is used for model training. Considering that the ScanNet20 dataset does not contain user text description information, the user text prompt information in the embodiment uses the text description information provided by the ScanRefer dataset. In the embodiment, the input three-dimensional point cloud is subjected to random flipping, random rotation and other data augmentation.

[0126] Reading three-dimensional point cloud data of an input scene The three-dimensional point cloud contains 215316 three-dimensional points, each of which is described by three-dimensional coordinates x, y, z and three-channel color r, g, b. A three-dimensional point cloud backbone network is used to extract three-dimensional point cloud features The feature depth of the three-dimensional point cloud features is 32.

[0127] In step 502, the input point cloud is divided into super points, and the features of the point cloud in the super points are pooled to obtain the features of the super points (i.e., super point features). Specifically, the following steps are included:

[0128] The three-dimensional point cloud is divided into 1632 super points according to the texture information of the point cloud, and the three-dimensional point features of the three-dimensional point cloud contained in each super point are pooled to obtain super point features

[0129] In step 503, for an interactive segmentation task, visual cue features encoding the visual cue information are sampled from the super point features according to the visual cue information input by the user. Specifically, the following steps are included:

[0130] The visual cue information is represented by three-dimensional coordinates. In step 502, the three-dimensional point cloud data is divided into super points, and the super point features of the super points containing the three-dimensional coordinates corresponding to the visual cue information are sampled as the visual cue features

[0131] In step 504, for a reference segmentation task, a text encoder is used to extract text cue features of the text cue information input by the user. Specifically, the following steps are included:

[0132] The text cue information is input into a tokenizer to obtain tokens corresponding to the text, represented as where l represents the length of the input text, and c represents the dimension of the token. The text token is input into a text encoder to obtain a text feature An MLP is applied to map the feature depth of the text feature to Thus, the text feature F with a feature depth of is obtained. T .

[0133] ​In step 505, the super point features, visual cue features, and text cue features are input into a mask decoder to obtain a multi-task mask, and multi-task three-dimensional point cloud segmentation is implemented, specifically including:

[0134] In this embodiment, 805 super point features are randomly sampled from the super point features F s to form a super point query, denoted as In the inference stage, all 1632 super point features are used to perform class prediction for each super point. The visual cue features and the text cue features obtained in steps 503 and 504 are directly used as visual cue queries and text cue queries, respectively, denoted as wherein the number of user-input visual cues is 21, and the number of user-input text cues is 19. To guide the model to distinguish between the super point query, the visual cue query, and the text cue query, this embodiment designs a task embedding, denoted as e u , e p , and e t :

[0135] q u = q ′ u + e u , q p = q ′ p + e p , q t = q ′ t + e t ,

[0136] wherein q , q , and q are super point query embedding, visual cue query embedding, and text cue query embedding, respectively. When the query is added to the embedding, the feature shapes of the super point query embedding, the visual cue query embedding, and the text cue query embedding are broadcasted to

[0137]

[0138] The visual Transformer is selected as the mask decoder, and the super point query, the visual cue query, and the text cue query with the above-mentioned stacked task embedding are input into the mask decoder for three-dimensional point cloud segmentation prediction, and the calculation formula is: f out = MaskDecoder(q = Concat(q u , q p , qt ) ; k = F s ; v = F s ),

[0139] wherein is the output feature of the mask decoder, d out is the feature depth. The super point query is used for three-dimensional instance segmentation, semantic segmentation, panoramic segmentation, open vocabulary semantic segmentation, such three-dimensional point cloud segmentation tasks cannot perceive the prompt information from the user in the inference process, therefore, the super point query needs to avoid feature interaction with the visual prompt query and the text prompt query in the training process, so as to ensure the consistency of the training and inference processes. In the embodiment, the visual Transformer is composed of a cross-attention layer and a self-attention layer, the self-attention layer performs feature interaction on the input query, and the cross-attention layer does not perform feature interaction on the input query. In the embodiment, the visual prompt query and the text prompt query are only sent into the cross-attention layer in the visual Transformer, and the super point query is sent into the cross-attention layer and the self-attention layer of the visual Transformer.

[0140] In the embodiment, the category information is encoded as category embedding, and the model predicts the category embedding to realize category prediction. The extraction steps of the category embedding are the same as the extraction steps of the text prompt feature, that is, the category name text is sent into the text encoder and the MLP in turn. The difference lies in that, for the category embedding, the output feature depth of the MLP is 256. In the embodiment, the category embedding is represented as The number of category names is 20, and the model outputs the mask mask pred and the category prediction result cls pred which is obtained by the following formula:

[0141]

[0142] wherein the MLP projects the feature depth to 256, and represent the i-th predicted mask and the i-th category prediction result respectively. The predicted mask and the category prediction result are both the segmentation results of the super point. In order to obtain the three-dimensional point cloud segmentation result, the super point needs to be projected back to the three-dimensional point cloud, and the segmentation result of the three-dimensional point contained in the super point is consistent with the super point.

[0143] In step 506, the knowledge distillation technology is used to constrain the training process of the instance segmentation, semantic segmentation, and reference segmentation tasks with the output information of the interactive segmentation task, specifically including:

[0144] The output result of the super point query is represented as A one-to-one matching algorithm is used to establish Pred u The label of the visual prompt segmentation task One-to-one matching is used to divide the output result of the matching into positive samples, represented as The mask prediction result of the visual prompt segmentation can be represented as The pixel coordinates of the 10% of the pixel points with the highest confidence in the predicted mask are selected as the learning area The BCE loss function is applied The mask prediction result of the super point query is constrained by the mask prediction result of the visual prompt segmentation task:

[0145]

[0146] Where mask pos (R) and mask p (R) are the predicted values of the positive sample prediction mask and the visual prompt segmentation task prediction mask in the learning area R, respectively.

[0147] The class prediction results of the visual prompt query and the text prompt query referring to the same instance individual are represented as cls p and cls t The BCE loss function is applied The class prediction result of the reference segmentation task is constrained by the class prediction result of the visual prompt segmentation task:

[0148]

[0149] In this embodiment, BCE and are used to optimize the three-dimensional point cloud segmentation accuracy.

[0150] In step 507, the contrast learning technique is used to constrain the alignment of the visual prompt features and the text prompt features, specifically including:

[0151] The visual prompt information and the text prompt information referring to the same instance individual are defined as a group of visual-text prompt pairs. It is assumed that there are 19 groups of visual-text prompt pairs in the current training batch. In this embodiment, the prompt features of the 19 groups of visual-text prompt pairs are projected into visual embedding and text embedding, represented as and The contrast learning loss function and is expressed by the formula:

[0152]

[0153] wherein s is a learnable parameter used to control the size of the contrastive learning loss. In this embodiment, the vector dot product between the i-th visual embedding and the j-th text embedding is taken as the similarity measure between the visual embedding and the text embedding, denoted as s i,j In this embodiment, the ranking rule s i,i i,j is obtained, and the ranking loss function is which is expressed by the formula as follows:

[0154]

[0155] In this embodiment, the contrastive learning loss function and the ranking loss function are used to optimize the accuracy of three-dimensional point cloud segmentation, and the specific implementation process is shown in Figure 5 , and the achieved effect is shown in Figure 6 This embodiment realizes six three-dimensional point cloud segmentation tasks based on a single model.

[0156] Embodiment 3:

[0157] Based on the text description information provided by the ScanRefer in the foregoing embodiment 2, the text is used as the input text of the reference segmentation task, and the model is trained on the ScanNet20 training set. This embodiment verifies the performance of a multi-task three-dimensional point cloud segmentation method on instance segmentation, semantic segmentation, panoramic segmentation, and interactive three-dimensional point cloud segmentation tasks on the ScanNet20 dataset. This embodiment verifies the performance of a multi-task three-dimensional point cloud segmentation method on the reference segmentation task on the ScanRefer dataset. This embodiment verifies the performance of a multi-task three-dimensional point cloud segmentation method on the open vocabulary semantic segmentation task on the ScanNet200 dataset.

[0158] In this embodiment, the mask prediction result of the open vocabulary semantic segmentation task is directly output by a multi-task three-dimensional point cloud segmentation method in this embodiment, and the class prediction result of the open vocabulary semantic segmentation task is referred to the existing Open3DIS open vocabulary semantic segmentation method. Specifically, first, the three-dimensional point cloud is projected into an RGB image, then the projected region on the image is cropped and sent into the image encoder of CLIP to obtain the image embedding, and at the same time, the list of class name texts is sent into the text encoder of CLIP to obtain the text embedding, then the similarity between the image embedding and the text embedding is measured, and the class name corresponding to the text embedding with the highest similarity is the predicted class of the three-dimensional point cloud. ​

[0159] ScanNet20 provides RGB-D images and three-dimensional point cloud data of 1613 scenes, including 18 instance classes and 2 semantic classes. ScanNet200 uses the same scene data as ScanNet20, but contains up to 198 instance classes and 2 semantic classes, and is therefore more challenging. ScanRefer contains 51583 natural language descriptions that point to 11046 three-dimensional point cloud instance targets selected from 800 scenes.

[0160] In terms of evaluation indicators, the embodiment uses PQ, mIoU and mAP to evaluate the performance of panoramic segmentation, semantic segmentation and instance segmentation. In addition, the embodiment uses AP and mIoU to evaluate the performance of interactive segmentation and reference segmentation tasks, respectively. For the open-vocabulary semantic segmentation task, the embodiment trains the model on ScanNet20 and uses the AP indicator to evaluate the performance on ScanNet200 without fine-tuning the model weights on the ScanNet200 dataset.

[0161] As shown in the following table, the embodiment can implement multiple three-dimensional point cloud segmentation tasks based on a single model, and the embodiment can achieve performance superior to other comparative methods on multiple tasks. The symbol "-" in the table represents that the method cannot be directly applied to the task, so the performance cannot be verified.

[0162]

[0163] As can be seen, the multiple indicators of the method described in the embodiment have a significant improvement over the prior art.

[0164] Embodiment 4:

[0165] As Figure 7 shown, is a schematic diagram of the architecture of the multi-task three-dimensional point cloud segmentation device of the embodiment of the application. The multi-task three-dimensional point cloud segmentation device of the embodiment includes one or more processors 21 and a memory 22. Among them, Figure 7 take a processor 21 as an example.

[0166] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 7 take the example of connection through the bus.

[0167] The memory 22 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs and non-volatile computer executable programs, such as the multi-task three-dimensional point cloud segmentation method in embodiment 1. The processor 21 executes the non-volatile software programs and instructions stored in the memory 22, thereby executing the multi-task three-dimensional point cloud segmentation method.

[0168] The memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 can optionally include a memory disposed remotely with respect to the processor 21, which can be connected to the processor 21 through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, perform the multi-task three-dimensional point cloud segmentation method in the above-mentioned embodiment 1, embodiment 2 or embodiment 3.

[0170] The processor 21 can be a multi-core processor to improve processing speed and efficiency; the memory 22 can be a cache or a dynamic random access memory (DRAM) to ensure fast access and processing of data; and the data bus can be a high-bandwidth data bus to support high data transmission rate. The instructions include obtaining and preprocessing three-dimensional point cloud data, training and reasoning of a multi-task learning model based on the preprocessed data, and post-processing of the training results to generate the final segmentation results.

[0171] It is worth noting that the information interaction, execution process, etc. between the modules and units in the above-mentioned apparatus and system, since based on the same concept as the processing method embodiments of the present application, the specific content can be referred to the description in the method embodiments of the present application, which will not be repeated here.

[0172] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, which can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0173] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-task 3D point cloud segmentation method, characterized in that, include: Extract 3D point cloud features; divide the input point cloud into superpoints, perform feature pooling on the point cloud within the superpoints, and use the pooled features as the superpoint features; For interactive segmentation tasks, visual cue features encoding visual cue information are obtained by sampling from superpoint features based on the visual cue information input by the user; for reference segmentation tasks, text cue features of text cue information input by the user are extracted using a text encoder. By inputting superpoint features, visual cue features, and text cue features into the mask decoder, a multi-task mask is obtained, enabling multi-task 3D point cloud segmentation. The knowledge distillation technique is used to constrain the training process of instance segmentation, semantic segmentation, and reference segmentation tasks with the output information of interactive segmentation tasks; A contrastive learning technique is used to constrain the alignment of visual cue features with text cue features.

2. The multi-task 3D point cloud segmentation method according to claim 1, characterized in that, The extraction of 3D point cloud features specifically includes: Read the 3D point cloud data of the input scene ,in This refers to the number of 3D points contained in a 3D point cloud, where each 3D point is represented by 3D coordinates. , , and three-channel color , , Description; 3D point cloud features are extracted using a 3D point cloud backbone network. ,in The feature depth of the 3D point cloud features.

3. The multi-task 3D point cloud segmentation method according to claim 2, characterized in that, The process of dividing the input point cloud into superpoints, performing feature pooling on the point cloud within each superpoint, and using the pooled features as the superpoint features specifically includes: Based on the texture information of the point cloud, the 3D point cloud is divided into... For each superpoint, feature pooling is performed on the 3D point features of the 3D point cloud contained within the superpoint to obtain the superpoint features. ,in .

4. The multi-task 3D point cloud segmentation method according to claim 2, characterized in that, For interactive segmentation tasks, based on the visual cue information input by the user, visual cue features encoding the visual cue information are sampled from the super-point features, specifically including: Sampling superpoint features that include the 3D coordinates of superpoints corresponding to visual cue information is used as visual cue features. ; For the reference segmentation task, a text encoder is used to extract text prompt features from the user-input text prompt information, specifically including: Extract the text prompt information input by the user, send the text prompt information to the tokenizer to obtain the token corresponding to the text, and represent it as... ,in Indicates the length of the input text. This represents the dimension of the token; the text token is fed into a text encoder to obtain text features. ; Applying MLP to map the feature depth of text features to Thus, the feature depth is obtained. Text features .

5. The multi-task 3D point cloud segmentation method according to claim 3, characterized in that, The process of inputting superpoint features, visual cue features, and text cue features into a mask decoder to obtain a multi-task mask and achieve multi-task 3D point cloud segmentation specifically includes: We employ a unified query representation for superpoint features, visual cue features, and text cue features, specifically: From superpoint features Random sampling A superpoint query is composed of several superpoint features, denoted as: ; Set during the training phase This reduces computational cost and allows for setting parameters during the inference phase. Therefore, category prediction is performed for each superpoint; the visual cue features and text cue features are directly used as the visual cue query and text cue query, respectively, as follows: ,in The number of visual cues input by the user. The number of text suggestions for user input; To guide the model in distinguishing between super-point queries, visual cue queries, and text cue queries, a task embedding is designed, represented as: , , ;in, in , , These are the super-point query embedding, the visual cue query embedding, and the text cue query embedding, respectively. When the query and embedding are added together, the feature shapes of the super-point query embedding, the visual cue query embedding, and the text cue query embedding are broadcast as follows: , , ; A visual Transformer is selected as the mask decoder. The superpoint query, visual cue query, and text cue query of the overlay task embedding are input into the mask decoder for 3D point cloud segmentation prediction.

6. The multi-task 3D point cloud segmentation method according to claim 5, characterized in that, The step of selecting a visual Transformer as the mask decoder involves inputting the superpoint query, visual cue query, and text cue query from the overlay task embedding into the mask decoder for 3D point cloud segmentation prediction. Specifically, this includes: Calculate the output features of the mask decoder in For the output features of the mask decoder, The feature depth is defined as follows: The visual Transformer consists of a cross-attention layer and a self-attention layer. The self-attention layer performs feature interaction on the input query, while the cross-attention layer does not perform feature interaction on the input query. Visual cue queries and text cue queries are only fed into the cross-attention layer of the visual Transformer, while superpoint queries are fed into both the cross-attention layer and the self-attention layer of the visual Transformer. Category information is encoded into category embeddings, and the model predicts the category embeddings to achieve category prediction. The category embeddings are extracted by sequentially feeding the category name text into a text encoder and then into an MLP. For the category embedding, the MLP's output feature depth is... ; Represent the category embedding as , The model outputs a mask based on the number of category names. Compared with category prediction results It can be obtained through the following formula: MLP projects the feature depth as , , ; and Representing the first The first prediction mask and the second Each category prediction result; the prediction mask and the category prediction result are both superpoint segmentation results. In order to obtain the 3D point cloud segmentation result, the superpoint is back-projected onto the 3D point cloud, and the segmentation result of the 3D points contained in the superpoint is consistent with the superpoint.

7. The multi-task 3D point cloud segmentation method according to claim 6, characterized in that, The training process for instance segmentation, semantic segmentation, and reference segmentation tasks, which employs knowledge distillation techniques to constrain the output information of interactive segmentation tasks, specifically includes: Knowledge distillation technology is used to optimize the segmentation accuracy of 3D point clouds. Specifically: The output of the superpoint query is represented as A one-to-one matching algorithm is used to establish Labels for segmentation tasks with visual cues One-to-one matching divides the matched output into positive samples, denoted as... ; The mask prediction results for visual cue segmentation can be represented as Select the one with the highest confidence from the predicted masks. The pixel coordinates of % of the pixels are used as the learning region. ; Applying the BCE loss function The mask prediction results of the visual cue segmentation task are used to constrain the mask prediction results of the super-point query: in and These are the positive sample prediction mask and the visual cue segmentation task prediction mask in the learning region, respectively. The predicted value; The category prediction results for visual cue queries and text cue queries referring to the same instance are represented as follows: and Applying the BCE loss function The category prediction results of the visual cue segmentation task are used as constraints to the category prediction results of the reference segmentation task. Specifically: application and Optimize the accuracy of 3D point cloud segmentation.

8. The multi-task 3D point cloud segmentation method according to claim 1, characterized in that, The method of employing contrastive learning techniques to constrain the alignment of visual and textual cue features specifically includes: Contrastive learning techniques are used to optimize the accuracy of 3D point cloud segmentation, specifically: A visual cue and a text cue representing the same instance are defined as a visual-text cue pair. The cue feature projections of a pair of visual-text cues are visual embeddings and text embeddings, denoted as follows: and ; wherein, the The group of visual-text cue pairs is the visual-text cue pair of the current training batch; Contrast learning loss function , and Expressed as a formula: ;in These are learnable parameters used to control the magnitude of the contrastive learning loss; The first The visual embedding and the first The vector dot product of the text embeddings is used as the similarity measure between the visual embedding and the text embedding, denoted as: Design sorting rules The sorting loss function is obtained. This can be expressed as a formula: ; Applying the contrastive learning loss function With sorting loss function Optimize the accuracy of 3D point cloud segmentation.

9. A multi-task 3D point cloud segmentation device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the multi-task 3D point cloud segmentation method according to any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions that are executed by one or more processors to perform the method described in any one of claims 1-8.

Images