An interactive 3D point cloud semantic segmentation refinement method based on diffusion model

Abstract

The present invention discloses an interactive three-dimensional point cloud semantic segmentation refinement method based on a diffusion model. The method fully mines high-dimensional point-by-point semantic features through a point cloud feature extraction network, designs an interactive click feature embedding method, obtains an interactive feature map, connects the point-by-point semantic features and the interactive feature map in series through a point cloud segmentation refinement network, outputs a predicted refined segmentation mask, and realizes the gradual refinement of the segmentation mask by constructing forward and backward diffusion processes under the correction of the interactive semantic segmentation refinement network. The method effectively solves the problem of insufficient local segmentation accuracy of the current semantic segmentation algorithm and broadens high-precision downstream application scenarios such as reconstruction and data annotation.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional computer vision, and in particular to an interactive three-dimensional point cloud semantic segmentation refinement method based on a diffusion model, which is used to improve the quality of three-dimensional point cloud semantic segmentation. Background Art

[0002] 3D point cloud semantic segmentation is an important task in the field of 3D computer vision. It aims to assign labels containing semantic information to each data point in the point cloud. It is the basis for downstream tasks such as autonomous driving, robot navigation, and 3D scene reconstruction. The quality of the 3D point cloud semantic segmentation mask is crucial to downstream tasks.

[0003] There are two main types of existing point cloud segmentation methods. One is the traditional point cloud segmentation based on heuristic algorithms such as graph cut or random walk algorithm. It can achieve segmentation of connected areas with similar colors by constructing a local relationship graph of the point cloud and using the minimum cut algorithm. It can also use the random walk algorithm to segment the process factory pipeline point cloud. The other is a three-dimensional point cloud semantic segmentation method based on deep learning, including a point cloud segmentation network based on positional relationship convolution and deep residual connection and a weakly supervised point cloud segmentation algorithm. The former achieves point cloud segmentation by embedding the point cloud into a high-dimensional space and performing convolution, deep residual, fully connected layer and decoding in sequence. The latter achieves point cloud segmentation based on sparse semantic annotation information.

[0004] The existing point cloud segmentation methods have the following problems:

[0005] 1. Point cloud segmentation based on heuristic algorithms can mostly only obtain good segmentation results on specific point cloud objects, lack of versatility, and rely on underlying features, making it difficult to perform semantic segmentation and unable to meet downstream task requirements.

[0006] 2. The 3D point cloud semantic segmentation method based on deep learning ignores the distribution difference between test data and training data, which easily leads to segmentation errors and cannot meet the requirements of high-precision segmentation masks. Summary of the invention

[0007] The purpose of the present invention is to provide an interactive 3D point cloud semantic segmentation refinement method based on a diffusion model, so as to provide an interactive 3D point cloud semantic segmentation refinement method based on user interaction correction, so as to obtain a refined high-precision 3D point cloud segmentation mask.

[0008] The technical solution to achieve the purpose of the present invention is:

[0009] An interactive 3D point cloud semantic segmentation refinement method based on a diffusion model, comprising:

[0010] Design a point cloud feature extraction network to extract point-by-point semantic features;

[0011] Design interactive click feature embedding method and obtain interactive feature map ;

[0012] Design a point cloud segmentation refinement network that outputs predicted refined segmentation masks and confidence probabilities based on coarse segmentation masks, point-by-point semantic features, and interaction feature maps.

[0013] Design an interactive semantic segmentation refinement network to simulate user interaction;

[0014] Design diffusion model and construct state transfer matrix;

[0015] Construct training data, design loss functions, train and optimize point cloud segmentation refinement network, interactive semantic segmentation refinement network and diffusion model parameters;

[0016] A back-diffusion process is performed to progressively refine the coarse segmentation mask.

[0017] Furthermore, the point cloud feature extraction network is designed to extract point-by-point semantic features, including the following specific steps:

[0018] Extract high-dimensional semantic features of point clouds through the Point Transformer network;

[0019] Based on the rough segmentation mask, locate the semantic segmentation boundary area;

[0020] The attention mechanism is used to enhance the high-dimensional semantic features and the features of the semantic segmentation boundary area to obtain point-by-point semantic features.

[0021] Furthermore, we design an interactive click feature embedding method to obtain the interactive feature map The specific steps include:

[0022] Place all interaction points in the spatial rectangular coordinate system and obtain the The coordinates of the interaction points , and the interaction feature map Middle Line Set the column elements to 0;

[0023] First The interaction point is the center, and the length of the edge is Cube, determine the The coordinates of the interaction points Is it in the cube? ;

[0024] Jordi The interaction points are in the cube, and the interaction feature map Middle Line Column elements and Line The column elements are set to 1, otherwise they are set to 0;

[0025] Traverse all interaction points in sequence and build an interaction feature map .

[0026] Furthermore, a point cloud segmentation refinement network is designed to output the predicted refined segmentation mask and confidence probability based on the coarse segmentation mask, point-by-point semantic features and interactive feature maps, including the following specific steps:

[0027] According to the coordinates of the interaction points, the rough segmentation boundary area is located;

[0028] Binarize the roughly segmented boundary area to transform the multi-class segmentation problem into a two-class segmentation problem;

[0029] The coarse segmentation boundary region, coarse segmentation mask, point-by-point semantic features and interaction feature maps are combined Perform feature concatenation;

[0030] The two-category segmentation result is obtained through a point-by-point binary classifier, and then mapped back to the multi-category segmentation space through back projection to obtain the predicted refined segmentation mask and confidence probability.

[0031] Furthermore, an interactive semantic segmentation refinement network is designed to simulate user interaction, including the following specific steps:

[0032] Obtaining an erroneous segmentation mask map and multiple erroneous segmentation regions based on the difference between the predicted refined segmentation mask and the actual refined segmentation mask;

[0033] Cluster each mis-segmented region separately and locate the region with the largest area Mis-segmented regions;

[0034] Perform local density estimation on the point cloud of a single mis-segmented area, obtain the point set to be interacted with, assign actual semantic labels, and simulate user interaction.

[0035] Furthermore, designing a diffusion model and constructing a state transfer matrix includes the following specific steps:

[0036] It is defined that the interaction point state has a refined state and a degenerate state, which correspond to the refined segmentation mask and the coarse segmentation mask respectively;

[0037] Design the forward diffusion process of the diffusion model to gradually degenerate the refined segmentation mask into a coarse segmentation mask to obtain the first The initial state of the interaction point ;

[0038] Based on the state transition probability Calculate the forward state transfer matrix , the time step Status Multiply the forward state transfer matrix Realize the single-step degradation of the forward diffusion process;

[0039] The state can be obtained by recursion The state probability distribution ;

[0040] Design the reverse diffusion process of the diffusion model to gradually refine the coarse segmentation mask into a refined segmentation mask, and obtain the predicted refined segmentation mask output by the point cloud segmentation refinement network and confidence probability ;

[0041] Based on the forward diffusion process and Bayesian theory, the posterior probability distribution of the reverse diffusion process is obtained The specific formula is as follows

[0042] ,

[0043] in, In the forward diffusion process The initial state of each interaction point;

[0044] Based on The confidence level of the interaction point Calculate the reverse state transfer matrix , the single-step refinement of the back diffusion process can be obtained by the time step Status Multiply by the inverse state transfer matrix accomplish.

[0045] Furthermore, constructing training data, designing loss functions, training and optimizing point cloud segmentation refinement network, interactive semantic segmentation refinement network and diffusion model parameters include the following specific steps:

[0046] Construct training data and use a benchmark segmentation network for training;

[0047] Record parameters at fixed intervals, obtain several point cloud segmentation refinement network models, and output predicted refined segmentation masks with different refinement levels;

[0048] The interactive semantic segmentation refinement network and diffusion model are further trained based on the predicted refined segmentation masks with different refinement levels, and the parameters are updated;

[0049] A combination of cross entropy and focal loss is chosen as the loss function.

[0050] Furthermore, the back diffusion process is performed to gradually refine the rough segmentation mask of the point cloud, including the following specific steps:

[0051] Set the state of the interaction point to a degenerate state;

[0052] Processing time steps through point cloud segmentation refinement network The rough segmentation mask of Get the predicted refined segmentation mask and confidence probability ;

[0053] Calculate the The reverse state transfer matrix of the interaction points and get the time step No. The state of the interaction point ;

[0054] Calculate the current time step The refined segmentation mask ;

[0055] The time step The refined segmentation mask Again as the time step The rough segmentation mask of is input into the point cloud segmentation refinement network, and the time step is obtained by single-step refinement of the back diffusion process. The refined segmentation mask ;

[0056] Repeat the above process until the final refined segmentation mask is obtained .

[0057] Compared with the prior art, the present invention has the following significant advantages:

[0058] 1. Design a point cloud feature network to fully explore the high-dimensional point-by-point semantic features of point cloud data, design an interactive click feature embedding method, consider user interactive clicks, and integrate the interactive feature map into the point-by-point semantic features;

[0059] 2. Design a point cloud segmentation refinement network, an interactive semantic segmentation refinement network, and a diffusion model joint segmentation refinement scheme. Design a loss function that takes into account both point cloud segmentation and user interaction. Use user interaction for real-time correction and obtain a more detailed point cloud segmentation mask through the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flow chart of an interactive 3D point cloud semantic segmentation refinement method based on a diffusion model;

[0061] Figure 2 Flowchart for constructing interactive feature maps;

[0062] Figure 3 Refine the network flow chart for designing point cloud segmentation;

[0063] Figure 4 Refining network flowchart for interactive semantic segmentation. DETAILED DESCRIPTION

[0064] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0065] The embodiment provided by the present invention is an interactive 3D point cloud semantic segmentation refinement method based on a diffusion model. Based on the diffusion model, under the guidance of user interaction, the segmentation mask is refined by constructing forward and backward diffusion processes, which effectively solves the problem of insufficient local segmentation accuracy of the current semantic segmentation algorithm and broadens high-precision downstream application scenarios such as reconstruction and data annotation, including the following steps:

[0066] Design a point cloud feature extraction network to extract point-by-point semantic features;

[0067] Design interactive click feature embedding method and obtain interactive feature map ;

[0068] Design a point cloud segmentation refinement network based on coarse segmentation masks, point-by-point semantic features, and interactive feature maps , output predicted refined segmentation mask and confidence probability;

[0069] Design an interactive semantic segmentation refinement network to simulate user interaction;

[0070] Design diffusion model and construct state transfer matrix;

[0071] Construct training data, design loss functions, train and optimize point cloud segmentation refinement network, interactive semantic segmentation refinement network and diffusion model parameters;

[0072] A back-diffusion process is performed to progressively refine the coarse segmentation mask.

[0073] Furthermore, the point cloud feature extraction network is designed to extract point-by-point semantic features, including the following specific steps:

[0074] Extract high-dimensional semantic features of point clouds through the Point Transformer network;

[0075] Based on the rough segmentation mask, locate the semantic segmentation boundary area;

[0076] The attention mechanism is used to reconstruct and represent high-dimensional semantic features, strengthen the features of semantic segmentation boundary areas, and obtain point-by-point semantic features.

[0077] Furthermore, we design an interactive click feature embedding method to obtain the interactive feature map The specific steps include:

[0078] Place all interaction points in the spatial rectangular coordinate system and obtain the The coordinates of the interaction points , and set ;

[0079] First The interaction point is the center, and the length of the edge is Cube;

[0080] Get the The coordinates of the interaction points , , respectively calculate the The interaction point and Interaction points Axis distance, Axis distance and The axis distance is , and ;

[0081] judge Axis distance , Axis distance and Axis distance Are all less than or equal to half the edge length? ;

[0082] If satisfied, then determine The interaction point is located at The interaction point is the center, In a cube with edge length of If not satisfied, set ;

[0083] Traverse all interaction points in sequence and build an interaction feature map .

[0084] Furthermore, a point cloud segmentation refinement network is designed to output the predicted refined segmentation mask and confidence probability based on the coarse segmentation mask, point-by-point semantic features and interactive feature maps, including the following specific steps:

[0085] According to the coordinates of the interaction points, the rough segmentation boundary area is located;

[0086] Binarize the roughly segmented boundary area, set the semantic categories of the areas on both sides of the roughly segmented boundary to 0 and 1 respectively, and transform the complex multi-category segmentation problem into a simple two-category segmentation problem;

[0087] The coarse segmentation boundary region, coarse segmentation mask, point-by-point semantic features and interaction feature maps are combined Perform feature concatenation;

[0088] The binary segmentation result is obtained through a point-by-point binary classifier, which is mapped back to the multi-class segmentation space through a back-projection operation to obtain the predicted refined segmentation mask and confidence probability.

[0089] Furthermore, an interactive semantic segmentation refinement network is designed to simulate user interaction, including the following specific steps:

[0090] Calculate the difference between the predicted refined segmentation mask and the actual refined segmentation mask to obtain the mis-segmentation mask map and multiple mis-segmentation regions;

[0091] Cluster each mis-segmented region based on the DBSCAN density clustering algorithm to obtain the clustering result of a single mis-segmented region;

[0092] Based on the clustering results of a single mis-segmented region, the largest area is located Mis-segmented regions;

[0093] Perform local density estimation on the point cloud of each mis-segmented area;

[0094] Based on the local density estimation results, the point set to be interacted is obtained, and actual semantic labels are assigned to simulate user interaction.

[0095] Furthermore, designing a diffusion model and constructing a state transfer matrix includes the following specific steps:

[0096] The interaction points are defined to have two states: a refined state and a degenerate state, where the refined state and the degenerate state correspond to a refined segmentation mask and a coarse segmentation mask, respectively.

[0097] The forward diffusion process of the diffusion model is designed. The forward diffusion process is to gradually degenerate the refined segmentation mask into a coarse segmentation mask. The initial state of the interaction point Defined as a refined state, at time step Status is defined as a degenerate state, represented by a Bernoulli variable, , ;

[0098] The single-step degradation in the forward diffusion process can be expressed as follows

[0099] ,

[0100] in, is the state transition probability, represents the forward state transfer matrix.

[0101] Then based on the initial state Get time step Status The state probability distribution The specific formula is as follows

[0102] ;

[0103] The reverse diffusion process of the diffusion model is designed. The reverse diffusion process is to gradually refine the rough segmentation mask into a refined segmentation mask. The processing process of the point cloud segmentation refinement network is defined as , then the output of the point cloud segmentation refinement network is as follows

[0104] ,

[0105] in, represents the predicted refined segmentation mask, represents the confidence probability, Represents point cloud data, represents the rough segmentation mask;

[0106] Based on the forward diffusion process, combined with Bayesian theory, the posterior probability distribution of the reverse diffusion process is obtained The specific formula is as follows

[0107] ,

[0108] in, In the forward diffusion process The initial state of each interaction point;

[0109] The single-step refinement of the reverse diffusion process can be expressed as

[0110] ,

[0111] in, For the The reverse state transfer matrix of the interaction points, For the The confidence level of each interaction point.

[0112] Furthermore, constructing training data, designing loss functions, training and optimizing point cloud segmentation refinement network, interactive semantic segmentation refinement network and diffusion model parameters include the following specific steps:

[0113] Construct training data, use a baseline segmentation network, and train parameters from scratch;

[0114] The parameters are recorded every fixed round to obtain several point cloud segmentation refinement network models, and then the predicted refined segmentation masks with different refinement levels are output for further training of the interactive semantic segmentation refinement network and diffusion model.

[0115] The interactive semantic segmentation refinement network obtains the point set to be interacted based on the predicted refined segmentation mask and assigns the actual semantic label;

[0116] The loss function is designed, using point-by-point cross entropy loss as the dominant loss function, and the focus loss defined at the interaction point as the auxiliary loss function to strengthen the response to user interaction.

[0117] Furthermore, the back diffusion process is performed to gradually refine the rough segmentation mask of the point cloud, including the following specific steps:

[0118] Set the states of all interaction points to degenerate states;

[0119] Through the point cloud segmentation refinement network according to the time step The rough segmentation mask of Get the predicted refined segmentation mask and confidence probability ;

[0120] Calculate the The reverse state transfer matrix of the interaction points and get the time step No. The state of the interaction point ;

[0121] Based on time step No. The state of the interaction point , predict refined segmentation mask and the reverse state transfer matrix , calculate the time step The refined segmentation mask ;

[0122] The time step The refined segmentation mask As a rough segmentation mask, it is input into the cloud segmentation refinement network again and the time step is obtained by a single-step refinement through the back diffusion process. The refined segmentation mask ;

[0123] Repeat the above steps until the final refined segmentation mask is obtained .

[0124] The present invention fully mines high-dimensional point-by-point semantic features through a point cloud feature extraction network, designs an interactive click feature embedding method, obtains an interactive feature map, connects point-by-point semantic features and interactive feature maps in series through a point cloud segmentation refinement network, outputs a predicted refined segmentation mask, and under the correction of the interactive semantic segmentation refinement network, realizes the gradual refinement of the segmentation mask by constructing forward and backward diffusion processes, effectively solving the problem of insufficient local segmentation accuracy of the current semantic segmentation algorithm and broadening high-precision downstream application scenarios such as reconstruction and data labeling.

[0125] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An interactive 3D point cloud semantic segmentation refinement method based on diffusion model, characterized in that: The following steps are involved: Design a point cloud feature extraction network, using the Point Transformer network combined with the attention mechanism to obtain point-by-point semantic features; Design an interactive click feature embedding method to obtain an interactive feature map based on the spatial relationship between the remaining interactive points and a cube centered on a single interactive point; Design a point cloud segmentation refinement network, locate and binarize the roughly segmented boundary area, obtain the two-category segmentation result through feature concatenation and point-by-point binary classifier, and use back projection to obtain the predicted refined segmentation mask and confidence probability; Design an interactive semantic segmentation refinement network, divide the mis-segmented areas and perform clustering, obtain the point set to be interacted through local density estimation and assign actual semantic labels; Design a diffusion model, construct a forward state transfer matrix based on the state transition probability to achieve single-step degradation of the forward diffusion process, obtain the confidence of a single interaction point through the point cloud segmentation refinement network and calculate the reverse state transfer matrix to achieve single-step refinement of the reverse diffusion process; Construct training data, use a benchmark segmentation network for training, use cross entropy loss combined with focus loss as the loss function, and optimize parameters through multiple rounds of training; The back diffusion process is performed, the state of the interaction point is set to a degenerate state, and the single-step refinement of the back diffusion process is repeated until the final refined segmentation mask is obtained.

2. The interactive 3D point cloud semantic segmentation refinement method based on diffusion model according to claim 1, characterized in that: The design of the point cloud segmentation refinement network to obtain the predicted refined segmentation mask and confidence probability includes the following specific steps: According to the coordinates of the interaction points, the rough segmentation boundary area is located; Performing binarization processing on the roughly segmented boundary area; Concatenating the roughly segmented boundary area, the roughly segmented mask, the point-by-point semantic features, and the interactive feature map; A two-category segmentation result is obtained by a point-by-point two-class classifier, and is mapped back to a multi-category segmentation space through back-projection to obtain the predicted refined segmentation mask and the confidence probability.

3. The interactive 3D point cloud semantic segmentation refinement method based on diffusion model according to claim 1, characterized in that: The design of the interactive semantic segmentation refinement network includes the following specific steps: Acquire an erroneous segmentation mask map and an erroneous segmentation region based on a difference between the predicted refined segmentation mask and the actual refined segmentation mask; Clustering the mis-segmented regions, and locating several mis-segmented regions with the largest areas; Perform local density estimation on the point cloud of the mis-segmented area, obtain the point set to be interacted, and assign actual semantic labels.

4. The interactive 3D point cloud semantic segmentation refinement method based on diffusion model according to claim 1, characterized in that: The design of the diffusion model and the construction of the forward state transfer matrix and the reverse state transfer matrix include the following specific steps: Get the The initial state of the interaction point, the single-step degradation of the forward diffusion process, and the time step The state transition probability Calculate the forward state transfer matrix , the time step Status Multiply the forward state transfer matrix Degenerate to time step Status ; The initial state Perform forward diffusion Single-step degradation, get the time step Status About the initial state The state probability distribution , for The product of the forward state transfer matrices of time steps; Process the point cloud data through the point cloud segmentation refinement network to obtain the time step The predicted refined segmentation mask and confidence probability ; Obtaining the posterior probability distribution of the reverse diffusion process based on Bayesian theory , the specific formula is as follows: ， in, for The product of the forward state transfer matrices of time steps; Based on The confidence level of the interaction point Calculate the reverse state transfer matrix , the time step Status Multiplying by the reverse state transfer matrix realizes the single-step refinement of the reverse diffusion process to the time step Status .

5. The interactive 3D point cloud semantic segmentation refinement method based on diffusion model according to claim 1, characterized in that: The performing of the back diffusion process to gradually refine the rough segmentation mask comprises the following steps: Set the state of the interaction point to a degenerate state; The time step The refined segmentation mask of is used as the coarse segmentation mask to input the point cloud segmentation refinement network to obtain the time step The predicted refined segmentation mask and confidence probability of ; Obtain the reverse state transfer matrix and perform a single-step refinement of the reverse diffusion process to obtain the time step Refined segmentation mask of ; The single-step refinement of the point cloud segmentation refinement network prediction and the back diffusion process is repeated until the final refined segmentation mask is obtained.

6. The interactive 3D point cloud semantic segmentation refinement method based on diffusion model according to claim 1, characterized in that: The design of the interactive click feature embedding method and obtaining the interactive feature map include the following steps: Place all interaction points in a spatial rectangular coordinate system, obtain the coordinates of a single interaction point, and set the diagonal elements in the interaction feature map to 0; Build a cube centered around a single interaction point; Based on whether the remaining single interaction points are in the cube, the corresponding row and column elements in the interaction feature graph are set to 1 or 0; Traverse all interaction points in sequence and build an interaction feature graph.

7. The interactive 3D point cloud semantic segmentation refinement method based on diffusion model according to claim 1, characterized in that: The design of the point cloud feature extraction network to extract point-by-point semantic features includes the following steps: Extract high-dimensional semantic features of point clouds through the Point Transformer network; Based on the rough segmentation mask, locate the semantic segmentation boundary area; The attention mechanism is used to enhance the high-dimensional semantic features and the features of the semantic segmentation boundary area to obtain point-by-point semantic features.

8. The interactive 3D point cloud semantic segmentation refinement method based on diffusion model according to any one of claims 1 to 7, characterized in that: The steps of constructing training data, designing loss functions, training and optimizing parameters include: Construct training data and use a benchmark segmentation network for training; Record parameters at fixed intervals, obtain several point cloud segmentation refinement network models, and output predicted refined segmentation masks with different refinement levels; The interactive semantic segmentation refinement network and diffusion model are further trained based on the predicted refined segmentation masks with different refinement levels, and the parameters are updated; A combination of cross entropy and focal loss is chosen as the loss function.

Images