Self-supervision point cloud object segmentation method based on double attention mechanism and contrast regularization

Abstract

The invention discloses a self-supervised point cloud object segmentation method based on a double attention mechanism and contrast regularization, and the method specifically comprises the steps: firstly, generating multi-view point cloud data through a geometric transformation enhancement strategy, and capturing the geometric invariance features in a mode through a contrast learning mechanism; then, using cross space attention and cross channel attention to realize deep fusion of image texture and geometric features of the point cloud, and introducing a learnable weight distribution strategy to dynamically perform weight distribution on the image and the point cloud; and finally, constructing a global response matrix by integrating global features through global channel mode and point mode descriptors, enhancing the interactive expression ability of local details and global semantics, and completing segmentation. The problem that feature extraction of an existing self-supervision method is insufficient is solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional point cloud semantic segmentation, and particularly relates to a self-supervised point cloud object segmentation method based on a double attention mechanism and contrast regularization. BACKGROUND

[0002] Three-dimensional point cloud object component segmentation is one of the core tasks in the field of computer vision and three-dimensional data processing, and has a wide range of applications in medical image analysis, industrial detection, robot grabbing and the like. However, due to the irregularity of point cloud data, the difficulty of manual labeling of point cloud data sets and the certain limitations of single modal data, the segmentation technology still faces multiple challenges.

[0003] At present, many works use self-supervised methods to solve the above problems, most of which are through the comparison method between the converted point clouds, or through the setting of occluded point clouds to learn features from themselves, but still lack certain effective feature support, resulting in the limitation of segmentation accuracy. And although the single point cloud data can provide geometric structure, it lacks texture information; and although the single image is rich in color details, the depth information is fuzzy. Therefore, Wu et al. proposed the CrossPoint method to combine the cross-modal implicit features of objects extracted from images and point cloud information, but there are deficiencies in the alignment of cross-modal features and the processing of cross-modal noise. SUMMARY

[0004] The purpose of the present application is to provide a self-supervised point cloud object segmentation method based on a double attention mechanism and contrast regularization, which solves the problem of insufficient feature extraction of existing self-supervised methods.

[0005] The technical solution adopted by the present application is a self-supervised point cloud object segmentation method based on a double attention mechanism and contrast regularization, specifically, first, through a geometric transformation enhancement strategy, multiple-view point cloud data is generated, and a contrast learning mechanism is used to capture the geometric invariance features within the modal; then, cross-space attention and cross-channel attention are used to realize the deep fusion of image texture and point cloud geometric features, and a learnable weight allocation strategy is introduced to dynamically allocate weights for images and point clouds; finally, a global channel mode and a point mode descriptor are used to comprehensively describe the global features to construct a global response matrix, enhance the interactive expression ability of local details and global semantics, and complete the segmentation.

[0006] The present application is characterized in that,

[0007] The present application is characterized in that,

[0008] Step 1, dynamic point cloud enhanced contrast learning: the input point cloud adopts a random geometric transformation strategy, and the original point cloud data is geometrically transformed to dynamically generate two point cloud representation data with different perspectives. Then, a contrast learning mechanism is constructed to compare the internal features of point clouds with different perspectives. Finally, the geometric features Z of the different perspective point clouds within the modal are extracted t ;

[0009] Step 2, dual attention multi-modal fusion: first, the geometric features f I of the image are extracted. Then, the mapping relationship between the image texture and the point cloud geometry in the spatial position is fused by combining the cross-space attention and the cross-channel attention. The semantic interaction between the cross-modal channels is deeply mined. The dynamic fusion mechanism is combined to obtain the point cloud and image fused features F fused ;

[0010] Step 3, global bilinear regularization feature extraction: the global response matrix G is constructed by comprehensively describing the global features through the global channel method and the point method descriptor. The global features F G are obtained by MLP. The nonlinearity is enhanced by using the activation function σ to obtain the point cloud features F out ;

[0011] Step 4, segmentation prediction: the final point cloud features F out are obtained by two multi-layer perceptrons and a Dropout layer, and the final segmentation result is obtained by Softmax.

[0012] Step 1 is specifically:

[0013] Step 1.1, the initial point cloud P = {p1, p2, …, p j , …, p N} is converted to by T , where N represents the total number of samples in the feature space, P j represents the jth point, as shown in equation (1):

[0014]

[0015] In equation (1), α and β represent two views of the point cloud;

[0016] Step 1.2, after constructing two views, the global point cloud features are extracted using feature encoding. First, a dynamic graph convolution of the local point is established, and a dynamic graph is constructed in the feature space using the k-nearest neighbor algorithm. The neighborhood of each point is dynamically updated according to the current feature similarity. Then, for each point and its neighborhood point pair (p i , p j ), the edge feature is calculated, as shown in equation (2):

[0017] eij =h(p i ,p j -p i ) (2)

[0018] In formula (2), e ij Represents edge features, h represents a multi-layer perceptron, which is used to fuse the center point p i and relative position information p j -p i ;

[0019] The maximum pooling symmetric function is used to aggregate neighborhood features, retaining local features and global information, as shown in formula (3):

[0020] d α ,f α =F P (p α ),d β ,f β =F P (p β ) (3)

[0021] In formula (3), d and f are the location and features of the downsampling point respectively, F P It is a point cloud feature extractor;

[0022] Step 1.3, project the extracted feature vectors. and The projection vectors are expressed as and The specific formula is as follows:

[0023]

[0024] In formula (4), g P For projection head, used to maximize and Similarity between them, minimize p j Similarity of projection vectors to other points;

[0025] In step 1.4, the Hungarian algorithm is used to construct positive sample pairs to achieve optimal bidirectional matching between cross-view points. Specifically, the downsampled point sets of the two views are regarded as two vertex sets of a bipartite graph, and the cost matrix is ​​constructed by calculating the distance between the point pairs, as shown in the following formula (5):

[0026] M=(cost=||T(d α )-(d β )||2) (5)

[0027] In formula (5), T is the geometric transformation in formula (1), and then the Euclidean distance between the points in view a and the points in view β is calculated by L2 norm to find the matching relationship of each point in the two views;

[0028] Step 1.5, after establishing the corresponding relationship, the loss of the adversarial contrast loss is calculated by using the following formula (6), (7),

[0029]

[0030] In formula (6)-(7), s(.) is the cosine similarity, τ is the temperature factor, and N represents the batchsize of the input point number, represents the loss of the a view to the β view, represents the loss of the β view to the a view L in represents the overall loss between the two views, that is, the loss within the mode.

[0031] Step 2 is specifically:

[0032] Step 2.1, first, the image features are extracted: using the ResNet network to embed the 2D image rendered by the point cloud data into the feature space, specifically: introducing the residual block, realizing the identity mapping through the jump connection, and gradually extracting different levels of features through the convolution layer, the shallow network extracts low-level features, and the deep network gradually extracts semantic high-level features through stacking residual blocks, as follows formula (8):

[0033] f I =F I (I) (8)

[0034] In formula (8), f I is the extracted image feature, F I is the image feature extractor, and I is the input image;

[0035] Step 2.2, on the basis of the obtained point cloud feature Z t and the image feature f I , the cross-space attention mechanism is used to fuse the mapping relationship of the image texture and the point cloud geometry in the spatial position, and to accurately align the spatial semantics of the two;

[0036] Step 2.3, then the cross-channel attention mechanism is used to deeply mine the semantic interaction between the cross-modal channels, and to realize the fusion at the feature channel level;

[0037] Step 2.4, using the adaptive dynamic fusion mechanism, dynamically adjusting the fusion ratio of the point cloud and the image features and the contribution weight of different modes, so as to maximize the complementarity of the point cloud features and the image features.

[0038] Step 2.2 is specifically as follows:

[0039] Step 2.2.1, for a given point cloud feature Z t , two-dimensional image feature is f I , by linear transformation to the same dimension Q, K, V space, align the semantic space of point cloud and image modal, formula as shown in equation (9):

[0040]

[0041] Step 2.2.2, matrix multiplication of Q p and K I , calculate the similarity matrix Scores p-I between point cloud features and image features

[0042] Scores p-I = Q p ·K I T

[0043] As equation (10), apply Softmax to normalize the similarity matrix, generate attention weight matrix Get the contribution of image features to each point cloud feature;

[0044] Similarly, matrix multiplication of Q I and K p get similarity matrix Scores I-p ,

[0045] Scores I-p = Q I ·K p T

[0046] And as equation (10) normalization get point cloud to image contribution

[0047] In equation (10), d k is a scaling factor;

[0048] Step 2.2.3, use attention weight to weighted sum of image point cloud feature, generate fusion cross-modal feature, image semantic feature embedding point cloud feature get fusion point cloud feature Point cloud semantic feature embedding image feature get fusion image feature As equation (11):

[0049]

[0050] Step 2.3 as follows:

[0051] Step 2.3.1, first fuse the cross-space attention of the point cloud features and image features Align the dimensions by a linear layer, and then perform global average pooling to obtain one-dimensional point cloud features and one-dimensional image features The formula is shown in equation (12) below, where Linear represents a fully connected layer:

[0052]

[0053] Step 2.3.2, two one-dimensional vectors obtained from step 2.3.1 are passed through an MLP multi-layer perceptron learning channel weight, then through a sigmoid activation function to generate channel weight, realize channel scaling, and finally the attention weight of each channel is multiplied by the respective input point cloud feature and image feature, to obtain the feature vector of the point cloud and image fused by channel attention mechanism The specific formula is shown in equation (13) below, where represents matrix multiplication,

[0054]

[0055] Step 2.4 is as follows:

[0056] Step 2.4.1, fuse the point cloud features and image features obtained in step 2.3.2, first, concatenate the point cloud features and image features in the channel dimension to serve as the basis for calculating the gating weight, then adjust the feature dimension through a fully connected layer, and use a Sigmoid activation function to generate the gating weight, the formula is shown in equation (14) below:

[0057]

[0058] In equation (14), Concat is a concatenation function used to concatenate and two features;

[0059] Step 2.4.2, element-wise fuse the point cloud and image features according to the gating weight to calculate the fused features F fused as shown in equation (15) below:

[0060]

[0061] In equations (14)-(15), represents matrix multiplication, and γ is a dynamically adjusted weight, γ∈[0,1], when γ is 1, the corresponding feature is dominated by the geometric structure of the point cloud, and when γ is 0, it is dominated by the texture feature of the image.

[0062] Step 3 is as follows:

[0063] Step 3.1, generating global descriptors in channel direction: on the dual-attention fused feature F fused Channel direction compression is performed, applying a weight matrix where r is the reduction factor to reduce the fused output F fused ∈R N×C dimension;

[0064] Step 3.2, using a ReLU function to provide nonlinear changes after the linear mapping of W c ; finally, the channel responses of all points are aggregated by an average pooling operation to compress the spatial information into global channel descriptors g c , the specific mathematical formula is as follows formula (16):

[0065] g c = avg N (ReLU(F fusec ·W c )) (16)

[0066] In formula (16), g c ∈R C / r , F fused ·W c is the matrix product between F fused and W c , avg N is the average pooling on N points, and the reduction factor is an integer satisfying r≥2, and in addition, represents the global average response of the jth channel in the feature map of the entire point cloud;

[0067] Step 3.3, generating global descriptors in point direction: using a weight matrix W p ∈R C×≤ / r , a ReLU function and an average pooling operation, the formula is as follows formula (17):

[0068]

[0069] In formula (17), g p ∈R N , g p = [ε1, …, ε i , …, ε N ], ε i represents the global response of the i-th in the feature map of the entire point cloud, and avg C / r represents the aggregation of the corresponding channel dimension by average pooling, capturing the global dependence between points;

[0070] Step 3.4, the channel descriptor gc and the point direction descriptor g obtained in step 3.3 p The outer product of and the square root of the outer product of generate a low-rank bilinear response matrix G ∈ R N×C / r where γ ij represents the global response value of the i th point and the j th channel, and the formula is as follows formula (18) :

[0071]

[0072] In formula (18), represents the outer product, and the point-channel cross matrix G is generated, and the square root is taken to balance the point and channel level contributions and avoid gradient explosion;

[0073] Step 3.5, using a shared MLP to map G back to the original feature space, and through a residual connection with the original features to retain the original features, to obtain the feature F G , as follows formula (19) ;

[0074] F G = MLP (G + F fused W c + F fused W p ) (19) ;

[0075] Step 3.6, in order to enhance the learning function, the dynamically fused feature F fused is subtracted from F G to suppress redundant common feature information, and finally, an activation function σ is used to enhance the nonlinearity to obtain the final point cloud feature F out , and the formula is as follows formula (20) :

[0076] F out = σ (F fused - F G ) (20).

[0077] The beneficial effects of the present application are:

[0078] The self-supervised point cloud object segmentation method based on double attention mechanism and contrast regularization of the present application is based on the CrossPoint method, adopts a self-supervised segmentation method of deep learning, uses a cross-modal contrast learning method, combines point clouds and 2D images, further explores the internal correlation information of points of the point cloud, and uses a double attention mechanism to further fuse cross-modal features, deeply analyzes the geometric shape features and topological correlation characteristics of the point cloud object, realizes part segmentation of the three-dimensional point cloud object, solves the problems of insufficient feature extraction of the existing self-supervised method and insufficient alignment operation processing of the cross-modal features, improves the complementarity of the point cloud features and the image features, and enhances the interactive expression ability of local details and global semantics. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 is the network overall framework diagram of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the application;

[0080] Figure 2 is the module structure diagram of cross-space attention;

[0081] Figure 3 is the module structure diagram of cross-channel attention;

[0082] Figure 4 is the module structure diagram of bilinear regularization;

[0083] Figure 5 is the visualization result of the chair category of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the application on the ShapeNet dataset, wherein, Figure 5(a) is the true value effect diagram, Figure 5(b) is the visualization result diagram of the DGCNN method, and Figure 5(c) is the visualization result diagram of the application;

[0084] Figure 6 is the visualization result of the car category of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the application on the ShapeNet dataset, wherein, Figure 6(a) is the true value effect diagram, Figure 6(b) is the visualization result diagram of the DGCNN method, and Figure 6(c) is the visualization result diagram of the application. DETAILED DESCRIPTION

[0085] The application will be described in detail below with reference to the accompanying drawings and specific embodiments. The described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0086] In the following description of the application, the terms "first", "second", "third" are only to distinguish similar objects, and do not represent the specific order of the objects. It can be understood that "first", "second", "third" can be interchanged in specific order or sequence as allowed, so that the application embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0087] Unless otherwise defined, all technical and scientific terms used in the application have the same meaning as commonly understood by those skilled in the art to which the application belongs. The terms used in the application are only for the purpose of describing the embodiments of the application and are not intended to limit the application.

[0088] Embodiment 1

[0089] The embodiment provides a self-supervised point cloud object segmentation method based on a double attention mechanism and contrast regularization, specifically, first, through a geometric transformation enhancement strategy, multi-view point cloud data is generated, and a contrast learning mechanism is used to capture intra-modal geometric invariance features; then, in order to solve the problem of insufficient cross-modal alignment of the crosspoint method, cross-space attention and cross-channel attention are used to realize deep fusion of image texture and point cloud geometric features, and a learnable weight allocation strategy is introduced to dynamically allocate weights for images and point clouds, thereby improving the complementarity of point cloud features and image features; finally, global features are integrated to construct a global response matrix through a global channel mode and a point mode descriptor, the interactive expression ability of local details and global semantics is enhanced, and segmentation is completed.

[0090] The embodiment only represents a preferred embodiment of the self-supervised point cloud object segmentation method based on the double attention mechanism and the contrast regularization of the application, and any point cloud object segmentation method designed by adopting similar technical features of the application will fall within the protection scope of the self-supervised point cloud object segmentation method based on the double attention mechanism and the contrast regularization of the application.

[0091] Embodiment 2

[0092] The embodiment provides a self-supervised point cloud object segmentation method based on a double attention mechanism and contrast regularization, and on the basis of embodiment 1, as shown in Figure 1 , the following steps are implemented:

[0093] Step 1, dynamic point cloud enhancement contrast learning: the input point cloud is subjected to random geometric transformation strategy, and original point cloud data is subjected to rotation, translation, scaling and other diversified geometric transformation operations, thereby dynamically generating two point cloud representation data of different views, then a contrast learning mechanism is constructed, the internal features of point clouds of different views are compared with each other, and finally the geometric features Z t of different view point clouds in the mode are extracted.

[0094] Step 2, double attention multi-modal fusion: first, the geometric features f I of the image are extracted, second, the cross-space attention and the cross-channel attention are combined to fuse the mapping relationship between the image texture and the point cloud geometry in the spatial position, the semantic interaction between the cross-modal channels is deeply mined, the dynamic fusion mechanism is combined to obtain the point cloud and image fused features F fused .

[0095] Step 3, global bilinear regularization feature extraction: global features are integrated to construct a global response matrix G through a global channel mode and a point mode descriptor, the global features F G are obtained through an MLP, an activation function sigma is used to enhance nonlinearity, and the point cloud features F out are obtained.

[0096] Step 4, split prediction: split the final point cloud feature F out Through two multi-layer perceptrons and a Dropout layer, and through Softmax to get the final segmentation result.

[0097] This embodiment only represents the preferred embodiment of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the present application, and any point cloud object segmentation method designed by adopting similar technical features to the present application will fall within the protection scope of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the present application.

[0098] Embodiment 3

[0099] This embodiment provides a self-supervised point cloud object segmentation method based on a dual attention mechanism and contrast regularization, and on the basis of embodiments 1-2, step 1 is specifically:

[0100] Step 1.1, after the initial point cloud P = {p1, p2, …, p j ,…,p N} is obtained, the point cloud is split into two views, and the two views are represented as P and P , where N represents the total number of samples in the feature space, P j represents the jth point, as shown in equation (1):

[0101]

[0102] In equation (1), α and β represent two views of the point cloud;

[0103] Step 1.2, after the two views are constructed, the global point cloud feature is extracted using feature encoding, first, the dynamic graph convolution of the local point is established, the dynamic graph is constructed in the feature space using the k-nearest neighbor algorithm, and the neighborhood of each point is dynamically updated according to the current feature similarity, then, for each point and its neighborhood point pair (p i ,p j ), the edge feature is calculated, as shown in equation (2):

[0104] e ij = h(p i ,p j -p i ) (2)

[0105] In equation (2), e ij represents the edge feature, and h represents a multi-layer perceptron used to fuse the center point p i and the relative position information p j -p i ;

[0106] The neighborhood features are aggregated by using a max-pooling symmetric function to preserve local features and global information, as shown in equation (3):

[0107] d α ,f α =F P (p α ),d β ,f β =F P (p β ) (3)

[0108] In equation (3), d and f are the positions and features of the down-sampled points, respectively, and F P is a point cloud feature extractor;

[0109] Step 1.3, project the extracted feature vectors, and the projection vectors of and are represented as and respectively. The specific formula is shown in equation (4):

[0110]

[0111] In equation (4), g P is a projection head used to maximize the similarity between and and minimize the similarity of p j to the projection vectors of other points;

[0112] Step 1.4, the down-sampled points of different views may cause asymmetric matching problems due to differences in spatial distribution. The Hungarian algorithm is used to construct positive sample pairs to achieve optimal bidirectional matching between cross-view points. Specifically, the down-sampled point sets of the two views are regarded as two vertex sets of a bipartite graph, and a cost matrix is constructed by calculating the distance between point pairs, as shown in equation (5):

[0113] M=(cost=||T(d α )-(d β )||2) (5)

[0114] In equation (5), T is the geometric transformation (such as rotation and translation) in equation (1), and then the Euclidean distance between the points in view α and the points in view β is calculated by L2 norm, and the matching relationship of each point in the two views is found;

[0115] Step 1.5, after establishing the corresponding relationship, the loss is calculated by using the following equations (6) and (7) for adversarial contrastive loss,

[0116]

[0117] In formula (6)-(7), s(.) is a cosine similarity, τ is a temperature factor, and N represents a batchsize of input points, represents a loss of the alpha view to the beta view, represents a loss of the beta view to the alpha view L in represents a loss of the overall between the two views, i.e., the loss within the modal.

[0118] This embodiment only represents a preferred embodiment of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the present application, and any point cloud object segmentation method designed by using similar technical features of the present application will fall within the protection scope of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the present application.

[0119] Embodiment 4

[0120] This embodiment provides a self-supervised point cloud object segmentation method based on a dual attention mechanism and contrast regularization. Based on embodiments 1-3, step 2 is specifically:

[0121] Step 2.1, first, image features are extracted: using a ResNet network to embed a 2D image rendered by point cloud data into a feature space, specifically: introducing a residual block, realizing an identity mapping through a skip connection, and gradually extracting different levels of features through a convolutional layer, a shallow network extracts low-level features such as edges and textures, and a deep network gradually extracts high-level semantic features through stacked residual blocks, as shown in formula (8):

[0122] f I = F I (I) (8)

[0123] In formula (8), f i is the extracted image feature, F i is an image feature extractor, and I is the input image.

[0124] Step 2.2, based on the obtained point cloud feature Z t and image feature f I , a cross-space attention mechanism is used to fuse the mapping relationship of image texture and point cloud geometry in spatial position, accurately align the spatial semantics of the two, and the module structure is as shown in Figure 2 .

[0125] Step 2.3, then a cross-channel attention mechanism is used to deeply mine the semantic interaction between cross-modal channels, realize the fusion at the feature channel level, and the module structure is as shown in Figure 3 .

[0126] Step 2.4, using an adaptive dynamic fusion mechanism, dynamically adjusts the fusion ratio of point cloud and image features and the contribution weight of different modalities, so as to maximize the complementarity of point cloud features and image features.

[0127] This embodiment only represents the preferred embodiment of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the present application. Any point cloud object segmentation method designed by using similar technical features as the present application will fall within the protection scope of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the present application.

[0128] Embodiment 5

[0129] This embodiment provides a self-supervised point cloud object segmentation method based on a dual attention mechanism and contrast regularization. Based on embodiments 1-4, step 2.2 is specifically as follows:

[0130] Step 2.2.1, for given point cloud features Z t , two-dimensional image features f I , project the two features into the same dimension Q, K, V space through linear transformation, align the semantic space of point cloud and image modalities, as shown in formula (9):

[0131]

[0132] Step 2.2.2, multiply Q p and K I matrices to calculate the similarity matrix Scores p-I between point cloud features and image features

[0133] Scores p-I = Q p ·K I T

[0134] As formula (10), apply Softmax to normalize the similarity matrix to generate attention weight matrix to get the contribution degree of image features to each point cloud feature;

[0135] Similarly, multiply Q I and K p matrices to get similarity matrix Scores I-p ,

[0136] Scores I-p = Q I ·K p T

[0137] And normalize to get the contribution of the point cloud to the image as formula (10)

[0138] In formula (10), d k is a scaling factor;

[0139] Step 2.2.3, the image point cloud features are weighted and summed using attention weights to generate fused cross-modal features, and the image semantic features are embedded into the point cloud features to obtain fused point cloud features The point cloud semantic features are embedded into the image features to obtain fused image features As shown in the following formula (11):

[0140]

[0141] Step 2.3 is as follows:

[0142] Step 2.3.1, first, the cross-space attention fused point cloud features and image features are dimensionally aligned through a linear layer, and then globally averaged pooling is performed to obtain one-dimensional point cloud features and one-dimensional image features The formula is shown in the following formula (12), where Linear represents a fully connected layer:

[0143]

[0144] Step 2.3.2, from step 2.3.1, two one-dimensional vectors are obtained, which are respectively passed through an MLP multi-layer perceptron learning channel weight, and then a channel weight is generated through a sigmoid activation function, realizing channel scaling. Finally, the attention weight of each channel is multiplied by the respective input point cloud feature and image feature to obtain the feature vector of the point cloud and image fused through the channel attention mechanism The specific formula is shown in the following formula (13), where represents matrix multiplication,

[0145]

[0146] Step 2.4 is as follows:

[0147] Step 2.4.1, the point cloud features and image features obtained in step 2.3.2 are fused, first, the point cloud features and image features are spliced in the channel dimension to serve as the basis for calculating the gating weight, then the feature dimension is adjusted through a fully connected layer, and a sigmoid activation function is used to generate the gating weight, as shown in the following formula (14):

[0148]

[0149] In formula (14), Concat is a concatenation function used to concatenate and two features;

[0150] Step 2.4.2, element-wise fusion of point cloud and image features according to the gating weight, to calculate the fused feature F fused as follows formula (15):

[0151]

[0152] In formula (14)-(15), represents matrix multiplication, and γ is a dynamically adjusted weight, γ∈[0,1], when γ is 1, the corresponding feature is dominated by the geometric structure of the point cloud, and when γ is 0, the corresponding feature is dominated by the texture feature of the image.

[0153] This embodiment only represents the preferred embodiment of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the present application, and any point cloud object segmentation method designed by using similar technical features of the present application will fall within the protection scope of the self-supervised point cloud object segmentation method based on the dual attention mechanism and contrast regularization of the present application.

[0154] Embodiment 6

[0155] This embodiment provides a self-supervised point cloud object segmentation method based on a dual attention mechanism and contrast regularization, and on the basis of embodiments 1-5, step 3 is specifically:

[0156] Step 3.1, generating a global descriptor in the channel direction, the module structure is as shown in Figure 4 : compressing the dual attention fusion feature F fused in the channel direction, and applying a weight matrix W , where r is a reduction factor to reduce the dimension of the fusion output F fused ∈R N×C ;

[0157] Step 3.2, using a ReLU function to provide nonlinear changes after the linear mapping of W c ; finally, the channel responses of all points are aggregated through an average pooling operation to compress the spatial information into a global channel descriptor g c , and the specific mathematical formula is as follows formula (16):

[0158] g c =avg N (ReLU(F fused ·W c )) (16)

[0159] In formula (16), g c ∈R C / r, F fused ·W c is F fused and W c between the matrix product, avg N is the average pooling on N points, the reduction factor is an integer satisfying r≥2, and in addition, represents the global average response of the j-th channel in the feature map of the entire point cloud;

[0160] Step 3.3, generating global descriptors of point direction: using the weight matrix W p ∈R C×C / r , ReLU function and average pooling operation, the formula is as follows formula (17):

[0161]

[0162] In formula (17), g p ∈R N ,g p =[ε1,…,ε i ,…,ε N ],ε i represents the global response of the i-th in the feature map of the entire point cloud, avg C / r indicates the corresponding aggregation of channel dimension by average pooling, capturing the global dependence between points;

[0163] Step 3.4, taking the square root of the outer product of the channel descriptor g c obtained in step 3.2 and the point direction descriptor g p obtained in step 3.3, to generate a low-rank bilinear response matrix G∈r N×C / r , where γ ij represents the global response value of the i-th point and the j-th channel, the formula is as follows formula (18):

[0164]

[0165] In formula (18), g represents the outer product, generating a point-channel cross matrix G, and taking the square root is to balance the point and channel level contribution and avoid gradient explosion;

[0166] Step 3.5, using a shared MLP to map G back to the original feature space, and through the residual connection and the original feature fusion to retain the original feature, to obtain the feature F G , as follows formula (19);

[0167] F G =MLP(G+F fused E c +F fusedW p ) (19);

[0168] Step 3.6, in order to enhance the learning function, the features F dynamically fused through the attention mechanism fused Subtract F G to suppress redundant common feature information, finally, using the activation function σ to enhance the nonlinearity, get the final point cloud feature F out , the formula is as follows formula (20):

[0169] F out = σ (F fusec -F G ) (20).

[0170] The point cloud object segmentation method of the embodiment is compared with the DGCNN method for chair and car segmentation effect, and the final chair and car point cloud object part segmentation results are shown in Figures 5 and 6. Comparing Figures 5(b), 5(c) with Figure 5(a), and comparing Figures 6(b), 6(c) with Figure 6(a), the segmentation results of the embodiment are closer to the true value. From the visualization results, it can be seen that the segmentation effect of the embodiment is better than the DGCNN method.

[0171] This embodiment only represents the preferred embodiment of the self-supervised point cloud object segmentation method based on the double attention mechanism and contrast regularization of the present application. Any point cloud object segmentation method designed by using similar technical features of the present application will fall within the protection scope of the self-supervised point cloud object segmentation method based on the double attention mechanism and contrast regularization of the present application.

[0172] The self-supervised point cloud object segmentation method based on the double attention mechanism and contrast regularization provided by the present application is described in detail above, and the embodiments of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the structure of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific embodiments and application scope will be changed; in conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A self-supervised point cloud object segmentation method based on dual attention mechanism and contrast regularization, characterized by: Specifically, first, a geometric transformation enhancement strategy is used to generate multi-perspective point cloud data, and the contrastive learning mechanism is used to capture the intra-modal geometric invariance features; then, cross-spatial attention and cross-channel attention are used to achieve deep fusion of image texture and point cloud geometric features, and a learnable weight allocation strategy is introduced to dynamically allocate weights for images and point clouds; finally, a global response matrix is ​​constructed by integrating global features through global channel-wise and point-wise descriptors, thereby enhancing the interactive expression ability of local details and global semantics and completing segmentation.

2. The self-supervised point cloud object segmentation method based on dual attention mechanism and contrast regularization according to claim 1, characterized in that: Please follow the steps below to implement: Step 1: Dynamic point cloud enhanced contrast learning: The input point cloud adopts a random geometric transformation strategy to perform geometric transformation operations on the original point cloud data, dynamically generate point cloud representation data of two different perspectives, and then build a contrast learning mechanism to mutually contrast and constrain the internal features of the point clouds of different perspectives, and finally extract the geometric features Z of the point clouds of different perspectives within the modality. t ; Step 2, dual attention multimodal fusion: First, extract the geometric features f of the image I , then combined with cross-spatial attention and cross-channel attention, the mapping relationship between image texture and point cloud geometry in spatial position is integrated, the semantic interaction between cross-modal channels is deeply mined, and the dynamic fusion mechanism is combined to obtain the feature F of point cloud and image fusion. fused ; Step 3, global bilinear regularization feature extraction: The global response matrix G is constructed by integrating global features through global channel mode and point mode descriptors, and the global feature F is obtained through MLP. G , use the activation function σ to enhance nonlinearity and obtain the point cloud feature F out ; Step 4, segmentation prediction: the final point cloud feature F out The final segmentation result is obtained through two multi-layer perceptrons and a Dropout layer, and then through Softmax.

3. The self-supervised point cloud object segmentation method based on dual attention mechanism and contrast regularization according to claim 1, characterized in that: The step 1 is specifically as follows: Step 1.1, the initial point cloud P = {p1,p2,…,p j ,…,p N }, Converts to T Where N represents the total number of samples in the feature space, P j Represents the jth point, as shown in formula (1): In formula (1), α and β represent two views of the point cloud; Step 1.2, after forming two views, use feature encoding to extract global point cloud features. First, establish dynamic graph convolution of local points and use k-nearest neighbor algorithm to build dynamic graph in feature space. The neighborhood of each point is dynamically updated according to the current feature similarity. Then, for each point and its neighborhood point pair (p i ,p j ), calculate the edge features, such as formula (2): e ij =h(p i ,p j -p i ) (2) In formula (2), e ij Represents edge features, h represents a multi-layer perceptron, which is used to fuse the center point p i and relative position information p j -p i ; The maximum pooling symmetric function is used to aggregate neighborhood features, retaining local features and global information, as shown in formula (3): d α ,f α =F P (p α ),d β ,f β =F P (p β ) (3) In formula (3), d and f are the location and features of the downsampling point respectively, F P It is a point cloud feature extractor; Step 1.3, project the extracted feature vectors. and The projection vectors are expressed as and The specific formula is as follows: In formula (4), g P For projection head, used to maximize and Similarity between them, minimize p j Similarity of projection vectors to other points; In step 1.4, the Hungarian algorithm is used to construct positive sample pairs to achieve optimal bidirectional matching between cross-view points. Specifically, the downsampled point sets of the two views are regarded as two vertex sets of a bipartite graph, and the cost matrix is ​​constructed by calculating the distance between the point pairs, as shown in the following formula (5): M=(cost=||T(d α )-(d β )||2) (5) In formula (5), T is the geometric transformation in formula (1). Then, the Euclidean distance between the points in view α and the points in view β is calculated using the L2 normal form to find the corresponding matching relationship between the points in the two views. Step 1.5: After establishing the corresponding relationship, the adversarial contrast loss is used to calculate the loss: In formulas (6)-(7), s(.) is the cosine similarity, τ is the temperature factor, and N is the batch size of the input points. represents the loss from α view to β view, Denotes the loss L from β view to α view in Represents the overall loss between the two views, that is, the intra-modal loss.

4. The self-supervised point cloud object segmentation method based on dual attention mechanism and contrast regularization according to claim 3, characterized in that: The step 2 is specifically as follows: Step 2.1: First, extract the image features: Use the ResNet network to embed the 2D image rendered by the point cloud data into the feature space. Specifically, introduce the residual block, realize the identity mapping through the jump connection, and gradually extract the features of different levels in the convolution layer. The shallow network extracts low-level features, and the deep network gradually extracts semantic high-level features by stacking residual blocks. The formula is as follows (8): f I =F I (I) (8) In formula (8), f I is the extracted image feature, F I is an image feature extractor, I is the input image; Step 2.2, the point cloud feature Z is obtained t and image features f I Based on this, we use the cross-spatial attention mechanism to fuse the mapping relationship between image texture and point cloud geometry in spatial position and accurately align the spatial semantics of the two. In step 2.3, we then use the cross-channel attention mechanism to deeply explore the semantic interactions between cross-modal channels and achieve fusion at the feature channel level. In step 2.4, an adaptive dynamic fusion mechanism is used to dynamically adjust the fusion ratio of point cloud and image features and the contribution weights of different modalities to maximize the complementarity between point cloud features and image features.

5. The self-supervised point cloud object segmentation method based on dual attention mechanism and contrast regularization according to claim 1, characterized in that: Step 2.2 is as follows: Step 2.2.1, for a given point cloud feature Z t , the two-dimensional image feature is f I , projecting the two features into the same dimension Q, K, V space through linear transformation, aligning the semantic space of point cloud and image modality, as shown in the following formula (9): Step 2.2.2, Q p and K I Perform matrix multiplication to calculate the similarity matrix Scores between point cloud features and image features p-I , Scores p-I =Q p ·K I T As shown in formula (10), Softmax is applied to normalize the similarity matrix to generate the attention weight matrix Get the contribution of image features to each point cloud feature; Similarly, Q I and K p Matrix multiplication obtains the similarity matrix Scores I-p , Scores I-p =Q I ·K p T And normalized as formula (10) to obtain the contribution of point cloud to the image In formula (10), d k is the scaling factor; Step 2.2.3: Use attention weights to perform weighted summation on the image point cloud features to generate fused cross-modal features, and embed the image semantic features into the point cloud features to obtain fused point cloud features. Embed point cloud semantic features into image features to obtain fused image features As shown in formula (11):

6. The self-supervised point cloud object segmentation method based on dual attention mechanism and contrast regularization according to claim 5, characterized in that: Step 2.3 is as follows: Step 2.3.1, first perform cross-spatial attention fusion on the point cloud features and image features Dimension alignment is performed through the linear layer, and then global average pooling is performed to obtain one-dimensional point cloud features. and one-dimensional image features The formula is shown below (12), where Linear represents the fully connected layer: Step 2.3.2, get two one-dimensional vectors from step 2.3.1, pass them through the MLP multi-layer perceptron to learn the channel weights, then generate channel weights through the sigmoid activation function to achieve channel scaling, and finally multiply the attention weights of each channel with their respective input point cloud features and image features to obtain the feature vectors of the point cloud and image fused by the channel attention mechanism The specific formula is as follows (13), where ⊙ represents matrix multiplication, 7. The self-supervised point cloud object segmentation method based on dual attention mechanism and contrast regularization according to claim 6, characterized in that: The step 2.4 is as follows: In step 2.4.1, the point cloud features and image features obtained in step 2.3.2 are fused. First, the point cloud features and image features are concatenated in the channel dimension to serve as the basis for the calculation of the gating weight. Then, the feature dimension is adjusted through the fully connected layer, and the gating weight is generated using the Sigmoid activation function. The formula is shown below (14): In formula (14), Concat is a concatenation function used to concatenate and Two features; Step 2.4.2, fuse the point cloud and image features element by element according to the gate weight, and calculate the fused feature F fused As shown in formula (15): In formulas (14) and (15), ⊙ represents matrix multiplication, and γ is a dynamically adjusted weight, γ∈[0,1]. When γ is 1, the corresponding features are dominated by the geometric structure of the point cloud, and when it is 0, they are dominated by the texture features of the image.

8. The self-supervised point cloud object segmentation method based on dual attention mechanism and contrast regularization according to claim 7, characterized in that: The step 3 is specifically as follows: Step 3.1, generate channel-wise global descriptor: fusion feature F fused Perform channel-wise compression and apply weight matrix Where r is the reduction factor to reduce the fusion output F fused ∈R N×C Dimensionality; Step 3.2, use the ReLU function to c After the linear mapping, nonlinear changes are provided; finally, the channel responses of all points are aggregated through the average pooling operation to compress the spatial information into a global channel descriptor g c , the specific mathematical formula is as follows (16): g c =avg N (ReLU(F fused ·W c )) (16) In formula (16), g c ∈R C / r , F fused W c It's F fused and W c The matrix product between avg N It is to perform average pooling on N points, and the reduction factor is an integer satisfying r≥2. In addition, Represents the global average response of the jth channel in the feature map of the entire point cloud; Step 3.3, generate a global descriptor of the point direction: use the weight matrix W p ∈R C×C / r , ReLU function and average pooling operation, the formula is as follows (17): In formula (17), g p ∈R N ,g p =[ε1,…,ε i ,…,ε N ],ε i Represents the global response of the i-th point in the feature map of the entire point cloud, avg C / r Represents the response of the channel dimension aggregated by average pooling, capturing the global dependency between points; Step 3.4: The channel descriptor g obtained in step 3.2 is c And the point direction descriptor g obtained in step 3.3 p Take the square root of the outer product of to generate a low-rank bilinear response matrix G∈R N×C / r , where γ ij represents the global response value of the i-th point and the j-th channel, and the formula is as follows (18): In formula (18), Represents the outer product, generating the point-channel cross matrix G. The square root is taken to balance the point and channel level contributions to avoid gradient explosion; Step 3.5, use the shared MLP to map G back to the original feature space, and fuse it with the original features through residual connection to retain the original features and obtain feature F G , as shown in formula (19); F G =MLP(G+F fused W c +F fused W p ) (19); Step 3.6, in order to enhance the learning function, the feature F is dynamically fused through the attention mechanism fused Subtract F G To suppress redundant common feature information, finally, the activation function σ is used to enhance nonlinearity and obtain the final point cloud feature F out , the formula is as follows (20): F out =σ(F fused -F G ) (20).

Images