Point cloud scene semantic instance joint segmentation method based on adaptive feature fusion

Abstract

The application discloses a point cloud scene semantic instance joint segmentation method based on adaptive feature fusion, and specifically comprises the following steps: constructing an LPR-AFFN network; constructing a local polar representation module LPR, and performing LPR operation on the original point cloud; performing coordinate conversion according to Z-axis rotation invariance, calculating a neighborhood centroid point matrix, updating a polar angle, and combining the polar coordinates, geometric distances and original point cloud information to obtain a feature matrix; constructing a feature representation module, inputting the feature matrix into the feature representation module, defining a local centroid point, finding the neighborhood points of the centroid point to construct a local group; preliminarily performing feature extraction on the input point cloud; constructing a feature extraction module, obtaining new discriminative features after the extracted features pass through the feature extraction module; and constructing an adaptive feature fusion module, and using the adaptive feature fusion module to integrate the new discriminative feature information to construct the global perception ability of the point cloud.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a point cloud scene semantic instance joint segmentation method based on adaptive feature fusion. BACKGROUND

[0002] With the rapid development of automatic driving, augmented reality and laser scanning technology, computer vision research has also made great progress. Humans can quickly obtain point cloud data of scenes or objects. As a main three-dimensional data representation form, point cloud can be directly obtained by 3D sensors, and point cloud can better represent the geometric structure of scenes or objects, and has been widely applied in indoor navigation, industrial detection and many other fields. However, point cloud data itself has the characteristics of irregularity, disorder and uneven distribution. Under the influence of complex scenes and other factors, it is still challenging to achieve accurate and efficient point cloud segmentation tasks.

[0003] The existing three-dimensional point cloud instance segmentation method based on deep learning can be divided into: candidate region-based instance segmentation method and candidate region-free instance segmentation method. The candidate region-based instance segmentation method: this method divides the three-dimensional point cloud instance segmentation into object prediction and instance prediction. The candidate region-free instance segmentation method: this method regards the three-dimensional point cloud instance segmentation as a subsequent step of semantic segmentation, and completes the instance segmentation task with the help of the results of semantic segmentation. However, the above-mentioned methods have poor perception ability of local context and global context. SUMMARY

[0004] The purpose of the present application is to provide a point cloud scene semantic instance joint segmentation method based on adaptive feature fusion, which can improve the perception ability of local context features and integrate instance and semantic information to enrich the global perception ability.

[0005] The technical solution adopted by the present application is a point cloud scene semantic instance joint segmentation method based on adaptive feature fusion, which is implemented according to the following steps:

[0006] Step 1, constructing an LPR-AFFN network;

[0007] Step 2, constructing a local polar representation module LPR and performing LPR operation;

[0008] Step 3, performing coordinate conversion, calculating a neighborhood centroid point matrix and updating a polar angle;

[0009] Step 4, constructing a feature representation module GPM, defining a local centroid point, and finding the neighborhood points of the centroid point to construct a local group;

[0010] Step 5, preliminarily extracting features from the input point cloud;

[0011] Step 6, build a feature extraction module PCFE and perform PCFE operation;

[0012] Step 7, build an adaptive feature fusion module AFF, and use AFF to build the global perception ability of the point cloud.

[0013] The application is also characterized in that,

[0014] In step 1, the LPR-AFFN network is composed of an encoding layer and a decoding layer and a skip connection; the encoding stage is composed of a shared encoder, and the shared encoder includes four layers of encoding, and the feature matrix size of each layer is N a ×64, N b ×128, N c ×256, and N d ×512 respectively; the decoding stage is composed of two parallel decoders, and each of the two parallel decoders includes four layers of decoding, and the feature matrix size of each layer is N c ×256, N b ×256, N a ×128, and N q ×128 respectively;

[0015] Wherein N a , N b , N c , N d , N q are the sampling points of each layer respectively;

[0016] The encoding layer and the decoding layer both use the same structure as PointNet++.

[0017] In step 2, the local polar coordinate representation module LPR is composed of K nearest neighbor algorithm, tensor maximum, neighborhood point, matrix subtraction, tensor tiling, matrix splicing, tensor mean, dimensionality reduction operation;

[0018] The specific process is as follows:

[0019] Step 2.1, the original point cloud F C is subjected to K nearest neighbor algorithm to collect k neighboring points, and k is set to 20 to obtain a neighboring point feature matrix F K .

[0020] F K = KNN (F C ) (1)

[0021] Step 2.2, the neighboring point feature matrix F K is subjected to neighborhood point operation to obtain a feature matrix F G .

[0022] F G= Gather_neighbour(F K ) (2)

[0023] Step 2.3, feature matrix F G Perform tensor max operation to get feature matrix F r .

[0024] F r = Reduce_max(F G ) (3)

[0025] Step 2.4, original point cloud F C and feature matrix F G , perform matrix subtraction operation, tensor sum operation and square root operation to get feature matrix F Q ;

[0026] F Q = Sqrt(Reduce_sum(F C -F G )) (4)

[0027] Step 2.5, original point cloud F C Perform one-dimensional tensor tiling operation to get feature matrix F T ;

[0028] F T = Tile(F C ) (5)

[0029] Step 2.6, feature matrix F Q and feature matrix F T , perform matrix concatenation operation to get feature matrix F c1 ;

[0030] F c1 = Concat(F Q ,F T ) (6)

[0031] Step 2.7, feature matrix F G and feature matrix F c1 , perform matrix concatenation operation to get feature matrix F c2 ;

[0032] F c2 = Concat(F G ,F c1 ) (7)

[0033] Step 2.8, feature matrix F c2 , perform twice tensor max operation to get feature matrix F M ;

[0034] F M = Reduce_max(Reduce_max(F c2 )) (8)

[0035] Step 2.9, power index operation is performed on the feature matrix F M , to obtain the feature matrix F x ;

[0036] F x = Pow(F M ) (9)

[0037] Step 2.10, matrix subtraction is performed on the neighboring point feature matrix F r and the original point cloud to obtain the matrix feature F1;

[0038] F1= F r - F C (10)

[0039] Step 2.11, one-dimensional tensor tiling operation is performed on the feature matrix F1 to obtain the feature matrix F t ;

[0040] F t = Tile(F1) (11).

[0041] The specific process of Step 3 is as follows:

[0042] Step 3.1, for any point p i , whose coordinate values are (x i , y i , z i ), k neighboring points p k , whose coordinate values are (x k , y k , z k ), are collected using the K-neighbor algorithm. The relative coordinates of the point p i and p k are represented as wherein The distance from any neighboring point p k to the point p i is denoted as The angle between the corresponding rays of the point p i and the point p k and the reference direction is denoted as

[0043]

[0044]

[0045]

[0046] Step 3.2, for any point p i , the tensor sum operation and square root operation are performed on the feature matrix F i by k neighboring points of p t , to obtain the neighborhood centroid point feature matrix F z ;

[0047] F z = sqrt(Reduce_sum(F t )) (15)

[0048] Step 3.3, coordinate conversion feature matrix F x and neighborhood centroid point feature matrix F z , after two matrix subtraction operations, the feature matrix is spliced, and the polar angle and According to the Z-axis rotation invariance, and do not change;

[0049] F2 = F x - F z (16)

[0050] F3 = Concat(F2, F2) (17)

[0051] F4 = Concat(F3, F x ) (18)

[0052] Step 3.4, the new feature matrix F4 generated after splicing twice is subjected to a tensor mean operation to obtain a new feature matrix F m ;

[0053] F m = Reduce_mean(F4) (19) Step 3.5, the new feature matrix F m is subjected to dimension reduction to obtain a feature matrix F lpr ;

[0054] Step 3.6, the original point cloud F C , polar coordinate conversion information and geometric distance information are spliced to output a feature matrix F a of size N LPR ×12;

[0055] F LPR = Concat(F lpr , F C ) (20).

[0056] In step 4, the feature representation module GPM consists of convolutional layers, ReLU activation function, multilayer perceptron (MLP), and gate operation for updating combined local features;

[0057] The specific process is as follows:

[0058] Step 4.1: Select a point A from the input point cloud as the query point according to the farthest sampling point algorithm, and select the farthest point B from the remaining point cloud. At this time, the query point is (A,B). Then select the farthest point C from the remaining point cloud. Select the minimum value between the distance from point A to point C and the distance from point B to point C as the distance d to the query point. Divide the sampled point cloud into N query point cloud sets. Randomly select a group of points from the query point cloud sets, define a local centroid in the query point set, and use the K-nearest neighbor algorithm to find the neighboring points of the centroid to construct local groups.

[0059] Step 4.2: After projecting the point cloud grouped in Step 4.1 using the MLP (32,32,64) operation, the feature matrix F is obtained. mlp ;

[0060] F mlp =MLP(F LPR ) (twenty one)

[0061] Step 4.3, for the characteristic matrix F mlp The feature matrix F is obtained by performing matrix dimension transformation and one-dimensional convolution operations. map ;

[0062] F map =Conv1D(reshape(F mlp )) (twenty two)

[0063] Step 4.4, for the characteristic matrix F map Two one-dimensional convolution operations are performed to obtain the same feature matrix F. w1 and F w2 ;

[0064] F w1 =Conv1D(F map ) (twenty three)

[0065] F w2 =Conv1D(F map ) (twenty four)

[0066] Step 4.5, for the feature matrix F w1 Perform the transpose operation to obtain the characteristic matrix F. t1 ;

[0067] F t1= Trranspose(F w1 ) (25)

[0068] Step 4.6, the feature matrix F t1 and F w2 are added to obtain the feature matrix F add ;

[0069] F add = Multiply(F t1 , F w2 ) (26)

[0070] Step 4.7, the feature matrix F add is subjected to ReLu function and Softmax operation to obtain the feature matrix F s ;

[0071] F s = Softmax(ReLu(F add )) (27)

[0072] Step 4.8, the feature matrix F s and F map are multiplied to obtain the feature matrix F x1 ;

[0073] F x1 = Multiply(F s , F map ) (28)

[0074] Step 4.9, the feature matrix F x1 is subjected to activation function and update combined local feature gate operation to obtain the feature matrix F g1 ;

[0075] F g1 = gate(F x1 ) (29)

[0076] Step 4.10, the feature matrix F mlp is subjected to matrix dimension change and update combined local feature gate operation to obtain the feature matrix F g2 ;

[0077] F g2 = gate(reshape(F mlp )) (30)

[0078] Step 4.11, the feature matrix F x1 is subjected to activation function, matrix dimension change operation, and the feature matrix F g2 is subjected to matrix multiplication operation to obtain the feature matrix F x2 ;

[0079] F x2 = Multiply(reshape(F x1 ), F g2 ) (31)

[0080] Step 4.12, matrix multiplication is performed on the feature matrix F mlp and F g1 after matrix dimension operation, respectively, to obtain a new feature matrix F x3 ;

[0081] F x3 = Multiply(reshape(F mlp ), reshape(F g1 )) (32)

[0082] Step 4.13, matrix splicing is performed on the feature matrix F x2 and F x3 to obtain a feature matrix F p with a size of N m x N c ;

[0083] F c = Concat(F x2 , F x3 ) (33).

[0084] The specific process of step 5 is as follows:

[0085] Step 5.1, the encoding mode adopted by the last three layers of the encoding layer is the same as PointNet++, and each encoding layer will complete the extraction of the input feature once, so the output dimensions of the local features extracted by the four encoding layers are (64, 128, 256, 512) respectively.

[0086] Step 5.2, the decoding layer adopts the same decoding mode as pointNet++, which is to update the features after interpolation operation and jump connection combination by upsampling, and to save the feature information generated by the encoding layer as much as possible. The encoding layer and the decoding layer correspond to each other, and the output dimensions of the updated and combined features after upsampling are (256, 256, 128, 128) respectively, that is, a feature matrix with a size of N a x 128 is obtained.

[0087] In step 6, the feature extraction module PCFE is composed of convolutional layers, upsampling operations and matrix splicing operations.

[0088] The specific process is as follows:

[0089] Step 6.1, two Na The eigenma matrix of 128 × 10⁸ is represented by F. I and F S To represent this, taking semantic segmentation as an example, firstly, for F... I Upsampling is performed to obtain a new feature matrix F′ I ;

[0090] F′ I =Upsample(F I (34)

[0091] Step 6.2, F′ I and F S Connecting them together yields a size of N. a The eigenmatrix F′ of 256 × 10⁶ SI ;

[0092] F′ SI =Concat(F′ I ,F S (35)

[0093] Step 6.3, for the characteristic matrix F′ SI Perform a one-dimensional convolution operation, with an output size of N. a The eigenmatrix F of 128 × 10⁸ SI ;

[0094] F SI =Conv1D(F′ SI (36)

[0095] Step 6.4: The entire decoder has two parallel branches. Similarly, for instance segmentation tasks, F... A and F B To represent N a A feature matrix of 128 × 10⁸;

[0096] Step 6.5, F A Upsampling is performed to obtain a new feature matrix F′ A ;

[0097] F′ A =Upsample(F A (37)

[0098] Step 6.6, F′ A and F B Connecting them together yields a size of N. a The eigenmatrix F′ of 256 × 10⁶ BA ;

[0099] F′ BA =Concat(F′ A ,F B(38)

[0100] Step 6.7, for the characteristic matrix F′ BA Perform a one-dimensional convolution operation, with an output size of N. a The eigenmatrix F of 128 × 10⁸ BA ;

[0101] F BA =Conv1D(F′ BA (39).

[0102] In step 7, the adaptive feature fusion module (AFF) consists of convolutional layers, softmax, matrix multiplication, and matrix addition operations.

[0103] The specific process is as follows:

[0104] Step 7.1, in the semantic segmentation task, the adaptive feature fusion module (AFF) first inputs a value of N to the feature extraction module (PCFE). a The semantic feature matrix F of 128 × SI Expand the dimensions, then perform a two-dimensional convolution operation to output the feature matrix F. SII ;

[0105] F SII =Conv2D(F SI (40)

[0106] Step 7.2, for F SII The feature matrix F output by the normalized Softmax operation SIS ;

[0107] F SIS =Softmax(F SII (41)

[0108] Step 7.3, Feature matrix F SIS With F SII Perform matrix dot product and output the characteristic matrix F. sem ;

[0109] F sem =Multiply(F SIS ,F SII (42)

[0110] Step 7.4, Feature matrix F sem The size generated after dimensionality reduction is N a The semantic feature matrix F of 128 × SEM semantic feature matrix F SEM After passing through a one-dimensional convolution operation in a fully connected layer, the output size is N. a ×N csemantic prediction P sem , for predicting the final semantic class, where N c is the number of semantic prediction classes,

[0111] P sem = Conv1D(F SEM ) (43)

[0112] Step 7.5, for the instance segmentation task, the instance feature matrix F a with the same size of N BA ×128 is expanded in dimension, and after the expansion, a two-dimensional convolution operation is performed to generate a feature matrix F IBA .

[0113] F IBA = Conv2D(F BA ) (44)

[0114] Step 7.6, the feature matrix F IBA is subjected to a normalization Softmax operation to generate a feature matrix F BAI .

[0115] F BAI = Softmax(F IBA ) (45)

[0116] Step 7.7, the feature matrix F IBA is subjected to a matrix point multiplication operation with F BAI , and a feature matrix F ins is output.

[0117] F ins = Multiply(F IBA , F BAI ) (46)

[0118] Step 7.8, the instance feature matrix F ins and the semantic feature matrix F sem are subjected to matrix addition, and a feature matrix F′ ins is output.

[0119] F′ ins = F ins + F sem (47)

[0120] Step 7.9, the feature matrix F′ ins is subjected to a dimension reduction operation to generate an instance feature matrix F a with a size of N INS ×128, and after the instance feature matrix F INS is subjected to a one-dimensional convolution operation through a full connection layer, an output with a size of N a ×N eExample embedding E ins where N e is the dimension of the example feature embedding;

[0121] F ins =Conv1D(F INS ) (48).

[0122] The beneficial effects of the present application are that the point cloud scene semantic instance joint segmentation method based on adaptive feature fusion solves the problem of direct learning of point feature direction sensitivity, improves the perception ability of local context features, and integrates instance and semantic information to enrich the global perception ability. BRIEF DESCRIPTION OF DRAWINGS

[0123] Figure 1 is the flowchart of the joint instance segmentation method based on adaptive feature fusion of the present application;

[0124] Figure 2 is the LPR-AFFN network model structure diagram proposed in the method of the present application;

[0125] Figure 3 is the local polar coordinate representation module LPR structure diagram proposed in the present application;

[0126] Figure 4 is the feature representation GPM module structure diagram in the present application;

[0127] Figure 5 is the feature extraction module PCFE structure diagram in the present application;

[0128] Figure 6 is the adaptive feature fusion module AFF structure diagram in the present application. DETAILED DESCRIPTION

[0129] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0130] The point cloud scene semantic instance joint segmentation method based on adaptive feature fusion of the present application is implemented according to the following steps, as shown in Figure 1 .

[0131] Embodiment 1

[0132] Step 1, construct LPR-AFFN network;

[0133] As shown in Figure 2 , the LPR-AFFN network is composed of an encoding layer and a decoding layer and a skip connection, wherein the encoding layer and the decoding layer both use the same structure as PointNet++, that is: the entire encoding stage is composed of a shared encoder, and the shared encoder includes four layers of encoding, and the feature matrix size output by each layer is Na ×64, N b ×128, N c ×256, N d ×512; similarly, the entire decoding stage consists of two parallel decoders, both of which include four layers of decoding, and the feature matrix output by each layer has a size of N c ×256, N b ×256, N a ×128, N q ×128, where N a , N b , N c , N d , N q are the number of sampling points of each layer, respectively;

[0134] Step 2, construct a local polar coordinate representation module LPR, and the original point cloud F C After the local polar coordinate representation module LPR operation, the feature matrix F t is obtained.

[0135] The specific process of step 2 is as follows:

[0136] Step 2.1, the original point cloud F C is collected using the K-nearest neighbor algorithm to collect k neighboring points, k is set to 20, and the neighboring point feature matrix F K is obtained.

[0137] F K = LNN(F C ) (1)

[0138] Step 2.2, the neighboring point feature matrix F K is operated to obtain the feature matrix F G .

[0139] F G = Gather_neighbour(F K ) (2)

[0140] Step 2.3, the feature matrix F G is operated to obtain the feature matrix F r .

[0141] F r = Reduce_max(F G ) (3)

[0142] Step 2.4, the original point cloud F C and the feature matrix F G , are operated to obtain the feature matrix FQ ;

[0143] F Q = Sqrt(Reduce_sum(F C -F G )) (4)

[0144] Step 2.5, the original point cloud F C is subjected to a one-dimensional tensor tiling operation to obtain a feature matrix F T ;

[0145] F T = Tile(F C ) (5)

[0146] Step 2.6, the feature matrix F Q and the feature matrix F T are subjected to a matrix concatenation operation to obtain a feature matrix F c1 ;

[0147] F c1 = Concat(F Q ,F T ) (6)

[0148] Step 2.7, the feature matrix F G and the feature matrix F c1 are subjected to a matrix concatenation operation to obtain a feature matrix F c2 ;

[0149] F c2 = Concat(F G ,F c1 ) (7)

[0150] Step 2.8, the feature matrix F c2 is subjected to two tensor maximum value operations to obtain a feature matrix F M ;

[0151] F M = Reduce_max(Reduce_max(F c2 )) (8) Step 2.9, the feature matrix F M is subjected to a power exponent operation to obtain a feature matrix F x ;

[0152] F x = Pow(F M ) (9)

[0153] Step 2.10, the adjacent point feature matrix F r and the original point cloud are subjected to a matrix subtraction to obtain a matrix feature F1;

[0154] F1 = F r -F C (10)

[0155] Step 2.11, the feature matrix F1 is subjected to one-dimensional tensor tiling operation to obtain the feature matrix F t ;

[0156] F t = Tile(F1) (11)

[0157] Step 3, according to the Z-axis rotation invariance, coordinate transformation is performed, the neighborhood centroid point matrix is calculated, the polar angle is updated, and the polar coordinates, geometric distance and original point cloud information are combined to obtain the feature matrix F LPR ;

[0158] Step 3.1, for any point p i , whose coordinate value is (x i , y i , z i ), k neighboring points p k , whose coordinate value is (x k , y k , z k ), are collected by using the K-neighbor algorithm. The relative coordinates of the points p i and p k are represented as wherein The distance from any neighboring point p k to the point p i is denoted as The angle between the corresponding rays of the points p i and p k and the reference direction is denoted as, and the amplitude angle of the complex form of any point in the polar coordinate system is denoted as, then the polar coordinates of the neighboring points are represented as

[0159]

[0160]

[0161]

[0162] Step 3.2, for any point p i , the feature matrix F i is subjected to tensor summation operation and square root operation by using the k neighboring points of p t to obtain the neighborhood centroid point feature matrix F z ;

[0163] F z = Sqrt(Reduce_sum(Ft )) (15)

[0164] Step 3.3, coordinate conversion feature matrix F x and neighborhood centroid point feature matrix F z , after two matrix subtraction operations, splice feature matrix, update polar angle and According to the Z-axis rotation invariance, and Will not change;

[0165] F2=F x -F z (16)

[0166] F3=Concat(F2,F2) (17)

[0167] F4=Concat(F3,F x ) (18)

[0168] Step 3.4, the new feature matrix F4 generated after splicing twice is subjected to tensor mean operation to obtain a new feature matrix F m ;

[0169] F m =Reduce_mean(F4) (19)

[0170] Step 3.5, the new feature matrix F m is subjected to dimension reduction to obtain a feature matrix F lpr ;

[0171] Step 3.6, the original point cloud F C , polar coordinate conversion information and geometric distance information are spliced to output a feature matrix F a with a size of N LPR ×12;

[0172] F LPR =Concat(F lpr ,F C ) (20)

[0173] Step 4, construct a feature representation module GPM, input the feature matrix F LPR obtained in step 3 into the feature representation module GPM, and define a local centroid point to find the neighborhood points of the centroid point to construct a local group;

[0174] The specific process of step 4 is as follows:

[0175] Step 4.1, select a point A as a query point from the input point cloud according to the farthest sampling point algorithm, select a farthest point B from the remaining point cloud, at this time the query point is (A, B), then select a farthest point C from the remaining point cloud, select the minimum value of the distance between point A and point C and the distance between point B and point C as the distance d to the query point, divide the sampled point cloud into N query point cloud sets, select a set of points from the query point cloud set, define a local centroid point in the query point set, and find the neighboring points of the centroid point to construct a local group by using the K nearest neighbor algorithm;

[0176] Step 4.2, after the point cloud grouped in step 4.1 is projected by MLP(32, 32, 64) operation, a feature matrix F is obtained mlp ;

[0177] F mlp = MLP(F LPR ) (21)

[0178] Step 4.3, perform matrix dimensionality reduction and one-dimensional convolution operation on the feature matrix F mlp to obtain a feature matrix F map ;

[0179] F map = Conv1D(reshape(F mlp )) (22)

[0180] Step 4.4, perform two one-dimensional convolution operations on the feature matrix F map respectively to obtain feature matrices F w1 and F w2 ;

[0181] F w1 = Conv1D(F map ) (23)

[0182] F w2 = Conv1D(F map ) (24)

[0183] Step 4.5, perform a transpose operation on the feature matrix F w1 to obtain a feature matrix F t1 ;

[0184] F t1 = Transpose(F w1 ) (25)

[0185] Step 4.6, perform matrix addition on the feature matrices F t1 and F w2 to obtain a feature matrix F add ;

[0186] F add = Multiply(F t1 ,F w2 ) (26)

[0187] Step 4.7, feature matrix F add After ReLu function and Softmax operation, the feature matrix F s is obtained.

[0188] F s = Softmax(ReLu(F add )) (27)

[0189] Step 4.8, feature matrix F s and F map are multiplied to obtain the feature matrix F x1 .

[0190] F x1 = Multiply(F s ,F map ) (28)

[0191] Step 4.9, feature matrix F x1 After the activation function and the update combined local feature gate operation, the feature matrix F g1 is obtained.

[0192] F g1 = gate(F x1 ) (29)

[0193] Step 4.10, feature matrix F mlp After the matrix dimensionality reduction and the update combined local feature gate operation, the feature matrix F g2 is obtained.

[0194] F g2 = gate(reshape(F mlp )) (30)

[0195] Step 4.11, feature matrix F x1 After the activation function, the matrix dimensionality reduction operation, and the matrix multiplication operation with the feature matrix F g2 , the feature matrix F x2 is obtained.

[0196] F x2 = Multiply(reshape(F x1 ),F g2 ) (31)

[0197] Step 4.12, feature matrix F mlpand F g1 Matrix multiplication is performed after matrix dimension operation respectively to obtain a new feature matrix F x3 ;

[0198] F x3 = Multiply(reshape(F mlp ), reshape(F g1 )) (32)

[0199] Step 4.13, matrix splicing is performed on the feature matrices F x2 and F x3 to obtain a feature matrix F p of size N m × N c ;

[0200] F c = Concat(F x2 , F x3 ) (33)

[0201] Step 5, replace the first layer encoder with the feature representation module GPM, retain the max pooling operation of the first layer of the shared encoder, and then preliminarily extract features from the input point cloud;

[0202] The specific process of step 5 is as follows:

[0203] Step 5.1, the encoding mode adopted by the last three layers of the encoding layer is the same as that of PointNet++, and each encoding layer will complete feature extraction of the input once, so the local feature output dimensions of the four encoding layers are (64, 128, 256, 512) respectively;

[0204] Step 5.2, the decoding layer adopts the same decoding mode as pointNet++, which is to upsample the features of the encoding layer, update the features combined after interpolation operation and jump connection, and save as much feature information generated by the encoding layer as possible. The encoding layer and the decoding layer correspond to each other, and the output dimensions of the features updated and combined after upsampling are (256, 256, 128, 128) respectively, that is, a feature matrix of N a ×128 is obtained;

[0205] Step 6, construct the feature extraction module PCFE, and obtain new discriminative features after the features extracted in step 5 pass through the feature extraction module PCFE;

[0206] The specific process of step 6 is as follows:

[0207] Step 6.1, two feature matrices of N a ×128 obtained in step 5.2 are F I and F STo represent this, taking semantic segmentation as an example, firstly, for F... I Upsampling is performed to obtain a new feature matrix F′ I ;

[0208] F′ I =Upsample(F I (34)

[0209] Step 6.2, F′ I and F S Connecting them together yields a size of N. a The eigenmatrix F′ of 256 × 10⁶ SI ;

[0210] F′ SI =Concat(F′ I ,F S (35)

[0211] Step 6.3, for the characteristic matrix F′ SI Perform a one-dimensional convolution operation, with an output size of N. a The eigenmatrix F of 128 × 10⁸ SI ;

[0212] F SI =Conv1D(F′ SI (36)

[0213] Step 6.4: The entire decoder has two parallel branches. Similarly, for instance segmentation tasks, F... A and F B To represent N a A feature matrix of 128 × 10⁸;

[0214] Step 6.5, F A Upsampling is performed to obtain a new feature matrix F′ A ;

[0215] F′ A =Upsample(F A (37)

[0216] Step 6.6, F′ A and F B Connecting them together yields a size of N. a The eigenmatrix F′ of 256 × 10⁶ BA ;

[0217] F′ BA =Concat(F′ A ,F B (38)

[0218] Step 6.7, for the characteristic matrix F′BA Perform a one-dimensional convolution operation, with an output size of N. a The eigenmatrix F of 128 × 10⁸ BA ;

[0219] F BA =Conv1D(F′ BA (39)

[0220] Step 7: Construct an adaptive feature fusion module (AFF). Use the AFF to integrate the new discriminative feature information obtained in Step 6 to build the global perception capability of the point cloud, thereby promoting instance embedding and semantic prediction.

[0221] The specific process of step 7 is as follows:

[0222] Step 7.1, in the semantic segmentation task, the adaptive feature fusion module (AFF) first inputs a value of N to the feature extraction module (PCFE). a The semantic feature matrix F of 128 × SI Expand the dimensions, then perform a two-dimensional convolution operation to output the feature matrix F. SII ;

[0223] F SII =Conv2D(F SI (40)

[0224] Step 7.2, for F SII The feature matrix F output by the normalized Softmax operation SIS ;

[0225] F SIS =Softmax(F SII (41)

[0226] Step 7.3, Feature matrix F SIS With F SII Perform matrix dot product and output the characteristic matrix F. sem ;

[0227] F sem =Multiply(F SIS ,F SII (42)

[0228] Step 7.4, Feature matrix F sem The size generated after dimensionality reduction is N a The semantic feature matrix F of 128 × SEM semantic feature matrix F SEM After passing through a one-dimensional convolution operation in a fully connected layer, the output size is N. a ×N c semantic prediction P sem, used to predict the final semantic category, where N c It is the number of categories in the semantic prediction.

[0229] P sem =Conv1D(F SEM (43)

[0230] Step 7.5, for instance segmentation tasks, for instances of the same size N... a The instance feature matrix F of ×128 BA The dimensions are expanded, and the resulting feature matrix F is generated through a two-dimensional convolution operation. IBA ;

[0231] F IBA =Conv2D(F BA (44)

[0232] Step 7.6, for the characteristic matrix F IBA Normalized Softmax operation is performed to generate the feature matrix F BAI ;

[0233] F BAI =Softmax(F IBA (45)

[0234] Step 7.7, Feature matrix F IBA With F BAI Perform matrix dot product and output the characteristic matrix F. ins ;

[0235] F ins =Multiply(F IBA ,F BAI (46)

[0236] Step 7.8, for the instance feature matrix F ins and semantic feature matrix F sem Perform matrix addition and output the characteristic matrix F′ ins ;

[0237] F′ ins =F ins +F sem (47)

[0238] Step 7.9, Feature matrix F′ ins After dimensionality reduction, a size of N is generated. a The instance feature matrix F of 128 × 10⁸ INS Instance feature matrix F INS After passing through a fully connected layer and undergoing a one-dimensional convolution operation, the output size is N. a ×N e Instance embedding E ins, where N e It is the dimension of instance feature embedding;

[0239] E ins =Conv1D(F INS (48).

[0240] Example 2

[0241] The difference from Example 1 is as follows: Figure 3 As shown, the Local Polar Representation (LPR) module consists of the K-nearest neighbor algorithm, tensor maximum value, neighborhood points, matrix subtraction, tensor tiling, matrix concatenation, tensor mean, and dimensionality reduction operations.

[0242] Example 3

[0243] The difference from Example 2 is as follows: Figure 4 As shown, the feature representation module GPM consists of convolutional layers, ReLU activation function, multilayer perceptron (MLP), and gate operation for updating combined local features.

[0244] Example 4

[0245] The difference from Example 3 is as follows: Figure 5 As shown, the feature extraction module PCFE consists of convolutional layers, upsampling operations, and matrix concatenation operations.

[0246] Example 5

[0247] The difference from Example 4 is that: Figure 6 As shown, the Adaptive Feature Fusion (AFF) module consists of convolutional layers, Softmax, matrix multiplication, and matrix addition operations.

Claims

1. A point cloud scene semantic instance joint segmentation method based on adaptive feature fusion, characterized in that, The specific implementation is as follows: Step 1, constructing LPR-AFFN network; Step 2, constructing local polar representation module LPR and performing LPR operation; Step 3, performing coordinate conversion, calculating neighborhood centroid point matrix, and updating polar angle; The specific process of step 3 is as follows: Step 3.1, for any point Its coordinates are ( , , ),use K The nearest neighbor algorithm collects its k Neighboring points Its coordinates are ( , , ),point and The relative coordinates are represented as ,in any neighboring point Time The distance is denoted as , Point and points The angle between the corresponding ray and the reference direction, Let the argument of the complex number form of any point in polar coordinates be denoted as Then the polar coordinates of the nearest neighboring points are expressed as . ; (12) (13) (14) Step 3.2, for any point , by the k neighbor points of the point, a tensor sum operation and a square root operation are performed on the feature matrix to obtain a neighborhood centroid point feature matrix ; = (15) Step 3.3, coordinate conversion feature matrix and neighborhood centroid feature matrix , after two matrix subtraction operations, splicing feature matrix, updating polar angle and According to the Z-axis rotation invariance, and will not change; = - (16) (17) (18) Step 3.4, performing tensor mean operation on the new feature matrix resulting from the two concatenations ;​ (19) Step 3.5, new feature matrix After dimension reduction, the feature matrix is obtained ; Step 3.6, splicing the original point cloud , polar coordinate conversion information and geometric distance information, and outputting a feature matrix with a size of ; (20)； Step 4, constructing feature representation module GPM, and defining local centroid point, finding neighborhood points of centroid point to construct local group; In step 4, the feature representation module GPM is composed of convolution layer, ReLu activation function, multi-layer perception MLP, and update combination local feature gate operation; The specific process is as follows: Step 4.1, according to the farthest sampling point algorithm, select a point A as the query point from the input point cloud, select a farthest point B from the remaining point cloud, at this time the query point is (A, B), then select the farthest distance point C from the remaining point cloud, select the minimum value of the distance between point A and point C and the distance between point B and point C as the distance d to the query point, divide the sampled point cloud into N query point cloud sets, select a group of points from the query point cloud set, define a local centroid point in the query point set, and find the neighborhood points of the centroid point to construct a local group using K nearest neighbor algorithm; Step 4.2, the grouped point cloud in step 4.1 is processed by After the operation projection, the feature matrix is obtained ; (21) Step 4.3, obtaining the feature matrix from the characteristic matrix performing matrix dimensionality reduction and one-dimensional convolution operation to obtain the feature matrix ; (22) Step 4.4, obtaining the feature matrix Two one-dimensional convolution operations are performed respectively to obtain the same type of feature matrix and ; (23) (24) Step 4.

5. Transpose the feature matrix Step 4.

6. Perform a transpose operation on the feature matrix ; (25) Step 4.6, matrix addition to feature matrix and ;​ (26) Step 4.7, feature matrix After ReLu function and Softmax operation, the feature matrix is obtained ; (27) Step 4.8, Feature Matrix and Do matrix multiplication to get the feature matrix ; (28) Step 4.9, feature matrix After the activation function and the update combination local feature gate operation, the feature matrix is obtained ; (29) Step 4.10, feature matrix After matrix dimensionality reduction and updating combination local feature gate operation, the feature matrix is obtained ; (30) Step 4.11, feature matrix After activation function, matrix dimension operation and feature matrix Matrix multiplication operation is performed to obtain the feature matrix ; (31) Step 4.12, matrix dimensionality reduction is performed on the feature matrix and respectively, and matrix multiplication is performed to obtain a new feature matrix ; (32) Step 4.13, for the feature matrix and Perform matrix concatenation to obtain a matrix of size . Feature matrix ; (33)； Step 5, performing preliminary feature extraction on the input point cloud; Step 6, constructing feature extraction module PCFE and performing PCFE operation; Step 7, constructing adaptive feature fusion module AFF, and using AFF to build global perception ability of point cloud; In step 7, the adaptive feature fusion module AFF is composed of convolution layer, Softmax, matrix point multiplication, and matrix addition operation; The specific process is as follows: Step 7.1, in the semantic segmentation task, the adaptive feature fusion module (AFF) first inputs a value of [value missing] to the feature extraction module (PCFE). semantic feature matrix Expand the dimensions, then perform a two-dimensional convolution operation to output the feature matrix. ; (40) Step 7.2, to normalize the feature matrix output by the Softmax operation ; (41) Step 7.3, Eigenmatrix With Do a matrix multiplication and output the Eigenmatrix ; (42) Step 7.4, feature matrix The size of the generated semantic feature matrix after dimension reduction is The semantic feature matrix After one-dimensional convolution operation in the full connection layer, the size of the semantic prediction is , which is used to predict the final semantic category, wherein is the number of semantic prediction categories,​​ (43) Step 7.5, for instance segmentation task, the same size as instance feature matrix is extended dimensionally, and the extended dimensionally feature matrix is generated by two-dimensional convolution operation ; (44) Step 7.

6. Perform a normalization Softmax operation to generate a feature matrix ;​ (45) Step 7.7, feature matrix With Do a matrix multiplication and output the feature matrix ; (46) Step 7.8, for example feature matrix and semantic feature matrix do matrix addition, output feature matrix ; (47) Step 7.

9. Feature matrix After dimensionality reduction operation, the size of the instance feature matrix is generated After one-dimensional convolution operation through the full connection layer, the size of the instance embedding is output, where is the dimension of the instance feature embedding;​​ (48)。 2. The point cloud scene semantic instance joint segmentation method based on adaptive feature fusion according to claim 1, characterized in that, In step 1, the LPR-AFFN network is composed of an encoding layer and a decoding layer and a skip connection; the encoding stage is composed of one shared encoder, and the one shared encoder includes four layers of encoding, and the feature matrix sizes of each layer output are 512 respectively; 、 、 、 512; the decoding stage is composed of two parallel decoders, and the two parallel decoders each include four layers of decoding, and the feature matrix sizes of each layer output are 、 、 、 ; wherein , , , , are the number of sampling points for each layer, respectively; Both the encoding layer and the decoding layer use the same structure as PointNet++.

3. The point cloud scene semantic instance joint segmentation method based on adaptive feature fusion according to claim 1, characterized in that, In step 2, the local polar representation module LPR is composed of K nearest neighbor algorithm, tensor maximum value, neighborhood point, matrix subtraction, tensor tiling, matrix splicing, tensor mean value, and dimension reduction operation; The specific process is as follows: Step 2.1, to the original point cloud Adopting K The most adjacent algorithm collects k The adjacent points, k Set to 20, get the adjacent point feature matrix ; (1) Step 2.2, neighborhood point feature matrix Performing neighborhood point operations to obtain a feature matrix ; (2) Step 2.3, Eigenmatrix Performing the tensor max operation, resulting in the eigenmatrix ; (3) Step 2.4, original point cloud and feature matrix , performing a matrix subtraction operation, a tensor sum operation, and a square root operation to obtain a feature matrix ; = (4) Step 2.5, original point cloud One-dimensional tensor tiling is performed to obtain a feature matrix ; (5) Step 2.

6. Perform matrix concatenation operation on the feature matrix and the feature matrix to obtain the feature matrix ; (6) Step 2.

7. Perform matrix concatenation operation on the feature matrix and the feature matrix to obtain the feature matrix ; (7) Step 2.

8. Obtaining the feature matrix by performing two tensor max operations ; (8) Step 2.

9. Perform the power exponent operation on the feature matrix to obtain the feature matrix ; (9) Step 2.10, Neighboring point feature matrix and the original point cloud, get the matrix feature ; = - (10) Step 2.11, feature matrix A one-dimensional tensor tiling operation is performed to obtain a feature matrix ; (11)。 4. The point cloud scene semantic instance joint segmentation method based on adaptive feature fusion according to claim 1, characterized in that, The specific process of step 5 is as follows: Step 5.1, the encoding mode adopted by the last three layers of the encoding layer is the same as PointNet++, each encoding layer will complete the extraction of the input features once, and the output dimensions of the local features extracted by the four encoding layers are ; Step 5.2, the decoding layer adopts the same decoding mode as pointNet++, that is, the features of the encoding layer are up-sampled, the features combined from the interpolation operation and the jump connection are updated, the feature information generated by the encoding layer is preserved as much as possible, the encoding layer and the decoding layer correspond to each other, and the feature output dimensions of the features combined after up-sampling and updating are , that is, the feature matrix of 128 is obtained. ​ 5. The point cloud scene semantic instance joint segmentation method based on adaptive feature fusion according to claim 1, characterized in that, In step 6, the feature extraction module PCFE is composed of convolution layer, up-sampling operation, and matrix splicing operation; The specific process is as follows: Step 6.1, the two feature matrices obtained in step 5.2 are denoted as 128, respectively, and are denoted as and Taking semantic segmentation as an example, first, up-sample to obtain a new feature matrix ; (34) Step 6.2, concatenate and to obtain a feature matrix of size ; and ; (35) Step 6.

3. One-dimensional convolution operation is performed on the feature matrix with an output size of the feature matrix ; (36) Step 6.4 There are two parallel branches in the whole decoder, and for the instance segmentation task, we use and to represent the feature matrix of 128; Step 6.5, Up-sampling is performed to obtain a new feature matrix ; (37) Step 6.6, concatenate and to obtain a feature matrix of size ;​ (38) Step 6.

7. Perform one-dimensional convolution operation on the feature matrix with output size of the feature matrix ; (39)。

Images