A 3D object recognition method based on multi-view grouping perception attention

Abstract

The present invention discloses a three-dimensional object recognition method based on multi-view grouping perception attention. First, multiple two-dimensional views of different three-dimensional object models at different shooting angles are obtained, and multiple views are processed by using a convolutional neural network to extract view feature vectors, and the view feature vectors are grouped according to the similarity score, and then the weight coefficients of each view feature vector in its group are calculated respectively, and then all view feature vectors in a group are fused into a group-level fused view feature vector by weighted summation; finally, the correlation between the group-level fused view feature vectors is mined through a recurrent neural network, and the group-level associated view feature vectors with different contributions are adaptively fused into a three-dimensional object descriptor for object classification through multi-perception group attention. The present invention combines the advantages of convolutional neural networks, recurrent neural networks and attention algorithms, and significantly improves the accuracy and efficiency of three-dimensional object recognition.

Description

Technical Field

[0001] The present invention relates to three-dimensional object recognition, and in particular to a three-dimensional object recognition method based on multi-view grouping perception attention. Background Art

[0002] Information is growing explosively, but people are becoming more and more demanding in their daily lives. There is a large amount of data in real life, such as audio and video data, a large amount of text data information, biomedical image data, large-scale map satellite remote sensing image data, face image data, traffic sign data, etc. Traditional image recognition mainly includes the following four processes: image acquisition, image preprocessing, feature extraction, and image recognition. Among them, image acquisition is to obtain image data sets; image preprocessing is to process the image data format and normalize it according to requirements; feature extraction is to describe and extract the features of the image; image recognition is to match and identify the target according to the features of the image in the trained classifier.

[0003] Three-dimensional object recognition is a hot topic in the current computer vision field, but existing image-related deep learning methods cannot accomplish this task well. On the one hand, three-dimensional data is organized in various forms, including view-based and model-based, and models can be represented by voxels, point clouds, slices, etc. Three-dimensional data lacks large-scale data sets similar to two-dimensional data, which places strict requirements on the learning and induction capabilities of the network. The view-based method uses the two-dimensional projection of the three-dimensional object as a data set and converts the three-dimensional object recognition into two-dimensional space, which produces better results. However, the simple use of convolutional neural networks combined with maximum pooling operations for three-dimensional object recognition is inefficient and will lose a lot of key information, and simply using the maximum pooling operation to fuse view features cannot discover the correlation between views. Therefore, how to improve the accuracy and efficiency of three-dimensional object recognition methods is the key to improving the performance of three-dimensional object recognition. Summary of the invention

[0004] In view of the deficiencies in the prior art, the technical problem that the present invention intends to solve is to provide a three-dimensional object recognition method based on multi-view grouping perception attention.

[0005] The technical solution of the present invention to solve the technical problem is to provide a three-dimensional object recognition method based on multi-view grouping perception attention, characterized in that the method comprises the following steps:

[0006] Step 1: Obtain a three-dimensional object data set; the data set includes several categories of three-dimensional object models, and each category includes several three-dimensional object models;

[0007] Step 2, obtaining a two-dimensional view of the three-dimensional object model: N virtual cameras are evenly arranged along the horizontal plane around the three-dimensional object model, each three-dimensional object model in the data set is photographed, and the three-dimensional object model is placed at the center of its view, and N two-dimensional views of each three-dimensional object model are obtained;

[0008] Step 3: Extract 2D view features using a convolutional neural network: For a certain 3D object model, the N 2D views obtained in step 2 are sequentially input into a convolutional neural network for view feature extraction to obtain N view feature vectors of the 3D object model.

[0009] Step 4, view grouping: Calculate the similarity scores between the N view feature vectors extracted in step 3, and then group the N view feature vectors according to the similarity scores, grouping the view feature vectors with high similarity into one group, and a total of M groups;

[0010] Step 5, view group fusion: For each group, the weight coefficient of each view feature vector in its group is calculated respectively, and then all view feature vectors in a group are fused into a group-level fused view feature vector by weighted summation, and a total of M group-level fused view feature vectors are obtained;

[0011] Step 6: Input the M group-level fusion view feature vectors obtained in step 5 into the Bi-LSTM recurrent neural network. The Bi-LSTM recurrent neural network regards the group-level fusion view feature vectors as a time series, and obtains the correlation between the M group-level fusion view feature vectors; then, based on the correlation, outputs M group-level associated view feature vectors;

[0012] Step 7: Aggregate the M group-level associated view feature vectors obtained in step 6 into a 3D object descriptor using multi-sensory group attention;

[0013] Step 8, 3D object recognition: The 3D object descriptor obtained in step 7 is sent to a fully connected layer whose number of neurons is the same as the number of types of 3D object models in the data set, and then a set of binary data is generated; the position with the number 1 in the set of binary data represents the recognition result of the 3D object model.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] 1. The present invention combines the advantages of convolutional neural networks, recurrent neural networks and attention algorithms, improves convolutional neural networks and recurrent neural networks, and adds multi-view group fusion and group attention mechanisms to them, which significantly improves the accuracy and efficiency of three-dimensional object recognition. First, multiple two-dimensional views of different three-dimensional object models at different shooting angles are obtained, and multiple views are processed by convolutional neural networks to extract view feature vectors, and then the view feature vectors are grouped according to the similarity score through a clustering algorithm, and then the weight coefficient of each view feature vector in its group is calculated respectively, and then all view feature vectors in a group are fused into a group-level fused view feature vector by weighted summation; there is a certain correlation between the fused view feature vectors, and finally the correlation between the group-level fused view feature vectors is mined through a recurrent neural network, and the group-level associated view feature vectors with different contributions are adaptively fused into a three-dimensional object descriptor for object classification through multi-perception group attention.

[0016] 2. The present invention adopts a convolutional neural network-recurrent neural network architecture. Compared with MVCNN and GVCNN, which only use convolutional neural networks, focusing on the correlation between multiple views to discover the contextual relationship of views is more conducive to improving the accuracy of 3D object recognition. While improving the convolutional neural network and recurrent neural network for better object recognition, multi-view grouping fusion and grouping attention mechanisms are added to highlight important views and key views and assign them higher weight ratios.

[0017] 3. The present invention introduces a group-aware group attention mechanism to replace the maximum pooling operation, which reduces the loss of view information and improves the efficiency of fusion view features. The group-aware group attention enables the view to adaptively fuse features and discover view correlation, thereby improving the accuracy of 3D object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the overall flow chart of the present invention;

[0019] Figure 2 A virtual camera arrangement diagram for obtaining a two-dimensional view of a three-dimensional object model according to the present invention;

[0020] Figure 3 A set of 12 two-dimensional views obtained for a three-dimensional object model of the present invention;

[0021] Figure 4 A network hierarchy diagram of the present invention;

[0022] Figure 5 The weight coefficient of the visualization view grouping and the total score distribution diagram of the attention perception area of ​​the multi-perception group of the present invention;

[0023] Figure 6This is a comparison chart of the recognition accuracy of the present invention and other algorithms on the ModelNet40 dataset. DETAILED DESCRIPTION

[0024] The specific embodiments of the present invention are given below. The specific embodiments are only used to further illustrate the present invention in detail and do not limit the protection scope of the claims of this application.

[0025] The present invention provides a three-dimensional object recognition method based on multi-view grouping perception attention (hereinafter referred to as method), characterized in that the method comprises the following steps:

[0026] Step 1: Obtain a three-dimensional object data set; the data set includes several categories of three-dimensional object models, and each category includes several three-dimensional object models;

[0027] Preferably, in step 1, the data set uses the ModelNet40 or ModelNet10 data set published by Princeton University; the ModelNet40 data set contains 40 general object types, and the ModelNet10 data set contains 10 general object types; this embodiment uses the ModelNet40 data set;

[0028] Step 2: Get a 2D view of the 3D object model. Figure 2 As shown in FIG. 1 , N virtual cameras are evenly arranged along the horizontal plane around the circumference of the 3D object model, and each 3D object model in the data set obtained in step 1 is photographed, and the 3D object model is placed at the center of its view by cropping, and N 2D views of each 3D object model are obtained (12 2D views in this embodiment are shown in FIG. 1 ). Figure 3 As shown), use it as the input of the convolutional neural network;

[0029] Preferably, step 2 specifically includes:

[0030] 1) The three-dimensional object model is placed upright on a horizontal plane according to a vertical line, and N virtual cameras are evenly arranged along the horizontal plane in the circumference of the three-dimensional object model according to a manually set horizontal interval to obtain N views; when shooting, the height of the virtual camera is higher than the center of the three-dimensional object model, and the angle between the line connecting the lens and the center of the three-dimensional object model and the horizontal plane is 30°, and the lens points to the center of the three-dimensional object model;

[0031] 2) For each of the N views, the 3D object model is placed at the center of the view by cropping. Then, in this embodiment, the view needs to be cropped to a size of 224×224 to obtain N 2D views.

[0032] Step 3: Extract 2D view features using a convolutional neural network: For a certain 3D object model, the N 2D views obtained in step 2 are sequentially input into a convolutional neural network for view feature extraction to obtain N view feature vectors of the 3D object model.

[0033] Preferably, in step 3, the convolutional neural network adopts an improved ResNet-18 network, and its specific structure is:

[0034] The first layer is a convolutional layer with a kernel size of 7×7 and a step size of 2. Relu is set as the activation function. The number of input channels is 3 and the input size is 224×224. The number of output channels is 64 and the output size is 112×112.

[0035] The second layer consists of a maximum pooling layer and two consecutive convolution blocks connected in sequence. The maximum pooling layer operation is performed first, and then the convolution operation is performed on the pooling result. The convolution kernel size of the maximum pooling layer is 3×3, and the step size is 2. The specifications of the two convolution blocks are the same, both including two convolution layers of the same size, the convolution kernel size is 3×3, the number of input channels is 64, the input size is 112×112, the number of output channels is 64, and the output size is 56×56.

[0036] The third layer consists of a global maximum pooling layer, a global average pooling layer and a convolution layer connected sequentially; the stride of the global maximum pooling layer and the global average pooling layer are both 1, and the number of output channels is 64; the convolution layer has a convolution kernel size of 7×7, a stride of 2, an input channel number of 64, an input size of 56×56, an output channel number of 64, and an output size of 56×56;

[0037] The fourth layer consists of two consecutive convolution blocks of the same size. Each convolution block includes two identical convolution layers, with a convolution kernel size of 3×3, 64 input channels, 56×56 input size, 128 output channels, and 28×28 output size.

[0038] The fifth layer consists of two consecutive convolution blocks of the same size. Each convolution block includes two identical convolution layers, with a convolution kernel size of 3×3, 128 input channels, 28×28 input size, 256 output channels, and 14×14 output size.

[0039] The sixth layer consists of two consecutive convolution blocks of the same size. Each convolution block includes two identical convolution layers, with a convolution kernel size of 3×3, 256 input channels, 14×14 input size, 512 output channels, and 7×7 output size.

[0040] The seventh layer is the BN layer, which performs batch normalization on the output of the sixth layer. The number of input channels is 512, the input size is 7×7, the number of output channels and the output size remain unchanged, but the numerical range of the feature is fixed between (0,1). The BN layer processes data to prevent overfitting.

[0041] Step 4: View grouping: Calculate the similarity scores between the N view feature vectors extracted in step 3 according to the Euclidean distance, and then group the N view feature vectors according to the similarity scores, grouping the view feature vectors with high similarity into one group, and divide them into M groups of indefinite length (M <N);

[0042] Preferably, in step 4, a similarity score matrix A(N, N) is constructed and calculated based on the N view feature vectors extracted in step 3; two matrices T(N, N) and G(N, N) are initialized at the same time to calculate the information of view grouping, and the matrices T(N, N) and G(N, N) are updated in the continuous iteration process; the attraction matrix T(N, N) is used to describe the degree to which the current view feature vector selects other view feature vectors as its grouping center, and the higher the calculated score, the more suitable it is to become a grouping center of the view feature vector; the attribution matrix G(N, N) is a reverse information propagation, indicating that the grouping center of the current view feature vector observes whether the other view feature vectors are suitable as its group members, and the higher the view feature vector matching score, the more suitable the view feature vector is to become this group;

[0043] The iterative process is to update the matrices T(N, N) and G(N, N). The initial values ​​of these two matrices are both 0. After each update, the value t in the attraction matrix T(N, N) is ij and the value g in the degree matrix G(N,N) ij The sum determines the group center feature vector in the current view feature vector; if i=j, it means that the view feature vector i is the group center feature vector of its group; if i≠j, it means that the view feature vector i belongs to the group of the view feature vector j; the condition for stopping the iteration is that the classification of the N view feature vectors no longer changes after multiple consecutive iterations or exceeds the maximum number of iterations.

[0044] Preferably, in step 4, the value a in the similarity score matrix A(N, N) ij It is the similarity score between view feature vector i and view feature vector j, which is calculated by the Euclidean distance of the feature sequence between view feature vector i and view feature vector j; the smaller the Euclidean distance value, the larger the similarity score and the higher the similarity.

[0045] Preferably, in step 4, the value t in the attraction matrix T(N, N) ijUsed to describe the degree to which view feature vector j is suitable as the grouping center of view feature vector i (that is, the attraction of view feature vector j to view feature vector i):

[0046]

[0047] In formula (1), a ij is the similarity score between view feature vector i and view feature vector j, g ij′ represents the degree of attribution of other view feature vectors to view feature vector i except view feature vector j, which is initially 0; a ij′ It represents the similarity of other view feature vectors to view feature vector i except view feature vector j.

[0048] Preferably, in step 4, the value g in the attribution matrix G(N,N) ij It is used to describe the suitability of view feature vector i to select view feature vector j as its center point (i.e., the degree of attribution of view feature vector i to view feature vector j):

[0049]

[0050] In formula (2), t jj Indicates the suitability of view feature vector j as the grouping center; a i′j represents the attractiveness of view feature vector j as the grouping center of other view feature vectors except view feature vector i; t i′j represents the similarity score of view feature vector j as the group center of view feature vectors other than view feature vector i.

[0051] Step 5, view group fusion (multi-layer perceptron): For each group, the weight coefficient (i.e. contribution ratio) of each view feature vector in its group is calculated respectively, and then all view feature vectors in a group are fused into a group-level fused view feature vector by weighted summation. For M groups, a total of M group-level fused view feature vectors are obtained;

[0052] Preferably, step 5 is specifically as follows: input the N view feature vectors into the first fully connected layer according to the groups, and the number of input channels and the number of output channels are both 512; then input the second fully connected layer after passing through the Relu function, and the number of channels of the second fully connected layer is set the same as the number of channels of the first fully connected layer, but the parameters are different; finally, input the Softmax function to generate a weight coefficient between (0,1) for each view feature vector, and then multiply the weight coefficient by the corresponding view feature vector, and then add and sum all the multiplication results according to the groups, and obtain a group-level fused view feature vector for each group, and obtain M group-level fused view feature vectors for M groups; wherein the two fully connected layers are of the same size and have 512 neurons; and a Relu function is added between the two fully connected layers to avoid gradient disappearance.

[0053] Step 6: Input the M group-level fusion view feature vectors obtained in step 5 into the Bi-LSTM recurrent neural network. The Bi-LSTM recurrent neural network regards the group-level fusion view feature vectors as a time series, and obtains the correlation between the M group-level fusion view feature vectors; then, based on the correlation, outputs M group-level associated view feature vectors;

[0054] Preferably, in step 6, the Bi-LSTM recurrent neural network is a double-layer LSTM network structure, consisting of a forward LSTM and a reverse LSTM; both the forward LSTM and the reverse LSTM are composed of a forget gate, an input gate, an output gate and a memory cell;

[0055] The forget gate consists of a fully connected layer and a sigmoid function; the input gate consists of two parts, the first part consists of a fully connected layer and a sigmoid function, and the second part consists of a fully connected layer and a tanh function; the output gate consists of a fully connected layer and a sigmoid function; the parameters of the fully connected layers of the forget gate, input gate, and output gate are different; then the specific operations of step 6 are as follows:

[0056] (6.1) Input the M group-level fusion view feature vectors into the forward LSTM and the backward LSTM respectively. The input processing of the forget gate, input gate and output gate adopts the same operation: the group-level fusion view feature vector of the current i-th group is concatenated with the output of the i-1-th group to obtain the output of the i-th group;

[0057] (6.2) Input the output of the i-th group into the forget gate, generate a coefficient between (0,1) through the fully connected layer and the sigmoid function, and then multiply the coefficient with the group-level fusion view feature vector of the i-th group to obtain a memory cell;

[0058] (6.3) Input the output of the i-th group into the two parts of the input gate at the same time, then multiply the output results of the two parts, and then add them to the memory cell obtained by the forget gate to obtain the memory cell update result;

[0059] (6.4) Input the output of the i-th group into the output gate, generate a coefficient between (0,1) through the fully connected layer and the sigmoid function, and then multiply the coefficient with the memory cell update result obtained by the input gate to obtain a group-level correlation view feature vector component of the forward LSTM and the reverse LSTM respectively;

[0060] (6.5) The two group-level association view feature vector components are concatenated to obtain M group-level association view feature vectors output by the Bi-LSTM recurrent neural network.

[0061] Step 7: Aggregate the M group-level associated view feature vectors obtained in step 6 into a 3D object descriptor using multi-sensory group attention;

[0062] Preferably, step 7 specifically includes:

[0063] (7.1) concatenate the M group-level correlation view feature vectors obtained in step 6 to obtain an overall group-level correlation view feature vector; then pass the overall group-level correlation view feature vectors through a fully connected layer respectively and output them as Q and V respectively;

[0064] The M group-level fusion view feature vectors obtained in step 5 are concatenated and passed through a fully connected layer to generate K;

[0065] (7.2) Split the M group-level associated view feature vectors in Q and the M group-level fused view feature vectors in K into t blocks of perceptual regions, and obtain a total of M×t blocks of perceptual regions respectively; then for each group, multiply the t blocks of perceptual regions in Q with the t blocks of perceptual regions in K, and then add up all the multiplication results to obtain a total perceptual region score for each group; a total of M perceptual region scores are obtained; t represents a hyperparameter;

[0066] (7.3) Multiply the M group-level associated view feature vectors in V with the total scores of the M perception regions, and then add up all the multiplication results to obtain a three-dimensional object descriptor.

[0067] In order to understand the weight coefficient of each view feature vector in each group in step 5 and the total score of the perceptual area of ​​M group-level associated view feature vectors in the multi-perceptual group attention in step 7, the present invention visualizes the specific values ​​obtained by the three 3D object models of guitar, car and bed in the ModelNet40 dataset in steps 5 and 7, as shown in FIG. Figure 5 shown. Figure 5The left side of the figure is a 3D object model example of a guitar, a car, and a bed, and the right side is 12 2D views of the 3D object model obtained by step 2. Take the 3D object model of a small guitar as an example: after the view grouping in step 4, the 12 guitar view feature vectors are divided into 4 groups, and then the weight coefficients of each view feature vector in the 4 groups are calculated by step 5 (the weight coefficients of the three view feature vectors of group 1 are 0.46, 0.35, and 0.36 respectively, the weight coefficients of the three view feature vectors of group 2 are 0.28, 0.34, and 0.25 respectively, the weight coefficients of the three view feature vectors of group 3 are 0.16, 0.11, and 0.18 respectively, and the weight coefficients of the three view feature vectors of group 4 are 0.14, 0.12, and 0.15 respectively). Finally, step 7 calculates the total perceptual area scores for the 4 groups (group 1 is 0.38, group 2 is 0.32, group 3 is 0.15, and group 4 is 0.15).

[0068] Step 8: Send the three-dimensional object descriptor obtained in step 7 to a fully connected layer, the number of neurons of which is the same as the number of types of three-dimensional object models in the data set (40 in this embodiment), and then generate a set of binary data (in this embodiment, the set of binary data is 40 numbers, [0,0,0,0, ..., 1,0]); the position where the number is 1 in the set of binary data represents the recognition result of the three-dimensional object model, that is, the category of the three-dimensional object model;

[0069] Step 9: The identification of other three-dimensional object models is similar, that is, steps 3 to 8 are repeated to obtain the categories of all three-dimensional object models.

[0070] In order to test the performance of the present invention and verify the effectiveness of the present invention, on the three-dimensional model dataset ModelNet40, the MVCNN algorithm (adopted from the document "Su H, Maji S, Kalogerakis E, et al. Multi-view convolutional neural networks for 3D shape recognition / / IEEE International Conference on Computer Vision, Santiago, 2015: 945-953."), the GVCNN algorithm (adopted from the document "Feng Y, Zhang Z, Zhao X, et al. GVCNN: Group-view convolutional neural networks for 3D shape recognition [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 264-272") and the MHBN algorithm (adopted from the document "Yu T, Meng J, Yuan J. Multi-view harmonized bilinear network for 3d object recognition [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 264-272") were respectively adopted. Recognition.2018:186-194》), MV-CRNN algorithm (using the literature 《Ma C, Guo Y, Yang J, et al. Learning multi-view representation with LSTM for 3-D shape recognition and retrieval[J]. IEEE Transactions on Multimedia, 2018, 21(5):1169-1182》) and the algorithm of the present invention for recognition. The recognition results are as follows: Figure 6 shown. Figure 6 In , the horizontal axis is the number of views used when training the network, and the vertical axis is the accuracy of 3D object recognition. Figure 6It can be seen that the accuracy of three-dimensional object recognition gradually increases with the increase in the number of two-dimensional views of the three-dimensional object model obtained in step 2; the accuracy of the method used in the present invention is higher than that of the other four algorithms when the number of two-dimensional views is 3, 6, 9, and 12, indicating that the present invention can significantly improve the accuracy of three-dimensional object recognition and has excellent performance.

[0071] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A three-dimensional object recognition method based on multi-view grouping perception attention, characterized in that: The method comprises the following steps: Step 1: Obtain a three-dimensional object data set; the data set includes several categories of three-dimensional object models, and each category includes several three-dimensional object models; Step 2, obtaining a two-dimensional view of the three-dimensional object model: N virtual cameras are evenly arranged along the horizontal plane around the three-dimensional object model, each three-dimensional object model in the data set is photographed, and the three-dimensional object model is placed at the center of its view, and N two-dimensional views of each three-dimensional object model are obtained; Step 3: Extract 2D view features using a convolutional neural network: For a certain 3D object model, the N 2D views obtained in step 2 are sequentially input into a convolutional neural network for view feature extraction to obtain N view feature vectors of the 3D object model. Step 4, view grouping: Calculate the similarity scores between the N view feature vectors extracted in step 3, and then group the N view feature vectors according to the similarity scores, grouping the view feature vectors with high similarity into one group, and a total of M groups; Step 5, view group fusion: For each group, the weight coefficient of each view feature vector in its group is calculated respectively, and then all view feature vectors in a group are fused into a group-level fused view feature vector by weighted summation, and a total of M group-level fused view feature vectors are obtained; Step 6: Input the M group-level fusion view feature vectors obtained in step 5 into the Bi-LSTM recurrent neural network. The Bi-LSTM recurrent neural network regards the group-level fusion view feature vectors as a time series, and obtains the correlation between the M group-level fusion view feature vectors; then, based on the correlation, outputs M group-level associated view feature vectors; Step 7: Aggregate the M group-level associated view feature vectors obtained in step 6 into a 3D object descriptor using multi-sensory group attention; Step 8, 3D object recognition: The 3D object descriptor obtained in step 7 is sent to a fully connected layer whose number of neurons is the same as the number of types of 3D object models in the data set, and then a set of binary data is generated; the position with the number 1 in the set of binary data represents the recognition result of the 3D object model.

2. The three-dimensional object recognition method based on multi-view grouping perception attention according to claim 1 is characterized in that: In step 1, the dataset uses the ModelNet40 or ModelNet10 dataset released by Princeton University; the ModelNet40 dataset contains 40 common object categories, and the ModelNet10 dataset contains 10 common object categories.

3. The three-dimensional object recognition method based on multi-view grouping perception attention according to claim 1 is characterized in that: Step 2 is specifically: 1) The three-dimensional object model is placed on a horizontal plane according to a vertical line, and N virtual cameras are evenly arranged along the horizontal plane in the circumference of the three-dimensional object model according to a manually set horizontal interval to obtain N views; when shooting, the height of the virtual camera is higher than the center of the three-dimensional object model, and the angle between the line connecting the lens and the center of the three-dimensional object model and the horizontal plane is 30°, and the lens points to the center of the three-dimensional object model; 2) For each of the N views, the 3D object model is placed at the center of the view by cropping to obtain N 2D views.

4. The three-dimensional object recognition method based on multi-view grouping perception attention according to claim 1 is characterized in that: In step 3, the convolutional neural network adopts an improved ResNet-18 network, and its specific structure is: The first layer is the convolutional layer, and Relu is set as the activation function; The second layer consists of a maximum pooling layer and two consecutive convolution blocks connected sequentially. The maximum pooling layer operation is performed first, and then the convolution operation is performed on the pooling result; The third layer consists of a global maximum pooling layer, a global average pooling layer, and a convolutional layer connected sequentially; The fourth layer consists of two consecutive convolutional blocks of the same size; each convolutional block includes two identical convolutional layers; The fifth layer consists of two consecutive convolutional blocks of the same size; each convolutional block includes two identical convolutional layers; The sixth layer consists of two consecutive convolutional blocks of the same size; each convolutional block includes two identical convolutional layers; The seventh layer is the BN layer, which performs batch normalization on the output of the sixth layer.

5. The three-dimensional object recognition method based on multi-view grouping perception attention according to claim 1 is characterized in that: In step 4, a similarity score matrix A(N, N) is constructed and calculated based on the N view feature vectors extracted in step 3; at the same time, the attraction matrix T(N, N) and the attribution matrix G(N, N) are initialized to calculate the information of view grouping, and the matrices T(N, N) and G(N, N) are updated in the continuous iteration process; The iterative process is to update the matrices T(N, N) and G(N, N). The initial values ​​of these two matrices are both 0. After each update, the value t in the attraction matrix T(N, N) is ij and the value g in the degree matrix G(N,N) ij The sum determines the group center feature vector in the current view feature vector; if i=j, it means that the view feature vector i is the group center feature vector of its group; if i≠j, it means that the view feature vector i belongs to the group of the view feature vector j; the condition for stopping the iteration is that the classification of the N view feature vectors no longer changes after multiple consecutive iterations or exceeds the maximum number of iterations.

6. The three-dimensional object recognition method based on multi-view grouping perception attention according to claim 5 is characterized in that: In step 4, the value a in the similarity score matrix A(N,N) ij It is the similarity score between view feature vector i and view feature vector j, which is calculated by the Euclidean distance of the feature sequence between view feature vector i and view feature vector j; the smaller the Euclidean distance value, the larger the similarity score and the higher the similarity.

7. The three-dimensional object recognition method based on multi-view grouping perception attention according to claim 5 is characterized in that: In step 4, the value t in the attraction matrix T(N, N) ij Used to describe the degree to which view feature vector j is suitable as the grouping center of view feature vector i: In formula (1), a ij is the similarity score between view feature vector i and view feature vector j, g ij′ represents the degree of attribution of other view feature vectors to view feature vector i except view feature vector j, which is initially 0; a ij′ represents the similarity of other view feature vectors to view feature vector i except view feature vector j; The value g in the attribute matrix G(N,N) ij Used to describe the suitability of view feature vector i to select view feature vector j as its center point: In formula (2), t jj Indicates the suitability of view feature vector j as the grouping center; a i′j represents the attractiveness of view feature vector j as the grouping center of other view feature vectors except view feature vector i; t i′j represents the similarity score of view feature vector j as the group center of view feature vectors other than view feature vector i.

8. The three-dimensional object recognition method based on multi-view grouping perception attention according to claim 1 is characterized in that: Step 5 is as follows: input the N view feature vectors into the first fully connected layer according to the groups, then input them into the second fully connected layer after passing through the Relu function, and finally input them into the Softmax function to generate a weight coefficient between (0,1) for each view feature vector, and then multiply the weight coefficient by the corresponding view feature vector, and then add and sum all the multiplication results according to the groups, and obtain a group-level fused view feature vector for each group, and obtain M group-level fused view feature vectors for M groups.

9. The three-dimensional object recognition method based on multi-view grouping perception attention according to claim 1 is characterized in that: In step 6, the Bi-LSTM recurrent neural network is a two-layer LSTM network structure, which is composed of a forward LSTM and a reverse LSTM; both the forward LSTM and the reverse LSTM are composed of a forget gate, an input gate, an output gate, and a memory cell; The forget gate consists of a fully connected layer and a sigmoid function; the input gate consists of two parts, the first part consists of a fully connected layer and a sigmoid function, and the second part consists of a fully connected layer and a tanh function; the output gate consists of a fully connected layer and a sigmoid function; then the specific operations of step 6 are as follows: (6.1) Input the M group-level fusion view feature vectors into the forward LSTM and the backward LSTM respectively. The forget gate, input gate and output gate all take the same operation: concatenate the group-level fusion view feature vector of the current i-th group with the output of the i-1-th group to obtain the output of the i-th group; (6.2) Input the output of the i-th group into the forget gate, generate a coefficient between (0,1) through the fully connected layer and the sigmoid function, and then multiply the coefficient with the group-level fusion view feature vector of the i-th group to obtain a memory cell; (6.3) Input the output of the i-th group into the two parts of the input gate at the same time, then multiply the output results of the two parts, and then add them to the memory cell obtained by the forget gate to obtain the memory cell update result; (6.4) Input the output of the i-th group into the output gate, generate a coefficient between (0,1) through the fully connected layer and the sigmoid function, and then multiply the coefficient with the memory cell update result obtained by the input gate to obtain a group-level correlation view feature vector component of the forward LSTM and the reverse LSTM respectively; (6.5) The two group-level correlation view feature vector components are concatenated to obtain M group-level correlation view feature vectors.

10. The three-dimensional object recognition method based on multi-view grouping perception attention according to claim 1 is characterized in that: Step 7 is as follows: (7.1) concatenate the M group-level correlation view feature vectors obtained in step 6 to obtain an overall group-level correlation view feature vector; then pass the overall group-level correlation view feature vectors through a fully connected layer respectively and output them as Q and V respectively; The M group-level fusion view feature vectors obtained in step 5 are concatenated and passed through a fully connected layer to generate K; (7.2) Split the M group-level associated view feature vectors in Q and the M group-level fused view feature vectors in K into t-block perception regions, and obtain a total of M×t-block perception regions respectively; For each group, multiply the t-block perception area in Q with the t-block perception area in K, and then add up all the multiplication results to get a total perception area score for each group; a total of M perception area scores are obtained; t represents a hyperparameter; (7.3) Multiply the M group-level associated view feature vectors in V with the total scores of the M perception regions, and then add up all the multiplication results to obtain a three-dimensional object descriptor.

Images