An application-oriented multi-view 3D object recognition method

Abstract

The present invention proposes an application-end oriented multi-view three-dimensional object recognition method, which relates to the technical field of three-dimensional object recognition. The method first groups multi-view features based on feature differences, groups features that may come from similar perspectives into one group and fuses them into multiple group features, converts multi-view data with differences into similar intermediate layer features, and then uses a complete view set to train a teacher model, and uses knowledge distillation to allow the teacher model trained with a complete multi-view data set to guide the student model training, so that the student model has the ability to adapt to situations with few views and uncertain perspectives. In actual application tasks, when any multi-view data is input, the student model can achieve good three-dimensional recognition effects, only requires view information, and has the characteristics of being lightweight. The present invention is conducive to solving the problem of three-dimensional recognition effects caused by a small number of views and insufficient information in practical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional object recognition, and more specifically, to an application-oriented multi-view Figure 3 dimensional object recognition method. Background Art

[0002] In recent years, with the rapid development of intelligent robotics, autonomous driving, virtual reality, medical imaging, and other fields, 3D object recognition has become a new research hotspot. In the era of deep learning, various deep neural networks have been widely applied to 3D object recognition. Among various methods, multi-view-based methods have garnered significant attention due to their readily available and easily processed data. After using large-scale datasets such as ImageNet for pre-training CNN models, multi-view-based 3D object recognition methods have achieved a leading position in recognition accuracy and have become the current mainstream approach.

[0003] MVCCN (Multi-view CNN) is a combination of multiple 2D projection features learned by convolutional neural network (CNN) in an end-to-end trainable manner. This method has become a milestone in 3D shape recognition and has achieved the current best performance. Since the birth of MVCNN (Multi-view CNN) method, multi-view Figure 3 There are many methods for three-dimensional recognition. This type of research mainly focuses on how to perform efficient feature fusion or reduce information redundancy to improve the accuracy of three-dimensional object recognition. However, an important factor affecting the effectiveness of three-dimensional recognition is often overlooked by researchers - the reliability of the dataset. At present, the acquisition of multi-view datasets is mainly based on known three-dimensional objects. Then, single views of the three-dimensional objects are rendered in sequence from multiple preset perspectives according to a certain rule. However, in real scenes, due to factors such as occlusion and uncertain view position information, the multi-view data obtained is often far from the ideal situation. On the application side, due to the limitations of equipment and specific scene application requirements, multi-view data often have problems such as a small number of views and uncertain view angles. These situations may seriously affect the three-dimensional recognition accuracy of objects.

[0004] Many methods have attempted to address these challenges. While they have achieved some success, they still have their own shortcomings. For example, MVCNN uses maximum pooling to fuse features from any number of views. For example, a prior art method for 3D model recognition based on visual saliency sharing proposes first obtaining a 3D model to be retrieved, then obtaining a 2D view sequence based on the 3D model, and finally obtaining a visual feature vector for the 2D view sequence. Next, the visual features are fed into the MVCNN branch and the visual saliency branch, where the complex features from the MVCNN branch are fused with the visual saliency features from the visual saliency branch to form a fused feature. Finally, the fused feature is used to retrieve or classify the 3D model to be retrieved, but this method suffers from poor performance due to the loss of a significant amount of information. Furthermore, the Rotationet method requires camera viewpoint information for 3D recognition, which is significantly out of touch with reality. Therefore, current 3D recognition methods still need to meet two requirements for practical application: 1. They must be able to achieve good results using any number of views, and 2. They must not require any information other than the view data. Both requirements must be met simultaneously on the application side. Summary of the Invention

[0005] In order to solve the problem of a small number of views of the object to be identified and uncertain input view angle in the actual application scenario of three-dimensional object recognition, the present invention proposes a multi-viewing algorithm for the application end. Figure 3 The 3D object recognition method improves the problem of poor machine learning training results caused by differences in actual scene data input when facing practical applications. It only requires view information and is lightweight.

[0006] In order to achieve the above technical effects, the technical solutions of the present invention are as follows:

[0007] An application-oriented multi-view Figure 3 The method for identifying a three-dimensional object comprises the following steps:

[0008] S1. For each 3D object, extract all view features from the multi-view dataset and select several features from the multi-view features as group representative features based on the feature difference distance metric;

[0009] S2. Divide the remaining features except the group representative feature into the group containing the group representative feature closest to the group representative feature, and perform feature fusion on all features within each group to obtain multiple group fusion features;

[0010] S3. Input the group fusion features into the graph convolutional network to perform global transfer of local information, obtain graph convolutional features with global information, and fuse all graph convolutional features into a three-dimensional feature descriptor;

[0011] S4. Construct a teacher model and a student model that have the functions of steps S1 to S3, use the complete multi-view dataset to train the teacher model, use the defective multi-view dataset and use the trained teacher model to guide the training of the student model, and obtain a trained student model;

[0012] S5. Input any multi-view data of the actual three-dimensional object to be recognized on the application side into the trained student model to obtain the three-dimensional recognition result.

[0013] This technical solution utilizes the feature differences of multi-view features to group them, thereby grouping features that may come from similar perspectives and fusing them into multiple group features. This converts the differing multi-view data into similar intermediate-layer features, improving the problem of poor machine learning training results caused by data input differences. Finally, a graph convolutional network is used to fuse the group features to form object descriptors for 3D recognition. To address the problem of poor 3D recognition results caused by a small number of views and insufficient information, this solution uses knowledge distillation to allow a teacher model trained on complete multi-view data to guide the training of a student model. The student model imitates real-world tasks and is trained using any multi-view data in the task. Ultimately, the student model can achieve good 3D recognition results when any multi-view data is input. The model only requires view information and is lightweight.

[0014] Preferably, in step S1, all views of all three-dimensional objects from all angles constitute a multi-view dataset, and all views of each three-dimensional object from all angles are input into a pre-trained feature extraction network to obtain all single-view image features of the multi-view dataset.

[0015] Preferably, for each 3D object, all its single-view image features are combined into a multi-view feature F, which is expressed as: F = {f1, f2, ..., f N}, where N represents the number of single-view image features of the three-dimensional object, f i Represents the i-th single-view image feature in the multi-view feature F, i = 1, 2, ..., N.

[0016] Preferably, in step S2, for each 3D object, a feature difference measure between the single view image features in its multi-view feature F is calculated: The calculation expression is:

[0017]

[0018] F G =argmax(max(V(F;θ i ))

[0019] Among them, θ i Represents the feature parameters of each single view image, Represents the feature difference measure, which is obtained by calculating the sum of the square differences of the corresponding features. V(·) represents the feature difference measure. Based on the recursive representative view extraction method, first randomly extract the feature f i , and then extract the feature f with the largest sum of feature difference metrics from the previously extracted features j Finally, repeat this step until M features F are extracted G ={f g1 ,f g2 ,...f gM}, as the representative feature of each group, M is the number of groups set, not greater than N,

[0020] Preferably, in step S3, the distance between the group representative feature and the remaining features is calculated, and the expression is:

[0021] d(f gu -f j )=||f gu -f j ||2

[0022] The remaining features are divided into the group with the smallest distance from the representative feature, which satisfies:

[0023] G l =argmin(min(d(F G ;f j )),l=0,1,2...M-1,j=0,1,2,..N-1

[0024] in, Finally, we get the feature group G l , G l M groups contain multiple features. All features within each group are subjected to maximum pooling operation to achieve local feature fusion and obtain multiple group fusion features. The expression is: Among them, maxpool represents the maximum pooling operation, N l Indicates the number of features contained in each group, G l,i Represents the i-th feature of the l-th, and the feature F obtained after fusion n ={f1,f2,...f M}Contains M fused group features.

[0025] Here, by grouping the multi-view features based on their feature differences, the drastically changing multi-view data can be converted into similar intermediate-layer features, thus avoiding poor training results due to data changes. This has a positive effect on extracting high-resolution three-dimensional object descriptors in practical applications.

[0026] Preferably, when calculating the distance between the group representative feature and the remaining features, if the distance to multiple group representative features is the same, the feature is simultaneously classified into the groups where the multiple group representative features are located.

[0027] Preferably, the process of inputting the group fusion features into the graph convolutional network for global transfer of local information is:

[0028] S41. Group feature F n The group fusion feature f in i As the nodes of the graph structure, the adjacency matrix S representing the neighbor relationship of the graph nodes is obtained through the intermediate layer containing multiple layers of MLP i,j :S i,j =φ(d ij θ s )

[0029] Among them, d ij =[f i ,f j ,f i -f j ,||f i -f j ||2]∈R 10 represents the spatial relationship between the fusion features of the two groups, It means that it contains multiple layers of MLP and performs vector fusion on the elements in the group features. s A representation parameter that indicates the correspondence between two sets of features;

[0030] S42. Use the KNN algorithm to determine the group features within the nearest neighbor range of each group feature, and only retain the related edges with the K nearest neighbor group features to obtain the sparse connection matrix A i,j :

[0031] A i,j =S i,j ·C{f ni ∈K(f nj )};

[0032] Among them, C(·) represents the nearest neighbor operation to determine whether a group feature belongs to another group feature, and multiplication represents the sparseness of the original adjacency matrix;

[0033] S43. Perform graph convolution on the graph structure to obtain graph convolution structure features

[0034]

[0035] Among them, Α l Represents the adjacency matrix of the l-th layer graph, F G l is the original group feature of the level graph, Wl is the learnable weight matrix of the layer graph, θ l is the parameter of the linear activation function, Ψ is a nonlinear transformation function, when the original group feature F is input G l , first pass the adjacency matrix Α l Propagate, and then update the group feature nodes by linear transformation of the learnable weight matrix, and obtain the local graph convolution feature of each level All of them need to be globally messaged:

[0036] Where i, j = 0, 1, 2, ... M-1, is the spatial relationship between two nodes, and σ represents the relationship function between them. The actual meaning is the set of node pair content messages output by the multi-layer MLP. Each layer of MLP contains multiple convolutional units and nonlinear activations.

[0037] S44. According to the newly obtained spatial relationship of all node pairs, it is fused into the local graph convolution feature to obtain a new local graph convolution feature with global information

[0038]

[0039] Among them, Ω is a single-layer MLP, which fuses features through batch normalization and outputs new local graph convolution features that are a fusion of global node pair information and original group features in the graph;

[0040] S45. Each level of graph structure obtains local graph convolution and performs global information transmission in the same way. Based on the feature difference distance metric, each level of graph structure gradually eliminates the group features with the smallest comprehensive distance metric. Finally, M graph convolution features are obtained. The M levels of local graph convolution features are fused together through the maximum pooling method to form a global descriptor F representing the 3D object GCN :

[0041]

[0042] Here, sparseening the adjacency matrix can improve the computational efficiency of the graph structure.

[0043] Preferably, in step S5, the constructed teacher model and student model both include an image feature extraction module, a feature grouping and fusion module, and a graph convolution module;

[0044] The image feature extraction module is used to extract all single-view image features of a multi-view data set of three-dimensional objects and combine them into multi-view features; the feature grouping and fusion module selects several features from the multi-view features as group representative features for each three-dimensional object based on the feature difference distance metric, and divides the remaining features except the group representative features into the group where the group representative features are closest to the group representative features, and performs feature fusion on all features within each group to obtain multiple group fusion features; the graph convolution module is used for the global transmission of local information, obtains graph convolution features with global information, and fuses all graph convolution features into a three-dimensional feature descriptor.

[0045] Preferably, the complete multi-view dataset in step S5 is a standard public dataset, and the defective multi-view dataset is a dataset obtained by reducing the number of views of the complete multi-view dataset and disrupting their order.

[0046] Preferably, the process of using the complete multi-view dataset to train the teacher model and the defective multi-view dataset to guide the training of the student model is as follows: extract the Logits layer of the teacher model and the student model respectively and use x i and y i The MSE mean square error is used to measure their feature differences, and the cross entropy function is used to represent the differences between the predicted results and the true labels in the teacher-student network:

[0047] Where N represents the total number of views, r represents the number of groups, and n i Indicates the number of view features for a group, represents the sample corrected variance, p and q represent the prediction result and the formal label respectively;

[0048]

[0049] The loss functions are added together to measure the similarity and prediction accuracy of the output layers of the teacher model and the student model, and are optimized so that the student model learns the generalization ability of the output layer of the teacher model. The expression is:

[0050]

[0051] By using the MSE distance between the teacher model and the student model corresponding to the intermediate layer features as part of the loss function, the loss function L is minimized. Feature Make the functions and structures of the middle layers of the two similar:

[0052]

[0053] in, and They are the corresponding intermediate layer features of a certain layer of graph structure in the teacher model and the student model respectively, n is the total number of BatchSize, M is the total number of levels of graph structure features, and the total loss function L required for knowledge distillation is formed by combining the two parts: L = L logits +L feature ;

[0054]

[0055] When the difference between the teacher model and the student model is too large, the amount of knowledge input to the teacher model is too large, which is not conducive to achieving good training results. The temperature hyperparameter T is introduced to adjust the size of the difference in the intermediate layer features and determine the amount of knowledge contained in the knowledge distillation. Setting its size balances the relationship between the difference between the teacher model and the student model and the amount of transferred knowledge, smoothes the training curve, and improves the training effect; setting the hyperparameter λ makes the logis layer loss and the hidden layer loss close in magnitude. By optimizing the total loss function, the loss function converges to obtain a well-trained student model.

[0056] Here, knowledge distillation is introduced into multi-view Figure 3 In the field of 3D recognition, the complete information of the teacher model after training is distilled into the student model, so that the student model can obtain 3D recognition effects close to those using a complete view set even when the number of views is small. A high-precision 3D recognition model can be obtained when any multi-view data is input. The model does not require any information other than the views and is lightweight.

[0057] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0058] The present invention proposes a multi-viewing Figure 3 3D object recognition method, which uses the feature differences of multi-view features to group them, groups features that may come from similar perspectives into one group and fuses them into multiple group features, and converts multi-view data with differences into similar intermediate-layer features. Based on this idea, a basic teacher model that can realize these functions and a student model with the same structure as the teacher model are constructed. In order to solve the problem of poor 3D recognition effect caused by a small number of views and insufficient information, the knowledge distillation method is used to let the teacher model trained with a complete multi-view dataset guide the student model training, and the student model imitates the real task. In actual application tasks, when any multi-view data is input, the student model can achieve good 3D recognition effect, only requires view information, and is lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The multi-viewing method proposed in the first embodiment of the present invention is Figure 3 Flowchart of the 3D object recognition method;

[0060] Figure 2 A structural diagram showing a teacher model or a student model proposed in Example 2 of the present invention;

[0061] Figure 3 A diagram showing the process of processing multi-view data by the teacher model and the student model proposed in Example 2 of the present invention;

[0062] Figure 4 A curve chart showing the comparison of the classification accuracy of the ModelNet40 dataset using the method of the present application proposed in Example 3 of the present invention and other methods under uncertain viewing angles is shown. DETAILED DESCRIPTION

[0063] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0064] In order to better illustrate this embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the actual size;

[0065] It is understandable to those skilled in the art that descriptions of certain well-known contents may be omitted in the drawings.

[0066] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0067] The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent;

[0068] Example 1

[0069] like Figure 1 As shown, this embodiment proposes a multi-viewing Figure 3 dimensional object recognition method, see Figure 1 , the method comprises the following steps:

[0070] S1. For each 3D object, extract all view features from the multi-view dataset and select several features from the multi-view features as group representative features based on the feature difference distance metric;

[0071] S2. Divide the remaining features except the group representative feature into the group containing the group representative feature closest to the group representative feature, and perform feature fusion on all features within each group to obtain multiple group fusion features;

[0072] S3. Input the group fusion features into the graph convolutional network to perform global transfer of local information, obtain graph convolutional features with global information, and fuse all graph convolutional features into a three-dimensional feature descriptor;

[0073] S4. Construct a teacher model and a student model that have the functions of steps S1 to S3, use the complete multi-view dataset to train the teacher model, use the defective multi-view dataset and use the trained teacher model to guide the training of the student model, and obtain a trained student model;

[0074] S5. Input any multi-view data of the actual three-dimensional object to be recognized on the application side into the trained student model to obtain the three-dimensional recognition result.

[0075] The method proposed in this embodiment is to group features using the feature differences of multi-view features, thereby grouping features that may come from similar perspectives into a group and fusing them into multiple group features. The multi-view data with differences is converted into similar intermediate-layer features. Finally, the group features are fused through a graph convolutional network to form an object descriptor for 3D recognition. A teacher model that can implement the above functional ideas and a student model with the same structure as the teacher model are constructed. Then, to solve the problem of poor 3D recognition caused by a small number of views and insufficient information, the teacher model is trained using a complete multi-view dataset using a knowledge distillation method. The teacher model is then used to guide the training of the student model, which imitates real tasks and is trained using any multi-view data in the task. Ultimately, when any multi-view data is input, the student model can achieve good 3D recognition results.

[0076] In step S1, all views of all three-dimensional objects from all angles constitute a multi-view dataset, and all views of each three-dimensional object from all angles are input into a pre-trained feature extraction network to obtain all single-view image features of the multi-view dataset. In this embodiment, the pre-trained feature extraction network can be ResNet-18, including but not limited to this network.

[0077] For each 3D object, all its single-view image features are combined into a multi-view feature F, which is expressed as: F = {f1,f2,...,f N}, where N represents the number of single-view image features of the three-dimensional object, f i Represents the i-th single-view image feature in the multi-view feature F, i = 1, 2, ..., N.

[0078] In step S2, for each 3D object, the feature difference measure between the single view image features in its multi-view feature F is calculated. The calculation expression is:

[0079]

[0080] F G =argmax(max(V(F;θ i ))

[0081] Among them, θi Represents the feature parameters of each single view image, Represents the feature difference measure, which is obtained by calculating the sum of the square differences of the corresponding features. V(·) represents the feature difference measure. Based on the recursive representative view extraction method, first randomly extract the feature f i , and then extract the feature f with the largest sum of feature difference metrics from the previously extracted features j Finally, repeat this step until M features F are extracted G ={f g1 ,f g2 ,...f gM}, as the representative feature of each group, M is the number of groups set, not greater than N,

[0082] In step S3, the distance between the group representative feature and the remaining features is calculated, and the expression is:

[0083] d(f gu -f j )=||f gu -f j ||2

[0084] The remaining features are divided into the group with the smallest distance from the representative feature, which satisfies:

[0085] G l =argmin(min(d(F G ;f j )),l=0,1,2...M-1,j=0,1,2,..N-1

[0086] in, Finally, we get the feature group G l , G l M groups of features contain multiple features. By calculating the distance between features, the view features F are divided into M groups. All features within each group are subjected to the maximum pooling operation to achieve local feature fusion and obtain multiple group fusion features. The expression is: Among them, maxpool represents the maximum pooling operation, N l Indicates the number of features contained in each group, G l,i Represents the i-th feature of the l-th, and the feature F obtained after fusion n ={f1,f2,...f MThe method consists of M fused group features. When calculating the distance between a group representative feature and the remaining features, if the distance to multiple group representative features is the same, the feature is assigned to the same group as multiple group representative features. This method utilizes the differences in multi-view features for grouping, converting drastically varying multi-view data into similar intermediate-level features. This avoids poor training results due to data variability and has a positive impact on extracting high-resolution 3D object descriptors in practical applications.

[0087] The process of inputting group fusion features into the graph convolutional network for global transmission of local information is as follows:

[0088] S41. Group feature F n The group fusion feature f in i As the nodes of the graph structure, the adjacency matrix S representing the neighbor relationship of the graph nodes is obtained through the intermediate layer containing multiple layers of MLP i,j :S i,j =φ(d ij θ s )

[0089] Among them, d ij =[f i ,f j ,f i -f j ,||f i -f j ||2]∈R 10 represents the spatial relationship between the fusion features of the two groups, It means that it contains multiple layers of MLP and performs vector fusion on the elements in the group features. s A representation parameter that indicates the correspondence between two sets of features;

[0090] S42. Use the KNN algorithm to determine the group features within the nearest neighbor range of each group feature, and only retain the related edges with the K nearest neighbor group features to obtain the sparse connection matrix A i,j :

[0091] A i,j =S i,j ·C{f ni ∈K(f nj )};

[0092] Among them, C(·) represents the nearest neighbor operation to determine whether a group feature belongs to another group feature, and multiplication represents the sparseness of the original adjacency matrix;

[0093] S43. Perform graph convolution on the graph structure to obtain graph convolution structure features

[0094]

[0095] Among them, Α l Represents the adjacency matrix of the l-th layer graph, F G l is the original group feature of the level graph, W l is the learnable weight matrix of the layer graph, θ l is the parameter of the linear activation function, Ψ is a nonlinear transformation function, when the original group feature F is input G l , first pass the adjacency matrix Α l Propagate, and then update the group feature nodes by linear transformation of the learnable weight matrix, and obtain the local graph convolution feature of each level All of them need to be globally messaged:

[0096] Where i, j = 0, 1, 2, ... M-1, is the spatial relationship between two nodes, and σ represents the relationship function between them. The actual meaning is the set of node pair content messages output by the multi-layer MLP. Each layer of MLP contains multiple convolutional units and nonlinear activations.

[0097] S44. According to the newly obtained spatial relationship of all node pairs, it is fused into the local graph convolution feature to obtain a new local graph convolution feature with global information

[0098]

[0099] Among them, Ω is a single-layer MLP, which fuses features through batch normalization and outputs new local graph convolution features that are a fusion of global node pair information and original group features in the graph;

[0100] S45. Each level of graph structure obtains local graph convolution and performs global information transmission in the same way. Based on the feature difference distance metric, each level of graph structure gradually eliminates the group features with the smallest comprehensive distance metric. Finally, M graph convolution features are obtained. The M levels of local graph convolution features are fused together through the maximum pooling method to form a global descriptor F representing the 3D object GCN : Sparseening the adjacency matrix can improve the computational efficiency of the graph structure. Here, the model uses the distance metric designed when selecting group representative features, and gradually removes the group features with the smallest comprehensive distance metric at each level, which can be called "minus-one sampling method" (M at the first level, M-1 at the second level, and so on). Through feature sampling and local graph convolution processing method, M graph convolution features can be obtained.

[0101] Example 2

[0102] In this embodiment, the structural diagram of the teacher model and the student model is as follows: Figure 2 As shown, see Figure 2 ,The constructed teacher model and student model both include image feature extraction module,,feature grouping and fusion module and graph convolution module;

[0103] Specifically, the image feature extraction module is used to extract all single-view image features of the multi-view dataset of three-dimensional objects and combine them into multi-view features; the feature grouping and fusion module selects several features from the multi-view features as group representative features for each three-dimensional object based on the feature difference distance metric, and divides the remaining features except the group representative features into the group where the group representative features closest to the group representative features are located, and performs feature fusion on all features within each group to obtain multiple group fusion features; the graph convolution module is used for the global transmission of local information to obtain graph convolution features with global information, and fuses all graph convolution features into a three-dimensional feature descriptor, that is, the teacher model and the student model are able to realize the functions of steps S1 to S4. The processing process can be seen in Figure 3 In addition, when training the teacher model and the student model, the complete multi-view dataset used by the teacher model is a standard public dataset, and the defective multi-view dataset is a dataset after the number of views of the complete multi-view dataset is reduced and the order is shuffled.

[0104] The process of using the complete multi-view dataset to train the teacher model and the defective multi-view dataset to guide the training of the student model is as follows: extract the Logits layer of the teacher model and the student model respectively with x i and y i The MSE mean square error is used to measure their feature differences, and the cross entropy function is used to represent the differences between the predicted results and the true labels in the teacher-student network:

[0105] Where N represents the total number of views, r represents the number of groups, and n i Indicates the number of view features for a group, represents the sample corrected variance, p and q represent the prediction result and the formal label respectively;

[0106]

[0107] The loss functions are added together to measure the similarity and prediction accuracy of the output layers of the teacher model and the student model, and are optimized so that the student model learns the generalization ability of the output layer of the teacher model. The expression is:

[0108]

[0109] By using the MSE distance between the teacher model and the student model corresponding to the intermediate layer features as part of the loss function, the loss function L is minimized. Feature Make the functions and structures of the middle layers of the two similar:

[0110]

[0111] in, and They are the corresponding intermediate layer features of a certain layer of graph structure in the teacher model and the student model respectively, n is the total number of BatchSize, M is the total number of levels of graph structure features, and the total loss function L required for knowledge distillation is formed by combining the two parts: L = L logits +L feature ;

[0112]

[0113] When the difference between the teacher model and the student model is too large, the amount of knowledge input to the teacher model is too large, which is not conducive to achieving good training results. The temperature hyperparameter T is introduced to adjust the size of the difference in the intermediate layer features and determine how much knowledge the knowledge distillation contains. Setting its size balances the relationship between the difference between the teacher model and the student model and the amount of transferred knowledge, smoothes the training curve, and improves the training effect. The hyperparameter λ is set to make the logis layer loss and the hidden layer loss close in magnitude. By optimizing the total loss function, the loss function converges and a well-trained student model is obtained. Figure 3 In the field of 3D recognition, the complete information of the teacher model after training is distilled into the student model, so that the student model can obtain 3D recognition effects close to those using a complete view set even when the number of views is small. A high-precision 3D recognition model can be obtained when any multi-view data is input. The model does not require any information other than the views and is lightweight.

[0114] Example 3

[0115] This embodiment uses the ModelNet40 and ModelNet10 datasets to evaluate the method proposed in the present invention, and the effect is further illustrated by the following simulation experiments.

[0116] The ModelNet40 and ModelNet10 datasets used in the experiment are multi-view datasets of three-dimensional objects. The ModelNet40 multi-view dataset has multi-view data of 12,311 three-dimensional objects (12 views or 20 views) from 40 categories. The segmentation of the ModelNet40 dataset follows the following operation: 9,843 objects in the dataset are divided into a training set and 2,468 objects are divided into a test set. When testing the ModelNet40 dataset, the three-dimensional recognition results of 20 views and 12 views are tested respectively. Compared with the ModelNet40 dataset, the ModelNet10 dataset is much smaller. The ModelNet10 multi-view dataset has multi-view data of 4,899 three-dimensional objects (12 views or 20 views) from 10 categories, of which 3,991 objects are used as training sets and 908 objects are used as test sets. The present invention also tests the three-dimensional recognition effects of 20 views and 12 views on it.

[0117] The comparison methods involved are: MVCNN, GVCNN, MHBN, MLVCNN, Rotationet, View-GCN, CAR-Net and other multi-view Figure 3 The main comparison metrics are classification and retrieval accuracy in 3D recognition. Classification accuracy is the ratio of the number of correctly predicted samples to the total number of samples. MAP ranks features by calculating the L2 distance between them, taking the 3D object with the smallest distance as the prediction result, and finally calculating the average retrieval accuracy. The comparison results are shown in Table 1.

[0118] Table 1

[0119]

[0120] As can be seen from Table 1, this method performs well in 3D recognition when full multi-view data is input. Table 2 shows the model size settings of this application, the MVCNN method, and View-GCN.

[0121] Table 2

[0122] method The method proposed by the present invention MVCNN View—GCN Model size 63.76MB 491.84MB 129.48MB

[0123] Assuming that the training images are numbered 1-20 in sequence, in the case of random input, the input multi-view data may be in the following form, combined with Figure 4 , Figure 4 The horizontal axis is the number of views, and the vertical axis represents the classification accuracy, for example: 8 views (the order and source of views may be: 13, 7, 2, 14, 3, 6, 7, 9). Figure 4 It can be seen that the model can still achieve good three-dimensional object recognition results in the case of disordered order and a small number of views, which verifies the effectiveness of the method proposed in this invention.

[0124] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A multi-view 3D object recognition method for an application end, characterized in that: The following steps are involved: S1. For each 3D object, extract all view features from the multi-view dataset and select several features from the multi-view features as group representative features based on the feature difference distance metric; S2. Divide the remaining features except the group representative feature into the group containing the group representative feature closest to the group representative feature, and perform feature fusion on all features within each group to obtain multiple group fusion features; S3. Input the group fusion features into the graph convolutional network to perform global transfer of local information, obtain graph convolutional features with global information, and fuse all graph convolutional features into a three-dimensional feature descriptor; S4. Construct a teacher model and a student model that have the functions of steps S1 to S3, use the complete multi-view dataset to train the teacher model, use the defective multi-view dataset and use the trained teacher model to guide the training of the student model, and obtain a trained student model; S5. Input any multi-view data of the actual three-dimensional object to be recognized on the application side into the trained student model to obtain the three-dimensional recognition result.

2. The application-oriented multi-view 3D object recognition method according to claim 1, characterized in that: In step S1, all views of all three-dimensional objects from all angles constitute a multi-view dataset, and all views of each three-dimensional object from all angles are input into a pre-trained feature extraction network to obtain all single-view image features of the multi-view dataset.

3. The application-oriented multi-view 3D object recognition method according to claim 2, characterized in that: For each 3D object, all its single-view image features are combined into a multi-view feature F, which is expressed as: F = {f1,f2,...,f N }, where N represents the number of single-view image features of the three-dimensional object, f i Represents the i-th single-view image feature in the multi-view feature F, i = 1, 2, ..., N.

4. The application-oriented multi-view 3D object recognition method according to claim 1, characterized in that: In step S2, for each 3D object, the feature difference measure between the single view image features in its multi-view feature F is calculated. The calculation expression is: F G =argmax(max(V(F;θ i )) Among them, θ i Represents the feature parameters of each single view image, Represents the feature difference measure, which is obtained by calculating the sum of the square differences of the corresponding features. V(·) represents the feature difference measure. Based on the recursive representative view extraction method, first randomly extract the feature f i , then extract the feature f with the largest sum of feature difference metrics from the previously extracted features j Finally, repeat this step until M features F are extracted G ={f g1 ,f g2 ,...f gM }, as the representative feature of each group, M is the number of groups set, not greater than N, 5. The application-oriented multi-view 3D object recognition method according to claim 4, characterized in that: In step S3, the distance between the group representative feature and the remaining features is calculated, and the expression is: d(f gu -f j )=||f gu -f j ||2 The remaining features are divided into the group with the smallest distance from the representative feature, which satisfies: G l =argmin(min(d(F G ;f j )),l=0,1,2...M-1,j=0,1,2,..N-1 in, Finally, we get the feature group G l , G l M groups contain multiple features. All features within each group are subjected to maximum pooling operation to achieve local feature fusion and obtain multiple group fusion features. The expression is: Among them, maxpool represents the maximum pooling operation, N l Indicates the number of features contained in each group, G l,i Represents the i-th feature of the l-th group, and the feature F obtained after fusion n ={f1,f2,...f M }Contains M fused group features.

6. The application-oriented multi-view 3D object recognition method according to claim 4, characterized in that: When calculating the distance between a group representative feature and other features, if the distance to multiple group representative features is the same, the feature is simultaneously classified into the groups where the multiple group representative features are located.

7. The application-oriented multi-view 3D object recognition method according to claim 5, characterized in that: The process of inputting group fusion features into the graph convolutional network for global transmission of local information is as follows: S41. Group feature F n The group fusion feature f in i As the nodes of the graph structure, the adjacency matrix S representing the neighbor relationship of the graph nodes is obtained through the intermediate layer containing multiple layers of MLP i,j :S i,j =φ(d ij θ s ) Among them, d ij =[f i ,f j ,f i -f j ,||f i -f j ||2]∈R 10 represents the spatial relationship between the fusion features of the two groups, It means that it contains multiple layers of MLP and performs vector fusion on the elements in the group features. s A representation parameter that indicates the correspondence between two sets of features; S42. Use the KNN algorithm to determine the group features within the nearest neighbor range of each group feature, and only retain the related edges with the K nearest neighbor group features to obtain the sparse connection matrix A i,j : A i,j =S i,j ·C{f ni ∈K(f nj )}; Among them, C(·) represents the nearest neighbor operation to determine whether a group feature belongs to another group feature, and multiplication represents the sparseness of the original adjacency matrix; S43. Perform graph convolution on the graph structure to obtain graph convolution structure features Among them, Α l Represents the adjacency matrix of the l-th layer graph, F G l is the original group feature of the level graph, W l is the learnable weight matrix of the layer graph, θ l is the parameter of the linear activation function, Ψ is a nonlinear transformation function, when the original group feature F is input G l , first pass the adjacency matrix Α l Propagate, and then update the group feature nodes by linear transformation of the learnable weight matrix, and obtain the local graph convolution feature of each level All of them need to be passed globally: Where i, j = 0, 1, 2, ... M-1, is the spatial relationship between two nodes, and σ represents the relationship function between them. The actual meaning is the set of node pair content messages output by the multi-layer MLP. Each layer of MLP contains multiple convolutional units and nonlinear activations. S44. According to the newly obtained spatial relationship of all node pairs, it is fused into the local graph convolution feature to obtain a new local graph convolution feature with global information Among them, Ω is a single-layer MLP, which fuses features through batch normalization and outputs new local graph convolution features that are a fusion of global node pair information and original group features in the graph; S45. Each level of graph structure obtains local graph convolution and performs global information transmission in the same way. Based on the feature difference distance metric, each level of graph structure gradually eliminates the group features with the smallest comprehensive distance metric. Finally, M graph convolution features are obtained. The M levels of local graph convolution features are fused together through the maximum pooling method to form a global descriptor F representing the 3D object GCN : l=0,1,...,M-1.

8. The application-oriented multi-view 3D object recognition method according to claim 1, characterized in that: In step S5, the constructed teacher model and student model both include an image feature extraction module, a feature grouping and fusion module, and a graph convolution module; The image feature extraction module is used to extract all single-view image features of the three-dimensional object multi-view dataset and combine them into multi-view features; For each three-dimensional object, the feature grouping and fusion module selects several features from the multi-view features as group representative features based on the feature difference distance measurement, and divides the remaining features except the group representative feature into the group containing the group representative feature closest to the group representative feature. All features within each group are fused to obtain multiple group fusion features; the graph convolution module is used for the global transmission of local information to obtain graph convolution features with global information, and all graph convolution features are fused into a three-dimensional feature descriptor.

9. The application-oriented multi-view 3D object recognition method according to claim 8, characterized in that: The complete multi-view dataset in step S5 is a standard public dataset, and the defective multi-view dataset is a dataset obtained by reducing the number of views of the complete multi-view dataset and disrupting their order.

10. The application-oriented multi-view 3D object recognition method according to claim 9, characterized in that: The process of using the complete multi-view dataset to train the teacher model and the defective multi-view dataset to guide the training of the student model is as follows: extract the Logits layer of the teacher model and the student model respectively with x i and y i Indicates that the difference in their features is measured using the mean square error (MSE), and the cross entropy function is used to represent the difference between the predicted results and the true labels in the teacher-student network: Where N represents the total number of views, r represents the number of groups, and n i Indicates the number of view features for a group, represents the sample corrected variance, p and q represent the prediction result and the formal label respectively; The loss functions are added together to measure the similarity and prediction accuracy of the output layers of the teacher model and the student model, and are optimized so that the student model learns the generalization ability of the output layer of the teacher model. The expression is: By using the MSE distance between the teacher model and the student model corresponding to the intermediate layer features as part of the loss function, the loss function L is minimized. Feature Make the functions and structures of the middle layers of the two similar: in, and They are the corresponding intermediate layer features of a certain layer of graph structure in the teacher model and the student model respectively, n is the total number of BatchSizes, M is the total number of levels of graph structure features, and the total loss function L required by the student model in knowledge distillation is formed by combining the two parts: L = L logits +L feature ; When the difference between the teacher model and the student model is too large, the amount of knowledge input to the teacher model is too large, which is not conducive to achieving good training results. The temperature hyperparameter T is introduced to adjust the size of the difference in the intermediate layer features and determine the amount of knowledge contained in the knowledge distillation. Setting its size balances the relationship between the difference between the teacher model and the student model and the amount of transferred knowledge, smoothes the training curve, and improves the training effect; setting the hyperparameter λ makes the logis layer loss and the hidden layer loss close in magnitude. By optimizing the total loss function, the loss function converges to obtain a well-trained student model.

Images