A fine-grained 3D model classification method based on deep integration and detail perception

Abstract

The present invention discloses a fine-grained three-dimensional model classification method based on deep integration and detail perception. The method realizes fine-grained classification of three-dimensional model multi-views based on a three-dimensional model fine-classification network. The three-dimensional model fine-classification network consists of a backbone network and an auxiliary network. The backbone network consists of an initial feature extraction layer, a group of base learners and a deep integration learner, which is used to learn the global structural features of the three-dimensional model contained in the multi-views. The auxiliary network consists of a context detail perception module, a view maximum pooling layer and a fully connected layer, which is used to capture the detail features contained in the view; the classification decisions of the backbone network and the auxiliary network are combined to form a comprehensive classification decision of the overall network, and a joint loss function is constructed based on this, so that the three-dimensional model fine-classification network can accurately classify fine-grained three-dimensional models. The present invention can effectively capture fine-grained discriminative information and has good classification effect and universality in three data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of computer graphics, computer vision and intelligent recognition, and in particular to a fine-grained three-dimensional model classification method based on depth integration and detail perception. Background Art

[0002] The fine-grained 3D model classification task is a very challenging problem in the field of computer vision. It is mainly used to distinguish sub-class 3D models with only local differences. At present, an ideal fine-grained 3D model classification network not only needs to capture the global shape features of the 3D model, but also needs to further extract discriminative local areas between sub-classes, which leads to poor classification ability of general 3D model classification networks when applied to fine-grained recognition. Existing general 3D model classification work cannot meet these current needs. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art. Aiming at the problem that the general 3D model classification method has poor classification ability when applied to fine-grained recognition, a fine-grained 3D model classification method based on deep integration and detail perception is proposed, which can effectively capture the overall shape features of the 3D model under multiple views and the detail perception features contained in its views. That is, in the backbone network and the auxiliary network, the global structural information of the 3D model and the fine-grained discriminative information contained in each view are extracted by using the deep integration learner and the context detail perception module, respectively, to achieve efficient recognition of fine-grained 3D models.

[0004] To achieve the above-mentioned purpose, the technical solution provided by the present invention is as follows: a fine-grained 3D model classification method based on deep integration and detail perception, the method is based on a 3D model fine-grained classification network to achieve fine-grained classification of 3D model multi-views, wherein the 3D model fine-grained classification network is composed of a backbone network and an auxiliary network, the backbone network is composed of an initial feature extraction layer, a group of base learners and a deep integration learner, and is used to learn the global structural features of the 3D model contained in the multi-views, the auxiliary network is composed of a context detail perception module, a view maximum pooling layer and a fully connected layer, and is used to capture the detail features contained in the view;

[0005] The specific implementation of the fine-grained 3D model classification method includes the following steps:

[0006] S1. Input the multi-view sequence of the 3D model into the backbone network, extract the feature information contained in each view through the initial feature extraction layer, and generate the initial feature map of the 3D model. Select the shallow feature map of the initial feature extraction layer As the input of the auxiliary network, k represents the feature map from the kth layer of the initial feature extraction layer, v iRepresents the i-th view of a 3D model. A 3D model has a total of V views.

[0007] S2, the initial feature map The information is sent to the corresponding base learners to calculate the initial classification decision of each view. After the initial decision information is concatenated, the final classification decision of the backbone network is obtained through the ensemble learner.

[0008] S3, shallow feature map Send it to the context detail perception module to calculate the context detail perception feature map CF mn ; After dimensionality reduction through average pooling operation, all CF mn Splice to get the detail perception feature DF of the corresponding view i ; Finally, the view max pooling layer is used to integrate the detail perception features, and the classification decision of the auxiliary network is calculated through the fully connected layer;

[0009] S4. Combine the classification decisions of the backbone network and the auxiliary network to form a comprehensive classification decision of the overall 3D model classification network, and construct a joint loss function based on this, so that the 3D model classification network can accurately classify fine-grained 3D models and complete the fine-grained classification of multiple views of 3D models.

[0010] Further, in step S1, the initial feature extraction layer uses a residual neural network constructed by a convolution and pooling design pattern to extract initial feature information from the view sequence; wherein the residual neural network includes four different stages, each stage is composed of a different number of residual blocks, the first stage includes a 7*7 convolution layer, a maximum pooling layer and three residual blocks, which are intended to perform initial preprocessing on the view and perform shallow feature extraction on it, the second stage is four residual blocks, which are intended to further abstract the information and features in the view, the third stage is six residual blocks, which are intended to learn the high-level semantic information contained in the view, and the fourth stage is three residual blocks, which are intended to extract highly abstract semantic features of the view; the initial feature extraction layer abstracts the input view sequence into an initial feature map of w×h×c Where w represents the width of the feature map, h represents the height of the feature map, and c represents the number of channels of the feature map; the initial feature map It contains abstract high-level semantic information, which is conducive to the network learning the global structure of the 3D model and using it as the input feature map of the base learner; the shallow feature map of the initial feature extraction layer has richer detailed feature information, which is beneficial for distinguishing fine-grained 3D models, so some shallow feature maps are selected. as the input of the auxiliary network.

[0011] Further, in step S2, the base learner is composed of a global average pooling layer and two fully connected layers; the global average pooling further abstracts and downsamples the highly abstract semantic features of each view; the initial feature map After being compressed by global average pooling, the information is further abstracted, and then the initial classification decision corresponding to each view is calculated through a (c, 256) and (256, L) fully connected layer; wherein c represents the number of channels in the feature map, and L represents the number of categories in the data set; the ensemble learner is composed of a Concat splicing layer and a fully connected layer; after the initial classification decision is sent to the ensemble learner, the Concat splicing operation is first used to integrate and splice all the initial classification decisions, and then a (V×L, L) fully connected layer is used to obtain the final classification decision of the global structure of the 3D model, wherein V represents the number of views of the 3D model;

[0012] In order to improve the accuracy of the initial prediction results of the deep ensemble learner, the classification loss function Loss of the base learner is defined m And the ensemble learner classification loss function Loss s , respectively solved by the following formulas:

[0013]

[0014]

[0015] Where N is the number of samples; χ{·} represents the indicator function. If the true label of the i-th sample is j, the function takes the value of 1, otherwise it takes the value of 0. represents the true label of the i-th view of the 3D model; t ij represents the predicted value that the ith sample belongs to the jth class; δ{·} represents the indicator function. If the true label of the ith sample is j, the function takes the value of 1, otherwise the function takes the value of -1, where represents the true label of the i-th 3D model; P represents the L1 regularization paradigm;

[0016] Then the classification loss of the backbone network is Loss main Solved by the following formula:

[0017]

[0018] In the formula, Loss m represents the classification prediction loss corresponding to the base learner corresponding to the m-th view, and α and β represent the weight distribution of the base learner loss and the ensemble learner loss.

[0019] Further, in step S3, the context detail perception module extracts the detail feature information contained in the view by establishing information interaction between features of different convolutional layers; first, a dot multiplication operation is performed on the corresponding positions of the convolutional layer features of the view to obtain the context detail perception feature map CF of the i-th view. mn ; Then use the splicing operation to fuse the contextual detail perception features between different levels to form the detail perception feature map DF of the corresponding view i Finally, the detailed features are reduced in dimension through the global average pooling layer. The calculation formula is as follows:

[0020]

[0021] DF i =Concat(g(CF 12 ),g(CF 13 ),g(CF 23 ))

[0022] Where m and n are the mth and nth residual blocks in the fourth stage of the initial feature extraction layer, respectively. and It represents the shallow feature map taken from the mth and nth residual blocks, γ is the matrix transformation operation, which transforms the input matrix of size c×h×w into c×n, and n=h×w, c represents the number of channels of the feature map, h represents the height of the feature map, w represents the width of the feature, and ⊙ represents the dot product operation between matrices; CF mn CF represents the context-detail-aware feature map after the information interaction of the shallow feature maps of the m-th and n-th residual blocks; 12 Represents the context-detail-aware feature map after the interaction of the shallow feature information of the first and second residual blocks, CF 13 Represents the context-detail-aware feature map after the interaction of the shallow feature information of the first and third residual blocks, CF 23 It represents the context-detail-aware feature map after the interaction of the shallow feature information of the second and third residual blocks; g(·) is the global average pooling, and Concat represents the concatenation operation.

[0023] Further, in step S3, after the auxiliary network captures the fine-grained discriminative information in the view through the context detail perception module, the view maximum pooling layer integrates the information of the detail features in the view, aiming to find the most significant detail features in the view so that they have category representativeness, and then obtains the classification decision corresponding to the fine-grained features through a (2048×3,L) fully connected layer, where L represents the number of categories;

[0024] In order to ensure that the detailed features captured by the network are discriminative, the loss function of the auxiliary network is Loss auxiSolved by the following formula:

[0025]

[0026] In the formula, N represents the number of samples; represents the indicator function. If the true label of the i-th sample is j, the function value is 1, otherwise the function value is 0, where l i represents the true label of the i-th sample; t ij Indicates the predicted value that the i-th sample belongs to the j-th class.

[0027] Further, in step S4, the classification decisions of the backbone network and the auxiliary network are fused to form a comprehensive classification decision, aiming to fuse the classification results of the two networks to obtain the final result of the three-dimensional model classification; wherein the classification decision of the backbone network and the classification decision of the auxiliary network are weighted and calculated, and the fusion ratio is set to 1:1 to obtain the correct category prediction value to which the three-dimensional model belongs; to ensure the classification accuracy of the method, the classification loss function Loss of the overall three-dimensional model detailed classification network is solved by the following formula:

[0028] Loss = λ1Loss main +λ2Loss auxi

[0029] In the formula, λ1 and λ2 represent the weight distribution of the main network and auxiliary network losses, Loss main Represents the classification loss function of the backbone network, Loss auxi Represents the classification loss function of the auxiliary network.

[0030] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0031] 1. The method of the present invention realizes fine-grained three-dimensional model classification based on deep integration and detail perception.

[0032] 2. A method combining the backbone network and the auxiliary network is proposed to classify fine-grained 3D models; the global structural information of the 3D model is learned through the backbone network to provide global structural guarantees for the detail features; the detail feature information in the view is learned through the auxiliary network to learn the detail features in the 3D model.

[0033] 3. An auxiliary network containing a context detail perception module is proposed, which makes full use of the context feature information to extract local features with fine-grained discrimination capabilities, so that the context information is effectively utilized and richer detail features are retained for classification.

[0034] 4. Experiments on different categories show that the proposed method has good universality and classification performance compared with the most advanced methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the overall architecture diagram of the present invention.

[0036] Figure 2 The main network structure diagram of the present invention; in the figure, ResNet is a residual neural network, FC is a fully connected layer, and Loss i Represents the classification loss corresponding to the i-th view, Concat is tensor concatenation, Loss main is the overall classification loss of the backbone network.

[0037] Figure 3 This is the structural diagram of the ResNet residual neural network.

[0038] Figure 4 It is the auxiliary network structure diagram of the present invention; in the figure, View-Maxpool is view maximum pooling, CDAM is context detail perception module, Reshape is dimensionality transformation, GAP is global average pooling, and Concat is tensor concatenation. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0040] This embodiment provides a fine-grained 3D model classification method based on deep integration and detail perception. The method implements fine-grained classification of 3D model multi-views based on a 3D model fine-classification network, wherein the 3D model fine-classification network is composed of a backbone network and an auxiliary network. Figure 1 As shown in the figure, the entire network framework is shown. The multi-view sequence learns the global features and detail-perceived features of the view through the backbone network and the auxiliary network. The backbone network extracts the global features of the 3D model and forms an initial decision of the global structure. The auxiliary network captures the detail-perceived features of the 3D model and forms an initial decision with detail-discriminative features. The initial decisions of the two networks are combined to form the final decision of the entire 3D model classification network, completing the learning of the fine-grained discriminative features of the 3D model and realizing the classification of the fine-grained 3D model. It includes the following steps:

[0041] 1) Input a set of V 3D model multi-view sequences, which come from 2D projections of the 3D model at different angles;

[0042] 2) The input multi-view sequence is passed to the initial feature extraction layer of the backbone network, and the feature information contained in each view is extracted through the initial feature extraction layer to generate the initial feature map of the 3D model. Select the shallow feature map of the initial feature extraction layer As the input of the auxiliary network, k represents the feature map from the kth layer of the initial feature extraction layer, v i Represents the i-th view of a 3D model. A 3D model has a total of V views.

[0043] The backbone network architecture is as follows Figure 2 As shown in FIG, it is composed of an initial feature extraction layer, a set of base learners and a deep ensemble learner. The initial feature extraction layer uses a residual neural network constructed by a convolution and pooling design pattern to extract the initial feature information in the view sequence; wherein the residual neural network contains four different stages, such as Figure 3 As shown in the figure, each stage consists of a different number of residual blocks. The first stage contains a 7*7 convolutional layer, a maximum pooling layer and three residual blocks, which are designed to perform initial preprocessing on the view and extract shallow features. The second stage consists of four residual blocks, which are designed to further abstract the information and features in the view. The third stage consists of six residual blocks, which are designed to learn the high-level semantic information contained in the view. The fourth stage consists of three residual blocks, which are designed to extract highly abstract semantic features of the view. The initial feature extraction layer abstracts the input view sequence into an initial feature map of w×h×c. Where w represents the width of the feature map, h represents the height of the feature map, c represents the number of channels of the feature map, and v i Represents the i-th view of the 3D model; initial feature map It contains abstract high-level semantic information, which is conducive to the network learning the global structure of the 3D model and using it as the input feature map of the base learner; the shallow feature map of the initial feature extraction layer has richer detailed feature information, which is beneficial for distinguishing fine-grained 3D models, so some shallow feature maps are selected. as the input of the auxiliary network.

[0044] The backbone network automatically extracts the global structural features of the 3D model. The samples obtained by each base learner for each view are compared with the true sample labels. The classification mapping information of the 3D model is compared with the true labels by the predicted values ​​obtained by the deep ensemble learner. In order to ensure the classification performance of the network for fine-grained 3D models, the classification loss function of the base learner is Loss m And the ensemble learner classification loss function Loss s , respectively solved by the following formulas:

[0045]

[0046]

[0047] Where N is the number of samples; L is the number of categories; χ{·} is the indicator function. If the true label of the i-th sample is j, the function takes the value of 1, otherwise it takes the value of 0. represents the true label of the i-th view of the 3D model; t ij represents the predicted value that the ith sample belongs to the jth class; δ{·} represents the indicator function. If the true label of the ith sample is j, the function takes the value of 1, otherwise the function takes the value of -1, where represents the true label of the i-th 3D model; P represents the L1 regularization paradigm;

[0048] Overall classification loss of the backbone network main It can be solved by the following formula:

[0049]

[0050] In the formula, Loss m represents the classification prediction loss corresponding to the base learner corresponding to the m-th view, V represents the 3D model has a total of V views, α and β represent the weight distribution of the base learner loss and the ensemble learner loss.

[0051] 3) Initial feature maps of each view The initial classification decision is calculated through the base learner, and then the deep ensemble learner is used to integrate the initial classification decisions of all views to obtain the final classification decision of the backbone network;

[0052] like Figure 2 As shown in Figure 2, the base learner consists of a global average pooling layer and two fully connected layers. In the base learner, the view passes through the initial feature extraction layer to obtain an initial feature map with high-level semantic information. The dimensionality is reduced through a global average pooling layer, and the classification decision information of each view is generated using the (c, 256) and (256, L) fully connected layers; where c represents the number of channels of the feature map, and L represents the number of categories in the data set; the ensemble learner consists of a Concat splicing layer and a fully connected layer; after the initial classification decision is sent to the ensemble learner, all the initial classification decisions are first integrated and spliced ​​using the Concat splicing operation, and then a (V×L, L) fully connected layer is used to obtain the final classification decision of the global structure of the 3D model, where V represents the number of 3D model views.

[0053] 4) Extract the shallow feature map of the initial feature extraction layer It is sent to the auxiliary network, and the detail perception features of the 3D model are captured through the context detail perception module, and the corresponding initial features are obtained;

[0054] The specific three-layer shallow features of the initial feature extraction layer are sent to the auxiliary network. The auxiliary network structure is as follows: Figure 4 As shown in (a), it consists of a context detail perception module, a view maximum pooling layer, and a fully connected layer to capture the detail features contained in the view. The structure of the context detail perception module is shown in Figure 4 As shown in (b), by establishing information interaction between features of different convolutional layers, the detailed feature information contained in the view is extracted; first, the shallow features of the view Perform a dot multiplication operation at the corresponding position to obtain the context detail perception feature map CF of the i-th view mn Then, the contextual detail perception features between different levels are fused by splicing operation to form the detail perception feature map DF of the corresponding view. i , and finally sent to the global average pooling layer to reduce the dimension of the obtained detail features. The calculation formula is as follows:

[0055]

[0056] DF i =Concat(g(CF 12 ),g(CF 13 ),g(CF 23 ))

[0057] Where m and n are the mth and nth residual blocks in the fourth stage of the initial feature extraction layer, respectively. and It represents the shallow feature map taken from the mth and nth residual blocks, γ is the matrix transformation operation, which transforms the input matrix of size c×h×w into c×n, and n=h×w, c represents the number of channels of the feature map, h represents the height of the feature map, w represents the width of the feature, and ⊙ represents the dot product operation between matrices; CF mn CF represents the context-detail-aware feature map after the information interaction of the shallow feature maps of the m-th and n-th residual blocks; 12 Represents the context-detail-aware feature map after the interaction of the shallow feature information of the first and second residual blocks, CF 13 Represents the context-detail-aware feature map after the interaction of the shallow feature information of the first and third residual blocks, CF 23 It represents the context-detail-aware feature map after the interaction of the shallow feature information of the second and third residual blocks; g(·) is the global average pooling, and Concat represents the concatenation operation.

[0058] After the auxiliary network captures the fine-grained discriminative information in the view through the contextual detail perception module, the view maximum pooling layer integrates the information of the detail features in the view, aiming to find the most significant detail features in the view and make them representative of the category. Then, a (2048×3,L) fully connected layer is used to obtain the classification decision corresponding to the fine-grained features.

[0059] In order to ensure that the detailed features captured by the network are discriminative, the loss function of the auxiliary network is Loss auxi It can be solved by the following formula:

[0060]

[0061] In the formula, N represents the number of samples; represents the indicator function. If the true label of the i-th sample is j, the function value is 1, otherwise the function value is 0, where l i represents the i-th sample; t ij Indicates the predicted value that the i-th sample belongs to the j-th class.

[0062] 5) Combine the classification decisions of the backbone network and the auxiliary network to form a comprehensive classification decision of the overall 3D model classification network; and construct a joint loss function based on this, so that the 3D model classification network can accurately classify fine-grained 3D models and complete the fine-grained classification of multiple views of 3D models.

[0063] The classification decisions of the backbone network and the auxiliary network are fused to form a comprehensive classification decision, aiming to fuse the classification results of the two networks to obtain the final result of the three-dimensional model classification; wherein, the classification decision of the backbone network and the classification decision of the auxiliary network are weighted and calculated, and the fusion ratio is set to 1:1 to obtain the correct category prediction value to which the three-dimensional model belongs.

[0064] In order to ensure the classification accuracy of the method, the classification loss function Loss of the overall three-dimensional model fine classification network is solved by the following formula:

[0065] Loss = λ1Loss main +λ2Loss auxi

[0066] In the formula, λ1 and λ2 represent the weight distribution of the main network and auxiliary network losses, Loss main Represents the classification loss function of the backbone network, Loss auxi Represents the classification loss function of the auxiliary network.

[0067] Experimental configuration: The hardware environment of this experiment is Intel Xeon Silver 4214+TESLA V100 32GB+128G RAM, and the software environment is Linux x64+CUDA 10.0+cuDNN 7.4+Pytorch+Python.

[0068] Dataset: The dataset used in this experiment is the fine-grained 3D model dataset FG3D (Fine-Grained 3D Datasets). The dataset contains three subcategories, namely Airplane, Car, and Chair. The Airplane sub-dataset contains 4174 models in 13 subcategories, the Car sub-dataset contains 8235 models in 20 subcategories, and the Chair sub-dataset contains 13054 models in 33 subcategories. The views in FG3D are all two-dimensional representations from 12 different perspectives after the three-dimensional model is tilted 30 degrees. Table 1 gives the basic information of the dataset selected in this paper.

[0069] Table 1 Basic information of the dataset

[0070] Subdataset Number of subcategories Total number of models Training set Test Set Airplane 13 4173 3441 732 Car 20 8235 7010 1315 Chair(chair) 33 13054 11124 1930

[0071] Table 2 Classification accuracy of fine-grained 3D models

[0072]

[0073]

[0074] Compared with the general 3D model classification method, it is not possible to achieve better results on this dataset. Existing methods only focus on learning and capturing the overall structure of the 3D model view. The method of the present invention uses deep ensemble learning to strengthen the overall structure learning of the 3D model, and designs the context detail perception module of the auxiliary network to capture the fine-grained discriminant information in the view, so as to learn the fine-grained features of the 3D model, effectively improving the performance of the fine-grained 3D model classification method, and the amount of calculation and parameters is moderate.

[0075] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fine-grained 3D model classification method based on deep integration and detail perception, characterized in that: The method realizes fine-grained classification of 3D model multi-views based on a 3D model fine-classification network, wherein the 3D model fine-classification network is composed of a backbone network and an auxiliary network, wherein the backbone network is composed of an initial feature extraction layer, a group of base learners and a deep ensemble learner, and is used to learn the global structural features of the 3D model contained in the multi-views, and the auxiliary network is composed of a context detail perception module, a view maximum pooling layer and a fully connected layer, and is used to capture the detail features contained in the view; The specific implementation of the fine-grained 3D model classification method includes the following steps: S1. Input the multi-view sequence of the 3D model into the backbone network, extract the feature information contained in each view through the initial feature extraction layer, and generate the initial feature map of the 3D model. Select the shallow feature map of the initial feature extraction layer As the input of the auxiliary network, k represents the feature map from the kth layer of the initial feature extraction layer, v i Represents the i-th view of a 3D model. A 3D model has a total of V views. S2, the initial feature map The information is sent to the corresponding base learners to calculate the initial classification decision of each view. After the initial decision information is concatenated, the final classification decision of the backbone network is obtained through the ensemble learner. S3, shallow feature map Send it to the context detail perception module to calculate the context detail perception feature map CF mn ; After dimensionality reduction through average pooling operation, all CF mn Splice to get the detail perception feature DF of the corresponding view i ; Finally, the view max pooling layer is used to integrate the detail perception features, and the classification decision of the auxiliary network is calculated through the fully connected layer; The context detail perception module extracts the detail feature information contained in the view by establishing information interaction between features of different convolutional layers. First, a dot multiplication operation is performed on the corresponding positions of the convolutional layer features of the view to obtain the context detail perception feature map CF of the i-th view. mn ; Then use the splicing operation to fuse the contextual detail perception features between different levels to form the detail perception feature map DF of the corresponding view i Finally, the detailed features are reduced in dimension through the global average pooling layer. The calculation formula is as follows: DF i =Concat(g(CF 12 ),g(CF 13 ),g(CF 23 )) Where m and n are the mth and nth residual blocks in the fourth stage of the initial feature extraction layer, respectively. and It represents the shallow feature map taken from the mth and nth residual blocks, γ is the matrix transformation operation, which transforms the input matrix of size c×h×w into c×n, and n=h×w, c represents the number of channels of the feature map, h represents the height of the feature map, w represents the width of the feature, and ⊙ represents the dot product operation between matrices; CF mn CF represents the context-detail-aware feature map after the information interaction of the shallow feature maps of the m-th and n-th residual blocks; 12 Represents the context-detail-aware feature map after the interaction of the shallow feature information of the first and second residual blocks, CF 13 Represents the context-detail-aware feature map after the interaction of the shallow feature information of the first and third residual blocks, CF 23 It represents the contextual detail-aware feature map after the interaction of the shallow feature information of the second and third residual blocks; g(·) is the global average pooling, and Concat represents the concatenation operation; S4. Combine the classification decisions of the backbone network and the auxiliary network to form a comprehensive classification decision of the overall 3D model classification network, and construct a joint loss function based on this, so that the 3D model classification network can accurately classify fine-grained 3D models and complete the fine-grained classification of multiple views of 3D models.

2. The method for fine-grained 3D model classification based on depth integration and detail perception according to claim 1, characterized in that: In step S1, the initial feature extraction layer uses a residual neural network constructed by a convolution and pooling design pattern to extract the initial feature information in the view sequence; wherein the residual neural network includes four different stages, each stage is composed of a different number of residual blocks, the first stage includes a 7*7 convolution layer, a maximum pooling layer and three residual blocks, which are intended to perform initial preprocessing on the view and perform shallow feature extraction on it, the second stage is four residual blocks, which are intended to further abstract the information and features in the view, the third stage is six residual blocks, which are intended to learn the high-level semantic information contained in the view, and the fourth stage is three residual blocks, which are intended to extract highly abstract semantic features of the view; the initial feature extraction layer abstracts the input view sequence into an initial feature map of w×h×c Where w represents the width of the feature map, h represents the height of the feature map, and c represents the number of channels of the feature map; the initial feature map It contains abstract high-level semantic information, which is conducive to the network learning the global structure of the 3D model and using it as the input feature map of the base learner; the shallow feature map of the initial feature extraction layer has richer detailed feature information, which is beneficial for distinguishing fine-grained 3D models, so some shallow feature maps are selected. as the input of the auxiliary network.

3. The method for fine-grained 3D model classification based on depth integration and detail perception according to claim 1, characterized in that: In step S2, the base learner consists of a global average pooling layer and two fully connected layers; the global average pooling further abstracts and downsamples the highly abstract semantic features of each view; the initial feature map After being compressed by global average pooling, the information is further abstracted, and then the initial classification decision corresponding to each view is calculated through a (c, 256) and (256, L) fully connected layer; wherein c represents the number of channels in the feature map, and L represents the number of categories in the data set; the ensemble learner is composed of a Concat splicing layer and a fully connected layer; after the initial classification decision is sent to the ensemble learner, the Concat splicing operation is first used to integrate and splice all the initial classification decisions, and then a (V×L, L) fully connected layer is used to obtain the final classification decision of the global structure of the 3D model, wherein V represents the number of views of the 3D model; In order to improve the accuracy of the initial prediction results of the deep ensemble learner, the classification loss function Loss of the base learner is defined m And the ensemble learner classification loss function Loss s , respectively solved by the following formulas: Where N is the number of samples; χ{·} represents the indicator function. If the true label of the i-th sample is j, the function takes the value of 1, otherwise it takes the value of 0. represents the true label of the i-th view of the 3D model; t ij represents the predicted value that the ith sample belongs to the jth class; δ{·} represents the indicator function. If the true label of the ith sample is j, the function takes the value of 1, otherwise the function takes the value of -1, where represents the true label of the i-th 3D model; P represents the L1 regularization paradigm; Then the classification loss of the backbone network is Loss main Solved by the following formula: In the formula, Loss m represents the classification prediction loss corresponding to the base learner corresponding to the m-th view, and α and β represent the weight distribution of the base learner loss and the ensemble learner loss.

4. The method for fine-grained 3D model classification based on depth integration and detail perception according to claim 1, characterized in that: In step S3, after the auxiliary network captures the fine-grained discriminative information in the view through the context detail perception module, the view maximum pooling layer integrates the information of the detail features in the view, aiming to find the most significant detail features in the view and make them representative of the category, and then obtains the classification decision corresponding to the fine-grained features through a (2048×3,L) fully connected layer, where L represents the number of categories; In order to ensure that the detailed features captured by the network are discriminative, the loss function of the auxiliary network is Loss auxi Solved by the following formula: In the formula, N represents the number of samples; represents the indicator function. If the true label of the i-th sample is j, the function value is 1, otherwise the function value is 0, where l i Represents the true label of the i-th sample; t ij Indicates the predicted value that the i-th sample belongs to the j-th class.

5. The method for fine-grained 3D model classification based on depth integration and detail perception according to claim 1, characterized in that: In step S4, the classification decisions of the backbone network and the auxiliary network are fused to form a comprehensive classification decision, aiming to fuse the classification results of the two networks to obtain the final result of the classification of the three-dimensional model; wherein the classification decision of the backbone network and the classification decision of the auxiliary network are weighted and calculated, and the fusion ratio is set to 1:1, so as to obtain the correct category prediction value to which the three-dimensional model belongs; In order to ensure the classification accuracy of the method, the classification loss function Loss of the overall three-dimensional model fine classification network is solved by the following formula: Loss=λ1Loss main +λ2Loss auxi In the formula, λ1 and λ2 represent the weight distribution of the main network and auxiliary network losses, Loss main Represents the classification loss function of the backbone network, Loss auxi Represents the classification loss function of the auxiliary network.

Images