A multi-view three-dimensional model classification method based on budget perception and dynamic routing

Abstract

The application relates to the technical field of three-dimensional model classification, and discloses a multi-view three-dimensional model classification method based on budget perception and dynamic routing. The method first projects a three-dimensional model from multiple perspectives to obtain a multi-view two-dimensional image set; then a lightweight convolution view routing network is used to quickly score each view and determine the processing priority, a feature extraction network based on ResNet50 is used to extract the deep features of each view, and a view self-adaptive aggregation module based on a multilayer perception machine is used to weight and fuse the deep features of each view to obtain global shape features; finally, the global shape features are input into a classifier to obtain category probabilities, and an early exit mechanism is triggered based on a confidence threshold to realize efficient classification of the three-dimensional model. While maintaining high classification accuracy, the application can reduce the number of network forward calculations and is suitable for multi-view three-dimensional model identification and classification.

Description

Technical Field

[0001] This invention relates to a multi-view system based on budget awareness and dynamic routing. Figure 3 This method is a 3D model classification method, which has been widely used in the field of 3D model classification. Background Technology

[0002] In recent years, computer vision and 3D geometric modeling technologies have made significant progress. With continuous technological innovation, the amount of available 3D model data has grown exponentially, covering an increasingly wide range of areas, and the diversity and complexity of models have continued to rise. 3D models, with their powerful ability to represent object shape information, have been widely used in key fields such as autonomous driving, remote sensing, virtual reality, mechanical manufacturing, and robot vision grasping. This makes the ability to quickly, accurately, and with low power consumption identify 3D models, and to efficiently manage and utilize them, an important research topic in the field of computer vision.

[0003] Projecting a 3D model into a set of 2D views, and then using deep learning techniques to extract multi-view features and perform classification, is a mainstream and efficient method for 3D model recognition. This method has high engineering feasibility due to the low dimensionality of the 2D view data and its ability to fully utilize existing mature 2D image network frameworks. However, existing multi-view... Figure 3 3D model classification methods still have certain limitations: On the one hand, existing networks usually adopt a static inference mode. Regardless of the recognition level or information redundancy of the input view, a heavyweight backbone network is required to process and aggregate all the views one by one. This leads to a linear increase in the amount of inference computation and the number of views, resulting in serious waste of computing resources and making it difficult to meet the real-time inference and low power consumption requirements of intelligent edge computing devices. On the other hand, 3D model dimensionality reduction and rendering into 2D views inevitably results in the loss of spatial geometric topology information. When some viewpoints are occluded, missing, or have high background noise, the model is prone to classification performance degradation and lacks the ability to implicitly complete and robustly infer the overall 3D structure under incomplete view conditions.

[0004] Furthermore, existing multi-view feature aggregation mechanisms often employ simple strategies such as average pooling or max pooling, which struggle to adaptively allocate decision weights based on the information content and quality differences of individual views. This makes it easy for noisy features from low-quality views to contaminate the effective representation of high-quality views, thus affecting the final classification accuracy and stability. Based on these issues, it is necessary to propose a multi-view-oriented approach. Figure 3A budget-aware inference and adaptive fusion technology for dimensional model classification is proposed: a lightweight view routing module is used to sort the global view at low cost, and a heavyweight backbone network is used for sequential feature extraction and early exit based on confidence, so as to reduce unnecessary computational overhead while ensuring classification accuracy. At the same time, a cross-level knowledge distillation mechanism is introduced to guide the lightweight routing module to learn the high-dimensional importance rules of the heavyweight network. Furthermore, a cross-view self-supervised consistency constraint based on momentum update is combined to improve the classification robustness and system stability under complex conditions such as occlusion, missing data and noise by applying feature alignment to heterogeneous view subsets. Summary of the Invention

[0005] To address the problems existing in the field of 3D model classification, this invention discloses a multi-view method based on budget awareness and dynamic routing. Figure 3 3D model classification method.

[0006] Therefore, the present invention provides the following technical solution:

[0007] 1. A multi-view approach based on budget awareness and dynamic routing Figure 3 A 3D model classification method, characterized by the following main steps:

[0008] Step 1: Preprocess the 3D model dataset, perform multi-view projection on each 3D model, and obtain multi-view... Figure 2 A collection of 3D images.

[0009] Step 2: Build a lightweight view router for handling multiple input views. Figure 2 Lightweight feature extraction and view importance estimation are performed on a set of 3D images, and the initial route score for each view is output.

[0010] Step 3: Construct a heavyweight feature extraction network for the multi-view... Figure 2 High-dimensional deep semantic features are extracted from each view in the dimensional image set to obtain a view feature set.

[0011] Step 4: Construct a view adaptive aggregation module to score the importance of each view feature in the view feature set, and calculate the adaptive weight corresponding to each view based on the scoring results, so as to perform weighted fusion of the depth features of each view to obtain the global shape features.

[0012] Step 5: Construct an end-to-end joint optimization system, and build a cross-view system based on the teacher-student momentum update mechanism. Figure 1 Consistency constraints are imposed, and the evaluation capability of heavyweight branches on the importance of the view is transferred to the ultra-lightweight view router based on the cross-level knowledge distillation mechanism. The network is trained by jointly using classification loss, consistency loss and distillation loss.

[0013] Step 6: During the inference phase, execute a confidence-aware dynamic budget inference mechanism, utilizing the trained ultra-lightweight view router for multi-view analysis of the test samples. Figure 2 The image set is quickly scored and a processing priority sequence is determined. According to the processing priority sequence, the heavyweight feature extraction network, the view adaptive aggregation module and the classifier are called in sequence to perform incremental feature accumulation and iterative prediction. When the real-time classification confidence reaches a preset threshold, the early termination mechanism is triggered to stop the subsequent view calculation and output the classification result.

[0014] 2. A multi-view system based on budget awareness and dynamic routing as described in claim 1 Figure 3 The 3D model classification method is characterized by preprocessing the 3D model in step 1, specifically the following steps:

[0015] Step 1-1: Set a circle with a fixed tilt angle above the model in the ModelNet10 3D model dataset;

[0016] Steps 1-2 involve uniformly arranging camera positions along the sampling trajectory and sampling at preset angle intervals to obtain K two-dimensional views. The set of two-dimensional views of the three-dimensional model is expressed by the formula V={V1, V2, …, V K}, where K is the number of projected views, V i Let i be the i-th view image.

[0017] 3. A multi-view system based on budget awareness and dynamic routing as described in claim 1 Figure 3 The dimensional model classification method is characterized by the following steps in step 2: constructing a lightweight view router and outputting the initial route score corresponding to each view.

[0018] Step 2-1 will input multiple views Figure 2 A set of 3D images V = {V1, V2, …, V} K} as input to a very lightweight view router;

[0019] Step 2-2 uses convolutional feature extraction units and corresponding nonlinear activation units to perform lightweight feature extraction on each view, obtaining lightweight feature maps for each view:

[0020]

[0021] Where G(⋅) is the lightweight feature extraction map in the ultra-lightweight view router, and F i r This is the lightweight feature map corresponding to the i-th view;

[0022] Steps 2-3 perform global pooling on the lightweight feature map to obtain compact feature vectors corresponding to each view:

[0023]

[0024] Where Pool(⋅) represents global pooling, g i This is the compact feature vector corresponding to the i-th view;

[0025] Steps 2-4 input the compact feature vector into the scoring layer and output the initial route score for each view:

[0026]

[0027] Among them, W r and b r These are the weight parameters and bias parameters of the scoring layer, s i r The initial score for the route corresponding to the i-th view;

[0028] Steps 2-5 combine the initial route scores corresponding to each view into an initial route score vector:

[0029]

[0030] Where R(⋅) is an ultra-lightweight view router, and S router This is the initial route score vector corresponding to the multi-view input set.

[0031] 4. A multi-view system based on budget awareness and dynamic routing as described in claim 1 Figure 3 The dimensional model classification method is characterized by the following steps in step 3: constructing a heavyweight feature extraction network to obtain a view feature set.

[0032] Step 3-1 will use the multi-view Figure 2 Each view in the 3D image set V is input into a heavyweight feature extraction network to extract the depth features corresponding to each view:

[0033]

[0034] Among them, f θ (⋅) is a heavyweight feature extraction network, e i The depth feature corresponding to the i-th view;

[0035] Step 3-2: Assemble the view feature set by arranging the depth features corresponding to each view in view order.

[0036]

[0037] 5. A multi-view system based on budget awareness and dynamic routing as described in claim 1 Figure 3The dimensional model classification method is characterized in that, in step 4, a view adaptive aggregation module is constructed to score the importance of each view feature in the view feature set, and to calculate the adaptive weight corresponding to each view based on the scoring results, so as to perform weighted fusion of the depth features of each view to obtain the global shape feature. The specific steps are as follows:

[0038] Step 4-1 assigns importance scores to each view feature in the view feature set to obtain the importance score for each view:

[0039]

[0040] Where S(⋅) is the view importance scoring function, s i h The importance score for the i-th view;

[0041] Step 4-2: Perform segmented gating modulation based on the importance score of each view, and calculate the adaptive weight for each view.

[0042]

[0043]

[0044] in, The importance score is the result of segmented gating modulation, β is the threshold for segmenting view importance, λ is the suppression coefficient for low-importance views, and 0 < λ < 1, α i For the adaptive weight corresponding to the i-th view, when s i h When s < β, it indicates that the importance of the i-th view is low. Therefore, its importance score is compressed and modulated to reduce the adverse effect of low-quality or redundant views on the construction of global shape features. i h When the value is ≥β, the importance score of the view remains unchanged to preserve the effective contribution of high-value views to classification decisions;

[0045] Step 4-3: Based on the adaptive weights, perform weighted fusion of the depth features of each view to obtain the global shape feature z:

[0046]

[0047] 6. A multi-view system based on budget awareness and dynamic routing as described in claim 1 Figure 3 The dimensional model classification method is characterized by the following steps in step 5: constructing an end-to-end joint optimization system to train the network.

[0048] Step 5-1: Construct a classifier by inputting the global shape features into the classifier to obtain the category score vector.

[0049]

[0050] Step 5-2 normalizes the category score vector to obtain the category probability distribution:

[0051]

[0052] Step 5-3: Construct cross-view based on teacher-student momentum update mechanism Figure 1 Consistency constraints divide the set of multiple view inputs corresponding to the same 3D model into two different view subsets V. A With V B The student network and teacher network are input separately to obtain student shape features and teacher shape features:

[0053]

[0054] Among them, F s (⋅) and F t (⋅) represent the feature extraction mappings for the student network and the teacher network, respectively;

[0055] Step 5-4: Teacher network parameters are updated from student network parameters using an exponential moving average.

[0056]

[0057] in, For teacher network parameters, Here are the student network parameters, and m is the momentum coefficient;

[0058] Step 5-5 applies cross-view to the global shape features of the student and the global shape features of the teacher. Figure 1 Consistency constraints, the consistency loss is expressed as:

[0059]

[0060] Steps 5-6, based on a cross-level knowledge distillation mechanism, transfer the view importance evaluation capability of the heavyweight branch to the ultra-lightweight view router. The view importance score vector output by the heavyweight branch and the routing score vector output by the ultra-lightweight view router are represented as follows:

[0061]

[0062] Steps 5-7 perform temperature normalization on the view importance score vector and route score vector respectively, and construct the teacher distribution p. h and student distribution p r :

[0063]

[0064] Where T is the distillation temperature coefficient, and stopgrad(⋅) represents the gradient cutoff operation;

[0065] Steps 5-8 approximate the teacher distribution by constraining the student distribution using distillation loss, whereby the distillation loss is expressed as:

[0066]

[0067] Steps 5-9 calculate the classification loss based on the complete multi-view input set, whereby the classification loss is expressed as:

[0068]

[0069] in, The true class label is CE(⋅), and the cross-entropy loss is CE(⋅).

[0070] Steps 5-10 combine the aforementioned classification loss and cross-view loss. Figure 1 Combining consistency loss and distillation loss, we obtain the total loss function, which is expressed as:

[0071]

[0072] Where, λ cons With λ router These are the weighting coefficients corresponding to consistency loss and distillation loss, respectively.

[0073] 7. A multi-view system based on budget awareness and dynamic routing as described in claim 1 Figure 3 The dimensional model classification method is characterized by the execution of a confidence-aware dynamic budget inference mechanism during step 6, specifically in the inference phase, with the following steps:

[0074] Step 6-1 utilizes the trained ultra-lightweight view router to perform multi-view analysis on the test samples. Figure 2 A set of 3D images is quickly scored, and a processing priority sequence is determined based on the scoring results:

[0075]

[0076] Step 6-2 calls the heavyweight feature extraction networks sequentially according to the priority sequence π to extract depth features from the currently unprocessed view, and adds the newly added depth features in the current step to the local feature set:

[0077]

[0078] Step 6-3 uses the view adaptive aggregation module to perform weighted fusion of the current local feature set to obtain the global fused features for the current step:

[0079]

[0080] Where, α i (t) The adaptive weight corresponding to the i-th processed view in step t;

[0081] Step 6-4 inputs the global fusion features from the current step into the classifier to obtain the probability distribution of the current class:

[0082]

[0083] Step 6-5: Calculate the confidence level of the current category based on the current category probability distribution.

[0084]

[0085] Step 6-6 triggers an early termination mechanism based on the relationship between the current category confidence level and the dynamic threshold, where the dynamic threshold is represented by a piecewise function:

[0086]

[0087] Where, τ t Let τ1, τ2, and τ3 be the dynamic confidence threshold corresponding to step t, and let τ1, τ2, and τ3 be the threshold parameters for different inference stages, satisfying τ1 ≥ τ2 ≥ τ3. Let T1 and T2 be the segmentation points for the inference stages, and T be the threshold value for the next step. max To maximize the number of processing steps, the early termination criterion for the current step is expressed as follows:

[0088]

[0089] When the confidence level of the current category satisfies c t ≥τ t When this happens, an early termination mechanism is triggered, stopping the calculation of the remaining view and outputting the current predicted category. This occurs when the confidence level of the current category does not reach a preset threshold τ. t If this happens, continue processing the next priority view until the maximum number of processing steps T is reached. max Alternatively, output predicted categories after processing all views:

[0090]

[0091] Beneficial effects:

[0092] This invention is a multi-view based on budget awareness and dynamic routing. Figure 3 3D model classification method.

[0093] 1. This invention addresses the problem of significant computational resource waste caused by static reasoning of the entire view in traditional multi-view networks, and proposes a multi-view network based on dynamic budget reasoning and an early termination mechanism. Figure 3 A dimensional model classification method is proposed. An ultra-lightweight view router is used to quickly score the input view, and a heavyweight backbone network is combined for sequential feature extraction and confidence assessment. When the confidence of the current classification reaches a preset threshold, the heavyweight computation of subsequent views can be terminated early, thereby reducing the actual forward pass of the heavyweight backbone network, lowering the overall computational overhead, and improving inference efficiency.

[0094] 2. To further improve the effectiveness and stability of dynamic view selection, this invention designs a cross-level knowledge distillation mechanism, transferring the view importance evaluation capability of heavyweight branches to a lightweight view router. During the training phase, the view importance score output by the heavyweight branches is used as the teacher signal. Temperature normalization and KL divergence distillation loss are used to constrain the output distribution of the lightweight view router, enabling it to have better view value evaluation and screening capabilities with lower computational overhead. This improves the prediction stability during early regression inference and reduces the number of invalid views processed.

[0095] 3. This invention designs a cross-view Figure 1 Consistency constraints and view adaptive aggregation strategies are employed to improve the robustness of 3D model classification. Firstly, a cross-view aggregation strategy based on teacher-student momentum updates is used. Figure 1 Consistency constraints align features extracted from different view subsets, which helps learn a consistent global shape representation across viewpoints, thereby improving classification stability under conditions of missing viewpoints, occlusion, or noise interference. Secondly, a view-adaptive aggregation module is used to dynamically assign weights and perform weighted fusion of depth features from each view. Compared with simple aggregation methods such as average pooling or max pooling, this can suppress the adverse effects of low-quality noisy views on the global representation and make fuller use of complementary information between multiple views, thereby improving classification performance and robustness under complex conditions.

[0096] 4. This invention is validated using the publicly available ModelNet10 model library as an example. The results of the embodiments show that the classification method proposed in this invention is effective. Attached Figure Description

[0097] Figure 1 This is an example diagram of a 3D model (toilet) to be classified in an embodiment of the present invention.

[0098] Figure 2 This is a diagram illustrating the three-dimensional model classification framework in an embodiment of the present invention.

[0099] Figure 3 This refers to the twelve angle views of a three-dimensional model obtained through a virtual camera in an embodiment of the present invention. Detailed Implementation

[0100] In order to clearly and completely describe the technical solutions in the embodiments of the present invention, Figure 1 The 3D model instance shown is an object to be classified, combined with Figure 2 The multi-view based on budget awareness and dynamic routing is shown. Figure 3 The dimensional model classification framework is used to describe the multi-view approach based on budget awareness and dynamic routing proposed in this invention. Figure 3 The 3D model classification method is explained in detail.

[0101] The method of the present invention mainly includes the following steps.

[0102] Step 1: Preprocess the 3D model. The specific steps are as follows:

[0103] Step 1-1: Set a circle with a fixed tilt angle above the model in the ModelNet10 3D model dataset;

[0104] Steps 1-2 involve uniformly distributing camera positions around the model and sampling at preset angle intervals to obtain K two-dimensional views, such as... Figure 3 As shown, the model's two-dimensional view set is represented by V = {V1, V2, …, V}. 12}, where K=12 is the number of projected views, V i Let i be the i-th view image.

[0105] Step 2: Construct an ultra-lightweight view router using a lightweight convolutional neural network structure. This lightweight convolutional neural network structure includes a first convolutional layer, a first non-linear activation layer, a second convolutional layer, a second non-linear activation layer, an adaptive average pooling layer, and a fully connected scoring layer connected sequentially. This is used to quickly extract features from each view and output the initial routing score for each view. The specific steps are as follows:

[0106] Step 2-1 will input multiple views Figure 2 A set of 3D images V = {V1, V2, …, V} 12} as input to a very lightweight view router;

[0107] Step 2-2 uses convolutional feature extraction units and corresponding nonlinear activation units to perform lightweight feature extraction on each view, obtaining lightweight feature maps for each view:

[0108]

[0109] Where G(⋅) is the lightweight feature extraction map in the ultra-lightweight view router, and F i r This is the lightweight feature map corresponding to the i-th view;

[0110] Steps 2-3 perform global pooling on the lightweight feature map to obtain compact feature vectors corresponding to each view:

[0111]

[0112] Where Pool(⋅) represents global pooling, g i The compact feature vector corresponding to the i-th view is shown below. The compact feature vectors obtained from the twelve views are as follows:

[0113] tensor([[1.9492e-01, 3.6390e-02, …, 1.0823e-04, 1.2931e-01, 1.3201e-02],

[0114] [1.4680e-01, 5.2334e-02, …, 2.6907e-04, 1.0540e-01, 1.3991e-02],

[0115] [7.8261e-02, 1.0008e-01, …, 1.9735e-04, 7.1005e-02, 1.4122e-02],

[0116] ...,

[0117] [1.4802e-01, 5.1604e-02, …, 2.3198e-04, 1.0597e-01, 1.3264e-02]]);

[0118] Steps 2-4 input the compact feature vector into the scoring layer and output the initial route score for each view:

[0119]

[0120] Among them, W r and b r These are the weight parameters and bias parameters of the scoring layer, s i r The initial score for the route corresponding to the i-th view;

[0121] Steps 2-5 combine the initial route scores corresponding to each view into an initial route score vector:

[0122]

[0123] Where R(⋅) is an ultra-lightweight view router, and S is the initial routing score vector S extracted from the twelve views of this 3D model sample. router As shown below:

[0124] tensor([1.5447e+00, 1.3330e+00, 1.0876e+00, 1.0805e+00, 1.0758e+00,1.3299e+00, 1.5571e+00, 1.3271e+00, 1.0839e+00, 1.0601e+00, 1.0657e+00,1.3339e+00]).

[0125] Step 3: Construct a heavyweight feature extraction network using a deep convolutional neural network. The deep convolutional neural network adopts a ResNet50 backbone structure, and its original classification layer is removed to output a single-view depth feature vector, which is used for high-dimensional feature extraction of each view. The specific steps are as follows:

[0126] Step 3-1 will use the multi-view Figure 2 Each view in the 3D image set V is input into a heavyweight feature extraction network to extract the depth features corresponding to each view:

[0127]

[0128] Among them, f θ (⋅) is a heavyweight feature extraction network, e i The depth feature corresponding to the i-th view;

[0129] Step 3-2: Assemble the view feature set by arranging the depth features corresponding to each view in view order.

[0130]

[0131] from Figure 3 The set of view features extracted from the two-dimensional view set shown is as follows:

[0132] tensor([[1.4006e-03, 1.2919e-01, …, 8.1070e-01, 9.2323e-01, 4.4066e-02],

[0133] [4.3976e-05, 4.2810e-01, …, 9.0611e-01, 1.3693e+00, 1.7738e-01],

[0134] [0.0000e+00, 3.0717e-01, …, 1.0743e+00, 1.3374e+00, 1.3691e-01],

[0135] ...,

[0136] [0.0000e+00, 5.0978e-01, …, 1.1027e+00, 1.4254e+00, 1.5300e-01]]).

[0137] Step 4: Construct a view adaptive aggregation module using a view scoring network based on a multilayer perceptron. The view scoring network includes a first fully connected layer, a GELU activation layer, and a second fully connected layer connected sequentially. This network is used to score the importance of each view's depth features and calculate the adaptive weights corresponding to each view based on the scoring results. These weighted weights are then used to fuse the depth features of each view to obtain the global shape features. The specific steps are as follows:

[0138] Step 4-1 assigns importance scores to each view feature in the view feature set to obtain the importance score for each view:

[0139]

[0140] Where S(⋅) is the view importance scoring function, s i h The importance score for the i-th view is given by the view adaptive aggregation module, and the heavy view importance score is shown below:

[0141] tensor([5.3645e+00, 5.9964e+00, 6.0512e+00, 4.5288e+00, 2.6289e+00,3.1590e+00, -1.1680e-01, 3.3301e+00, 3.4390e+00, 3.1405e+00, 5.1000e+00,5.9835e+00]);

[0142] Step 4-2: Perform segmented gating modulation based on the importance score of each view, and calculate the adaptive weight for each view.

[0143]

[0144]

[0145] in, The importance score is the result of segmented gating modulation, β is the threshold for segmenting view importance, λ is the suppression coefficient for low-importance views, and 0 < λ < 1, α i For the adaptive weight corresponding to the i-th view, when s i h When s < β, it indicates that the importance of the i-th view is low. Therefore, its importance score is compressed and modulated to reduce the adverse effect of low-quality or redundant views on the construction of global shape features. ih When ≥β, the importance score of the view remains unchanged to preserve the effective contribution of high-value views to the classification decision. The importance scores after segmented gating are shown below:

[0146] tensor([5.3645e+00, 5.9964e+00, 6.0512e+00, 4.5288e+00, 2.6289e+00,3.1590e+00, -5.8398e-02, 3.3301e+00, 3.4390e+00, 3.1405e+00, 5.1000e+00,5.9835e+00])

[0147] The adaptive weight vector α obtained from the complete set of views is shown below:

[0148] tensor([1.1777e-01, 2.2156e-01, 2.3404e-01, 5.1066e-02, 7.6389e-03,1.2979e-02, 5.1991e-04, 1.5401e-02, 1.7172e-02, 1.2741e-02, 9.0402e-02,2.1871e-01]);

[0149] Step 4-3: Based on the adaptive weights, perform weighted fusion of the depth features of each view to obtain the global shape feature z:

[0150]

[0151] The first 12 dimensions of the global shape feature z obtained by weighted fusion are shown below:

[0152] tensor([2.7996e-03, 4.0534e-01, 2.3857e+00, 2.6030e+00, 2.7042e+00,1.7043e-02, 4.8708e-02, 5.6346e-02, 6.9632e-02, 9.8222e-01, 1.2710e+00,1.6979e-01]).

[0153] Step 5: Construct an end-to-end joint optimization system and train the network. The specific steps are as follows:

[0154] Step 5-1: Construct a classifier by inputting the global shape features into the classifier to obtain the category score vector.

[0155]

[0156] Step 5-2 normalizes the category score vector to obtain the category probability distribution:

[0157]

[0158] The probability distribution matrix p is shown below:

[0159] tensor([3.0791e-05, 7.7238e-06, 2.6911e-07, 2.0118e-06, 1.6245e-06,8.5685e-07, 3.0258e-05, 4.9185e-05, 7.5362e-05, 9.9980e-01]);

[0160] Step 5-3: Construct cross-view based on teacher-student momentum update mechanism Figure 1 Consistency constraints divide the set of multiple view inputs corresponding to the same 3D model into two different view subsets V. A With V B The student network and teacher network are input separately to obtain student shape features and teacher shape features:

[0161]

[0162] Among them, F s (⋅) and F t (⋅) represent the feature extraction mappings for the student network and the teacher network, respectively;

[0163] Step 5-4: Teacher network parameters are updated from student network parameters using an exponential moving average.

[0164]

[0165] in, For teacher network parameters, Here are the student network parameters, and m is the momentum coefficient;

[0166] Step 5-5 applies cross-view to the global shape features of the student and the global shape features of the teacher. Figure 1 Consistency constraints, the consistency loss is expressed as:

[0167]

[0168] Steps 5-6, based on a cross-level knowledge distillation mechanism, transfer the view importance evaluation capability of the heavyweight branch to the ultra-lightweight view router. The view importance score vector output by the heavyweight branch and the routing score vector output by the ultra-lightweight view router are represented as follows:

[0169]

[0170] Steps 5-7 perform temperature normalization on the view importance score vector and route score vector respectively, and construct the teacher distribution p. h and student distribution p r :

[0171]

[0172] Where T is the distillation temperature coefficient, and stopgrad(⋅) represents the gradient cutoff operation;

[0173] Steps 5-8 approximate the teacher distribution by constraining the student distribution using distillation loss, whereby the distillation loss is expressed as:

[0174]

[0175] Steps 5-9 calculate the classification loss based on the complete multi-view input set, whereby the classification loss is expressed as:

[0176]

[0177] in, The true class label is CE(⋅), and the cross-entropy loss is CE(⋅).

[0178] Steps 5-10 combine the aforementioned classification loss and cross-view loss. Figure 1 Combining consistency loss and distillation loss, we obtain the total loss function, which is expressed as:

[0179]

[0180] Where, λ cons With λ router These are the weighting coefficients corresponding to consistency loss and distillation loss, respectively.

[0181] Step 6: Execute a confidence-aware dynamic budget inference mechanism during the inference phase. The specific steps are as follows:

[0182] Step 6-1 utilizes the trained ultra-lightweight view router to perform multi-view analysis on the test samples. Figure 2 A set of 3D images is quickly scored, and a processing priority sequence is determined based on the scoring results:

[0183]

[0184] For S router After sorting in descending order, the view priority sequence π={7, 1, 12, 2, 6, 8, 3, 9, 4,5, 11, 10} is obtained, which indicates that the 7th and 1st views are given the highest processing priority by the ultra-lightweight router.

[0185] Step 6-2 calls the heavyweight feature extraction networks sequentially according to the priority sequence π to extract depth features from the currently unprocessed view, and adds the newly added depth features in the current step to the local feature set:

[0186]

[0187] The first view, V7, is processed according to the priority sequence π. The newly added depth feature e1 is obtained through a heavyweight feature extraction network. An example of its first 12 dimensions is shown below:

[0188] tensor([5.2149e-03, 6.7215e-01, 1.1456e+00, 6.7250e-01, 1.2459e+00,7.2423e-02, 1.1408e-01, 2.4040e-04, 2.2259e-01, 1.1118e-01, 7.9842e-01,6.0195e-01])

[0189] Continuing to process the second view, V1, according to the priority sequence π, the newly added depth feature e2 is obtained through a heavyweight feature extraction network. An example of its first 12 dimensions is shown below:

[0190] tensor([1.3928e-03, 1.2919e-01, 1.9721e+00, 2.1601e+00, 2.5722e+00,4.3661e-03, 1.2919e-01, 2.2562e-02, 8.6250e-02, 8.1072e-01, 9.2329e-01,4.4097e-02]);

[0191] Step 6-3 uses the view adaptive aggregation module to perform weighted fusion of the current local feature set to obtain the global fused features for the current step:

[0192]

[0193] Where, α i (t) Let α be the adaptive weight corresponding to the i-th processed view in step t, and let α be the gated importance score of the local feature set in step 1, which is tensor([-5.8398e-02]). (1) =tensor([1.0000e+00]), where the gated importance score corresponding to the local feature set in step 2 is tensor([-5.8011e-02, 5.3645e+00]), and the corresponding adaptive weight is α. (2)=tensor([4.3974e-03, 9.9560e-01]);

[0194] Step 6-4 inputs the global fusion features from the current step into the classifier to obtain the probability distribution of the current class:

[0195]

[0196] The class probability distribution p1 of step 1 is obtained by inputting the fused features of the current step into the classifier as shown below:

[0197] tensor([7.6491e-05, 3.2407e-05, 3.3556e-03, 2.1213e-02, 7.1264e-05,8.0768e-06, 3.7963e-03, 7.0947e-07, 8.6416e-01, 1.0728e-01])

[0198] The category probability distribution p2 in step 2 is shown below:

[0199] tensor([3.9030e-04, 1.3911e-04, 2.7752e-05, 1.1376e-05, 1.7856e-05,2.4684e-05, 8.0903e-05, 6.5261e-04, 1.9682e-03, 9.9669e-01]);

[0200] Step 6-5: Calculate the confidence level of the current category based on the current category probability distribution.

[0201]

[0202] The maximum class confidence score for step 1 is c1 = 0.8642, and the maximum class confidence score for step 2 is c2 = 0.9967.

[0203] Step 6-6 triggers an early termination mechanism based on the relationship between the current category confidence level and the dynamic threshold, where the dynamic threshold is represented by a piecewise function:

[0204]

[0205] Where, τ t Let τ1, τ2, and τ3 be the dynamic confidence threshold corresponding to step t, and let τ1, τ2, and τ3 be the threshold parameters for different inference stages, satisfying τ1 ≥ τ2 ≥ τ3. Let T1 and T2 be the segmentation points for the inference stages, and T be the threshold value for the next step. max To maximize the number of processing steps, the early termination criterion for the current step is expressed as follows:

[0206]

[0207] When the confidence level of the current category satisfies c t ≥τ t When this happens, an early termination mechanism is triggered, stopping the calculation of the remaining view and outputting the current predicted category. This occurs when the confidence level of the current category does not reach a preset threshold τ. t If this happens, continue processing the next priority view until the maximum number of processing steps T is reached. max Alternatively, output predicted categories after processing all views:

[0208]

[0209] In this embodiment, the dynamic confidence threshold for step 1 is τ1 = 0.95. Since c1 = 0.8642 < τ1, the early termination mechanism is not triggered, and the next priority view is processed. The dynamic confidence threshold for step 2 is τ2 = 0.90. Since c2 = 0.9967 ≥ τ2, the early termination mechanism is triggered, and the calculation of the remaining views is stopped. The maximum probability in the current category probability distribution is p. 2,9 =0.9967, its corresponding predicted category index is 9, and the corresponding category is toilet, therefore Figure 1 The prediction result for the 3D model instance shown is toilet.

[0210] Validated on the ModelNet10 test set, in the full-view inference mode (i.e., classifying each 3D model using all 12 views), the overall classification accuracy is 94.16%, with a 100% recognition rate for the toilet category. In the dynamic inference mode, when using segmented dynamic confidence thresholds τ1=0.95, τ2=0.90, and τ3=0.85, the overall classification accuracy is 90.53%, requiring only 2.29 views to be processed per 3D model on average, resulting in a computational saving of 80.9%. Therefore, the method of this invention can significantly reduce the number of forward computations of the heavyweight backbone network while maintaining high classification accuracy, thus achieving efficient 3D model classification.

[0211] Furthermore, when taking τ1=τ2=τ3=τ as a special case of segmented dynamic threshold, the dynamic inference results of the method of the present invention on the ModelNet10 test set are as follows: When τ=0.70, the classification accuracy is 80.95%, the average number of views processed is 1.49 / 12, and the calculated saving ratio is 87.6%; when τ=0.80, the classification accuracy is 85.13%, the average number of views processed is 1.74 / 12, and the calculated saving ratio is 85.5%; when τ=0.85, the classification accuracy is 87.56%, the average number of views processed is 1.96 / 12, and the calculated saving ratio is 83.7%; when τ=0.90, the classification accuracy is 89.87%, the average number of views processed is 2.28 / 12, and the calculated saving ratio is 81.0%; when τ=0.95, the classification accuracy is 91.74%, the average number of views processed is 2.86 / 12, and the calculated saving ratio is 76.2%. The above results demonstrate that the method of the present invention can significantly reduce the number of forward processing steps in a heavyweight network while maintaining high classification accuracy through lightweight view routing and an early exit mechanism, and has good budget-aware reasoning capabilities.

[0212] The above description, in conjunction with the accompanying drawings, provides a detailed account of the embodiments of the present invention. The specific implementation methods described herein are merely illustrative of the invention. Those skilled in the art will recognize that variations and modifications can be made to the specific implementation methods and applications based on the principles of the present invention; therefore, this specification should not be construed as a limitation of the invention.

Claims

1. A multi-view 3D model classification method based on budget awareness and dynamic routing, characterized in that, This method mainly includes the following steps: Step 1: Preprocess the 3D model dataset and perform multi-view projection on each 3D model to obtain a set of multi-view 2D images. Step 2: Construct a lightweight view router to perform lightweight feature extraction and view importance estimation on the input multi-view 2D image set, and output the initial route score for each view. Step 3: Construct a heavyweight feature extraction network to extract high-dimensional deep semantic features from each view in the multi-view two-dimensional image set, thereby obtaining a view feature set. Step 4: Construct a view adaptive aggregation module to score the importance of each view feature in the view feature set, and calculate the adaptive weight corresponding to each view based on the scoring results, so as to perform weighted fusion of the depth features of each view to obtain the global shape features. Step 5: Construct an end-to-end joint optimization system. Based on the teacher-student momentum update mechanism, construct cross-view consistency constraints. Based on the cross-level knowledge distillation mechanism, transfer the evaluation capability of heavyweight branches on view importance to the ultra-lightweight view router. Train the network through joint classification loss, consistency loss and distillation loss. Step 6: During the inference phase, a confidence-aware dynamic budget inference mechanism is executed. The trained ultra-lightweight view router is used to quickly score the multi-view 2D image set of the test samples and determine the processing priority sequence. The heavyweight feature extraction network, view adaptive aggregation module and classifier are called in sequence according to the processing priority sequence to perform incremental feature accumulation and iterative prediction. When the real-time classification confidence reaches the preset threshold, the early termination mechanism is triggered to stop the subsequent view calculation and output the classification result.

2. The multi-view 3D model classification method based on budget awareness and dynamic routing according to claim 1, characterized in that, In step 1, the 3D model is preprocessed, and the specific steps are as follows: Step 1-1: Set a circle with a fixed tilt angle above the model in the ModelNet10 3D model dataset; Steps 1-2 involve uniformly arranging camera positions along the sampling trajectory and sampling at preset angle intervals to obtain K two-dimensional views. The set of two-dimensional views of the three-dimensional model is expressed by the formula V={V1, V2, …, V K }, where K is the number of projected views, V i Let i be the i-th view image.

3. The multi-view 3D model classification method based on budget awareness and dynamic routing according to claim 1, characterized in that, In step 2, a lightweight view router is constructed, and the initial routing score corresponding to each view is output. The specific steps are as follows: Step 2-1: Input the multi-view 2D image set V={V1, V2, …, V K } as input to a very lightweight view router; Step 2-2 uses convolutional feature extraction units and corresponding nonlinear activation units to perform lightweight feature extraction on each view, obtaining lightweight feature maps for each view: Where G(⋅) is the lightweight feature extraction map in the ultra-lightweight view router, and F i r This is the lightweight feature map corresponding to the i-th view; Steps 2-3 perform global pooling on the lightweight feature map to obtain compact feature vectors corresponding to each view: Where Pool(⋅) represents global pooling, g i This is the compact feature vector corresponding to the i-th view; Steps 2-4 input the compact feature vector into the scoring layer and output the initial route score for each view: Among them, W r and b r These are the weight parameters and bias parameters of the scoring layer, s i r The initial score for the route corresponding to the i-th view; Steps 2-5 combine the initial route scores corresponding to each view into an initial route score vector: Where R(⋅) is an ultra-lightweight view router, and S router This is the initial route score vector corresponding to the multi-view input set.

4. The multi-view 3D model classification method based on budget awareness and dynamic routing according to claim 1, characterized in that, In step 3, a heavyweight feature extraction network is constructed to obtain the view feature set. The specific steps are as follows: Step 3-1: Input each view in the multi-view 2D image set V into a heavyweight feature extraction network to extract the depth features corresponding to each view. Among them, f θ (⋅) is a heavyweight feature extraction network, e i The depth feature corresponding to the i-th view; Step 3-2: Assemble the view feature set by arranging the depth features corresponding to each view in view order.

5. The multi-view 3D model classification method based on budget awareness and dynamic routing according to claim 1, characterized in that, In step 4, a view adaptive aggregation module is constructed to score the importance of each view feature in the view feature set, and to calculate the adaptive weight corresponding to each view based on the scoring results, so as to perform weighted fusion of the depth features of each view to obtain the global shape feature. The specific steps are as follows: Step 4-1 assigns importance scores to each view feature in the view feature set to obtain the importance score for each view: Where S(⋅) is the view importance scoring function, s i h The importance score for the i-th view; Step 4-2: Perform segmented gating modulation based on the importance score of each view, and calculate the adaptive weight for each view. in, The importance score is the result of segmented gating modulation, β is the threshold for segmenting view importance, λ is the suppression coefficient for low-importance views, and 0 < λ < 1, α i For the adaptive weight corresponding to the i-th view, when s i h When s < β, it indicates that the importance of the i-th view is low. Therefore, its importance score is compressed and modulated to reduce the adverse effect of low-quality or redundant views on the construction of global shape features. i h When the value is ≥β, the importance score of the view remains unchanged to preserve the effective contribution of high-value views to classification decisions; Step 4-3: Based on the adaptive weights, perform weighted fusion of the depth features of each view to obtain the global shape feature z:

6. The multi-view 3D model classification method based on budget awareness and dynamic routing according to claim 1, characterized in that, In step 5, an end-to-end joint optimization system is constructed to train the network. The specific steps are as follows: Step 5-1: Construct a classifier by inputting the global shape features into the classifier to obtain the category score vector. Step 5-2 normalizes the category score vector to obtain the category probability distribution: Step 5-3 constructs cross-view consistency constraints based on the teacher-student momentum update mechanism, dividing the multi-view input set corresponding to the same 3D model into two different view subsets V. A With V B The student network and teacher network are input separately to obtain student shape features and teacher shape features: Among them, F s (⋅) and F t (⋅) represent the feature extraction mappings for the student network and the teacher network, respectively; Step 5-4: Teacher network parameters are updated from student network parameters using an exponential moving average. in, For teacher network parameters, Here are the student network parameters, and m is the momentum coefficient; Step 5-5 applies cross-view consistency constraints to the student's global shape features and the teacher's global shape features. The consistency loss is expressed as: Steps 5-6, based on a cross-level knowledge distillation mechanism, transfer the view importance evaluation capability of the heavyweight branch to the ultra-lightweight view router. The view importance score vector output by the heavyweight branch and the routing score vector output by the ultra-lightweight view router are represented as follows: Steps 5-7 perform temperature normalization on the view importance score vector and route score vector respectively, and construct the teacher distribution p. h and student distribution p r : Where T is the distillation temperature coefficient, and stopgrad(⋅) represents the gradient cutoff operation; Steps 5-8 approximate the teacher distribution by constraining the student distribution using distillation loss, whereby the distillation loss is expressed as: Steps 5-9 calculate the classification loss based on the complete multi-view input set, whereby the classification loss is expressed as: in, The true class label is CE(⋅), and the cross-entropy loss is CE(⋅). Steps 5-10 combine the classification loss, cross-view consistency loss, and distillation loss to obtain the total loss function, expressed as: Where, λ cons With λ router These are the weighting coefficients corresponding to consistency loss and distillation loss, respectively.

7. The multi-view 3D model classification method based on budget awareness and dynamic routing according to claim 1, characterized in that, In step 6, a confidence-aware dynamic budget inference mechanism is executed during the inference phase. The specific steps are as follows: Step 6-1 utilizes the trained ultra-lightweight view router to quickly score the multi-view 2D image set of the test samples, and determines the processing priority sequence based on the scoring results: Step 6-2 calls the heavyweight feature extraction networks sequentially according to the priority sequence π to extract depth features from the currently unprocessed view, and adds the newly added depth features in the current step to the local feature set: Step 6-3 uses the view adaptive aggregation module to perform weighted fusion of the current local feature set to obtain the global fused features for the current step: Where, α i (t) The adaptive weight corresponding to the i-th processed view in step t; Step 6-4 inputs the global fusion features from the current step into the classifier to obtain the probability distribution of the current class: Step 6-5: Calculate the confidence level of the current category based on the current category probability distribution. Step 6-6 triggers an early termination mechanism based on the relationship between the current category confidence level and the dynamic threshold, where the dynamic threshold is represented by a piecewise function: Where, τ t Let τ1, τ2, and τ3 be the dynamic confidence threshold corresponding to step t, and let τ1, τ2, and τ3 be the threshold parameters for different inference stages, satisfying τ1 ≥ τ2 ≥ τ3. Let T1 and T2 be the segmentation points for the inference stages, and T be the threshold value for the next step. max To maximize the number of processing steps, the early termination criterion for the current step is expressed as follows: When the confidence level of the current category satisfies c t ≥τ t When this happens, an early termination mechanism is triggered, stopping the calculation of the remaining view and outputting the current predicted category. This occurs when the confidence level of the current category does not reach a preset threshold τ. t If this happens, continue processing the next priority view until the maximum number of processing steps T is reached. max Alternatively, output predicted categories after processing all views:

Images