A method for constructing a lightweight dynamic resolution image classification model

Abstract

The present invention discloses a method for constructing a lightweight dynamic resolution image classification model, which is characterized in that a relatively small network module is used to predict the optimal resolution of a sample, in which an optimal transmission problem is introduced to model it, and the most suitable resolution for each sample is iteratively predicted, and then it is input into a multi-scale network model for classification prediction, thereby realizing a dynamic resolution image classification network. Compared with the prior art, the present invention has the advantages of small amount of calculation, high recognition accuracy and fast speed, which not only solves the problem of unbalanced resolution selection in the dynamic resolution network, but also achieves a higher classification accuracy result, greatly reduces the amount of calculation in the image classification process, and greatly improves the speed and recognition accuracy in the actual reasoning process.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification in computer vision, and in particular to a lightweight construction method of a dynamic resolution image classification model based on an optimal transmission problem. Background Art

[0002] With the rapid development of computing devices such as GPU hardware and the advent of the big data era, deep learning is being applied to various fields of artificial intelligence, especially computer vision, and has achieved very good results, such as image classification, target detection, semantic segmentation, etc., which has greatly facilitated people's daily lives. Compared with traditional computer vision technology, deep learning methods require large amounts of data and high-performance GPU hardware computing devices. As the performance of deep neural networks continues to improve, the depth of the network is getting deeper and deeper, and the number of parameters is getting larger and larger. In the actual production practice, the resulting high storage and high computing volume of the network model seriously restricts the application of neural networks on edge devices, such as mobile phones and other edge devices. Therefore, compressing or lightweighting the model, reducing its computing volume as much as possible without reducing the accuracy effect, and then accelerating its reasoning process has become a problem that we urgently need to solve. Dynamic networks are one of the ways to solve such problems.

[0003] The so-called dynamic refers to the network processing method when the model faces different data. The parameters and model architecture of the traditional static model are fixed, and the same reasoning mode is maintained regardless of the data. In other words, it has the same computational overhead when facing any input data. The characteristic of the dynamic network is that it can dynamically adjust the network parameters or network structure according to the different input data, perform simple processing on simple data, and allocate more computing resources to complex data for processing, thereby saving computing power and playing the role of model compression. At present, the dynamic network design is mainly divided into two types. The network structure is changed based on the overall recognition difficulty of the sample, such as the network depth and width, or different computing amounts are allocated based on the spatial redundancy within the image to process it. The dynamic network design scheme based on image resolution belongs to the latter.

[0004] The resolution of an image has a huge impact on the amount of model inference calculations and the speed of inference, so some related work has focused on dynamic networks based on image resolution. However, the existing technology has a more obvious problem: the mode collapse phenomenon often occurs in the selection of image resolution, making the prediction selection of image resolution extremely unbalanced. This phenomenon results in a large number of images not being assigned to the optimal resolution, which greatly affects the accuracy of image classification. Summary of the invention

[0005] The purpose of the present invention is to construct a lightweight dynamic resolution image classification model designed to address the deficiencies of the prior art. The modeling method of the convolutional neural network that introduces the optimal transmission problem is adopted, and a smaller network module is used to predict the optimal resolution of the sample. The optimal transmission problem is introduced in the process to model it, and the most suitable resolution for each sample is iteratively predicted, and then it is input into the multi-scale network model for classification prediction, thereby realizing a dynamic resolution image classification network. The present invention models the prediction process of dynamic resolution into an optimal transmission problem, iteratively predicts the optimal resolution allocation scheme for a batch of data at the minimum cost, allocates the optimal resolution path selection for each data, and then inputs the data scaled according to the optimal resolution into the multi-scale benchmark classification network to obtain the final prediction result. In the process of actual reasoning test, this method can greatly reduce the amount of calculation required for the image classification process, improve the reasoning speed and still maintain a high accuracy rate. It is particularly suitable for a variety of current classic neural network structures and can achieve good results.

[0006] The present invention focuses on the problems encountered in dynamic resolution networks, starting from the image classification task, and strives to solve this problem. As one of the most basic subtasks in computer vision, the image classification task currently has too much computational complexity in high-precision image classification models, which has caused certain difficulties in the deployment and efficient reasoning of edge devices. The research on dynamic network technology in this field is not only conducive to the lightweight work of models in current tasks, but also conducive to its expansion to more downstream tasks of computer vision. In addition, for some difficult-to-identify samples, multiple small-resolution paths can be selected for reasoning at the same time, and the prediction results of multiple branches are weighted and summed to obtain the final prediction result. The multi-path reasoning scheme can further improve the accuracy of the model, and can still reduce a certain amount of computation compared to the reasoning process of conventional resolution.

[0007] The purpose of the present invention is achieved as follows: a lightweight method for dynamic resolution image classification model based on optimal transmission problem, characterized in that a lightweight resolution prediction module is used to predict the optimal resolution of the image, and the optimal transmission problem is introduced to model this process. Each data sample is regarded as each supplier, and each resolution candidate is regarded as each demander. The resolution prediction module is used to output the cost required to assign each data sample to each resolution selection, thereby generating a cost matrix corresponding to each data and each resolution selection. The optimal transmission plan is iteratively predicted based on the supplier, demander and cost matrix, so as to assign the most appropriate resolution to each data. The image data scaled according to the optimal resolution is input into the benchmark multi-scale classification network to obtain the final prediction result. The overall process specifically includes the following steps:

[0008] Step 1: Dataset preparation and preprocessing

[0009] 1-1: Collect a large amount of image data to be classified, divide it into training set and test set in an 8:2 ratio, and the image format is JPG;

[0010] 1-2: Scale the above training set images according to the preset resolution list r, scale each image to K different sizes, generate the corresponding multi-scale image list, and then use random horizontal flipping to enhance the training set data; scale the test set images according to the preset resolution list r.

[0011] Step 2: Multi-scale classification network construction

[0012] 2-1: Based on the basic classification convolutional neural network, a multi-scale classification network module is constructed, a parallel network structure is adopted, and different batch processing layers (BN layers) are used to process data of different resolution sizes;

[0013] 2-2: Input the multi-scale image data list processed in step 1 into the multi-scale classification network, extract features for each image in the list, and obtain the prediction results.

[0014] Step 3: Multi-scale classification network training

[0015] 3-1: Use the prediction results obtained from the input data in step 2 to train the multi-scale classification network, and calculate the cross entropy loss and self-distillation loss based on its prediction results and the real image labels;

[0016] 3-2: Set appropriate learning rate and optimization strategy for the training process;

[0017] 3-3: When the number of training iterations reaches the set threshold, stop training;

[0018] 3-4: Use the test set data reserved in step 1 to test the multi-scale classification network, and determine the model training results based on the accuracy results.

[0019] Step 4: Resolution prediction module construction

[0020] 4-1: Construct a lightweight resolution prediction module consisting of two convolutional layers;

[0021] 4-2: List the multi-scale image data obtained in step 1 and select the small-size image x l The output of the above-constructed resolution prediction module is used for dynamic resolution prediction.

[0022] Step 5: Prediction of optimal transmission dynamic resolution

[0023] 5-1: Optimal transmission modeling for the resolution prediction problem, considering the N data in the same batch as N suppliers, the K candidate resolutions to be selected as K demanders, and modeling the dynamic resolution allocation problem as an optimal transmission problem;

[0024] 5-2: Use the resolution prediction module to output the cost matrix of the transmission process, which indicates the amount of calculation required to allocate each sample to each resolution and the required overhead for accuracy;

[0025] 5-3: Through algorithm iteration, the optimal transportation plan is calculated based on the lowest total cost, that is, the optimal resolution selection result for each image sample.

[0026] Step 6: Multi-path dynamic resolution prediction

[0027] 6-1: In the process of resolution prediction, for some samples that are difficult to identify, multiple small resolution branches are dynamically selected for simultaneous testing and reasoning based on the optimal transportation plan and a certain threshold judgment;

[0028] 6-2: The prediction results of multiple resolution branches are weighted and summed as the final prediction result.

[0029] Step 7: Resolution prediction module training:

[0030] 7-1: Fix the model parameters of the multi-scale classification network and use the output prediction results of each resolution branch in step 2 to construct the ideal cost matrix;

[0031] 7-2: The resolution prediction module in step 4 is trained using the constructed loss function. The training process uses the ideal cost matrix to constrain the output results.

[0032] 7-3: Set appropriate learning rate and optimization strategy for the training process;

[0033] 7-4: When the number of training iterations reaches the set threshold, stop training;

[0034] 7-5: Use the test set data reserved in step 1 to perform an overall test on the resolution prediction module and the multi-scale classification network, and scale the resolution of the test set data according to the optimal transportation solution in step 5. Input the scaled data into the multi-scale classification network, and determine the model training results based on the accuracy results to obtain a lightweight dynamic resolution image classification model based on the optimal transmission problem.

[0035] The loss function calculation process of the multi-branch classification network training process in step 3 specifically includes:

[0036] 1) Calculate the cross entropy loss function value between each branch prediction result and the true label;

[0037] 2) After weighting the prediction results of each branch, calculate the KL divergence value of each branch.

[0038] The loss function calculation process of the resolution prediction module training process in step 7 specifically includes:

[0039] 1) Calculate the L2 distance between the ideal cost matrix and the output cost matrix of the resolution prediction module;

[0040] 2) Calculate the KL divergence between the ideal cost matrix and the output cost matrix of the resolution prediction module;

[0041] 3) Generate a “false path” label based on the ideal cost matrix, and calculate the cross entropy between the cost matrix output by the resolution prediction module and the “false path” label.

[0042] Compared with the prior art, the present invention has better solved the problem of unbalanced resolution selection in the original dynamic resolution network, and achieved higher classification accuracy results. Compared with the standard classification network, it greatly reduces the amount of calculation in the image classification process, and the speed and recognition accuracy in the actual reasoning process are greatly improved. It is not only beneficial to the lightweight work of the model in the current task, but also conducive to its expansion to more downstream computer vision tasks, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of the present invention;

[0044] Figure 2 Flowchart for dynamic resolution image classification;

[0045] Figure 3 It is the algorithm model framework diagram;

[0046] Figure 4 It is a schematic diagram of multi-branch dynamic prediction;

[0047] Figure 5 This is a comparison chart of accuracy performance;

[0048] Figure 6 This is a comparison chart of the resolution prediction distribution;

[0049] Figure 7 Predict visualization effects for dynamic resolution;

[0050] Figure 8 This is a comparison chart of acceleration performance. DETAILED DESCRIPTION

[0051] See also Figure 1The construction of the lightweight dynamic resolution image classification model of the present invention specifically includes the following steps:

[0052] Step 1: Dataset preparation and preprocessing

[0053] 1-1: Collect a large amount of image data to be classified, divide it into training set and test set in an 8:2 ratio, and the image format is JPG;

[0054] 1-2: Scale the above training set images according to the preset resolution list r, scale each image to K different sizes, generate the corresponding multi-scale image list, and then use random horizontal flipping to enhance the training set data; scale the test set images according to the preset resolution list r.

[0055] Step 2: Construction of multi-scale classification network

[0056] 2-1: A parallel network structure is used on the basic classification convolutional neural network, and different batch layers (BN layers) are used to process data of different resolution sizes to build a multi-scale classification network module;

[0057] 2-2: Input the multi-scale image data list in step 1 into the multi-scale classification network, perform feature extraction for each image in the list, and obtain the prediction result.

[0058] Step 3: Training of multi-scale classification network

[0059] 3-1: Use the prediction results obtained in step 2 to train the multi-scale classification network, and calculate the cross entropy loss and self-distillation loss based on its prediction results and the real image labels;

[0060] 3-2: Set appropriate learning rate and optimization strategy for the training process;

[0061] 3-3: When the number of training iterations reaches the set threshold, stop training;

[0062] 3-4: Use the test set data scaled in step 1 to test the multi-scale classification network and determine the model training results based on the accuracy.

[0063] Step 4: Construction of resolution prediction module

[0064] 4-1: Use two convolutional layers to build a lightweight resolution prediction module;

[0065] 4-2: List the multi-scale image data in step 1 and select the small-size image x l The output is used for dynamic resolution prediction.

[0066] Step 5: Prediction of optimal transmission dynamic resolution

[0067] 5-1: Consider the N data in the same batch as N suppliers and the K candidate resolutions to be selected as K demanders, and perform optimal transmission modeling for the dynamic resolution allocation problem, that is, modeling the resolution prediction problem into an optimal transmission problem;

[0068] 5-2: Use the resolution prediction module to output the cost matrix of the transmission process, which represents the amount of computation required to assign each sample to each resolution and the overhead required for the accuracy;

[0069] 5-3: The optimal transportation plan is calculated based on the lowest total cost through algorithm iteration, that is, the optimal resolution selection result for each image sample.

[0070] Step 6: Prediction of multi-path dynamic resolution

[0071] 6-1: Based on the optimal transportation plan and threshold judgment, multiple small-resolution branches are dynamically selected for synchronous test reasoning;

[0072] 6-2: Based on the resolution prediction results, a multi-branch dynamic reasoning scheme is introduced, and the prediction results of multiple resolution branches are weighted and summed as the final prediction result.

[0073] Step 7: Training of resolution prediction module

[0074] 7-1: Fix the model parameters of the multi-scale classification network and use the output prediction results of each resolution branch in step 2 to construct the ideal cost matrix;

[0075] 7-2: The resolution prediction module in step 4 is trained using the constructed loss function. The training process uses the ideal cost matrix to constrain the output results.

[0076] 7-3: Set appropriate learning rate and optimization strategy for the training process;

[0077] 7-4: When the number of training iterations reaches the set threshold, stop training;

[0078] 7-5: Use the test set data in step 1 to test the resolution prediction module and the multi-scale classification network as a whole, and scale the resolution of the test set data according to the optimal transportation solution in step 5, and input it into the multi-scale classification network. Determine the model training result based on the accuracy rate, and obtain a lightweight dynamic resolution image classification model based on the optimal transmission problem.

[0079] The present invention is further described in detail below by taking the candidate resolution list r=[224, 192, 160, 128, 96] and using ResNet50 as the backbone network structure of the multi-scale classification network as an example.

[0080] Example 1

[0081] See also Figure 2 ,The present invention is mainly divided into three parts in the training phase of data set preprocessing module, the first stage of multi-scale classification network training and the second stage of resolution prediction module training.

[0082] The specific operations of the data set preprocessing module are as follows:

[0083] A1: The dataset used is the public image classification dataset ImageNet, which contains 1,281,167,435 training set images and 50,000 validation set images.

[0084] A2: All training set and validation set images are cropped according to the candidate resolution list, and each original image data x i Cutting Generation Wherein N represents the size of the data set. In this embodiment, according to the length of the list r, K=5, and the training set images are enhanced by randomly flipping horizontally.

[0085] See also Figure 3 , the algorithm model framework built based on the standard network ResNet50. The specific training process of the multi-scale network training module in this network model is as follows:

[0086] B1: Construct a multi-scale classification network and design a corresponding BN layer for each candidate resolution, while data of all scales share the convolutional layer and the final linear classification layer.

[0087] B2: Input the preprocessed training set image data X into the multi-scale classification network In the above example, we get the prediction results p of each branch. 1 ,p 2 ,…,p K , and weightedly aggregate the prediction results of each branch to generate p 0 For the prediction results of each branch, the cross entropy loss is constructed using the real label for training, and p 0 A self-distillation loss is constructed for each branch, and the multi-scale network is trained. The loss function of this process is shown in the following formula (1):

[0088]

[0089] This formula defines the training loss function of the multi-scale classification network. The SGD optimizer is used during training, the initial learning rate is set to 0.1, and 4 NVIDIA RTX 3090 graphics cards are used for training with 120 iterations.

[0090] The resolution prediction module is used to output the cost of allocating each image to each resolution path, generate a cost matrix C, and fix the parameters of the trained multi-scale classification network. The specific training process of this module is as follows:

[0091] C1: Construct a resolution prediction module. Considering the original intention of lightweight model, the resolution prediction module is designed as a two-layer convolutional layer module with convolution kernel sizes of 7 and 3 respectively.

[0092] C2: Input the preprocessed training set image data X into the multi-scale classification network According to the prediction results p of each branch j The cross entropy loss value obtained between the true label y and the calculation amount f corresponding to each resolution branch j , construct the ideal true cost matrix C′ ij , where λ F =0.5: the true cost matrix C′ ij Calculated by the following formula (2):

[0093]

[0094] C3: In order to ensure the lightweight of the model, the input of the resolution prediction module is the low-resolution image data x l , after entering this module, we get Using the true cost matrix C′ obtained in the previous step ij , get the pseudo label of each image resolution path The pseudo path label and the cost matrix C output by the resolution prediction module are used to construct the cross entropy loss for training. The true cost matrix C′ is directly learned using KL divergence and L2 loss. ij , the loss function used in this process is shown in the following formula (3):

[0095]

[0096] This formula defines the training loss function of the resolution prediction module, where λ K =100,λ L = 10. The SGD optimizer was used during training, the initial learning rate was set to 0.02, and one NVIDIA RTX 3090 graphics card was used for training with 60 iterations.

[0097] In the inference stage, the present invention mainly includes two inference schemes: conventional single-path dynamic resolution inference and multi-path dynamic resolution inference. The specific inference process is as follows:

[0098] D1: After preprocessing the dataset, select a low-resolution image x from the data list l Input the resolution prediction module and output the cost matrix C. Consider each image data as a supplier. Each image should be assigned to a resolution option, and define the supply vector α = 1. Set the demand vector β to represent the number of images assigned to each resolution candidate. During the training process of D2, in order to ensure that the number of samples accepted by each resolution path is the same, each batch of data is evenly distributed to each candidate resolution, and the demand vector β = N / K·1 is obtained.

[0099] D2: According to the above steps, we can obtain the cost matrix C, supply vector α and demand vector β, and then we can model the dynamic resolution prediction problem as an optimal transmission problem for solution. By finding the optimal transmission solution π corresponding to the lowest cost * , determine the resolution selection for each sample, which is solved by the Sinkhorn-Knopp iterative algorithm of the following formula (4):

[0100]

[0101] D3: Usually, according to the optimal transportation solution π * The maximum value of the corresponding distribution of each sample in determines the optimal resolution allocated. The present invention has found in the experimental process that if two relatively small resolution paths (for example, 96 and 128) are selected, the amount of calculation used is still smaller than the amount of calculation spent in the original resolution case (for example, 224).

[0102] See also Figure 4 For some samples that are difficult to identify, multiple paths can be selected for simultaneous reasoning. Although this will bring some additional computational complexity, it is still more efficient than the original case and can further improve classification accuracy. A specific judgment basis is set for this situation, which is based on the following two parts: 1) The sum of the computational complexity of multiple resolution paths should be less than the original 224 resolution computational complexity; 2) If the prediction result of a single path is sufficient to predict correctly, there is no need to use multiple paths for simultaneous reasoning. Here, the optimal transportation plan π is considered comprehensively. * The confidence corresponding to the prediction result of a single path is screened by threshold judgment. In actual situations, at most two resolution paths are allowed for inference.

[0103] At the same time, the third stage of training is introduced to enable the dynamic model to adapt to any computing requirements. According to the inference stage, the dynamic resolution allocation of the model depends on the demand vector β, and the resolution prediction allocation determines the overall average computing amount of the model. Therefore, by fine-tuning the demand vector β to adapt to the theoretical computing amount, the model can be flexibly applied to various computing-restricted environments. The specific operations are as follows:

[0104] E1: During the fine-tuning training process at this stage, the network parameters of the multi-scale classification network and the resolution prediction module are fixed, and only the demand vector β is trained. The theoretical calculation amount F is calculated based on it. theory It is expressed by the following formula (5):

[0105]

[0106] The loss function of this training phase is calculated by the following formula (5):

[0107]

[0108] Among them, γ represents the set computing limit. By changing the value of γ, multiple classification models suitable for different computing resource scenarios can be obtained. The number of iterations in this training stage is 5, and the learning rate is set to 0.001.

[0109] This embodiment is tested under ImageNet, and compared with the standard ResNet50 model in terms of accuracy, number of parameters, and amount of calculation. The classification model of the present invention is named OTD-Net. The test results are shown in Table 1 below:

[0110] Table 1: Performance comparison between OTD-ResNet-50 and baseline models

[0111]

[0112]

[0113] As can be seen from the above table, compared with the baseline ResNet-50 model, the present invention can save nearly 40% of the computational effort while still maintaining a high level of accuracy. Among them, γ is randomly selected and represents the theoretical computational effort. It can be seen that according to different γ values, the present invention can generate a set of models suitable for a variety of resource scenarios. At the same time, it can also be seen that the multi-path reasoning process further improves the accuracy level of the model at the cost of only a very small amount of additional computation.

[0114] See also Figure 5 By comparing the accuracy of this embodiment with other classic dynamic networks, it can be seen that under similar computational levels, its accuracy far exceeds that of existing dynamic multi-state classification networks.

[0115] See also Figure 6 ,Through the comparison of the resolution prediction results, it can be seen that the ,resolution prediction problem is modeled as an optimal transmission problem in the ,presentation. Compared with the previous DRNet dynamic resolution model, ,the obtained resolution prediction results are smoother and the accuracy is maintained at a ,higher level.

[0116] See also Figure 7 ,It can be seen from the dynamic resolution visualization results that for images that are relatively simple to judge with the naked eye, they can be predicted to a smaller resolution. For images with a small area of ​​interest or occlusion or the influence of other objects, they can be considered to be more difficult to identify and assigned to a larger resolution, which further proves the rationality of the dynamic resolution in the present invention.

[0117] The present invention can also be used in other convolutional neural network model structures such as ResNet-18 and MobileNetV2. A similar model structure is built, and training and testing are performed according to the above steps. The comparison results of the test results obtained with the original standard model are shown in Table 2 below:

[0118] Table 2: Performance comparison of the present invention on other model structures

[0119]

[0120]

[0121] See also Figure 8 , in the model structure with ResNet-18 as an example, it can be seen from the comparison of the inference speed under the actual Intel(R) Core(TM) i9-10920XCPU with the standard model that this model can play a more obvious role in accelerating inference in actual deployment and use.

[0122] The above embodiments prove that the present invention can improve the existing convolutional neural network model into a dynamic resolution network model based on optimal transmission, which can improve the model reasoning speed, reduce the amount of calculation, and ensure the classification accuracy of the model, and can be applied to a variety of different resource-constrained environments. The present invention adopts a method of modeling the dynamic resolution prediction side problem as an optimal transmission problem, which can also be used in other dynamic network path allocations with known solution spaces. The proposed multi-path dynamic reasoning method can also be introduced into other dynamic networks for research and discussion. Starting from the image classification task, the present invention effectively solves the problems of unbalanced resolution selection and low image classification accuracy in the dynamic resolution network, and well avoids the pattern collapse phenomenon that occurs in the image resolution selection, making the prediction selection of image resolution extremely unbalanced. Such a phenomenon results in a large number of pictures not being assigned to the optimal resolution, thereby greatly affecting the accuracy of image classification.

[0123] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.

Claims

1. A method for constructing a lightweight dynamic resolution image classification model, characterized in that The method of introducing the optimal transmission strategy into the convolutional neural network model and using the lightweight resolution prediction module to predict the optimal resolution of the image, and realizing the lightweight dynamic resolution image classification model specifically includes the following steps: Step 1: Dataset preparation and preprocessing 1-1: Collect a large amount of image data to be classified, divide it into training set and test set in an 8:2 ratio, and the image format is JPG; 1-2: Scale the above training set images according to the preset resolution list r, scale each image to K different sizes, generate the corresponding multi-scale image list, and then use random horizontal flipping to perform data enhancement on the training set data; scale the test set images according to the preset resolution list r; Step 2: Construction of multi-scale classification network 2-1: Adopt a parallel network structure on the basic classification convolutional neural network, use different batch layers to process data of different resolution sizes, and build a multi-scale classification network module; 2-2: Input the multi-scale image data list in step 1 into the multi-scale classification network, extract features for each image in the list, and obtain the prediction results; Step 3: Training of multi-scale classification network 3-1: Use the prediction results obtained in step 2 to train the multi-scale classification network, and calculate the cross entropy loss and self-distillation loss based on its prediction results and the real image labels; 3-2: Set appropriate learning rate and optimization strategy for the training process; 3-3: When the number of training iterations reaches the set threshold, stop training; 3-4: Use the test set data scaled in step 1 to test the multi-scale classification network and determine the model training results based on the accuracy; Step 4: Construction of resolution prediction module 4-1: Use two convolutional layers to build a lightweight resolution prediction module; 4-2: List the multi-scale image data in step 1 and select the small-size image x l The output result is input into the resolution prediction module constructed above, and is used for dynamic resolution prediction; Step 5: Prediction of optimal transmission dynamic resolution 5-1: Consider the N data in the same batch as N suppliers and the K candidate resolutions to be selected as K demanders, and perform optimal transmission modeling for the dynamic resolution allocation problem, that is, modeling the resolution prediction problem into an optimal transmission problem; 5-2: Use the resolution prediction module to output the cost matrix of the transmission process, which represents the amount of computation required to assign each sample to each resolution and the overhead required for the accuracy; 5-3: Calculate the optimal transportation plan based on the lowest total cost through algorithm iteration, that is, the optimal resolution selection result for each image sample; Step 6: Prediction of multi-path dynamic resolution 6-1: Based on the optimal transportation plan and threshold judgment, multiple small-resolution branches are dynamically selected for synchronous test reasoning; 6-2: Introduce a multi-branch dynamic reasoning scheme based on the resolution prediction results, and take the weighted sum of the prediction results of multiple resolution branches as the final prediction result; Step 7: Training of resolution prediction module 7-1: Fix the model parameters of the multi-scale classification network and use the output prediction results of each resolution branch in step 2 to construct the ideal cost matrix; 7-2: The resolution prediction module in step 4 is trained using the constructed loss function. The training process uses the ideal cost matrix to constrain the output results. 7-3: Set appropriate learning rate and optimization strategy for the training process; 7-4: When the number of training iterations reaches the set threshold, stop training; 7-5: Use the test set data in step 1 to test the resolution prediction module and the multi-scale classification network as a whole, and scale the resolution of the test set data according to the optimal transportation solution in step 5, and input it into the multi-scale classification network. Determine the model training result based on the accuracy rate, and obtain a lightweight dynamic resolution image classification model based on the optimal transmission problem.

2. The method for constructing a lightweight dynamic resolution image classification model according to claim 1 is characterized in that The loss function calculation of the multi-branch classification network training process in step 3 specifically includes the following steps: A1: Calculate the cross entropy loss function value between the prediction results of each branch and the true label; A2: After weighting the prediction results of each branch, calculate the KL divergence value of each branch.

3. The method for constructing a lightweight dynamic resolution image classification model according to claim 1 is characterized in that The loss function calculation process of the resolution prediction module training process in step 7 specifically includes: A1: Calculate the L2 distance between the ideal cost matrix and the output cost matrix of the resolution prediction module; A2: Calculate the KL divergence between the ideal cost matrix and the output cost matrix of the resolution prediction module; A3: Generate a "pseudo path" label based on the ideal cost matrix, and calculate the cross entropy between the cost matrix output by the resolution prediction module and the "pseudo path" label.

Images