Image classification method and device of early exit network based on optimal combination strategy

Abstract

The invention discloses an image classification method of an early exit network based on an optimal combination strategy. The method comprises the following steps: firstly, selecting and training a backbone network; and then constructing and training a multi-level multi-exit structure, and counting measurement standards and prediction correctness of data at all exits. And calculating the average accuracy rate and the average calculation cost of the model on the basis of measurement standards and prediction correctness of all outlets through grid search traversal configuration and threshold combination. And finally, determining a model expression measurement standard, analyzing to obtain an optimal solution by combining a Pareto optimal algorithm, and realizing image classification based on the obtained optimal solution. Image recognition with the best balance between accuracy and calculation cost is achieved through an improved early outlet network.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology, and in particular relates to an image recognition method that achieves an optimal balance between accuracy and computational cost by improving an original early exit network. Background Art

[0002] With the widespread application of deep learning technology in image recognition, the demand for computing resources by deep neural networks (DNNs) is increasing. For image classification, deep neural networks suffer from high computational cost and slow inference speed on resource-constrained devices. Traditional static networks cannot dynamically adjust computational load based on sample complexity, resulting in wasted computing resources. Dynamic networks can selectively execute their structures based on the sample. Early exit networks are a branch of dynamic networks. Early exit networks add internal classifiers (also known as exits) at different depths in the backbone network. If inference reaches an exit and meets certain conditions (such as a confidence level greater than a threshold), the exit result is used as the final result, bypassing subsequent layers and reducing computational cost.

[0003] Existing early exit networks are difficult to give the optimal combination at one time. Therefore, the early exit network based on the optimization combination strategy of the present invention provides a feasible solution for finding the optimal network structure to achieve image classification tasks. Summary of the Invention

[0004] The present invention provides an image classification method for an early-stage exit network based on an optimized combination strategy. The core of the method is to solve the problems of high computational cost and slow inference speed of deep neural networks on resource-constrained devices in image classification problems at a low cost by setting multiple internal classifiers at different layers and exploring the optimal combination of classifiers and thresholds.

[0005] This method proposes a systematic strategy for optimizing the design and training of early-stage exit networks, achieving an optimal balance between accuracy and computational cost through a multi-level, multi-exit architecture, a performance prediction mechanism, and a computational optimization strategy. The performance prediction mechanism estimates the average accuracy and computational cost of the model, given a given classifier combination and threshold value, by combining statistical data on the confidence and correctness of the data at all exits. The computational optimization strategy leverages the shared inputs of multiple internal classifiers at the same layer to train multiple internal classifiers simultaneously, reducing duplicate computations. Furthermore, the strategy significantly reduces computational effort during the grid search phase by storing the metrics and inference correctness of each sample at each exit.

[0006] In a first aspect, an embodiment of the present application provides an image classification method based on an early exit network of an optimized combination strategy, comprising the following steps:

[0007] Step (1): Select and train the backbone network.

[0008] Step (2): Build and train a multi-level multi-output structure.

[0009] Step (3): Statistics are collected at all outlets to measure the accuracy of the predictions.

[0010] Step (4): Traverse the configuration and threshold combinations through grid search and calculate the average model accuracy and average computational cost based on the metrics and prediction correctness at all exits.

[0011] Step (5): Determine the model performance measurement criteria, combine with the Pareto optimal algorithm, and analyze to obtain the optimal solution.

[0012] Step (6): Implement image classification based on the obtained optimal solution.

[0013] In one possible implementation, the optional backbone networks include ResNet, WideResNet, and MobileNet. A training method and training data set are determined based on the selected backbone network and training is performed.

[0014] In a possible implementation, a multi-level multi-outlet structure is constructed based on the determined outlet positions and the internal classifier structure in combination with a trained backbone network.

[0015] The backbone network is divided into several blocks and final exits based on the locations where early exits can be added. There are several alternative exit structures for each exit location.

[0016] The output of the final output is used as the teacher, and the output as the student, transferring the teacher's knowledge to the student. The final loss function is a weighted sum of the classification loss and the distillation loss. The distillation loss weight of each layer is modified, and the number of layers is combined to make the distillation loss weight of deeper layers smaller.

[0017] In one possible implementation, it is determined based on actual conditions whether to save the output of the input data after passing through each backbone network block and use it as the training input for all exits of the corresponding backbone network block, thereby using additional storage space in exchange for reduced computational costs during training.

[0018] In one possible implementation, the entropy or confidence of each exit result is used as an alternative exit criterion. Both metrics are uniformly mapped to the interval [0, 1], with a larger value indicating a more confident exit result.

[0019] For a given training dataset, as the network is forward propagated through the early exits, the metric (normalized entropy or confidence) and the prediction correctness are recorded at each exit.

[0020] In one possible implementation, a performance prediction algorithm is used to estimate the average accuracy and average computational cost of the model based on metrics and prediction accuracy at all exits, as well as classifier combinations and threshold values ​​in a grid search.

[0021] Based on the granularity of the threshold value, a threshold encompassing all exit locations is defined. A dynamic exit strategy is employed, whereby the input sample exits at the first exit that satisfies the criterion greater than or equal to the threshold. If all exits fail to meet the criteria, the input sample exits at the final exit. For a given threshold, the model's average accuracy and average computational cost are used to evaluate overall performance. A grid search is performed across all classifier combinations and threshold values. A hierarchical, coarse-grained pre-screening approach is employed to quickly identify potential areas before refining the search.

[0022] In one possible implementation, the hierarchical coarse-grained pre-screening includes the following sub-steps:

[0023] (1) Determine the initial granularity.

[0024] Determine the maximum total number of combinations allowed. When the proximity of the number of exits on each layer is less than or equal to the proximity threshold, use the same granularity G at each layer; otherwise, allocate granularity G based on the number of exits on each layer.

[0025] (2) Dynamic adjustment based on confidence distribution.

[0026] Statistical training data set confidence distribution histogram at each exit.

[0027] If the confidence distribution of the sample is highly uniform, G quantiles are taken on 0-1 as the threshold value.

[0028] If the confidence distribution of the sample is highly non-uniform, all confidences are sorted and G quantiles are taken as the threshold value.

[0029] In one possible implementation, based on the obtained average accuracy and average computational cost of the models under different classifier combinations and threshold values, a Pareto optimal algorithm is used to find the Pareto optimal combination (i.e., no other combination is better in both accuracy and computational cost).

[0030] In one possible implementation, when the Pareto front needs to be refined, pruning is performed based on the model performance and the search space is narrowed. The average accuracy and average computational cost of the model are repeatedly estimated again, and then the analysis is performed again to obtain a new optimal solution.

[0031] Prune and narrow the search space as follows: Plot the average computational cost and average model accuracy for each classifier combination and threshold value as a scatter plot. Each classifier combination corresponds to a "curve segment." When "curve segment" A is above "curve segment" B, the classifier combination corresponding to B is removed from the search space in the next grid search. Simultaneously, in the next search round, the threshold value range is narrowed based on the threshold value in the previous search round.

[0032] In one possible implementation, image classification is performed based on the obtained optimal solution as follows:

[0033] The final model only uses the internal classifiers from the optimal solution and removes any unused internal classifiers. After preprocessing, the image is fed into the final model, which then uses the internal classifiers or the final classifier to generate the image classification result.

[0034] In a second aspect, an embodiment of the present application provides an image classification device based on an early exit network of an optimized combination strategy, comprising the following modules:

[0035] Training module: Select and train the backbone network.

[0036] Building module: Build and train a multi-layer multi-output structure.

[0037] Statistics module: Statistics on the metrics and prediction correctness at all outlets.

[0038] Performance prediction module: It traverses configuration and threshold combinations through grid search and calculates the average model accuracy and average computational cost based on the metrics and prediction correctness at all exits.

[0039] Solution module: Determine the model performance measurement criteria, combine with the Pareto optimal algorithm, and analyze to obtain the optimal solution.

[0040] Classification module: classifies images based on the optimal solution obtained.

[0041] In a possible implementation, the training module determines a training method and a training data set according to the selected backbone network and performs training.

[0042] In a possible implementation, the construction module constructs a multi-level multi-outlet structure based on the determined outlet positions and the internal classifier structure in combination with a trained backbone network.

[0043] The backbone network is divided into several blocks and final exits based on the locations where early exits can be added. There are several alternative exit structures for each exit location.

[0044] The output of the final output is used as the teacher, and the output as the student, transferring the teacher's knowledge to the student. The final loss function is a weighted sum of the classification loss and the distillation loss. The distillation loss weight of each layer is modified, and the number of layers is combined to make the distillation loss weight of deeper layers smaller.

[0045] In one possible implementation, the construction module decides, based on actual conditions, whether to save the output of the input data after passing through each backbone network block and use it as the training input for all exits of the corresponding backbone network block, thereby using additional storage space in exchange for reduced computational costs during training.

[0046] In one possible implementation, the statistical module uses the entropy or confidence of each exit result as an alternative exit criterion, and uniformly maps the two metrics to the interval [0, 1], with a larger value indicating a more confident exit result.

[0047] For a given training dataset, as the network is forward propagated through the early exits, the metric (normalized entropy or confidence) and the prediction correctness are recorded at each exit.

[0048] In one possible implementation, the performance prediction module estimates the average accuracy and average computational cost of the model based on metrics and prediction accuracy at all exits, as well as classifier combinations and threshold values ​​in a grid search.

[0049] Based on the granularity of the threshold value, a threshold encompassing all exit locations is defined. A dynamic exit strategy is employed, whereby the input sample exits at the first exit that satisfies the criterion greater than or equal to the threshold. If all exits fail to meet the criteria, the input sample exits at the final exit. For a given threshold, the model's average accuracy and average computational cost are used to evaluate overall performance. A grid search is performed across all classifier combinations and threshold values. A hierarchical, coarse-grained pre-screening approach is employed to quickly identify potential areas before refining the search.

[0050] In one possible implementation, the hierarchical coarse-grained pre-screening of the performance prediction module includes the following sub-steps:

[0051] (1) Determine the initial granularity.

[0052] Determine the maximum total number of combinations allowed. When the proximity of the number of exits on each layer is less than or equal to the proximity threshold, use the same granularity G at each layer; otherwise, allocate granularity G based on the number of exits on each layer.

[0053] (2) Dynamic adjustment based on confidence distribution.

[0054] Statistical training data set confidence distribution histogram at each exit.

[0055] If the confidence distribution of the sample is highly uniform, G quantiles are taken on 0-1 as the threshold value.

[0056] If the confidence distribution of the sample is highly non-uniform, all confidences are sorted and G quantiles are taken as the threshold value.

[0057] In one possible implementation, the solution module uses a Pareto optimal algorithm to find the Pareto optimal combination (i.e., there is no other combination that is better in both accuracy and computational cost) based on the average accuracy and average computational cost of the model under different classifier combinations and threshold values.

[0058] In one possible implementation, when the Pareto front needs to be refined, pruning is performed based on the model performance and the search space is narrowed. The average accuracy and average computational cost of the model are repeatedly estimated through the performance prediction module, and then the solution module is used to analyze again to obtain a new optimal solution.

[0059] Prune and narrow the search space as follows: Plot the average computational cost and average model accuracy for each classifier combination and threshold value as a scatter plot. Each classifier combination corresponds to a "curve segment." When "curve segment" A is above "curve segment" B, the classifier combination corresponding to B is removed from the search space in the next grid search. Simultaneously, in the next search round, the threshold value range is narrowed based on the threshold value in the previous search round.

[0060] In one possible implementation, the image classification module implements image classification through the final model:

[0061] The final model only uses the internal classifiers from the optimal solution and removes any unused internal classifiers. After preprocessing, the image is fed into the final model, which then uses the internal classifiers or the final classifier to generate the image classification result.

[0062] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

[0063] The memory is used to store computer programs;

[0064] The processor is used to implement any image classification method described in this application when executing the program stored in the memory.

[0065] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any image classification method described in the present application.

[0066] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any image classification method described in the present application.

[0067] The beneficial effects of the present invention are as follows:

[0068] The present invention achieves improved performance of image classification models on the accuracy-computational cost curve through the combination of internal classifiers, while also improving the efficiency of training these classifiers. A performance prediction algorithm allows for low-computational estimation of model performance, eliminating the need for accuracy testing on a test set. A grid search and pruning algorithm are used to find the optimal solution for classifier combinations and thresholds at a manageable cost. An improved early exit network achieves image recognition with an optimal balance between accuracy and computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 : Flowchart of the method according to an embodiment of the present invention.

[0070] Figure 2 : Schematic diagram of the multi-level and multi-export structure of an embodiment of the present invention.

[0071] Figure 3 : Accuracy-computational cost diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0073] For image classification tasks, deep neural networks suffer from high inference costs. Early-egress networks address this cost issue to some extent, but their performance still needs improvement and is not flexible enough. Compared to early-egress networks with a single egress per layer, the method proposed in this paper is more flexible, adaptable to different metrics, and outperforms traditional early-egress networks in various scenarios.

[0074] Existing work lacks a method for determining and selecting exit structures. A randomly determined exit structure is not necessarily optimal, and it is even more difficult to determine the optimal combination in a single pass when multiple exit locations are possible. Furthermore, existing early exit networks often default to adding exits at fixed locations, ignoring the case where no exits are added. In reality, if an exit is too simple, the inference accuracy may not meet the required standards, or if an exit is too complex and computationally expensive, the overall model performance may degrade after adding the exit. Furthermore, most existing work evaluates the performance of a model based on a single metric, whereas the real world is often complex, with requirements varying significantly across scenarios. Even within the same scenario, the metric may frequently change over time due to factors such as policy, shifting the weighting between accuracy and computational cost. This necessitates revising the model's thresholds and configuration to optimize overall performance under the new metric. Therefore, the model must cover a wide range on the accuracy-computational cost curve to ensure sufficient adaptability. A single internal classifier struggles to achieve both a sufficiently low computational cost and a sufficiently high accuracy.

[0075] In summary, exploring the optimal solution for the setting and combination of early export network export structure and thresholds has always been a key issue in related fields.

[0076] To address these issues, this paper proposes a method that explores how to determine the optimal combination strategy to ensure optimal model performance under different criteria, given that each layer has multiple exits to choose from in an early-stage exit network architecture. This approach also considers the case where no exits are provided at a particular layer. Furthermore, techniques are employed to reduce the computational cost of training and predicting model performance.

[0077] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0078] Step (1): Select and train the backbone network.

[0079] Step (2): Build and train a multi-level multi-output structure.

[0080] Step (3): Statistics are collected at all outlets to measure the accuracy of the predictions.

[0081] Step (4): Traverse the configuration and threshold combinations through grid search and calculate the average model accuracy and average computational cost based on the metrics and prediction correctness at all exits.

[0082] Step (5): Determine the model performance measurement criteria, combine with the Pareto optimal algorithm, and analyze to obtain the optimal solution.

[0083] Step (6): Implement image classification based on the obtained optimal solution.

[0084] In a possible implementation, the backbone network includes ResNet, WideResNet, MobileNet, etc. A training method and a training data set are determined based on the selected backbone network and training is performed.

[0085] In one possible implementation, a multi-level multi-export structure is constructed and trained as follows:

[0086] Based on the determined exit locations and internal classifier structure, combined with the trained backbone network, a multi-layer, multi-exit structure is constructed. First, the exit locations are determined, selecting the end of each backbone network block. Next, the internal classifier (i.e., exit) structure is defined, using a Flatten plus fully connected architecture or a CNN-like convolutional layer, pooling, and fully connected architecture.

[0087] Assume that the size of the training dataset is N, the task is a K-classification problem, the input sample is X, and the label of the sample is Y. The multi-level multi-export structure is as follows Figure 2 As shown, the backbone network has M locations where early exits can be added. The backbone network is divided into M blocks and final exits. The computational cost of each block and final exit of the backbone network is , when i is M+1, it represents the computational cost of the final exit. There are several alternative exit structures at each exit position, using represents the j-th exit of the l-th layer, and its computational cost is Assume that the input x∈X, the corresponding true label is y. The output of x after passing through the first l backbone network blocks is , the output at the j-th exit of the l-th layer is .

[0088] Knowledge distillation is a technique that approximates the behavior of a larger, more complex "teacher" model by training a smaller "student" model. In this paper, the output of the final output is used as the teacher, and the output is used as the student, transferring the knowledge learned by the teacher to the student. The classification loss is defined as:

[0089]

[0090] in, Represents the output of an outlet.

[0091] Distillation loss is defined as:

[0092]

[0093] in, Output of the final export; Represents the output at the j-th exit of the l-th layer.

[0094] The final loss function is the weighted sum of the above two losses:

[0095]

[0096] in, Is the balance coefficient, which is used to adjust the weight between the two losses. The square term of temperature T 2 This is because when the temperature is high, the gradient will be reduced, so multiplying by T 2 To compensate for this effect. Experiments also observed that using the final output as a teacher improved the output of shallow layers, but in certain specific cases, it was ineffective for the output of deeper layers, which are more computationally expensive. Therefore, the distillation loss weight of each layer was modified, and the number of layers l was combined to make the distillation loss weight of deeper layers smaller.

[0097]

[0098] From the form of the loss function, we can see that different exit ICs in the same layer share the embedding generated in the backbone network, that is, the exit at the lth exit position receives the output ZB after the lth backbone network block of the backbone network. l As the input of the export, this data can be shared during export training, sharing the cost of the backbone network and improving the efficiency of training. Compared with the traditional IC-only method, the cost of this method at the l-th layer export is reduced by the traditional becomes , the increase is only the additional computational cost of the export, the computational efficiency .

[0099] Since IC-only does not change the backbone network parameters, ZB l The value of ZB is also unchanged. l After saving, this is used as the training input for all outputs of layer l, trading additional storage space for reduced computational cost during training. However, since the output size after the convolutional layer may be much larger than the original dataset size, the decision to use this operation depends on the actual situation.

[0100] In one possible embodiment, the present invention provides a statistical method for measuring the accuracy of standards and predictions:

[0101] The entropy or confidence of the output results logits of each exit is used as an alternative exit criterion. The higher the complexity and disorder, the less credible the predicted result. The entropy calculation formula is:

[0102]

[0103] in, The probability that the export prediction sample belongs to the kth category; the entropy value range is [0, log K].

[0104] The confidence calculation formula is:

[0105] The confidence range is .

[0106] In order to facilitate switching between different metrics and fully utilize the search space, the two metrics are uniformly mapped to the interval [0, 1], and the larger the value, the more confident the export result. The calculation formula of the normalized entropy is:

[0107]

[0108] The calculation formula of normalized confidence is:

[0109]

[0110] For a given training dataset, when forward propagating through the early exit network, at each exit (for the j-th exit of the l-th layer ) records the following statistics: measurement standard (normalized entropy value or confidence = or ), prediction accuracy ( ). For a network with m exits, the statistical matrix is ​​finally obtained: N represents the total number of training data, where for the xith (xi=1, 2, ..., N) data x, the exits of each layer and the final exit are sorted in sequence, and the jth exit or the final exit of the lth layer corresponds to the sequence number i' (i'=1, 2, ..., m+1), i' is m+1 represents the final exit; the element recorded at the i'th exit ;

[0111] In one possible implementation, the present invention provides a performance prediction algorithm, including:

[0112] The average accuracy and average computational cost of the model are estimated based on the metrics and prediction accuracy at all exits, as well as the classifier combination and threshold value in the grid search. The specific operations are as follows: define the threshold , contains the thresholds of M exit positions, and the granularity of the threshold value is G, that is, the threshold has G optional values. The dynamic exit strategy is: That is, the input sample x first satisfies Export Exit at the final exit if all exits are not satisfied. , the overall performance evaluation is as follows: Using ECl Indicates the number of samples exiting the lth layer, ET l It represents the number of correctly classified samples exiting the l-th layer. The cumulative computational cost of reasoning to the l-th layer backbone network is In particular, when a layer does not have an exit, the number of exit samples and the correct number of exit samples of the layer are set to 0, and the computational cost of the exit of the layer is set to 0.

[0113] The average accuracy of the model is:

[0114]

[0115] The average computational cost of the model is:

[0116]

[0117] All classifier combinations and threshold values ​​are traversed through grid search. Assume that the lth layer contains NIC l exits, the total number of classifier-threshold combinations in this layer is (Including the case where no classifier is set). Therefore, the total number of classifier combinations is As the number of layers increases, the computational complexity rises rapidly; since the number of exits in each layer is usually limited, the choice of granularity G has a decisive influence on the overall complexity.

[0118] In one possible implementation, to optimize grid search efficiency and efficiently explore optimal solutions, the present invention provides a Hierarchical Coarse-to-Fine Search (HCFS) method. Its core concept is to first quickly locate potential areas and then refine the search. This method includes:

[0119] (1) Determine the initial granularity G.

[0120] Let the maximum total number of combinations allowed be K max (e.g. 10 4 ), then the following conditions must be met: .

[0121] If the number of NICs at each layer l Similar to Th is the NIC proximity threshold, which is 2 in the embodiment. Then the same granularity is used in each layer. .

[0122] If NIC l Big differences, such as , then the layer with more outlets (such as shallow layer) is given priority to allocate smaller G l . Take the first layer

[0123] (2) Dynamic adjustment based on confidence distribution.

[0124] Statistical training data set confidence distribution histogram at each exit.

[0125] If the confidence distribution of the sample is highly uniform, G quantiles are taken on 0-1 as the threshold value.

[0126] If the confidence distribution of the samples is highly non-uniform, more sampling points are desirable in densely sampled areas. Using a uniform threshold value in the interval [0, 1] can lead to rapid changes in model accuracy and computational cost in densely sampled areas. The solution is to sort all confidence levels and then use G quantiles as the threshold value.

[0127] For all classifier combinations and threshold values ​​in the grid search, combined with the metrics and prediction accuracy at all exits obtained in step (3), the corresponding average model accuracy and average computational cost are calculated.

[0128] In one possible implementation, based on the average accuracy and average computational cost of the models under different classifier combinations and threshold values, a Pareto optimal algorithm is used to find the Pareto optimal combination (i.e., no other combination is better in both accuracy and computational cost).

[0129] In one possible implementation, if the Pareto front is to be refined, pruning is performed based on the model performance and the search space is narrowed. The performance prediction algorithm is then used again to estimate the average accuracy and average computational cost of the model, and a new analysis is performed to obtain a new optimal solution.

[0130] The average computational cost and average model accuracy for each classifier combination and threshold value are plotted as a scatter plot. For points on the same classifier combination, the computational cost and model accuracy change smoothly as the threshold value changes, meaning each classifier combination corresponds to a "curve segment." Curve segments may intersect. Because the performance prediction algorithm uses a grid search, the points on the curve segments are discrete, making it difficult to accurately locate the intersection of two curve segments. To refine the Pareto front and further determine the exact location of the intersection, the performance prediction algorithm is used again to estimate the average model accuracy and average computational cost. Before doing so, pruning is performed to reduce the search space for the next grid search round. The pruning method is as follows: Based on the plotted scatter plot, if "curve segment A" is above "curve segment B," this indicates that the classifier combination corresponding to A performs better than the classifier combination corresponding to B in the corresponding computational cost range. In this case, the classifier combination corresponding to B is removed from the search space in the next grid search round. At the same time, in the next round of search, the threshold value range is narrowed according to the threshold value in the previous round of search results, and a fine-grained search is performed to make the final Pareto optimal curve more in line with the actual optimal situation.

[0131] In one possible implementation, the final model uses only the internal classifiers from the optimal solution and removes any unused internal classifiers. The pre-processed image is then fed into the final model, which then obtains the image classification result from either the internal classifier or the final classifier.

[0132] Through the above steps and technical means, this invention provides a flexible and efficient early exit network design solution that can not only adapt to the needs of various application scenarios, but also achieve an optimal balance between accuracy and computational cost. This innovation provides strong support for the practical deployment of deep neural networks, especially showing great potential in resource-constrained environments. In addition, this method also lays the foundation for future research directions such as automated search for exit structures, cross-modal early exit mechanisms, and customized optimization for specific hardware architectures.

[0133] After obtaining the Pareto front, substitute all points in the Pareto front into multiple metrics to find the optimal solution.

[0134] See also Figure 1 , Figure 1 This is a flow chart of an image classification method according to an embodiment of the present application, including:

[0135] Step (1): Select and train the backbone network as follows:

[0136] Use CIFAR10 or CIFAR100 as the dataset, ResNet56 as the backbone network, train the model until convergence or use a pre-trained model.

[0137] Step (2): Build and train a multi-level multi-export structure as follows:

[0138] Select four positions before and after the three residual blocks of ResNet56 as the exit positions, and define the internal classifier, i.e., the exit, based on the vector shape and total number of classifications of the backbone network at these positions. The definition method selected in the embodiment is Flatten layer plus MLP. Assume that the size of the Flatten layer is S f , the number of hidden layers is N, the task is a K classification task, then log(MLP hidden layer shape) takes logS f For ease of storage, the N equally divided points between logK are converted to values ​​closest to powers of 2. The definition of outputs for structures similar to CNNs needs to be tailored to specific circumstances and will not be expanded here.

[0139] Input the dataset data into the backbone network and get the output of the middle layer of the backbone network , as The input trains all the outputs of a layer at the same time. The loss function is , use SGD to update the export parameters.

[0140] Step (3): Statistically calculate the metrics and prediction accuracy at all outputs. Specifically, the data in the dataset is input into a multi-level, multi-output structure. Logits are output at each internal classification. The normalized entropy and confidence level of the output are calculated, and the class corresponding to the maximum value of the output probability is recorded as the correctness of the inference. The data is shown in the following table.

[0141] <![CDATA[IC1 1 ]]> <![CDATA[IC1 2 ]]> <![CDATA[IC2 1 ]]> <![CDATA[IC2 2 ]]> <![CDATA[IC2 3 ]]> Final_IC 1 (0.8，T) (0.7，T) (0.18，F) (0.7，F) (0.15，F) (-1，F) 2 (0.5，T) (0.16，F) (0.14，F) (0.12，F) (0.9，T) (-1，T)

[0142] The final exit does not need to record confidence or entropy, because no matter what the confidence is, the remaining samples will exit at the final exit.

[0143] In a possible embodiment, the metrics recorded simultaneously may not be limited to one, and using different metrics at different layers is also a feasible solution. However, to meet the requirements, the customized metrics may be normalized like entropy and confidence and not depend on the results of other layers.

[0144] Step (4): Traverse the configuration and threshold combinations through grid search, and calculate the average accuracy and average computational cost of the model based on the metrics and prediction accuracy of all exits obtained in step (3). The specific approach is to first determine a coarser threshold granularity G, traverse the exit index and threshold combination of the lth layer, and the total number of cases is the product of G and the total number of exits in the lth layer. Considering the case where no classifier is set, add 1 to this result. Then multiply the number of combinations of all layers to get the total number of combinations. In the embodiment, M=4, and each layer NICl =3, K max =10000, then:

[0145] (divided into 4 equal parts)

[0146] Since both confidence and entropy are classified into the range of 0 to 1, the threshold value is the quarter point of the range of 0 to 1, that is, [0.25, 0.5, 0.75]. There are three exits on each layer, so the exit index of the first layer is one of [0, 1, 2]. Therefore, the combination of exit index and threshold is (0, 0.25) (1, 0.25) (2, 0.25) (0, 0.5) (1, 0.5) (2, 0.5) (0, 0.75) (1, 0.75) (2, 0.75) nine kinds plus (-1, -1) (indicating that the classifier is not used) for a total of 10 kinds. The total number of combinations of the four layers is 10 4 Each type contains four export indexes and export thresholds, in the form of [first-level index, second-level index, third-level index, fourth-level index][first-level threshold, second-level threshold, third-level threshold, fourth-level threshold].

[0147] Step (5): Combine the Pareto front and the model performance metric to analyze the optimal structure. If you want to make the Pareto front more refined, prune the model based on the performance and narrow the search space. Repeat step (4) and analyze again to obtain a new optimal solution.

[0148] One solution to Pareto optimality is to set a variable to record the highest accuracy, initially set to 0. All points are sorted by computational cost, and all points are traversed in ascending order of computational cost. If the accuracy of a point is greater than the highest accuracy, the point is added to the Pareto front and the highest accuracy is updated. The quality of the exits on the same layer can be roughly seen in the accuracy-computational cost graph, as shown in the following example: Figure 3 As shown, the search space is narrowed down in the next round of grid search.

[0149] In this example, we take the CIFAR10 dataset as an example and first determine the model performance metric F( , ), where the average accuracy of the model The higher the average computational cost of the model The lower the value of the model performance metric, the higher the value should be. Secondly, the points in the Pareto front are substituted into the model performance metric to obtain the optimal classifier combination and threshold configuration under the model performance metric.

[0150] Step (6): Implement image classification based on the obtained optimal structure.

[0151] If the model performance metric remains constant over time, the final model uses only the internal classifiers from the optimal solution, removing any unused internal classifiers. If the metric changes over time, all internal classifiers involved in the Pareto front are retained, and the optimal solution is re-searched when the metric changes. Once the metric is determined, the model behaves like a conventional early-stage network. During inference, the preprocessed image is fed into the model, and the model generates inference results from either the internal classifiers or the final classifier.

[0152] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. An image classification method based on an early exit network with an optimized combination strategy, characterized in that: The steps include: Step (1): Select and train the backbone network; Step (2): Build and train a multi-level multi-output structure; Step (3): Statistics on the measurement standards and prediction accuracy at all outlets; Step (4): Traverse the configuration and threshold combinations through grid search and calculate the average model accuracy and average computational cost based on the metrics and prediction correctness at all exits; Step (5): Determine the model performance measurement standard, combine it with the Pareto optimal algorithm, and analyze to obtain the optimal solution; Step (6): Implement image classification based on the obtained optimal solution.

2. The image classification method based on the early exit network of the optimization combination strategy according to claim 1 is characterized in that: Determine the training method and training data set based on the selected backbone network and perform training.

3. The image classification method based on the early exit network of the optimization combination strategy according to claim 1 or 2, characterized in that: Construct a multi-level multi-export structure based on the determined exit location and internal classifier structure combined with the trained backbone network; The backbone network is divided into several blocks and final exits based on the locations where early exits can be added; each exit location has several alternative exit structures; The output of the final export is used as the teacher, and the export is used as the student to transfer the knowledge learned by the teacher to the student. The final loss function is the weighted sum of the classification loss and the distillation loss. The proportion of the distillation loss of each layer is modified, and the number of layers is combined to make the weight of the deep distillation loss smaller.

4. The image classification method based on the early exit network of the optimization combination strategy according to claim 1 is characterized in that: The entropy or confidence of each export output result is used as an alternative exit criterion; the two measurement criteria are uniformly mapped to the interval [0, 1], and the larger the value, the more confident the export result; For a given training dataset, we forward propagate through the early exit network, recording the metrics and prediction correctness at each exit.

5. The image classification method based on the early exit network of the optimization combination strategy according to claim 1 is characterized in that: A performance prediction algorithm is used to estimate the average accuracy and average computational cost of the model based on the metrics and prediction accuracy at all exits, as well as the classifier combination and threshold value in the grid search. A threshold that includes all exit locations is defined based on the granularity of the threshold value. A dynamic exit strategy is adopted, that is, the input sample exits at the first exit that meets the measurement criteria greater than or equal to the threshold. If all exits do not meet the criteria, the input sample exits at the final exit. For a given threshold, the model's average accuracy and average computational cost are used to evaluate the overall performance. All classifier combinations and threshold values ​​are traversed through grid search. A hierarchical coarse-grained pre-screening is used to quickly locate potential areas before refining the search.

6. The image classification method based on the early exit network of the optimization combination strategy according to claim 5 is characterized in that: The stratified coarse-grained pre-screening process includes the following sub-steps: (1) Determine the initial granularity division; Determine the maximum total number of combinations allowed; when the degree of proximity of the number of exits on each layer is less than or equal to the proximity threshold, use the same granularity G on each layer; Otherwise, the allocation granularity G is allocated according to the number of outlets at each layer; (2) Dynamic adjustment based on confidence distribution; Calculate the confidence distribution histogram of the training data set at each exit; If the confidence distribution of the sample is highly uniform, G quantiles are taken on 0-1 as the threshold value; If the confidence distribution of the sample is highly non-uniform, all confidences are sorted and G quantiles are taken as the threshold value.

7. The image classification method based on the early exit network of the optimization combination strategy according to claim 1 is characterized in that: Based on the obtained average accuracy and average computational cost of the models under different classifier combinations and threshold values, the Pareto optimal combination is obtained using the Pareto optimal algorithm.

8. The image classification method based on the early exit network of the optimization combination strategy according to claim 1 is characterized in that: Image classification is implemented based on the obtained optimal solution, as follows: The final model only uses the internal classifiers in the optimal solution and removes the unused internal classifiers. The preprocessed image is input into the final model, and the model obtains the image classification result at the internal classifier or the final classifier.

9. An image classification device based on an early exit network of an optimized combination strategy, characterized in that: Includes the following modules: Training module: select and train the backbone network; Building module: build and train multi-level and multi-output structures; Statistics module: Statistics on the measurement standards and prediction accuracy at all outlets; Performance prediction module: traverses configuration and threshold combinations through grid search and calculates the average model accuracy and average computational cost based on the metrics and prediction correctness at all exits; Solution module: Determine the model performance measurement standard, combine it with the Pareto optimal algorithm, and analyze and obtain the optimal solution; Classification module: classifies images based on the optimal solution obtained.

10. An electronic device, characterized in that: including processor and memory; The memory is used to store computer programs; The processor is used to implement the image classification method of the early exit network based on the optimized combination strategy described in any one of claims 1-8 when executing the program stored in the memory.

Images