Multi-dimensional composite scaling method for convolutional neural network

Abstract

The invention discloses a multi-dimensional composite scaling method for a convolutional neural network, and mainly solves the problems of parameter redundancy and slow search during network scaling in the prior art. According to the implementation scheme, a baseline network is selected, and the FLOPs upper limit of the baseline network is set; selecting scaling relation constraints, and determining network resolution, width and depth scaling factors; adjusting the baseline network to generate a plurality of candidate networks; constructing a delay evaluation model of the candidate network; screening a candidate network by using the model and network precision to obtain an optimal scaling configuration network resolution scaling factor er, a network width scaling factor ew and a network depth scaling factor ed of the baseline network; and scaling the baseline network according to the optimal scaling configuration. According to the method, the parameter redundancy of the scaling network is reduced, higher parameter efficiency is realized, the generation speed of the candidate network is improved, the process of evaluating the candidate network architecture is accelerated, the overall search efficiency is improved, and the method can be used for deploying the neural network model to hardware equipment with limited resources.

Description

Technical Field

[0001] The present invention belongs to the field of neural network technology, and further relates to a multi-dimensional composite scaling method, which can be used to deploy a neural network model to a hardware device with limited resources. Background Art

[0002] Convolutional neural network is a feedforward neural network with deep structure and convolutional calculation. It has been widely used in intelligent applications such as image recognition and target detection. With the diversification of workloads, neural networks need to run efficiently on different devices, which requires the network to have not only high precision but also low latency and low energy consumption. Therefore, it is necessary to scale the width, depth and resolution dimensions of the network according to the resources of the hardware device. Traditional network scale adjustment methods are mainly divided into two categories: single-dimensional scaling and unified scaling. Single-dimensional scaling only adjusts one of the width, depth or resolution, while unified scaling simultaneously expands or reduces the width, depth and resolution of the network. These two methods do not consider the correlation between different dimensions of the network, which will lead to waste of hardware resources and loss of network accuracy.

[0003] The patent document with application number CN202080010508.3 discloses a "composite model scaling method for a neural network", whose implementation scheme is: (1) receiving a baseline architecture of the neural network, wherein the baseline architecture has been trained to perform the specific machine learning task, and wherein the baseline architecture has a network width dimension, a network depth dimension, and a resolution dimension; (2) receiving data defining a composite coefficient, wherein the composite coefficient controls additional computing resources used to scale the baseline architecture; (3) performing a search to determine a baseline width coefficient, a baseline depth coefficient, and a baseline resolution coefficient, wherein the baseline width coefficient, the baseline depth coefficient, and the baseline resolution coefficient specify how to assign the additional computing resources to the network width dimension, the network depth dimension, and the resolution dimension of the baseline architecture, respectively; (4) determining a width coefficient, a depth coefficient, and a resolution coefficient based on the baseline width coefficient, the baseline depth coefficient, the baseline resolution coefficient, and the composite coefficient; (5) generating the final architecture that scales the network width dimension, the network depth dimension, and the resolution dimension of the baseline architecture based on the corresponding width coefficient, depth coefficient, and resolution coefficient. This method uses compound coefficients to linearly and uniformly scale the network width, depth, and resolution of the baseline model. Although it can avoid spending a lot of computing resources on hyperparameter adjustment, it has three shortcomings: First, this method uses unified compound scaling, that is, it simultaneously expands or reduces the network width, depth, and resolution of the baseline model. It does not consider the quantitative relationship between network width and network resolution during network scaling, resulting in parameter redundancy in the scaled network and failure to achieve optimal parameter efficiency. Second, this method only constrains the search space by the network's computational FLOPs, resulting in an overly large search space for the scaled network, which affects the speed of generating candidate networks. Third, this method obtains the performance score of the candidate network by training on specific machine learning tasks and measuring it on specific hardware. It lacks an efficient delayed evaluation method for the candidate network architecture, resulting in a slow process of evaluating the candidate network architecture and reducing the overall search efficiency. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a multi-dimensional composite scaling method for convolutional neural networks to reduce parameter redundancy of the scaled network, achieve optimal parameter efficiency, increase the speed of generating candidate networks, speed up the process of evaluating candidate network architectures, and improve overall search efficiency.

[0005] The technical idea for achieving the purpose of the present invention is to input high-resolution images and reduce the network width to effectively improve the accuracy with less increase in model parameters; to quantify the relationship between network width and resolution and use this relationship to constrain the search space to find a scaling model with smaller parameters but higher accuracy, thereby improving the parameter efficiency of the network; to analyze the different computing properties of each layer of the neural network, establish the relationship between the network inference delay and the different dimensions of the network, construct a delay prediction model, and accurately measure the impact of dimensional scaling on the inference speed, thereby guiding the dimensional scaling strategy of the network.

[0006] The technical solution for achieving the purpose of the present invention comprises the following steps:

[0007] (1) Select an existing trained baseline network and set a scaled target floating-point operation FLOPs upper limit for it;

[0008] (2) Set the scaling relationship set of network resolution and width to {0.9, 1.0, 1.1, 1.2, 1.3}, and select a scaling relationship constraint f(e w ,e r ) value, when 0.3≤e r Randomly select a network resolution scaling factor e in the range ≤ 2.8 r ;

[0009] (3) According to the scaling constraint f(e w ,e r ) and the floating point operation FLOPs upper limit constraint, the network width scaling factor e is calculated w and the depth scaling factor e d ;

[0010] (4) Use the network width scaling factor e w , depth scaling factor e d and resolution scaling factor e r , adjust the original width w, depth d, and resolution r of the baseline network to get the scaled width w new 、Depth w and resolution r new , generate a new candidate network architecture;

[0011] (5) Select other unused scaling relationship constraints f(e) from the scaling relationship set w ,e r ), at 0.3≤e r Randomly select an unused network resolution scaling factor e in the range ≤ 2.8 r ; Use the scaling constraint f(e w ,e r) and the network resolution scaling factor e r Generate multiple new candidate networks Nets;

[0012] (6) Calculate the computationally intensive layer inference latency T of all candidate networks based on the network depth d, width w, and resolution r CB and the inference latency T of the memory-intensive layer MB , the relationship between the network dimension and the overall reasoning delay is established by combining the two delays, and the network delay evaluation model R is obtained pro ;

[0013] (7) Using network delay to evaluate model R pro The best scaling configuration for the baseline network is obtained by comparing the measured network accuracy:

[0014] (7a) Evaluating model R using network delay pro Calculate the inference delay Run(Nets) of all candidate networks;

[0015] (7b) Retrain the candidate network using the training set and validation set of the existing dataset, and use the test set to infer the accuracy Acc(Nets) of all candidate networks on the trained candidate network;

[0016] (7c) Calculate the evaluation value of all candidate networks Tar = Acc(Nets) - γ × Run(Nets), where γ is the calculation weight of the inference delay Run(Nets);

[0017] (7d) Select the network N with the largest Tar among all candidate networks as the best scaling network. The resolution scaling factor of the best scaling network is e r , width scaling factor e w and the depth scaling factor e d This is the optimal scaling configuration for the baseline network;

[0018] (8) According to the optimal scaling configuration parameter e r 、e w 、e d Scaling the baseline network.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] First, the present invention fully considers the interaction between different dimensions of the network, quantifies the relationship between network width and network resolution, and constrains f(e w ,e r ) and the floating point operation FLOPs upper limit to generate the width scaling factor e of the baseline network w , depth scaling factor e d and resolution scaling factor er Compared with the single-dimensional scaling method and the unified scaling method used in the prior art, the accuracy of the network can be improved without significantly increasing the computational complexity, the redundancy of network parameters can be effectively reduced, and better parameter efficiency can be achieved.

[0021] Second, the present invention analyzes the inference delay of the computationally intensive layer and the memory intensive layer, and integrates the two types of delays to establish the relationship between the network dimension and the overall inference delay, and obtains the network delay evaluation model R pro ; and use network delay to evaluate model R pro The optimal scaling configuration is performed for the baseline network based on the measured network accuracy. Compared with the existing technology that only uses a single indicator such as FLOPs to evaluate the inference delay of the scaling model, the evaluation error of the network inference delay is reduced, the impact of dimensional scaling on the network inference efficiency can be accurately measured, the process of evaluating candidate network architectures is accelerated, and the search efficiency of the scaling configuration is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart for implementing the present invention. DETAILED DESCRIPTION

[0023] The embodiments and effects of the present invention are further described in detail below with reference to the accompanying drawings.

[0024] Reference Figure 1 , the present invention provides the following two embodiments:

[0025] Embodiment 1: Scaling of network for image classification task.

[0026] The implementation steps of this example are as follows:

[0027] Step 1: Select a baseline network and set the FLOPs upper limit.

[0028] According to the requirements of the image classification task, the trained baseline network EffectiveNet-B0 is selected from the existing publicly available baseline networks. The baseline network includes a computationally intensive layer and a memory intensive layer.

[0029] The floating-point operation volume (FLOPs) upper limit is set for the selected baseline network based on the application requirements of the image classification task, the hardware processing capabilities, and the energy cost requirements. The application requirements include the accuracy and inference time requirements of the application; the hardware processing capabilities include the computing and memory resources of the hardware; and the energy cost requirements include the network operation power consumption and the network operation cost.

[0030] This example assumes but is not limited to an upper limit of FLOPs of 210M.

[0031] Step 2: Select scaling constraints and determine network resolution, width, and depth scaling factors.

[0032] 2.1) Set the scaling relationship set of network resolution and width to {0.9, 1.0, 1.1, 1.2, 1.3}, and select a scaling relationship constraint f(e w ,e r ) value, when 0.3≤e r Randomly select a network resolution scaling factor e in the range ≤ 2.8 r ;

[0033] 2.2) According to the scaling constraint f(e w ,e r ) and the resolution scaling factor e r Calculate the width scaling factor e of the network w :

[0034]

[0035] Among them, 0.3≤e w ≤2.8, if e w If there is no solution in this range, reselect the network resolution scaling factor e r And the scaling constraint f(e w ,e r );

[0036] 2.3) Calculate the network's depth scaling factor e based on the FLOPs upper limit d :

[0037] 2.3.1) Calculate the floating point operations FLOPs of a single convolutional layer:

[0038] FLOPS(Layer)=w 2 r 2 k 2

[0039] Among them, w represents the width of the network layer, that is, the number of convolution filters, k is the size of the filter, and r represents the spatial resolution of the layer;

[0040] 2.3.2) The network depth scaling factor e is obtained by solving the following inequality group: d The value range of is:

[0041]

[0042] Where d is the depth of the baseline network, e d *d represents the depth of the scaling network, Indicates d i FLOPs of the convolutional layer;

[0043] 2.3.3) From the solved depth scaling factor e d Randomly select a value within the selection range as the depth scaling factor e d .

[0044] Step 3: Adjust the baseline network to generate candidate network architecture.

[0045] 3.1) Use network width scaling factor e w , depth scaling factor e d and resolution scaling factor e r , adjust the original width w, depth d, and resolution r of the baseline network, and get the scaled width w by the following formula new , depth d new and resolution r new :

[0046] d new =e d *d

[0047] w new =e w *w

[0048] r new =e r *r

[0049] 3.2) According to the depth d of the new candidate network new , width w new , resolution r new Adjust the baseline network to generate a candidate network.

[0050] Step 4: Select other scaling constraints and network resolutions to generate multiple new candidate networks.

[0051] 4.1) Select other unused scaling constraints f(e) from the scaling set w ,e r ), at 0.3≤e r Randomly select an unused network resolution scaling factor e in the range ≤ 2.8 r ;

[0052] 4.2) Using scaling relations to constrain f(e w ,e r ) and the network resolution scaling factor e r Generate multiple e w ,e r ,e d Scaling configuration combinations and adjusting baseline networks to generate multiple new candidate networks Nets.

[0053] Step 5: Build a network delay evaluation model.

[0054] 5.1) Calculate the overall delay of the network:

[0055] 5.1.1) Calculate the computationally intensive layer inference latency T of the candidate network CB :

[0056]

[0057] Among them, Layer ops =w 2 r 2 k 2 represents the floating point operations FLOPs required for a single convolutional layer in a neural network, w represents the network width, k is the size of the filter, r represents the spatial resolution of the layer, P max Indicates the maximum computing power of the GPU per second, U til represents resource utilization, P max *U til Represents the effective computing power of the hardware, denoted as constant b;

[0058] 5.1.2) Calculate the inference latency T of the memory-intensive layer of the candidate network MB :

[0059]

[0060] Among them, D io =wr 2 M represents the amount of data of intermediate activation features transmitted between network layers. BW Indicates the maximum memory bandwidth of the GPU, M BW *U til represents the effective memory bandwidth of the hardware, denoted as a constant;

[0061] 5.2) According to T CB and T MB Calculate the total inference latency T of the entire neural network Net :

[0062] T Net =d1T CB +d2T MB

[0063]

[0064] Where d1 and d2 represent the number of memory-intensive layers and computation-intensive layers, respectively; w1 and w2 represent the average channel width of memory-intensive layers and computation-intensive layers, respectively; r1 and r2 represent the resolution of memory-intensive layers and computation-intensive layers, respectively;

[0065] 5.3) Total inference delay T for step 5.2) Net To organize:

[0066] Let d = d1 + d2 represent the total number of network layers, that is, the depth of the network;

[0067] Let the total average channel width of the network be That is, the network width;

[0068] Let r = r1 + r2 represent the input resolution of the network;

[0069] Substituting d, w, and r into the above formula and rearranging it, we get:

[0070]

[0071] 5.4) The network inference time T is obtained from the result of step 5.3) Net The complexity of is related to the network dimensions d, w, and r as follows:

[0072] O(T Net (d,w,r))=O(dw 1.5 r 2 )

[0073] Where O represents the relationship between the network inference time and the network depth d, width w and resolution r in the worst case;

[0074] 5.5) Use the relationship in step 5.4) to get the delayed evaluation model R pro :

[0075] R pro =dw 1.5 r 2 .

[0076] Step 6: Perform the best scaling configuration for the selected baseline network.

[0077] 6.1) Evaluating Model R using Network Latency pro Calculate the inference delay Run(Nets) of all candidate networks;

[0078] The depth d of each candidate network i , width w i , resolution r i Substitute the delayed evaluation model R pro Calculate the inference delay of all candidate networks:

[0079] Run(Nets) = {R pro (d i ,w i ,r i )|i∈Nets};

[0080] 6.2) Retrain the candidate network and use the test set to infer the accuracy Acc(Nets) of all candidate networks on the trained candidate network:

[0081] 6.2.1) Divide the dataset:

[0082] The ImageNet-100 dataset is obtained from the public network, and data enhancement processing such as random cropping, random inversion, and color jitter is performed on it. The enhanced dataset is then divided into training set, validation set, and test set in a ratio of 7:2:1.

[0083] 6.2.2) Use the deep learning framework PyTorch to build a candidate network model and define a training loop;

[0084] 6.2.3) Initialize network parameters, use gradient descent method to process training set data batch by batch and update network weights, use validation set to evaluate network performance and adjust network parameters after each training cycle;

[0085] 6.2.4) Repeat step 6.2.3) until the network converges and stops training to obtain a trained network model;

[0086] 6.2.5) Repeat steps 6.2.2) to 6.2.4) for all candidate networks to obtain all trained candidate networks;

[0087] 6.2.6) Input each sample of the test set into the trained candidate network and output the prediction result of each sample;

[0088] 6.2.7) Compare the predicted results of all samples with their true labels, and record the samples whose predicted results and true labels are the same as those with correct predictions;

[0089] 6.2.8) Calculate the classification accuracy of all test samples to obtain the accuracy of a candidate network;

[0090] 6.2.9) Repeat steps 6.2.6) to 6.2.8) for all trained candidate networks to obtain the accuracy Acc(Nets) of all candidate networks;

[0091] 6.3) Calculate the evaluation value of all candidate networks Tar = Acc(Nets) - γ × Run(Nets), where γ is the calculation weight of the inference delay Run(Nets);

[0092] 6.4) Select the network N with the largest Tar among all candidate networks as the best scaling network according to the following formula:

[0093]

[0094] Among them, Nets represents all candidate networks, and γ is 5×10 -6 .

[0095] 6.5) Scale the resolution of the optimal scaled network by a factor e r , width scaling factor e w and the depth scaling factor e d As the optimal scaling configuration parameter of the selected baseline network, according to the optimal scaling configuration parameter e r 、e w 、e d Scaling the baseline network.

[0096] Example 2: Scaling of the network for target detection tasks.

[0097] The implementation steps of this example include the following:

[0098] Step 1: Select a baseline network and set the FLOPs upper limit.

[0099] According to the requirements of the target detection task, the trained baseline network EfficientNet-B0 is selected from the existing publicly available baseline networks as the feature extractor of the target detection network SSDLite;

[0100] This example assumes but is not limited to a FLOPs upper limit of 400M;

[0101] Step 2: Select scaling constraints and determine network resolution, width, and depth scaling factors.

[0102] The implementation of this step is the same as step 2 of embodiment 1.

[0103] Step 3: Adjust the baseline network to generate a candidate network architecture.

[0104] The implementation of this step is the same as step 3 of embodiment 1.

[0105] Step 4: Select other scaling constraints and network resolutions to generate multiple new candidate networks.

[0106] The implementation of this step is the same as step 4 of the first embodiment.

[0107] Step 5: Build a network delay evaluation model.

[0108] The implementation of this step is the same as step 5 of the first embodiment.

[0109] Step 6: Perform the best scaling configuration for the selected baseline network.

[0110] 6-1) Using network delay to evaluate model R pro Calculate the inference delay Run(Nets) of all candidate networks;

[0111] The depth d of each candidate network i , width w i , resolution r i Substitute the delayed evaluation model R pro Calculate the inference delay of all candidate networks:

[0112] Run(Nets) = {R pro (d i ,w i ,r i )|i∈Nets};

[0113] 6-2) Retrain the candidate network and use the test set to infer the accuracy Acc(Nets) of all candidate networks on the trained candidate network:

[0114] 6-2a) Divide the data set:

[0115] The MSCOCO dataset is obtained from the public network, and data enhancement processing such as random cropping, random inversion, and color jitter is performed on it. The enhanced dataset is then divided into training set, validation set, and test set in a ratio of 7:2:1.

[0116] 6-2b) Use the deep learning framework PyTorch to build a candidate network model and define a training loop;

[0117] 6-2c) Initialize network parameters, use gradient descent method to process training set data batch by batch and update network weights, and use validation set to evaluate network performance and adjust network parameters after each training cycle;

[0118] 6-2d) Repeat step 6-2c) until the network converges and stops training to obtain a trained network model;

[0119] 6-2e) Repeat steps 6-2b) to 6-2d) for all candidate networks to obtain all trained candidate networks;

[0120] 6-2f) Input each sample of the test set into the trained candidate network and output the target detection result of each sample, including bounding box, category and confidence;

[0121] 6-2g) For each predicted bounding box, calculate its intersection over union (IoU) with the true bounding box and compare it with the set threshold of 0.5:

[0122] If the IoU is higher than this threshold, the bounding box is determined to be correctly detected and step 6-2h is executed;

[0123] Otherwise, this bounding box is considered as an incorrect detection;

[0124] 6-2h) According to the number of correctly predicted samples and the total number of samples, calculate the AP of each category and the average precision mAP of multiple categories respectively, and take it as the accuracy of a candidate network;

[0125] 6-2i) Repeat steps 6-2f) to 6-2h) for all trained candidate networks to obtain the accuracy Acc(Nets) of all candidate networks;

[0126] 6-3) Calculate the evaluation value of all candidate networks Tar = Acc(Nets) - γ × Run(Nets), where γ is the calculation weight of the inference delay Run(Nets);

[0127] 6-4) Select the network N with the largest Tar among all candidate networks as the best scaling network according to the following formula:

[0128]

[0129] Among them, Nets represents all candidate networks, and γ is 5×10 -6 .

[0130] 6-5) Scale the resolution of the optimal scaled network by factor e r , width scaling factor e w and the depth scaling factor e d As the optimal scaling configuration parameter of the selected baseline network, according to the optimal scaling configuration parameter e r 、e w 、e d Scaling the baseline network.

[0131] The effect of the present invention can be further illustrated by the following simulation experiment:

[0132] 1. Simulation Experiment Conditions

[0133] 1) The experimental data set uses ImageNet-100, which includes 55,000 training images with a total of 100 categories. The images are subjected to data enhancement processing such as random cropping, random inversion, and color jittering. The images are divided into training set, validation set, and test set in a ratio of 7:2:1.

[0134] 2) Build a network model based on the deep learning framework PyTorch and the PyTorch Image Models library;

[0135] 3) The experiments were conducted on an Nvidia RTX 2080 GPU, using the RMSProp optimizer with momentum 0.9 and decay 0.9, and the weight decay was set to 10 -5, the batch normalization momentum is 0.99, and the training is done for 400 epochs with a batch size of 384. The initial learning rate is set to 0.048 and decreased by 0.97 every 2.4 epochs.

[0136] 2. Simulation experiment content

[0137] Simulation experiment 1, under the above simulation experiment conditions, the floating point calculation amount FLOPs is limited to ≤ 210M, and the baseline network EfficientNet-B0 is scaled by using the first embodiment of the present invention and the existing single dimension scaling and unified dimension scaling methods, wherein the single dimension scaling scales the depth d, width w and resolution r of the baseline network respectively to obtain three scaled networks, and the other methods each obtain a scaled network, and the floating point calculation amount FLOPs, network accuracy and inference delay of each scaled network are calculated. The results are shown in Table 1:

[0138] Table 1 Performance comparison of different methods of scaling networks under the limit of floating point calculation FLOPs ≤ 210M

[0139]

[0140] As can be seen from Table 1, under the limitation of floating-point computing amount FLOPs≤210M, the scaling network obtained by using the present invention is superior to the scaling network generated by using the single-dimensional scaling method and the unified-dimensional scaling method in terms of inference delay and network accuracy.

[0141] Simulation experiment 2, under the above simulation experiment conditions, the floating point calculation amount FLOPs is limited to ≤ 600M, and the baseline network EfficientNet-B0 is scaled using the first embodiment of the present invention and the existing single dimension scaling and unified dimension scaling methods, wherein the single dimension scaling scales the depth d, width w and resolution r of the baseline network respectively to obtain three scaled networks, and the other methods each obtain a scaled network, and the floating point calculation amount FLOPs, network accuracy and inference delay of each scaled network are calculated. The results are shown in Table 2:

[0142] Table 2 Performance comparison of different methods of scaling networks under the limitation of floating point calculation FLOPs ≤ 600M

[0143]

[0144] As can be seen from Table 2, under the limitation of floating-point computing amount FLOPs≤600M, the scaling network obtained by using the present invention is superior to the scaling network generated by using the single-dimensional scaling method and the unified-dimensional scaling method in terms of inference delay and network accuracy.

[0145] The above simulation results show that the scaling performance of the present invention is better than the existing single-dimensional scaling method and unified-dimensional scaling method.

[0146] The above descriptions are only two specific examples of the present invention and do not constitute any limitation to the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without departing from the principles and structures of the present invention. For example, the present invention can also be used for image segmentation tasks, image super-resolution tasks and video tracking tasks, etc. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0147] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention to facilitate understanding, and the order of the step numbers is not limited.

Claims

1. A multi-dimensional composite scaling method for convolutional neural networks, characterized in that: The steps include: (1) Select an existing trained baseline network and set a scaled target floating-point operation FLOPs upper limit for it; (2) Set the scaling relationship set of network resolution and width to {0.9, 1.0, 1.1, 1.2, 1.3}, and select a scaling relationship constraint f(e w ,e r ) value, when 0.3≤e r Randomly select a network resolution scaling factor e in the range ≤ 2.8 r ; (3) According to the scaling constraint f(e w ,e r ) and the floating point operation FLOPs upper limit constraint, the network width scaling factor e is calculated w and the depth scaling factor e d ; (4) Use the network width scaling factor e w , depth scaling factor e d and resolution scaling factor e r , adjust the original width w, depth d, and resolution r of the baseline network to get the scaled width w new , depth d new and resolution r new , generate a new candidate network architecture; (5) Select other unused scaling relationship constraints f(e) from the scaling relationship set w ,e r ), at 0.3≤e r Randomly select an unused network resolution scaling factor e in the range ≤ 2.8 r ; Use the scaling constraint f(e w ,e r ) and the network resolution scaling factor e r Generate multiple new candidate networks Nets; (6) Calculate the computationally intensive layer inference latency T of all candidate networks based on the network depth d, width w, and resolution r CB and the inference latency T of the memory-intensive layer MB , the relationship between the network dimension and the overall reasoning delay is established by combining the two delays, and the network delay evaluation model R is obtained pro ; (7) Using network delay to evaluate model R pro The best scaling configuration for the baseline network is obtained by comparing the measured network accuracy: (7a) Evaluating model R using network delay pro Calculate the inference delay Run(Nets) of all candidate networks; (7b) Retrain the candidate network using the training set and validation set of the existing dataset, and use the test set to infer the accuracy Acc(Nets) of all candidate networks on the trained candidate network; (7c) Calculate the evaluation value of all candidate networks Tar = Acc(Nets) - γ × Run(Nets), where γ is the calculation weight of the inference delay Run(Nets); (7d) Select the network N with the largest Tar among all candidate networks as the best scaling network. The resolution scaling factor of the best scaling network is e r , width scaling factor e w and the depth scaling factor e d This is the optimal scaling configuration for the baseline network; (8) According to the optimal scaling configuration parameter e r 、e w 、e d Scaling the baseline network.

2. The method according to claim 1, characterized in that: Step (1) sets a scaled target floating-point operation FLOPs upper limit for the trained baseline network, which is determined based on application requirements, hardware processing capabilities, and energy consumption cost requirements. The application requirements include application accuracy and inference time requirements; the hardware processing capabilities include hardware computing and memory resources; and the energy consumption cost requirements include network operation power consumption and network operation costs.

3. The method according to claim 1, characterized in that: In the step (2), a scaling relation constraint f(e) is selected from the scaling relation set. w ,e r ) value, the formula is as follows: Among them, e w represents the network width scaling factor, e r Indicates the network resolution scaling factor; The network width scaling factor e is calculated in step (3) w , the formula is as follows: Among them, e r represents the network resolution scaling factor, f(e w ,e r ) represents the value of the scaling relationship constraint; 0.3≤e w ≤2.8, if e w If there is no solution in this range, reselect the network resolution scaling factor e r And the scaling constraint f(e w ,e r ).

4. The method according to claim 1, characterized in that: The depth scaling factor e of the network is calculated in step (3) d , the implementation steps include the following: (3a) Calculate the floating point operations FLOPs of a single convolutional layer: FLOPs(Layer)=w 2 r 2 k 2 Among them, w represents the width of the network layer, that is, the number of convolution filters, k is the size of the filter, and r represents the spatial resolution of the layer; (3b) The network depth scaling factor e is obtained by solving the following inequality group: d The value range of is: Where d is the depth of the baseline network, e d *d represents the depth of the scaling network, Indicates d i FLOPs of the convolutional layer; (3c) From the solved depth scaling factor e d Randomly select a value within the selection range as the depth scaling factor e d .

5. The method according to claim 1, characterized in that The scaled width w is obtained in step (4) new , depth d new and resolution r new , respectively expressed as follows: w new =e w *w d new =e d *d r new ＝e r *r 6. The method according to claim 1, characterized in that In step (6), the computationally intensive layer inference latency T of all candidate networks is calculated. CB and the inference latency T of the memory-intensive layer MB , the implementation steps include the following: (6a) Calculate the inference latency T of the candidate network’s computationally intensive layer CB : Among them, layer pps =w 2 r 2 k 2 represents the floating point operations FLOPs required for a single convolutional layer in a neural network, w represents the network width, k is the size of the filter, r represents the spatial resolution of the layer, P max Indicates the maximum computing power of the GPU per second, U til represents resource utilization, P max *U til Represents the effective computing power of the hardware, denoted as constant b; (6b) Calculate the inference latency T of the memory-intensive layer of the candidate network MB : Among them, D io =wr 2 M represents the amount of data of intermediate activation features transmitted between network layers. BW Indicates the maximum memory bandwidth of the GPU, M BW *U til Represents the effective memory bandwidth of the hardware, denoted as a constant.

7. The method according to claim 1, characterized in that Step (6) Combine the two delays to establish the relationship between the network dimension and the overall reasoning delay, and obtain the network delay evaluation model R pro , the implementation steps include the following: (6c) Calculate the total inference delay T of the entire neural network Net : Where d1 and d2 represent the number of memory-intensive layers and computation-intensive layers, respectively; w1 and w2 represent the average channel width of memory-intensive layers and computation-intensive layers, respectively; r1 and r2 represent the resolution of memory-intensive layers and computation-intensive layers, respectively; Let d = d1 + d2 represent the total number of network layers, that is, the depth of the network. Represents the total average channel width of the network, that is, the network width; r=r1+r2 represents the input resolution of the network. Substituting d, w, and r into the above formula and sorting it out, we get: (6d) The network inference time T is obtained from the result of step (6c) Net The complexity of is related to the network dimensions d, w, and r as follows: GRASS Net (d,w,r))=O(dw 1.5 r 2 ) Where O represents the relationship between the network inference time and the network depth d, width w and resolution r in the worst case; (6e) Using the relationship in step (6d) we obtain the delayed evaluation model R pro : R pro =dw 1.5 r 2 。 8. The method according to claim 1, characterized in that Step (7a) uses the network delay evaluation model R pro The inference delay Run(Nets) of all candidate networks is calculated by subtracting the depth d of each candidate network from the i , width w i , resolution r i Substitute the delayed evaluation model R pro Calculate the inference delay of all candidate networks: Run(Nets)={R pro (d i ,w i ,r i )∣i∈Nets}。 9. The method according to claim 1, characterized in that: Step (7b) obtains the accuracy Acc(Nets) of all candidate networks. The implementation steps include the following: (7b1) Divide the data set: Obtain the ImageNet-100 dataset from the Internet and perform data augmentation on it, including random cropping, random inversion, and color jittering; divide the augmented dataset into training set, validation set, and test set in a ratio of 7:2:1; (7b2) Train the candidate network: (7b2a) Use the deep learning framework PyTorch to build a candidate network model and define a training loop; (7b2b) Initialize network parameters, use gradient descent method to process training set data batch by batch and update network weights, and use validation set to evaluate network performance and adjust network parameters after each training cycle; (7b2c) Repeat step (7b2b) until the network converges and then stop training to obtain a trained network model. (7b2d) Repeat steps (7b2a), (7b2b), and (7b2c) for all candidate networks to obtain all trained candidate networks; (7b3) Test the candidate network accuracy: (7b3a) Input each sample of the test set into the trained candidate network and output the prediction result of each sample; (7b3b) Compare the predicted results of all samples with the true labels: For classification tasks, samples with the same predicted result and true label are predicted as correct. For regression tasks, the error between the predicted result and the true label is calculated; (7b3c) Calculate the accuracy of all test samples in the classification task or the average error of all test samples in the regression task to obtain the accuracy of the candidate network; (7b3d) Repeat steps (7b3a), (7b3b), and (7b3c) for all trained candidate networks to obtain the accuracy Acc(Nets) of all candidate networks.

10. The method according to claim 1, characterized in that In step (7d), select the network N with the largest Tar among all candidate networks. The formula is as follows: Among them, Nets represents all candidate networks, Acc(Nets) represents the accuracy of all candidate networks; Run(Nets) represents the inference time of all candidate networks; γ represents the weight of inference time; N is the network with the largest Tar.

Images