Edge device inference acceleration method, apparatus, and data processing system

Abstract

The present disclosure provides an edge device inference acceleration method, device, system, medium and program product, which can be applied to the field of artificial intelligence technology. The method comprises: obtaining a pre-trained first neural network model; pruning at least one convolution layer and / or at least one convolution kernel in the first neural network model to obtain a second neural network model; obtaining a first inference time of the second neural network model running on an edge device to be deployed; and if the first inference time is less than a target inference time, determining that the second neural network model is a model meeting the deployment requirements of the edge device. The neural network pruning method can be used for model compression with the target inference time as a constraint, and a model meeting the deployment requirements can be pruned for the edge device to be deployed, so as to realize inference acceleration of a specific edge device. The present disclosure also provides a data processing system deployed on an edge device.

Description

[0001] This application is a divisional application. The parent application number is 202210785413.X, the application date is July 4, 2022, and the invention title is: Edge Device Inference Acceleration Method, Apparatus and Data Processing System. Technical Field

[0002] This disclosure relates to the field of artificial intelligence, and more specifically, to a method, apparatus, data processing system, edge device inference acceleration system, medium, and program product for accelerating inference on edge devices. Background Technology

[0003] Neural network models that meet accuracy requirements typically have a large number of parameters and high floating-point computational cost, making them difficult to deploy on edge devices such as wearable devices and automotive systems. Related technologies can compress neural network models to reduce the number of parameters and floating-point computational cost. However, the inference time of neural network models on edge devices is closely related to many characteristics of the edge devices, such as device model, input data type, memory access bandwidth, and data alignment. Therefore, deploying compressed neural network models on specific edge devices results in slow inference speeds. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a method, apparatus, data processing system, edge device inference acceleration system, medium, and program product.

[0005] One aspect of this disclosure provides a method for accelerating inference on an edge device, comprising: obtaining a pre-trained first neural network model, wherein the first neural network model includes N convolutional layers, each convolutional layer including at least one convolutional kernel, and N being an integer greater than or equal to 1; pruning at least one convolutional layer and / or at least one convolutional kernel in the first neural network model to obtain a second neural network model; obtaining a first inference time of the second neural network model running on an edge device to be deployed, wherein the first inference time includes the time to process specific input data using the second neural network model and obtain a processing result; if the first inference time is less than a target inference time, determining that the second neural network model is a model that meets the deployment requirements of the edge device, wherein the deployment requirements include inference acceleration requirements for the edge device to be deployed, and the target inference time is preset according to the performance of the edge device to be deployed.

[0006] According to embodiments of this disclosure, before pruning at least one convolutional layer and / or at least one convolutional kernel in the first neural network model, the method further includes: obtaining a first score for each convolutional layer based on the importance of each convolutional layer, wherein the importance is used to characterize the degree of influence on the first loss function value of the first neural network model on the validation set; and sorting the N convolutional layers according to the magnitude of their first scores to obtain a first sorting result, wherein the first sorting result is used as the basis for pruning each convolutional layer.

[0007] According to an embodiment of this disclosure, the i-th convolutional layer is any one of the N convolutional layers, i is greater than or equal to 1, and i is less than or equal to N. Obtaining the first score of each convolutional layer includes: setting the initial value of i to 1, and performing the following operations in a loop: pruning the i-th convolutional layer from the first neural network model to obtain a third neural network model; obtaining the third loss function value of the third neural network model on the validation set; obtaining the first score of the i-th convolutional layer based on the third loss function value; and incrementing the value of i by 1 if the value of i is less than N.

[0008] According to embodiments of this disclosure, the method further includes: pruning the convolutional kernels in the i-th convolutional layer according to M pruning ratios to obtain M fourth neural network models, where M is an integer greater than or equal to 1; obtaining M fourth loss function values ​​of the M fourth neural network models on the validation set; wherein obtaining the first score of the i-th convolutional layer based on the third loss function value includes: obtaining the first score of the i-th convolutional layer based on the third loss function value and the M fourth loss function values.

[0009] According to an embodiment of this disclosure, the first score of each convolutional layer is positively correlated with its importance. Pruning at least one convolutional layer and / or at least one convolutional kernel in the first neural network model to obtain a second neural network model includes: pruning each convolutional layer sequentially according to the ascending order of the first sorting results.

[0010] According to an embodiment of this disclosure, the j-th order is any one of the first sorting results, j is greater than or equal to 1, and j is less than or equal to N. The step of pruning each convolutional layer in ascending order according to the first sorting results includes: setting the initial value of j to 1, and performing the following operations in a loop: pruning the convolutional layer of the j-th order in the first neural network model based on the current model to be pruned, to obtain a fifth neural network model; obtaining the fifth loss function value of the fifth neural network model on the validation set; if the first change between the first loss function value and the fifth loss function value is less than the target change, using the fifth neural network model as the current model to be pruned, and if the value of j is less than N, incrementing the value of j by 1 and executing the next loop.

[0011] According to embodiments of this disclosure, if the first change between the first loss function value and the fifth loss function value is greater than or equal to the target change, before incrementing the value of j by 1, the method further includes: performing convolution kernel pruning on the j-th sequential convolutional layer to obtain a sixth neural network model; obtaining the sixth loss function value of the sixth neural network model on the validation set; if the second change between the first loss function value and the sixth loss function value is less than the target change, using the sixth neural network model as the current model to be pruned, and incrementing the value of j by 1 and executing the next loop if the value of j is less than N; or if the second change between the first loss function value and the sixth loss function value is greater than or equal to the target change, not using the sixth neural network model as the current model to be pruned, and incrementing the value of j by 1 and executing the next loop if the value of j is less than N.

[0012] According to an embodiment of this disclosure, before pruning the convolutional kernels of the j-th sequential convolutional layer, the method includes: obtaining a second score for each convolutional kernel based on the importance of each convolutional kernel; sorting the second scores of all convolutional kernels in each convolutional layer to obtain N second sorting results, wherein the N second sorting results are used as the basis for pruning each convolutional kernel.

[0013] According to an embodiment of this disclosure, the i-th convolutional layer is any one of the N convolutional layers, i is greater than or equal to 1, and i is less than or equal to N. Obtaining the second score of each convolutional kernel includes: setting the initial value of i to 1, and performing the following operations in a loop: using the validation set as input to the first neural network model to obtain the parameter pair of each convolutional kernel in the i-th convolutional layer, wherein the parameter pair includes a gradient value and a weight value, and the gradient value is obtained according to the backpropagation algorithm; obtaining the second score of each convolutional kernel in the i-th convolutional layer according to the parameter pair of each convolutional kernel in the i-th convolutional layer; if the value of i is less than N, incrementing the value of i by 1.

[0014] According to an embodiment of this disclosure, the validation set includes S subsets, where S is greater than or equal to 2. The step of using the validation set as input to the first neural network model to obtain parameter pairs for each convolutional kernel in the i-th convolutional layer includes: using at least two subsets from the S subsets as input to the first neural network model to obtain at least two parameter pairs for each convolutional kernel in the i-th convolutional layer, wherein each parameter pair is obtained based on each subset from the at least two subsets.

[0015] According to an embodiment of this disclosure, obtaining the second score of each convolutional kernel in the i-th convolutional layer based on the parameter pairs of each convolutional kernel in the i-th convolutional layer includes: obtaining at least two third scores for each convolutional kernel in the i-th convolutional layer based on the at least two parameter pairs; and obtaining the second score of each convolutional kernel in the i-th convolutional layer based on the at least two third scores.

[0016] According to an embodiment of this disclosure, the second score of each convolutional kernel is positively correlated with its importance, and the convolutional kernel pruning of the j-th sequential convolutional layer includes: pruning at least one convolutional kernel in the j-th sequential convolutional layer according to the ascending order in the second sorting result.

[0017] According to an embodiment of this disclosure, the j-th sequential convolutional layer includes Q convolutional kernels, where Q is greater than or equal to 1. The step of pruning at least one convolutional kernel in the j-th sequential convolutional layer according to the ascending order of the second sorting result includes: looping through the following operations, wherein the model obtained after the loop is used as the sixth neural network model: obtaining p convolutional kernels according to the ascending order of the second sorting result; pruning the p convolutional kernels when the value of p is greater than or equal to 1 and less than Q to obtain a seventh neural network model; obtaining the seventh loss function value of the seventh neural network model on the validation set; and increasing the value of p by a first preset value if the third change between the first loss function value and the seventh loss function value is less than the target change.

[0018] According to embodiments of this disclosure, the method further includes: if a third change between the first loss function value and the seventh loss function value is greater than or equal to the target change, the value of p is reduced by a second preset value.

[0019] According to an embodiment of this disclosure, obtaining the second neural network model includes: each time the current model to be pruned is updated, the updated current model to be pruned is used as a second neural network model.

[0020] According to an embodiment of this disclosure, obtaining the first inference time of the second neural network model running on the edge device to be deployed includes: obtaining the first inference time of each of the second neural networks; wherein the method further includes: determining the second neural network model with the minimum loss function value from at least one second neural network model that meets the deployment requirements; and deploying the second neural network model with the minimum loss function value on the edge device.

[0021] According to embodiments of this disclosure, the method further includes: if the value of j is equal to N, obtaining the third inference time of the current pruning model running on the edge device to be deployed; wherein, the method further includes updating the target change amount, specifically including: if the third inference time is less than the target inference time, reducing the target change amount by a third preset value.

[0022] According to an embodiment of this disclosure, updating the target change further includes: if the third inference time is greater than or equal to the target inference time, increasing the target change by a fourth preset value.

[0023] According to embodiments of this disclosure, before obtaining the first inference time of the second neural network model running on the edge device to be deployed, the method further includes: training the second neural network model using a training set, wherein the training data in the training set includes any one of image data, speech data, natural language data, or video data; and running the trained second neural network model on the edge device to be deployed.

[0024] According to embodiments of this disclosure, the specific input data includes any one of image data, voice data, natural language data, or video data.

[0025] Another aspect of this disclosure provides an edge device inference acceleration apparatus, comprising: a model acquisition module for acquiring a pre-trained first neural network model, wherein the first neural network model includes N convolutional layers, each convolutional layer including at least one convolutional kernel, and N being an integer greater than or equal to 1; a model pruning module for pruning at least one convolutional layer and / or at least one convolutional kernel in the first neural network model to obtain a second neural network model; a time acquisition module for acquiring a first inference time of the second neural network model running on an edge device to be deployed, wherein the first inference time includes the time for processing specific input data using the second neural network model and obtaining a processing result; and a model determination module for determining that the second neural network model meets the deployment requirements of the edge device if the first inference time is less than a target inference time, wherein the deployment requirements include inference acceleration requirements for the edge device to be deployed, and the target inference time is preset according to the performance of the edge device to be deployed.

[0026] Another aspect of this disclosure provides a data processing system deployed on an edge device, wherein: a second neural network model for processing specific input data in the data processing system is obtained according to the method of claim 1, the specific input data including any one of image data, voice data, natural language data, or video data.

[0027] Another aspect of this disclosure provides an edge device inference acceleration system, including an electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0028] According to embodiments of this disclosure, the edge device inference acceleration system further includes an edge device. The edge device is used to run a second neural network model sent by the one or more processors to obtain a first inference time; wherein the one or more processors are used to obtain the first inference time from the edge device.

[0029] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0030] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0031] The above one or more embodiments have the following beneficial effects: By using a neural network pruning method to compress the model, constrained by the target inference time on the edge device to be deployed, inference acceleration is achieved for specific edge devices. First, a second neural network model is obtained by pruning at least one convolutional layer and / or at least one convolutional kernel in the first neural network model. Then, the first inference time of the second neural network model running on the edge device to be deployed is obtained. Finally, the magnitude of the first inference time is compared with the target inference time to determine whether the second neural network model meets the deployment requirements of the edge device. Thus, while obtaining a high compression ratio model, this method also solves, to some extent, the problem that conventional pruning algorithms cannot effectively accelerate model inference on some hardware platforms due to factors such as memory access bandwidth limitations. Attached Figure Description

[0032] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0033] Figure 1 This diagram schematically illustrates an application scenario of the edge device inference acceleration method according to embodiments of the present disclosure.

[0034] Figure 2 A flowchart illustrating an edge device inference acceleration method according to an embodiment of the present disclosure is shown schematically.

[0035] Figure 3 A flowchart illustrating the training of a second neural network model according to an embodiment of the present disclosure is shown schematically.

[0036] Figure 4 A flowchart illustrating the process of obtaining a first sorting result according to an embodiment of the present disclosure is shown schematically;

[0037] Figure 5 A flowchart illustrating the process of obtaining a first score according to an embodiment of the present disclosure is shown schematically;

[0038] Figure 6 A flowchart illustrating the process of obtaining a first score according to another embodiment of the present disclosure is shown schematically;

[0039] Figure 7 A flowchart illustrating model pruning according to an embodiment of the present disclosure is shown schematically;

[0040] Figure 8 A flowchart illustrating model pruning according to another embodiment of the present disclosure is shown schematically;

[0041] Figure 9 A flowchart illustrating the process of obtaining a second sorting result according to an embodiment of the present disclosure is shown schematically;

[0042] Figure 10 A flowchart illustrating the process of obtaining a second score according to an embodiment of the present disclosure is shown schematically;

[0043] Figure 11 A flowchart illustrating model pruning according to another embodiment of the present disclosure is shown schematically;

[0044] Figure 12 A flowchart illustrating model pruning according to another embodiment of the present disclosure is shown schematically;

[0045] Figure 13 A flowchart illustrating an edge device inference acceleration method according to another embodiment of the present disclosure is shown schematically;

[0046] Figure 14 A flowchart illustrating an edge device inference acceleration method according to another embodiment of the present disclosure is shown schematically;

[0047] Figure 15 A schematic block diagram of an edge device inference acceleration apparatus according to an embodiment of the present disclosure is shown; and

[0048] Figure 16 A block diagram schematically illustrates an electronic device suitable for implementing an edge device inference acceleration method according to an embodiment of the present disclosure. Detailed Implementation

[0049] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0050] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0051] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0052] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0053] To address the problems existing in related technologies, embodiments of this disclosure provide a method for accelerating inference on edge devices. By using a neural network pruning method to compress the model with a target inference time as a constraint, a model that meets the requirements of the edge device to be deployed can be pruned, thereby accelerating inference on specific edge devices. First, a second neural network model is obtained by pruning at least one convolutional layer and / or at least one convolutional kernel in a first neural network model. Then, the first inference time of the second neural network model running on the edge device to be deployed is obtained. Finally, the magnitude of the first inference time is compared with the target inference time to determine whether the second neural network model meets the deployment requirements of the edge device. Thus, while obtaining a model with a high compression ratio, this method also solves to some extent the problem that conventional pruning algorithms cannot effectively accelerate model inference on some hardware platforms due to factors such as memory access bandwidth limitations.

[0054] Figure 1 The illustration shows an application scenario diagram of the edge device inference acceleration method according to an embodiment of the present disclosure.

[0055] like Figure 1 As shown, application scenario 100 according to this embodiment may include edge devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing a communication link between edge devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0056] Users can use edge devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on edge devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0057] Edge devices are hardware devices with edge computing capabilities, used to perform different data processing tasks based on configured software applications or functions. In other words, data, data processing, and applications can be centralized in edge devices, giving them the ability to store data and process tasks. Edge devices can be used in scenarios such as the Internet of Things (IoT), robotics, and smart devices. Edge devices 101, 102, and 103 include, but are not limited to, autonomous driving platforms, wearable devices, implantable medical devices, smartphones, tablets, laptops, and desktop computers, etc.

[0058] Server 105 can be a server that provides various services, such as an edge server that supports data packets transmitted by users using edge devices 101, 102, and 103 (for example only). The edge server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0059] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0060] The following will be based on Figure 1 The described scene, through Figures 2-14 The edge device inference acceleration method according to the embodiments of this disclosure will be described in detail.

[0061] Figure 2 A flowchart illustrating an edge device inference acceleration method according to an embodiment of the present disclosure is shown schematically.

[0062] like Figure 2 As shown, the edge device inference acceleration method of this embodiment includes operations S210 to S240.

[0063] In operation S210, a pre-trained first neural network model is obtained, wherein the first neural network model includes N convolutional layers, each convolutional layer includes at least one convolutional kernel, and N is an integer greater than or equal to 1.

[0064] For example, the first neural network model can be pre-trained using a training set. The training data in the training set includes any one of image data, speech data, natural language data, or video data. The first neural network model is a convolutional neural network model.

[0065] In operation S220, at least one convolutional layer and / or at least one convolutional kernel in the first neural network model are pruned to obtain the second neural network model.

[0066] For example, model pruning is a method of model compression that can remove some unimportant connections from the original model and use the remaining connections as a new model. The first neural network model is the original model, and the second neural network model is the new model.

[0067] In some embodiments, each pruning step can use one or more convolutional layers or kernels as the pruning unit to compress the first neural network model. The specific convolutional layers or kernels within a given convolutional layer to be removed can be determined based on the accuracy of the new model after pruning. Accuracy can be determined by the loss function value obtained by processing validation set data with the new model. For example, a smaller loss function value indicates higher accuracy, meaning better model convergence.

[0068] In operation S230, the first inference time of the second neural network model running on the edge device to be deployed is obtained, wherein the first inference time includes the time to process specific input data using the second neural network model and obtain the processing result.

[0069] According to embodiments of this disclosure, specific input data includes any one of image data, voice data, natural language data, or video data.

[0070] For example, in image recognition, the specific input data is one or more images. The time it takes for the second neural network model to process these images and obtain the image recognition result is the first inference time. In speech recognition, the specific input data is audio data over a period of time. The time it takes for the second neural network model to process this audio and obtain the speech recognition result is the first inference time. In natural language processing scenarios, such as text topic recognition, the specific input data is a piece of text. The time it takes for the second neural network model to process this text and obtain the topic recognition result is the first inference time. In object re-identification scenarios, the specific input data is video data over a period of time. The time it takes for the second neural network model to process this video data and obtain the object tracking result is the first inference time.

[0071] In some embodiments, a second neural network model can be run on the edge device to be deployed to process images for speed testing in the following manner:

[0072] Step 1: Read the second neural network model obtained during the pruning process.

[0073] Step 2: Process the second neural network model according to the platform and framework configuration of the edge device. For example, if a specific edge device does not support SyncBatchNorm, it needs to be converted to BatchNorm2d for processing. Both SyncBatchNorm and BatchNorm2d are used for batch normalization or batch standardization of the input. Convert the second neural network model into data formats supported by different frameworks, such as Cambricon format; ONNX (Open Neural Network Exchange) format; and file formats supported by TensorRT. TensorRT is a deep learning inference engine (GPU Inference Engine) developed by NVIDIA, a complete tool from model acquisition to model optimization and compilation, and then to deployment.

[0074] Step 3: Set the batch size (e.g., 64 images, for example only) and the shape (channels and dimensions, e.g., 3*224*224, for example only) of the input images for the second neural network model during speed testing.

[0075] Step 4: Read an image, resize / padding it to fit the shape of the input image, and copy it an equal number of times as batch_size.

[0076] Step 5: Feed the entire batch into the model for forward inference, repeating this several times (e.g., 20 times, just an example). This step is not timed in order to allow the device state to stabilize.

[0077] Step 6: Feed the entire batch into the model for forward inference, repeat several times (e.g., 100 times, just an example), use the system timing function to calculate the difference to get the total inference time, that is, get the two times before and after the model runs, and then calculate the difference, such as endtime (run end time) minus starttime (run start time).

[0078] Step 7: Calculate the average inference time based on the total inference time in Step 6, such as (endtime-starttime) / 100.

[0079] Step 8: Return the average inference time obtained in Step 7 as the speed measurement result (i.e., the first inference time).

[0080] In operation S240, if the first inference time is less than the target inference time, the second neural network model is determined to be a model that meets the deployment requirements of the edge device. The deployment requirements include the inference acceleration requirements of the edge device to be deployed, and the target inference time is preset according to the performance of the edge device to be deployed.

[0081] For example, if the first inference time is less than the target inference time, it means that the pruned second neural network model meets the requirements, and its parameter count and floating-point computation are smaller than those of the first neural network model, which is more conducive to improving the inference speed of the edge device to be deployed.

[0082] For example, different edge devices vary in model, purpose, input data type, memory access bandwidth, and data alignment, resulting in different inference performance. By using the inference time ultimately deployed on a specific edge device as a constraint, searching for pruned models that meet the deployment requirements within this time constraint can more accurately obtain models that match the specific edge device, thereby achieving better inference acceleration.

[0083] According to embodiments of this disclosure, model compression is performed using a neural network pruning method, with the target inference time as a constraint. This allows for the pruning of models to meet the requirements of the edge devices to be deployed, thereby accelerating inference on specific edge devices. While obtaining a high compression ratio model, this method also addresses, to some extent, the problem that conventional pruning algorithms cannot effectively accelerate model inference on some hardware platforms due to factors such as memory access bandwidth limitations.

[0084] Figure 3 A flowchart illustrating the training of a second neural network model according to an embodiment of the present disclosure is shown.

[0085] like Figure 3 As shown, operations S310 to S320 may be included before operation S230.

[0086] In operation S310, the second neural network model is trained using a training set, wherein the training data in the training set includes any one of image data, speech data, natural language data, or video data.

[0087] After pruning the first neural network model, the parameters in the original model may no longer be applicable. Therefore, a second neural network model is trained before determining its feasibility.

[0088] First, one or more training data sets are input into the second neural network model. The training data includes samples and labels, such as {x1, y1}, {x2, y2}, ..., {xn, yn}. Here, x1~xn are n samples to be processed, and y1~yn are the labels corresponding to each sample.

[0089] Secondly, for example, in a face recognition task, x1~xn represent n face images, and y1~yn represent the actual results corresponding to each sample. The predicted results output by the second neural network model and the labels are input into the loss function to calculate the contrast loss between the two.

[0090] Finally, backpropagation is performed to update the parameters based on the output of the loss function. When the loss function tends to converge, the robustness of the current second neural network model is considered to be high, thus obtaining the trained second neural network model.

[0091] The S320 is used to run the trained second neural network model on the edge device to be deployed.

[0092] According to embodiments of this disclosure, after determining that the accuracy of the second neural network model meets the requirements, it is then run on the edge device to be deployed, which can ensure that the model that meets the deployment requirements also has high reliability.

[0093] Figure 4 A flowchart illustrating the process of obtaining a first sorting result according to an embodiment of the present disclosure is shown.

[0094] like Figure 4 As shown, the first sorting result can be obtained before operation S220 in this embodiment, specifically including operations S410 to S420.

[0095] In operation S410, a first score for each convolutional layer is obtained based on the importance of each convolutional layer, where the importance is used to characterize the degree of influence on the loss function value of the first neural network model.

[0096] For example, the degree of influence refers to the change in the loss function values ​​of two models when they process the same batch of validation data before and after pruning. For instance, a first neural network model processes data from the validation set, resulting in a loss function value A1. Pruning a certain convolutional layer yields a new model, which processes the same validation set, resulting in a loss function value A2. The change in A2 relative to A1 (such as the absolute value of the difference between the two) can be used to characterize importance.

[0097] The validation data in the validation set is of the same type as the training data, and each validation data also includes a sample and a label.

[0098] In operation S420, the first sorting result is obtained by sorting the N convolutional layers according to the size of their first scores. The first sorting result is used as the basis for pruning each convolutional layer.

[0099] According to embodiments of this disclosure, the importance of a convolutional layer is characterized by a first score, thereby enabling the determination of the pruning order for the N convolutional layers based on the first ranking result, thus improving the efficiency of model compression.

[0100] The following is combined Figure 5 and Figure 6 This section introduces some implementations of obtaining the first score for each convolutional layer in operation S410.

[0101] Figure 5 A flowchart illustrating the process of obtaining a first score according to an embodiment of the present disclosure is shown.

[0102] like Figure 5 As shown, obtaining the first score for each convolutional layer in operation S410 includes: setting the initial value of i to 1, and repeatedly executing operations S510 to S540. Here, the i-th convolutional layer is any one of the N convolutional layers, where i is greater than or equal to 1 and less than or equal to N.

[0103] In operation S510, the i-th convolutional layer is pruned from the first neural network model to obtain the third neural network model.

[0104] The i-th convolutional layer is removed from the first neural network model, leaving the third neural network model with the remaining N-1 convolutional layers. When i is 1, the first convolutional layer can be determined according to the order of forward computation (the computation process from input to output). As i increases, the analysis proceeds sequentially, taking one convolutional layer at a time for analysis.

[0105] By operating the S520, the third loss function value of the third neural network model on the validation set is obtained.

[0106] The third neural network model can be trained to achieve a high degree of convergence. Then, the trained third neural network model processes data from the validation set to obtain the third loss function value.

[0107] In operation S530, the first score of the i-th convolutional layer is obtained based on the value of the third loss function.

[0108] For example, the validation set can be divided into multiple subsets, each subset obtaining a third loss function value, and the average of the multiple third loss function values ​​is used as the first score.

[0109] In operation S540, determine if i is less than N. If the value of i is less than N, increment the value of i by 1. If the value of i is not less than N, end the loop.

[0110] According to embodiments of this disclosure, the first score of the convolutional layer can be obtained by removing the change in the model loss function value before and after the i-th convolutional layer, which can reflect the impact on model accuracy.

[0111] Figure 6 A flowchart illustrating the process of obtaining a first score according to another embodiment of the present disclosure is shown.

[0112] like Figure 6As shown, obtaining the first score for each convolutional layer in operation S410 includes operations S510, S520, and S540, and may also include operations S610 to S630. Operation S630 is one embodiment of operation S530. Operations S510 and S520 are described above and will not be repeated here.

[0113] In operation S610, according to M pruning ratios, the convolution kernels in the i-th convolutional layer are pruned respectively to obtain M fourth neural network models, where M is an integer greater than or equal to 1.

[0114] The M pruning ratios represent M different pruning rates, corresponding to M different numbers of convolutional kernels. For example, pruning 20%, 50%, and 80% of the convolutional kernels in the i-th convolutional layer respectively yields three fourth neural network models (for example only).

[0115] For example, convolutional kernels can be randomly selected for pruning. Alternatively, the importance of the convolutional kernels can be sorted, and kernels can be selected for pruning in ascending order of importance.

[0116] In operation S620, M fourth loss function values ​​of M fourth neural network models on the validation set are obtained respectively.

[0117] For example, the fourth neural network model obtained with different pruning ratios is trained on the training set, and the loss function value is calculated and saved on the validation set.

[0118] In operation S630, the first score of the i-th convolutional layer is obtained based on the third loss function value and M fourth loss function values.

[0119] For the i-th convolutional layer, after obtaining the loss function value when the layer is completely pruned and the loss function values ​​at different pruning rates, the first score is obtained by weighted summation of multiple loss function values.

[0120] Then, the first scores of the N convolutional layers are sorted from low to high to obtain the final importance ranking of each convolutional layer. The higher the ranking, the smaller the first score, indicating that the kernel pruning or layer pruning of the convolutional layer has a smaller impact on the model's loss value, that is, the convolutional layer has a lower importance.

[0121] According to embodiments of this disclosure, a comprehensive importance is obtained by performing layer pruning and convolution kernel pruning on each layer, which is used to determine the order of pruning analysis of the convolutional layers.

[0122] According to embodiments of this disclosure, the first score of each convolutional layer is positively correlated with its importance. Operation S220, which prunes at least one convolutional layer and / or at least one convolutional kernel in the first neural network model to obtain the second neural network model, includes: pruning each convolutional layer sequentially according to the ascending order of the first sorting results. For example, layer pruning can be performed on a certain convolutional layer first. If the pruned model meets the inference acceleration requirements of the edge device to be deployed, then the next convolutional layer is pruned. If the pruned model does not meet the inference acceleration requirements, kernel pruning is performed on the convolutional kernels in that convolutional layer to further determine whether the pruned model meets the inference acceleration requirements. After kernel pruning is completed, the next convolutional layer is pruned. The pruning steps for the next convolutional layer are repeated. A detailed description follows.

[0123] Figure 7 A flowchart illustrating model pruning according to an embodiment of the present disclosure is shown schematically.

[0124] like Figure 7 As shown, according to the ascending order of the first sorting result, the pruning of each convolutional layer includes: setting the initial value of j to 1, and repeatedly executing operations S710 to S750. Here, the j-th order is any one of the first sorting results, j is greater than or equal to 1, and j is less than or equal to N.

[0125] In operation S710, based on the current model to be pruned, the j-th sequential convolutional layer in the first neural network model is pruned to obtain the fifth neural network model.

[0126] When performing convolutional layer pruning for the first time, the first neural network model is the model to be pruned.

[0127] By operating the S720, the fifth loss function value of the fifth neural network model on the validation set is obtained.

[0128] The fifth neural network model can be trained to achieve a high degree of convergence. Then, the trained fifth neural network model processes the data in the validation set, yielding the fifth loss function value, test_loss.

[0129] For example, the first change (ΔLoss1) between the fifth loss function value test_loss and the first loss function value test_loss_ori can be calculated, as shown in Equation 1).

[0130] △Loss1= abs(test_loss_ori - test_loss) Formula 1)

[0131] Where abs represents the absolute value. It should be noted that Equation 1) is only one embodiment of the calculation of the first change in this disclosure. For example, ΔLoss1 can also be set equal to test_loss minus test_loss_ori. This disclosure does not impose any specific limitations.

[0132] In operation S730, it is determined whether the first change (△Loss1) between the first loss function value and the fifth loss function value is less than the target change (△Loss2). If yes, operation S740 is executed; otherwise, operation S750 is terminated.

[0133] In operation S740, if the first change between the first loss function value and the fifth loss function value is less than the target change, the fifth neural network model is taken as the current model to be pruned.

[0134] If ΔLoss1 < ΔLoss2 holds true, then this layer can be pruned under the current threshold condition; otherwise, this layer cannot be pruned. Therefore, the current model to be pruned is updated, and the model to be pruned in the next iteration is the fifth neural network model.

[0135] In operation S750, determine if j is less than N. If j is less than N, increment j by 1 and execute operation S710. If j is not less than N, end the loop.

[0136] According to embodiments of this disclosure, by determining whether the j-th sequential convolutional layer can be directly and completely pruned under the constraint of a loss threshold of ΔLoss2.

[0137] Figure 8 A flowchart illustrating model pruning according to another embodiment of the present disclosure is shown schematically.

[0138] like Figure 8 As shown, the model pruning in this embodiment may include operations S710 to S750. If the first change between the first loss function value and the fifth loss function value is greater than or equal to the target change, the loop may further include operations S810 to S840. Operations S710 to S740 are as described in the above embodiment and will not be repeated here.

[0139] In operation S810, the kernel of the j-th sequential convolutional layer is pruned to obtain the sixth neural network model.

[0140] By operating the S820, the sixth loss function value of the sixth neural network model on the validation set is obtained.

[0141] The sixth neural network model can be trained to achieve a high degree of convergence. Then, the trained sixth neural network model processes the data in the validation set, and the sixth loss function value is obtained as test_loss. The second change (ΔLoss3) is obtained according to Equation 1.

[0142] In operation S830, it is determined whether ΔLoss3 between the first loss function value and the sixth loss function value is less than ΔLoss2. If yes, operation S840 is executed; otherwise, the sixth neural network model is not used as the current model to be pruned, and operation S750 is executed. If the value of j is less than N, the value of j is incremented by 1 and the next loop is executed. If Loss3 is greater than or equal to ΔLoss2, the current model to be pruned in the next loop is the same as the current model to be pruned in the current loop.

[0143] In operation S840, if the second change between the first loss function value and the sixth loss function value is less than the target change, the sixth neural network model is taken as the current model to be pruned, and the sixth neural network model is the current model to be pruned in the next loop.

[0144] If ΔLoss3 < ΔLoss2 holds true, then this layer can undergo convolution kernel pruning under the current threshold condition. Therefore, update the current model to be pruned.

[0145] In operation S750, determine if j is less than N. If j is less than N, increment j by 1 and execute operation S710. If j is not less than N, end the loop.

[0146] According to an embodiment of this disclosure, if it is determined that the j-th sequential convolutional layer cannot be completely pruned directly under the constraint of a loss threshold of ΔLoss2, then the convolutional kernels in that convolutional layer are further pruned.

[0147] The following is combined Figures 9-11 Furthermore, some embodiments of kernel pruning for the j-th sequential convolutional layer based on the importance of the convolutional kernel are introduced.

[0148] Figure 9 A flowchart illustrating the process of obtaining a second sorting result according to an embodiment of the present disclosure is shown.

[0149] like Figure 9 As shown, operations S910 to S920 may be included before operation S810.

[0150] In the S910 operation, a second score for each convolutional kernel is obtained based on the importance of each kernel.

[0151] For example, a first neural network model processes data from the validation set to obtain a loss function value A1. A new model is obtained by pruning one or more convolutional kernels in a convolutional layer and processing data from the same validation set to obtain a loss function value A3. The change of A3 relative to A1 is used to characterize importance.

[0152] In operation S920, the second scores of all convolutional kernels in each convolutional layer are sorted to obtain N second sorting results, which are used as the basis for pruning each convolutional kernel.

[0153] According to embodiments of this disclosure, the importance of a convolutional kernel in a convolutional layer is characterized by a second score, thereby enabling the determination of the pruning order of convolutional kernels in a convolutional layer based on the second ranking result, thus improving the efficiency of model compression.

[0154] In some embodiments, the second score may be determined based on the number of non-zero values ​​in the convolution kernel, or based on the eigenvalues ​​of the convolution kernel, or based on the weights and gradients of the convolution kernel.

[0155] The following is combined Figure 10 An example of determining the second score based on the weights and gradients of the convolution kernel is described.

[0156] Figure 10 A flowchart illustrating the process of obtaining a second score according to an embodiment of the present disclosure is shown.

[0157] like Figure 10 As shown, obtaining the second score in operation S910 includes operations S1010 to S1030.

[0158] In operation S1010, the validation set is used as input to the first neural network model to obtain parameter pairs for each convolutional kernel in the i-th convolutional layer. Each parameter pair includes gradient values ​​and weight values, with the gradient values ​​obtained using the backpropagation algorithm. The weight values ​​consist of matrix element values ​​from each convolutional kernel.

[0159] After training the first neural network model using the training set, forward and backward computations can be performed using the validation set. Backward computation involves obtaining the gradient values ​​of each convolutional kernel using the backpropagation method.

[0160] The embodiments of this disclosure utilize a validation set for back-calculation to obtain gradient values, thus avoiding the situation where the gradient values ​​obtained in the training set are unreliable if the trained model is overfitted.

[0161] In operation S1020, based on the parameter pairs of each convolution kernel in the i-th convolutional layer, the second score of each convolution kernel in the i-th convolutional layer is obtained.

[0162] For example, the L1 norm can be obtained from the gradient and weight values ​​of each convolutional kernel and used as a second score.

[0163] In some embodiments, the validation set can be divided into multiple batches, with each batch being a subset. Therefore, the validation set can include S subsets, where S is greater than or equal to 2. At least two subsets from the S subsets are used as input to the first neural network model, corresponding to at least two parameter pairs for each convolutional kernel in the i-th convolutional layer. Each parameter pair is obtained based on each subset within the at least two subsets; that is, operation S1010 is performed once on each subset to obtain a corresponding parameter pair.

[0164] First, based on at least two parameter pairs, at least two third scores are obtained for each convolutional kernel in the i-th convolutional layer. In some embodiments, an L1 norm is calculated for each parameter pair of a convolutional kernel, and this L1 norm is directly used as a third score. In other embodiments, when a parameter pair for each convolutional kernel in the i-th convolutional layer is obtained based on a subset, the L1 norm of each convolutional kernel is first obtained. Then, the L1 norm of each convolutional kernel is sorted using a rank function to obtain the ranking of each convolutional kernel. Finally, the third score of each convolutional kernel is assigned according to the ranking.

[0165] Secondly, based on at least two third scores, a second score is obtained for each convolutional kernel in the i-th convolutional layer. For example, the second score can be obtained by summing multiple third scores. Alternatively, the second score can be obtained by averaging multiple third scores. The purpose of averaging is to avoid interference from data noise; for example, gradient values ​​obtained from data in a subset that deviate from normal levels may be due to noise.

[0166] In some embodiments, the validation set may not be divided into multiple batches, but can be treated as a single dataset. In this embodiment, the validation set can be used as input to the first neural network model to obtain a parameter pair for each convolutional kernel in the i-th convolutional layer. First, the L1 norm of each convolutional kernel is obtained. Then, the L1 norm of each convolutional kernel is sorted using a rank function to obtain the ranking of each convolutional kernel. Finally, a third score is assigned to each convolutional kernel based on its ranking. The third score of each convolutional kernel obtained in this step can be directly used as the second score.

[0167] In operation S1030, determine if the value of i is less than N. If the value of i is less than N, increment the value of i by 1. If the value of i is not less than N, end the loop.

[0168] According to embodiments of this disclosure, the order of convolution kernel pruning analysis is determined based on the importance of each convolution kernel.

[0169] According to embodiments of this disclosure, the second score of each convolutional kernel is positively correlated with its importance. Operation S810, which prunes the convolutional kernels of the j-th sequential convolutional layer, includes pruning at least one convolutional kernel in the j-th sequential convolutional layer according to the ascending order of the second sorting result. The j-th sequential convolutional layer comprises Q convolutional kernels, where Q is greater than or equal to 1.

[0170] Figure 11 A flowchart illustrating model pruning according to another embodiment of the present disclosure is shown schematically.

[0171] like Figure 11 As shown, one embodiment of operation S810 may include cyclically executing operations S1110 to S1170, wherein the model obtained after the loop ends serves as the sixth neural network model.

[0172] In operation S1110, p convolutional kernels are obtained according to the ascending order of the second sorting results.

[0173] For example, during the initial pruning, the initial value of p can be predetermined. For instance, if an initial ratio of 20% is determined based on expert experience, then the value of p is 20% of Q.

[0174] In some embodiments, the value of p can be determined using a bisection method. For example, the initial value of L is set to 0, the initial value of R is set to Q, and the value of p is (L+R)*0.5 each time pruning is performed.

[0175] In operation S1120, determine whether 1 ≤ p < Q is true. If yes, then execute operation S1130. If no, then end the loop.

[0176] In some embodiments, operation S1120 may also determine whether L is equal to R. If so, it indicates that the bisection method is no longer suitable for updating the value of p.

[0177] In operation S1130, when the value of p is greater than or equal to 1 and less than Q, the p convolutional kernels are pruned to obtain the seventh neural network model.

[0178] In operation S1140, the seventh loss function value of the seventh neural network model on the validation set is obtained.

[0179] The seventh neural network model can be trained to achieve a high degree of convergence. Then, the trained seventh neural network model processes the data in the validation set, and the seventh loss function value is obtained as the test_loss. The third change (ΔLoss4) between the first and seventh loss function values ​​can be obtained according to Equation 1).

[0180] In operation S1150, determine whether the third change (ΔLoss4) between the first loss function value and the seventh loss function value is less than the target change (ΔLoss2). If yes, proceed to operation S1160. If no, proceed to operation S1170.

[0181] In operation S1160, if the third change between the first loss function value and the seventh loss function value is less than the target change, the value of p is increased by the first preset value, and the next loop is executed. That is, after increasing the first preset value, the operation starts again from operation S1110 based on the latest value of p.

[0182] If ΔLoss4 < ΔLoss2 holds true, then this layer can undergo kernel pruning at the current pruning rate under the current threshold conditions. Therefore, update the value of p.

[0183] In some embodiments, a first preset value, such as 10% (for example only), can be preset. If a binary search is used, the value of L can be incremented by 1 to obtain the current value of p, and (L+R)*0.5 can be updated to obtain the new value of p.

[0184] In operation S1170, if the third change between the first loss function value and the seventh loss function value is greater than or equal to the target change, the value of p is reduced by the second preset value, and the next loop is executed. That is, after reducing the second preset value, the operation starts again from operation S1110 based on the latest value of p.

[0185] In some embodiments, a second preset value, such as 10% (for example only), can be preset. If a binary search is used, the value of R can be reduced by 1 from the current value of p, and (L+R)*0.5 can be updated to obtain the new value of p.

[0186] According to an embodiment of this disclosure, under the premise of meeting the target change amount, the convolution kernels in a certain layer are gradually pruned, and the resulting seventh neural network model is used as the sixth neural network model.

[0187] In some embodiments, after each iteration of the pruning analysis for each convolutional layer in the first neural network model, the target change can be updated based on the final result. The following section combines... Figure 12 Further details.

[0188] Figure 12A flowchart illustrating model pruning according to another embodiment of the present disclosure is shown schematically.

[0189] like Figure 12 As shown, the model pruning in this embodiment may include operations S710-S750 and S810-S840. If the value of j is N, the target change amount can also be updated, specifically including operations S1210-S1240. Operations S710-S750 and S810-S840 are the same as those in the above embodiment and will not be repeated here.

[0190] In operation S1210, if the value of j is equal to N, obtain the third inference time of the current pruning model running on the edge device to be deployed.

[0191] Because the value of j increases by 1 in each loop, during the increment process, if the condition that j is less than N is not true, the value of j is equal to N.

[0192] The time it takes to run the current pruning model on the edge device to be deployed, process specific input data using the model, and obtain the processing result is the third inference time.

[0193] In operation S1220, if the third inference time is less than the target inference time, the target change amount is reduced by the third preset value.

[0194] If the third inference time is less than the target inference time, it means that the current model satisfies both the LOSS condition (i.e., less than the target change) and the inference acceleration condition (i.e., less than the target inference time), thus reducing the target change.

[0195] In some embodiments, a third preset value, such as 5 seconds, can be preset (example only). Alternatively, an intermediate value can be taken based on the current target change.

[0196] In operation S1230, if the third inference time is greater than or equal to the target inference time, the target change amount is increased by a fourth preset value.

[0197] If the third inference time is greater than or equal to the target inference time, it means that there is no model that satisfies both the LOSS condition and the inference acceleration condition. Therefore, the target change amount should be increased.

[0198] In some embodiments, a third preset value, such as 5 seconds, can be preset (for example only). The third preset value can also be flexibly determined based on the current target change.

[0199] In operation S1240, determine whether 0 ≤ △Loss2 < W is true. If yes, then execute operation S710. If no, then end the loop.

[0200] In some embodiments, the target change is not unlimited in terms of increase or decrease; its minimum value is 0 when the target change is an absolute value. If the target change is too large, it indicates a significant performance change in the new model compared to the original model after pruning; in this case, the new model is not considered. The value of W can be determined based on the specific circumstances.

[0201] According to embodiments of this disclosure, by flexibly adjusting the target variation amount, the efficiency of obtaining a model that satisfies both the LOSS condition and the inference acceleration condition can be improved.

[0202] like Figure 12 As shown, the edge device inference acceleration method of this disclosure embodiment may further include operation S1250. In operation S1250, each time the current model to be pruned is updated, the updated current model to be pruned is used as a second neural network model. For example, in a loop, if the fourth neural network model is used as the current model to be pruned in operation S740, then the fourth neural network model is correspondingly used as a second neural network model. If the fifth neural network model is used as the current model to be pruned in operation S840, then the fifth neural network model is correspondingly used as a second neural network model.

[0203] According to embodiments of this disclosure, after model pruning, one or more second neural network models can be obtained, and one of these models can be selected for deployment. The following describes... Figure 13 Let me introduce it.

[0204] Figure 13 A flowchart illustrating an edge device inference acceleration method according to another embodiment of the present disclosure is shown schematically.

[0205] like Figure 13 As shown, the edge device inference acceleration method of this embodiment may include operations S210, S220, and S240, and may also include operations S1310 to S1330. Operation S1310 is one embodiment of operation S230. Operations S210, S220, and S240 can be referred to the above embodiments and will not be repeated here.

[0206] In operation S1310, the first inference time of each second neural network is obtained.

[0207] In operation S1320, a second neural network model with the minimum loss function value is determined from at least one second neural network model that meets the deployment requirements.

[0208] In operation S1330, a second neural network model with the minimum loss function value is deployed on the edge device.

[0209] According to embodiments of this disclosure, the final deployed model meets the inference acceleration conditions and is close to the accuracy of the first neural network model, thus achieving more stable data processing results.

[0210] In other embodiments, a second neural network model with a minimum third change can be determined from at least one second neural network model that meets the deployment requirements, wherein the third change includes either the first change or the second change. The second neural network model with the minimum third change is then deployed on an edge device.

[0211] Embodiments of this disclosure also provide a data processing system deployed on an edge device, wherein: a second neural network model in the data processing system for processing specific input data is based on the above... Figures 2-13 The specific input data obtained, as described in one or more embodiments, includes any one of image data, voice data, natural language data, or video data. (See also...) Figure 1 The data processing system can be deployed on edge devices 101, 102 and 103.

[0212] Figure 14 A flowchart illustrating an edge device inference acceleration method according to another embodiment of the present disclosure is shown schematically.

[0213] like Figure 14 As shown, the method of this embodiment includes operations S1401 to S1416.

[0214] In operation S1401, the original model is built and the weights are imported. The training and validation datasets are read, and the loss value test_loss_ori of the original model is obtained by testing on the validation set. The inference time constraint target_time is set, and the left boundary value of the loss threshold is initialized to LOSS_L=0.0 and the right boundary value is LOSS_R=0.1 (for example only). The result set is set to an empty set.

[0215] In operation S1402, determine whether the absolute value of the difference between LOSS_L and LOSS_R is less than 0.005 (for example only). If yes, then execute operation S1403. If no, then execute operation S1404.

[0216] In operation S1403, if the inequality holds, then select the model with the smallest loss from the result set as the compressed model, save it, and exit the analysis.

[0217] In operation S1404, Find_flag=False, Delta_loss=LOSS_R, and STEP=(LOSS_R-LOSS_L). Find_flag serves as an identifier for the change in the target, and Delta_loss represents the change in the target.

[0218] In operation S1405, read the comprehensive importance of the convolutional layer, that is, the first score, and refer to operations S510 to S530 and operations S610 to S630. Sort the comprehensive importance to obtain the first sorting result, and start analyzing from the layer with the lowest importance.

[0219] In operation S1406, for a certain layer in the model, first perform layer pruning. If this layer can be layer-pruned under the constraint of the Delta_loss threshold, then execute operation S1407; otherwise, execute operation S1408.

[0220] In operation S1407, if this layer can be layer-pruned, then send it to the target hardware platform to test the model speed temp_time.

[0221] In operation S1408, if this layer cannot be layer-pruned, then perform convolutional kernel pruning on this layer.

[0222] In operation S1409, if there is a model that satisfies Deta_loss during the convolutional kernel pruning process, then send it to the target hardware platform to test the model speed temp_time.

[0223] In operation S1410, judge whether temp_time < target_time holds. If so, then execute operations S1411 and S1412. If not, then execute operation S1412.

[0224] In operation S1411, add the current model to the analysis result set and set Find_flag = True.

[0225] In operation S1412, judge whether this layer is the layer with the highest importance score. If it is the layer with the highest importance, then it may be the last convolutional layer. If so, then execute operation S1414. If not, then execute operation S1413.

[0226] In operation S1413, start analyzing the next layer.

[0227] In operation S1414, judge whether Find_flag is True. If so, then execute operation S1415. If not, then execute operation S1416.

[0228] In operation S1415, if Find_flag is True, it means that there exists a model that satisfies both the LOSS condition and the inference speed condition between the current LOSS_L and LOSS_R thresholds. Therefore, the threshold is reduced, i.e., LOSS_R = (LOSS_L + LOSS_R) / 2, and the current LOSS_R is recorded as MaxR (i.e., subsequent searches do not need to exceed this maxR value). Then, operation S1402 is executed.

[0229] In operation S1416, if Find_flag is False, it means that there is no pruned model in the current threshold range that satisfies both the LOSS threshold and the inference speed condition, so the threshold needs to be expanded. Check if MaxR is defined. If yes, execute operation S1418. If no, execute operation S1417.

[0230] In operation S1417, set LOSS_L = LOSS_R and LOSS_R = LOSS_R + 2 * STEP. Then execute operation S1402.

[0231] In operation S1418, let LOSS_L = LOSS_R, LOSS_R = (LOSS_R + MaxR) / 2. Then execute operation S1402.

[0232] Based on the above-described edge device inference acceleration method, this disclosure also provides an edge device inference acceleration device. The following will be combined with... Figure 15 The device is described in detail.

[0233] Figure 15 A schematic block diagram of an edge device inference acceleration apparatus according to an embodiment of the present disclosure is shown.

[0234] like Figure 15 As shown, the edge device inference acceleration device 1500 of this embodiment includes a model acquisition module 1510, a model pruning module 1520, a time acquisition module 1530, and a model determination module 1540.

[0235] The model acquisition module 1510 can perform operation S210 to obtain a pre-trained first neural network model, wherein the first neural network model includes N convolutional layers, each convolutional layer includes at least one convolutional kernel, and N is an integer greater than or equal to 1.

[0236] The model pruning module 1520 can perform operation S220 to prune at least one convolutional layer and / or at least one convolutional kernel in the first neural network model to obtain a second neural network model.

[0237] The time acquisition module 1530 can perform operation S230 to obtain the first inference time of the second neural network model running on the edge device to be deployed, wherein the first inference time includes the time to process specific input data and obtain the processing result using the second neural network model.

[0238] The model determination module 1540 can perform operation S240, which determines the second neural network model as a model that meets the deployment requirements of the edge device if the first inference time is less than the target inference time. The deployment requirements include the inference acceleration requirements of the edge device to be deployed, and the target inference time is preset according to the performance of the edge device to be deployed.

[0239] According to embodiments of the present disclosure, the edge device inference acceleration device 1500 may further include a model training module for performing operations S310 to S320.

[0240] According to embodiments of the present disclosure, the edge device inference acceleration device 1500 may further include a first sorting module for performing operations S410-S420, S510-S540, and S610-S630.

[0241] According to embodiments of this disclosure, the model pruning module 1520 can also be used to prune each convolutional layer sequentially according to the ascending order in the first sorting result.

[0242] According to embodiments of this disclosure, the model pruning module 1520 can also perform operations S710 to S750 and operations S810 to S840.

[0243] According to embodiments of this disclosure, the edge device inference acceleration device 1500 may further include a second sorting module for performing operations S910-S920 and operations S1010-S1030.

[0244] According to embodiments of this disclosure, the model pruning module 1520 can also perform operations S1110 to S1170.

[0245] According to embodiments of the present disclosure, the edge device inference acceleration device 1500 may further include a target change update module for performing operations S1210 to S1240.

[0246] According to embodiments of the present disclosure, the edge device inference acceleration device 1500 may further include a model deployment module for performing operations S1250, S1310 to S1330.

[0247] It should be noted that the implementation methods, technical problems solved, functions achieved, and technical effects of each module / unit / subunit in the device embodiments are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of each corresponding step in the method embodiments, and will not be repeated here.

[0248] According to embodiments of this disclosure, any multiple modules among the model acquisition module 1510, model pruning module 1520, time acquisition module 1530, and model determination module 1540 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in one module.

[0249] According to embodiments of this disclosure, at least one of the model acquisition module 1510, model pruning module 1520, time acquisition module 1530, and model determination module 1540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the model acquisition module 1510, model pruning module 1520, time acquisition module 1530, and model determination module 1540 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0250] Embodiments of this disclosure provide an edge device inference acceleration system, including an electronic device and an edge device.

[0251] Figure 16 A block diagram schematically illustrates an electronic device suitable for implementing an edge device inference acceleration method according to an embodiment of the present disclosure.

[0252] like Figure 16As shown, an electronic device 1600 according to an embodiment of the present disclosure includes a processor 1601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage portion 1608 into a random access memory (RAM) 1603. The processor 1601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1601 may also include onboard memory for caching purposes. The processor 1601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0253] RAM 1603 stores various programs and data required for the operation of electronic device 1600. Processor 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Processor 1601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1602 and / or RAM 1603. It should be noted that programs may also be stored in one or more memories other than ROM 1602 and RAM 1603. Processor 1601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0254] According to embodiments of this disclosure, the electronic device 1600 may further include an input / output (I / O) interface 1605, which is also connected to a bus 1604. The electronic device 1600 may also include one or more of the following components connected to the I / O interface 1605: an input section 1606 including a keyboard, mouse, etc.; an output section 1607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN card, modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the I / O interface 1605 as needed. A removable medium 1611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1610 as needed so that computer programs read from it can be installed into the storage section 1608 as needed.

[0255] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0256] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1602 and / or RAM 1603 and / or one or more memories other than ROM 1602 and RAM 1603 described above.

[0257] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0258] When the computer program is executed by the processor 1601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0259] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1609, and / or installed from a removable medium 1611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0260] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1609, and / or installed from the removable medium 1611. When the computer program is executed by the processor 1601, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0261] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0262] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0263] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0264] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for accelerating inference on edge devices, comprising: Obtain a pre-trained first neural network model, wherein the first neural network model includes N convolutional layers, each convolutional layer includes at least one convolutional kernel, and N is an integer greater than or equal to 1; Prune at least one convolutional layer and / or at least one convolutional kernel in the first neural network model to obtain a second neural network model; The first inference time of the second neural network model running on the edge device to be deployed is obtained, wherein the first inference time includes the time to process input data and obtain processing results using the second neural network model; If the first inference time is less than the target inference time, the second neural network model is determined to be a model that meets the deployment requirements of the edge device. The deployment requirements include inference acceleration requirements for the edge device to be deployed. The target inference time is preset according to the performance of the edge device to be deployed. The performance is related to at least one of the following factors: the model, purpose, input data type, memory access bandwidth, and data alignment of the edge device. Prior to pruning at least one convolutional layer and / or at least one convolutional kernel in the first neural network model, the method further includes: Based on the importance of each convolutional layer, a first score is obtained for each convolutional layer, wherein the importance is used to characterize the degree of influence on the first loss function value of the first neural network model on the validation set; Obtaining the first score for each convolutional layer includes: For each convolutional layer, after obtaining the loss function value when the layer is completely pruned and the loss function value of the convolutional kernel with M pruning ratios, the first score is obtained based on multiple loss function values, where M is an integer greater than or equal to 1.

2. The method according to claim 1, wherein, Before pruning at least one convolutional layer and / or at least one convolutional kernel in the first neural network model, the method further includes: The first sorting result is obtained by sorting the N convolutional layers according to their first scores, and the first sorting result is used as the basis for pruning each convolutional layer.

3. The method according to claim 2, wherein, The i-th convolutional layer is any one of the N convolutional layers, where i is greater than or equal to 1 and i is less than or equal to N. Obtaining the first score for each convolutional layer includes: setting the initial value of i to 1, and performing the following operations in a loop. The third neural network model is obtained by pruning the i-th convolutional layer from the first neural network model; Obtain the third loss function value of the third neural network model on the validation set; The first score of the i-th convolutional layer is obtained based on the value of the third loss function. If the value of i is less than N, increment the value of i by 1.

4. The method according to claim 3, wherein, The first score is obtained based on multiple loss function values, including: Based on the M pruning ratios, the convolution kernels in the i-th convolutional layer are pruned respectively to obtain M fourth neural network models; Obtain the M fourth loss function values ​​of the M fourth neural network models on the validation set; The first score of the i-th convolutional layer is obtained based on the third loss function value, including: The first score of the i-th convolutional layer is obtained based on the third loss function value and the M fourth loss function values.

5. The method according to claim 2, wherein, The first score of each convolutional layer is positively correlated with its importance. Pruning at least one convolutional layer and / or at least one convolutional kernel in the first neural network model yields the second neural network model, which includes: Pruning is performed on each convolutional layer in ascending order according to the first sorting result.

6. The method according to claim 5, wherein, The j-th order is any one of the first sorting results, where j is greater than or equal to 1 and j is less than or equal to N. The step of pruning each convolutional layer sequentially according to the ascending order of the first sorting results includes: setting the initial value of j to 1, and looping through the following operations: Based on the current model to be pruned, the j-th sequential convolutional layer in the first neural network model is pruned to obtain the fifth neural network model; Obtain the fifth loss function value of the fifth neural network model on the validation set; If the first change between the first loss function value and the fifth loss function value is less than the target change, the fifth neural network model is taken as the current model to be pruned, and if the value of j is less than N, the value of j is incremented by 1 before the next loop is executed.

7. The method according to claim 6, wherein, If the first change between the first loss function value and the fifth loss function value is greater than or equal to the target change, the method further includes: Pruning the kernels of the j-th sequential convolutional layer yields the sixth neural network model; Obtain the sixth loss function value of the sixth neural network model on the validation set; If the second change between the first loss function value and the sixth loss function value is less than the target change, the sixth neural network model is taken as the current model to be pruned, and if the value of j is less than N, the value of j is incremented by 1 before the next iteration is executed; or If the second change between the first loss function value and the sixth loss function value is greater than or equal to the target change, the sixth neural network model is not used as the current model to be pruned, and if the value of j is less than N, the value of j is incremented by 1 before the next loop is executed.

8. The method according to claim 7, wherein, Before performing kernel pruning on the j-th sequential convolutional layer, the following is included: A second score is obtained for each convolutional kernel based on the importance of each kernel. The second scores of all convolutional kernels in each convolutional layer are sorted to obtain N second sorting results, which are used as the basis for pruning each convolutional kernel.

9. The method according to claim 8, wherein, The i-th convolutional layer is any one of the N convolutional layers, where i is greater than or equal to 1 and less than or equal to N. Obtaining the second score for each convolutional kernel includes: setting the initial value of i to 1, and performing the following operations in a loop. Using the validation set as input to the first neural network model, parameter pairs for each convolutional kernel in the i-th convolutional layer are obtained, wherein the parameter pairs include gradient values ​​and weight values, and the gradient values ​​are obtained according to the backpropagation algorithm; Based on the parameter pairs of each convolution kernel in the i-th convolutional layer, the second score of each convolution kernel in the i-th convolutional layer is obtained; If the value of i is less than N, increment the value of i by 1.

10. The method according to claim 9, wherein, The validation set includes S subsets, where S is greater than or equal to 2. The step of using the validation set as input to the first neural network model to obtain the parameter pairs for each convolutional kernel in the i-th convolutional layer includes: Using at least two subsets from the S subsets as input to the first neural network model, at least two parameter pairs are obtained for each convolutional kernel in the i-th convolutional layer, wherein each parameter pair is obtained based on each subset from the at least two subsets.

11. The method according to claim 10, wherein, The step of obtaining the second score of each convolutional kernel in the i-th convolutional layer based on the parameter pairs of each convolutional kernel in the i-th convolutional layer includes: Based on the at least two parameter pairs, at least two third scores are obtained for each convolutional kernel in the i-th convolutional layer; Based on the at least two third scores, the second score of each convolutional kernel in the i-th convolutional layer is obtained.

12. The method according to claim 8, wherein, The second score of each convolutional kernel is positively correlated with its importance. Convolutional kernel pruning for the j-th sequential convolutional layer includes: Based on the ascending order of the second sorting results, at least one convolutional kernel in the convolutional layer of the j-th order is pruned.

13. The method according to claim 12, wherein, The j-th sequential convolutional layer includes Q convolutional kernels, where Q is greater than or equal to 1. The step of pruning at least one convolutional kernel in the j-th sequential convolutional layer according to the ascending order of the second sorting result includes: The following operations are performed repeatedly, wherein the model obtained after the loop ends serves as the sixth neural network model: Based on the ascending order of the second sorting results, p convolutional kernels are obtained; If the value of p is greater than or equal to 1 and less than Q, the p convolutional kernels are pruned to obtain the seventh neural network model; Obtain the seventh loss function value of the seventh neural network model on the validation set; If the third change between the first loss function value and the seventh loss function value is less than the target change, the value of p is increased by a first preset value, and the next loop is executed.

14. The method of claim 13, wherein: If the third change between the first loss function value and the seventh loss function value is greater than or equal to the target change, the value of p is reduced by a second preset value, and the next loop is executed.

15. The method according to claim 6 or 7, wherein, The obtained second neural network model includes: Each time the current model to be pruned is updated, the updated current model to be pruned is used as a second neural network model.

16. The method according to claim 15, wherein, The process of obtaining the first inference time of the second neural network model running on the edge device to be deployed includes: Obtain the first inference time for each of the second neural networks; The method further includes: From at least one second neural network model that meets the deployment requirements, determine the second neural network model with the minimum loss function value; The second neural network model with the minimum loss function value is deployed on the edge device.

17. The method according to claim 6 or 7, wherein, Also includes: If the value of j is equal to N, obtain the third inference time of the current pruning model running on the edge device to be deployed; The method further includes updating the target change, specifically including: If the third inference time is less than the target inference time, the target change amount is reduced by a third preset value.

18. The method according to claim 17, wherein, The update of the target change also includes: If the third inference time is greater than or equal to the target inference time, the target change amount is increased by a fourth preset value.

19. The method according to claim 1, wherein, Before obtaining the first inference time of the second neural network model running on the edge device to be deployed, the following is also included: The second neural network model is trained using a training set, wherein the training data in the training set includes any one of image data, speech data, natural language data, or video data. The trained second neural network model is then run on the edge device to be deployed.

20. The method according to claim 1, wherein, The input data includes any one of image data, voice data, natural language data, or video data.

21. An edge device inference acceleration device, comprising: The model acquisition module is used to acquire a pre-trained first neural network model, wherein the first neural network model includes N convolutional layers, each convolutional layer includes at least one convolutional kernel, and N is an integer greater than or equal to 1; The model pruning module is used to prune at least one convolutional layer and / or at least one convolutional kernel in the first neural network model to obtain a second neural network model. The time acquisition module is used to acquire the first inference time of the second neural network model running on the edge device to be deployed, wherein the first inference time includes the time to process input data and obtain processing results using the second neural network model; The model determination module is used to determine that the second neural network model meets the deployment requirements of the edge device if the first inference time is less than the target inference time. The deployment requirements include inference acceleration requirements for the edge device to be deployed. The target inference time is preset according to the performance of the edge device to be deployed. The performance is related to at least one of the following factors: the model, purpose, input data type, memory access bandwidth, and data alignment of the edge device. Prior to pruning at least one convolutional layer and / or at least one convolutional kernel in the first neural network model, the method further includes: Based on the importance of each convolutional layer, a first score is obtained for each convolutional layer, wherein the importance is used to characterize the degree of influence on the first loss function value of the first neural network model on the validation set; Obtaining the first score for each convolutional layer includes: For each convolutional layer, after obtaining the loss function value when the layer is completely pruned and the loss function values ​​of convolutional kernels with different pruning ratios, the first score is obtained based on multiple loss function values.

22. A data processing system deployed on an edge device, wherein: The second neural network model used to process input data in the data processing system is obtained according to the method described in claim 1, wherein the input data includes any one of image data, voice data, natural language data, or video data.

23. An edge device inference acceleration system, comprising electronic devices, wherein, The electronic device includes: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 20.

24. The system according to claim 23, wherein, Also includes: An edge device is used to run a second neural network model sent by the one or more processors to obtain a first inference time; The one or more processors are used to obtain the first inference time from the edge device.

25. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 20.

Images