A residual network memory optimization method, device, equipment and medium

Abstract

The application discloses a residual network memory optimization method and device, equipment and medium, and relates to the technical fields of residual network and embedded device. The method comprises the following steps: obtaining a trained deep learning model, setting the input quantization type of each Add node in the residual network as an INT32 type, and building a deep learning inference framework of an embedded device; performing inference on the deep learning model by using the deep learning inference framework, so as to determine a target Add node causing a memory bottleneck from each Add node; setting the input quantization type of the target Add node as an INT7 type, updating the corresponding quantization factor in the deep learning inference framework, adding a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function in the deep learning inference framework, and obtaining an optimized deep learning inference framework. The method solves the memory bottleneck problem of the residual network on the embedded device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of residual network and embedded device, and particularly relates to a residual network memory optimization method, device, equipment and medium. BACKGROUND

[0002] In recent years, deep learning neural network models are widely used in many fields and have achieved very good results. Model inference, as a stage of model use, is closely related to the hardware, environment, etc. of running the model, and model inference frameworks that adapt to different hardware have appeared. In addition, embedded devices have low power consumption, limited computing power and memory resources, and high requirements for the deployment of deep learning models. The most prominent problem in the model inference stage is large memory occupation, especially the large memory occupation of activation values, which conflicts with the limited memory resources of embedded devices. Residual network is an important module in convolutional neural network, and appears in many important convolutional neural networks, including commonly used convolutional neural networks such as MobileNetV2, ResNet, etc. In embedded devices, memory resources are very limited, and the Add operator of the residual network is often a bottleneck of memory occupation because two output tensors need to be added, and the output tensor is the part with the highest memory occupation in model inference. In order to reduce the memory occupation of the output tensor, a quantization method is usually used, that is, in each convolutional node, the convolution is calculated using quantized weights and inputs, and is inversely quantized to INT8 type as output. Compared with 32-bit floating point numbers, 8-bit integers greatly reduce the memory occupation. However, for the Add node in the residual network, if the INT8 outputs of the two convolutional nodes are added, data overflow will occur. Therefore, for the input of the Add node, only 32-bit data can be used, and the convolutional nodes in front are often inversely quantized to INT32 data type as output, which avoids data overflow but causes a memory occupation bottleneck problem.

[0003] In summary, how to avoid the memory bottleneck problem of the residual network when it is inferred on the embedded device is a problem to be solved at present. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a residual network memory optimization method, device, equipment and medium, which can avoid the memory bottleneck problem of the residual network when it is inferred on the embedded device. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a residual network memory optimization method applied to an embedded device, comprising:

[0006] obtaining a trained deep learning model, setting the input quantization type of each Add node of the residual network in the deep learning model to an INT32 type, and building a deep learning inference framework based on the embedded device;

[0007] inference on the deep learning model by using the deep learning inference framework on the server to determine a target Add node causing the memory bottleneck from the Add nodes;

[0008] setting an input quantization type of the target Add node as an INT7 type, updating a corresponding quantization factor in the deep learning inference framework based on the INT7 type, and adding a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function in the deep learning inference framework to obtain an optimized deep learning inference framework.

[0009] Optionally, the updating of the corresponding quantization factor in the deep learning inference framework based on the INT7 type comprises:

[0010] updating a quantization factor corresponding to an input of the target Add node and updating an inverse quantization factor corresponding to an output of an upper node of the target Add node in the deep learning inference framework based on the INT7 type.

[0011] Optionally, after the obtaining of the optimized deep learning inference framework, the method further comprises:

[0012] inference on the deep learning model by using the optimized deep learning inference framework.

[0013] Optionally, in the inference on the deep learning model by using the optimized deep learning inference framework, the method further comprises:

[0014] obtaining target data output by an upper node based on an input quantization type set for a current Add node in a residual network, so as to use the target data as input data of the current Add node;

[0015] determining a target Add operator based on the input quantization type of the current Add node, and processing the input data by using the target Add operator to obtain an output result.

[0016] Optionally, the determining of the target Add operator based on the input quantization type of the current Add node and the processing of the input data by using the target Add operator to obtain the output result comprise:

[0017] if the input quantization type of the current Add node is an INT32 type, the corresponding target Add operator is a first Add operator with an INT32 type data addition function, and the input data is processed by using the first Add operator to obtain an output result of an INT32 type;

[0018] If the input quantization type of the current Add node is of the INT7 type, a corresponding target Add operator is a second Add operator having an INT8 type data addition function, and the input data is processed using the second Add operator to obtain an output result of the INT8 type.

[0019] Optionally, after processing the input data using the target Add operator to obtain an output result, the method further includes:

[0020] If the output result is of the INT32 type, the output result is quantized according to the IN8 type, and the quantized output result is input as a lower layer node;

[0021] If the output result is of the INT8 type, the output result is directly input as a lower layer node.

[0022] Optionally, the determining, using the deep learning inference framework on the server, of the target Add node causing the memory bottleneck from the Add nodes includes:

[0023] The deep learning inference framework is used on the server to infer the deep learning model, to determine an Add node occupying higher memory and corresponding memory occupation information from the Add nodes;

[0024] The memory occupation information is compared with memory information of the embedded device, to determine the target Add node causing the memory bottleneck from the Add node occupying higher memory.

[0025] In a second aspect, the present application discloses a residual network memory optimization device, applied to an embedded device, including:

[0026] A framework building module is configured to obtain a trained deep learning model, set input quantization types of each Add node of a residual network in the deep learning model to be of an INT32 type, and build a deep learning inference framework based on an embedded device;

[0027] A model inference module is configured to use the deep learning inference framework on a server to infer the deep learning model, to determine a target Add node causing a memory bottleneck from each Add node;

[0028] The framework optimization module is configured to set an input quantization type of the target Add node to an INT7 type, update a corresponding quantization factor in the deep learning inference framework based on the INT7 type, and add a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function in the deep learning inference framework to obtain an optimized deep learning inference framework.

[0029] In a third aspect, the present application discloses an electronic device, comprising:

[0030] A memory for storing a computer program;

[0031] A processor for executing the computer program to implement the steps of the residual network memory optimization method disclosed above.

[0032] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the steps of the residual network memory optimization method disclosed above.

[0033] It can be seen that the application discloses a residual network memory optimization method applied to an embedded device. The method comprises the following steps: obtaining a trained deep learning model, setting an input quantization type of each Add node of a residual network in the deep learning model as an INT32 type, and building a deep learning inference framework based on the embedded device; performing inference on the deep learning model by using the deep learning inference framework on a server to determine a target Add node causing a memory bottleneck from each Add node; setting the input quantization type of the target Add node as an INT7 type, updating a corresponding quantization factor in the deep learning inference framework based on the INT7 type, and adding a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function in the deep learning inference framework to obtain an optimized deep learning inference framework. It can be seen that the application firstly obtains the trained deep learning model, and sets the input quantization type of each Add node of the residual network in the deep learning model as the INT32 type to avoid data overflow problems; then the deep learning inference framework of the embedded device is built, and the deep learning model is inferred by using the deep learning inference framework on the server to determine the target Add node causing the memory bottleneck from each Add node, so that the input quantization type of the target Add node is set as the INT7 type, that is, the application does not use the INT7 quantization for all Add nodes, but uses the INT7 quantization for the target Add node with higher memory occupation and causing the memory bottleneck, so that the memory bottleneck can be avoided, and the precision decline problem caused by the INT7 quantization can be avoided. In addition, the corresponding quantization factor in the deep learning inference framework is updated based on the INT7 type, and the first Add operator with the INT32 type data addition function and the second Add operator with the INT8 type data addition function are added in the deep learning inference framework to obtain the optimized deep learning inference framework, so that the deep learning model can be inferred by using the optimized deep learning inference framework. Through the above scheme, the application can effectively solve the memory bottleneck problem of the residual network when the residual network is inferred on the embedded device. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings.

[0035] Figure 1 A residual network memory optimization method flow chart is disclosed in the application.

[0036] Figure 2 This application discloses a flowchart of a specific residual network memory optimization method.

[0037] Figure 3 This is a schematic diagram of a specific residual network memory optimization process disclosed in this application;

[0038] Figure 4 This is a schematic diagram of the structure of a residual network memory optimization device disclosed in this application;

[0039] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0041] Currently, to reduce the memory footprint of the output tensor of the Add node in a residual network, quantization is commonly used. This involves calculating the convolution with quantized weights and input in each convolutional node layer, and then dequantizing the result to INT8 type as the output. Compared to 32-bit floating-point numbers, 8-bit integers significantly reduce memory usage. However, for the Add node in a residual network, adding the INT8 outputs of two convolutional nodes can lead to data overflow when writing. Therefore, the input to the Add node must be 32-bit data, and the preceding convolutional nodes are often dequantized to INT32 data type for output. This avoids data overflow but introduces a memory bottleneck. To address this, this application discloses a method, apparatus, device, and medium for optimizing residual network memory, which can prevent memory bottlenecks during inference on embedded devices.

[0042] See Figure 1 As shown in the figure, this application discloses a residual network memory optimization method, applied to an embedded device, the method comprising:

[0043] Step S11: Obtain the trained deep learning model, set the input quantization type of each Add node of the residual network in the deep learning model to INT32 type, and build a deep learning inference framework based on embedded devices.

[0044] In this embodiment, first, the trained deep learning model is obtained, and the input quantization type of each Add node of the residual network in the deep learning model is set to the INT32 type to avoid overflow caused by adding two INT8 type data, so the Add node of the residual network calculates the data of the inverse quantization to the INT32 type, and then the deep learning inference framework based on the embedded device is built. In addition, the above-mentioned trained deep learning model can also be obtained by currently training the deep learning model.

[0045] Step S12: Inference is performed on the deep learning model on the server by using the deep learning inference framework to determine the target Add node causing the memory bottleneck from each Add node.

[0046] In this embodiment, the memory occupation of the Add node is further evaluated, specifically, the deep learning model is inferred on the server by using the deep learning inference framework to determine the target Add node causing the memory bottleneck from each Add node.

[0047] Step S13: The input quantization type of the target Add node is set to the INT7 type, and the corresponding quantization factor is updated in the deep learning inference framework based on the INT7 type, and a first Add operator with the function of adding INT32 type data and a second Add operator with the function of adding INT8 type data are added in the deep learning inference framework to obtain an optimized deep learning inference framework.

[0048] In this embodiment, the target Add node of the memory occupation bottleneck is set to have an input quantization type of INT7 type. That is, the present application does not use INT7 quantization for all Add nodes, but uses INT7 quantization for the target Add node that occupies higher memory and causes memory bottleneck. In this way, the memory bottleneck can be avoided, and the precision degradation problem caused by INT7 quantization can also be avoided. It should be pointed out that here INT7 only represents the quantization range [-63, 63], but the data type still uses INT8 data type to support general embedded devices. Then, the corresponding quantization factor is updated in the deep learning inference framework based on the INT7 type, that is, for the node with operator type Add and set to INT7 quantization type, the corresponding quantization factor needs to be updated. In the specific embodiment, the above updating the corresponding quantization factor in the deep learning inference framework based on the INT7 type includes: updating the quantization factor corresponding to the input of the target Add node in the deep learning inference framework based on the INT7 type, and updating the inverse quantization factor corresponding to the output of the upper node of the target Add node. It can be understood that when updating the corresponding quantization factor, the quantization factor corresponding to the input of the target Add node needs to be updated, and the inverse quantization factor corresponding to the output of the upper node of the target Add node also needs to be updated. Specifically, the quantization factor of the Add node input is divided by 2, converted from INT8 quantization to INT7 quantization, and the inverse quantization factor of the node taking the Add node input as the output, that is, the inverse quantization factor of the upper node, is updated, that is, the inverse quantization factor is multiplied by 2.

[0049] Further, a first Add operator with INT32 type data addition function and a second Add operator with INT8 type data addition function need to be added in the deep learning inference framework to obtain an optimized deep learning inference framework. That is, in addition to the first Add operator supporting INT32 type data addition in the deep learning inference framework, a second Add operator supporting INT8 type data addition also needs to be added to obtain an optimized deep learning inference framework. It should be pointed out that subsequent attention needs to be paid to check whether the input data is within the INT7 data range to avoid data overflow.

[0050] It can be seen that the application discloses a residual network memory optimization method applied to an embedded device. The method comprises the following steps: obtaining a trained deep learning model, setting an input quantization type of each Add node of a residual network in the deep learning model as an INT32 type, and building a deep learning inference framework based on an embedded device; performing inference on the deep learning model by using the deep learning inference framework on a server, so as to determine a target Add node causing a memory bottleneck from each Add node; setting the input quantization type of the target Add node as an INT7 type, updating a corresponding quantization factor in the deep learning inference framework based on the INT7 type, and adding a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function in the deep learning inference framework, so as to obtain an optimized deep learning inference framework. It can be seen that the application firstly obtains a trained deep learning model, and sets an input quantization type of each Add node of a residual network in the deep learning model as an INT32 type, so as to avoid data overflow problems; then a deep learning inference framework of an embedded device is built, and the deep learning model is inferred by using the deep learning inference framework on a server, so as to determine a target Add node causing a memory bottleneck from each Add node, and the input quantization type of the target Add node is set as an INT7 type. That is, the application does not use INT7 quantization for all Add nodes, but uses INT7 quantization for a target Add node occupying higher memory and causing a memory bottleneck. In this way, the memory bottleneck can be avoided, and the precision decline problem caused by INT7 quantization can also be avoided. In addition, a corresponding quantization factor is updated in the deep learning inference framework based on the INT7 type, and a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function are added in the deep learning inference framework, so as to obtain an optimized deep learning inference framework, so as to infer the deep learning model by using the optimized deep learning inference framework. Through the above scheme, the application can effectively solve the memory bottleneck problem of the residual network when the residual network is inferred on the embedded device.

[0051] Referring to Figure 2 and Figure 3 As shown in the application embodiment, the application embodiment discloses a specific residual network memory optimization method. Compared with the previous embodiment, the application embodiment further describes and optimizes the technical scheme. Specifically, the application embodiment comprises the following steps:

[0052] Step S21: obtaining a trained deep learning model, setting an input quantization type of each Add node of a residual network in the deep learning model as an INT32 type, and building a deep learning inference framework based on an embedded device.

[0053] Step S22: On the server, use the deep learning inference framework to perform inference on the deep learning model to determine the Add nodes with high memory usage and their corresponding memory usage information from each Add node.

[0054] In this embodiment, the memory usage bottleneck is evaluated by using a deep learning inference framework on the server to perform inference on the deep learning model, so as to output the Add nodes with high memory usage and the corresponding memory usage information. That is, the Add nodes with high memory usage and the corresponding memory usage information are determined from each Add node.

[0055] Step S23: Compare the memory usage information with the memory information of the embedded device to identify the target Add node causing the memory bottleneck from the Add nodes with high memory usage.

[0056] In this embodiment, the memory information of the currently selected embedded device is obtained and compared with the aforementioned memory usage information to determine the target Add node causing the memory bottleneck. It is understood that this embodiment first identifies the Add nodes with high memory usage, and then determines the target Add node causing the memory bottleneck based on the embedded device's memory information. That is, this application only uses INT7 quantization on a small number of target Add nodes with high memory usage that cause the memory bottleneck, avoiding the precision degradation problem caused by INT7 quantization and also avoiding the memory bottleneck. It is understood that in actual use, the number of Add nodes that become bottlenecks is often very small; for example, in MobileNetV2, only the first Add node has a very high memory usage.

[0057] Step S24: Set the input quantization type of the target Add node to INT7 type, update the corresponding quantization factor in the deep learning inference framework based on the INT7 type, and add a first Add operator with INT32 type data addition function and a second Add operator with INT8 type data addition function to the deep learning inference framework to obtain the optimized deep learning inference framework.

[0058] In this embodiment, the quantization module and the Add operator implementation module in the deep learning inference framework are optimized, namely, the quantization factor is updated and the Add operator is added.

[0059] Step S25: Use the optimized deep learning inference framework to perform inference on the deep learning model.

[0060] In this embodiment, after obtaining the optimized deep learning inference framework, the deep learning model is inferred using the optimized deep learning inference framework. In the specific implementation, the process of inferring the deep learning model using the optimized deep learning inference framework further includes: obtaining target data output by an upper node based on an input quantization type set for a current Add node in the residual network, so as to take the target data as input data of the current Add node; determining a target Add operator based on the input quantization type of the current Add node, and processing the input data using the target Add operator to obtain an output result. It can be understood that in the convolutional neural network, the output of the upper convolutional node is the input of the current convolutional node. In this embodiment, the target data output by the upper node is obtained based on the input quantization type set for the current Add node in the residual network, so as to take the target data as the input data of the current Add node. That is, the type of the input data of the current Add node and the type of the output data of the upper node can be determined based on the input quantization type set for the current Add node in the residual network. Then, the corresponding target Add operator is determined based on the input quantization type of the current Add node, and the input data is processed using the target Add operator to obtain an output result.

[0061] Further, the process of determining the target Add operator based on the input quantization type of the current Add node and processing the input data using the target Add operator to obtain an output result includes: if the input quantization type of the current Add node is an INT32 type, the corresponding target Add operator is a first Add operator having an INT32 type data addition function, and the input data is processed using the first Add operator to obtain an output result of an INT32 type; if the input quantization type of the current Add node is an INT7 type, the corresponding target Add operator is a second Add operator having an INT8 type data addition function, and the input data is processed using the second Add operator to obtain an output result of an INT8 type. That is, if the input quantization type set for the current Add node is an INT32 type, the corresponding target Add operator is a first Add operator having an INT32 type data addition function, and the output result obtained by processing the input data using the first Add operator is data of an INT32 type, so as to ensure the overall model precision; if the input quantization type of the current Add node is an INT7 type, the corresponding target Add operator is a second Add operator having an INT8 type data addition function, and the output result obtained by processing the input data using the second Add operator is data of an INT8 type.

[0062] Further, after the input data is processed by the target Add operator to obtain an output result, the method further includes: if the output result is of an INT32 type, quantizing the output result according to an IN8 type, and inputting the quantized output result as a lower layer node; and if the output result is of an INT8 type, directly inputting the output result as the lower layer node. That is, for a lower layer node of a target Add node, if the output data type of the target Add node is INT32, the output data needs to be quantized to INT8 and then calculated; and if the output data type of the Add node is INT8, the output data is directly calculated.

[0063] The specific process of steps S21 and S24 will be described below.

[0064] It can be seen that, in the embodiments of the present application, when the target Add node causing the memory bottleneck is determined, the deep learning model is first inferred by using the deep learning inference framework to output the Add node with high memory occupation and the corresponding memory occupation information, and then the memory occupation information is compared with the memory information of the currently selected embedded device to determine the target Add node causing the memory bottleneck, so that only the target Add node is quantized by INT7 to avoid the precision decline caused by INT7 quantization. In addition, after the optimized deep learning inference framework is obtained, the deep learning model is inferred by using the optimized deep learning inference framework, the target data output by the upper layer node is obtained based on the input quantization type set by the current Add node, the target data is used as the input data of the current Add node, and the input data is processed by using the target Add operator corresponding to the input quantization type to obtain an output result. If INT7 quantization is set, the second Add operator of INT8 type data addition is used, otherwise, the first Add operator of INT32 type data addition is continuously used to ensure the overall model precision. Through the above scheme, the present application can effectively solve the memory bottleneck problem of the residual network when the residual network is inferred on the embedded device.

[0065] Referring to Figure 4 The embodiments of the present application disclose a residual network memory optimization device applied to an embedded device, and the device comprises:

[0066] The framework building module 11 is configured to obtain a trained deep learning model, set the input quantization type of each Add node of a residual network in the deep learning model to an INT32 type, and build a deep learning inference framework based on an embedded device.

[0067] The model inference module 12 is configured to perform inference on the deep learning model by using the deep learning inference framework on the server to determine a target Add node causing the memory bottleneck from each of the Add nodes.

[0068] The framework optimization module 13 is configured to set an input quantization type of the target Add node to an INT7 type, update a corresponding quantization factor in the deep learning inference framework based on the INT7 type, and add a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function in the deep learning inference framework to obtain an optimized deep learning inference framework.

[0069] It can be seen that the application discloses a residual network memory optimization method applied to an embedded device. The method comprises the following steps: obtaining a trained deep learning model, setting an input quantization type of each Add node of a residual network in the deep learning model as an INT32 type, and building a deep learning inference framework based on the embedded device; performing inference on the deep learning model by using the deep learning inference framework on a server to determine a target Add node causing a memory bottleneck from each Add node; setting the input quantization type of the target Add node as an INT7 type, updating a corresponding quantization factor in the deep learning inference framework based on the INT7 type, and adding a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function in the deep learning inference framework to obtain an optimized deep learning inference framework. It can be seen that the application firstly obtains the trained deep learning model, and sets the input quantization type of each Add node of the residual network in the deep learning model as the INT32 type to avoid data overflow problems; then the deep learning inference framework of the embedded device is built, and the deep learning model is inferred by using the deep learning inference framework on the server to determine the target Add node causing the memory bottleneck from each Add node, so as to set the input quantization type of the target Add node as the INT7 type. That is, the application does not use the INT7 quantization for all Add nodes, but uses the INT7 quantization for the target Add node with higher memory occupation and causing the memory bottleneck. In this way, the memory bottleneck can be avoided, and the precision decline problem caused by the INT7 quantization can also be avoided. In addition, the corresponding quantization factor is updated in the deep learning inference framework based on the INT7 type, and the first Add operator with the INT32 type data addition function and the second Add operator with the INT8 type data addition function are added in the deep learning inference framework to obtain the optimized deep learning inference framework, so as to infer the deep learning model by using the optimized deep learning inference framework. Through the above scheme, the application can effectively solve the memory bottleneck problem of the residual network when the residual network is inferred on the embedded device.

[0070] Figure 5 A structural schematic diagram of an electronic device provided by an embodiment of the application is provided. Specifically, it can include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the related steps in the residual network memory optimization method performed by the electronic device disclosed in any of the preceding embodiments.

[0071] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input and output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited herein.

[0072] The processor 21 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 21 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 can also include an AI (Artificial Intelligence) processor configured to process machine learning-related computing operations.

[0073] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223, etc. The storage mode can be temporary storage or permanent storage.

[0074] The operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, so as to realize the operation and processing of the processor 21 on the mass data 223 in the memory 22, and can be Windows, Unix, Linux, etc. In addition to the computer program capable of completing the residual network memory optimization method disclosed by the electronic device 20 in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work. In addition to the data 223 that can include the data received by the electronic device from the external device, the data 223 can also include the data collected by the self input and output interface 25, etc.

[0075] Further, the embodiment of the present application further discloses a computer readable storage medium, the storage medium stores a computer program, and the computer program is loaded and executed by the processor to realize the method steps executed in the residual network memory optimization process disclosed by any of the foregoing embodiments.

[0076] In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed by the embodiment, since it corresponds to the method disclosed by the embodiment, the description is relatively simple, and the related parts can be referred to the method part.

[0077] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or combination of the two. In order to clearly show the interchangeability of hardware and software, the composition and steps of each example have been described in the above description. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0078] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, software module executed by the processor, or combination of the two. The software module can be placed in random access memory (RAM), memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0079] Finally, it needs to be pointed out that in this article, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0080] The above describes in detail the residual network memory optimization method, device, equipment and storage medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples. The above example is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for residual network memory optimization, the method comprising: The application is applied to an embedded device, comprising: obtaining a trained deep learning model, setting an input quantization type of each Add node of a residual network in the deep learning model to an INT32 type, and building a deep learning inference framework based on an embedded device; inference of the deep learning model on a server by using the deep learning inference framework to determine a target Add node causing a memory bottleneck from each Add node; setting the input quantization type of the target Add node to an INT7 type, updating a corresponding quantization factor in the deep learning inference framework based on the INT7 type, and adding a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function in the deep learning inference framework to obtain an optimized deep learning inference framework; wherein the updating of the corresponding quantization factor in the deep learning inference framework based on the INT7 type comprises: updating a quantization factor corresponding to an input of the target Add node and updating an inverse quantization factor corresponding to an output of an upper node of the target Add node in the deep learning inference framework based on the INT7 type.

2. The residual network memory optimization method of claim 1, wherein, after obtaining the optimized deep learning inference framework, further comprising: inference of the deep learning model by using the optimized deep learning inference framework.

3. The residual network memory optimization method of claim 2, wherein, in the process of inference of the deep learning model by using the optimized deep learning inference framework, further comprising: obtaining target data output by an upper node based on an input quantization type set by a current Add node in a residual network, so as to take the target data as input data of the current Add node; determining a target Add operator based on the input quantization type of the current Add node, and processing the input data by using the target Add operator to obtain an output result.

4. The residual network memory optimization method of claim 3, wherein, the determining of the target Add operator based on the input quantization type of the current Add node and the processing of the input data by using the target Add operator to obtain the output result comprise: if the input quantization type of the current Add node is an INT32 type, a corresponding target Add operator is a first Add operator with an INT32 type data addition function, and the input data is processed by using the first Add operator to obtain an output result of an INT32 type; if the input quantization type of the current Add node is an INT7 type, a corresponding target Add operator is a second Add operator with an INT8 type data addition function, and the input data is processed by using the second Add operator to obtain an output result of an INT8 type.

5. The residual network memory optimization method of claim 4, wherein, after the processing of the input data by using the target Add operator to obtain the output result, further comprising: if the output result is of an INT32 type, the output result is quantized according to an IN8 type, and the quantized output result is taken as input of a lower node. If the output result is of INT8 type, the output result is directly input as a lower node.

6. The residual network memory optimization method of any one of claims 1 to 5, wherein, The step of inferring the deep learning model on the server by using the deep learning inference framework to determine a target Add node causing the memory bottleneck from each Add node includes: The step of inferring the deep learning model on the server by using the deep learning inference framework to determine an Add node with high memory occupation and corresponding memory occupation information from each Add node includes: The step of comparing the memory occupation information with memory information of the embedded device to determine the target Add node causing the memory bottleneck from the Add node with high memory occupation includes:

7. A residual network memory optimization apparatus, comprising: The application applied to an embedded device includes: A framework building module is configured to acquire a trained deep learning model, set an input quantization type of each Add node of a residual network in the deep learning model to be an INT32 type, and build a deep learning inference framework based on the embedded device; A model inference module is configured to infer the deep learning model on a server by using the deep learning inference framework to determine a target Add node causing a memory bottleneck from each Add node; A framework optimization module is configured to set an input quantization type of the target Add node to be an INT7 type, update a corresponding quantization factor in the deep learning inference framework based on the INT7 type, and add a first Add operator with an INT32 type data addition function and a second Add operator with an INT8 type data addition function in the deep learning inference framework to obtain an optimized deep learning inference framework. The framework optimization module is specifically configured to update a quantization factor corresponding to an input of the target Add node in the deep learning inference framework based on the INT7 type, and update an inverse quantization factor corresponding to an output of an upper node of the target Add node.

8. An electronic device, comprising: It includes: A memory is configured to save a computer program; A processor is configured to execute the computer program to implement steps of the residual network memory optimization method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored in the memory, and when executed by the processor, the computer program implements steps of the residual network memory optimization method according to any one of claims 1 to 6.

Images