A network model inference method, apparatus, device, and medium

Abstract

The application discloses a network model inference method, system, device and medium, relates to the technical field of model processing, and comprises the following steps: acquiring a plurality of subgraphs formed by operator division corresponding to each network layer in a target network model; the subgraph type comprises a first type of subgraph supporting running on a neural network processor in a front-end device and a second type of subgraph not supporting running on the neural network processor; the first type of subgraph is added to a task queue of the neural network processor, and the second type of subgraph is added to a task queue of a non-neural network processor; subgraph processing is performed in the task queue according to the queue order, and subgraph processing is performed between the task queues based on the order of the plurality of subgraphs, until the last subgraph in all the task queues is processed, and the inference of the target network model is completed. The application completes the inference of the model by constructing different task queues and processing various subgraphs in a specified order, thereby improving the orderliness and inference efficiency.

Description

Technical Field

[0001] This invention relates to the field of model processing technology, and in particular to a network model inference method, system, device and medium. Background Technology

[0002] With the rise of deep learning technology, neural network models have been applied to various aspects of daily life, such as face detection, face comparison, speech recognition, and fingerprint recognition. After constructing each network layer of the neural network model, each network layer needs to be converted into a corresponding operator, and the hardware inference engine processes the operators corresponding to each network layer to complete the inference process of the deep neural network model.

[0003] Currently, an increasing number of front-end devices can directly execute the inference process of complex models, thereby saving time in data transmission from cloud devices to user devices. However, due to differences in the underlying implementation architecture and operator libraries of different hardware inference engines, the process of front-end devices executing network model inference is often chaotic and inefficient.

[0004] In summary, ensuring the orderliness of operator inference in front-end devices and improving the inference efficiency of network models are problems that need to be solved. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a network model reasoning method, system, device and medium that can ensure the orderliness of operator reasoning and improve the reasoning efficiency of network models.

[0006] Firstly, this application discloses a network model inference method, including:

[0007] Obtain several subgraphs formed by partitioning the network layers in the target network model using operators; wherein, the types of the subgraphs include a first type of subgraph that supports operation on a neural network processor in a front-end device and a second type of subgraph that does not support operation on the neural network processor;

[0008] The first type of subgraph is added to the task queue of the neural network processor in the front-end device, and the second type of subgraph is added to the task queue of the non-neural network processor in the front-end device;

[0009] Subgraph processing is performed within the task queue in the order of the queue, and subgraph processing is performed between the task queues based on the order of several subgraphs, until the last subgraph in all task queues is processed, at which point the inference of the target network model is completed.

[0010] Optionally, adding the first type of subgraph to the task queue of the neural network processor in the front-end device and adding the second type of subgraph to the task queue of the non-neural network processor in the front-end device includes:

[0011] The order of several subgraphs is determined based on the model structure of the target network model;

[0012] In accordance with the order of the subgraphs, the first type of subgraph is added to the task queue of the neural network processor, and the second type of subgraph is added to the task queue of the non-neural network processor.

[0013] The subgraph processing based on the order of several subgraphs within the task queue includes:

[0014] If it is determined that the second target subgraph after the first target subgraph has completed subgraph processing is of a different type from the first target subgraph, a notification event is generated to interact with the task queue where the second target subgraph is located, so as to start the subgraph processing of the second target subgraph. The second target subgraph is the next target subgraph after the first target subgraph in the sequence of several subgraphs.

[0015] Optionally, the target network model includes multiple network models;

[0016] When multiple network models perform subgraph processing, they can reuse working memory.

[0017] Optionally, the subgraph processing within the task queue according to queue order includes:

[0018] The processing of subgraphs in the task queue of non-neural network processors is performed using cache, while the processing of subgraphs in the task queue of neural network processors is performed using DDR memory.

[0019] Optionally, when forming the third target subgraph, the operators involved in each network layer include target operators in floating-point form, and the third target subgraph is the second type of subgraph;

[0020] The process of forming the third target subgraph includes:

[0021] Based on the order of the operators, an inverse quantization operator is added before the target operator, and a quantization operator is added after the target operator;

[0022] The inverse quantization operator, the target operator, and the quantization operator are divided to form the third target subgraph.

[0023] Optionally, obtaining the several sub-graphs formed by the operator partitioning corresponding to each network layer in the target network model includes:

[0024] Obtain the model file corresponding to the target network model;

[0025] Obtain several sub-graphs formed by dividing each network layer into its corresponding operators from the model file;

[0026] The model file also includes dynamic library files and model configuration parameters;

[0027] After obtaining the subgraphs formed by the operator partitioning corresponding to each network layer from the model file, the process further includes:

[0028] Load the dynamic library file to obtain indication information of the exported function corresponding to the second type of subgraph in the dynamic library file. The indication information is used to indicate the position of the exported function corresponding to the second type of subgraph.

[0029] The subgraph processing within the task queue according to queue order includes:

[0030] In the task queue of the non-neural network processor, according to the order in which each second type of subgraph is arranged in the task queue, the derived function corresponding to each second type of subgraph is found according to the indication information one by one, so as to perform inference of the second type of subgraph based on the derived function corresponding to each second type of subgraph and the model configuration parameters.

[0031] Optionally, the model file may also include target instructions;

[0032] After obtaining the subgraphs formed by the operator partitioning corresponding to each network layer from the model file, the process further includes:

[0033] The target instruction is stored in memory, and the storage location information of the target instruction in memory is obtained;

[0034] The subgraph processing within the task queue according to queue order includes:

[0035] In the task queue of the neural network processor, according to the order in which each first type of subgraph is arranged in the task queue, the target instruction corresponding to each first type of subgraph is obtained from the memory according to the storage location information, so as to perform inference of the first type of subgraph according to the target instruction and the model configuration parameters.

[0036] Secondly, this application discloses a network model inference system, including a server and a front-end device; wherein,

[0037] The server is used to divide the operators corresponding to each network layer in the target network model into several subgraphs. The types of the subgraphs include a first type of subgraph that supports running on the neural network processor in the front-end device and a second type of subgraph that does not support running on the neural network processor.

[0038] The front-end device is configured to obtain several subgraphs from the server, add the first type of subgraphs to the task queue of the neural network processor in the front-end device, and add the second type of subgraphs to the task queue of the non-neural network processor in the front-end device;

[0039] The front-end device is also used to perform subgraph processing in the task queue according to the queue order, and to perform subgraph processing in the task queue based on the order of several subgraphs, until the last subgraph in all task queues is processed, thereby completing the inference of the target network model.

[0040] Thirdly, this application discloses an electronic device, including:

[0041] Memory, used to store computer programs;

[0042] A processor is used to execute the computer program to implement the steps of the aforementioned disclosed network model inference method.

[0043] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed network model inference method.

[0044] As can be seen, this application obtains several subgraphs formed by the operator partitioning of each network layer in the target network model; wherein, the types of the subgraphs include a first type of subgraph that supports operation on the neural network processor in the front-end device and a second type of subgraph that does not support operation on the neural network processor; the first type of subgraph is added to the task queue of the neural network processor in the front-end device, and the second type of subgraph is added to the task queue of the non-neural network processor in the front-end device; subgraph processing is performed in the queue order within the task queue, and subgraph processing is performed between the task queues based on the order of several subgraphs, until the last subgraph in all task queues is processed, thus completing the inference of the target network model.

[0045] Therefore, this application divides the operators corresponding to each network layer in the target network model into several subgraphs based on whether the operators support running on the neural network processor in the front-end device. This subgraph division allows operators that support running on the same type of processor to be grouped into the same subgraph. Correspondingly, the types of subgraphs include a first type of subgraph that supports running on the neural network processor in the front-end device and a second type of subgraph that does not support running on the neural network processor. After obtaining the divided subgraphs, the first type of subgraphs are added to the task queue of the neural network processor in the front-end device, and the second type of subgraphs are added to the task queue of the non-neural network processor in the front-end device. In other words, this application constructs separate task queues for different types of processors and adds different types of subgraphs to their corresponding task queues for subgraph processing, thus enabling the deployment of operators corresponding to each network layer in the network model to different hardware inference engines for processing. Within the task queue, subgraph processing is performed according to the queue order. Between task queues, subgraph processing is performed based on the order of several subgraphs until the last subgraph in all task queues is processed, thus completing the inference of the target network model. In other words, this application performs subgraph processing in a prescribed order both within and between different task queues, thereby ensuring the correctness and orderliness of data flow and control flow between subgraphs, and finally completing the inference process of the network model on the front-end device, improving the inference efficiency of the network model. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 This is a flowchart of a network model inference method disclosed in this application;

[0048] Figure 2 This is a schematic diagram of subgraph partitioning of a CRNN network disclosed in this application;

[0049] Figure 3 This is a specific subgraph partitioning diagram disclosed in this application;

[0050] Figure 4 This is a flowchart of a specific network model inference method disclosed in this application;

[0051] Figure 5 This is a schematic diagram of a model loading process disclosed in this application;

[0052] Figure 6 This is a schematic diagram of the structure of a network model inference system disclosed in this application;

[0053] Figure 7 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0055] During the deployment of network models, an increasing number of front-end devices can directly execute the inference process of complex models, thereby saving time in data transmission from cloud devices to user devices. However, compared to cloud devices and edge devices, front-end devices have limited computing power, and complex operators may exceed the support range of the front-end device's hardware inference engine. In addition, due to differences in the underlying implementation architecture and operator libraries of different hardware inference engines, operators such as convolution, pooling, and fully connected operations can be executed on the NPU, but operators such as LSTM and GRU in recurrent neural networks cannot be executed on the NPU. This results in a messy inference process and low inference efficiency for front-end devices.

[0056] To this end, this application discloses a network model inference method, system, device, and medium, which can deploy the operators corresponding to each network layer in the network model to different hardware inference engines on the front-end device for processing, ensuring the orderliness of operator inference on the front-end device and improving the inference efficiency of the network model.

[0057] See Figure 1 As shown in the embodiments of this application, a network model inference method is disclosed, the method comprising:

[0058] Step S11: Obtain several subgraphs formed by the operator partitioning of each network layer in the target network model; wherein, the types of the subgraphs include a first type of subgraph that supports running on the neural network processor in the front-end device and a second type of subgraph that does not support running on the neural network processor.

[0059] In this embodiment, the front-end device can obtain several subgraphs formed by partitioning the operators corresponding to each network layer in the target network model from a server or a specified address. It should be noted that in this embodiment, the server partitions the operators corresponding to each network layer in the target network model into several subgraphs based on whether the operators support running on the neural network processor in the front-end device. Through this subgraph partitioning, operators that support running on the same type of processor can be grouped into the same or the same type of subgraph. Correspondingly, the types of subgraphs include a first type of subgraph that supports running on the neural network processor in the front-end device and a second type of subgraph that does not support running on the neural network processor.

[0060] It should be noted that operators that are not supported by the neural network processor are typically deployed to the central processing unit (CPU) for processing. Based on the operators supported by the neural network processor in the front-end device, operators not supported by the neural network processor in the network model can be filtered out from all the converted operators. Using these unsupported operators as a dividing point, the target network model is divided into subgraphs. Supported operators are assigned to the neural network processor, while unsupported operators are assigned to the CPU, thus obtaining several subgraphs for neural network processors and CPUs.

[0061] Subgraphs that support operation on neural network processors in front-end devices are classified as the first type of subgraph, while subgraphs that support operation on non-neural network processors in front-end devices, such as central processing units, are classified as the second type of subgraph.

[0062] For example, with Figure 2 Taking the CRNN (Convolutional Recurrent Neural Network, an image-based sequence recognition network) network for Chinese text recognition as an example, assuming that its convolution, pooling, and dense operators support running in a neural network processor with int8 precision, while the Long Short-Term Memory (LSTM) operator, dequantize, and quantize operator do not support running in a neural network processor.

[0063] So Figure 2 Convolution, pooling, and fully connected operations will be classified into the first type of subgraph, while LSTM, dequantization, and quantization will be classified into the second type of subgraph. Specifically, the network model is divided into subgraphs based on operators not supported by the neural network processor, with the four operators after the input node—convolution, pooling, convolution, and pooling—classified into the second type. Figure 1 Dequantization, LSTM, and quantization are divided into sub-processes. Figure 2 Divide the fully connected layer into sub-layers. Figure 3 Among them, sub Figure 2 The input is sub Figure 1 The output, sub Figure 3 The input is sub Figure 2 The output, sub Figure 1 Kazuko Figure 3 Belongs to the first type of subgraph, sub Figure 2 It belongs to the second type of subgraph.

[0064] For example Figure 3 The network model shown assumes that operators A, B, E, F, and G support running on a neural network processor, and will be classified as a first-class subgraph, while operators C and D do not support running on a neural network processor, and will be classified as a second-class subgraph. Operators C and D can be specified to run on the central processing unit.

[0065] Using operators not supported by the neural network processor as the dividing point, after subgraph partitioning, operators A and B are divided into subgraphs. Figure 1 Operators C and D are divided into sub-operators. Figure 2 Operators E, F, and G are divided into sub-operators. Figure 3 Among them, sub Figure 2 The input is sub Figure 1 The output, sub Figure 3 The input is sub Figure 1 Kazuko Figure 2 The output, sub Figure 1 Kazuko Figure 3 Belongs to the first type of subgraph, sub Figure 2 It belongs to the second type of subgraph.

[0066] Step S12: Add the first type of subgraph to the task queue of the neural network processor in the front-end device, and add the second type of subgraph to the task queue of the non-neural network processor in the front-end device.

[0067] In this embodiment, after obtaining the subgraphs partitioned by the server, the front-end device adds the first type of subgraphs to the task queue of the neural network processor in the front-end device, and adds the second type of subgraphs to the task queue of the non-neural network processor in the front-end device. That is, this embodiment constructs separate task queues for different types of processors, wherein the number of task queues is related to the type of processor, i.e., one type of processor can correspond to at least one task queue; then, different types of subgraphs are added to their corresponding task queues for subgraph processing, thereby enabling the deployment of operators corresponding to each network layer in the network model to different hardware inference engines for processing.

[0068] The above scheme is based on subgraph partitioning, which divides several operators of the same processor type into the same subgraph and then submits them to the specified processor and its corresponding task queue. This avoids the submission of tasks in the form of single operators at the user level and completes the transformation of complex models into executable models for front-end devices.

[0069] Step S13: Subgraph processing is performed within the task queue according to the queue order, and subgraph processing is performed between the task queues based on the order of several subgraphs, until the last subgraph in all task queues is processed, thus completing the inference of the target network model.

[0070] In this embodiment, subgraph processing is performed within the task queue in a first-in-first-out (FIFO) order. Between task queues, subgraph processing is performed based on the order of several subgraphs until the last subgraph in all task queues is processed, at which point the inference of the target network model is completed. That is, this application performs subgraph processing in a prescribed order both within and between different task queues, thereby ensuring the correctness and orderliness of data flow and control flow between subgraphs, ultimately completing the inference process of the network model on the front-end device and improving the inference efficiency of the network model.

[0071] In a specific implementation, the above-mentioned acquisition of several subgraphs formed by the operator partitioning corresponding to each network layer in the target network model includes: acquiring the model file corresponding to the target network model; and acquiring several subgraphs formed by the operator partitioning corresponding to each network layer from the model file.

[0072] Understandably, after completing the subgraph partitioning, the server packages the subgraph information to generate a model file corresponding to the target network model and stores the model file locally on the server or at a specified address. The subgraph information specifically includes the number of subgraphs, subgraph type, number of operators, etc. Therefore, the front-end device obtains the model file from the server or the specified address, and then extracts several subgraphs formed by partitioning each network layer using the corresponding operators from the model file.

[0073] In one specific embodiment, the model file further includes a dynamic library file and model configuration parameters; after obtaining the several subgraphs formed by the operator partitioning corresponding to each network layer from the model file, the method further includes:

[0074] Load the dynamic library file to obtain indication information of the exported function corresponding to the second type of subgraph in the dynamic library file. The indication information is used to indicate the position of the exported function corresponding to the second type of subgraph.

[0075] The subgraph processing within the task queue according to the queue order includes: in the task queue of the non-neural network processor, according to the arrangement order of each second type of subgraph in the task queue, searching for the derived function corresponding to each second type of subgraph according to the indication information, so as to perform inference of the second type of subgraph according to the derived function corresponding to each second type of subgraph and the model configuration parameters.

[0076] In another specific embodiment, the model file further includes target instructions and model configuration parameters; after obtaining several subgraphs formed by the operator partitioning corresponding to each network layer from the model file, the method further includes: storing the target instructions in memory, and obtaining the storage location information of the target instructions in the memory.

[0077] The subgraph processing within the task queue according to the queue order includes: in the task queue of the neural network processor, according to the arrangement order of each first type of subgraph in the task queue, retrieving the target instruction corresponding to each first type of subgraph from the memory based on the storage location information, so as to perform inference of the first type of subgraph based on the target instruction and the model configuration parameters.

[0078] Understandably, when packaging model files, the server executes each operator node of the first type of subgraph running on a neural network processor, generates target instructions, and packages them into the model file. For the second type of subgraph running on a non-neural network processor, such as a central processing unit (CPU), the functions corresponding to each operator in the second type of subgraph are packaged into a dynamic library file, which contains all the CPU's exported functions. Furthermore, model configuration parameters and subgraph information also need to be packaged into the model file. Specifically, the model configuration parameters include weight parameters and bias parameters.

[0079] The front-end device receives the model file, first loads the packaged model file, verifies its header information, and after successful verification, obtains the inputs, outputs, model configuration parameters, and working space memory required for the entire network model to allocate memory space. Inputs and outputs refer to the input and output space of the entire network model, excluding the input and output space of operators within memory. Model configuration parameters are constant values ​​such as weights and biases during model operation. The working space refers to the temporary space allocated for the inputs and outputs of each operator during model operation.

[0080] Therefore, it can be understood that in this embodiment, the memory space is divided into four parts: input, output, constant values, and working space. Further, after obtaining several subgraphs formed by the operator partitioning corresponding to each network layer from the model file, for the first type of subgraph, the name of the first type of subgraph is obtained from the model file, the target instruction is stored in memory, and the storage location information of the target instruction in memory is obtained. Specifically, the storage location information includes the start address and length information of the instruction.

[0081] For the second type of subgraph, the dynamic library file in the model file is loaded, and the exported functions are parsed according to the operator names to obtain the indication information of the exported functions corresponding to the second type of subgraph in the dynamic library file. The indication information is mainly used to indicate the location of the exported functions corresponding to the second type of subgraph, such as function pointers. In addition, the model configuration parameters obtained from the model file are also stored in the memory corresponding to the parameters for reading and use during inference.

[0082] Furthermore, when processing subgraphs according to the queue order within the task queue, there are two scenarios: In the task queue of the neural network processor, the target instructions corresponding to each first-class subgraph are retrieved from memory based on the storage location information, according to the order in which each first-class subgraph is arranged in the task queue, so as to perform inference on the first-class subgraphs based on the target instructions and model configuration parameters; In the task queue of the neural network processor, the target instructions corresponding to each first-class subgraph are retrieved from memory based on the storage location information, according to the order in which each first-class subgraph is arranged in the task queue, so as to perform inference on the first-class subgraphs based on the target instructions and model configuration parameters.

[0083] Furthermore, the target network model mentioned above includes multiple network models, and these multiple network models reuse working memory when performing subgraph processing.

[0084] This means that the working space of multiple network models can be shared when performing subgraph processing. For example, the multiple network models may include a first target network model and a second target network model; the working memory called by the first target network model when performing subgraph processing is the same as the working memory called by the second target network model when performing subgraph processing.

[0085] It is understood that the target network model includes multiple network models, that is, the embodiments of this application support the simultaneous loading of multiple models, and multiple models share the working area memory space. In other words, when multiple models are loaded simultaneously in this application, since the working area space is the temporary space for the input and output of each operator during the model operation, multiple network models can reuse the memory of the same working area, thereby saving the memory space for multiple model operation.

[0086] The working space memory size is the maximum value among the memory spaces allocated by multiple target network models to store the input and output values ​​of operators. For example, when loading ResNet18 and YOLOv5n simultaneously, ResNet18 requires 0x93000 units of working space memory, while YOLOv5n requires 0x151800 units. However, by reusing working space memory, only 0x15800 units of memory are needed to allow ResNet18 and YOLOv5n to run simultaneously, thus saving 0x93000 units of memory.

[0087] Therefore, this application divides the operators corresponding to each network layer in the target network model into several subgraphs based on whether the operators support running on the neural network processor in the front-end device. This subgraph division allows operators that support running on the same type of processor to be grouped into the same subgraph. Correspondingly, the types of subgraphs include a first type of subgraph that supports running on the neural network processor in the front-end device and a second type of subgraph that does not support running on the neural network processor. After obtaining the divided subgraphs, the first type of subgraphs are added to the task queue of the neural network processor in the front-end device, and the second type of subgraphs are added to the task queue of the non-neural network processor in the front-end device. In other words, this application constructs separate task queues for different types of processors and adds different types of subgraphs to their corresponding task queues for subgraph processing, thus enabling the deployment of operators corresponding to each network layer in the network model to different hardware inference engines for processing.

[0088] Within the task queue, subgraph processing is performed according to the queue order. Between task queues, subgraph processing is performed based on the order of several subgraphs until the last subgraph in all task queues is processed, thus completing the inference of the target network model. In other words, this application performs subgraph processing in a prescribed order both within the task queue and between different task queues, thereby ensuring the correctness and orderliness of data flow and control flow between subgraphs, and finally completing the inference process of the network model on the front-end device, improving the inference efficiency of the network model.

[0089] See Figure 4 As shown, this application discloses a specific network model inference method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes:

[0090] Step S21: Obtain several subgraphs formed by the operator partitioning of each network layer in the target network model; wherein, the types of the subgraphs include a first type of subgraph that supports running on the neural network processor in the front-end device and a second type of subgraph that does not support running on the neural network processor.

[0091] Step S22: Determine the order of several subgraphs based on the model structure of the target network model.

[0092] In this embodiment, the order of several subgraphs can be determined based on the model structure of the target network model, for example... Figure 2 The network model shown can be used to determine the sub-models based on its structure. Figure 2 The input is sub Figure 1 The output, sub Figure 3 The input is sub Figure 2 If the output is given, then the subgraph deployment order is: Figure 1 ,son Figure 2 ,son Figure 3 ;according to Figure 3 The network model shown can be used to determine the sub-models based on its structure. Figure 2 The input is sub Figure 1 The output, sub Figure 3 The input is sub Figure 1 Kazuko Figure 2 If the output is , then the subgraph deployment order is also . Figure 1 ,son Figure 2 ,son Figure 3 In other words, this application determines the input and output information between each operator based on the model structure, and further determines the deployment order between subgraphs.

[0093] Step S23: According to the order of the subgraphs, add the first type of subgraph to the task queue of the neural network processor and add the second type of subgraph to the task queue of the non-neural network processor.

[0094] In this embodiment, the model operation supports synchronous and asynchronous operation. According to the order of several subgraphs, the first type of subgraph is added to the task queue of the neural network processor, and the second type of subgraph is added to the task queue of the non-neural network processor.

[0095] Step S24: Subgraph processing is performed within the task queue according to the queue order, and subgraph processing is performed between the task queues based on the order of several subgraphs, until the last subgraph in all task queues is processed, thus completing the inference of the target network model.

[0096] In a specific implementation, the above-mentioned subgraph processing based on the order of several subgraphs in the task queue includes: if it is determined that the second target subgraph after the first target subgraph that has completed subgraph processing is of a different type from the first target subgraph, a notification event is generated to interact with the task queue where the second target subgraph is located, so as to start the subgraph processing of the second target subgraph, wherein the second target subgraph is the next target subgraph after the first target subgraph in the order of several subgraphs.

[0097] Understandably, both within and between task queues, it's stipulated that the next subgraph can only be processed after the current subgraph has been processed. Within a task queue, subgraphs are processed sequentially according to the first-in-first-out (FIFO) rule. However, when processing subgraphs between task queues, assuming the second target subgraph is the next target subgraph after the first target subgraph in a given sequence, if it's determined that the second target subgraph after the currently processed first target subgraph is of a different type than the first target subgraph, a notification event needs to be generated to interact with the task queue containing the second target subgraph in order to initiate its processing.

[0098] As stated above, both within and between task queues, it is stipulated that the next subgraph can only be processed after the current subgraph has been processed. Therefore, for a first target subgraph and a second target subgraph loaded into different task queues, the output of the first target subgraph after operator processing will serve as the input of the second target subgraph. Furthermore, the interaction of notification events between task queues allows the task queue containing the second target subgraph to receive its input and initiate subgraph processing. This approach enables smooth and efficient deployment of the network model, improving model inference efficiency.

[0099] In this way, by using the event-based notification interaction between tasks in the task queue, this application ensures the orderly execution of tasks across multiple devices and also ensures the correctness of data flow and control flow between subgraphs, thereby completing the inference process of complex network models on the front-end device and improving the inference efficiency of the network model.

[0100] Specifically, the subgraph processing within the task queue according to the queue order includes: using a cache to process subgraphs in the task queue of non-neural network processors, and using DDR memory to process subgraphs in the task queue of neural network processors.

[0101] like Figure 5 As shown, the central processing unit's inference occurs in the cache, while the neural network processor's inference occurs in DDR (Double Data Rate) memory. Figure 2 Taking the CRNN network in the example, due to the sub Figure 2 The dequantization, LSTM, and quantization operators in the process need to be executed on the central processing unit (CPU). Therefore, a work area memory with cache (high-speed cache memory) can be allocated to accelerate the CPU's processing speed. Furthermore, in the sub-... Figure 2A `flush_cache` operator is added after the last operator to flush the inference results of the central processing unit to the DDR memory, ensuring that the subprocessor... Figure 3 Obtaining correct input information is essential to ensure the accuracy of the network model's inference results.

[0102] In a specific embodiment, the second type of subgraph may include a third target subgraph, or in other words, the third target subgraph is the second type of subgraph. When forming the third target subgraph, the operators involved in each network layer include target operators in floating-point form. The process of forming the third target subgraph includes: according to the order of the operators, adding an inverse quantization operator before the target operator and adding a quantization operator after the target operator; dividing the inverse quantization operator, the target operator, and the quantization operator to form the third target subgraph.

[0103] It is understood that this application provides a specific example of the partitioning of the third target subgraph. The third target subgraph is a second type of subgraph that supports operation on the central processing unit. When the server forms the third target subgraph, the operators involved in each network layer include target operators in floating-point form. By adding an inverse quantization operator before the target operator and a quantization operator after the target operator according to the order of the operators, the inverse quantization operator, the target operator, and the quantization operator are then partitioned to form the third target subgraph.

[0104] Still with Figure 2 Taking the CRNN network as an example, since neural network processors cannot support the fixed-point (int8) operations of LSTM, the LSTM operators need to be assigned to the central processing unit in floating-point (float32) form. Therefore, dequantization and quantization operators need to be inserted before and after the LSTM operators to achieve mutual conversion between int8 and float32 types. These two operators also need to run on the CPU, thus dividing the process into sub-operations. Figure 2 .

[0105] For more detailed processing of steps S21 and S24, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0106] As can be seen, this application's embodiments, based on the concept of heterogeneity, support not only operators running on neural network processors but also operators running on central processing units. By constructing subgraphs, the correct flow of data and control is ensured, ultimately guaranteeing the correctness of the inference results. Furthermore, different task queues need to be constructed for different processor devices. During the process of adding subgraphs to task queues, the order of several subgraphs determined by the model structure of the target network model is used. First-type subgraphs are added to the task queue of the neural network processor, and second-type subgraphs are added to the task queue of the non-neural network processor. In addition, both within and between task queues, it is stipulated that the next subgraph can only be processed after the current subgraph has been processed. If two consecutively executed subgraphs are of different types, a notification event is generated to the task queue of the next subgraph after the previous subgraph is processed, to initiate the corresponding subgraph processing flow. This ensures the orderly execution of tasks across multiple devices and also guarantees the correctness of data and control flow between subgraphs.

[0107] See Figure 6 As shown in the figure, this application discloses a network model inference system, including a server 1 and a front-end device 2; wherein,

[0108] The server 1 is used to divide the operators corresponding to each network layer in the target network model into several subgraphs. The types of the subgraphs include a first type of subgraph that supports running on the neural network processor in the front-end device and a second type of subgraph that does not support running on the neural network processor.

[0109] The front-end device 2 is used to obtain several subgraphs from the server, add the first type of subgraphs to the task queue of the neural network processor in the front-end device, and add the second type of subgraphs to the task queue of the non-neural network processor in the front-end device;

[0110] The front-end device 2 is also used to perform subgraph processing in the task queue according to the queue order, and to perform subgraph processing in the task queue based on the order of several subgraphs, until the last subgraph in all task queues is processed, thereby completing the inference of the target network model.

[0111] Therefore, in this application, the server divides the operators corresponding to each network layer in the target network model into several subgraphs based on whether the operators support running on the neural network processor in the front-end device. This subgraph division allows operators that support running on the same type of processor to be grouped into the same subgraph. Correspondingly, the types of subgraphs include a first type of subgraph that supports running on the neural network processor in the front-end device and a second type of subgraph that does not support running on the neural network processor. After obtaining the divided subgraphs from the server, the front-end device adds the first type of subgraphs to the task queue of the neural network processor in the front-end device and adds the second type of subgraphs to the task queue of the non-neural network processor in the front-end device. In other words, this application constructs separate task queues for different types of processors and adds different types of subgraphs to their corresponding task queues for subgraph processing, thus deploying the operators corresponding to each network layer in the network model to different hardware inference engines for processing. In this process, the front-end device processes subgraphs in the order of the task queue within the task queue, and processes subgraphs in the order of several subgraphs between task queues, until the last subgraph in all task queues is processed, thus completing the inference of the target network model. That is, this application processes subgraphs in the prescribed order both within the task queue and between different task queues, thereby ensuring the correctness and orderliness of data flow and control flow between subgraphs, and finally completing the inference process of the network model on the front-end device, improving the inference efficiency of the network model.

[0112] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the network model inference method performed by the electronic device disclosed in any of the foregoing embodiments.

[0113] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0114] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0115] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0116] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the network model inference method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0117] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program, which, when loaded and executed by a processor, implements the network model inference method steps disclosed in any of the foregoing embodiments.

[0118] This application also discloses a computer program product, which, when executed, implements the network model inference method steps disclosed in any of the foregoing embodiments.

[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems, media, and program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant details can be found in the method section.

[0120] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0122] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0123] The foregoing has provided a detailed description of the network model inference method, system, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A network model inference method, characterized in that, include: Obtain several subgraphs formed by partitioning the network layers in the target network model using operators; wherein, the types of the subgraphs include a first type of subgraph that supports operation on a neural network processor in a front-end device and a second type of subgraph that does not support operation on the neural network processor; The first type of subgraph is added to the task queue of the neural network processor in the front-end device, and the second type of subgraph is added to the task queue of the non-neural network processor in the front-end device; Subgraph processing is performed within the task queue in the order of the queue, and subgraph processing is performed between the task queues based on the order of several subgraphs, until the last subgraph in all task queues is processed, at which point the inference of the target network model is completed. The subgraph processing within the task queue according to queue order includes: The processing of subgraphs in the task queue of non-neural network processors is performed using cache, while the processing of subgraphs in the task queue of neural network processors is performed using DDR memory. When forming the third target subgraph, the operators involved in each network layer include target operators in floating-point form, and the third target subgraph is the second type of subgraph; The process of forming the third target subgraph includes: Based on the order of the operators, an inverse quantization operator is added before the target operator, and a quantization operator is added after the target operator; The inverse quantization operator, the target operator, and the quantization operator are divided to form the third target sub-graph; The model files corresponding to the target network model include dynamic library files and model configuration parameters; After obtaining the several subgraphs formed by the operator partitioning corresponding to each network layer in the target network model, the process further includes: Load the dynamic library file to obtain indication information of the exported function corresponding to the second type of subgraph in the dynamic library file. The indication information is used to indicate the position of the exported function corresponding to the second type of subgraph. The subgraph processing within the task queue according to queue order includes: In the task queue of the non-neural network processor, according to the order in which each second type of subgraph is arranged in the task queue, the derived function corresponding to each second type of subgraph is found according to the indication information one by one, so as to perform inference of the second type of subgraph based on the derived function corresponding to each second type of subgraph and the model configuration parameters.

2. The network model inference method according to claim 1, characterized in that, The step of adding the first type of subgraph to the task queue of the neural network processor in the front-end device and adding the second type of subgraph to the task queue of the non-neural network processor in the front-end device includes: The order of several subgraphs is determined based on the model structure of the target network model; In accordance with the order of the subgraphs, the first type of subgraph is added to the task queue of the neural network processor, and the second type of subgraph is added to the task queue of the non-neural network processor. The subgraph processing based on the order of several subgraphs within the task queue includes: If it is determined that the second target subgraph after the first target subgraph has completed subgraph processing is of a different type from the first target subgraph, a notification event is generated to interact with the task queue where the second target subgraph is located, so as to start the subgraph processing of the second target subgraph. The second target subgraph is the next target subgraph after the first target subgraph in the sequence of several subgraphs.

3. The network model inference method according to claim 1, characterized in that, The target network model includes multiple network models; When multiple network models perform subgraph processing, they can reuse working memory.

4. The network model inference method according to claim 1, characterized in that, The model file also includes target instructions; After obtaining several subgraphs formed by the operator partitioning corresponding to each network layer from the model file, the process also includes: The target instruction is stored in memory, and the storage location information of the target instruction in memory is obtained; The subgraph processing within the task queue according to queue order includes: In the task queue of the neural network processor, according to the order in which each first type of subgraph is arranged in the task queue, the target instruction corresponding to each first type of subgraph is obtained from the memory according to the storage location information, so as to perform inference of the first type of subgraph according to the target instruction and the model configuration parameters.

5. The network model inference method according to any one of claims 1 to 4, characterized in that, The process of obtaining several sub-graphs formed by the operator partitioning corresponding to each network layer in the target network model includes: Obtain the model file corresponding to the target network model; Obtain several subgraphs formed by dividing the network layers into their corresponding operators from the model file.

6. A network model inference system, characterized in that, This includes servers and front-end devices; among which, The server is used to divide the operators corresponding to each network layer in the target network model into several subgraphs. The types of the subgraphs include a first type of subgraph that supports running on the neural network processor in the front-end device and a second type of subgraph that does not support running on the neural network processor. The front-end device is configured to obtain several subgraphs from the server, add the first type of subgraphs to the task queue of the neural network processor in the front-end device, and add the second type of subgraphs to the task queue of the non-neural network processor in the front-end device; The front-end device is also used to perform subgraph processing in the task queue according to the queue order, and to perform subgraph processing based on the order of several subgraphs between the task queues, until the last subgraph in all task queues is processed, thereby completing the inference of the target network model. The front-end device is used to process subgraphs in the task queue of a non-neural network processor using a cache, and to process subgraphs in the task queue of a neural network processor using a DDR memory. When forming the third target subgraph, the operators involved in each network layer include target operators in floating-point form, and the third target subgraph is the second type of subgraph; The process by which the server forms the third target subgraph includes: Based on the order of the operators, an inverse quantization operator is added before the target operator, and a quantization operator is added after the target operator; the inverse quantization operator, the target operator, and the quantization operator are divided to form the third target subgraph; The model files corresponding to the target network model include dynamic library files and model configuration parameters; The front-end device is also used to load the dynamic library file to obtain indication information of the exported function corresponding to the second type of subgraph in the dynamic library file. The indication information is used to indicate the position of the exported function corresponding to the second type of subgraph. The front-end device is used to search for the derived function corresponding to each second-type subgraph in the task queue of the non-neural network processor according to the order of each second-type subgraph in the task queue, based on the indication information, so as to perform inference of the second-type subgraph based on the derived function corresponding to each second-type subgraph and the model configuration parameters.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the network model inference method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the network model inference method as described in any one of claims 1 to 5.

Images