Neural network model inference method and apparatus, computer device, and storage medium

Abstract

A neural network model inference method includes: obtaining a neural network model inference task, the neural network model inference task including a model identifier (202); obtaining a neural network model corresponding to the model identifier, parsing the neural network model to obtain a computation graph corresponding to the neural network model, the computation graph including a connection layer (204); obtaining a pre-constructed data structure template, and generating a target sub-data structure corresponding to the computation graph according to the data structure template (206); determining to-be-optimized connection layer data in the computation graph according to the target sub-data structure and the connection layer (208); performing optimization processing on the to-be-optimized connection layer data according to the target sub-data structure to obtain an optimized neural network model (210); and performing inference according to the optimized neural network model to obtain a model inference result (212).

Description

Technical Field

[0001] This application relates to a neural network model inference method, apparatus, computer device, and storage medium. Background Technology

[0002] Inference in neural network models refers to deploying a pre-trained neural network model into real-world business scenarios, such as image classification, object detection, and online translation. This involves inputting data into the neural network model and obtaining output data through it. As the network structure of neural networks becomes more complex, inference takes significantly more time. To improve the inference speed, a traditional approach is to fuse connection layers. This involves directly writing the input tensors of a connection layer into its output tensor, and then deleting the input tensor and the connection layer itself. This reduces memory usage and memory copying time.

[0003] However, the inventors realized that traditional methods are only suitable for simple connection layer structures. As the network structure of neural network models becomes more complex, the connection operations of the connection layers also become more complex. If the traditional method is used for connection layer fusion, it will lead to errors in the inference results. Therefore, how to improve the inference speed while obtaining correct inference results has become a technical problem that needs to be solved. Summary of the Invention

[0004] According to various embodiments disclosed in this application, a neural network model inference method, apparatus, computer device, and storage medium are provided.

[0005] A neural network model inference method, comprising:

[0006] Obtain a neural network model inference task, wherein the neural network model inference task includes a model identifier;

[0007] Obtain the neural network model corresponding to the model identifier, parse the neural network model, and obtain the computation graph corresponding to the neural network model, wherein the computation graph includes connection layers;

[0008] Obtain a pre-built data structure template, and generate the target sub-data structure corresponding to the computation graph based on the data structure template;

[0009] Based on the target sub-data structure and the connection layer, the connection layer data to be optimized is determined in the computation graph;

[0010] The data of the connection layer to be optimized is optimized based on the target substructure data to obtain the optimized neural network model; and

[0011] The optimized neural network model is used to perform inference, and the model inference result is obtained.

[0012] A neural network model inference device, comprising:

[0013] The task acquisition module is used to acquire neural network model inference tasks, wherein the neural network model inference tasks include model identifiers.

[0014] The model parsing module is used to obtain the neural network model corresponding to the model identifier, parse the neural network model, and obtain the computation graph corresponding to the neural network model, wherein the computation graph includes connection layers;

[0015] The structure generation module is used to obtain a pre-built data structure template and generate the target sub-data structure corresponding to the computation graph based on the data structure template.

[0016] The data determination module is used to determine the connection layer data to be optimized in the computation graph based on the target sub-data structure and the connection layer.

[0017] The model optimization module is used to optimize the connection layer data to be optimized based on the target substructure data, thereby obtaining an optimized neural network model; and

[0018] The model inference module is used to perform inference based on the optimized neural network model and obtain the model inference result.

[0019] A computer device includes a memory and one or more processors, the memory storing computer-readable instructions that, when executed by the processors, cause the one or more processors to perform the following steps:

[0020] Obtain a neural network model inference task, wherein the neural network model inference task includes a model identifier;

[0021] Obtain the neural network model corresponding to the model identifier, parse the neural network model, and obtain the computation graph corresponding to the neural network model, wherein the computation graph includes connection layers;

[0022] Obtain a pre-built data structure template, and generate the target sub-data structure corresponding to the computation graph based on the data structure template;

[0023] Based on the target sub-data structure and the connection layer, the connection layer data to be optimized is determined in the computation graph;

[0024] The data of the connection layer to be optimized is optimized based on the target substructure data to obtain the optimized neural network model; and

[0025] The optimized neural network model is used to perform inference, and the model inference result is obtained.

[0026] One or more computer storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the following steps:

[0027] Obtain a neural network model inference task, wherein the neural network model inference task includes a model identifier;

[0028] Obtain the neural network model corresponding to the model identifier, parse the neural network model, and obtain the computation graph corresponding to the neural network model, wherein the computation graph includes connection layers;

[0029] Obtain a pre-built data structure template, and generate the target sub-data structure corresponding to the computation graph based on the data structure template;

[0030] Based on the target sub-data structure and the connection layer, the connection layer data to be optimized is determined in the computation graph;

[0031] The data of the connection layer to be optimized is optimized based on the target substructure data to obtain the optimized neural network model; and

[0032] The optimized neural network model is used to perform inference, and the model inference result is obtained.

[0033] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0034] Figure 1 This is a diagram illustrating the application environment of the neural network model inference method in one or more embodiments.

[0035] Figure 2 This is a flowchart illustrating the neural network model inference method in one or more embodiments.

[0036] Figure 3 This is a schematic diagram of a computation graph in one or more embodiments.

[0037] Figure 4 For one or more embodiments Figure 3 The diagram shows the computation graph obtained after the computation graph is fused with the connection layer.

[0038] Figure 5 This is a schematic diagram of a computational graph containing complex connection operations in one or more embodiments.

[0039] Figure 6This is a schematic diagram of a data structure template in one or more embodiments.

[0040] Figure 7 This is a flowchart illustrating the steps of generating the target sub-data structure corresponding to the computation graph based on the data structure template in one or more embodiments.

[0041] Figure 8 This is a flowchart illustrating the steps of generating the optimized data structure corresponding to the output tensor of the connection layer based on the data structure template and the computation graph in one or more embodiments.

[0042] Figure 9 This is a block diagram of a neural network model inference device in one or more embodiments.

[0043] Figure 10 This is a block diagram of a computer device in one or more embodiments. Detailed Implementation

[0044] The neural network model inference method provided in this application can be applied to computer devices, which can be terminals or servers. It is understood that the neural network model inference method provided in this application can be applied to terminals, servers, or systems including both terminals and servers, and is implemented through interaction between the terminal and the server.

[0045] The neural network model inference method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 and server 104 communicate via a network. Terminal 102 can obtain a model inference task, which carries a model identifier. Terminal 102 obtains the neural network model corresponding to the model identifier, parses the neural network model to obtain the corresponding computation graph, which includes connection layers, thereby obtaining a pre-constructed data structure template. Based on the data structure template, a target sub-data structure corresponding to the computation graph is generated. Based on the target sub-data structure and the connection layers, the connection layer data to be optimized is determined in the computation graph. The connection layer data to be optimized is optimized based on the target sub-data structure data to obtain an optimized neural network model. Then, inference is performed based on the optimized neural network model to obtain the model inference result. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0046] As is understood, the neural network model inference method provided in this application enables inference on neural network models and can be applied to various application environments. Neural network models can include various types. For example, neural network models can include convolutional neural network models, recurrent neural network models, and recurrent neural network models. Neural network models can be used to process various types of data. For example, specific neural network models can include image recognition models, feature extraction models, speech recognition models, text recognition models, and scene classification models.

[0047] In one embodiment, the inference method of the neural network model provided in this application can be specifically applied to the field of autonomous driving. The neural network model can specifically include at least one of image recognition model or trajectory prediction model.

[0048] In one embodiment, the inference method of the neural network model provided in this application can be specifically applied to the text domain, and the neural network model can specifically include at least one of image recognition model, behavior prediction model or risk assessment model.

[0049] In one embodiment, such as Figure 2 As shown, a neural network model inference method is provided. The method is illustrated using a computer device as an example. This computer device can be... Figure 1 The terminal or server in the process includes the following steps:

[0050] Step 202: Obtain the model inference task, which carries the model identifier.

[0051] Model inference refers to the process of performing calculations on the data input to a neural network model according to the arithmetic operations corresponding to each of the network layers, based on the network structure of the model, to obtain the inference result output by the neural network model. Model inference tasks instruct computer devices to perform inference on the corresponding neural network models. The computer device can be a terminal or a server. A model identifier is a unique identifier used to label and distinguish neural network models.

[0052] When model inference is required, the computer device can acquire a model inference task, parse it, and obtain the model identifier carried within the task. Specifically, when a user needs to perform model inference, the computer device can determine the user-specified neural network model based on received user operation instructions and generate a model inference task carrying the model identifier. Alternatively, the computer device can determine the neural network model to be invoked based on actual operational needs and generate a model inference task. For example, in image recognition, when an image recognition model needs to be invoked to process an image, the computer device can generate a model inference task. Based on this task, it can then infer the image recognition model after inputting the image, obtaining the image recognition result output by the model.

[0053] In one embodiment, the computer device may pre-store an inference engine, which can be used to perform model inference tasks and infer the neural network model corresponding to the model identifier. The inference engine refers to the functional module in the computer device used to perform inference.

[0054] Step 204: Obtain the neural network model corresponding to the model identifier, parse the neural network model, and obtain the computation graph corresponding to the neural network model. The computation graph includes connection layers.

[0055] Computer devices pre-store neural network models, which are trained using a large amount of sample data. Therefore, the corresponding neural network model can be obtained based on the model identifier. The neural network model corresponding to the model identifier can include at least one of several types of neural network models. For example, depending on the network structure, it can specifically include at least one of Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and other recurrent neural network models. Depending on the function of the neural network model, it can specifically include at least one of image recognition models, feature extraction models, speech recognition models, text recognition models, and scene classification models.

[0056] Computer equipment analyzes the acquired neural network model to obtain the corresponding computation graph. The computation graph can be an abstract representation of the neural network model during the inference process, and may include multiple operation layers, tensors corresponding to each operation layer, and directed edges between operation layers and their corresponding tensors. Operation layers represent the network layers in the neural network model's structure and determine the arithmetic operations performed by those layers, such as convolution, fully connected operations, and connection operations. A tensor is a data structure that can be understood as a vector or an array matrix. The shape of a tensor can be represented by its dimensions; a one-dimensional tensor is called a vector, and tensors with two or more dimensions are called array matrices. Tensors include input tensors and output tensors. Input tensors represent the input data corresponding to the operation layer, and output tensors represent the output data of the operation layer. Each operation layer may include multiple pointers to different input tensors and multiple pointers to different output tensors. Each tensor may include one pointer to the production layer and multiple pointers to different demand layers. The production layer refers to which operation layers compute the tensor, and the demand layer refers to which operation layers the tensor serves as the input tensor for. Directed edges between the operation layer and the corresponding tensor in the computation graph can be generated using pointers in the operation layer and the tensor.

[0057] Computer devices can refer to the operation layer corresponding to the connection operation as the connection layer. The connection layer is used to concatenate multiple input tensors. The data stored in the output tensor of the connection layer is exactly the same as the data stored in the multiple input tensors.

[0058] In one embodiment, the schematic diagram of the computation graph can be as follows: Figure 3 As shown, the computation graph includes operation layer 1, operation layer 2, operation layer 3, and the corresponding input and output tensors for each operation layer. The output tensor 1 of operation layer 1 and the output tensor 2 of operation layer 2 are used as inputs to operation layer 3. Operation layer 3 concatenates output tensor 1 and output tensor 2 to obtain output tensor 3. The shapes in output tensor 1, output tensor 2, and output tensor 3 represent the shapes of the tensors, which can be represented by dimensions. For example, the shape of output tensor 1 is 1×3×40×40, representing the 0th, 1st, 2nd, and 3rd dimensions from left to right. The connection dimension in operation layer 3 is 1, indicating that output tensor 1 and output tensor 2 are concatenated along the 1st dimension.

[0059] Step 206: Obtain the pre-built data structure template and generate the target sub-data structure corresponding to the computation graph based on the data structure template.

[0060] A pre-built data structure template refers to the data structure template required for optimizing the computation graph. The data structure template is a new data structure template built upon the traditional data structure of the computation graph. The target sub-data structure refers to the optimized tensor data structure within the computation graph.

[0061] Neural network models include numerous operational layers, which may contain connection layers. Connection layers concatenate multiple input tensors, and the data stored in the output tensor of these connection layers is identical to the data stored in the input tensors. However, during inference in the neural network model, these connection operations result in duplicate data stored in memory and unnecessary memory copying time. As the amount of data increases, these connection operations consume more memory, and the memory copying time also increases, leading to longer inference times and slower inference speeds, which is detrimental to data processing processes with high real-time requirements. For example, in the field of autonomous driving, it is necessary to quickly derive data processing results from neural network model inference. Traditional methods aim to accelerate inference by fusing connection layers and removing duplicate data to reduce memory usage and copying time. However, traditional methods are only suitable for simple connection operations, such as when all input tensors of a connection layer are fed into the connection layer for concatenation. When the connection operations in a neural network model are complex, such as when a single tensor serves as input to multiple operation layers, and some of these operation layers are not involved in connection operations, traditional connection layer fusion methods can lead to incorrect inference results. Furthermore, when consecutive connections exist, traditional connection layer fusion methods cannot determine the location of the tensor that was optimized out in the output tensor of the last connection layer, hindering model inference. Additionally, when the output tensor of one operation layer is simultaneously connected to the output tensors of multiple operation layers (i.e., the output tensor needs to be input to multiple connection layers), traditional connection layer fusion methods cannot simultaneously store the output tensor of that operation layer, resulting in incorrect model inference results. Therefore, computer devices pre-construct and store a new data structure template. When model inference is needed, the pre-constructed data structure template is retrieved, and the data structure of the computation graph is optimized based on the template. This allows for accurate fusion of connection layers in all cases, ensuring correct inference results, while reducing memory usage and memory copying time, thus accelerating inference speed.

[0062] In one embodiment, a conventional method is used to... Figure 3 The schematic diagram of the computation graph obtained after fusing the connection layers shown can be as follows: Figure 4As shown. Since the content stored in output tensor 3, which serves as the connection layer, is completely identical to the content in output tensor 1 and output tensor 2, the connection operation can be called a memory copy operation. By directly writing the output results of operation layer 1 and operation layer 2 into output tensor 3, and deleting output tensor 1, output tensor 2, and operation layer 3, we can reduce the allocation of two memory spaces for output tensor 1 and output tensor 2, and reduce one memory copy operation in operation layer 3.

[0063] In one embodiment, a computational graph diagram containing complex connection operations can be as follows: Figure 5 As shown. Operation layers 1-7 are all operation layers, and operation layers 3, 5, and 7 are connection layers. The dashed arrows pointing to output tensor 1, output tensor 3, and output tensor 5 indicate the demand layers other than the connection layers corresponding to each output tensor.

[0064] In one embodiment, the pre-constructed data structure template can be the output structure template of the output tensor corresponding to the connection layer in the computation graph. A schematic diagram of the data structure template can be shown as follows: Figure 6 As shown, it includes multiple template items: a production layer for storing output tensors, a demand layer for storing output tensors, a data layer for storing the shapes of output tensors, an index table for storing the corresponding output tensors of each production layer, an index table for storing the input tensors of each demand layer, and a table for storing the sizes of the input tensors of each demand layer. Specifically, the production layer for storing output tensors is used to store the output tensors corresponding to the connection layers, including the production layer for storing the input tensors that need to be stored; the demand layer for storing output tensors is used to store the demand layers for storing the output tensors corresponding to the connection layers, including the demand layers for storing the input tensors that need to be stored; the data layer for storing the shapes of output tensors is used to store the shapes of the output tensors corresponding to the connection layers, including the shapes of the input tensors that need to be stored; the index table for storing the corresponding production layers is used to store the indexes corresponding to each production layer; the index table for storing the input tensors of each demand layer is used to store the indexes of the input tensors corresponding to each demand layer; and the table for storing the sizes of the input tensors of each demand layer is used to store the sizes of the input tensors corresponding to each demand layer.

[0065] Because the obtained computation graph data structure contains multiple pointers to different input tensors and multiple pointers to different output tensors in each operational layer, when the connection operations in the neural network model are complex, such as when a certain tensor is used as the input to multiple operational layers, and other operational layers (excluding the connection layer) do not participate in the connection operation, after connection layer optimization, these other operational layers lose their original input tensors and will use the output tensor of that connection layer as the output tensor of other operational layers, thus leading to errors in the output results of other operational layers. Furthermore, when the computation graph is... Figure 5As shown, there are multiple consecutive connection layers. Merging these connection layers using traditional methods not only results in incorrect output results but also fails to determine the positions of output tensors 1, 2, 4, and 6 within the merged output tensor 7. Consequently, the deleted connection layers 3, 5, and 7, as well as the demand layer, cannot determine their corresponding input tensors, leading to errors in the inference results.

[0066] The computer device can optimize the data structure of the computation graph based on the pre-built data structure template. Specifically, the computer device can obtain the original data structure of the output tensor corresponding to each connection layer in the computation graph based on the data structure template. The original data structure may include the output tensor itself, a pointer to the production layer, and multiple pointers to different demand layers. The computer device can optimize the original data structure based on the data structure template, complete the data structure template according to the computation graph and the output tensor corresponding to each connection layer in the computation graph, thereby obtaining the optimized data structure of the output tensor corresponding to each connection layer in the computation graph. The optimized data structure is the optimized data structure, and the target sub-data structure corresponding to the computation graph is determined based on the generated optimized data structure. The target sub-data structure may include the production layer, demand layer, shape, index of each production layer, input tensor index of each demand layer, and size of the input tensor corresponding to each demand layer of the output tensor corresponding to the connection layer. There can be one or more connection layers. The production layer and demand layer of the output tensor corresponding to the connection layer are used to determine the original input tensor of the demand layer that does not participate in the connection operation, ensuring that the correct output result is obtained. The shape of the output tensor is used to ensure the correctness of the connection operation performed by the connection layer. The index of the input tensor corresponding to each demand layer and the size of the input tensor corresponding to each demand layer are used to represent the position of the input tensor in the output tensor. This allows the input tensor corresponding to each demand layer to be found based on its position when there are consecutive connection layers.

[0067] Step 208: Determine the connection layer data to be optimized based on the target sub-data structure and the connection layer in the computation graph.

[0068] Step 210: Optimize the data of the connection layer to be optimized based on the target substructure data to obtain the optimized neural network model.

[0069] The target sub-data structure refers to an optimized data structure that can correctly integrate the connection layer.

[0070] Computer equipment can determine the data of the connection layers to be optimized in the computational graph based on the target sub-data structure and the connection layers in the computational graph. The data of the connection layers to be optimized may include the connection layers in the computational graph and the corresponding input tensors of the connection layers. The computer equipment can optimize the data of the connection layers to be optimized based on the target sub-structure data. The optimization method may be to merge the connection layers, that is, to delete the data of the connection layers to be optimized.

[0071] In one embodiment, the optimization process is performed on the connection layer data to be optimized based on the target substructure data to obtain an optimized neural network model. This includes: deleting the connection layer data to be optimized to obtain a computation graph after deletion; and connecting the deleted computation graph according to the target data structure to obtain the optimized neural network model. After deleting the connection layer data to be optimized, the computer device can sequentially connect the deleted computation graph according to the operation layer order in the target data structure to obtain the optimized computation graph, and then obtain the optimized neural network model based on the optimized computation graph. By deleting the connection layer data to be optimized and connecting the deleted computation graph, the memory space occupied by the inference process can be reduced, the memory copy time can be reduced, and the inference speed can be accelerated.

[0072] Step 212: Perform inference based on the optimized neural network model to obtain the model inference result.

[0073] After obtaining an optimized neural network model, the computer device, compared to a traditional neural network model, reduces the number of connection operations during inference, decreases the memory usage of repetitive data during inference, and reduces memory copy time, thereby improving the model's inference speed. The computer device can perform inference based on the optimized neural network model, sequentially performing arithmetic operations according to the corresponding operation layers to obtain the inferred data result. For example, the computer device can perform operations on the input image according to the order of arithmetic operations corresponding to the optimized neural network model to obtain the recognized image result.

[0074] In this embodiment, a neural network model inference task is obtained, and the neural network model corresponding to the model identifier in the neural network model inference task is obtained. The neural network model is parsed to obtain the corresponding computation graph. The computation graph includes connection layers, thereby obtaining a pre-constructed data structure template. Based on the data structure template, a target sub-data structure corresponding to the computation graph is generated. Based on the target sub-data structure and the connection layers in the computation graph, the connection layer data to be optimized is determined. Then, the connection layer data to be optimized is optimized based on the target sub-data structure data to obtain an optimized neural network model. Inference is performed based on the optimized neural network model to obtain the model inference result. By pre-constructing a data structure template and generating the target sub-data structure corresponding to the computation graph based on the template, the data structure of the computation graph can be optimized. This allows the optimized target sub-data structure to correctly integrate connection layers in all cases, thereby obtaining the correct inference result. At the same time, it reduces the memory space occupied during the inference process, reduces memory copy time, and speeds up the inference process.

[0075] In one embodiment, such as Figure 7 As shown, the steps for generating the target sub-data structure corresponding to the computation graph based on the data structure template include:

[0076] Step 702: Traverse the operation layers in the computation graph and identify the connection layers in the operation layers.

[0077] Step 704: Generate an optimized data structure for the output tensor of the connection layer based on the data structure template and the computation graph.

[0078] Step 706: Determine the target sub-data structure corresponding to the computation graph based on the optimized data structure of the output tensor corresponding to the connection layer.

[0079] The computation graph consists of operation layers and tensors. Operation layers include connection layers, and tensors can be either input or output tensors of the operation layers. The computer device traverses all operation layers in the computation graph, identifying connection layers within them. When a connection layer is identified, it retrieves the template item data for the corresponding output tensor from the computation graph based on multiple template items in the data structure template. Each retrieved template item data is then added to the corresponding template item, thereby generating an optimized data structure for the output tensor of that connection layer. This process continues until the optimized data structure for the output tensor of the last connection layer is generated. The optimized data structure refers to the data structure obtained after optimizing the original data structure of the output tensor.

[0080] Computer equipment can determine the target sub-data structure of the computation graph based on the optimized data structure of the output tensor corresponding to the generated connection layer. When there is only one connection layer in the computation graph, the optimized data structure of the output tensor corresponding to that connection layer is directly used to determine the target sub-data structure of the computation graph. When there are multiple consecutive connection layers in the computation graph, such as... Figure 5 As shown in the computation graph, the computer device can determine the target sub-data structure of the computation graph by optimizing the output tensor corresponding to the last connection layer. When the computation graph includes multiple parallel connection layers, that is, when the output tensor of one operation layer needs to be input to multiple connection layers simultaneously, the computer device can determine the optimized data structures of the output tensors corresponding to multiple connection layers as the target sub-data structure of the computation graph. In this case, the target sub-data structure includes the optimized data structures of the output tensors corresponding to multiple connection layers.

[0081] In this embodiment, the operation layers in the computation graph are traversed to identify the connection layers. It is only necessary to generate the optimized data structure of the output tensor corresponding to the connection layer based on the data structure template and the computation graph. The target sub-data structure corresponding to the computation graph is determined based on the optimized data structure of the output tensor corresponding to the connection layer. This allows for the rapid acquisition of the data structure required to correctly fuse the connection layer in all complex situations.

[0082] In one embodiment, the method further includes: performing topological sorting on the operation layers in the computation graph to obtain a topological sequence; identifying whether each operation layer is a connection layer according to the topological sequence; skipping the operation layer if it is not a connection layer; and generating an optimized data structure for the output tensor of the connection layer according to the data structure template and the computation graph if it is a connection layer.

[0083] A computation graph can include multiple operation layers. A computer device can perform topological sorting on these operation layers to obtain a topological sequence. Topological sorting refers to arranging the operation layers in a directed computation graph into a sequence that satisfies a topological order based on the dependencies between them. The resulting sequence is a one-dimensional linear sequence. Specifically, the computer device first finds the operation layer with an in-degree of 0 (i.e., no input edges) in the computation graph, stores this operation layer in a stack, and removes the operation layer and its associated directed edges from the computation graph. The in-degree of the deleted operation layer is adjusted, for example, by decrementing it by 1. This process of finding operation layers with an in-degree of 0 and deleting / adjusting them is repeated until all operation layers in the computation graph have been saved in the stack. All operation layers in the stack can then be output sequentially according to their storage order, thus obtaining the topological sequence. The order of operation layers in the topological sequence can be determined based on the aforementioned storage order. The computer device can access each operation layer according to its order in the topological sequence, identifying whether the arithmetic operation corresponding to each operation layer is a join operation. Arithmetic operations refer to the data processing operations performed by each operation layer. When the operation is not a join operation, the operation layer is not a join layer, and the computer can skip this operation layer without performing data structure optimization. When the operation is a join operation, the operation layer is a join layer, and data structure optimization is required. The computer generates an optimized data structure for the output tensor of the join layer based on the data structure template and the computation graph.

[0084] In this embodiment, by topologically sorting the operation layers in the computation graph, the computer device can ensure that when reasoning is performed on a certain operation layer, the arithmetic operation corresponding to the previous operation layer has been reasoned out, thereby improving the recognition accuracy of the connection layer and facilitating the rapid acquisition of the optimized data structure of the output tensor corresponding to the connection layer.

[0085] In one embodiment, such as Figure 8 As shown, the steps for generating the optimized data structure for the output tensor of the connection layer based on the data structure template and the computation graph include:

[0086] Step 802: Obtain the current connection layer and identify whether there is an optimized input tensor in the input tensor corresponding to the current connection layer.

[0087] Step 804: If it exists, obtain the optimized data structure of the optimized output tensor, generate the optimized data structure of the output tensor corresponding to the current connection layer based on the optimized data structure of the optimized output tensor, the computation graph and the data structure template, update the next connection layer to the current connection layer, return to the step of identifying whether there is an optimized input tensor in the input tensor corresponding to the current connection layer, until the traversal is completed, and generate the optimized data structure of the output tensor corresponding to the connection layer in the operation layer.

[0088] Step 806: When there is no optimized input tensor in the input tensor corresponding to the current connection layer, extract the template data corresponding to the output tensor of the current connection layer in the computation graph according to the data structure template, and add the extracted template data to the data structure template to obtain the optimized data structure of the output tensor corresponding to the current connection layer.

[0089] The current connection layer refers to the currently accessed connection layer. An optimized input tensor means that the production layer of the input tensor is a connection layer, and the data structure of the input tensor is an optimized data structure generated based on the data structure template and the computation graph.

[0090] The computer device acquires the current connection layer and identifies whether there is an optimized input tensor in the input tensor corresponding to the current connection layer. When an optimized input tensor is identified, it indicates the existence of a continuous connection layer in the computation graph. An optimized data structure for the output tensor corresponding to the current connection layer can be generated based on the optimized data structure of the optimized input tensor. Specifically, since the optimized data structure of the input tensor is an optimized data structure, the computer device can acquire the optimized data structure of the optimized input tensor and combine it with the unoptimized input tensor corresponding to the current connection layer. The computer device, based on the optimized data structure of the optimized input tensor and the computation graph, acquires the template data corresponding to the output tensor of the current connection layer and adds the acquired template data to the data structure template, thereby obtaining the optimized data structure corresponding to the current connection layer. The optimized data structure corresponding to the current connection layer may include the production layer, demand layer, shape, index of each production layer, index of each demand layer, and size of the input tensor corresponding to each demand layer stored in the output tensor of the current connection layer. The computer device can then identify whether the next operation layer is a connection layer. If it is, the next operation layer is used as the next connection layer, and the next connection layer is updated to the current connection layer. It then returns to the step of identifying whether there is an optimized input tensor in the input tensor corresponding to the current connection layer, continuing until optimized data structures for the output tensors corresponding to all connection layers in the operation layer are generated. By combining the optimized data structures of the optimized input tensors with the unoptimized input tensors corresponding to the current connection layer, the generation efficiency of optimized data structures can be improved, thereby accelerating the fusion speed of connection layers and improving the model inference speed.

[0091] In one embodiment, when there are consecutive connection layers in the computation graph, that is, the input tensors corresponding to the second connection layer and subsequent connection layers all contain optimized input tensors, the computer device can generate the optimized data structure of the output tensor corresponding to each connection layer according to the above-described output method of the optimized data structure, and take the optimized data structure of the output tensor corresponding to the last connection layer as the target sub-data structure corresponding to the computation graph.

[0092] When it is detected that there is no optimized input tensor in the input tensor corresponding to the current connection layer, it indicates that the input tensor of the current connection layer is a basic tensor. The computer device can directly extract the template data corresponding to the output tensor of the current connection layer from the computation graph according to the data structure template. Specifically, the data structure template includes multiple template items. The computer device can sequentially extract the template item data corresponding to the output tensor of the current connection layer according to the multiple template items in the data structure template, and add the extracted template item data to the corresponding template item to obtain the optimized data structure of the output tensor of the current connection layer. This allows for the correct fusion of the optimized data structure of the connection layers.

[0093] In one embodiment, extracting template data corresponding to the output tensor of the current connection layer from the computation graph according to the data structure template, and adding the extracted template data to the data structure template to obtain the optimized data structure of the output tensor corresponding to the current connection layer includes: extracting the production layer, demand layer, and dimension data corresponding to each input tensor sequentially from the computation graph according to the data structure template; adding the extracted production layer, demand layer, and dimension data to the data structure template; establishing a production layer index corresponding to the production layer in the production layer index table of the data structure template; establishing a demand layer index corresponding to the demand layer in the demand layer data table of the data structure template, and calculating the size of the input tensor corresponding to each demand layer; and attaching the demand layers other than the connection layers in the demand layers corresponding to the input tensors to the demand layer data table to obtain the optimized data structure of the output tensor corresponding to the current connection layer.

[0094] The template data corresponding to the output tensor of the current connection layer includes the production layer, demand layer, shape, index of each production layer, input tensor index of each demand layer, and size of the input tensor of each demand layer. During the fusion process of the connection layers, the input tensors of the connection layers need to be deleted, but the input tensors of the connection layers need to be saved; therefore, they can only be saved in the output tensors of the connection layers. Specifically, the computer device extracts the production layer corresponding to each input tensor of the connection layer from the computation graph, using it as the production layer of the current connection layer's output tensor. It also extracts the demand layer corresponding to each input tensor of the connection layer from the computation graph, using it as the demand layer of the current connection layer's output tensor. Finally, it extracts the shape corresponding to each input tensor of the connection layer from the computation graph, using it as the shape of the current connection layer's output tensor. After extracting the template data, the computer device can add the extracted template data to the corresponding template item in the data structure template. The computer device can also create a production layer index corresponding to each production layer in the production layer index table of the data structure template. The production layer index is used to distinguish multiple production layers, indicating the position of the production layer in the computation graph, ensuring the accuracy of the position of each production layer in the optimized neural network model. The computer device establishes a requirement layer index corresponding to each requirement layer in the requirement layer data table of the data structure template, and also counts the size of the input tensor corresponding to each requirement layer. The requirement layer index is used to determine where the requirement layer needs to locate its own input tensor in the output tensor, indicating the position of each input tensor in the output tensor. This facilitates finding the corresponding input tensor of each requirement layer based on this position after the join layer is fused. The size of the input tensor corresponding to each requirement layer can be used to represent the number of basic tensors required by each requirement layer. The computer device attaches the requirement layers other than the join layer corresponding to each input tensor of the join layer to the requirement layer data table. The attached requirement layers are operation layers that do not participate in the join operation, avoiding the problem of directly using the output tensor as the input tensor of the operation layer that does not participate in the join operation after the join layer is optimized, thus improving the correctness of the join layer fusion.

[0095] by Figure 5Taking the computation graph as an example, the process of generating the target sub-data structure corresponding to the computation graph is explained. Layers 1-7 are operation layers, layers 3, 5, and 7 are connection layers. The dashed arrows pointing to the consumer layers of tensor1, tensor3, and tensor5 refer to the requirement layers other than the connection layers corresponding to each tensor. The computer device can traverse each operation layer in the order of layers 1-7. When identifying layers 1 and 2, if they are not identified as connection layers, layers 1 and 2 are skipped. Continuing to identify layer 3, which is a connection layer, and both input tensors of layer 3, tensor1 and tensor2, are unoptimized input tensors, the steps to generate the optimized data structure of tensor3 can be illustrated as follows (A1-F1):

[0096] A1.sub_producer_layers=[layer1,layer2]

[0097] B1.sub_consumer_layers=[tensor1's consumer layers, tensor 2'sconsumer layers]

[0098] C1.sub_shapes=[tensor1's shape, tensor2's shape]

[0099] subproducer name layer1 layer2 index 0 1

[0100] sub consumer name tensor1's consumers tensor2's consumers index 0 1 stride 1 1

[0101] F1. Connect the other consumers in the requirement layer corresponding to tensor1 and tensor2, except for the connection layer layer3, to table e.

[0102] In step B1, `tensor1's consumer layers` represents the demand layers corresponding to `tensor1`, and `tensor2's consumer layers` represents the demand layers corresponding to `tensor2`. The table in step E1 refers to the merged table of `sub_consumer_index_table` and `sub_consumer_stride_table`, which can be called the demand layer table. In the demand layer table, `tensor1's consumers` represents the demand layers corresponding to `tensor1`. An index of 0 for `tensor1's consumers` indicates that the demand layers corresponding to `tensor1` can search for the input tensor `tensor1` at the 0th position of the output tensor. A stride of 0 for `tensor1's consumers` indicates that the size of `tensor1` is 1 unit. Similarly, an index of 1 for `tensor2's consumers` indicates that the demand layers corresponding to `tensor2` can search for the input tensor `tensor2` at the 1st position of the output tensor. A stride of 1 for `tensor2's consumers` indicates that the size of `tensor2` is 1 unit.

[0103] The computer continues to identify layer 4. If it finds that layer 4 is not a connection layer, it skips it. It then continues to identify layer 5. Layer 5 is a connection layer, and its input tensor 3 is an optimized input tensor. The steps for generating the optimized data structure of tensor 5 can be illustrated as follows (A2-F2):

[0104] A2. Obtain the optimized data structure of tensor3, and combine the optimized data structure of tensor3 with tensor4.

[0105] B2.sub_producer_layers=[layer1, layer2, layer4]

[0106] C2.sub_consumer_layers=[tensor1's consumer layers, tensor2's consumer layers, tensor3's consumer layers, tensor4's consumer layers]

[0107] D2.sub_shapes=[tensor 1's shape, tensor2's shape, tensor4's shape]

[0108] E2.sub_producer_index_table:

[0109]

[0110] F2.sub_consumer_index_table and sub_consumer_stride_table:

[0111]

[0112] G2. Connect the other consumers in the requirement layers corresponding to tensor1 to tensor4, except for the connection layer layer3, to table e.

[0113] The computer continues to identify layer 6. If it finds that layer 6 is not a connection layer, it skips it. It then continues to identify layer 7. Layer 7 is a connection layer, and its input tensor 5 is an optimized input tensor. The steps for generating the optimized data structure of tensor 7 can be illustrated in diagrams A3-F3 below:

[0114] A3. Obtain the optimized data structure of tensor5, and combine the optimized data structure of tensor5 with tensor6.

[0115] B3.sub_producer_layers=[layer1, layer2, layer4, layer6]

[0116] C3.sub_consumer_layers=[tensor1's consumer layers, tensor2's consumer layers, tensor3's consumer layers, tensor4's consumer layers, tensor5's consumer layers, tensor6's consumer layers]

[0117] D3.sub_shapes=[tensor1's shape, tensor2's shape, tensor4's shape, tensor6's shape]

[0118] E3.sub_producer_index_table:

[0119]

[0120] F3.sub_consumer_index_table:

[0121]

[0122] G3.sub_consumer_stride_table:

[0123]

[0124] H3. Connect the consumers in the requirement layers corresponding to tensors 1 to 6, except for the connection layers layer3 and layer5, to sub_consumer_index_table and sub_consumer_stride_table.

[0125] The computer device uses the optimized data structure of tensor7 obtained as the target sub-data structure corresponding to the computation graph.

[0126] In one embodiment, determining the connection layer data to be optimized based on the target sub-data structure and the connection layer in the computation graph includes: determining the connection layer to be optimized in the connection layer based on the target sub-data structure; obtaining the input tensor corresponding to each connection layer in the connection layer to be optimized in the computation graph; and using the connection layer to be optimized and the obtained input tensor as the connection layer data to be optimized.

[0127] The computer device identifies the connection layers to be optimized within the connection layer based on the target sub-data structure. These connection layers are those that need to be deleted. Since the connection layers concatenate the input tensors of the connection layer, the data stored in the output tensors of the connection layer is identical to the data stored in the input tensors. Therefore, the connection layer and its corresponding input tensors can be deleted. Consequently, the computer device retrieves the input tensors corresponding to each connection layer to be optimized from the computational graph, and uses the connection layer to be optimized and the retrieved input tensors as the data for the connection layer to be optimized.

[0128] by Figure 7 Taking this as an example, the computer device can determine the connection layers to be optimized as layer 3, layer 5, and layer 7 based on the optimized data structure of tensor 7. The input tensors corresponding to layer 3, layer 5, and layer 7 are obtained from the computation graph, resulting in tensors 1 to 6. Therefore, layer 3, layer 5, layer 7, and tensors 1 to 6 are used as the connection layer data to be optimized, and the connection layer data to be optimized is deleted.

[0129] In this embodiment, since the obtained target sub-data structure is a data structure that can be correctly fused into the connection layer, the data to be optimized in the connection layer can be determined based on the target sub-data structure, which can improve the accuracy of the data to be optimized into the connection layer. Thus, the connection layer can be correctly fused based on the data to be optimized into the connection layer. At the same time, after the connection layer is fused, the model inference speed can also be improved.

[0130] In one embodiment, such as Figure 9 As shown, a neural network model inference device is provided, comprising: a task acquisition module 902, a model parsing module 904, a structure generation module 906, a data determination module 908, a model optimization module 910, and a model inference module 912, wherein:

[0131] The task acquisition module 902 is used to acquire neural network model inference tasks, which include model identifiers.

[0132] The model parsing module 904 is used to obtain the neural network model corresponding to the model identifier, parse the neural network model, and obtain the computation graph corresponding to the neural network model. The computation graph includes connection layers.

[0133] The structure generation module 906 is used to obtain a pre-built data structure template and generate the target sub-data structure corresponding to the computation graph based on the data structure template.

[0134] The data determination module 908 is used to determine the data of the connection layer to be optimized based on the target sub-data structure and the connection layer in the computation graph.

[0135] The model optimization module 910 is used to optimize the data of the connection layer to be optimized based on the target substructure data, so as to obtain the optimized neural network model.

[0136] The model inference module 912 is used to perform inference based on the optimized neural network model and obtain the model inference result.

[0137] In one embodiment, the computation graph includes an operation layer and tensors. The operation layer includes a connection layer, and the tensors are either input tensors or output tensors of the operation layer. The structure generation module 906 is further configured to traverse the operation layer in the computation graph, identify the connection layer in the operation layer, generate an optimized data structure for the output tensor corresponding to the connection layer based on the data structure template and the computation graph, and determine the target sub-data structure corresponding to the computation graph based on the optimized data structure for the output tensor corresponding to the connection layer.

[0138] In one embodiment, the above-mentioned apparatus further includes: an identification module, configured to perform topological sorting on the operation layers in the computation graph to obtain a topological sequence; to identify whether each operation layer is a connection layer according to the topological sequence; to skip the operation layer if it is not a connection layer; and to generate an optimized data structure for the output tensor corresponding to the connection layer according to the data structure template and the computation graph if it is a connection layer.

[0139] In one embodiment, the structure generation module 906 is further configured to obtain the current connection layer, identify whether there is an optimized input tensor in the input tensor corresponding to the current connection layer; if there is, obtain the optimized data structure of the optimized output tensor, generate the optimized data structure of the output tensor corresponding to the current connection layer based on the optimized data structure of the optimized output tensor, the computation graph and the data structure template, update the next connection layer to the current connection layer, return to the step of identifying whether there is an optimized input tensor in the input tensor corresponding to the current connection layer, until the optimized data structure of the output tensor corresponding to all connection layers in the operation layer is generated.

[0140] In one embodiment, the structure generation module 906 is further configured to extract template data corresponding to the output tensor of the current connection layer from the computation graph according to the data structure template when there is no optimized input tensor in the input tensor corresponding to the current connection layer, and add the extracted template data to the data structure template to obtain the optimized data structure of the output tensor corresponding to the current connection layer.

[0141] In one embodiment, the structure generation module 906 is further configured to extract the production layer, demand layer, and shape corresponding to each input tensor sequentially from the computation graph according to the data structure template; add the extracted production layer, demand layer, and shape to the data structure template; establish a production layer index corresponding to the production layer in the production layer index table of the data structure template; establish a demand layer index corresponding to the demand layer in the demand layer data table of the data structure template, and count the size of the input tensor corresponding to each demand layer; attach the demand layers other than the connection layers in the demand layers corresponding to the input tensors to the demand layer data table to obtain the optimized data structure of the output tensor corresponding to the current connection layer.

[0142] In one embodiment, the data determination module 908 is further configured to determine the connection layer to be optimized in the connection layer according to the target sub-data structure; obtain the input tensor corresponding to each connection layer in the connection layer to be optimized in the computation graph; and use the connection layer to be optimized and the obtained input tensor as the connection layer data to be optimized.

[0143] In one embodiment, the model optimization module 910 is further configured to delete the data of the connection layer to be optimized, thereby obtaining a computational graph after deletion; and to connect the deleted computational graph according to the target data structure to obtain an optimized neural network model.

[0144] Specific limitations regarding the neural network model inference device can be found in the limitations of the neural network model inference method described above, and will not be repeated here. Each module in the aforementioned neural network model inference device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0145] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 10 As shown, the computer device includes a processor, memory, communication interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions stored in the non-volatile storage media. The database stores data for a neural network model inference method. The communication interface connects to an external terminal for communication. When the computer-readable instructions are executed by the processor, they implement a neural network model inference method.

[0146] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] A computer device includes a memory and one or more processors. The memory stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps described in the various method embodiments above.

[0148] One or more computer storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the various method embodiments described above.

[0149] The computer storage medium is a readable storage medium, which can be either non-volatile or volatile.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A neural network model inference method, comprising: Obtain a neural network model inference task, wherein the neural network model inference task includes a model identifier; Obtain the neural network model corresponding to the model identifier, obtain the input image, parse the neural network model using the input image, and obtain the computation graph corresponding to the neural network model. The computation graph includes connection layers, operation layers, and tensors. The operation layer includes connection layers, and the tensors are either input tensors or output tensors of the operation layer. The neural network model is an image recognition model. Obtaining a pre-constructed data structure template and generating a target sub-data structure corresponding to the computation graph based on the data structure template includes: traversing the operation layers in the computation graph and identifying the connection layers in the operation layers; generating an optimized data structure for the output tensor corresponding to the connection layer based on the data structure template and the computation graph; and determining the target sub-data structure corresponding to the computation graph based on the optimized data structure for the output tensor corresponding to the connection layer. Based on the target sub-data structure and the connection layer, the connection layer data to be optimized is determined in the computation graph; The data of the connection layer to be optimized is optimized according to the target sub-data structure to obtain the optimized neural network model; and The optimized neural network model is used to perform calculations on the image to be recognized to obtain the model inference result. The optimized data structure for generating the output tensor corresponding to the connection layer based on the data structure template and the computation graph includes: Obtain the current connection layer and identify whether there is an optimized input tensor in the input tensor corresponding to the current connection layer; When there is no optimized input tensor in the input tensor corresponding to the current connection layer, the production layer, demand layer and shape corresponding to each input tensor are extracted sequentially in the computation graph according to the data structure template. Add the extracted production layer, demand layer, and shape to the data structure template; Establish a production layer index corresponding to the production layer in the production layer index table of the data structure template; In the requirement layer data table of the data structure template, establish the requirement layer index corresponding to the requirement layer, and calculate the size of the input tensor corresponding to each requirement layer; and The requirement layers other than the connection layers in the requirement layer corresponding to the input tensor are attached to the requirement layer data table to obtain the optimized data structure of the output tensor corresponding to the current connection layer.

2. The method according to claim 1, characterized in that, The method further includes: The operation layers in the computation graph are topologically sorted to obtain a topological sequence; Based on the topology sequence, each operational layer is sequentially identified as a connection layer; If it is not a connection layer, then skip the operation layer; and When it is a connection layer, an optimized data structure for the output tensor of the connection layer is generated based on the data structure template and the computation graph.

3. The method according to any one of claims 1 to 2, characterized in that, The optimized data structure for generating the output tensor corresponding to the connection layer based on the data structure template and the computation graph further includes: When there is an optimized input tensor in the input tensor corresponding to the current connection layer, the optimized data structure of the optimized output tensor is obtained. Based on the optimized data structure of the optimized output tensor, the computation graph, and the data structure template, the optimized data structure of the output tensor corresponding to the current connection layer is generated. The next connection layer is updated to the current connection layer. The step of identifying whether there is an optimized input tensor in the input tensor corresponding to the current connection layer is returned until the optimized data structures of the output tensors corresponding to all connection layers in the operation layer are generated.

4. The method according to claim 1, characterized in that, The step of determining the connection layer data to be optimized in the computation graph based on the target sub-data structure and the connection layer includes: Based on the target sub-data structure, the connection layer to be optimized is determined in the connection layer; Obtain the input tensor corresponding to each connection layer in the connection layer to be optimized from the computation graph; and The connection layer to be optimized and the obtained input tensor are used as the connection layer data to be optimized.

5. The method according to claim 1, characterized in that, The step of optimizing the connection layer data to be optimized according to the target sub-data structure to obtain the optimized neural network model includes: Delete the data of the connection layer to be optimized to obtain the computation graph after deletion; and The computational graph after deletion is connected according to the target sub-data structure to obtain the optimized neural network model.

6. A neural network model inference device, comprising: The task acquisition module is used to acquire neural network model inference tasks, wherein the neural network model inference tasks include model identifiers. The model parsing module is used to obtain the neural network model corresponding to the model identifier, acquire the input image, parse the neural network model through the input image, and obtain the computation graph corresponding to the neural network model. The computation graph includes connection layers, operation layers, and tensors. The operation layer includes connection layers, and the tensors are either input tensors or output tensors of the operation layer. The neural network model is an image recognition model. A structure generation module is used to obtain a pre-constructed data structure template and generate a target sub-data structure corresponding to the computation graph based on the data structure template. This includes: traversing the operation layers in the computation graph and identifying the connection layers within the operation layers; generating an optimized data structure for the output tensors corresponding to the connection layers based on the data structure template and the computation graph; and determining the target sub-data structure corresponding to the computation graph based on the optimized data structure for the output tensors corresponding to the connection layers. Specifically, generating the optimized data structure for the output tensors corresponding to the connection layers based on the data structure template and the computation graph includes: obtaining the current connection layer and identifying whether there are optimized input tensors among the input tensors corresponding to the current connection layer; when... When there is no optimized input tensor among the input tensors corresponding to the current connection layer, the production layer, demand layer, and shape corresponding to each input tensor are extracted sequentially in the computation graph according to the data structure template; the extracted production layer, demand layer, and shape are added to the data structure template; a production layer index corresponding to the production layer is established in the production layer index table of the data structure template; a demand layer index corresponding to the demand layer is established in the demand layer data table of the data structure template, and the size of the input tensor corresponding to each demand layer is counted; and the demand layers other than the connection layer in the demand layer corresponding to the input tensor are attached to the demand layer data table to obtain the optimized data structure of the output tensor corresponding to the current connection layer. The data determination module is used to determine the connection layer data to be optimized in the computation graph based on the target sub-data structure and the connection layer. The model optimization module is used to optimize the connection layer data to be optimized according to the target sub-data structure to obtain an optimized neural network model; and The model inference module is used to perform calculations on the image to be recognized based on the optimized neural network model to obtain the model inference result.

7. A computer device comprising a memory and one or more processors, the memory storing computer-readable instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: Obtain a neural network model inference task, wherein the neural network model inference task includes a model identifier; Obtain the neural network model corresponding to the model identifier, obtain the input image, parse the neural network model using the input image, and obtain the computation graph corresponding to the neural network model. The computation graph includes connection layers, operation layers, and tensors. The operation layer includes connection layers, and the tensors are either input tensors or output tensors of the operation layer. The neural network model is an image recognition model. Obtain a pre-constructed data structure template, and generate the target sub-data structure corresponding to the computation graph based on the data structure template, including: The operation layers in the computation graph are traversed to identify the connection layers within the operation layers. The optimized data structure of the output tensor corresponding to the connection layer is generated according to the data structure template and the computation graph; the target sub-data structure corresponding to the computation graph is determined according to the optimized data structure of the output tensor corresponding to the connection layer; wherein, generating the optimized data structure of the output tensor corresponding to the connection layer according to the data structure template and the computation graph includes: obtaining the current connection layer, identifying whether there is an optimized input tensor in the input tensor corresponding to the current connection layer; when there is no optimized input tensor in the input tensor corresponding to the current connection layer, extracting the production layer, demand layer and shape corresponding to each input tensor sequentially in the computation graph according to the data structure template; adding the extracted production layer, demand layer and shape to the data structure template; establishing a production layer index corresponding to the production layer in the production layer index table of the data structure template; establishing a demand layer index corresponding to the demand layer in the demand layer data table of the data structure template, and counting the size of the input tensor corresponding to each demand layer; and attaching the demand layers other than the connection layer in the demand layer corresponding to the input tensor to the demand layer data table to obtain the optimized data structure of the output tensor corresponding to the current connection layer. Based on the target sub-data structure and the connection layer, the connection layer data to be optimized is determined in the computation graph; The data of the connection layer to be optimized is optimized according to the target sub-data structure to obtain the optimized neural network model; and The optimized neural network model is used to perform calculations on the image to be recognized, and the model inference results are obtained.

8. The computer device according to claim 7, characterized in that, When the processor executes the computer-readable instructions, it further performs the following steps: topologically sorting the operation layers in the computation graph to obtain a topological sequence; sequentially identifying whether each operation layer is a connection layer according to the topological sequence; skipping the operation layer if it is not a connection layer; and generating an optimized data structure for the output tensor corresponding to the connection layer according to the data structure template and the computation graph if it is a connection layer.

9. The computer device according to any one of claims 7 to 8, characterized in that, When the processor executes the computer-readable instructions, it further performs the following steps: when there is an optimized input tensor in the input tensor corresponding to the current connection layer, it obtains the optimized data structure of the optimized output tensor, generates the optimized data structure of the output tensor corresponding to the current connection layer based on the optimized data structure of the optimized output tensor, the computation graph, and the data structure template, updates the next connection layer to the current connection layer, and returns to the step of identifying whether there is an optimized input tensor in the input tensor corresponding to the current connection layer, until the optimized data structures of the output tensors corresponding to all connection layers in the operation layer are generated.

10. One or more computer storage media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the following steps: Obtain a neural network model inference task, wherein the neural network model inference task includes a model identifier; Obtain the neural network model corresponding to the model identifier, obtain the input image, parse the neural network model using the input image, and obtain the computation graph corresponding to the neural network model. The computation graph includes connection layers, operation layers, and tensors. The operation layer includes connection layers, and the tensors are either input tensors or output tensors of the operation layer. The neural network model is an image recognition model. Obtain a pre-constructed data structure template, and generate the target sub-data structure corresponding to the computation graph based on the data structure template, including: The operation layers in the computation graph are traversed to identify the connection layers within the operation layers. The optimized data structure of the output tensor corresponding to the connection layer is generated according to the data structure template and the computation graph; the target sub-data structure corresponding to the computation graph is determined according to the optimized data structure of the output tensor corresponding to the connection layer; wherein, generating the optimized data structure of the output tensor corresponding to the connection layer according to the data structure template and the computation graph includes: obtaining the current connection layer, identifying whether there is an optimized input tensor in the input tensor corresponding to the current connection layer; when there is no optimized input tensor in the input tensor corresponding to the current connection layer, extracting the production layer, demand layer and shape corresponding to each input tensor sequentially in the computation graph according to the data structure template; adding the extracted production layer, demand layer and shape to the data structure template; establishing a production layer index corresponding to the production layer in the production layer index table of the data structure template; establishing a demand layer index corresponding to the demand layer in the demand layer data table of the data structure template, and counting the size of the input tensor corresponding to each demand layer; and attaching the demand layers other than the connection layer in the demand layer corresponding to the input tensor to the demand layer data table to obtain the optimized data structure of the output tensor corresponding to the current connection layer. Based on the target sub-data structure and the connection layer, the connection layer data to be optimized is determined in the computation graph; The data of the connection layer to be optimized is optimized according to the target sub-data structure to obtain the optimized neural network model; and The optimized neural network model is used to perform calculations on the image to be recognized, and the model inference results are obtained.

11. The storage medium according to claim 10, characterized in that, When the computer-readable instructions are executed by the processor, the following steps are also performed: topologically sorting the operation layers in the computation graph to obtain a topological sequence; sequentially identifying whether each operation layer is a connection layer according to the topological sequence; skipping the operation layer if it is not a connection layer; and generating an optimized data structure for the output tensor corresponding to the connection layer according to the data structure template and the computation graph if it is a connection layer.

12. The storage medium according to claim 10 or 11, characterized in that, When the computer-readable instructions are executed by the processor, the following steps are also performed: when there is an optimized input tensor in the input tensor corresponding to the current connection layer, the optimized data structure of the optimized output tensor is obtained; the optimized data structure of the output tensor corresponding to the current connection layer is generated based on the optimized data structure of the optimized output tensor, the computation graph, and the data structure template; the next connection layer is updated to the current connection layer; the step of identifying whether there is an optimized input tensor in the input tensor corresponding to the current connection layer is returned, until the optimized data structures of the output tensors corresponding to all connection layers in the operation layer are generated.

Images