A neural network model-based inference method, device, equipment and medium

Abstract

The application discloses a reasoning method and device based on a neural network model, equipment and a medium, comprising: determining a first target network layer with the same structure and a second target network layer with different structures in the neural network model; performing network layer splitting on the neural network model to obtain a first calculation subgraph and a second calculation subgraph corresponding to the first target network layer and the second target network layer respectively; generating a shared engine file corresponding to the first target network layer and weight parameters of each first target network layer according to the first calculation subgraph; generating an independent engine file corresponding to the second target network layer according to the second calculation subgraph; the independent engine file includes the weight parameters of the second target network layer; and obtaining a reasoning result through the shared engine file, the weight parameters of each first target network layer and the independent engine file according to the execution order of each calculation subgraph. Through the shared engine file reasoning of the network layers with the same structure, data storage can be reduced, and single-card reasoning capability can be improved.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a reasoning method, apparatus, device, and medium based on a neural network model. Background Technology

[0002] With the development of intelligent technologies, neural network models have been widely used in many fields. For example, large models are used in dialogue, image understanding, image generation, code generation, and multimodal applications.

[0003] However, as network models become increasingly complex, the size of neural network models, especially large models, far exceeds the storage capacity of computing devices. Current technologies employ model parallelism, using multiple computing devices running simultaneously to complete model inference.

[0004] However, the simultaneous operation of multiple computing devices increases the cost of large model inference, making large model applications prohibitive for cost-sensitive users. Summary of the Invention

[0005] This invention provides a reasoning method, apparatus, device, and medium based on a neural network model to improve single-card reasoning capabilities, enabling the neural network model to be completed in a single computing device and reducing reasoning costs.

[0006] According to one aspect of the present invention, a reasoning method based on a neural network model is provided, the method comprising:

[0007] Structural analysis is performed on the neural network model to determine the first target network layer with the same structure and the second target network layer with a different structure in the neural network model;

[0008] The neural network model is split into network layers based on the first target network layer and the second target network layer to obtain a first computational subgraph and a second computational subgraph corresponding to the first target network layer and the second target network layer, respectively.

[0009] Generate a shared engine file corresponding to the first target network layer based on the first computational subgraph, as well as the weight parameters of each of the first target network layers;

[0010] An independent engine file corresponding to the second target network layer is generated based on the second computational subgraph; wherein, the independent engine file includes the weight parameters of the second target network layer;

[0011] Based on the execution order of each computational subgraph, inference calculations are performed using the shared engine file, the weight parameters of each of the first target network layers, and the independent engine file to obtain the inference result.

[0012] According to another aspect of the present invention, a reasoning device based on a neural network model is provided, the device comprising:

[0013] The structural analysis module is used to perform structural analysis on the neural network model to determine the first target network layer with the same structure and the second target network layer with different structures in the neural network model.

[0014] The network layer splitting module is used to split the neural network model according to the first target network layer and the second target network layer to obtain a first computational subgraph and a second computational subgraph corresponding to the first target network layer and the second target network layer, respectively.

[0015] The shared engine file generation module is used to generate a shared engine file corresponding to the first target network layer and the weight parameters of each first target network layer based on the first computation subgraph.

[0016] An independent engine file generation module is used to generate an independent engine file corresponding to the second target network layer based on the second computational subgraph; wherein, the independent engine file includes the weight parameters of the second target network layer;

[0017] The inference result determination module is used to perform inference calculations based on the execution order of each computation subgraph, using the shared engine file, the weight parameters of each first target network layer, and the independent engine file, to obtain the inference result.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the inference method based on a neural network model according to any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the inference method based on a neural network model as described in any embodiment of the present invention.

[0023] The technical solution of this invention, through structural analysis of a neural network model, identifies a first target network layer with the same structure and a second target network layer with a different structure. The neural network model is then split into network layers based on the first and second target network layers, resulting in a first computational subgraph and a second computational subgraph corresponding to the first and second target network layers, respectively. A shared engine file corresponding to the first target network layer and weight parameters for each of the first target network layers are generated based on the first computational subgraph. An independent engine file corresponding to the second target network layer is generated based on the second computational subgraph, wherein the independent engine file includes the weight parameters for the second target network layer. Based on the execution order of each computational subgraph, inference calculations are performed using the shared engine file, the weight parameters for each of the first target network layers, and the independent engine file to obtain the inference result. This solves the problem that a single computing device cannot perform inference on large-scale neural network models. By splitting the network layers, network layers with the same structure can share an engine file for inference, reducing data storage requirements and enabling the neural network model to run on a single GPU, thus improving single-GPU inference capabilities.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a reasoning method based on a neural network model provided in Embodiment 1 of the present invention;

[0027] Figure 2 This is a schematic diagram of a transformer model structure provided in Embodiment 1 of the present invention;

[0028] Figure 3 This is a schematic diagram of a computational subgraph provided in Embodiment 1 of the present invention;

[0029] Figure 4 This is a flowchart of the inference process of a first target network layer according to Embodiment 1 of the present invention;

[0030] Figure 5This is a flowchart of a reasoning method based on a neural network model according to Embodiment 2 of the present invention;

[0031] Figure 6 This is a schematic diagram of the structure of an inference device based on a neural network model according to Embodiment 3 of the present invention;

[0032] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the reasoning method based on a neural network model according to an embodiment of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] Example 1

[0036] Figure 1 This is a flowchart of an inference method based on a neural network model according to Embodiment 1 of the present invention. This embodiment is applicable to situations where a single computing device performs neural network model inference. The method can be executed by an inference device based on a neural network model, which can be implemented in hardware and / or software and can be configured in an electronic device such as a computer. Figure 1 As shown, the method includes:

[0037] Step 110: Perform structural analysis on the neural network model to determine the first target network layer with the same structure and the second target network layer with a different structure.

[0038] Neural network models can be composed of network layers with the same network structure. For example, neural network models include the Transformer model. With the development of models, the Transformer model has evolved from GPT, Bloom, and Falcon to LLAMA2. While the implementation of network layers has changed significantly, the overall structure has remained relatively stable.

[0039] Figure 2 This is a schematic diagram of a transformer model structure provided according to Embodiment 1 of the present invention. Figure 2 As shown, although the Transformer model has undergone various changes, it still includes an embedding layer, a decoder layer, a layer normalization layer, and a fully connected linear layer. In the Transformer model, there is typically only one embedding layer, and only one output layer consisting of layer normalization and linear layers, while there are multiple decoders, such as n layers.

[0040] Therefore, after structural analysis of the transformer model, it can be determined that the transformer model consists of a decoder layer with the same structure, and an embedding layer and an output layer with different structures. Specifically, the first target network layer includes a decoding layer; the second target network layer includes an embedding layer, and an output layer consisting of layer normalization and fully connected layers.

[0041] Step 120: Decompose the neural network model into network layers according to the first target network layer and the second target network layer to obtain the first computational subgraph and the second computational subgraph corresponding to the first target network layer and the second target network layer, respectively.

[0042] Figure 3 This is a schematic diagram of a computational subgraph provided according to Embodiment 1 of the present invention. Figure 3 As shown, the transformer model can be decomposed into three computational subgraphs: subgraph A corresponding to the embedding layer, subgraph B corresponding to the decoder layer, and subgraph C corresponding to the output layer consisting of the layer normal layer and the linear layer. That is, when the neural network model includes a transformer model, the first computational subgraph includes subgraph B; the second computational subgraph includes subgraphs A and C.

[0043] Step 130: Generate a shared engine file corresponding to the first target network layer and the weight parameters of each first target network layer based on the first computational subgraph.

[0044] There can be multiple first target network layers. Multiple first target network layers can share the same common engine file. The common engine file can be generated from the computational subgraph of subgraph B, i.e., the decoder layer. The engine file can be a file generated by compiling the computational subgraph into an executable program. For example, the computational subgraph can be optimized, including but not limited to: constant folding, operator fusion, and automatic mixing, and then compiled to generate the engine file.

[0045] Specifically, the shared engine file can include executable programs corresponding to one or more decoder layers. That is, for n decoder layers in a transformer model, a shared engine file can be generated for a single decoder layer, and each decoder layer performs inference based on the shared engine file; or, for n decoder layers in a transformer model, a shared engine file can be generated for m decoder layers, and all decoder layers can be divided into multiple groups (e.g., n / m groups), with inference performed on a group-by-group basis based on the shared engine file. m is a natural number greater than 1 and less than n.

[0046] By sharing engine files, the amount of storage required for the engine files corresponding to the decoder layer during neural network model processing can be reduced, saving storage space occupied during neural network model inference and thus enabling computing devices to have higher inference capabilities.

[0047] In the shared engine file, the weight parameters of the decoder layers can be set to general parameters, while different decoder layers can have different weight parameters. Replacing the general parameters in the shared engine file with the weight parameters used by the decoder layers yields the decoder layer's engine file. Alternatively, the shared engine file can include the weight parameters for the first or the first m decoder layers; these weight parameters are then replaced when generating engine files for subsequent decoder layers.

[0048] The weight parameters of the first target network layer can include the parameters used by the attention mechanism layer and the MLP layer in the decoder layer, such as the parameters of the fully connected layer for calculating KQV.

[0049] Step 140: Generate an independent engine file corresponding to the second target network layer based on the second computational subgraph; wherein, the independent engine file includes the weight parameters of the second target network layer.

[0050] The independent engine file can be an executable program file generated based on the second computational subgraph. Specifically, a first independent engine file can be generated based on the computational subgraph of the embedding layer; a second independent engine file can be generated based on the computational subgraphs of the output layers, namely layernorm and linear. The independent engine file includes the weight parameters of the second target network layer. When the first independent engine file is executed, the inference results of the embedding layer are obtained; when the second independent engine file is executed, the inference results of layernorm and linear are obtained.

[0051] Step 150: Based on the execution order of each computational subgraph, inference calculations are performed using the shared engine file, the weight parameters of each first target network layer, and the independent engine file to obtain the inference results.

[0052] During the inference process, the execution order must be maintained the same as the original neural network model. Specifically, the inference results of the first independent engine file need to be transferred to the shared engine file. Based on the shared engine file and the weight parameters of each first target network layer, the engine files corresponding to each first target network layer can be obtained. Each engine file is executed sequentially according to the order in the original network model, that is, the obtained inference results are transferred to the engine file corresponding to the next first target network layer for continued execution, so as to obtain the inference results of all first target network layers. Then, the inference results obtained from the execution of all first target network layers are transferred to the second independent engine file for inference to obtain the final inference result.

[0053] To illustrate the generation process of the inference result more specifically, in an optional embodiment of the present invention, inference calculations are performed according to the execution order of each computational subgraph, using a shared engine file, the weight parameters of each first target network layer, and an independent engine file to obtain the inference result. This includes: obtaining the previous inference result according to the execution order of each computational subgraph, and selecting either a shared engine file or an independent engine file; if a shared engine file is selected, determining the current network layer corresponding to the current inference calculation in each first target network layer, and loading the current weight parameters of the current network layer; generating the current engine file for the current network layer based on the current weight parameters and the shared engine file; performing inference calculations based on the current engine file and the previous inference result to obtain the current inference result; if an independent engine file is selected, performing inference calculations based on the independent engine file and the previous inference result to obtain the current inference result.

[0054] Optionally, the current engine file for the current network layer is generated based on the current weight parameters and the shared engine file, including: generating the current engine file for the current network layer through weight refitting based on the current weight parameters and the shared engine file.

[0055] Weight refitting is an executable program editing technique. The technical solution of this invention generates the current engine file for the current network layer using weight refitting technology, eliminating the need to recompile and generate an executable program. Therefore, its execution speed exceeds that of compiling and generating an executable program, improving the computational speed of executing neural network models on a single computing device.

[0056] To further improve the computational speed of executing neural network models on a single computing device, based on the above implementation method, optionally, the computing device can be virtualized into at least two virtual devices; according to the execution order of each computational subgraph, inference calculations are performed using a shared engine file, the weight parameters of each first target network layer, and an independent engine file to obtain inference results, including: if a shared engine file is selected in both the current inference calculation and the next inference calculation, then the current network layer and the next network layer corresponding to the current inference calculation and the next inference calculation are determined in each first target network layer; the current weight parameters of the current network layer are loaded through the first virtual device, and the current engine file of the current network layer is generated based on the current weight parameters and the shared engine file; while performing inference calculations through the first virtual device based on the current engine file and the previous inference result to obtain the current inference result, the next weight parameters of the next network layer are loaded through the second virtual device, and the next engine file of the next network layer is generated based on the next weight parameters and the shared engine file; the current inference result is obtained through the second virtual device, and inference calculations are performed based on the current inference result and the next engine file to obtain the next inference result.

[0057] Figure 4 This is a flowchart of the inference process for a first target network layer according to Embodiment 1 of the present invention. Figure 4 As shown, the example illustrates the use of a computing device virtualized into two virtual devices, namely the first virtual device and the second virtual device. The execution method remains the same when the computing device is virtualized into more than one virtual device.

[0058] like Figure 4 As shown, after executing the first independent engine file, the inference result can be sent to the first virtual device. The first virtual device obtains the shared engine file corresponding to the decoder of the first layer or the first m layers and performs inference. If the shared engine file does not include weight parameters, the first virtual device obtains the current weight parameters of the current network layer corresponding to the current inference calculation and generates the current engine file based on the shared engine file. The first virtual device can generate the current engine file using weight refitting technology. The current network layer can be one or m decoder layers.

[0059] While the first virtual device performs inference, the second virtual device loads the next weight parameters for the next network layer corresponding to the next inference calculation. Based on these next weight parameters, the second virtual device can generate the next engine file using weight refitting technology. Afterward, the second virtual device can obtain the inference results from the first virtual device and perform inference based on the next engine file.

[0060] If, based on the execution order of the computation subgraph, it is determined after inference on the second virtual device that there are still decoder layers that have not been executed, then... Figure 4 As shown, while the second virtual device performs inference, the first virtual device continues to load the weight parameters of the network layer corresponding to the next inference calculation to generate the engine file. The first virtual device obtains the inference result from the second virtual device and performs inference based on the engine file.

[0061] In the above exemplary process, the first virtual device and the second virtual device execute in parallel until the inference of the decoder layer ends, and then the inference of the output layer is performed based on the inference result. Figure 4 The inference process shown hides the time spent loading weight parameters and generating engine files using WeightRefitting technology during the inference process, thereby improving the efficiency of model inference on a single computing device.

[0062] The technical solution of this embodiment analyzes the structure of a neural network model to identify a first target network layer with the same structure and a second target network layer with a different structure. The neural network model is then split into network layers based on the first and second target network layers, resulting in a first computational subgraph and a second computational subgraph corresponding to the first and second target network layers, respectively. A shared engine file corresponding to the first target network layer and weight parameters for each first target network layer are generated based on the first computational subgraph. An independent engine file corresponding to the second target network layer is generated based on the second computational subgraph, including the weight parameters for the second target network layer. Based on the execution order of each computational subgraph, inference calculations are performed using the shared engine file, the weight parameters for each first target network layer, and the independent engine file to obtain the inference result. This solves the problem that a single computing device cannot perform neural network model inference. By splitting the network layers, network layers with the same structure can share an engine file for inference, reducing data storage requirements and enabling the neural network model to run on a single GPU, thus improving single-GPU inference capabilities.

[0063] Example 2

[0064] Figure 5This is a flowchart of a reasoning method based on a neural network model according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 5 As shown, the method includes:

[0065] Step 510: Perform structural analysis on the neural network model to determine the first target network layer with the same structure and the second target network layer with a different structure.

[0066] Step 520: Decompose the neural network model into network layers according to the first target network layer and the second target network layer to obtain the first computation subgraph and the second computation subgraph corresponding to the first target network layer and the second target network layer, respectively.

[0067] Step 530: Estimate the static and dynamic storage capacity required for a single first target network layer in neural network model inference.

[0068] Static storage capacity refers to the storage space occupied by a single first target network layer in the computing device when generating the engine file, i.e., the size of the executable program corresponding to a single first target network layer. Dynamic storage capacity consists of two parts. One part is the stack space occupied by the engine file during runtime. The other part is the space occupied by additional data structures needed by the first target network layer during incremental inference. For example, when the neural network model is a transformer model, the decoder layer needs to use the KVcache data structure during incremental inference. The KVcache size is determined by several factors. For example, the KVcache size can vary depending on the attention mechanism. Attention mechanisms can include multi-head attention, multi-query attention, and group query attention. Furthermore, factors affecting the KVcache size may include: inference batch size, the number of attention mechanism heads, the length of the inference sequence, and the dimension of the decoder layer's latent vectors.

[0069] Step 540: Determine the number of device-accommodating layers that the computing device can accommodate for the first target network layer based on the device storage capacity, static storage capacity, and dynamic storage capacity of the computing device.

[0070] The storage capacity required for the network layer in a computing device should be less than or equal to the device's total storage capacity. Therefore, the number of layers a device can accommodate can be determined by dividing the device's storage capacity by (static storage capacity + dynamic storage capacity). The " / " sign indicates taking the quotient. The number of layers a device can accommodate can be greater than or equal to 1, but less than or equal to this quotient value.

[0071] Step 550: Based on the first calculation subgraph and the number of layers that the device can accommodate, generate a common engine file corresponding to the first target network layer, as well as the weight parameters of each first target network layer.

[0072] It should be noted that the more decoder layers contained in the shared engine file, the larger the storage capacity occupied by the shared engine file, and the faster the inference speed of a single computing device, but the shorter the inference sequence length will be. To ensure good inference performance, the longest inference sequence length can be used when determining the KVcache size.

[0073] In an optional embodiment of the present invention, a shared engine file corresponding to the first target network layer and weight parameters of each first target network layer are generated based on the first computational subgraph and the number of layers that the device can accommodate. This includes: determining the number of model network layers of the first target network layer in the neural network model; selecting a number of target network layers that is less than or equal to the number of layers that the device can accommodate and that can be divided evenly by the number of model network layers; and generating a shared engine file corresponding to the first target network layer and weight parameters of each first target network layer based on the first computational subgraph and the number of target network layers.

[0074] When a shared engine file includes multiple network layers, it is difficult to output the inference result from a single network layer. Therefore, in this embodiment of the invention, when determining the number of network layers included in the shared engine file, the number of network layers can be set as a factor of the number of model network layers of the first target network layer. That is, among values ​​less than or equal to the number of layers the device can accommodate, the number of target network layers is selected that is divisible by the number of model network layers. The shared engine file can contain the number of first target network layers equal to the number of target network layers. For example, the shared engine file can be generated based on the executable program of the number of decoder layers of the first target network layer in the transformer model.

[0075] The above method allows for the selection of an appropriate number of target network layers to generate a shared engine file. This ensures that the generated shared engine file can be accommodated on a single computing device, while also achieving a larger number of target network layers to improve the inference speed of a single computing device.

[0076] Step 560: Generate an independent engine file corresponding to the second target network layer based on the second computational subgraph; wherein, the independent engine file includes the weight parameters of the second target network layer.

[0077] Step 570: Based on the execution order of each computational subgraph, inference calculations are performed using the shared engine file, the weight parameters of each first target network layer, and the independent engine file to obtain the inference results.

[0078] In an optional embodiment of the present invention, inference calculations are performed according to the execution order of each computational subgraph, using a shared engine file, weight parameters of each first target network layer, and an independent engine file to obtain an inference result. This includes: obtaining the previous inference result according to the execution order of each computational subgraph, and selecting either a shared engine file or an independent engine file; if a shared engine file is selected, determining the current network layer corresponding to the current inference calculation in each first target network layer, and loading the current weight parameters of the current network layer; generating the current engine file for the current network layer based on the current weight parameters and the shared engine file; performing inference calculations based on the current engine file and the previous inference result to obtain the current inference result; if an independent engine file is selected, performing inference calculations based on the independent engine file and the previous inference result to obtain the current inference result.

[0079] In an optional embodiment of the present invention, the computing device can be virtualized into at least two virtual devices. Based on the execution order of each computational subgraph, inference calculations are performed using a shared engine file, weight parameters of each first target network layer, and independent engine files to obtain inference results. This includes: if a shared engine file is selected in both the current and next inference calculations, then the current network layer and the next network layer corresponding to the current and next inference calculations are determined in each first target network layer; the current weight parameters of the current network layer are loaded through the first virtual device, and a current engine file for the current network layer is generated based on the current weight parameters and the shared engine file; while performing inference calculations through the first virtual device based on the current engine file and the previous inference result to obtain the current inference result, the next weight parameters of the next network layer are loaded through the second virtual device, and a next engine file for the next network layer is generated based on the next weight parameters and the shared engine file; the current inference result is obtained through the second virtual device, and inference calculations are performed based on the current inference result and the next engine file to obtain the next inference result.

[0080] In an optional embodiment of the present invention, generating the current engine file of the current network layer based on the current weight parameters and the shared engine file includes: generating the current engine file of the current network layer through weight refitting based on the current weight parameters and the shared engine file.

[0081] The technical solution of this invention involves structural analysis of a neural network model to identify a first target network layer with the same structure and a second target network layer with a different structure. The neural network model is then split into network layers based on the first and second target network layers to obtain a first computational subgraph and a second computational subgraph corresponding to the first and second target network layers, respectively. The static and dynamic storage capacities required for a single first target network layer in neural network model inference are estimated. Based on the device storage capacity, static storage capacity, and dynamic storage capacity of the computing device, the number of device-compatible layers that the computing device can accommodate for the first target network layer is determined. Finally, based on the first computational subgraph and the number of device-compatible layers, a shared engine file corresponding to the first target network layer and each first target network layer are generated. The system calculates the weight parameters of the network layers; generates independent engine files corresponding to the second target network layer based on the second computation subgraph; the independent engine files include the weight parameters of the second target network layer; and performs inference calculations based on the execution order of each computation subgraph, using the shared engine files, the weight parameters of each first target network layer, and the independent engine files to obtain the inference results. This solves the problem that a single computing device cannot perform neural network model inference. By splitting the network layers, network layers with the same structure can share engine files for inference, which can reduce data storage requirements, allow neural network models to run on a single card, and improve the inference capability of a single card. Furthermore, by reasonably determining the number of target network layers to generate a shared engine file of the most suitable size, the system can improve the execution speed of a single device while ensuring that a single device can execute neural network models.

[0082] Example 3

[0083] Figure 6 This is a schematic diagram of the structure of an inference device based on a neural network model according to Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a structure analysis module 610, a network layer splitting module 620, a shared engine file generation module 630, an independent engine file generation module 640, and an inference result determination module 650. Wherein:

[0084] The structural analysis module 610 is used to perform structural analysis on the neural network model to determine the first target network layer with the same structure and the second target network layer with different structures in the neural network model.

[0085] The network layer splitting module 620 is used to split the neural network model according to the first target network layer and the second target network layer to obtain the first computation subgraph and the second computation subgraph corresponding to the first target network layer and the second target network layer, respectively.

[0086] The shared engine file generation module 630 is used to generate a shared engine file corresponding to the first target network layer and the weight parameters of each first target network layer based on the first computation subgraph.

[0087] Independent engine file generation module 640 is used to generate an independent engine file corresponding to the second target network layer based on the second computation subgraph; wherein, the independent engine file includes the weight parameters of the second target network layer;

[0088] The inference result determination module 650 is used to perform inference calculations based on the execution order of each computation subgraph, using a shared engine file, the weight parameters of each first target network layer, and an independent engine file, to obtain the inference result.

[0089] Optionally, the shared engine file generation module 630 includes:

[0090] A storage capacity determination unit is used to estimate the static and dynamic storage capacity required by a single first target network layer in neural network model inference.

[0091] The device layer capacity determination unit is used to determine the number of device layers that the computing device can accommodate for the first target network layer based on the device storage capacity, static storage capacity and dynamic storage capacity of the computing device.

[0092] The shared engine file generation unit is used to generate a shared engine file corresponding to the first target network layer, as well as the weight parameters of each first target network layer, based on the first computational subgraph and the number of layers that the device can accommodate.

[0093] Optionally, a shared engine file generation unit is used specifically for:

[0094] Determine the number of network layers in the model of the first target network layer in the neural network model;

[0095] Select a target network layer number that is less than or equal to the number of layers the device can accommodate and that is divisible by the number of network layers in the model.

[0096] Based on the first computational subgraph and the number of target network layers, a common engine file corresponding to the first target network layer and the weight parameters of each first target network layer are generated.

[0097] Optionally, the inference result determination module 650 includes:

[0098] The engine file selection unit is used to obtain the previous inference result according to the execution order of each computation subgraph, and select a shared engine file or an independent engine file;

[0099] The current weight parameter loading unit is used to determine the current network layer corresponding to the current inference calculation in each first target network layer if a common engine file is selected, and to load the current weight parameters of the current network layer.

[0100] The current engine file generation unit is used to generate the current engine file for the current network layer based on the current weight parameters and the shared engine file.

[0101] The current inference result determination unit is used to perform inference calculations based on the current engine file and the previous inference result to obtain the current inference result;

[0102] The current inference result determination unit is used to perform inference calculations based on the independent engine file and the previous inference result if an independent engine file is selected, and to obtain the current inference result.

[0103] Optionally, the computing device can be virtualized into at least two virtual devices;

[0104] The reasoning result determination module 650 includes:

[0105] The network layer determination unit is used to determine the current network layer and the next network layer corresponding to the current inference calculation and the next inference calculation respectively in each first target network layer if a common engine file is selected in both the current inference calculation and the next inference calculation.

[0106] Another current engine file generation unit is used to load the current weight parameters of the current network layer through the first virtual device, and generate the current engine file of the current network layer based on the current weight parameters and the shared engine file;

[0107] The next engine file generation unit is used to perform inference calculations based on the current engine file and the previous inference result through the first virtual device to obtain the current inference result, while loading the next weight parameters of the next network layer through the second virtual device, and generating the next engine file of the next network layer based on the next weight parameters and the shared engine file.

[0108] The next inference result determination unit is used to obtain the current inference result through the second virtual device, and perform inference calculations based on the current inference result and the next engine file to obtain the next inference result.

[0109] Optionally, the current engine file generation unit includes:

[0110] The current engine file generation subunit is used to generate the current engine file for the current network layer based on the current weight parameters and the shared engine file through weight refitting.

[0111] Optional, neural network models, including: transformer models;

[0112] The first target network layer includes a decoding layer; the second target network layer includes an embedding layer and an output layer consisting of layer normalization and fully connected layers.

[0113] The inference device based on a neural network model provided in this embodiment of the invention can execute the inference method based on a neural network model provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0114] Example 4

[0115] Figure 7 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0116] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0117] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0118] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as inference methods based on neural network models.

[0119] In some embodiments, the neural network model-based inference method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the neural network model-based inference method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the neural network model-based inference method by any other suitable means (e.g., by means of firmware).

[0120] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0124] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0125] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0126] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0127] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A neural network model based inference method, characterized in that, The method comprises the following steps: performing structural analysis on a neural network model to determine first target network layers with the same structure and second target network layers with different structures in the neural network model; performing network layer splitting on the neural network model according to the first target network layers and the second target network layers to obtain first calculation subgraphs and second calculation subgraphs corresponding to the first target network layers and the second target network layers respectively; generating a shared engine file corresponding to the first target network layers and weight parameters of each first target network layer according to the first calculation subgraph; generating an independent engine file corresponding to the second target network layers, wherein the independent engine file comprises weight parameters of the second target network layers; performing inference calculation through the shared engine file, the weight parameters of each first target network layer, and the independent engine file according to the execution order of each calculation subgraph to obtain an inference result; The method for generating a shared engine file corresponding to the first target network layers and weight parameters of each first target network layer according to the first calculation subgraph comprises the following steps: estimating the static storage capacity and dynamic storage capacity required by a single first target network layer in neural network model inference; determining the number of device-accommodating layers of the first target network layers that can be accommodated by a computing device according to the device storage capacity of the computing device, the static storage capacity, and the dynamic storage capacity; generating a shared engine file corresponding to the first target network layers and weight parameters of each first target network layer according to the first calculation subgraph and the number of device-accommodating layers; The computing device can be virtually divided into at least two virtual devices; The method for performing inference calculation through the shared engine file, the weight parameters of each first target network layer, and the independent engine file according to the execution order of each calculation subgraph to obtain an inference result comprises the following steps: if the shared engine file is selected in the current inference calculation and the next inference calculation, determining a current network layer and a next network layer corresponding to the current inference calculation and the next inference calculation respectively in each first target network layer; loading the current weight parameters of the current network layer through a first virtual device and generating a current engine file of the current network layer according to the current weight parameters and the shared engine file; loading the next weight parameters of the next network layer through a second virtual device and generating a next engine file of the next network layer according to the next weight parameters and the shared engine file while performing inference calculation through the first virtual device according to the current engine file and a previous inference result to obtain a current inference result; obtaining the current inference result through the second virtual device and performing inference calculation according to the current inference result and the next engine file to obtain a next inference result.

2. The method of claim 1, wherein, The method for generating a shared engine file corresponding to the first target network layers and weight parameters of each first target network layer according to the first calculation subgraph and the number of device-accommodating layers comprises the following steps: determining a model network layer number of the first target network layer in the neural network model; selecting a target network layer number less than or equal to the device accommodating layer number and divisible by the model network layer number; generating, according to the first calculation subgraph and the target network layer number, a shared engine file corresponding to the first target network layer and weight parameters of each first target network layer.

3. The method of claim 1, wherein, According to the execution order of each calculation subgraph, the shared engine file, the weight parameters of each first target network layer, and the independent engine file are used to perform inference calculation to obtain an inference result, including: According to the execution order of each calculation subgraph, a previous inference result is obtained, and the shared engine file or the independent engine file is selected; If the shared engine file is selected, a current network layer corresponding to a current inference calculation is determined in each first target network layer, and a current weight parameter of the current network layer is loaded; According to the current weight parameter and the shared engine file, a current engine file of the current network layer is generated; According to the current engine file and the previous inference result, inference calculation is performed to obtain a current inference result; If the independent engine file is selected, according to the independent engine file and the previous inference result, inference calculation is performed to obtain a current inference result.

4. The method of claim 3, wherein, According to the current weight parameter and the shared engine file, a current engine file of the current network layer is generated, including: According to the current weight parameter and the shared engine file, the current engine file of the current network layer is generated by weight refitting.

5. The method according to any one of claims 1 to 4, characterized in that, The neural network model includes a transformer model; The first target network layer includes a decoding layer, and the second target network layer includes an embedding layer and an output layer composed of layer normalization and a fully connected layer.

6. A neural network model based inference apparatus, characterized by comprising: including: The structure analysis module is configured to analyze the structure of the neural network model, determine the first target network layer with the same structure and the second target network layer with different structures in the neural network model; The network layer splitting module is configured to split the network layer of the neural network model according to the first target network layer and the second target network layer to obtain a first calculation subgraph and a second calculation subgraph corresponding to the first target network layer and the second target network layer, respectively; The shared engine file generation module is configured to generate a shared engine file corresponding to the first target network layer and weight parameters of each first target network layer according to the first calculation subgraph; The independent engine file generation module is configured to generate an independent engine file corresponding to the second target network layer according to the second calculation subgraph; wherein the independent engine file includes weight parameters of the second target network layer; The inference result determination module is configured to perform inference calculation according to the execution order of each calculation subgraph, the shared engine file, the weight parameters of each first target network layer, and the independent engine file to obtain an inference result. The shared engine file generation module comprises: a storage capacity determination unit configured to estimate static storage capacity and dynamic storage capacity required by a single first target network layer in neural network model inference; a device accommodation layer determination unit configured to determine a device accommodation layer number of the computing device capable of accommodating the first target network layer according to device storage capacity of the computing device, the static storage capacity and the dynamic storage capacity; and a shared engine file generation unit configured to generate a shared engine file corresponding to the first target network layer according to the first computing subgraph and the device accommodation layer number, and weight parameters of each first target network layer. The computing device can be virtually formed into at least two virtual devices. The inference result determination module comprises: A network layer determination unit configured to determine a current network layer and a next network layer corresponding to a current inference calculation and a next inference calculation in each first target network layer if the shared engine file is selected in the current inference calculation and the next inference calculation. A further current engine file generation unit configured to load current weight parameters of the current network layer through the first virtual device, and generate a current engine file of the current network layer according to the current weight parameters and the shared engine file. A next engine file generation unit configured to load next weight parameters of the next network layer through the second virtual device while the first virtual device performs inference calculation according to the current engine file and a previous inference result to obtain a current inference result, and generate a next engine file of the next network layer according to the next weight parameters and the shared engine file. A next inference result determination unit configured to obtain the current inference result through the second virtual device, and perform inference calculation according to the current inference result and the next engine file to obtain a next inference result.

7. An electronic device, comprising: The electronic device comprises: At least one processor; and A memory connected in communication with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the neural network model-based inference method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to implement the neural network model-based inference method in any one of claims 1-5 when executed.

Images