Model reasoning method and device
By preloading the model information of the preset model in the inference engine and determining the required model information based on the model identification, the problem of large time overhead of model inference in the prior art is solved, and a more efficient inference process is achieved.
Patent Information
- Application Number
- CN202311433179.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-30
- Publication Date
- 2025-05-02
AI Technical Summary
The existing inference engines need to frequently load and deserialize the trained neural network model during model inference, resulting in an increase in time overhead. How to reduce the time overhead of model inference has become a problem.
By preloading the model information of the preset model in the inference engine and determining the required model information from the preset model based on the model identification, directly performing inference is avoided, the step of frequently obtaining model information from the loading engine is avoided.
This method can significantly improve the speed of model inference, reduce time overhead, and make the inference process more efficient.
Smart Images
Figure CN119918650A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model reasoning method and device. Background Art
[0002] With the rapid development of computer technology, the application scope of deep learning is becoming wider and wider. Generally speaking, after building a neural network model, users can train the neural network model and use the trained neural network model for inference calls.
[0003] When an existing inference engine calls a trained neural network model for inference, the inference engine first loads the trained neural network model and deserializes the trained neural network model to obtain the information of each operator in the neural network model, and then infers the inference data based on the information of each operator to obtain the inference result.
[0004] Since the existing inference engine needs to obtain the deserialization results corresponding to the trained neural network model from the loading engine each time it performs an inference task, this will increase the time overhead brought by model inference. How to reduce the time overhead brought by model inference is the problem that needs to be solved at present. Summary of the invention
[0005] The present application provides a model reasoning method and device for reducing the time overhead caused by model reasoning.
[0006] In the first aspect, an embodiment of the present application provides a model reasoning method, which can be executed by a reasoning engine of a computer device. The method specifically includes the following steps: the reasoning engine receives the identifier of the first model and the data to be reasoned sent by the loading engine, and determines the model information of the first model from the model information of at least one preset model according to the identifier of the first model, wherein the model information of the first model includes the operator descriptor and weight corresponding to at least one operator in the first model, and the model information of at least one preset model is pre-loaded into the reasoning engine by the loading engine, and then, the reasoning engine reasons on the data to be reasoned based on the operator descriptor and weight corresponding to at least one operator in the first model to obtain the reasoning result.
[0007] Through the above method, when performing an inference task, the inference engine can directly determine the model information of the first model from the model information of at least one pre-stored preset model, and infer the inference data based on the model information of the first model, without obtaining the model information of the first model from the loading engine, thereby improving the model inference speed.
[0008] In a possible design, the inference engine determines the model information of the first model from the model information of at least one preset model according to the identifier of the first model, which may include: the inference engine determines the model descriptor corresponding to the identifier of the first model according to the first mapping relationship, wherein the first mapping relationship includes a mapping relationship between the identifier of at least one preset model and the model descriptor of at least one preset model, and the first mapping relationship is pre-loaded into the inference engine by the loading engine, and then the inference engine determines the model information of the first model from the model information of at least one preset model according to the model descriptor of the first model. Through this design, a method for determining the model information of the first model is provided.
[0009] In a possible design, the model descriptor of the first model includes the storage address of the operator descriptor corresponding to the first operator in the first model, and the storage address of the weight corresponding to the first operator, and the first operator in the first model is the first operator to be executed in the first model; the inference engine determines the model information of the first model from the model information of at least one preset model according to the model descriptor of the first model, which may include: taking the storage address of the operator descriptor corresponding to the first operator as the starting address, the inference engine sequentially obtains the operator descriptor corresponding to at least one operator in the first model; taking the storage address of the weight corresponding to the first operator as the starting address, the inference engine sequentially obtains the weight corresponding to at least one operator in the first model. Through this design, a method for determining the operator descriptor and weight corresponding to at least one operator is provided.
[0010] In one possible design, the inference engine receives the identifier of the first model and the data to be inferred sent by the loading engine, which may include: the inference engine receives the identifier of the first model and the first storage address sent by the loading engine, wherein the first storage address is used to store the data to be inferred.
[0011] In a possible design, the method further includes: the inference engine receiving a second storage address sent by the loading engine, wherein the second storage address is used to store the inference result.
[0012] In a second aspect, an embodiment of the present application provides a model reasoning method, which can be executed by a loading engine of a computer device. The method specifically includes the following steps: the loading engine receives the identifier of the first model and the data to be reasoned input by the client, and sends the identifier of the first model and the data to be reasoned to the reasoning engine.
[0013] Through the above method, the loading engine sends the identifier of the first model and the data to be inferred to the inference engine, so as to facilitate the inference engine to perform subsequent model inference.
[0014] In a possible design, after the loading engine receives the identifier of the first model and the data to be inferred input by the client, it may also include: the loading engine determines the first storage address and the second storage address based on the data to be inferred, wherein the first storage address is used to store the data to be inferred, and the second storage address is used to store the inference result corresponding to the data to be inferred; the loading engine sends the identifier of the first model and the data to be inferred to the inference engine, which may include: the loading engine sends the identifier of the first model, the first storage address, and the second storage address to the inference engine.
[0015] In a third aspect, an embodiment of the present application provides a model reasoning method, which can be executed by a loading engine of a computer device. The method specifically includes the following steps: the loading engine receives a model file of a preset model input by a client, and generates an identifier corresponding to the preset model, the loading engine obtains the model information of the preset model and the model descriptor of the preset model from the model file of the preset model, wherein the model information of the preset model includes an operator descriptor and a weight corresponding to at least one operator in the preset model, the loading engine generates a first mapping relationship based on the identifier and model descriptor of the preset model, and finally, the loading engine sends the first mapping relationship, the model information and model descriptor of the preset model to the reasoning engine for storage, and the loading engine outputs the identifier of the preset model to the client.
[0016] Through the above method, the loading engine can send the first mapping relationship, the model information of the preset model, and the model descriptor of the preset model to the inference engine and store them, so that when the subsequent inference engine performs the inference task, it can directly obtain the above content from the inference engine instead of obtaining the above content from the loading engine, which is conducive to reducing the time required for model inference.
[0017] In one possible design, the model descriptor of the preset model includes the storage address of the operator descriptor corresponding to the first operator in the preset model and the storage address of the weight corresponding to the first operator, and the first operator in the preset model is the first operator executed in the preset model.
[0018] In a possible design, the model file of the preset model is in an om model format.
[0019] In a fourth aspect, an embodiment of the present application provides a model reasoning method, which can be interactively executed by a loading engine and a reasoning engine of a computer device. The method specifically includes the following steps: the loading engine receives the identifier of the first model and the data to be reasoned input by the client, the loading engine sends the identifier of the first model and the data to be reasoned to the reasoning engine, and the reasoning engine determines the model information of the first model from the model information of at least one preset model according to the identifier of the first model, wherein the model information of the first model includes the operator descriptor and weight corresponding to at least one operator in the first model, and the model information of at least one preset model is pre-loaded into the reasoning engine by the loading engine, and the reasoning engine reasons on the data to be reasoned based on the operator descriptor and weight corresponding to at least one operator in the first model to obtain the reasoning result.
[0020] Through the above method, when performing an inference task, the inference engine can directly determine the model information of the first model from the model information of at least one pre-stored preset model, and infer the inference data based on the model information of the first model, without obtaining the model information of the first model from the loading engine, thereby improving the model inference speed.
[0021] In a fifth aspect, an embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the memory is used to store programs and data, and the processor is used to run the programs stored in the memory, and execute the method provided in any one of the first to fourth aspects above according to the data stored in the memory.
[0022] In a sixth aspect, an embodiment of the present application further provides a computer storage medium, in which a computer program is stored. When the computer program is executed by a computer, the computer executes the method provided in any one of the first to fourth aspects above.
[0023] In the seventh aspect, an embodiment of the present application further provides a chip, which is used to read a computer program stored in a memory to implement the method provided in any one of the first to fourth aspects above.
[0024] In an eighth aspect, an embodiment of the present application provides a chip system, which includes a processor for supporting a computer device to implement the functions involved in any one of the first to fourth aspects above. In a possible design, the chip system also includes a memory, which is used to store the necessary programs and data for the computer device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1A schematic diagram of a neural network provided in an embodiment of the present application;
[0026] Figure 2 A flowchart of a model reasoning method provided in an embodiment of the present application;
[0027] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;
[0028] Figure 4 A flowchart of a model reasoning method provided in an embodiment of the present application;
[0029] Figure 5 A schematic diagram of the content structure stored in an inference engine provided in an embodiment of the present application;
[0030] Figure 6 A flowchart of a model reasoning method provided in an embodiment of the present application;
[0031] Figure 7 A flowchart of a method for determining model information of a first model provided in an embodiment of the present application;
[0032] Figure 8 A schematic diagram of the structure of a model reasoning device provided in an embodiment of the present application;
[0033] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and advantages of the present application more clear, the present application will be further described in detail below with reference to the accompanying drawings. The specific operation methods, functional descriptions, etc. in the method embodiments can also be applied to the device embodiments or system embodiments.
[0035] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.
[0036] A neural network processing unit (NPU), an inference engine, can support neural network models such as onnx model format and om model format to run on the NPU.
[0037] om model format, an inference architecture model format, deployed on AI servers or other inference hardware based on AI chips.
[0038] The following is a brief introduction to neural networks:
[0039] Neural networks imitate the behavioral characteristics of animal neural networks and process data in a structure similar to the synaptic connections in the brain. As a mathematical operation model, neural networks are composed of a large number of nodes (or neurons) that are connected to each other.
[0040] The nodes in a neural network are Figure 1 As shown in Figure (a), a1~an on the left side of the circle in the figure are the input data of the node, w1~wn are the weights of the node, and b is the bias of the node. The circle in the figure represents the internal calculation of the node, that is, a specific output function f, also called the activation function. Each node can have multiple output data z.
[0041] like Figure 1 As shown in Figure (b), the neural network consists of an input layer, a hidden layer, and an output layer. The input layer is the input data of the neural network; the output layer is the output data of the neural network; and the hidden layer is composed of many nodes connected between the input layer and the output layer, which is used to perform operations on the input data. The hidden layer can be composed of one or more layers. The number of layers and nodes of the hidden layer in the neural network is directly related to the complexity of the problem actually solved by the neural network, the number of nodes in the input layer, and the number of nodes in the output layer.
[0042] Generally speaking, a neural network model can be used to reason about inference data through TensorRT, where TensorRT includes a loading engine and an inference engine.
[0043] Currently, when reasoning about inference data, Figure 2 As shown, the loading engine in TensorRT obtains the trained neural network model, deserializes the trained neural network model, and obtains the operator information corresponding to the N operators in the neural network model, where N is a positive integer, and the operator information corresponding to the N operators includes operator information 1, ..., operator information N. The loading engine then sends the operator information corresponding to the N operators to the inference engine in TensorRT, and the inference engine infers the inference data according to the operator information corresponding to the N operators to obtain the inference result.
[0044] If the client inputs a trained neural network model and M data to be inferred into TensorRT for inference, where M is an integer greater than 0, the loading engine in TensorRT first deserializes the trained neural network model to obtain the operator information corresponding to the N operators in the neural network model.
[0045] When reasoning for each data to be inferred, the loading engine in TensorRT needs to send the operator information corresponding to each of the N operators to the inference engine, and then the inference engine infers the data to be inferred based on the operator information corresponding to each of the N operators to obtain the inference result. Since there are M data to be inferred, the loading engine needs to send the operator information corresponding to each of the N operators M times to the inference engine, which will increase the inference time. In view of this, an embodiment of the present application provides a model inference method and device.
[0046] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0047] Figure 3 FIG. 1 shows a possible computer device structure diagram applicable to the model reasoning method provided in the embodiment of the present application. Figure 3 As shown, the computer device includes: a processor 310, a memory 320, a communication module 330, an input unit 340, a display unit 350, a power supply 360 and other components, wherein the processor includes a loading engine 3101 and an inference engine 3102. Those skilled in the art can understand that Figure 3 The structure of the computer device shown in the figure does not constitute a limitation on the computer device. The computer device provided in the embodiment of the present application may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0048] Combine the following Figure 3 A detailed introduction to the various components of a computer device:
[0049] The communication module 330 can be connected to other devices by wireless connection or physical connection to realize data transmission and reception of the computer device. Optionally, the communication module 330 can include any one or a combination of radio frequency (RF) circuit, wireless fidelity (WiFi) module, communication interface, Bluetooth module, etc., which is not limited in the embodiment of the present application.
[0050] The memory 320 can be used to store program instructions and data. The processor 310 executes various functional applications and data processing of the computer device by running the program instructions stored in the memory 320. Among them, the program instructions include program instructions that enable the processor 310 to execute the model reasoning method provided in the following embodiments of the present application.
[0051] Optionally, the memory 320 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, various application programs, and program instructions, etc.; the data storage area may store various data such as the first model file. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as a disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0052] The input unit 340 may be used to receive information such as data or operation instructions input by a user. Optionally, the input unit 340 may include input devices such as a touch panel, function keys, a physical keyboard, a mouse, a camera, and a monitor.
[0053] The display unit 350 can realize human-computer interaction and is used to display information input by the user, information provided to the user, etc. through the user interface. The display unit 350 may include a display panel 351. Optionally, the display panel 351 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.
[0054] Furthermore, when the input unit includes a touch panel, the touch panel may cover the display panel 351. When the touch panel detects a touch event on or near it, it transmits the touch event to the processor 310 to determine the type of the touch event and perform a corresponding operation.
[0055] The processor 310 is the control center of the computer device, and uses various interfaces and lines to connect the above components. The processor 310 can complete various functions of the computer device by executing program instructions stored in the memory 320 and calling data stored in the memory 320 to implement the model reasoning method provided in the embodiment of the present application.
[0056] Optionally, the loading engine 3101 in the processor 310 may include a CPU or other hardware device capable of floating-point operations. When the loading engine 3101 of the computer device implements the model reasoning method, the loading engine 3101 may read the model file of the preset model from the storage data area of the memory 320, generate an identifier corresponding to the preset model according to the model file of the preset model, and obtain the model information of the preset model and the model descriptor of the preset model from the model file of the preset model, and then generate a first mapping relationship based on the identifier and model descriptor of the preset model, send the first mapping relationship, the model information and model descriptor of the preset model to the reasoning engine, and send the identifier of the preset model to the client. The loading engine 3101 can also read the identifier of the first model and the data to be inferred from the storage data area of the memory 320, and send the identifier of the first model and the data to be inferred to the reasoning engine 3102.
[0057] The inference engine 3102 in the processor 310 may include a hardware device capable of floating-point operations such as an NPU or a GPU. When the inference engine 3102 of the computer device implements the model inference method, the inference engine 3102 may read the identifier of the first model and the data to be inferred from the storage data area of the memory 320, and infer the data to be inferred according to the identifier of the first model to obtain an inference result.
[0058] The computer device also includes a power supply 360 (such as a battery) for supplying power to various components. Optionally, the power supply 360 can be logically connected to the processor 310 through a power management system, so that the power management system can realize functions such as charging and discharging of the computer device.
[0059] Although not shown, the computer device may also include components such as a camera, a sensor, and an audio collector, which will not be described in detail here.
[0060] Before introducing the model reasoning method in the embodiment of the present application, the neural network operator in the neural network model is first introduced, where the neural network operator can also be simply referred to as an operator.
[0061] Below we take the traditional single shot multibox detector (SSD) neural network model as an example. The SSD neural network model contains the following operators: Convolution operator, BatchNorm operator, Scale operator, Prior_Box operator, SoftMax operator, Permute operator, Flatten operator, Concat operator, and Eltwise operator.
[0062] First, the main functions of the above operators are introduced:
[0063] Convolution operator: This operator mainly performs convolution operations on the input feature map to obtain feature maps related to the number of convolution kernels. The main calculations of this layer are matrix multiplication of the input feature map and the convolution kernel data, summing the multiplication results, and summing the summation results with the bias of each convolution kernel.
[0064] BatchNorm operator and Scale operator: mainly normalize the feature map and then perform a linear transformation of the trainable value.
[0065] Prior_Box operator: Prior_Box operator is mainly used for the generation of default box. In the specific calculation, first obtain the parameters corresponding to the operator, including two different sizes (min_size in the parameters) and aspect_ratio, etc. Before calculation, it is also necessary to obtain the compression ratio of the feature map of the layer where the operator is located with the size of the original input image. For each pixel point on the feature map, calculate its mapping to the original image (obtain the center point), and then use the obtained min_size and aspect_ratio to calculate each default box (xmin, ymin, xmax, ymax). After the traversal is completed, process the boundary points to ensure that the generated default boxes fall within the image. The main calculations of this layer are floating-point multiplication and square root.
[0066] SoftMax operator: The main function of this operator is to normalize the probability values of multiple classes of labels so that the sum of all probabilities is 1. The main calculation of this operator is to calculate the power of all inputs with e as the base, sum all power values, and divide the power by the sum of all powers.
[0067] Permute operator: The main function of this operator is to rearrange the data. For example, for an input feature map of N×C×H×W, this layer can rearrange the data into a feature map of the format of N×H×W×C. The value of the data is not changed in this process, that is, the operator only rearranges the data to facilitate the processing and understanding of the data by subsequent operators.
[0068] Flatten operator: The Flatten operator is similar to the Permute operator and also rearranges the data to a certain extent. The main function of the Flatten operator is to convert the feature map in the format of N×H×W×C after the Permute operator processing into a feature map of N×(H×W×C)×1×1, that is, it does not change the storage format and order of the data, but only changes the dimension of the feature map to which the data belongs (does not change the data, only changes the pointer). The significance of this operator is that different operators in the neural network have predictions of different resolutions. After the calculation of the Flatten operator, the predictions of these different layers can be easily combined.
[0069] Concat operator: The main function of the Concat operator is to concatenate multiple input feature maps into a large feature map. This operator can combine the prediction information of objects of different resolutions in the neural network to obtain the overall prediction information.
[0070] Eltwise operator: The main function of this operator is to calculate the sum of two feature maps of the same dimension. The function completed is the combination of two different feature maps, that is, the calculation of F(x)+x.
[0071] The embodiment of the present application provides a model reasoning method, which can be divided into two stages, namely, the model loading stage and the model scheduling stage. In the model loading stage, Figure 3 In the model scheduling stage, the loading engine in the computer device shown in FIG. Figure 3 The loading engine and the reasoning engine in the computer device shown are executed interactively. The execution process of the two different stages is described in detail below.
[0072] During the model loading phase, see Figure 4 As shown in Figure 2, the process of the model reasoning method includes:
[0073] S401, the loading engine receives a model file of a preset model input by a client, and generates an identifier corresponding to the preset model.
[0074] In an embodiment of the present application, the preset model may be a trained neural network model, wherein the trained neural network model may be applied to any scenario such as image processing, natural language processing, and speech recognition.
[0075] The model file of the preset model can be in om model format or non-om model format. When the model file of the preset model is in om model format, the loading engine can directly receive the model file of the preset model without formatting the preset model.
[0076] In the case where the model file of the preset model is not in the om model format, such as the model file of the preset model is in the onnx model format, the loading engine converts the model file of the preset model from the onnx model format to the om model format.
[0077] When the loading engine determines that the model file of the preset model is in the om model format, the loading engine can generate an identifier corresponding to the preset model. The loading engine can use a randomly generated sequence as the identifier corresponding to the preset model, or can encode the model information in the preset model to obtain the identifier corresponding to the preset model, which is not limited here. It should be understood that the loading engine generates a unique identifier for each preset model, and different preset models have different corresponding identifiers.
[0078] The loading engine may receive a single model file of a preset model input by a client, or may receive a plurality of model files of different preset models input by a client, which is not limited here.
[0079] S402, the loading engine obtains model information of the preset model and a model descriptor of the preset model from the model file of the preset model.
[0080] In an embodiment of the present application, the model information of the preset model includes an operator descriptor and a weight corresponding to at least one operator in the preset model.
[0081] It should be understood that the operator descriptor can also be called a task descriptor, and the operator descriptor of each operator can include the executable code corresponding to each operator. The operator descriptor of each operator can also include the concurrency corresponding to each operator, that is, the number of cores in the inference engine required when each operator is executed.
[0082] The model descriptor of the preset model includes the storage address of the operator descriptor corresponding to the first operator in the preset model and the storage address of the weight corresponding to the first operator. The first operator in the preset model is the first operator executed in the preset model.
[0083] S403: The loading engine generates a first mapping relationship based on the identifier of the preset model and the model descriptor.
[0084] In the embodiment of the present application, the loading engine matches the identifier of the preset model with the model descriptor to generate a first mapping relationship.
[0085] S404, the loading engine sends the first mapping relationship, the model information of the preset model, and the model descriptor to the inference engine for storage.
[0086] In the embodiment of the present application, the operator descriptors corresponding to at least one operator in the model information of the preset model are stored in the inference engine in sequence according to the order in which each operator is executed. Similarly, the weights corresponding to at least one operator in the model information of the preset model are stored in the inference engine in sequence according to the order in which each operator is executed.
[0087] For example, the preset model is ResNet50, which includes N operators. The loading engine determines that the identifier corresponding to ResNet50 is ID1, and the first operator in ResNet50 is Convolution operator 1. The content stored in the inference engine is as follows: Figure 5 As shown, the inference engine can store a first mapping relationship, a model descriptor of ResNet50, operator descriptors corresponding to N operators in ResNet50, and weights corresponding to N operators in ResNet50, wherein the first mapping relationship includes ID1 and the model descriptor of ResNet50. Since Convolution operator 1 is the first operator in ResNet50, the operator descriptor corresponding to Convolution operator 1 is operator descriptor 1, and the weight corresponding to Convolution operator 1 is weight 1, the model descriptor of ResNet50 includes the storage address of operator descriptor 1 and the storage address of weight 1. The operator descriptors corresponding to N operators in ResNet50 include operator descriptor 1, ..., operator descriptor N, and the weights corresponding to N operators in ResNet50 include weight 1..., weight N.
[0088] S405, the loading engine outputs the identifier of the preset model to the client.
[0089] In the above method, the loading engine can send the first mapping relationship, the model information of the preset model, and the model descriptor of the preset model to the inference engine and store them, so that when the subsequent inference engine performs the inference task, it can directly obtain the above content from the inference engine instead of obtaining the above content from the loading engine, which is conducive to reducing the time required for model inference.
[0090] During the model scheduling phase, see Figure 6 As shown in Figure 2, the process of the model reasoning method includes:
[0091] S601, the loading engine receives the identifier of the first model and the data to be inferred input by the client.
[0092] In the embodiment of the present application, the data to be inferred may be any data such as image, text, voice, etc.
[0093] S602: The loading engine sends the identifier of the first model and the data to be inferred to the inference engine. Correspondingly, the inference engine receives the identifier of the first model and the data to be inferred sent by the loading engine.
[0094] In an embodiment of the present application, the loading engine can generate an execution descriptor of the first model based on the identifier of the first model and the data to be inferred, and send the execution descriptor of the first model to the inference engine.
[0095] The execution descriptor of the first model may include the following two possible generation methods.
[0096] A first possible generation method is to use the identifier of the first model and the data to be inferred as the execution descriptor of the first model.
[0097] In a second possible generation method, the loading engine stores the data to be inferred, obtains a first storage address, and uses the identifier of the first model and the first storage address as the execution descriptor of the first model, wherein the first storage address is used to store the data to be inferred, and the first storage address may be a base address for storing the data to be inferred. The first storage address may be located in the loading engine or in the inference engine, which is not limited here.
[0098] Optionally, the loading engine can also allocate storage space for the inference result of the data to be inferred, so the execution descriptor of the first model can also include a second storage address, the second storage address is used to store the inference result, and the second storage address can be a base address for storing the inference result. The second storage address can be located in the loading engine or in the inference engine, which is not limited here.
[0099] The loading engine can also allocate storage space for the intermediate results of the data to be inferred, so the execution descriptor of the first model can also include a third storage address, which is used to store the intermediate results of the data to be inferred. The third storage address can be the base address for storing the intermediate results of the inference data, where the intermediate results of the data to be inferred can be understood as the results of each operator processing the data to be inferred. The third storage address can be located in the loading engine or in the inference engine, which is not limited here.
[0100] In a possible implementation, in order to facilitate the management of execution descriptors of different models, the loading engine may add the execution descriptor of the first model to the task queue of the inference engine, and the inference engine obtains the execution descriptor of the first model from the task queue.
[0101] S603: The inference engine determines the model information of the first model from the model information of at least one preset model according to the identifier of the first model.
[0102] In the embodiment of the present application, determining the model information of the first model may include: Figure 7 The following steps are shown:
[0103] S701: The inference engine determines, according to the first mapping relationship, a model descriptor corresponding to the identifier of the first model.
[0104] In an embodiment of the present application, the first mapping relationship includes a mapping relationship between an identifier of at least one preset model and a model descriptor of at least one preset model, and the first mapping relationship is pre-loaded into the inference engine by the loading engine.
[0105] The inference engine determines, from the first mapping relationship, an identifier of a preset model that is identical to an identifier of the first model as the identifier of the target preset model, and uses a model descriptor corresponding to the identifier of the target preset model as the model descriptor of the first model.
[0106] In an embodiment of the present application, at least one operator in the first model includes a first operator, the first operator in the first model is the first operator executed in the first model, and the model descriptor of the first model may include the storage address of the operator descriptor corresponding to the first operator in the first model, and the storage address of the weight corresponding to the first operator.
[0107] S702: The inference engine determines model information of the first model from model information of at least one preset model according to the model descriptor of the first model.
[0108] In the embodiment of the present application, the model information of at least one preset model is pre-loaded into the inference engine by the loading engine, and the method for loading the model information of the preset model is as follows: Figure 4 S401-S405 have been explained and will not be repeated here.
[0109] The model information of the first model includes operator descriptors and weights corresponding to at least one operator in the first model. The operator descriptors corresponding to at least one operator in the first model are determined in the following manner: the inference engine uses the storage address of the operator descriptor corresponding to the first operator as the starting address, and sequentially obtains the operator descriptors corresponding to at least one operator in the first model.
[0110] The weights corresponding to at least one operator in the first model are determined in the following manner: the inference engine uses the storage address of the weight corresponding to the first operator as the starting address, and sequentially obtains the weights corresponding to at least one operator in the first model.
[0111] S604: The inference engine performs inference on the inference data based on the operator descriptors and weights respectively corresponding to at least one operator in the first model to obtain an inference result.
[0112] In the embodiment of the present application, the inference engine obtains the operator descriptor and weight corresponding to each operator in the order in which the operators in the first model are executed, performs inference on the inference data, and obtains the inference result.
[0113] In a possible implementation, the first model includes N operators, the i-th operator is the i-th executed operator in the first model, and i is an integer ranging from [1, N]. The first operator in the above text is the first executed operator in the first model.
[0114] The inference engine obtains the operator descriptor and weight corresponding to the first (i=1) operator in the first model, determines the executable code of the first operator according to the operator descriptor corresponding to the first operator, executes the executable code of the first operator, processes the inference data using the weight of the first operator, and obtains the intermediate result corresponding to the first operator. The intermediate result corresponding to the first operator can be stored in the third storage address.
[0115] The inference engine obtains the operator descriptor and weight corresponding to the i-th (i is an integer ranging from [2, N]) operator in the first model, determines the executable code of the i-th operator according to the operator descriptor corresponding to the i-th operator, executes the executable code of the i-th operator, processes the intermediate result corresponding to the i-1-th operator using the weight of the i-th operator, and obtains the intermediate result corresponding to the i-th operator. The intermediate result corresponding to the i-th operator can be stored in the third storage address.
[0116] Finally, the intermediate result corresponding to the Nth (i=N) operator is taken as the inference result.
[0117] In the above method, when executing an inference task, the inference engine can directly determine the model information of the first model from the model information of at least one preset model stored in advance, and infer the inference data based on the model information of the first model, without obtaining the model information of the first model from the loading engine, thereby improving the model inference speed.
[0118] Based on the above embodiments, the present application also provides a model reasoning device 800, which is applied to Figure 2 The processor of the computer device shown is used to implement Figure 4 or Figure 6 The embodiment shown provides a model reasoning method. Figure 8 As shown, the device includes: a transceiver module 801 and a processing module 802.
[0119] In one implementation, the model reasoning device 800 may perform the following functions:
[0120] The transceiver module 801 is used to receive the identifier of the first model and the data to be inferred sent by the loading engine.
[0121] Processing module 802 is used to determine the model information of the first model from the model information of at least one preset model according to the identifier of the first model, wherein the model information of the first model includes operator descriptors and weights corresponding to at least one operator in the first model, and the model information of at least one preset model is pre-loaded into the inference engine by the loading engine.
[0122] The processing module 802 is further used to perform reasoning on the inference data based on the operator descriptors and weights respectively corresponding to at least one operator in the first model to obtain a reasoning result.
[0123] In one possible design, the processing module 802 is specifically used to: determine the model descriptor corresponding to the identifier of the first model based on a first mapping relationship, wherein the first mapping relationship includes a mapping relationship between the identifier of at least one preset model and the model descriptor of at least one preset model, and the first mapping relationship is pre-loaded into the inference engine by the loading engine; determine the model information of the first model from the model information of at least one preset model based on the model descriptor of the first model.
[0124] In one possible design, the model descriptor of the first model includes the storage address of the operator descriptor corresponding to the first operator in the first model, and the storage address of the weight corresponding to the first operator, wherein the first operator in the first model is the first operator executed in the first model; the processing module 802 is specifically used to: use the storage address of the operator descriptor corresponding to the first operator as the starting address, and sequentially obtain the operator descriptors corresponding to at least one operator in the first model; use the storage address of the weight corresponding to the first operator as the starting address, and sequentially obtain the weights corresponding to at least one operator in the first model.
[0125] In a possible design, the transceiver module 801 is specifically used to: receive an identifier of a first model and a first storage address sent by a loading engine, wherein the first storage address is used to store data to be inferred.
[0126] In one possible design, the transceiver module 801 is specifically used to: receive a second storage address sent by the loading engine, wherein the second storage address is used to store the inference result.
[0127] In one implementation, the model reasoning device 800 may perform the following functions:
[0128] The transceiver module 801 is used to receive the identifier of the first model and the data to be inferred input by the client.
[0129] The transceiver module 801 is also used to send the identifier of the first model and the data to be inferred to the inference engine.
[0130] In one possible design, after receiving the identifier of the first model and the data to be inferred input by the client, the processing module 802 is specifically used to: determine the first storage address and the second storage address according to the data to be inferred, wherein the first storage address is used to store the data to be inferred, and the second storage address is used to store the inference result corresponding to the data to be inferred; send the identifier of the first model, the first storage address and the second storage address to the inference engine.
[0131] In one implementation, the model reasoning device 800 may perform the following functions:
[0132] The transceiver module 801 is used to receive a model file of a preset model input by a client, and generate an identifier corresponding to the preset model.
[0133] The processing module 802 is used to obtain model information of the preset model and the model descriptor of the preset model from the model file of the preset model, wherein the model information of the preset model includes an operator descriptor and a weight corresponding to at least one operator in the preset model.
[0134] The processing module 802 is used to generate a first mapping relationship based on the identifier of the preset model and the model descriptor.
[0135] The processing module 802 is used to send the first mapping relationship, the model information of the preset model and the model descriptor to the inference engine for storage.
[0136] The processing module 802 is used to output the identifier of the preset model to the client.
[0137] In one possible design, the model descriptor of the preset model includes the storage address of the operator descriptor corresponding to the first operator in the preset model and the storage address of the weight corresponding to the first operator, wherein the first operator in the preset model is the first operator executed in the preset model.
[0138] In a possible design, the model file of the preset model is in the om model format.
[0139] In one implementation, the model reasoning device 800 may perform the following functions:
[0140] The transceiver module 801 is used to receive the identifier of the first model and the data to be inferred input by the client.
[0141] The transceiver module 801 is also used to send the identifier of the first model and the data to be inferred to the inference engine.
[0142] Processing module 802 is used to determine the model information of the first model from the model information of at least one preset model according to the identifier of the first model, wherein the model information of the first model includes operator descriptors and weights corresponding to at least one operator in the first model, and the model information of at least one preset model is pre-loaded into the inference engine by the loading engine.
[0143] The processing module 802 is used to perform reasoning on the inference data based on the operator descriptors and weights respectively corresponding to at least one operator in the first model to obtain a reasoning result.
[0144] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0146] Based on the above embodiments, the present application also provides a computer device, which is used to implement the following Figure 4 or Figure 6 The model inference method shown has the following Figure 8 The model inference device functions as shown. Fig. 9 As shown, the computer device includes: a processor 901 and a memory 902.
[0147] The processor 901 and the memory 902 are connected to each other. Optionally, the processor 901 and the memory 902 can be connected to each other via a bus 903; the bus 903 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0148] Optionally, the computer device 900 further includes a communication module 904 for communicating and interacting with other devices. Optionally, the communication module 904 can communicate with other devices via wireless connection, for example, the communication module 904 can be an RF circuit, a WiFi module, etc. The communication module 904 can also communicate with other devices via physical connection, for example, the communication module 904 can be a communication interface.
[0149] The processor 901 is used to implement Figure 4 or Figure 6 The specific process of the model reasoning method shown can refer to the specific description in the above embodiment and will not be repeated here.
[0150] The memory 902 is used to store programs and data. Specifically, the program may include program code, which includes instructions for computer operations. The memory 902 may include random access memory (RAM), and may also include non-volatile memory (non-volatile memory), such as at least one disk storage. The processor 901 executes the program stored in the memory 902 to implement the above functions, thereby achieving the following: Figure 4 or Figure 6 The model inference method shown.
[0151] The present application also provides a computer storage medium, wherein a computer program is stored in the computer storage medium, and when the computer program is executed by a computer, the computer executes Figure 4 or Figure 6 The model inference method shown.
[0152] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0153] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0154] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0156] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A model reasoning method, characterized in that: The method is applied to an inference engine, and the method comprises: Receiving the identifier of the first model and the data to be inferred sent by the loading engine; According to the identifier of the first model, determining the model information of the first model from the model information of at least one preset model, the model information of the first model including an operator descriptor and a weight respectively corresponding to at least one operator in the first model; the model information of the at least one preset model is pre-loaded into the inference engine by the loading engine; Based on the operator descriptors and weights respectively corresponding to at least one operator in the first model, the data to be inferred is inferred to obtain an inference result.
2. The method according to claim 1, characterized in that The determining, according to the identifier of the first model, model information of the first model from model information of at least one preset model includes: Determine, according to a first mapping relationship, a model descriptor corresponding to the identifier of the first model, wherein the first mapping relationship includes a mapping relationship between the identifier of the at least one preset model and the model descriptor of the at least one preset model; the first mapping relationship is pre-loaded into the inference engine by the loading engine; According to the model descriptor of the first model, the model information of the first model is determined from the model information of the at least one preset model.
3. The method according to claim 2, characterized in that The model descriptor of the first model includes a storage address of an operator descriptor corresponding to a first operator in the first model and a storage address of a weight corresponding to the first operator; the first operator in the first model is the first operator executed in the first model; The determining, according to the model descriptor of the first model, the model information of the first model from the model information of the at least one preset model comprises: Taking the storage address of the operator descriptor corresponding to the first operator as the starting address, sequentially obtaining the operator descriptors corresponding to at least one operator in the first model; Taking the storage address of the weight corresponding to the first operator as the starting address, the weights corresponding to at least one operator in the first model are obtained in sequence.
4. The method according to any one of claims 1 to 3, characterized in that: The receiving the identifier of the first model and the data to be inferred sent by the loading engine includes: An identifier of a first model and a first storage address sent by the loading engine are received, where the first storage address is used to store the data to be inferred.
5. The method according to any one of claims 1 to 4, characterized in that: Also includes: A second storage address sent by the loading engine is received, where the second storage address is used to store the inference result.
6. A model reasoning method, characterized in that: The method is applied to a loading engine, and the method comprises: Receive an identifier of a first model and data to be inferred input by a client; The identifier of the first model and the data to be inferred are sent to an inference engine.
7. The method according to claim 6, characterized in that After receiving the identifier of the first model and the data to be inferred input by the client, the method further includes: Determine a first storage address and a second storage address according to the data to be inferred, the first storage address is used to store the data to be inferred, and the second storage address is used to store the inference result corresponding to the data to be inferred; The step of sending the identifier of the first model and the data to be inferred to the inference engine includes: An identifier of the first model, the first storage address, and the second storage address are sent to the inference engine.
8. A model reasoning method, characterized in that: The method is applied to a loading engine, and the method comprises: Receive a model file of a preset model input by a client, and generate an identifier corresponding to the preset model; Acquire model information of the preset model and a model descriptor of the preset model from a model file of the preset model, wherein the model information of the preset model includes an operator descriptor and a weight corresponding to at least one operator in the preset model; Generate a first mapping relationship based on the identifier of the preset model and the model descriptor; Sending the first mapping relationship, the model information of the preset model and the model descriptor to the inference engine for storage; The identifier of the preset model is output to the client.
9. The method according to claim 8, characterized in that The model descriptor of the preset model includes the storage address of the operator descriptor corresponding to the first operator in the preset model and the storage address of the weight corresponding to the first operator; the first operator in the preset model is the first operator executed in the preset model.
10. The method according to claim 8 or 9, characterized in that The model file of the preset model is in om model format.
11. A model reasoning method, characterized in that: The method is applied to a model reasoning system, the reasoning system includes a loading engine and a reasoning engine, and the method includes: The loading engine receives an identifier of a first model and data to be inferred input by a client; The loading engine sends the identifier of the first model and the data to be inferred to the inference engine; The inference engine determines the model information of the first model from the model information of at least one preset model according to the identifier of the first model, wherein the model information of the first model includes an operator descriptor and a weight corresponding to at least one operator in the first model; the model information of the at least one preset model is pre-loaded into the inference engine by the loading engine; The inference engine infers the data to be inferred based on the operator descriptors and weights respectively corresponding to at least one operator in the first model to obtain an inference result.
12. A model reasoning device, characterized in that: The device comprises: A transceiver module, used for receiving the identifier of the first model and the data to be inferred sent by the loading engine; a processing module, configured to determine, according to the identifier of the first model, model information of the first model from model information of at least one preset model, wherein the model information of the first model includes an operator descriptor and a weight respectively corresponding to at least one operator in the first model; the model information of the at least one preset model is pre-loaded into the inference engine by the loading engine; The processing module is further used to infer the data to be inferred based on the operator descriptors and weights respectively corresponding to at least one operator in the first model to obtain an inference result.
13. A model reasoning device, characterized in that: The device comprises: A transceiver module, used for receiving an identifier of a first model and data to be inferred input by a client; The transceiver module is also used to send the identifier of the first model and the data to be inferred to the inference engine.
14. A model reasoning device, characterized in that: The device comprises: The transceiver module is used to receive the model file of the preset model input by the client and generate an identifier corresponding to the preset model; A processing module, used to obtain model information of the preset model and a model descriptor of the preset model from a model file of the preset model, wherein the model information of the preset model includes an operator descriptor and a weight corresponding to at least one operator in the preset model; A processing module, configured to generate a first mapping relationship based on the identifier of the preset model and the model descriptor; A processing module, used for sending the first mapping relationship, the model information of the preset model and the model descriptor to the inference engine for storage; A processing module is used to output the identifier of the preset model to the client.
15. A model reasoning device, characterized in that: The device comprises: A transceiver module, used for receiving an identifier of a first model and data to be inferred input by a client; The transceiver module is further used to send the identifier of the first model and the data to be inferred to the inference engine; a processing module, configured to determine, according to the identifier of the first model, model information of the first model from model information of at least one preset model, wherein the model information of the first model includes an operator descriptor and a weight respectively corresponding to at least one operator in the first model; the model information of the at least one preset model is pre-loaded into the inference engine by the loading engine; The processing module is used to infer the data to be inferred based on the operator descriptors and weights respectively corresponding to at least one operator in the first model to obtain an inference result.
16. A computer device, characterized in that: include: Memory, used to store programs and data; A processor is used to run the program stored in the memory, and execute the method according to any one of claims 1 to 5, the method according to any one of 6 to 7, the method according to any one of 8 to 10, or the method according to 11 according to the data stored in the memory.
17. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a computer, the computer executes the method according to any one of claims 1 to 5, the method according to any one of claims 6 to 7, the method according to any one of claims 8 to 10, or the method according to claim 11.