Method, apparatus, computer device, storage medium and program for invoking inference engine

Abstract

The present application provides a method, apparatus, computer equipment, storage medium, and program for inferencing engine invocation, which relate to the fields of artificial intelligence, smart transportation, assisted driving, cloud technology, and other technologies. By determining first data to be inferred and a target model corresponding to the first data, the target structure of the first data can be an engine structure or a general structure. Based on a preprocessing function corresponding to the target structure, second data after preprocessing the first data is obtained, and a target inferencing engine corresponding to the target model is determined. Based on the second data and the target structure, an inference interface corresponding to the target inferencing engine is called to obtain inference data of the target inferencing engine for the second data. Since the inference interface includes structural parameters for carrying structural identifiers of either a general structure or an engine structure, it can support both engine and general structure data to be inferred, thereby improving the practicality of the inferencing engine invocation process.

Description

Technical Field

[0001] The present application relates to technical fields such as artificial intelligence and cloud computing, and relates to an inference engine calling method, apparatus, computer equipment, storage medium, and program. Background Art

[0002] An inference engine is a component that applies logical rules to infer new information. After inputting data into a model, the inference engine typically needs to reason about that data to produce a final result. Therefore, invoking the inference engine is a crucial step in the process of using the model to output results.

[0003] In related technologies, the inference engine calling process includes: first converting the data to be inferred into data of a general structure, and then calling the inference engine based on the data of the general structure, so as to execute the inference process on the data through the inference engine to obtain the inference result.

[0004] The above-mentioned inference engine calling process is limited to data of a general structure, and each inference engine has its own supported data structure. When a new inference engine is introduced, staff are required to develop conversion code for converting the data structure supported by the new inference engine and the general structure, which results in the poor practicality of the above-mentioned inference engine calling process. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, storage medium, and program for invoking an inference engine, which can solve the problem of poor practicality of the inference engine invocation process in related technologies. The technical solution is as follows:

[0006] In one aspect, a method for invoking an inference engine is provided, the method comprising:

[0007] Determining first data to be inferred and a target model corresponding to the first data, where the target structure of the first data is either an engine structure or a general structure, and the engine structure refers to a data structure that matches any preconfigured inference engine;

[0008] Based on a preprocessing function corresponding to the target structure, obtaining second data after preprocessing the first data, and determining a target inference engine corresponding to the target model;

[0009] Based on the second data and the target structure, calling the inference interface corresponding to the target inference engine to obtain inference data of the target inference engine for the second data;

[0010] The reasoning interface includes structural parameters for carrying any structural identifier of a general structure or an engine structure.

[0011] In another aspect, a device for invoking an inference engine is provided, the device comprising:

[0012] a determination module, configured to determine first data to be inferred and a target model corresponding to the first data, wherein the target structure of the first data is either an engine structure or a general structure, and the engine structure refers to a data structure that matches any preconfigured inference engine;

[0013] an acquisition module, configured to acquire second data obtained by preprocessing the first data based on a preprocessing function corresponding to the target structure, and determine a target inference engine corresponding to the target model;

[0014] A calling module, configured to call an inference interface corresponding to the target inference engine based on the second data and the target structure, to obtain inference data of the target inference engine for the second data;

[0015] The reasoning interface includes structural parameters for carrying any structural identifier of a general structure or an engine structure.

[0016] In one possible implementation, the calling module is also used to replace the structural parameters of the reasoning interface with the structural identifier of the target structure; replace the request parameters of the reasoning interface with the second data; and based on the second data and the structural identifier, execute the reasoning process of the target reasoning engine for the second data to obtain the reasoning data.

[0017] In one possible implementation, the calling module is also used to, when the target structure is a general structure, infer third data through the target inference engine to obtain the inference data, and the third data is data obtained by converting the second data from a general structure to an engine structure; when the target structure is an engine structure, infer the second data through the target inference engine to obtain the inference data.

[0018] In one possible implementation, the apparatus further includes:

[0019] A conversion module is used to convert the inference data into fourth data of a general structure when the target structure is a general structure, and use the fourth data as the inference result of the first data; when the target structure is an engine structure, use the inference data as the inference result of the first data.

[0020] In one possible implementation, the apparatus further includes:

[0021] The acquisition module is further configured to acquire a creation interface of the target inference engine from a registry of the target structure;

[0022] A creation module is used to call the creation interface of the target reasoning engine based on the creation parameters of the target reasoning engine to create a running object of the target reasoning engine.

[0023] In one possible implementation, the apparatus further includes:

[0024] An adding module is used to create a registry of the data structure for each data structure, and to add a creation interface of at least one inference engine in the registry.

[0025] In one possible implementation, the determination module is further configured to, in response to a target request from a client, obtain first data carried in the target request, and determine a target model for executing a target task, wherein the target request is configured to request execution of the target task based on the first data and return an execution result.

[0026] Accordingly, the device further includes:

[0027] The return module is used to obtain the execution result corresponding to the target request based on the inference data and the task processing logic of the target task preconfigured in the target model, and return the execution result to the client.

[0028] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-mentioned inference engine calling method.

[0029] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned inference engine calling method is implemented.

[0030] On the other hand, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned inference engine calling method.

[0031] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0032] The inference engine calling method provided by the embodiment of the present application determines the first data to be inferred and the model identifier corresponding to the first data. The target structure of the first data can be an engine structure or a general structure. The engine structure refers to a data structure that matches any preconfigured inference engine; based on the preprocessing function corresponding to the target structure, the second data after preprocessing the first data is obtained, and the target inference engine corresponding to the model identifier is determined; based on the second data and the target structure, the inference interface corresponding to the target inference engine is called to obtain the inference data of the target inference engine for the second data; wherein, since the inference interface includes a structural parameter for carrying any structural identifier in the general structure or the engine structure; thus, the inference engine calling method of the present application can support both engine structure and general structure data to be inferred. Compared with the related art that only supports general structures, the present application greatly expands the usage scenarios and improves the practicality of the inference engine calling process. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0034] Figure 1 A schematic diagram of an implementation environment for implementing a method for calling an inference engine provided in an embodiment of the present application;

[0035] Figure 2 A flowchart of an inference engine calling method provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of the design principle of an inference engine provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of a reasoning framework provided in an embodiment of the present application;

[0038] Figure 5 A flowchart of an inference engine call provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of the structure of an inference engine calling device provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0042] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0043] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0044] The inference engine calling method provided in this application involves the above-mentioned artificial intelligence technology, such as using the above-mentioned machine learning technology to build an inference engine, and using cloud computing technology to perform structural conversion and preprocessing on the inference data.

[0045] Figure 1 This is a schematic diagram of the implementation environment of an inference engine calling method provided by this application. Figure 1 As shown, the implementation environment includes: a first device 101 and a second device 102. The first device 101 can provide a model calling service to the second device 102.

[0046] The first device 101 may be pre-configured with at least one trained model, such as an object recognition model. The first device 101 may also be configured with associations between the various models and the inference engine. Each model may implement its specific functionality based on the inference process of the corresponding inference engine. For example, the object recognition model may provide object recognition functionality based on the inference process of the inference engine. Based on this association, the first device 101 may provide the specific functionality of each model to the second device 102.

[0047] Embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving. In one possible scenario, the second device 102 may send a target request to the first device 101, requesting that a target task be executed based on first data and that an execution result be returned. Based on the target request, the first device 101 may obtain the first data and determine a target model for executing the target task. Based on the association between preconfigured models and inference engines, the first device 101 may determine the inference engine corresponding to the target model, and based on the inference engine's invocation, implement the target task and return the execution result to the second device 102. Exemplarily, the first device 101 may preprocess the first data based on its data structure to obtain second data; and based on this data structure and the second data, invoke an inference interface to perform inference on the second data through the inference engine, thereby obtaining an execution result corresponding to the target task based on the inferenced data. The first device 101 returns the execution result to the second device 102.

[0048] It should be noted that the first device 101 and the second device 102 can both be any computer device such as a server or a terminal. Figure 1 Only the first device 101 is used as a server and the second device 102 is used as a terminal for example; of course, the first device 102 can also be a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The above-mentioned network may include but is not limited to: wired networks, wireless networks, wherein the wired network includes: local area networks, metropolitan area networks and wide area networks, and the wireless network includes: Bluetooth, Wi-Fi and other networks that realize wireless communication. The terminal can be a smartphone (such as an Android phone, iOS phone, etc.), a tablet computer, a laptop computer, a digital broadcast receiver, a MID (Mobile Internet Device), a PDA (Personal Digital Assistant), a desktop computer, an intelligent voice interaction device, a smart home appliance, an in-vehicle terminal (such as an in-vehicle navigation terminal, an in-vehicle computer, etc.), an aircraft, a smart speaker, a smart watch, etc. The terminal and the server can be directly or indirectly connected via wired or wireless communication, but are not limited to this. The specific connection can also be determined based on the actual application scenario requirements and is not limited here.

[0049] Figure 2This is a flow chart of a method for calling an inference engine provided by an embodiment of the present application. The execution subject of this method can be a computer device, which can be any device that can implement the inference engine calling process, such as a server, a service cluster, etc. Figure 2 As shown, the method includes the following steps.

[0050] Step 201: The computer device determines first data to be inferred and a target model corresponding to the first data.

[0051] The target structure of the first data is any one of an engine structure or a universal structure, and the engine structure refers to a data structure that matches any preconfigured inference engine. Each inference engine can infer data of the engine structure that matches it. The universal structure is a configured universal data structure that does not match any preconfigured inference engine. If the data to be inferred is a universal structure, the data to be inferred of the universal structure can be converted into an engine structure that matches the required inference engine to execute the inference process of the inference engine on the data to be inferred. For example, the universal structure can be a Mulan::Tensor (Mulan, tensor) structure; in addition, an MNN (Mobile Neural Network) inference engine, a tensorRT inference engine, a libtorch inference engine, a TNN inference engine, etc. can also be preconfigured; for example, the engine structure corresponding to the MNN inference engine can be expressed as MNN::Tensor; the engine structure corresponding to the libtorch inference engine can be expressed as at::Tensor.

[0052] The inference engine is used to provide the inference data required for the model to implement a specific function; that is, each model implements the function of the model based on the inference process of the corresponding inference engine; illustratively, the inference data is obtained by executing the inference process of the inference engine on the inference data, and the model further outputs the execution result of the corresponding task based on the inference data; for example, the inference engine can infer the object image data of the engine structure that matches it, and execute the recognition task corresponding to the object recognition model based on the inference data obtained by inference to obtain the object recognition result.

[0053] In one possible implementation, the target model is used to execute the target task based on the first data and return the execution result. The computer device can determine the model to be called based on the client's request so as to call the inference engine corresponding to the model. Step 201 may include: the computer device responds to the client's target request, obtains the first data carried by the target request, and determines the target model for executing the target task. The target request is used to request to execute the target task based on the first data and return the execution result. Exemplarily, the target request carries the first data, and the computer device can extract the first data carried by the target request from the target request and determine the corresponding target model based on the target task. For example, the target request is an object recognition request for requesting to recognize an object image, that is, the target task can be an object recognition task; the execution result is an object recognition result, for example, identifying the location of the object, the outline of the object part, etc.; the first data can be an object image, and the computer device can determine the model identifier of the object recognition model corresponding to the object image.

[0054] Among them, Tensor is the data structure used by model parameters, indicating the input and output data dimensions, data arrangement method, data storage format and other information. In one possible technical implementation, this application can pre-encapsulate a tensor (Mulan::Tensor) with a general structure to facilitate upper-level users. Without having to understand the data structure and usage methods of various engines, they can use various engines to infer models. Among them, the function code corresponding to the Tensor structure is as follows:

[0055] class Tensor{

[0056] public:

[0057] Tensor(DataTypedataType,DataFormatdataFormat,const std::vector<int32_t> dimensions,uint8_t*host);

[0058] std::vector <int>shape()const; / / Get the shape of this Tensor

[0059] std::string toString()const; / / Print Tensor

[0060] int32_t byteSize()const; / / Number of bytes occupied by storage

[0061] int32_t elemSize()const; / / Number of elements stored

[0062] template<typename T>

[0063] T*host()const{

[0064] return (T*) this->_host;

[0065] }

[0066] private:

[0067] DataType_dataType; / / data type

[0068] DataFormat_dataFormat; / / data format

[0069] std::vector<int32_t> _dimensions; / / Dimension information such as: [1,3,224,224]

[0070] uint8_t *_host; / / metadata

[0071] Based on Mulan::Tensor, this application implements common preprocessing functions, such as decoding image data into Mulan::Tensor, modifying the size, copying, and cropping of Mulan::Tensor, etc., to facilitate upper-level users to preprocess the input data.

[0072] Step 202: The computer device obtains second data after preprocessing the first data based on the preprocessing function corresponding to the target structure, and determines the target inference engine corresponding to the target model.

[0073] The computer device can preprocess the first data based on the preprocessing function corresponding to the target structure. When the target structure is a general structure, the computer device uses the preprocessing function corresponding to the general structure to preprocess the first data to obtain second data. When the target structure is an engine structure, the computer device uses the preprocessing function corresponding to any inference engine that matches the engine structure to preprocess the first data to obtain second data. Exemplarily, the preprocessing function can be encapsulated in a preprocessing interface, and the computer device can perform preprocessing operations on the first data by calling the preprocessing interface corresponding to the target structure. For example, for data to be inferred of a general structure, a set of encapsulated preprocessing interfaces can be used to preprocess the data. For data to be inferred of an engine structure, the preprocessing interface of the corresponding inference engine can be used to perform corresponding preprocessing operations. Exemplarily, the preprocessing corresponding to the general structure or engine structure can include but is not limited to preprocessing operations such as rotation, mirroring, cropping, and scaling.

[0074] In one possible implementation, the computer device is preconfigured with an association between a model identifier of a model and an inference engine that provides an inference process for the model. Based on the model identifier of the target model, the computer device can determine the target inference engine corresponding to the model identifier of the target model from the association between the model identifier and the engine identifier of the inference engine. For example, the model identifier can be a model ID, and the engine identifier can be an engine ID. The engine ID corresponding to each model ID can be preconfigured.

[0075] In one possible implementation, after the computer device determines the target inference engine, it may also create an execution object for the target inference engine. This process may include: the computer device obtaining a creation interface for the target inference engine from a registry of the target structure; and the computer device, based on the creation parameters of the target inference engine, calling the creation interface of the target inference engine to create the execution object for the target inference engine. Exemplarily, the computer device is pre-configured with a registry of various data structures, each of which includes a creation interface for an inference engine that provides an inference process for the data of that data structure. Therefore, before the computer device obtains the creation interface for the target inference engine from the registry of the target structure, it may first create a registry of various data structures. This process may include: the computer device creating a registry for each data structure and adding a creation interface for at least one inference engine to the registry. The at least one inference engine is used to provide an inference process for the data of that data structure. Exemplarily, the computer device may configure a registry of various data structures as needed. For example, a registry of a general structure may include a creation interface for at least one inference engine. For example, an engine factory of a general structure may register creation interfaces for all inference engines, indicating that after converting the inferenced data of the general structure into an engine structure that matches any inference engine among all inference engines, any inference engine may be used to execute an inference process on the inferenced data. A registry of a particular engine structure may include a creation interface for providing an inference engine that performs an inference process on that type of engine structure.

[0076] Figure 3 A schematic diagram of the design principle of an inference engine provided in an embodiment of the present application is shown as follows: Figure 3 As shown, an engine template Engine can be pre-defined <t>Each inference engine can inherit from the engine template Engine <t>The engine template predefines the functional codes such as initializing the inference engine, checking whether the engine is loaded successfully, and executing model inference. For example, the MNNEngine class implemented for the MNN inference engine and the TorchEngine class implemented for the pytorch engine can both inherit from the engine template class Engine. <t>, thereby providing the reasoning capabilities of different inference engines. For each data structure, a corresponding registry can be configured, which acts as an engine factory for that data structure. The same inference engine can register its corresponding engine construction class in the registry of the supported data structure, depending on the data structure it supports. Figure 3 The data structures supported by the MNNEngine engine include the MNN engine structure MNN::Tensor and the general structure Mulan::Tensor. Therefore, the creation interface of the MNN inference engine, that is, the Creator object, can be registered to the engine factory of the MNN engine structure (that is, EngineFactory <mnn::tensor>Engine Factory) and the creation interface for creating the inference engine, that is, the Creator object is registered to the engine factory of the general structure Mulan::Tensor (that is, EngineFactory <mulan::tensor>In the engine factory). When reasoning is needed on a certain data structure, the engine factory for that data structure is called to create an interface to run the corresponding inference engine, thereby loading and running the inference engine in memory. By adopting this registration method, the inference engine object is constructed only when it is needed, preventing the resource consumption caused by creating many different engines at device startup. This enables a pluggable design for the inference engine, lazy loading, and compatibility with multiple data structures, including general structures and engine structures, improving the practicality of inference engine calls.

[0077] like Figure 3 As shown, for example, inheriting Engine <t>The engine TorchEngine can call the creation interface EngineCreator based on the following parameter information in the engine <t>, to construct a specific engine, thereby loading the object of the corresponding engine into memory;

[0078] Among them, the template Engine <t>The engine's function code is as follows:

[0079] +Engine(EngineOptions)

[0080] +Init()int / / Initialize the inference engine

[0081] +Check()bool / / Check whether the engine is loaded successfully

[0082] +Inference(EngineReq <t>&req,EngineRsp <t>&rsp)status / / Execute model reasoning;

[0083] For example, the TorchEngine engine parameter information is as follows:

[0084] +engine_type:EngineType / / Selected background inference engine type

[0085] +engine_env:EngineEnv / / Selected inference engine environment cpu / gpu

[0086] -engine_versionst::string / / Selected inference engine version

[0087] -name:std::string / / Model name

[0088] -path:std::string / / Model path

[0089] -thread.num:int32_t / / Number of threads used

[0090] For example, inheriting Engine <t>The engine MNNEngine can call the creation interface EngineCreator based on the following parameter information in the engine <t>, to construct a specific engine, thereby loading the corresponding engine object into memory; among them, the engine MNNEngine parameter information is as follows

[0091] +engine_type:EngineType / / Selected background inference engine type

[0092] +engine_env:EngineEnv / / Selected inference engine environment cpu / gpu

[0093] -engine_version:std::string / / Selected inference engine version

[0094] -name:std::string / / Model name

[0095] -path:std::string / / Model path

[0096] -thread num:int32_t / / Number of threads used

[0097] Step 203: The computer device calls the inference interface corresponding to the target inference engine based on the second data and the target structure to obtain the inference data of the target inference engine for the second data.

[0098] The reasoning interface includes a structure parameter for carrying a structure identifier of either a general structure or an engine structure; wherein the engine structure is a data structure that matches the target reasoning engine. When the target structure is the engine structure, the computer device can directly execute the reasoning process of the reasoning engine on the second data.

[0099] In one possible implementation, the computer device may call the reasoning interface based on the structure identifier and the second data to execute the reasoning process of the target reasoning engine. Step 203 may include: the computer device replaces the structure parameters of the reasoning interface with the structure identifier of the target structure; the computer device replaces the request parameters of the reasoning interface with the second data; and the computer device executes the reasoning process of the target reasoning engine on the second data based on the second data and the structure identifier to obtain the reasoning data. Exemplarily, the step of the computer device executing the reasoning process of the target reasoning engine on the second data may include: when the target structure is a general structure, the computer device uses the target reasoning engine to reason on third data to obtain the reasoning data, the third data being data obtained by converting the second data from a general structure to an engine structure; when the target structure is an engine structure, the computer device uses the target reasoning engine to reason on the second data to obtain the reasoning data. Exemplarily, when the target structure is a general structure, the computer device may perform structural conversion on the second data, converting the second data from a general structure to an engine structure to obtain the third data, and then execute the reasoning process of the reasoning engine on the third data. The structure parameter is used to carry the data structure of the second data; the request parameter is used to carry the data to be inferred, for example, the number to be inferred can be the second data. Exemplarily, the inference interface can be: virtual Status Inference (const EngineReq <t>&req,EngineRsp <t>&rsp)=0; wherein, the request parameter may be Req, that is, Req is replaced by the second data; the structure parameter may be <t>, that is, <t>Replaced with the structure identifier of the target structure. Exemplarily, a unified reasoning interface can be encapsulated using template technology. Template technology is the basis of generic programming. For classes or functions with the same appearance, type-independent code can be written, thereby supporting the generation of different types of classes or functions with the same code. Among them, the structure parameter can be a template parameter T. Through the template parameter T, only this unique reasoning interface can be exposed to the outside world to support different data structures for model reasoning. The interface receives a template parameter T. The same interface can support both a general structure and the engine structure of each engine, thereby supporting the input of data of different data structures into the reasoning layer where the reasoning engine is located. Different data structures are replaced by parameter T, thereby shielding the differences in the caller's data structure. Of course, if a general structure is used, the general structure can be converted into the engine structure of each engine through special logic at the reasoning layer, so that the corresponding reasoning engine can be used for reasoning.

[0100] In one technical implementation, when calling the inference engine to perform inference operations, the engine can call a specific implementation function inherited from the template interface Inference, that is, an instantiation of the template inference function. For example, the MNN inference engine needs to instantiate the parameter T as MNN::Tensor and Mulan::Tensor, so that the engine supports inference on two input structures: MNN::Tensor and Mulan::Tensor. The MNN engine instantiates the template inference function as the following two functions:

[0101] The reasoning interface of the MNN engine is instantiated as a call to the engine data of the MNN structure:

[0102] StatusInference(const EngineReq <mnn::tensor>&req,EngineRsp <mnn::tensor>&rsp);

[0103] The inference interface of the MNN engine is instantiated as a call to the engine data of the Mulan::Tensor structure: Status Inference(const EngineReq <mulan::tensor>&req,EngineRsp <mulan::tensor>&rsp);

[0104] Here, req may be used to carry the data to be inferred, MNN::Tensor represents the structure identifier of the engine structure of the MNN inference engine; Mulan::Tensor represents the structure identifier of the general structure.

[0105] In one possible implementation, the computer device can also convert the data obtained from the reasoning process for different data structures before outputting it. After the computer device obtains the inference data from the target inference engine for the second data, when the target structure is a general structure, the computer device converts the inference data into fourth data of a general structure, and uses the fourth data as the inference result of the first data; when the target structure is an engine structure, the computer device uses the inference data as the inference result of the first data. That is, when the data structure of the first data is a general structure, the obtained inference data is first converted into a general structure before output; when the target structure of the first data is an engine structure, the inference data can be directly output.

[0106] Since the inference engine calling method of the embodiment of the present application provides a pre-processing function and optimizes the engine structure of the native matching of the inference engine, the inference interface including the structural parameters is encapsulated through template technology, so that the inference engine calling method of the embodiment of the present application not only supports the engine structure, but also supports the pre-processing and model reasoning of the data to be inferred of the general structure such as Mulan::Tensor, which greatly facilitates the use of multi-engine reasoning. At the same time, the inference engine calling method of the embodiment of the present application adopts a pluggable and lazy loading method, and the background inference engine can be dynamically configured. If a certain inference engine is not currently used, the inference engine will not be created in the memory, thereby avoiding resource consumption, facilitating the self-creation of the inference engine object when using it, and saving online framework operation resources.

[0107] Step 204: The computer device obtains the execution result corresponding to the target request based on the inference data and the task processing logic of the target task preconfigured in the target model, and returns the execution result to the client.

[0108] The target model is preconfigured with the task processing logic for the target task. Based on the inference data obtained by invoking the inference engine, the computer device executes the task processing logic for the target task preconfigured in the target model, obtains the execution result corresponding to the target request, and can return the execution result to the client. For example, based on the object image data and MNN structure identifier of the MNN inference engine, the computer device invokes the inference interface of the MNN inference engine to obtain inference data. The computer device can then further execute the corresponding object recognition task based on the inference data, obtaining execution results such as facial feature locations and facial contours.

[0109] Figure 3 A schematic diagram of the design principle of an inference engine provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the XXXModelMNN in the upper left corner box represents a certain model, where the function code of the model XXXModelMNN is as follows:

[0110] +inputSize:int=224 / / Model input size

[0111] +enginePtr:shared_ptr <Engine <mnn::tensor>> / / Inference engine pointer

[0112] +predict(inputs,outputs) / / model inference

[0113] -paramCheck(inputs)

[0114] -preprocess(inputs)tensor / / model preprocessing

[0115] Such as Figure 3 As shown, after the MNN engine is created, the function code for executing the corresponding reasoning process is as follows:

[0116] +Init()int / / Initialize the inference engine

[0117] +Check()bool / / Check whether the engine is loaded successfully

[0118] +Inference(EngineReq <mnn:tensor>&,EngineRsp <mnn:tensor>&) / / Native tensor reasoning, that is, reasoning about the engine structure corresponding to the MNN engine

[0119] +Inference(EngineReq <mulan::tensor>&,EngineRsp <mulan:tensor>&) / / General structural reasoning

[0120] like Figure 3 As shown, after the TorchEngine engine is created, the functional code for executing the corresponding reasoning process is as follows:

[0121] +Init()int / / Initialize the inference engine

[0122] +Check()bool / / Check whether the engine is loaded successfully

[0123] +Inference(EngineReqat::Tensor>&,EngineRsp <at:tensor>&) / / Native tensor reasoning, that is, reasoning about the engine structure corresponding to the MNN engine

[0124] +Inference(EngineReq <mulan::tensor>&,EngineRsp <mulan:tensor>&) / / General structural reasoning

[0125] Figure 4 This is a schematic diagram of an inference framework provided by an embodiment of the present application. The inference framework is a software package that implements the inference engine calling process. Figure 4 As shown in Figure 1, the overall architecture of this framework can be divided into the model layer, the preprocessing layer, and the backend inference layer. The model layer manages the specific models requiring inference. This layer primarily facilitates unified model management and deployment, providing specific model capabilities, such as object recognition and detection. The preprocessing layer serves as a bridge between input data to be inferred and model inference data. The format and data dimensions of the input data to be inferred often differ from those required for model inference, often requiring structural transformation and alignment. This framework supports two types of preprocessing: using the engine structure and preprocessing functions specific to each engine, and using the generic structure and preprocessing functions encapsulated within the framework. The backend inference layer registers and manages the various inference engines supported by the framework, such as tensorRT, libtorch, and TNN. This layer provides a unified inference interface for the preprocessing layer, allowing inference to be performed directly with the corresponding inference engine. When the preprocessing layer uses the engine structure specific to a particular engine, inference can be performed directly with the corresponding inference engine. When using the generic structure, format conversion is required at this layer to the engine structure specific to each engine before inference can be performed.

[0126] When receiving a target request from a client, the model layer can preprocess the input data using the preprocessing functions of the corresponding inference engine, or it can preprocess the data using the preprocessing functions of the general interface. After preprocessing is complete, the unified inference interface is called to the backend inference layer to perform model inference using a specific inference engine.

[0127] The inference engine calling method of the embodiment of the present application uses template technology to encapsulate a unified inference interface Inference, and uses the structural parameter T in the inference structure to replace the structural identifiers of different data structures, thereby shielding the different data structures used by the caller at the inference layer. When calling the unified inference interface Inference, the model developer can pre-configure the correspondence between the model and the inference engine, as well as the inference engine to which the general structure can correspond. When using the engine structure of a certain inference engine, the inference engine can be selected for inference. When using the general structure Mulan::Tensor, any inference engine that supports data of the general structure can be used for inference.

[0128] Figure 5 This is a flowchart of an inference engine call provided by an embodiment of the present application, such as Figure 5 As shown, when a target request from a client is received, the data to be inferred can be obtained and the data structure used by the data to be inferred can be determined. When the data to be inferred uses the general structure of the platform, the preprocessing function of the general structure is used to preprocess the data to be inferred; when the data to be inferred uses the engine structure matched by the original engine, the preprocessing process corresponding to the engine structure is used. After the data to be inferred is preprocessed, the same inference interface can be called; further, based on the data structure used, the inference engine to be called can be further determined, such as calling preconfigured inference engine 1, inference engine 2, inference engine 3, etc., and the called inference engine is used to execute the corresponding inference function to obtain the corresponding inference result. Finally, the computer device can also further output based on the data structure used by the data to be inferred. For example, when the data to be inferred uses the engine structure, the inference result of the engine structure is directly output; when the data to be inferred uses the general structure, the computer device converts the inference result into the general structure and then outputs it.

[0129] In related technologies, the multitude of inference engines poses significant challenges for rapid platform deployment and maintenance. Two approaches are currently underway. One is to develop a new inference engine and provide conversion tools to support the conversion of models trained with existing training frameworks to the new inference engine, thereby providing inference capabilities. The other approach is to provide a platform that encapsulates a common structure to mask the differences between different inference engines, allowing callers to use only the common structure to provide inference capabilities, such as the HIE framework. Currently, the most widely used framework on the market is the HIE framework, which uses this common structure for model request and preprocessing. The HIE framework provides a unified entry point, Predict, to externally encapsulate the inference capabilities of different engines and support the invocation of multiple inference engines, such as OpenVino, TensorFlow, and TensorRT. After inference is completed by calling an inference engine, the inference results are converted to the common structure and output. For callers, simply using the HIE framework's common structure encapsulation to call the inference engine and complete model inference. However, for framework developers, when introducing a new inference engine, they must complete the conversion between the common structure and the engine structure that matches the new inference engine to provide the new engine's inference capabilities. The HIE framework has the following deficiencies: (1) The HIE reasoning framework in the related art only encapsulates a universal structure. Developers must use the preprocessing operations corresponding to the universal structure and cannot use the engine structure of the reasoning engine, let alone the preprocessing operations corresponding to the engine structure matched by the reasoning engine. (2) Because the preprocessing operations corresponding to the unified universal structure must be used, the native engine structure of the reasoning engine cannot be used for model reasoning. Therefore, switching to this framework for deployment requires a lot of additional development work. (3) The interface layer corresponding to the unified universal structure must be developed. This also means that when a new reasoning engine is integrated into the HIE framework, the framework maintainer needs to develop the corresponding conversion code in advance, making it impossible to deploy the new reasoning engine in the first place.

[0130] The inference engine calling method provided by the embodiment of the present application introduces a structural parameter in the inference interface provided to the outside world. The structural parameter can be replaced by the structure identifier of the engine structure or the structure identifier of the general structure. Therefore, the inference interface provided by the embodiment of the present application can not only support the inference engine calling of the inference engine data to be inferred of the general structure, but also support the inference engine calling of the inference engine data to be inferred of the native engine structure of the corresponding inference engine; this can meet a variety of needs. For example, if you do not want to understand the specific data structure and operation of the back-end inference engine, you can use the encapsulated general structure, and the registry of the general structure can be pre-configured with the creation interface of multiple inference engines, so that you can quickly switch between different inference engines. If you are already proficient in using a specific inference engine, or in some scenarios where the general structure is difficult to meet the performance requirements, or the preprocessing operation corresponding to the general structure is not implemented accordingly, you can use the native engine structure of the inference engine for inference. By using template technology to encapsulate the inference interface including the structural parameter, the inference engine calling process under different requirements in multiple scenarios and multiple requirements is met. Compared with the framework in the related art that can only use the general structure, the usage scenario is greatly expanded.

[0131] Moreover, since the inference engine calling method provided in the embodiment of the present application supports the native engine structure of the inference engine for preprocessing and reasoning, when a new inference engine needs to be connected, there is no need to rush to implement the support of the new inference engine calling process for the general structure in the early stage. Instead, the native engine structure of the new inference engine is directly used to perform reasoning, which greatly facilitates the deployment and launch of the new inference engine. Moreover, in order to optimize the model loading speed and facilitate the introduction of new inference engines, the inference engine calling method provided in the embodiment of the present application adopts the engine factory mode to register the inference engine, and supports on-demand loading and lazy loading. At the same time, the inference engine calling method provided in the embodiment of the present application also supports being embedded in the program code in the form of a component. When it is used specifically, it only needs to use the mirror image of this service, and then introduce the service code in the code project, and then the inference engine calling method provided in the embodiment of the present application can be used.

[0132] The inference engine calling method provided by the embodiment of the present application determines the first data to be inferred and the model identifier corresponding to the first data. The target structure of the first data can be an engine structure or a general structure. The engine structure refers to a data structure that matches any preconfigured inference engine; based on the preprocessing function corresponding to the target structure, the second data after preprocessing the first data is obtained, and the target inference engine corresponding to the model identifier is determined; based on the second data and the target structure, the inference interface corresponding to the target inference engine is called to obtain the inference data of the target inference engine for the second data; wherein, since the inference interface includes structural parameters for carrying any structural identifier in the general structure or the engine structure; thereby, the inference engine calling method of the present application can support both engine structure and general structure data to be inferred. Compared with the related technology that only supports general structures, the present application greatly expands the usage scenarios and improves the practicality of the inference engine calling process.

[0133] Figure 6 This is a structural diagram of an inference engine calling device provided in an embodiment of the present application. Figure 6 As shown, the device includes:

[0134] Determination module 601, for determining first data to be inferred and a target model corresponding to the first data, wherein the target structure of the first data is either an engine structure or a universal structure, and the engine structure refers to a data structure that matches any preconfigured inference engine;

[0135] An acquisition module 602 is configured to acquire second data after preprocessing the first data based on a preprocessing function corresponding to the target structure, and determine a target inference engine corresponding to the target model;

[0136] A calling module 603 is configured to call an inference interface corresponding to the target inference engine based on the second data and the target structure to obtain inference data of the target inference engine for the second data;

[0137] The reasoning interface includes structural parameters for carrying any structural identifier of a general structure or an engine structure.

[0138] In one possible implementation, the calling module is also used to replace the structural parameters of the reasoning interface with the structural identifier of the target structure; replace the request parameters of the reasoning interface with the second data; and based on the second data and the structural identifier, execute the reasoning process of the target reasoning engine for the second data to obtain the reasoning data.

[0139] In one possible implementation, the calling module 603 is also used to, when the target structure is a general structure, infer the third data through the target inference engine to obtain the inference data, and the third data is the data obtained by converting the second data from the general structure to the engine structure; when the target structure is an engine structure, infer the second data through the target inference engine to obtain the inference data.

[0140] In one possible implementation, the device further includes:

[0141] A conversion module is used to convert the inference data into fourth data of a general structure when the target structure is a general structure, and use the fourth data as the inference result of the first data; when the target structure is an engine structure, use the inference data as the inference result of the first data.

[0142] In one possible implementation, the device further includes:

[0143] The acquisition module 602 is further configured to acquire a creation interface of the target inference engine from a registry of the target structure;

[0144] The creation module is used to call the creation interface of the target reasoning engine based on the creation parameters of the target reasoning engine to create a running object of the target reasoning engine.

[0145] In one possible implementation, the device further includes:

[0146] An adding module is used to create a registry of the data structure for each data structure, and to add a creation interface of at least one reasoning engine in the registry, where the at least one reasoning engine is used to provide a reasoning process for the data of the data structure.

[0147] In one possible implementation, the determining module 601 is further configured to, in response to a target request from a client, obtain first data carried in the target request, and determine a target model for executing a target task, wherein the target request is configured to request execution of the target task based on the first data and return an execution result.

[0148] Accordingly, the device further comprises:

[0149] The return module is used to obtain the execution result corresponding to the target request based on the inference data and the task processing logic of the target task preconfigured in the target model, and return the execution result to the client.

[0150] The inference engine calling method provided by the embodiment of the present application determines the first data to be inferred and the target model corresponding to the first data. The target structure of the first data can be an engine structure or a general structure. The engine structure refers to a data structure that matches any preconfigured inference engine; based on the preprocessing function corresponding to the target structure, the second data after preprocessing the first data is obtained, and the target inference engine corresponding to the target model is determined; based on the second data and the target structure, the inference interface corresponding to the target inference engine is called to obtain the inference data of the target inference engine for the second data; wherein, since the inference interface includes structural parameters for carrying any structural identifier of the general structure or the engine structure; thereby, the inference engine calling method of the present application can support both engine structure and general structure data to be inferred. Compared with the related technology that only supports general structures, the present application greatly expands the usage scenarios and improves the practicality of the inference engine calling process.

[0151] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0152] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 7 As shown, the computer device includes: a memory, a processor, and a computer program stored in the memory. The processor executes the above computer program to implement the steps of the inference engine calling method. Compared with the related art, the processor can implement an inference engine calling method, which includes:

[0153] Determining first data to be inferred and a target model corresponding to the first data, where the target structure of the first data is either an engine structure or a universal structure, and the engine structure refers to a data structure that matches any preconfigured inference engine;

[0154] Based on the preprocessing function corresponding to the target structure, obtaining second data after preprocessing the first data, and determining a target inference engine corresponding to the target model;

[0155] Based on the second data and the target structure, calling the inference interface corresponding to the target inference engine to obtain inference data of the target inference engine for the second data;

[0156] The reasoning interface includes structural parameters for carrying any structural identifier of a general structure or an engine structure.

[0157] In one possible implementation, the processor may further implement, based on the second data and the target structure, calling a reasoning interface corresponding to the target reasoning engine to obtain reasoning data of the target reasoning engine for the second data, including:

[0158] Replacing the structure parameters of the inference interface with the structure identifier of the target structure;

[0159] Replacing the request parameter of the inference interface with the second data;

[0160] Based on the second data and the structure identifier, the target reasoning engine executes a reasoning process on the second data to obtain the reasoned data.

[0161] In one possible implementation, the processor may further execute, based on the second data and the structure identifier, a reasoning process of the target reasoning engine on the second data to obtain the reasoning data, including:

[0162] When the target structure is a general structure, the target inference engine is used to infer the third data to obtain the inference data, where the third data is data obtained by converting the second data from the general structure into the engine structure;

[0163] When the target structure is an engine structure, the target reasoning engine is used to reason the second data to obtain the reasoning data.

[0164] In one possible implementation, based on the second data and the target structure, after calling the inference interface corresponding to the target inference engine and obtaining the inference data of the target inference engine for the second data, the processor may further implement:

[0165] When the target structure is a universal structure, converting the inference data into fourth data of the universal structure, and using the fourth data as the inference result of the first data;

[0166] When the target structure is an engine structure, the inference data is used as the inference result of the first data.

[0167] In one possible implementation, after determining the target inference engine corresponding to the target model, the processor may further implement:

[0168] Obtain the target inference engine creation interface from the target structure registry;

[0169] Based on the creation parameters of the target reasoning engine, the creation interface of the target reasoning engine is called to create a running object of the target reasoning engine.

[0170] In a possible implementation, before obtaining the creation interface of the target inference engine from the registry of the target structure, the processor may further implement:

[0171] For each data structure, a registry of the data structure is created, and a creation interface of at least one inference engine is added to the registry. The at least one inference engine is used to provide an inference process for the data of the data structure.

[0172] In one possible implementation, the processor may further determine first data to be inferred and a target model corresponding to the first data, including:

[0173] In response to a target request from a client, obtaining first data carried in the target request and determining a target model for executing a target task, wherein the target request is used to request execution of the target task based on the first data and return an execution result;

[0174] Accordingly, based on the second data and the target structure, after calling the inference interface corresponding to the target inference engine and obtaining the inference data of the target inference engine for the second data, the processor may further implement:

[0175] Based on the inference data and the task processing logic of the target task preconfigured in the target model, an execution result corresponding to the target request is obtained, and the execution result is returned to the client.

[0176] The inference engine calling method provided by the embodiment of the present application determines the first data to be inferred and the model identifier corresponding to the first data. The target structure of the first data can be an engine structure or a general structure. The engine structure refers to a data structure that matches any preconfigured inference engine; based on the preprocessing function corresponding to the target structure, the second data after preprocessing the first data is obtained, and the target inference engine corresponding to the model identifier is determined; based on the second data and the target structure, the inference interface corresponding to the target inference engine is called to obtain the inference data of the target inference engine for the second data; wherein, since the inference interface includes structural parameters for carrying any structural identifier in the general structure or the engine structure; thereby, the inference engine calling method of the present application can support both engine structure and general structure data to be inferred. Compared with the related technology that only supports general structures, the present application greatly expands the usage scenarios and improves the practicality of the inference engine calling process.

[0177] In an alternative embodiment, a computer device is provided, such as Figure 7 As shown, Figure 7 The computer device 700 shown includes a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, for example, via a bus 702. Optionally, the computer device 700 may also include a transceiver 704, which can be used for data exchange between the computer device and other computer devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 704 is not limited to one, and the structure of the computer device 700 does not constitute a limitation on the embodiments of this application.

[0178] The processor 701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 701 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0179] The bus 702 may include a path for transmitting information between the above components. The bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 702 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0180] The memory 703 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media\other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0181] The memory 703 is used to store the computer program for executing the embodiments of the present application, and the execution is controlled by the processor 701. The processor 701 is used to execute the computer program stored in the memory 703 to implement the steps shown in the above method embodiments.

[0182] Among them, electronic equipment includes but is not limited to: servers, terminals, service clusters, cloud computing equipment, etc.

[0183] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0184] An embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0185] Those skilled in the art will understand that, unless otherwise specified, the singular forms "a," "an," "the," and "the" used herein may also include the plural forms. The terms "including" and "comprising" used in the embodiments of this application mean that the corresponding features can be implemented as the presented features, information, data, steps, and operations, but do not exclude the implementation of other features, information, data, steps, operations, etc. supported by the technical field.

[0186] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," "fourth," "1," "2," and the like (if any) are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be practiced in an order other than that shown or described.

[0187] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0188] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.< / mulan:tensor> < / mulan::tensor> < / at:tensor> < / mulan:tensor> < / mulan::tensor> < / mnn:tensor> < / mnn:tensor> < / mnn::tensor> < / mulan::tensor> < / mulan::tensor> < / mnn::tensor> < / mnn::tensor> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / mulan::tensor> < / mnn::tensor> < / t> < / t> < / t> < / int>

Claims

1. A method for calling an inference engine, characterized in that: The method comprises: Determining first data to be inferred and a target model corresponding to the first data, wherein the target structure of the first data is either an engine structure or a general structure, wherein the engine structure refers to a data structure that matches any preconfigured inference engine; and the general structure does not match any preconfigured inference engine; obtaining second data after preprocessing the first data based on a preprocessing function corresponding to the target structure; Determining a target inference engine corresponding to the target model; Based on the second data and the target structure, calling the inference interface corresponding to the target inference engine to obtain inference data of the target inference engine for the second data, including: replacing the structural parameters of the inference interface with the structure identifier of the target structure, replacing the request parameters of the inference interface with the second data, and executing the inference process of the target inference engine for the second data based on the second data and the structure identifier to obtain the inference data; The reasoning interface includes structural parameters for carrying any structural identifier of a general structure or an engine structure.

2. The method according to claim 1, characterized in that The step of executing the inference process of the target inference engine on the second data based on the second data and the structure identifier to obtain the inference data includes: When the target structure is a general structure, the target inference engine performs inference on third data to obtain the inference data, where the third data is data obtained by converting the second data from the general structure to the engine structure; When the target structure is an engine structure, the target inference engine is used to infer the second data to obtain the inference data.

3. The method according to claim 1, characterized in that After calling the inference interface corresponding to the target inference engine based on the second data and the target structure to obtain inference data of the target inference engine for the second data, the method further includes: When the target structure is a general structure, converting the inference data into fourth data of the general structure, and using the fourth data as the inference result of the first data; When the target structure is an engine structure, the inference data is used as the inference result of the first data.

4. The method according to claim 1, wherein After determining the target inference engine corresponding to the target model, the method further includes: Obtaining a creation interface for the target reasoning engine from a registry of the target structure; Based on the creation parameters of the target reasoning engine, the creation interface of the target reasoning engine is called to create a running object of the target reasoning engine, thereby loading and running the target reasoning engine in the memory.

5. The method according to claim 4, characterized in that Before obtaining the creation interface of the target inference engine from the registry of the target structure, the method further includes: For each data structure, a registry of the data structure is created, and a creation interface of at least one inference engine is added to the registry.

6. The method according to claim 1, characterized in that The determining of the first data to be inferred and the target model corresponding to the first data includes: In response to a target request from a client, obtaining first data carried in the target request and determining a target model for executing a target task, wherein the target request is used to request execution of the target task based on the first data and return an execution result; Accordingly, after calling the inference interface corresponding to the target inference engine based on the second data and the target structure to obtain the inference data of the target inference engine for the second data, the method further includes: Based on the inference data and the task processing logic of the target task preconfigured in the target model, an execution result corresponding to the target request is obtained, and the execution result is returned to the client.

7. A device for inference engine calling, characterized in that: The device comprises: a determination module, configured to determine first data to be inferred and a target model corresponding to the first data, wherein the target structure of the first data is either an engine structure or a universal structure, wherein the engine structure refers to a data structure that matches any preconfigured inference engine; and the universal structure does not match any preconfigured inference engine; an acquisition module, configured to acquire second data obtained by preprocessing the first data based on a preprocessing function corresponding to the target structure, and determine a target inference engine corresponding to the target model; a calling module, configured to call an inference interface corresponding to the target inference engine based on the second data and the target structure to obtain inference data of the target inference engine for the second data, including: replacing a structural parameter of the inference interface with a structural identifier of the target structure, replacing a request parameter of the inference interface with the second data, and executing an inference process of the target inference engine for the second data based on the second data and the structural identifier to obtain the inference data; The reasoning interface includes structural parameters for carrying any structural identifier of a general structure or an engine structure.

8. The device according to claim 7, characterized in that The calling module is further used to: When the target structure is a general structure, the target inference engine performs inference on third data to obtain the inference data, where the third data is data obtained by converting the second data from the general structure to the engine structure; When the target structure is an engine structure, the target inference engine is used to infer the second data to obtain the inference data.

9. The device according to claim 7, characterized in that The device further comprises: a conversion module, configured to, when the target structure is a universal structure, convert the inference data into fourth data of a universal structure, and use the fourth data as an inference result of the first data; The conversion module is further configured to use the inference data as an inference result of the first data when the target structure is an engine structure.

10. The device according to claim 7, characterized in that The acquisition module is further configured to acquire a creation interface of the target inference engine from a registry of the target structure; The device further comprises: A creation module is used to call the creation interface of the target reasoning engine based on the creation parameters of the target reasoning engine to create a running object of the target reasoning engine.

11. The device according to claim 10, characterized in that The device further comprises: An adding module is used to create a registry of the data structure for each data structure, and to add a creation interface of at least one inference engine in the registry.

12. The device according to claim 7, characterized in that The determination module is further configured to respond to a target request from a client, obtain first data carried in the target request, and determine a target model for executing a target task, wherein the target request is configured to request execution of the target task based on the first data and return an execution result; Accordingly, the device further includes: The return module is used to obtain the execution result corresponding to the target request based on the inference data and the task processing logic of the target task preconfigured in the target model, and return the execution result to the client.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the inference engine calling method according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the inference engine calling method according to any one of claims 1 to 6 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the inference engine calling method according to any one of claims 1 to 6 is implemented.

Images