Adaptation Method, Device and Electronic Device of Inference Engine

Abstract

An embodiment of this specification proposes an adaptation method, device, and electronic device for an inference engine. The method includes: obtaining data to be inferred; determining an inference engine for performing inference calculations on the data to be inferred according to a preset engine adaptation strategy; and calling the inference engine to complete the inference calculations and output inference result data. Since the inference interface is unified in this embodiment of the specification, algorithm implementers can use the unified inference interface to implement the inference of deep learning models without having to pay attention to the details of each inference engine, thus realizing the reuse of the inference interface. On the other hand, the inference engine side only needs to implement the unified engine interface to support existing inference schemes.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and more specifically, to a method for adapting an inference engine, an apparatus for adapting an inference engine, an electronic device, and a computer-readable storage medium. Background Art

[0002] Currently, more and more deep learning scenarios are applied to AIoT (artificial intelligence & internet of things). In order to achieve better results for deep learning models on low-cost solutions, various companies have developed different inference engines, and these inference engines have different focuses.

[0003] An inference engine is a program module for inferring deep learning models. Some inference engines support a comprehensive range of deep learning models, while some inference engines only have outstanding effects on the Graphics Processing Unit (GPU) side.

[0004] In actual applications, in order to optimize the performance of deep learning models, different deep learning models may use different inference engines at different times. However, in this solution, when implementing deep learning model inference, it is necessary to design inference engine interfaces separately for different inference engines, and the inference engine interfaces cannot be reused, resulting in poor compatibility. Summary of the Invention

[0005] The embodiments of this specification provide a new technical solution for adapting an inference engine.

[0006] According to the first aspect of this specification, a method for adapting an inference engine is provided, including:

[0007] Obtain data to be inferred;

[0008] Determine an inference engine for performing inference calculation on the data to be inferred according to a preset engine adaptation strategy;

[0009] Call the inference engine to complete the inference calculation and output the inference result data.

[0010] Optionally, the determining an inference engine for performing inference calculation on the data to be inferred according to a preset engine adaptation strategy includes:

[0011] Determine the inference engine specified in the data to be inferred as the inference engine for performing inference calculation on the data to be inferred.

[0012] Optionally, the determining, according to a preset engine adaptation policy, of an inference engine for performing inference calculation on the data to be inferred includes:

[0013] Determining an inference engine for performing inference calculation on the data to be inferred according to the type of the inference engine.

[0014] Optionally, the determining, according to a preset engine adaptation policy, of an inference engine for performing inference calculation on the data to be inferred includes:

[0015] Determining an inference engine for performing inference calculation on the data to be inferred according to the current performance metrics of the inference engine.

[0016] Optionally, the current performance metrics of the inference engine at least include: the current inference speed of the inference engine, and / or, the current load size of the inference engine.

[0017] Optionally, before obtaining the data to be inferred, the method further includes:

[0018] Initializing a plurality of inference engines to be selected.

[0019] Optionally, the process of calling the inference engine to complete the inference calculation includes:

[0020] Real-time monitoring of the performance metrics of each of the inference engines to be selected;

[0021] If, according to the performance metrics, it is determined that there is a new inference engine whose performance is better than that of the inference engine currently performing inference calculation, then calling the new inference engine to continue the inference calculation; until the inference calculation process is completed.

[0022] According to a second aspect of this specification, there is also provided an adaptation device for an inference engine, including:

[0023] An obtaining module, configured to obtain data to be inferred;

[0024] A determining module, configured to determine an inference engine for performing inference calculation on the data to be inferred according to a preset engine adaptation policy;

[0025] A calling module, configured to call the inference engine to complete the inference calculation and output inference result data.

[0026] According to a third aspect of this specification, there is also provided an electronic device, which includes:

[0027] The adaptation device for an inference engine as described in the second aspect of this specification; or,

[0028] A processor and a memory, the memory being configured to store instructions for controlling the processor to execute the method according to any one of the first aspects of this specification.

[0029] According to the fourth aspect of this specification, there is also provided a computer-readable storage medium storing executable instructions that, when executed by a processor, perform the method according to any one of the first aspects of this specification.

[0030] In one embodiment, by setting up an inference interface, the inference interface obtains data to be inferred; according to a preset engine adaptation strategy, an inference engine for performing inference calculations on the data to be inferred is determined statically or dynamically; then the inference engine is called to complete the inference calculation and output inference result data. Since the inference interface is unified in this embodiment, algorithm implementers can use the unified inference interface to implement the inference of deep learning models without having to pay attention to the details of each inference engine, realizing the reuse of the inference interface; on the other hand, the inference engine side only needs to implement a unified engine interface to support existing inference schemes.

[0031] Other features and advantages of the present invention will become clear from the following detailed description of the exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings incorporated in and constituting a part of this specification illustrate embodiments of the present invention and, together with the description, are used to explain the principles of the present invention.

[0033] Figure 1 The structural schematic diagram of a terminal device capable of implementing the adaptation method of an inference engine for one embodiment is shown;

[0034] Figure 2 It is a flowchart of the adaptation method of an inference engine according to an embodiment of this specification;

[0035] Figure 3 It is an architecture diagram that can be used to implement the adaptation method of an inference engine according to an embodiment of this specification;

[0036] Figure 4 It is a flowchart of the dynamic adaptation method of an inference engine that can be used to implement an embodiment of this specification;

[0037] Figure 5 It is a principle block diagram of a dynamic adaptation device of an inference engine that can be used in an embodiment of this specification;

[0038] Figure 6 It is a principle block diagram of an electronic device that can be used to implement an embodiment of this specification. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.

[0040] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0041] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0042] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0043] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] <Application Scenario>

[0045] With the continuous improvement of mobile computing power and the rapid development of deep learning, especially the maturation of small network models, inference and prediction previously performed in the cloud can now be moved to the device. Edge intelligence, which deploys and runs AI algorithms on the device, offers advantages over server-side intelligence, such as low latency, data privacy, and cloud resource conservation. Edge intelligence is already demonstrating significant value in scenarios such as AI photography and visual effects.

[0046] Deep neural network inference engines, which load deep neural network models on the device for inference and prediction, are currently widely used in a variety of applications, covering scenarios such as live streaming, short videos, search recommendations, product image search, interactive marketing, equity issuance, and security risk control. They also have several applications in scenarios such as AIoT. Specifically, the deep neural network inference engine is responsible for loading the network model on the device, performing inference and prediction, and returning relevant results. The entire inference process can be divided into three stages: model loading and parsing, computational graph scheduling, and efficient operation on heterogeneous backends.

[0047] It can be seen that the edge-side inference engine is the core module of edge-side intelligent applications. Under the limitations of limited computing power, limited memory, etc., it needs to efficiently utilize limited resources to quickly complete inference and prediction. Based on this, the embodiments of this specification set up an inference interface on the edge side and enable the inference interface to obtain data to be inferred; according to a preset engine adaptation strategy, statically or dynamically determine an inference engine for performing inference calculations on the data to be inferred; then call the inference engine to complete the inference calculation and output inference result data. Thus, the inference of deep learning models can be realized through a unified inference interface, without separately designing inference engine interfaces for different inference engines, achieving the reuse of the inference interface.

[0048] <Hardware device>

[0049] Figure 1 The composition structure diagram of a terminal device capable of implementing the adaptation method of the inference engine according to an embodiment is shown.

[0050] As Figure 1 shown, the terminal device 1000 in this embodiment may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and so on.

[0051] Among them, the processor 1100 is used to execute program instructions, and the program instructions can adopt instruction sets such as x86, Arm, RISC, MIPS, SSE, etc. The memory 1200 includes, for example, ROM (read-only memory), RAM (random access memory), non-volatile memory such as a hard disk, etc. The interface device 1300 includes, for example, a USB interface, a headphone interface, etc. The communication device 1400 can perform wired or wireless communication, for example. The display device 1500 is, for example, a liquid crystal display screen, a touch display screen, etc. The input device 1600 can include, for example, a touch screen, a keyboard, etc. The speaker 1700 is used to output voice information. The microphone 1800 is used to collect voice information.

[0052] The terminal device 1000 can be any device such as a smart phone, a portable computer, a desktop computer, a tablet computer, etc.

[0053] In this embodiment, the memory 1200 of the terminal device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to operate to support the implementation of the adaptation method of the inference engine according to any embodiment of this specification. Those skilled in the art can design instructions according to the solutions disclosed in this specification. How the instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0054] Those skilled in the art should understand that although in Figure 1Multiple devices of the terminal device 1000 are shown, however, the terminal device 1000 in the embodiments of this specification may only involve some of the devices, for example, only involve the processor 1100, the memory 1200, the display device 1500, the input device 1600, etc.

[0055] Figure 1 The shown terminal device 1000 is only illustrative and is in no way intended to limit this specification, its applications, or uses.

[0056] <Method Embodiment>

[0057] Figure 2 is a schematic flowchart of a method for adapting an inference engine according to an embodiment of this specification. This method can be implemented by a terminal device, such as Figure 1 the shown terminal device 1000.

[0058] As Figure 2 shown, the method for adapting the inference engine in this embodiment may include the following steps 2100 to 2300:

[0059] Step 2100, obtain data to be inferred.

[0060] Specifically, the data to be inferred may be, for example, a deep neural network model.

[0061] Before this step, the terminal device 1000 may first initialize multiple inference engines to be selected. For example Figure 3 as shown, the inference engines to be selected may be, for example, a compilation stack (Tensor Virtual Machine, TVM) engine for deep learning inference optimization, a NeuroPliot engine, or other existing inference engines.

[0062] Step 2200, determine an inference engine for performing inference calculations on the data to be inferred according to a preset engine adaptation strategy.

[0063] In this step, the terminal device 1000 may statically or dynamically select different inference engines from multiple inference engines to be selected to perform inference calculations.

[0064] In the scenario of statically selecting an inference engine, a specific inference engine may be specified for use according to prior knowledge in the data to be inferred, that is, the terminal device 1000 may specifically determine the inference engine specified by the data to be inferred as the inference engine for performing inference calculations on the data to be inferred. In this application scenario, the terminal device uses the specified inference engine to complete the inference calculations.

[0065] For example, if the TVM engine is specified for inference calculation in the data to be inferred, the terminal device 1000 calls the TVM engine to complete the inference calculation and outputs the inference result data.

[0066] In the scenario of dynamically selecting an inference engine, the inference engine for inference calculation can be specifically determined according to the type of the inference engine or the current performance metrics of the inference engine.

[0067] In one example, the terminal device 1000 can specifically determine the inference engine for performing inference calculation on the data to be inferred according to the type of the inference engine.

[0068] For example, the terminal device 1000 can randomly select one of the multiple inference engines to be selected. For example, it selects the NeuroPliot engine for inference calculation. This embodiment does not make specific limitations on this.

[0069] In another example, the terminal device 1000 can specifically determine the inference engine for performing inference calculation on the data to be inferred according to the current performance metrics of the inference engine. Among them, the current performance metrics of the inference engine at least include: the current inference speed of the inference engine, and / or, the current load size of the inference engine.

[0070] For example, there are multiple inference engines to be selected. If the terminal device 1000 determines that the inference speed of the TVM engine is faster than that of the NeuroPliot engine, the terminal device 1000 can determine the TVM engine as the inference engine for performing inference calculation on the data to be inferred.

[0071] For another example, there are multiple inference engines to be selected. If the terminal device 1000 determines that the load of the TVM engine is larger than that of the NeuroPliot engine, the terminal device 1000 can determine the NeuroPliot engine as the inference engine for performing inference calculation on the data to be inferred.

[0072] For another example, there are multiple inference engines to be selected. If the terminal device 1000 determines that the load of the TVM engine is larger than that of the NeuroPliot engine, and the inference speed of the TVM engine is lower than that of the NeuroPliot engine, in terms of overall performance, the performance of the NeuroPliot engine is better than that of the TVM engine. Then the terminal device 1000 can determine the NeuroPliot engine as the inference engine for performing inference calculation on the data to be inferred.

[0073] Step 2300, call the inference engine to complete the inference calculation and output the inference result data.

[0074] After determining the inference engine for performing inference calculations on the data to be inferred, the terminal device 1000 invokes the determined inference engine to complete the inference calculations, thereby outputting inference result data. For example, the determined inference engine can be invoked by a command to complete the inference calculations to obtain the inference result data at the merchant side.

[0075] Furthermore, in practical applications, for the scenario of dynamically selecting an inference engine, during the process of the terminal device 1000 invoking the inference engine to complete the inference calculations, it can also monitor the performance metrics of each of the inference engines to be selected in real time; if, based on the performance metrics, it is determined that there is a new inference engine whose performance is better than the inference engine currently performing the inference calculations, then the new inference engine is invoked to continue the inference calculations; until the inference calculation process is completed.

[0076] That is to say, during the process of the terminal device 1000 performing inference calculations, if it is found that in addition to the current inference engine among the inference engines to be selected, there is also an inference engine with better current performance, then the inference engine can be dynamically switched.

[0077] As Figure 4 shown, the Inference Dispatcher in the terminal device 1000 first obtains the available inference engines. For example, the available inference engines are obtained by invoking the GetAvaiableInference function, and then the corresponding inference engine is invoked to input the data to be inferred into a tensor (Tensor). For example, the data to be inferred is imported by invoking the GetInputTensor function, and the inference engine (Inference Engine) performs the inference calculations (Run). At this time, the inference engine (Inference Engine) performing the inference calculations can be the inference engine (Inference Engine) initially invoked, or it can be dynamically switched to another inference engine (Inference Engine) according to the performance metrics of each inference engine (Inference Engine) during the inference calculation process to complete the inference calculations. After the inference calculations are completed, the inference result data is output. For example, the inference result data is obtained by invoking the function to obtain the output tensor (GetOnputTensor), and the inference result data is returned through the inference engine interface.

[0078] In this embodiment, by setting up an inference interface, the inference interface obtains the data to be inferred; according to a preset engine adaptation strategy, a static or dynamic inference engine for performing inference calculations on the data to be inferred is determined; then the inference engine is called to complete the inference calculation and output the inference result data. Since the inference interface is unified in this embodiment, the algorithm implementation party can use the unified inference interface to implement the inference of the deep learning model without paying attention to the details of each inference engine, realizing the reuse of the inference interface; on the other hand, the inference engine party only needs to implement the unified engine interface to support the existing inference solutions.

[0079] <Device Embodiment>

[0080] In this embodiment, an adaptation device for an inference engine is further provided. The adaptation device for the inference engine can be set, for example, in the Figure 1 terminal device 1000 as shown.

[0081] As Figure 5 shown, the adaptation device 5000 for the inference engine includes an acquisition module 5100, a determination module 5200, and a call module 5300.

[0082] Among them, the acquisition module 5100 is used to acquire the data to be inferred.

[0083] The determination module 5200 is used to determine an inference engine for performing inference calculations on the data to be inferred according to a preset engine adaptation strategy.

[0084] The call module 5300 is used to call the inference engine to complete the inference calculation and output the inference result data.

[0085] In one example, the determination module 5200 may specifically be used to determine the inference engine specified by the data to be inferred as the inference engine for performing inference calculations on the data to be inferred.

[0086] In another example, the determination module 5200 may specifically be used to determine an inference engine for performing inference calculations on the data to be inferred according to the type of the inference engine.

[0087] In yet another example, the determination module 5200 may specifically be used to determine an inference engine for performing inference calculations on the data to be inferred according to the current performance metrics of the inference engine.

[0088] Among them, the current performance metrics of the inference engine at least include: the current inference speed of the inference engine, and / or, the current load size of the inference engine.

[0089] Furthermore, the adaptation device for the inference engine may further include an initialization module for initializing a plurality of inference engines to be selected.

[0090] In a feasible embodiment, the calling module 5300 may specifically be configured to monitor in real time the performance metrics of each of the to-be-selected inference engines; if, based on the performance metrics, a new inference engine whose performance is superior to that of the inference engine currently performing inference calculation is determined, then call the new inference engine to continue the inference calculation; until the inference calculation process is completed.

[0091] The inference engine adaptation device of this embodiment can be used to implement the technical solutions of the above method embodiments, and its implementation principle and technical effects are similar, so details are not described herein again.

[0092] <Device Embodiment>

[0093] In this embodiment, an electronic device is further provided. The electronic device includes the inference engine adaptation device 5000 described in the device embodiment of this specification; or, the electronic device is Figure 6 the electronic device 6000 shown in the figure, including:

[0094] A memory 6100, configured to store executable commands.

[0095] A processor 6200, configured to execute the method described in any method embodiment of this specification under the control of the executable commands stored in the memory 6100.

[0096] The implementation subject of the method embodiment executed by the electronic device may be a server or a terminal device.

[0097] <Computer-Readable Storage Medium Embodiment>

[0098] This embodiment provides a computer-readable storage medium, in which executable commands are stored. When the executable commands are executed by a processor, the method described in any method embodiment of this specification is executed.

[0099] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0100] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as being a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0101] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or can be downloaded to an external computer or an external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0102] The computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present invention.

[0103] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0104] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium, which instructions cause a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises a manufacture including instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0105] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0106] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions. As is well known to those skilled in the art, implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

[0107] The embodiments of the present invention have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An adaptation method for an inference engine, comprising: Initializing a plurality of inference engines to be selected; Obtaining data to be inferred, where the data to be inferred is a deep neural network model; Determining an inference engine for performing inference calculations on the data to be inferred according to a preset engine adaptation strategy; Invoking the inference engine to complete the inference calculation and outputting inference result data, including: Real-time monitoring of the performance metrics of each of the inference engines to be selected; If, based on the performance metrics, a new inference engine is determined that has better performance than the inference engine currently performing the inference calculation, then invoking the new inference engine to continue the inference calculation; until the inference calculation process is completed.

2. The method according to claim 1, wherein The determining an inference engine for performing inference calculations on the data to be inferred according to a preset engine adaptation strategy includes: Determining the specified inference engine in the data to be inferred as the inference engine for performing inference calculations on the data to be inferred.

3. The method according to claim 1, wherein, The determining an inference engine for performing inference calculations on the data to be inferred according to a preset engine adaptation strategy includes: Determining an inference engine for performing inference calculations on the data to be inferred according to the type of inference engine.

4. The method according to claim 1, wherein The determining an inference engine for performing inference calculations on the data to be inferred according to a preset engine adaptation strategy includes: Determining an inference engine for performing inference calculations on the data to be inferred according to the current performance metrics of the inference engine.

5. The method according to claim 4, wherein, The current performance metrics of the inference engine at least include: the current inference speed of the inference engine, and / or, the current load size of the inference engine.

6. An adaptation device for an inference engine, comprising: An acquisition module for acquiring data to be inferred, where the data to be inferred is a deep neural network model; A determination module for determining an inference engine for performing inference calculations on the data to be inferred according to a preset engine adaptation strategy; An invocation module for invoking the inference engine to complete the inference calculation and outputting inference result data; An initialization module for initializing a plurality of inference engines to be selected; The invocation module is specifically configured to: Real-time monitor the performance metrics of each of the inference engines to be selected; If, based on the performance metrics, a new inference engine is determined that has better performance than the inference engine currently performing the inference calculation, then invoke the new inference engine to continue the inference calculation; Until the inference calculation process is completed.

7. An electronic device, wherein, Including: The adaptation device for an inference engine according to claim 6; Or, A processor and a memory, where the memory is used to store instructions for controlling the processor to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing executable instructions, which when executed by a processor, execute the method according to any one of claims 1 to 5.

Images