Model calling method and related equipment

Abstract

The invention discloses a model calling method and related equipment to solve the problem that in the prior art, when cluster deployment is adopted to achieve concurrent reasoning, due to the fact that each node needs to independently load an operator and an operation environment needed by a model reasoning module, resource waste is likely to be caused. The method comprises the following steps: creating an algorithm model resource pool, wherein the algorithm model resource pool comprises operators and operation environments required when different algorithm models execute reasoning tasks; loading a plurality of algorithm models to be used for executing the reasoning task to an algorithm model resource pool; creating at least two API service threads, and activating communication links between the API service threads and the algorithm model resource pool; when a model calling request sent based on the API service thread is received, selecting a target algorithm model from the algorithm model resource pool, and loading the target algorithm model as a model instance based on the operator and the operation environment; and returning the model instance to the API service thread, so that the API service thread can execute the reasoning task based on the model instance.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model calling method and related equipment. Background Art

[0002] In order to enable the trained algorithm model to interact with the application system or client and provide reasoning services, it is usually necessary to package or integrate the trained algorithm model through an application programming interface (API). At present, when packaging or integrating through an API interface, an API interface needs to start at least an API request processing module and an algorithm model reasoning module. The API request processing module is responsible for receiving and parsing requests from the client; the algorithm model reasoning module is responsible for loading the trained algorithm model and putting it in a standby state for reasoning.

[0003] In actual applications, although the API request processing module can process multiple requests in parallel, the algorithm model reasoning module can usually only process requests one by one in serial, which makes it difficult for the API interface that encapsulates the algorithm model reasoning to achieve true concurrency, thereby extending the response time and reducing the user experience.

[0004] In the prior art, in order to solve the above technical problems, a cluster deployment method is usually adopted, that is, multiple groups of "1 API request processing module + 1 algorithm model reasoning module" nodes are deployed at the same time, and the request tasks are evenly distributed to each node with the help of application middleware, thereby realizing concurrent reasoning.

[0005] However, although the cluster deployment method achieves concurrency, each node needs to independently load the operators and operating environment required by the model reasoning module, and these operators and operating environments usually take up a lot of resources, which easily leads to resource waste. Among them, operators refer to the various basic computing operations that the algorithm model needs to perform during the reasoning process, such as matrix multiplication, convolution, pooling, activation function, batch normalization, and full connection. The operating environment includes all the underlying support libraries and framework environments that support the execution of operators. For example, for GPU reasoning, the operating environment mainly involves GPU drivers, CUDA libraries, cuDNN libraries, etc. These libraries can provide optimized acceleration support for various operators.

[0006] Based on this, how to achieve model reasoning concurrency while avoiding the waste of resources of processors such as GPU, CPU and TPU is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0007] An embodiment of the present application provides a model calling method to solve the problem that when cluster deployment is used to implement concurrent reasoning in the prior art, each node needs to independently load the operators and operating environment required by the model reasoning module, and these operators and operating environment usually occupy a large amount of resources, which easily leads to resource waste.

[0008] Embodiments of the present application also provide a model calling device, an electronic device, a computer-readable storage medium, and a computer program product.

[0009] The present application embodiment adopts the following technical solutions: In a first aspect, the present application provides a model calling method, comprising: Create an algorithm model resource pool, which includes operators and operating environments required by different algorithm models to perform reasoning tasks; Loading multiple algorithm models to be used for performing reasoning tasks into an algorithm model resource pool; Create at least two API service threads and activate the communication link between the API service threads and the algorithm model resource pool; When receiving a model call request sent based on the API service thread, select the target algorithm model from the algorithm model resource pool, and load the target algorithm model as a model instance based on the operator and the operating environment; The model instance is returned to the API service thread so that the API service thread can perform reasoning tasks based on the model instance.

[0010] Optionally, the algorithm model resource pool includes an idle chain and a scheduler; Among them, the idle chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; The scheduler is used to schedule and allocate model instances of the algorithm model in the algorithm model resource pool to respond to model call requests from the API service thread.

[0011] Optionally, when a model call request sent based on an API service thread is received, a target algorithm model is selected from an algorithm model resource pool, and the target algorithm model is loaded as a model instance based on an operator and a running environment, including: When receiving a model call request sent based on the API service thread, determine whether there is an idle algorithm model in the algorithm model resource pool; If there is an idle algorithm model, the target algorithm model is selected from the algorithm model resource pool, and the target algorithm model is loaded as a model instance based on the operator and the operating environment.

[0012] Optionally, the method further includes: If there is no idle algorithm model, an algorithm model in execution state is randomly selected from the algorithm model resource pool as the target algorithm model; When the target algorithm model ends its execution state, the model instance of the target algorithm model is returned to the API service thread.

[0013] Optionally, the algorithm model resource pool includes a free chain, and the free chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; then, When a model call request is received based on the API service thread, it is determined whether there is an idle algorithm model in the algorithm model resource pool, including: When receiving a model call request sent by an API service thread, determine whether the idle chain is empty; If the idle chain is not empty, it is confirmed that there are idle algorithm models in the algorithm model resource pool.

[0014] Optionally, the algorithm model resource pool includes a free chain, and the free chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; then, If there is an idle algorithm model, the target algorithm model is selected from the algorithm model resource pool and loaded as a model instance based on the operator and operating environment, including: If there is an idle algorithm model, get the model number at the end of the idle chain; Based on the operator and the operating environment, the target algorithm model corresponding to the model number is loaded as a model instance.

[0015] Optionally, the method further includes: Remove the model number at the end of the queue from the free chain; When the inference task of the API service thread is completed, the model number at the end of the idle chain is added back to the idle chain.

[0016] In a second aspect, the present application provides a model calling device, including a creation module, a loading module, an activation module, a processing module and a sending module, wherein: A creation module is used to create an algorithm model resource pool, which includes operators and operating environments required by different algorithm models to perform reasoning tasks; A loading module, used to load multiple algorithm models to be used for performing reasoning tasks into an algorithm model resource pool; An activation module, used to create at least two API service threads and activate the communication link between the API service threads and the algorithm model resource pool; A processing module, which is used to select a target algorithm model from an algorithm model resource pool when a model call request is received based on an API service thread, and load the target algorithm model as a model instance based on an operator and a running environment; The sending module is used to return the model instance to the API service thread so that the API service thread can perform reasoning tasks based on the model instance.

[0017] Optionally, the algorithm model resource pool includes an idle chain and a scheduler; Among them, the idle chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; The scheduler is used to schedule and allocate model instances of the algorithm model in the algorithm model resource pool to respond to model call requests from the API service thread.

[0018] Optional, processing module, including: A judging unit, configured to judge whether there is an idle algorithm model in the algorithm model resource pool when receiving a model call request sent based on the API service thread; The processing unit is used to select a target algorithm model from an algorithm model resource pool if there is an idle algorithm model, and load the target algorithm model as a model instance based on the operator and the operating environment.

[0019] Optionally, the device is also used for: If there is no idle algorithm model, an algorithm model in execution state is randomly selected from the algorithm model resource pool as the target algorithm model; When the target algorithm model ends its execution state, the model instance of the target algorithm model is returned to the API service thread.

[0020] Optionally, the algorithm model resource pool includes a free chain, and the free chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; then, the judgment unit is used to: When receiving a model call request sent by an API service thread, determine whether the idle chain is empty; If the idle chain is not empty, it is confirmed that there are idle algorithm models in the algorithm model resource pool.

[0021] Optionally, the algorithm model resource pool includes a free chain, and the free chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; then, the processing unit is used to: If there is an idle algorithm model, get the model number at the end of the idle chain; Based on the operator and the operating environment, the target algorithm model corresponding to the model number is loaded as a model instance.

[0022] Optionally, the device is also used for: Remove the model number at the end of the queue from the free chain; When the inference task of the API service thread is completed, the model number at the end of the idle chain is added back to the idle chain.

[0023] In a third aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the model calling method as described above when executed by the processor.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the model calling method as described above are implemented.

[0025] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the model calling method as described above when executed by a processor.

[0026] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: By adopting the method provided in the embodiment of the present application, by creating an algorithm model resource pool containing operators and operating environments required for different algorithm models to perform reasoning tasks, it is possible to achieve shared use of operators and operating environments by various API service threads, thereby reducing repeated loading of processor resources. Secondly, by creating at least two API service threads and activating the communication link between the API service thread and the algorithm model resource pool, multiple API service threads can simultaneously initiate model call requests to the algorithm model resource pool, and when multiple model call requests sent based on the API service thread are received, the target algorithm model can be selected from the algorithm model resource pool, and the model instance of the target algorithm model can be returned to each API service thread, thereby achieving parallel request processing. Compared with the prior art, it is possible to achieve model reasoning concurrency while avoiding the problem of processor resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of an implementation flow of a model calling method provided in an embodiment of the present application; Figure 2 A specific structural diagram of a model calling device is provided for an embodiment of the present application; Figure 3A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0029] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0030] Example 1 In order to solve the problem that when cluster deployment is used to implement concurrent reasoning in the prior art, each node needs to independently load the operators and operating environment required by the model reasoning module, and these operators and operating environment usually occupy a large amount of resources, which easily leads to resource waste. An embodiment of the present application provides a model calling method.

[0031] The execution subject of the method may be various types of computing devices, or an application or application (Application, APP) installed on the computing device. The computing device may be, for example, a user terminal such as a mobile phone, a tablet computer, a smart wearable device, or a server.

[0032] For ease of description, the present application embodiment takes the execution subject of the method as an example of a server to introduce the method. Those skilled in the art can understand that the present application embodiment takes the server as an example to introduce the method, which is only an exemplary description and does not limit the scope of protection of the claims corresponding to the present solution.

[0033] Specifically, the implementation process of the method provided in the embodiment of the present application is as follows: Figure 1 As shown, the following steps are included: Step 11: Create an algorithm model resource pool, which includes the operators and operating environment required for different algorithm models to perform reasoning tasks.

[0034] The algorithm model resource pool is a virtual space that centrally stores and manages different algorithm models, and is used to support resource allocation and scheduling when various algorithm models perform reasoning tasks. The algorithm model resource pool includes operators and operating environments required by various different algorithm models when performing reasoning tasks.

[0035] The operators required by the algorithm model to perform reasoning tasks can be understood as the basic computing units or various basic computing operations required by the algorithm model to perform reasoning tasks. For example, taking the algorithm model as a neural network model, the operators may include matrix multiplication, convolution, pooling, activation functions (such as ReLU, Sigmoid), batch normalization, fully connected, etc.

[0036] Correspondingly, the operating environment may include underlying support libraries, drivers and / or framework environments that support operator execution, to ensure that the algorithm model can be successfully executed on hardware (such as CPU, GPU).

[0037] For example, taking GPU reasoning as an example, the operating environment of the algorithm model may include GPU driver, CUDA library, cuDNN library, etc. Or, taking CPU reasoning as an example, the operating environment of the algorithm model may include linear algebra libraries such as MKL and OpenBLAS.

[0038] In the embodiment of the present application, the algorithm model resource pool can be created in the following ways: (1) Define the resources required by different algorithm models: First, according to the characteristics of different algorithm models, determine the operators and operating environment they need when performing reasoning tasks. You can list the operator requirements of each algorithm model in detail, such as matrix multiplication, convolution, pooling, etc. of neural networks; at the same time, list their operating environment requirements, such as CUDA and cuDNN libraries that support GPU reasoning or MKL libraries that support CPU reasoning.

[0039] (2) Configure the operating environment: Configure the required underlying support libraries and hardware drivers for the algorithm model resource pool. For example, if the target hardware is a GPU, preload the GPU-related driver and library files (such as CUDA and cuDNN) in the environment; if the target hardware is a CPU, load the linear algebra library suitable for the CPU (such as MKL, OpenBLAS). The configuration of these environments can ensure that the algorithm model can directly access the required resources when it is called without the need for dynamic loading.

[0040] (3) Allocate operator resources: According to the inference requirements of different algorithm models, load the required operator resources (such as convolution operations, activation functions, batch normalization, etc.) into the algorithm model resource pool. Ensure that each operator can be called quickly to improve model inference efficiency.

[0041] Optionally, the algorithm model resource pool may include a free chain and a scheduler; wherein the free chain is used to record the model numbers of the algorithm models that are currently unoccupied in the algorithm model resource pool, so that the algorithm model resource pool can quickly determine the algorithm models that are currently unoccupied, thereby improving the allocation efficiency of the algorithm models.

[0042] Optionally, the idle chain may specifically be a linked list or queue structure for recording currently unoccupied algorithm model numbers.

[0043] The scheduler is a component used to manage the allocation of algorithm models. It is used to schedule and allocate model instances of algorithm models in the algorithm model resource pool in response to model call requests from the API service thread.

[0044] Step 12: Load multiple algorithm models to be used to perform reasoning tasks into an algorithm model resource pool.

[0045] Among them, the number and type of algorithm models to be used to perform reasoning tasks can be determined according to actual needs.

[0046] For example, suppose an insurance company has the following three business requirements: (1) Identify fraudulent claims; (2) Identify potential churn customers; (3) Predict the number and amount of future claims. Furthermore, to achieve these business requirements, a fraud detection model, a customer churn prediction model, and a claims prediction model are required. In this case, it can be assumed that there are three algorithm models to be used to perform the reasoning task, namely, a fraud detection model, a customer churn prediction model, and a claims prediction model.

[0047] In an embodiment of the present application, before loading multiple algorithm models to be used to perform reasoning tasks into an algorithm model resource pool, an algorithm model registration request can be sent to the algorithm model resource pool; when the algorithm model resource pool receives the registration request sent by the algorithm model, the algorithm model can be loaded into the algorithm model resource pool.

[0048] Optionally, after multiple algorithm models to be used to perform reasoning tasks are loaded into the algorithm model resource pool, the algorithm model resource pool can also generate a unique number for each algorithm model, and the unique number is used to identify the identity information of the algorithm model.

[0049] Optionally, if the algorithm model resource pool includes a free chain, after the algorithm model resource pool generates a unique number for each algorithm model, the unique number of each algorithm model can be added to the free chain in sequence.

[0050] For example, following the above example, after the fraud detection model, customer churn prediction model, and claim prediction model are loaded into the algorithm model resource pool, unique numbers can be generated for the fraud detection model, customer churn prediction model, and claim prediction model, respectively, such as: The unique ID of the fraud detection model is ID-001 The unique ID of the customer churn prediction model is ID-002 The unique ID of the claims prediction model is ID-003 After the algorithm model is loaded and a unique number is generated, if the algorithm model resource pool includes an idle chain, these generated unique numbers can be added to the idle chain in sequence to manage the idle algorithm model, which facilitates subsequent reuse or resource scheduling of the idle algorithm model.

[0051] For example, the free chain contains the following in order: The idle chain head node points to ID-001 (fraud detection model) The idle chain node points to ID-002 (customer churn prediction model) The end node of the free chain points to ID-003 (claim prediction model).

[0052] Step 13, create at least two API service threads, and activate the communication link between the API service thread and the algorithm model resource pool.

[0053] The API service thread can be understood as a thread used to process external reasoning requests. Each API service thread represents an independent task processing channel, which completes a specific reasoning task by calling the algorithm model in the algorithm model resource pool.

[0054] In the embodiment of the present application, the functions of the API service thread include: (1) Receive and process requests: Receive reasoning task requests from users or systems, and be responsible for managing and controlling the entire reasoning task execution process; (2) Interacting with the algorithm model resource pool: When a certain reasoning task needs to be performed, the API service thread sends a request to the algorithm model resource pool to obtain a suitable algorithm model instance.

[0055] (3) Concurrent task processing: By creating multiple API service threads, multiple different reasoning task requests can be processed simultaneously.

[0056] (4) Resource management and release: After completing the inference task, the API service thread will return the used algorithm model instance to the algorithm model resource pool and put the model number back to the idle chain so that subsequent requests can reuse these resources, thereby improving resource utilization.

[0057] Optionally, the number of API service threads may be greater than or equal to the number of algorithm models in the algorithm model resource pool. For example, assuming that the algorithm model resource pool includes 6 algorithm models, a number of API service threads greater than or equal to 6 may be created.

[0058] Secondly, the communication link between the API service thread and the algorithm model resource pool is used to connect the API service thread and the algorithm model resource pool to ensure that the requests and responses between them can be transmitted quickly and accurately.

[0059] Step 14, when a model call request sent based on the API service thread is received, a target algorithm model is selected from the algorithm model resource pool, and the target algorithm model is loaded as a model instance based on the operator and the operating environment.

[0060] Among them, the model instance refers to an independent copy or object after the algorithm model is loaded, including the parameters, structure and execution status of the algorithm model, which can be directly used to perform reasoning tasks.

[0061] In an embodiment of the present application, when a model call request sent based on an API service thread is received, it can be determined whether there is an idle algorithm model in the algorithm model resource pool; if there is an idle algorithm model, the target algorithm model is selected from the algorithm model resource pool, and the target algorithm model is loaded as a model instance based on the operator and the operating environment.

[0062] Alternatively, if there is no idle algorithm model, an algorithm model in execution state is randomly selected from the algorithm model resource pool as the target algorithm model; when the target algorithm model ends the execution state, the model instance of the target algorithm model is returned to the API service thread.

[0063] Optionally, if the algorithm model resource pool includes an idle chain, when determining whether there is an idle algorithm model in the algorithm model resource pool, it can be determined whether the idle chain is empty, thereby determining whether there is an idle algorithm model in the algorithm model resource pool; if the idle chain is not empty, it is confirmed that there is an idle algorithm model in the algorithm model resource pool.

[0064] Optionally, if the algorithm model resource pool includes an idle chain, when there is an idle algorithm model, the model number at the end of the idle chain can be obtained; based on the operator and the operating environment, the target algorithm model corresponding to the model number is loaded as a model instance.

[0065] In an optional implementation, if the algorithm model resource pool includes an idle chain and a scheduler, then when there is an idle algorithm model, the algorithm model resource pool can perform a POP operation, that is, extract a model number from the end of the idle chain. Note that the POP operation here can be understood as a "pop-up" operation, which means that after extracting the model number at the end of the idle chain, this model number will be removed from the idle chain, that is, the algorithm model corresponding to this model number will no longer be considered to be in an idle state. In this way, the present application can ensure that each model number / algorithm model will only be used by one API service thread at the same time to avoid duplicate allocation.

[0066] After extracting the model number at the end of the queue from the free chain, the scheduler in the algorithm model resource pool can determine the algorithm model corresponding to the model number based on the model number, and return the model instance of the algorithm model to the API service thread that initiated the request.

[0067] Alternatively, in an optional implementation, if the algorithm model resource pool includes an idle chain and a scheduler, if there is no idle algorithm model, that is, the idle chain is empty, then a random natural number less than N (N is the number of algorithm models in the algorithm model resource pool) can be generated as the model number for providing reasoning service this time, and the target algorithm model corresponding to the model number is queued and waits for the target algorithm model to complete the previous round of reasoning tasks, and then the model instance of the target algorithm model is returned to the API service thread, so that the API service thread can perform this reasoning task based on the model instance.

[0068] Optionally, after extracting the model number at the end of the queue from the free chain, the process also includes removing the model number at the end of the queue from the free chain; when the inference task of the API service thread ends, the model number at the end of the queue of the free chain is added back to the free chain.

[0069] In an optional implementation, when the reasoning task of the API service thread ends, it can first check whether the model number already exists in the free chain; if the model number already exists in the free chain, the API service thread can directly release the model instance of the algorithm model corresponding to the model number, and there is no need to re-add the model number to the free chain. Alternatively, if the model number does not exist in the free chain, a PUSH operation can be performed, that is, the model number is added to the end of the free chain to ensure that the algorithm model resources can be reused. Then, the API service thread releases the model instance to ensure that the model instance in the algorithm model resource pool remains available.

[0070] Step 15: Return the model instance to the API service thread so that the API service thread can perform reasoning tasks based on the model instance.

[0071] By adopting the method provided in the embodiment of the present application, by creating an algorithm model resource pool containing operators and operating environments required for different algorithm models to perform reasoning tasks, it is possible to achieve shared use of operators and operating environments by various API service threads, thereby reducing repeated loading of processor resources. Secondly, by creating at least two API service threads and activating the communication link between the API service thread and the algorithm model resource pool, multiple API service threads can simultaneously initiate model call requests to the algorithm model resource pool, and when multiple model call requests sent based on the API service thread are received, the target algorithm model can be selected from the algorithm model resource pool, and the model instance of the target algorithm model can be returned to each API service thread, thereby achieving parallel request processing. Compared with the prior art, it is possible to achieve model reasoning concurrency while avoiding the problem of processor resource waste.

[0072] Example 2 In order to solve the problem that when cluster deployment is used to implement concurrent reasoning in the prior art, each node needs to independently load the operators and operating environment required by the model reasoning module, and these operators and operating environments usually occupy a large amount of resources, which easily leads to resource waste, the embodiment of the present application provides a model calling device, the specific structural diagram of the device is as follows Figure 2 As shown, it includes a creation module 21, a loading module 22, an activation module 23, a processing module 24 and a sending module 25. The functions of each unit are as follows: A creation module 21 is used to create an algorithm model resource pool, which includes operators and operating environments required for different algorithm models to perform reasoning tasks; A loading module 22, used to load multiple algorithm models to be used for performing reasoning tasks into an algorithm model resource pool; An activation module 23, used to create at least two API service threads and activate a communication link between the API service threads and the algorithm model resource pool; The processing module 24 is used to select a target algorithm model from the algorithm model resource pool when receiving a model call request sent based on the API service thread, and load the target algorithm model as a model instance based on the operator and the operating environment; The sending module 25 is used to return the model instance to the API service thread so that the API service thread can perform the reasoning task based on the model instance.

[0073] Optionally, the algorithm model resource pool includes a free chain and a scheduler; wherein the free chain is used to record the model numbers of the algorithm models that are currently not occupied in the algorithm model resource pool; the scheduler is used to schedule and allocate model instances of the algorithm models in the algorithm model resource pool to respond to model call requests from the API service thread.

[0074] Optionally, the processing module 24 includes: A judging unit, configured to judge whether there is an idle algorithm model in the algorithm model resource pool when receiving a model call request sent based on the API service thread; The processing unit is used to select a target algorithm model from an algorithm model resource pool if there is an idle algorithm model, and load the target algorithm model as a model instance based on the operator and the operating environment.

[0075] Optionally, the device is also used for: If there is no idle algorithm model, an algorithm model in execution state is randomly selected from the algorithm model resource pool as the target algorithm model; When the target algorithm model ends its execution state, the model instance of the target algorithm model is returned to the API service thread.

[0076] Optionally, the algorithm model resource pool includes a free chain, and the free chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; then, the judgment unit is used to: When receiving a model call request sent by an API service thread, determine whether the idle chain is empty; If the idle chain is not empty, it is confirmed that there are idle algorithm models in the algorithm model resource pool.

[0077] Optionally, the algorithm model resource pool includes a free chain, and the free chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; then, the processing unit is used to: If there is an idle algorithm model, get the model number at the end of the idle chain; Based on the operator and the operating environment, the target algorithm model corresponding to the model number is loaded as a model instance.

[0078] Optionally, the device is also used for: Remove the model number at the end of the queue from the free chain; When the inference task of the API service thread is completed, the model number at the end of the idle chain is added back to the idle chain.

[0079] By adopting the device provided by the embodiment of the present application, by creating an algorithm model resource pool containing operators and operating environments required for different algorithm models to perform reasoning tasks, it is possible to achieve shared use of operators and operating environments by various API service threads, thereby reducing repeated loading of processor resources. Secondly, by creating at least two API service threads and activating the communication link between the API service thread and the algorithm model resource pool, multiple API service threads can simultaneously initiate model call requests to the algorithm model resource pool, and when multiple model call requests sent based on the API service thread are received, the target algorithm model can be selected from the algorithm model resource pool, and the model instance of the target algorithm model can be returned to each API service thread, thereby achieving parallel request processing. Compared with the prior art, it is possible to achieve model reasoning concurrency while avoiding the problem of processor resource waste.

[0080] Example 3 Figure 3 A hardware structure diagram of an electronic device for implementing various embodiments of the present application, the electronic device may include a processor 301 and a memory 302 storing computer program instructions. Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0081] The memory 302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 302 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 302 may be inside or outside the electronic device. In a particular embodiment, the memory 302 may be a non-volatile solid-state memory.

[0082] In one embodiment, the memory 302 may be a read-only memory (ROM). In one embodiment, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0083] The processor 301 implements any one of the time series data compression methods or time series data decompression methods in the above embodiments by reading and executing computer program instructions stored in the memory 302 .

[0084] In one example, the electronic device may further include a communication interface 303 and a bus 310. Figure 3 As shown, the processor 301, the memory 302, and the communication interface 303 are connected via a bus 310 and communicate with each other.

[0085] The communication interface 303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0086] Bus 310 includes hardware, software or both, coupling the components of the electronic device to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnect, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, bus 710 may include one or more buses. Although the present application embodiment describes and illustrates a specific bus, the present application considers any suitable bus or interconnect.

[0087] In addition, in combination with the time series data compression method or the time series data decompression method in the above embodiments, the embodiments of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any time series data compression method or time series data decompression method in the above embodiments is implemented.

[0088] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0089] The above is only a specific implementation example of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, module and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A model calling method, characterized in that: include: Creating an algorithm model resource pool, wherein the algorithm model resource pool includes operators and operating environments required for different algorithm models to perform reasoning tasks; Loading multiple algorithm models to be used for performing reasoning tasks into the algorithm model resource pool; Create at least two API service threads, and activate the communication link between the API service threads and the algorithm model resource pool; When a model call request sent based on the API service thread is received, a target algorithm model is selected from the algorithm model resource pool, and the target algorithm model is loaded as a model instance based on the operator and the operating environment; The model instance is returned to the API service thread, so that the API service thread can perform reasoning tasks based on the model instance.

2. The method according to claim 1, characterized in that The algorithm model resource pool includes an idle chain and a scheduler; The idle chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; The scheduler is used to schedule and allocate model instances of the algorithm model in the algorithm model resource pool to respond to model call requests from the API service thread.

3. The method according to claim 1 or 2, characterized in that When a model call request sent based on the API service thread is received, a target algorithm model is selected from the algorithm model resource pool, and the target algorithm model is loaded as a model instance based on the operator and the operating environment, including: When receiving a model call request sent based on the API service thread, determining whether there is an idle algorithm model in the algorithm model resource pool; If there is an idle algorithm model, a target algorithm model is selected from the algorithm model resource pool, and the target algorithm model is loaded as the model instance based on the operator and the operating environment.

4. The method according to claim 3, characterized in that The method further comprises: If there is no idle algorithm model, randomly selecting an algorithm model in execution state from the algorithm model resource pool as the target algorithm model; When the target algorithm model ends the execution state, the model instance of the target algorithm model is returned to the API service thread.

5. The method according to claim 3, characterized in that The algorithm model resource pool includes a free chain, and the free chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; then, When a model call request sent based on the API service thread is received, determining whether there is an idle algorithm model in the algorithm model resource pool includes: When receiving the model call request sent based on the API service thread, determining whether the idle chain is empty; If the idle chain is not empty, it is confirmed that there are idle algorithm models in the algorithm model resource pool.

6. The method according to claim 3, characterized in that The algorithm model resource pool includes a free chain, and the free chain is used to record the model numbers of the algorithm models that are not currently occupied in the algorithm model resource pool; then, If there is an idle algorithm model, selecting a target algorithm model from the algorithm model resource pool, and loading the target algorithm model as the model instance based on the operator and the operating environment, including: If there is an idle algorithm model, obtain the model number of the tail of the idle chain; The target algorithm model corresponding to the model number is loaded as the model instance based on the operator and the operating environment.

7. The method according to claim 6, characterized in that The method further comprises: Remove the model number at the end of the queue from the free chain; When the reasoning task of the API service thread is completed, the model number at the end of the idle chain is added back to the idle chain.

8. A model calling device, characterized in that: It includes creating module, loading module, activating module, processing module and sending module, among which: A creation module is used to create an algorithm model resource pool, wherein the algorithm model resource pool includes operators and operating environments required for different algorithm models to perform reasoning tasks; A loading module, used to load multiple algorithm models to be used for performing reasoning tasks into the algorithm model resource pool; An activation module, used to create at least two API service threads and activate a communication link between the API service threads and the algorithm model resource pool; A processing module, configured to select a target algorithm model from the algorithm model resource pool when receiving a model call request sent based on the API service thread, and load the target algorithm model as a model instance based on the operator and the operating environment; A sending module is used to return the model instance to the API service thread so that the API service thread can perform reasoning tasks based on the model instance.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the model calling method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the model calling method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the model calling method according to any one of claims 1 to 7 when executed by a processor.

Images