Multi-model inference method, inference system, terminal device and computer readable storage medium

Abstract

The application is suitable for the field of artificial intelligence technology, and provides a multi-model reasoning method, a reasoning system, a terminal device and a computer readable storage medium, which comprises the following steps: receiving a first user request; in the case that a first model is in a dormant state, switching the first model from the dormant state to a wake-up state; the first model is a model required to be called by the first user request in the plurality of models; triggering the processor to acquire first target historical data from the memory, and reasoning the first user request based on the first target historical data and the first model to obtain a first reasoning result. Through the above method, the effective balance of large model reasoning efficiency and resource utilization rate can be realized.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a multi-model reasoning method, reasoning system, terminal device and computer-readable storage medium. Background Technology

[0002] With the development of large-scale model technology, its application areas are becoming increasingly widespread, and the performance requirements for server-side devices (such as central processing units and image processors) deploying large-scale models are also increasing. Currently, high-bandwidth memory (HBM) space allocation technology and computing resource allocation technology can enable the deployment of multiple models on the server side.

[0003] In some application scenarios, inference requests for some models among multiple large models deployed on the server are relatively sparse, meaning that these models are called less frequently. This results in low utilization of server-side computing resources for these models, leading to wasted computing resources. Therefore, how to rationally allocate server-side computing resources and efficiently run multiple large models is an important issue. Summary of the Invention

[0004] This application provides a multi-model inference method, inference system, terminal device, and computer-readable storage medium, which can achieve an effective balance between large model inference efficiency and resource utilization.

[0005] In a first aspect, embodiments of this application provide a multi-model inference method applied to an inference system, the inference system including a processor and a memory, the processor deploying multiple models, and the memory storing historical inference data for each of the multiple models; the multi-model inference method includes: Receive the first user request; When the first model is in a dormant state, the first model is switched from the dormant state to the awakened state; the first model is the model that the first user request needs to call among the plurality of models; The processor is triggered to retrieve first target historical data from the memory, and to perform reasoning on the first user request based on the first target historical data and the first model to obtain a first reasoning result; wherein, the processor stores model parameters of the first model, and the first target historical data is historical reasoning data that matches the first user request from the historical reasoning data of the first model.

[0006] In this embodiment, the processor stores model parameters in both the sleep and wake-up states. This eliminates the need to transmit model parameters during the switch between sleep and wake-up states, reducing transmission time and improving inference efficiency. In the sleep state, historical inference data is stored in memory. Upon switching to the wake-up state, the historical inference data matching the user request (target historical data) is transmitted from memory to the processor for model retrieval. This method reduces the processor's space usage for historical inference data in the sleep state. Furthermore, upon switching to the wake-up state, only the portion of historical inference data matching the user request is transmitted, eliminating the need to transmit all historical inference data from the first model. This reduces the transmission time of redundant information and further improves inference efficiency. By employing this method, we can reduce the time spent transmitting data during inference, improve inference efficiency, reduce processor space usage, save computing resources, and achieve an effective balance between inference efficiency and resource utilization.

[0007] In one implementation of the first aspect, switching the first model from a dormant state to a awakened state includes: Determine whether the processor currently has any idle resources; If the processor currently has idle resources, a first request is sent to the memory to trigger the memory to filter out the first target historical data according to the first request; wherein, the first request includes the first user request and a first identifier corresponding to the first user request; Switch the first model from sleep state to wake state.

[0008] In this embodiment, the process of waking up the first model and the process of the memory filtering the historical data of the first target are processed in parallel. When the first model executes the inference process, the memory has completed or is about to complete the filtering of the historical data of the first target, which can reduce the time for the first model to load the historical data of the first target from the memory, thereby improving the inference efficiency.

[0009] In one implementation of the first aspect, after determining whether the processor currently has idle resources, the method further includes: If the processor currently has no idle resources, then the first user request is added to the request queue; When the first user request is the first request in the request queue and the second model completes the inference task, the second target historical data is stored in the memory; wherein, the second model is the model called by the second user request, and the second user request is the previous completed user request of the first user request; the second target historical data is the inference data obtained by the second model executing the inference task corresponding to the second user request; Release the second target historical data from the processor; switch the second model from the wake-up state to the sleep state; Send the first request to the memory; Switch the first model from sleep state to wake state.

[0010] In this embodiment, when there are no idle resources, the scheduler can automatically allocate resources according to the scheduling policy, and wake up and hibernate related models as needed according to the resource allocation, thereby achieving a reasonable allocation of processor resources and saving computing resources.

[0011] In one implementation of the first aspect, after adding the first user request to the request queue, the method further includes: If the request queue does not contain any pending tasks corresponding to the second model, or if the number of skips corresponding to the second model reaches a preset threshold, then the second model is determined to have completed the inference task. The number of times to skip indicates the number of times the first user request is skipped and requests in the request queue that follow the first user request are processed first.

[0012] In one implementation of the first aspect, releasing the second target historical data in the processor includes: If the second model does not belong to the long activation class model, then release the second target historical data in the processor; The long activation class model is configured to remain in a wake-up state, and the processor includes the model parameters of the long activation class model and the historical inference data of the long activation class model.

[0013] In the above approach, for long-activation models, since they remain in a wake-up state, the transmission overhead caused by transferring historical inference data of long-activation models between the processor and memory can be reduced, which is beneficial to improving the inference efficiency of long-activation models. In particular, for some models with high call frequency, configuring them as long-activation models can greatly improve inference efficiency.

[0014] In one implementation of the first aspect, determining whether the processor currently has idle resources includes: If the number of third models is less than or equal to the preset number, or if there are no pending requests in the request queue, then it is determined that the processor currently has idle resources. The third model is the model that is currently awake in the processor, and the preset number is determined according to the number of models that the processor can execute in parallel.

[0015] In one implementation of the first aspect, the step of triggering the processor to retrieve first target historical data from the memory, and performing reasoning on the first user request based on the first target historical data and the first model to obtain a first reasoning result includes: The processor sends the first request to trigger the processor to retrieve the first target historical data from the memory based on the first identifier, and to infer the first user request based on the first model and the first target historical data to obtain the first inference result.

[0016] Understandably, the first user request typically involves a large number of bytes. If the processor obtains the first target historical data by sending the first user request to the memory, the transmission overhead would be significant. However, in the method described above, the processor only needs to send the first identifier to the memory to obtain the first target historical data. Since the first identifier is relatively small, this method can greatly reduce transmission overhead, thereby improving inference efficiency.

[0017] Secondly, embodiments of this application provide a reasoning system, including: The processor is used to deploy multiple models and perform inference tasks based on the deployed models. A memory for storing the historical inference data of each of the multiple models; A scheduler for executing the multi-model inference method described in any one of the first aspects above.

[0018] Thirdly, a multi-model inference device is provided, including: The receiving unit is used to receive the first user request; The switching unit is used to switch the first model from a dormant state to a wake-up state when the first model is in a dormant state; the first model is the model that the first user request needs to call from among the plurality of models; A triggering unit is configured to trigger the processor to retrieve first target historical data from the memory, and to perform reasoning on the first user request based on the first target historical data and the first model to obtain a first reasoning result; wherein, the processor stores model parameters of the first model, and the first target historical data is historical reasoning data that matches the first user request from the historical reasoning data of the first model.

[0019] Fourthly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-model inference method as described in any one of the first aspects above.

[0020] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-model inference method as described in any one of the first aspects above.

[0021] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the multi-model inference method described in any one of the first aspects.

[0022] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the inference system provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the multi-model inference method provided in an embodiment of this application; Figure 3 This is a schematic diagram of HBM occupancy provided in an embodiment of this application; Figure 4 This is a schematic diagram of the processing flow when there are idle resources provided in the embodiments of this application; Figure 5 This is a schematic diagram of the processing flow when there are no idle resources provided in the embodiments of this application; Figure 6 This is a structural block diagram of the multi-model inference device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0031] With the development of large-scale model technology, its application areas are becoming increasingly widespread, and the performance requirements for server-side devices (such as central processing units and image processors) deploying large-scale models are also increasing. Currently, high-bandwidth memory (HBM) space allocation technology and computing resource allocation technology can enable the deployment of multiple models on the server side.

[0032] In some application scenarios, inference requests for some models among multiple large models deployed on the server are relatively sparse, meaning that these models are called less frequently. This results in low utilization of server-side computing resources for these models, leading to wasted computing resources. Therefore, how to rationally allocate server-side computing resources and efficiently run multiple large models is an important issue.

[0033] Based on this, this application provides a multi-model inference method. In this embodiment, the processor stores model parameters in both the sleep and wake states. This eliminates the need to transmit model parameters during the switching between sleep and wake states, reducing transmission time and improving inference efficiency. In the sleep state, historical inference data of the model is stored in memory. When switching to the wake state, the historical inference data matching the user request (target historical data) is transmitted from memory to the processor for model retrieval. This method reduces the processor's space usage for historical inference data in the sleep state. Furthermore, when switching to the wake state, only the portion of historical inference data matching the user request is transmitted, without transmitting all historical inference data of the first model, reducing the transmission time of redundant information and further improving inference efficiency. This method reduces data transmission time during inference, improves inference efficiency, reduces processor space usage, saves computing resources, and achieves an effective balance between inference efficiency and resource utilization.

[0034] See Figure 1 This is a schematic diagram of an inference system provided in an embodiment of this application. The inference system includes a processor 11, a memory 12, and a scheduler 13. The processor 11 is connected to both the memory 12 and the scheduler 13.

[0035] The processor 11 deploys multiple models and performs inference tasks based on the deployed models. The memory 12 stores historical inference data for each of the multiple models. The scheduler 13 executes the multi-model inference method provided in this embodiment.

[0036] In some application scenarios, multiple models deployed in processor 11 can be used to execute different subtasks within the same inference task. For example, processor 11 deploys three models: model1, model2, and model3. For inference task Q, the first subtask Q1 of inference task Q is executed first using the first model, model1. Then, the next model is selected based on the result of subtask Q1. If the next model is determined to be model3, then model3 further executes subtask Q2 based on the result of subtask Q1. In this inference task, model1 and model3 are called, while model2 is not called. In the above example, model1 is called every time an inference task is performed, while model2 and model3 are called selectively. Therefore, in the above application scenario, there may be situations where one or more models are called less frequently.

[0037] In other application scenarios, multiple models deployed in processor 11 can be used to execute different inference tasks. For example, processor 11 may deploy three models: model1, model2, and model3. Model1 is used to execute inference task A, model2 is used to execute inference task B, and model3 is used to execute inference task C. In this application scenario, because users have different needs for different inference tasks, there may be situations where the execution frequency of one or more inference tasks is low, resulting in a low call frequency of the models corresponding to these inference tasks.

[0038] As described in the background section above, each model in processor 11 occupies a certain amount of HBM space. For models with low call frequency, continuously occupying HBM space can easily waste processor resources.

[0039] In some embodiments, the scheduler 13 may include a proxy module, a model invocation module, a scheduling module, and a request queue module. The proxy module receives user requests and related inference resources. The model scheduling module forwards user requests, performs inference, and forwards inference results. The scheduling module schedules inference resources. The request queue module stores pending user requests.

[0040] based on Figure 1 The inference system shown below illustrates the multi-model inference method provided in the embodiments of this application. See also... Figure 2 This is a flowchart illustrating the multi-model inference method provided in the embodiments of this application. Figure 2 The multi-model inference method shown can be derived from... Figure 1 The scheduler shown executes. As an example and not a limitation, the method may include the following steps: S101, Receive the first user request.

[0041] In this embodiment, the first user request is used to instruct the processor to perform a reasoning task. For example, the first user request may be user input information (such as voice information, text information, or image information), or the first user request may be a prompt generated based on the user input information.

[0042] S102, when the first model is in a dormant state, switch the first model from the dormant state to the awake state.

[0043] Wherein, the first model is the model that the first user request needs to call among the plurality of models.

[0044] In this embodiment, the model deployed in the processor 11 has sleep and wake-up capabilities, meaning the model can switch from a sleep state to a wake-up state, or vice versa. When the model is in a wake-up state, the processor 11 stores the model parameters and historical inference data. When the model switches from a wake-up state to a sleep state, the processor 11 still stores the model parameters and unloads the model's historical inference data to the memory 12, thereby saving HBM space in the processor 11.

[0045] In some implementations, the multiple models deployed in processor 11 can be divided into two types: long-activation models and hibernation-on-demand start models. Long-activation models are configured to remain in a wake-up state. Hibernation-on-demand start models are configured to switch between hibernation and wake-up states. For example, in the example above, processor 11 deploys three models: model1, model2, and model3. Each inference task necessarily calls model1, while model2 and model3 are called selectively. In this case, model1 can be configured as a long-activation model, while model2 and model3 can be configured as hibernation-on-demand start models.

[0046] Understandably, for long activation models, the processor 11 always stores the model parameters and historical inference data of the long activation model.

[0047] For example, see Figure 3 This is a schematic diagram of HBM occupancy provided in an embodiment of this application. Figure 3 As shown, Model 1 is a long-activation model, and Model 2 is a hibernation-on-demand startup model. Since the model parameters and historical inference data of Model 1 are always stored in the processor, the HBM usage of Model 1 in the processor remains constant. During the transition of Model 2 from wake-up to hibernation, the model parameters of Model 2 are always stored in the processor, meaning the HBM usage of Model 2's model parameters in the processor remains constant. However, as the historical inference data of Model 2 is gradually unloaded into memory, the HBM usage of Model 2's historical inference data in the processor gradually decreases to 0. During the transition of Model 2 from hibernation to wake-up, the model parameters of Model 2 are always stored in the processor, meaning the HBM usage of Model 2's model parameters in the processor remains constant. However, as the historical inference data of Model 2 is gradually loaded from memory into the processor, the HBM usage of Model 2's historical inference data in the processor gradually increases from 0.

[0048] The above approach allows for flexible implementation by configuring different models. Furthermore, for long-activation models, since they remain in a wake-up state, the transmission overhead of transferring historical inference data between the processor and memory can be reduced, thus improving the inference efficiency of long-activation models. This is especially true for models with high call frequencies; configuring them as long-activation models can significantly enhance inference efficiency.

[0049] S103, trigger the processor to retrieve the first target historical data from the memory, and perform reasoning on the first user request based on the first target historical data and the first model to obtain the first reasoning result.

[0050] The processor stores the model parameters of the first model, and the first target historical data is the historical inference data of the first model that matches the first user request.

[0051] In this embodiment, the processor stores model parameters in both the sleep and wake-up states. This eliminates the need to transmit model parameters during the switch between sleep and wake-up states, reducing transmission time and improving inference efficiency. In the sleep state, historical inference data is stored in memory. Upon switching to the wake-up state, the historical inference data matching the user request (target historical data) is transmitted from memory to the processor for model retrieval. This method reduces the processor's space usage for historical inference data in the sleep state. Furthermore, upon switching to the wake-up state, only the portion of historical inference data matching the user request is transmitted, eliminating the need to transmit all historical inference data from the first model. This reduces the transmission time of redundant information and further improves inference efficiency. By employing this method, we can reduce the time spent transmitting data during inference, improve inference efficiency, reduce processor space usage, save computing resources, and achieve an effective balance between inference efficiency and resource utilization.

[0052] In one embodiment, S102 includes: S201, determine whether there are any idle resources for the current processor.

[0053] In one implementation, S201 includes: If the number of third models is less than or equal to the preset number, or if there are no pending requests in the request queue, then it is determined that the current processor has idle resources.

[0054] The third model is the model that is currently awake in the processor, and the preset number is determined according to the number of models that the processor can execute in parallel.

[0055] S202, if the current processor has idle resources, a first request is sent to the memory to trigger the memory to filter out the first target historical data.

[0056] Understandably, during the inference process of a large model, one or more tokens are generated. In the next inference iteration, the next token can be predicted based on historically generated tokens, and then the newly generated token is added to the historical token list. Each token has a key (representing a context feature vector) and a value (representing a context content vector), i.e., KV. In the embodiments of this application, the historical inference data can refer to the KV of historical tokens.

[0057] The first request includes a first user request and a first identifier corresponding to the first user request, wherein the information length of the first identifier is less than the information length of the first user request.

[0058] In some implementations, the process of filtering the first target historical data in the memory includes: generating a first hash value from the first user request; searching in the stored historical reasoning data for a key that matches the first hash value; if a key that matches the first hash value and its corresponding value are identified as the first target historical data of the first user request.

[0059] Accordingly, in one implementation, S103 includes: A first request is sent to the processor to trigger the processor to retrieve the first target historical data from the memory based on the first identifier, and to infer the first user request based on the first model and the first target historical data to obtain the first inference result.

[0060] Understandably, the first user request typically involves a large number of bytes. If the processor obtains the first target historical data by sending the first user request to the memory, the transmission overhead would be significant. However, in the method described above, the processor only needs to send the first identifier to the memory to obtain the first target historical data. Since the first identifier is relatively small, this method can greatly reduce transmission overhead, thereby improving inference efficiency.

[0061] S203, switch the first model from sleep state to wake-up state.

[0062] For example, see Figure 4 This is a schematic diagram of the processing flow when idle resources exist, provided in an embodiment of this application. The processing flow may include the following steps: ① The scheduler's agent module receives the first user request.

[0063] ② The scheduler's agent module requests inference resources from the scheduler's scheduling module.

[0064] ③ If there are idle resources, the scheduling module sends a first request to the memory and wakes up the first model.

[0065] ④ After the proxy module obtains the resources, it sends the first user request to the model invocation module of the scheduler and notifies the model invocation module to execute inference; the model invocation module sends the first user request to the processor and notifies the processor to invoke the first model to execute inference.

[0066] ⑤ The processor retrieves the first target historical data that matches the first user request from memory.

[0067] ⑥ The processor infers the first user request based on the first target's historical data and outputs the inference result.

[0068] ⑦ After the inference is completed, the model calling module notifies the scheduling module that the inference requested by the first user has been completed, so that the scheduling model can release the relevant resources.

[0069] ⑧ The scheduling module switches the first model from the wake-up state to the sleep state. Accordingly, the processor sends the inference data of this time to the memory for storage, and then releases the historical inference data of the first model in the processor.

[0070] Through the above S201-S203 embodiments, the process of waking up the first model and the process of the memory filtering the first target historical data are processed in parallel. When the first model executes the inference process, the memory has completed or is about to complete the filtering of the first target historical data, which can reduce the time for the first model to load the first target historical data from the memory, thereby improving inference efficiency.

[0071] In one embodiment, after determining in S201 whether there are idle resources in the current processor, the method further includes: S204. If there are no idle resources on the current processor, add the first user request to the request queue.

[0072] S205, when the first user request is the first request in the request queue and the second model completes the inference task, the second target historical data is stored in the memory.

[0073] The embodiments of this application include the following two resource scheduling strategies.

[0074] Strategy 1: If the first request in the current request queue uses the same model as the most recently completed request, then process the first request in the current request queue. If the first request in the current request queue uses a different model than the most recently completed request, then check if there are any pending requests in the current request queue that use the same model as the most recently completed request. If so, prioritize processing the pending requests in the current request queue that use the same model as the most recently completed request. If not, switch the model called by the most recently completed request to a dormant state and switch the model called by the first request in the current request queue to a wake-up state.

[0075] For example, the request queue includes request 1, request 2, and request 3 in sequence. The most recently completed request 0 calls model a. If the model called by request 1 is also model a, then request 1 is processed directly; if the model called by request 1 is not model a, then it is determined whether the model called by request 2 is model a; if the model called by request 2 is model a, then request 1 is skipped and request 2 is processed first; after request 2 is processed, it is determined whether the model called by request 3 is model a; if the model called by request 3 is not model a, then model a is switched from the awakened state to the dormant state, model b called by request 1 is switched from the dormant state to the awakened state, and request 1 is processed.

[0076] Strategy 2: If there is a pending request in the current request queue that uses the same model as the most recently completed request, and that pending request is not the first request in the request queue, then the pending request will be processed first, and the skip count will be incremented by 1. When the skip count reaches a preset threshold, the currently called model will be put to sleep, the model to be called by the first request in the current request queue will be woken up, and the first request in the current request queue will be processed.

[0077] For example, the request queue includes request 1, request 2, request 3, and request 4 in sequence. The most recently completed request 0 calls model a. If the model required by request 1 is not model a, but the model required by request 2 is model a, then request 1 is skipped, request 2 is processed first, and the skip count is incremented by 1 (at this point, the skip count is 1). After request 2 is processed, if the model required by request 3 is also model a, then request 1 is skipped again, request 3 is processed first, and the skip count is incremented by 1 (at this point, the skip count is 2). When the skip count reaches the preset threshold of 2, it is no longer determined whether request 4 needs to call model a. Instead, model a is put to sleep, model b required by request 1 is awakened, request 1 is processed, and the skip count is reset (e.g., set to 0).

[0078] In one implementation, the methods for determining whether the reasoning task has been completed include: If the request queue does not contain any pending tasks corresponding to the second model, or if the number of skips corresponding to the second model reaches a preset threshold, then the second model is determined to have completed the inference task; where the number of skips represents the number of times the first user request is skipped and requests in the request queue that follow the first user request are processed first.

[0079] Understandably, in Strategy 1, if the request queue does not contain any pending tasks corresponding to the second model, then the second model is considered to have completed the inference task. In Strategy 2, if the number of skips corresponding to the second model reaches a preset threshold, then the second model is considered to have completed the inference task.

[0080] In this embodiment of the application, the second model is the model invoked by the second user request, and the second user request is the previous completed user request of the first user request; the second target historical data is the inference data obtained by the second model executing the inference task corresponding to the second user request.

[0081] It is understandable that the second user request may not necessarily be positioned before the first user request in the request queue. For example, in the example of Strategy 2 above, after request 0 is processed, request 2 is processed first, then request 3, and then request 1; if request 1 is recorded as the first user request, then request 3 is recorded as the second user request. However, request 3 is located after request 1 in the request queue.

[0082] S206, release the second target historical data in the processor; switch the second model from the wake-up state to the sleep state.

[0083] In one implementation, S206 includes: If the second model does not belong to the long activation class model, the second target historical data in the processor is released; wherein the long activation class model is configured to remain in the wake state, and the processor includes the model parameters of the long activation class model and the historical inference data of the long activation class model.

[0084] In the above approach, for long-activation models, since they remain in a wake-up state, the transmission overhead caused by transferring historical inference data of long-activation models between the processor and memory can be reduced, which is beneficial to improving the inference efficiency of long-activation models. In particular, for some models with high call frequency, configuring them as long-activation models can greatly improve inference efficiency.

[0085] S207, send the first request to the memory.

[0086] S208, switch the first model from sleep state to wake-up state.

[0087] For example, see Figure 5 This is a schematic diagram of the processing flow when there are no idle resources, provided in an embodiment of this application. The processing flow may include the following steps: 1) The scheduler's agent module receives the first user request.

[0088] 2) The scheduler's agent module requests inference resources from the scheduler's scheduling module.

[0089] 3) If no idle resources are available, the scheduling module will add the first user request to the request queue.

[0090] 4) The model calling module notifies the scheduling module that the second user has requested the inference to be completed.

[0091] 5) The scheduling module schedules requests and determines the first user request to be processed based on the scheduling strategy.

[0092] 6) The scheduling module hibernates the second model and wakes up the first model.

[0093] 7) The scheduling module sends the first request to the memory.

[0094] Steps 6) and 7) can be processed in parallel.

[0095] 8) After the proxy module obtains the resources, it sends the first user request to the model invocation module of the scheduler and notifies the model invocation module to execute inference; the model invocation module sends the first user request to the processor and notifies the processor to invoke the first model to execute inference.

[0096] 9) The processor retrieves the first target historical data that matches the first user request from memory.

[0097] 10) The processor infers the first user request based on the first target's historical data and outputs the inference result.

[0098] 11) After the inference is completed, the model calling module notifies the scheduling module that the inference requested by the first user has been completed.

[0099] The scheduling module determines, based on the scheduling policy, whether the model to be invoked for the next pending request is the first model. If so, the scheduling module continues to trigger the relevant process for the next pending request; otherwise, the scheduling module switches the first model from the awakened state to the sleep state. Accordingly, the processor sends the inference data of this time to the memory for storage, and then releases the historical inference data of the first model from the processor.

[0100] In the embodiments of steps S204-S208, when there are no idle resources, the scheduler can automatically allocate resources according to the scheduling policy, and wake up and hibernate related models as needed according to the resource allocation, thereby achieving a reasonable allocation of processor resources and saving computing resources.

[0101] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0102] Corresponding to the multi-model inference method described in the above embodiments, Figure 6 This is a structural block diagram of the multi-model inference device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0103] Reference Figure 6 The device 6 includes: The receiving unit 61 is used to receive the first user request.

[0104] The switching unit 62 is used to switch the first model from a dormant state to a wake-up state when the first model is in a dormant state; the first model is the model that the first user request needs to call among the plurality of models.

[0105] Triggering unit 63 is used to trigger the processor to retrieve first target historical data from the memory, and to perform reasoning on the first user request based on the first target historical data and the first model to obtain a first reasoning result; wherein, the processor stores model parameters of the first model, and the first target historical data is historical reasoning data that matches the first user request in the historical reasoning data of the first model.

[0106] Optionally, the switching unit 62 is also used for: Determine whether the processor currently has any idle resources; If the processor currently has idle resources, a first request is sent to the memory to trigger the memory to filter out the first target historical data according to the first request; wherein, the first request includes the first user request and a first identifier corresponding to the first user request; Switch the first model from sleep state to wake state.

[0107] Optionally, the switching unit 62 is also used for: After determining whether the processor has any idle resources, if the processor does not have any idle resources, the first user request is added to the request queue. When the first user request is the first request in the request queue and the second model completes the inference task, the second target historical data is stored in the memory; wherein, the second model is the model called by the second user request, and the second user request is the previous completed user request of the first user request; the second target historical data is the inference data obtained by the second model executing the inference task corresponding to the second user request; Release the second target historical data from the processor; switch the second model from the wake-up state to the sleep state; Send the first request to the memory; Switch the first model from sleep state to wake state.

[0108] Optionally, the switching unit 62 is also used for: After the first user request is added to the request queue, if the request queue does not contain the pending task corresponding to the second model, or if the number of skips corresponding to the second model reaches a preset threshold, then the second model is determined to have completed the inference task. The number of times to skip indicates the number of times the first user request is skipped and requests in the request queue that follow the first user request are processed first.

[0109] Optionally, the switching unit 62 is also used for: If the second model does not belong to the long activation class model, then release the second target historical data in the processor; The long activation class model is configured to remain in a wake-up state, and the processor includes the model parameters of the long activation class model and the historical inference data of the long activation class model.

[0110] Optionally, the switching unit 62 is also used for: If the number of third models is less than or equal to the preset number, or if there are no pending requests in the request queue, then it is determined that the processor currently has idle resources. The third model is the model that is currently awake in the processor, and the preset number is determined according to the number of models that the processor can execute in parallel.

[0111] Optionally, the trigger unit 63 is also used for: The processor sends the first request to trigger the processor to retrieve the first target historical data from the memory based on the first identifier, and to infer the first user request based on the first model and the first target historical data to obtain the first inference result.

[0112] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0113] in addition, Figure 6 The multi-model inference device shown can be a software unit, a hardware unit, or a combination of software and hardware built into an existing terminal device, or it can be integrated into the terminal device as an independent component, or it can exist as an independent terminal device.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0115] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 7 As shown, the terminal device 7 of this embodiment includes: at least one processor 70 ( Figure 7 (Only one is shown in the diagram) a processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, wherein the processor 70 executes the computer program 72 to implement the steps in any of the above-described multi-model inference method embodiments.

[0116] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 7 The example of terminal device 7 is merely an illustration and does not constitute a limitation on terminal device 7. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0117] The processor 70 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0118] In some embodiments, the memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. In other embodiments, the memory 71 may be an external storage device of the terminal device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 7. Furthermore, the memory 71 may include both internal and external storage units of the terminal device 7. The memory 71 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0119] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0120] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments.

[0121] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0124] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multi-model inference method, characterized in that, The multi-model inference method is applied to an inference system, which includes a processor and a memory. The processor deploys multiple models, and the memory stores historical inference data for each of the multiple models. Receive the first user request; When the first model is in a dormant state, the first model is switched from the dormant state to the awakened state; the first model is the model that the first user request needs to call among the plurality of models; The processor is triggered to retrieve first target historical data from the memory, and to perform reasoning on the first user request based on the first target historical data and the first model to obtain a first reasoning result; wherein, the processor stores model parameters of the first model, and the first target historical data is historical reasoning data that matches the first user request from the historical reasoning data of the first model.

2. The multi-model inference method as described in claim 1, characterized in that, Switching the first model from a dormant state to a wake-up state includes: Determine whether the processor currently has any idle resources; If the processor has idle resources, a first request is sent to the memory to trigger the memory to filter out the first target historical data; Switch the first model from sleep state to wake state.

3. The multi-model inference method as described in claim 2, characterized in that, After determining whether the processor currently has idle resources, the method further includes: If the processor currently has no idle resources, then the first user request is added to the request queue; When the first user request is the first request in the request queue and the second model completes the inference task, the second target historical data is stored in the memory; wherein, the second model is the model called by the second user request, and the second user request is the previous completed user request of the first user request; the second target historical data is the inference data obtained by the second model executing the inference task corresponding to the second user request; Release the second target historical data from the processor; switch the second model from the wake-up state to the sleep state; Send the first request to the memory; Switch the first model from sleep state to wake state.

4. The multi-model inference method as described in claim 3, characterized in that, After adding the first user request to the request queue, the method further includes: If the request queue does not contain any pending tasks corresponding to the second model, or if the number of skips corresponding to the second model reaches a preset threshold, then the second model is determined to have completed the inference task. The number of times to skip indicates the number of times the first user request is skipped and requests in the request queue that follow the first user request are processed first.

5. The multi-model inference method as described in claim 3, characterized in that, The release of the second target historical data in the processor includes: If the second model does not belong to the long activation class model, then release the second target historical data in the processor; The long activation class model is configured to remain in a wake-up state, and the processor includes the model parameters of the long activation class model and the historical inference data of the long activation class model.

6. The multi-model inference method as described in claim 2, characterized in that, The determination of whether the processor currently has idle resources includes: If the number of third models is less than or equal to the preset number, or if there are no pending requests in the request queue, then it is determined that the processor currently has idle resources. The third model is the model that is currently awake in the processor, and the preset number is determined according to the number of models that the processor can execute in parallel.

7. The multi-model inference method as described in claim 2 or 3, characterized in that, The triggering of the processor to retrieve first target historical data from the memory, and to perform reasoning on the first user request based on the first target historical data and the first model to obtain a first reasoning result, includes: The processor is sent the first request to trigger the processor to retrieve the first target historical data from the memory according to the first identifier, and to infer the first user request based on the first model and the first target historical data to obtain the first inference result; The first request includes the first user request and a first identifier corresponding to the first user request; the information length of the first identifier is less than the information length of the first user request.

8. A reasoning system, characterized in that, include: The processor is used to deploy multiple models and perform inference tasks based on the deployed models. A memory for storing the historical inference data of each of the multiple models; A scheduler for executing the multi-model inference method as described in any one of claims 1 to 7.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images