Model reasoning method, reasoning system, equipment, storage medium and program product
By integrating the model inference system on the server side, and self-provisioning and allocating model and computing resources, the problem of low efficiency in large-scale model deployment and fine-tuning is solved, and efficient model inference and resource utilization is achieved.
Patent Information
- Application Number
- CN202411999417.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
In the prior art, the deployment and fine-tuning of large models are low, especially on terminal devices, where resource redundancy and low utilization of graphics card resources are problems.
It provides a model inference method and inference system, integrating the resources required by different demanders in model inference. Through the data management module, resource scheduling module and inference module, the model, training data and computing resources are allocated and allocated by itself to realize intelligent model inference deployment and processing.
It greatly reduces the difficulty of model deployment, improves model deployment efficiency and reasoning efficiency, maximizes resource utilization, reduces resource waste, and eliminates manual operations.
Smart Images
Figure CN119918672A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to but is not limited to the field of computer technology, and in particular to a model reasoning method and reasoning system, device, storage medium, and program product. Background Art
[0002] A large model refers to a natural language processing model trained based on a deep learning algorithm. After pre-training, a large model can acquire general capabilities to solve various tasks. However, to truly apply a large model to actual business, it is necessary to fine-tune the capabilities of the large model according to the business scenario, such as fine-tuning and adapting the domain. Currently, the best way to fine-tune a large model is to use Low-Rank Adaptation of Large Language Models (Lora) based on large language models.
[0003] In the related technology, model fine-tuning based on Lora is implemented locally, and the codes used in deployment on different terminals are often repeated, resulting in low model deployment efficiency and low model fine-tuning efficiency. Summary of the invention
[0004] In view of this, the embodiments of the present disclosure at least provide a model reasoning method and reasoning system, device, storage medium, and program product.
[0005] The technical solution of the embodiment of the present disclosure is implemented as follows:
[0006] On the one hand, an embodiment of the present disclosure provides a model reasoning method, which is applied to a reasoning system, wherein the reasoning system is deployed on a server, and the reasoning system integrates resources required by different demand parties during model reasoning. The model reasoning method includes:
[0007] In response to a first request for model inference from any demander, based on the indication information of the first request, respectively deploy the first type of model, the second type of model, and the training data that match the first request; the first type of model is used to control the model's processing capability for overall features, and the second type of model is used to control the model's processing capability for local features;
[0008] Allocate a first computing resource to the first request based on the amount of the training data and the amount of parameters of the first model and the second model;
[0009] Based on the training data, the first type of model, the second type of model, and the first computing resource, a model inference result of the first request is determined.
[0010] On the other hand, an embodiment of the present disclosure provides an inference system, which is deployed on a server, and which integrates resources required by different demand parties during model inference. The inference system includes: a data management module, a resource scheduling module, and an inference module;
[0011] In response to a first request for model inference from any demander, the data management module is used to allocate the first type of model, the second type of model, and the training data matching the first request based on the indication information of the first request; the first type of model is used to control the processing capability of the model for overall features, and the second type of model is used to control the processing capability of the model for local features;
[0012] The resource scheduling module is used to allocate a first computing resource to the first request based on the data volume of the training data and the parameter volume of the first type of model and the second type of model;
[0013] The reasoning module is used to determine the model reasoning result of the first request based on the training data, the first type of model, the second type of model, and the first computing resource.
[0014] On the other hand, an embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements some or all of the steps in the above method when executing the program.
[0015] On the other hand, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements part or all of the steps in the above method when executed by a processor.
[0016] On the other hand, an embodiment of the present disclosure provides a computer program, including computer-readable codes. When the computer-readable codes are executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.
[0017] On the other hand, an embodiment of the present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, it implements some or all of the steps in the above method.
[0018] In the disclosed embodiment, the inference system integrates the resources required by different demand parties for model inference; in this way, the model inference resources of different demand parties are concentrated in the non-local inference system, so as to uniformly allocate the model inference resources through the inference system. In response to the first request for model inference, the inference system can allocate the various resources (first type of model, second type of model, and training data) required for the first request by itself, and complete the deployment and processing of model inference based on the allocated resources. The whole process is intelligent and does not require manual operation, which greatly reduces the difficulty of deployment and improves the efficiency of model deployment and model inference. In addition, the first computing resource is allocated to the first request based on the data volume of the training data and the parameter volume of the first type of model and the second type of model, so that resource scheduling can be performed according to the actual resource demand of the first request, so as to maximize resource utilization and reduce resource waste.
[0019] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.
[0021] Figure 1 A schematic diagram of the structure of an inference system provided in the embodiment of the present disclosure Figure 1 ;
[0022] Figure 2 A schematic diagram of the structure of an inference system provided in the embodiment of the present disclosure Figure 2 ;
[0023] Figure 3 A schematic diagram of the implementation process of a model reasoning method provided in the embodiment of the present disclosure Figure 1 ;
[0024] Figure 4 A schematic diagram of the implementation process of a model reasoning method provided in the embodiment of the present disclosure Figure 2 ;
[0025] Figure 5 A schematic diagram of the implementation process of a model reasoning method provided in the embodiment of the present disclosure Figure 3 ;
[0026] Figure 6 A schematic diagram of the implementation process of a model reasoning method provided in the embodiment of the present disclosure Figure 4 ;
[0027] Figure 7 A schematic diagram of the structure of an inference system provided in the embodiment of the present disclosure Figure 3 ;
[0028] Figure 8 A schematic diagram of the implementation process of a model reasoning method provided in the embodiment of the present disclosure Figure 5 ;
[0029] Fig. 9 A schematic diagram of the implementation process of a model reasoning method provided in the embodiment of the present disclosure Figure 6 ;
[0030] Fig.10 A schematic diagram of the structure of a model inference device provided in an embodiment of the present disclosure;
[0031] Fig.11 A hardware entity diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0033] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0034] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.
[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the present disclosure and are not intended to limit the present disclosure.
[0036] In order to better understand the model reasoning method provided by the embodiment of the present disclosure, the solution in the related technology is first described below.
[0037] Lora fine-tuning is also called Lora inference. When deployed locally, users need to log in to the server first, configure the large model and Lora model on the server, upload training data, and check whether there are graphics card resources on the server. If the graphics card resources are occupied, you need to manually monitor the status of the graphics card resources. Model inference will only be performed when the graphics card resources are idle.
[0038] There are at least the following technical problems in the related technology:
[0039] 1. Local deployment has the problem of resource redundancy, which is mainly reflected in:
[0040] First, when deploying Lora fine-tuning on different terminals, you need to compile a set of codes, which are often repeated and have code redundancy problems.
[0041] Second, during the Lora inference process, the same large model may be loaded multiple times, resulting in resource redundancy.
[0042] 2. The utilization rate of the graphics card resources of the local terminal is low during model inference, which is mainly reflected in:
[0043] First, the number of graphics cards is specified by the user, and some training does not require so many graphics cards. For example, to fine-tune a 6B model, only one graphics card is enough, but users often set it to 8 graphics cards, resulting in a serious waste of graphics card resources.
[0044] Second, users need to monitor manually. After detecting that the graphics card is idle, they can start their own model inference, but manual monitoring cannot be achieved 24 hours a day.
[0045] To this end, the embodiment of the present disclosure provides an inference system, which is deployed on a server and integrates the resources required by different demand parties during model inference, such as Figure 1 As shown, the reasoning system 10 includes: a data management module 11, a resource scheduling module 12 and a reasoning module 13;
[0046] In response to a first request for model inference from any demander, the data management module 11 is used to allocate the first type of model, the second type of model, and the training data matching the first request based on the indication information of the first request; the first type of model is used to control the processing capability of the model for overall features, and the second type of model is used to control the processing capability of the model for local features;
[0047] The resource scheduling module 12 is used to allocate a first computing resource to the first request based on the data volume of the training data and the parameter volume of the first type of model and the second type of model;
[0048] The reasoning module 13 is used to determine the model reasoning result of the first request based on the training data, the first type of model, the second type of model, and the first computing resource.
[0049] Here, the inference system 10 is used to deploy model inference tasks online and execute model inference tasks. The inference system 10 includes multiple graphics card resources, each of which can be used to provide video memory resources and computing power resources. During implementation, since the server cluster has huge computing resources and can support complex computing tasks, the inference system 10 can be deployed in the server cluster.
[0050] The inference system 10 integrates multiple types of resources required by different demand parties for model inference. Multiple types of resources refer to multiple different types of resources such as graphics cards, models, training data, etc. required for model inference. For example, multiple types of resources may include but are not limited to: multiple models, multiple training data, and graphics card resources, etc.; wherein graphics card resources may include video memory resources and computing resources.
[0051] In some embodiments, the multiple models may include a first type of model and a second type of model. The first type of model may be a large model; for example, the large model may refer to one or more of the Language Learning with Adaptive Multi-task Architecture (Llama), Baichuan, Chat General Language Model (ChatGLM), Generative Pre-trained Transformer 3 (GPT-3), Big Science Large Open-science Open-access Multilingual (Bloom), MusaChat model, etc. The second type of model may be a Lora model; for example, a Lora model may refer to a variety of adapters built on the basis of the basic Lora model.
[0052] In some implementations, the various training data may include, but are not limited to: voice data, image data, text data, video data, etc.
[0053] The data management module 11, resource scheduling module 12 and reasoning module 13 can be collectively referred to as the backend of the reasoning system 10, which is used to respond to requests for model reasoning sent by users and complete corresponding model reasoning tasks by executing the underlying code of model reasoning.
[0054] The data management module 11 is used to manage data resources in the inference system. Data resources include: models, training data, model parameters, etc. Specifically, the large model can be managed by uploading, modifying, and deleting the large model; the training data can be managed by uploading, modifying, and deleting the training data; the Lora model can be managed by uploading, modifying, and deleting the Lora model. In addition, the data management module 11 is also used to allocate the first type model, the second type model, and the training data required by any demand party.
[0055] The resource scheduling module 12 is used to schedule computing resources in the reasoning system. Specifically, the resource scheduling module 12 is used to allocate computing resources such as video memory resources and computing power resources to any demand party.
[0056] The reasoning module 13 is used to perform model reasoning tasks. Specifically, the reasoning module 13 is used to perform model reasoning tasks of any demand side based on training data, the first type of model, the second type of model, and computing resources.
[0057] In some embodiments, Figure 2 As shown, the reasoning system 10 may further include a human-computer interaction interface 14. The human-computer interaction interface 14 is used to display multiple types of resources in the reasoning system to the user, and trigger a request for model reasoning based on the user's selected operation on the multiple types of resources.
[0058] The human-computer interaction interface 14 refers to the front end of the reasoning system 10 and is used to interact with the user. Specifically, the human-computer interaction interface 14 is used to display multiple types of resources in the reasoning system to the user for viewing and selection; and trigger a request for model reasoning based on the resources selected by the user.
[0059] It should be noted that the reasoning system 10 integrates the resources required by different demand parties during model reasoning in order to uniformly store the resources required by different demand parties during model reasoning in the reasoning system so as to uniformly allocate the relevant resources for model reasoning through the reasoning system. This method of uniformly allocating resources only requires retaining a copy of the resources and execution code on the server side. Compared with the method of configuring resources locally in the related technology, there will be no resource redundancy and resources can be maximized.
[0060] In the disclosed embodiment, the inference system integrates the resources required by different demand parties for model inference; in this way, the model inference resources of different demand parties are concentrated in the non-local inference system, so that the model inference resources can be uniformly allocated through the inference system, without the need for additional deployment and without resource redundancy; furthermore, resources can be scheduled according to the actual resource requirements of the model inference task, and an appropriate amount of resources can be allocated to each model inference task to maximize resource utilization and reduce resource waste. Moreover, in response to a request for model inference (i.e., a model inference task), the inference system can allocate the resources required for the model inference task by itself, and complete the deployment and processing of the model inference based on the allocated resources. The entire process is intelligent and does not require manual operation, which greatly reduces the difficulty of deployment and improves the efficiency of model deployment and model inference.
[0061] The embodiment of the present disclosure provides a model reasoning method, which can be applied to the above-mentioned reasoning system.
[0062] like Figure 3 As shown, the method includes the following steps 301 to 303:
[0063] Step 301, respond to any demand party's first request for model inference, and based on the indication information of the first request, respectively deploy the first type of model, the second type of model, and the training data that match the first request; the first type of model is used to control the model's processing capability for overall features, and the second type of model is used to control the model's processing capability for local features.
[0064] The demander refers to an object with model reasoning requirements. The first request refers to a request for model reasoning triggered by any demander. A request for model reasoning corresponds to one or more model reasoning tasks. The indication information of the first request is used to reflect the model reasoning requirements of the demander.
[0065] Specifically, the indication information may include: indication features of the first type of model, indication features of the second type of model, and indication features of the training data. Among them, the indication features of the first type of model may be the field of the first type of model, the identifier of the first type of model; the field of the first type of model may include: graphics processing field, text processing field, speech processing field, etc. The indication features of the second type of model may be the identifier of the second type of model. The indication features of the training data may be the field of the training data, the identifier of the training data. For example: the first type of model may refer to a large model, and the second type of model may refer to a Lora model.
[0066] In some embodiments, the specific implementation method for determining the first request regarding model reasoning can be: after the user operates multiple types of resources on the human-computer interaction interface, the backend automatically generates a first request based on the multiple types of resources operated by the user; the first request carries the resource identifier of the user operation.
[0067] In some implementations, the operations performed by the user on the multiple types of resources on the human-computer interaction interface may include, but are not limited to: input operations, selection operations, and adjustment operations.
[0068] In some implementations, step 301 may be specifically implemented by respectively allocating the first type of model, the second type of model, and the training data that match the first request based on the resource identifier of the user operation.
[0069] Step 302: Allocate a first computing resource to the first request based on the amount of the training data and the amount of parameters of the first type of model and the second type of model.
[0070] The first computing resource refers to a computing resource required for the first request, and the computing resource may refer to a graphics card resource.
[0071] In some embodiments, the specific implementation method of step 302 can be: based on the data volume of training data and the parameter volume of the first type model and the second type model, determine the first computing resources required for processing the first request; and allocate the first computing resources to the first request.
[0072] Step 303: determine the model inference result of the first request based on the training data, the first type of model, the second type of model, and the first computing resource.
[0073] The model inference result can refer to the inferred model parameters, the inferred model, or the application result inferred by the model. Specifically, it can be set according to actual business needs. If the focus is to obtain a model that can be used directly, then the inference result can be the inferred model; if the focus is to obtain a model parameter with reference, then the inference result can be the inferred model parameter; if the focus is to obtain an image segmentation result, then the inference result can be the inferred image segmentation result.
[0074] In some embodiments, if the first type of model and the second type of model indicated by the first request are both trained, model training is not required, and the first type of model and the second type of model can be directly used to infer the data selected by the first request to obtain model inference results.
[0075] Since the second type of model in the embodiment of the present disclosure is to enable the first type of model to be suitable for different application scenarios, during training, the model parameters of the second type of model can be adjusted, while the model parameters of the first type of model are not adjusted.
[0076] In some embodiments, if the second type of model indicated by the first request is an untrained model, model training is required. At this time, the second type of model is optimized based on the training data and the first type of model, and the first type of model and the optimized second type of model are used to infer the data selected by the first request to obtain the model inference result.
[0077] For example: If the first request is to obtain a LoRa model suitable for the target field, the specific implementation method of step 303 can be: call the first computing resource, optimize the model parameters of the second type of model based on the training data (training data of the target field) and the first type of model; and use the optimized second type of model as the LoRa model suitable for the target field.
[0078] For example: if the first request is to obtain the segmentation result of the image to be processed, and the second type of model selected in the first request is a trained model, the specific implementation method of step 303 can be: use the second type of model and the first type of model to process the image to be processed respectively; merge the output of the first type of model and the output of the second type of model to obtain the segmentation result of the image to be processed.
[0079] For example: if the first request is to obtain the segmentation result of the image to be processed, and the second type of model selected in the first request is an untrained model, the specific implementation method of step 303 can be: calling the first computing resource, optimizing the model parameters of the second type of model based on the training data and the first type of model; using the optimized second type of model and the first type of model to process the image to be processed respectively; merging the output of the first type of model and the output of the second type of model to obtain the segmentation result of the image to be processed.
[0080] In the disclosed embodiment, the inference system integrates the resources required by different demand parties for model inference; in this way, the model inference resources of different demand parties are concentrated in the non-local inference system, so as to uniformly allocate the model inference resources through the inference system. In response to the first request for model inference, the inference system can allocate the various resources (first type of model, second type of model, and training data) required for the first request by itself, and complete the deployment and processing of model inference based on the allocated resources. The whole process is intelligent and does not require manual operation, which greatly reduces the difficulty of deployment and improves the efficiency of model deployment and model inference. In addition, the first computing resource is allocated to the first request based on the data volume of the training data and the parameter volume of the first type of model and the second type of model, so that resource scheduling can be performed according to the actual resource demand of the first request, so as to maximize resource utilization and reduce resource waste.
[0081] The present disclosure provides a model reasoning method, such as Figure 4 As shown, the method includes the following steps 401 to 406:
[0082] Step 401, respond to a first request for model inference from any demander, and based on the indication information of the first request, respectively deploy the first type of model, the second type of model, and the training data that match the first request; the first type of model is used to control the model's processing capability for overall features, and the second type of model is used to control the model's processing capability for local features.
[0083] Here, step 401 corresponds to the aforementioned step 301 , and the specific implementation of the aforementioned step 301 may be referred to during implementation.
[0084] In some embodiments, the specific implementation method of step 401 may be: based on the user's selection operation on multiple first-category models on the human-computer interaction interface, determine the first-category model; based on the user's selection operation on multiple second-category models on the human-computer interaction interface, determine the second-category model; based on the user's input operation on the model parameters on the human-computer interaction interface, determine the model parameters; based on the user's selection operation on multiple types of training data on the human-computer interaction interface, determine the training data corresponding to the first model.
[0085] Step 402: Determine the first graphics memory resources and the first computing power resources required for processing the first request based on the amount of the training data and the amount of parameters of the first model and the second model.
[0086] The first video memory resource refers to the video memory resource required for processing the first request. The first computing power resource refers to the computing power resource required for processing the first request.
[0087] Step 403: Determine the first computing resource based on the first graphics memory resource and the first computing power resource.
[0088] Step 404: Allocate a first computing resource to the first request.
[0089] Here, step 402 to step 404 correspond to the aforementioned step 302, and the specific implementation of the aforementioned step 302 may be referred to during implementation.
[0090] When processing model inference tasks, not only video memory resources but also computing power resources are required. Therefore, when allocating resources, both video memory resources and computing power resources need to be allocated at the same time.
[0091] In some implementations, a resource allocation strategy may be used to allocate the first computing resource to the first request. The resource allocation strategy may include, but is not limited to, priority allocation, fair allocation, and maximum utilization strategies.
[0092] Step 405: optimize the second type of model based on the training data, the first type of model, and the first computing resources to obtain an optimized second type of model.
[0093] In some implementations, step 405 may be implemented by steps 4051 to 4052 as follows:
[0094] Step 4051: call the first computing resource to train the first type of model and the second type of model respectively based on the training data.
[0095] The first type of model is a general large model, so it does not need to be trained. Here, the first type of model is trained based on the training data in order to obtain the output of the first type of model so as to optimize the second type of model based on the output of the first type of model. If the first type of model also needs to be trained, then the first type of model and the second type of model need to be optimized at the same time.
[0096] In some embodiments, if the first type of model and the second type of model are deployed on different devices, the first type of model and the second type of model are trained simultaneously based on the training data; if the first type of model and the second type of model are deployed on the same device, the first type of model and the second type of model are trained serially.
[0097] Specifically, if the first type of model and the second type of model are deployed on the same device, the first type of model can be trained based on the training data first, and then the second type of model can be trained based on the training data; or, the second type of model can be trained based on the training data first, and then the first type of model can be trained based on the training data.
[0098] In some implementations, batch processing techniques may be used to optimize the operation of training the first type of model based on the training data and to optimize the operation of training the second type of model based on the training data.
[0099] Step 4052: During the training process, based on the output of the first type of model and the output of the second type of model, the model parameters of the second type of model are continuously optimized until the iteration stop condition is reached, thereby obtaining the optimized second type of model.
[0100] The model parameters of the second type of model may include two parameter matrices. The iteration stop condition may be that the number of iterations meets a preset number threshold, or the accuracy of the second type of model meets a preset accuracy threshold, etc.
[0101] In some embodiments, the specific implementation method of step 4052 may be: during the training process, determining the loss value based on the output of the first type of model and the output of the second type of model; based on the loss value, adjusting the model parameters of the second type of model until the iteration stop condition is reached, thereby obtaining the optimized second type of model.
[0102] Step 406: Infer the to-be-processed data of the first request based on the optimized second-type model and the first-type model to obtain the model inference result.
[0103] The data to be processed can be data input by the user or data specified from training data.
[0104] In some implementations, step 406 may be implemented by steps 4061 to 4063 as follows:
[0105] Step 4061: Use the first type of model to infer the data to be processed to obtain a first inference result.
[0106] The first inference result refers to the processing result of the first type of model on the data to be processed.
[0107] In some embodiments, the specific implementation method of step 4061 may be: using a first optimization algorithm to optimize the operation of using the first type of model to infer the data to be processed.
[0108] The first optimization algorithm is used to improve the model reasoning rate. For example, the first optimization algorithm may be a kv cache technology.
[0109] During implementation, the kv cache technology can be used to optimize the process of using the first type of model to perform reasoning on the data to be processed, so as to quickly obtain the reasoning results.
[0110] Step 4062: Use the optimized second type model to infer the data to be processed to obtain a second inference result.
[0111] The second inference result refers to the processing result of the second type of model on the data to be processed.
[0112] In some embodiments, the specific implementation method of step 4062 may be: using a second optimization algorithm to optimize the operation of using the optimized second type model to infer the data to be processed to obtain a second inference result.
[0113] The second optimization algorithm may be the same as or different from the first optimization algorithm.
[0114] In some embodiments, if the first type of model and the second type of model are deployed on different devices, the first type of model is used to infer the data to be processed, and the optimized second type of model is used to infer the data to be processed at the same time.
[0115] Specifically, if the first type of model and the second type of model are deployed on the same device, the first type of model can be trained based on the training data first, and then the second type of model can be trained based on the training data; or, the second type of model can be trained based on the training data first, and then the first type of model can be trained based on the training data.
[0116] Step 4063: Determine the model reasoning result based on the first reasoning result and the second reasoning result.
[0117] In some implementations, step 4063 may be specifically implemented by merging the first reasoning result and the second reasoning result to obtain a model reasoning result.
[0118] Based on the above embodiments, the model reasoning method provided by the embodiment of the present disclosure further includes the following steps 501 to 506:
[0119] Step 501: When the first request is received, cache the first request in a message queue.
[0120] Since multiple requests may be received in the same time period, the inference system may maintain a message queue to be processed, which stores unprocessed requests for model inference.
[0121] It should be noted that caching the first request in the message queue can reduce the situation where the request is lost due to untimely processing and improve the reliability of request processing.
[0122] Step 502: Determine the first computing resources required for processing the first request based on the amount of the training data and the amount of parameters of the first type of model and the second type of model.
[0123] In some embodiments, the specific implementation method of step 502 may be: determining the association relationship between the data volume and the computing resources, and the association relationship between the model parameter quantity and the computing resources; from the association relationship between the data volume and the computing resources, determining the computing resources that match the data volume of the training data; from the association relationship between the model parameter quantity and the computing resources, determining the computing resources that match the parameter quantities of the first type of model and the second type of model; based on the computing resources that match twice, determining the first computing resource.
[0124] For example, the relationship between model parameter quantity and computing resources can be: the computing resources required for a model with 1M (mega) parameters are 4M. At this time, if the parameter quantity of the first type model and the second type model is 10, then the computing resources corresponding to the first type model and the second type model are 8M.
[0125] For example, the correlation between the amount of training data and computing resources can be: 8M computing resources are required for 1M of training data. At this time, if the amount of training data is 2M, the computing resources corresponding to the training data are 16M. In the case where the computing resources corresponding to the first type of model and the second type of model are 8M and the computing resources corresponding to the training data are 16M, the first computing resource can be 24M.
[0126] In some embodiments, the specific implementation of step 502 may also be: determining the first computing resource based on a configuration operation of the computing resource by the user on the computer interaction interface.
[0127] For example, the specific implementation method of determining the first computing resource may be: the user directly configures the computing resource through the human-computer interaction interface, and the computing resource configured by the user is used as the first computing resource. For example, if the computing resource required for the first model during training is 1G, but the user considers the need for additional training data in the future and configures the computing resource to 3G, then the first computing resource is 3G.
[0128] Step 503: Determine whether a second idle computing resource in the reasoning system matches the first computing resource.
[0129] The second computing resource refers to the computing resource that is currently idle in the inference system. In implementation, if the inference system includes 5 graphics cards, the computing resource of each graphics card is 2GB (gigabytes), and these computing resources are all idle, then the second computing resource can be 10GB.
[0130] Whether the second computing resource matches the first computing resource refers to whether the second computing resource is greater than or equal to the first computing resource. If the second computing resource matches the first computing resource, it means that the second computing resource is greater than or equal to the first computing resource, and the idle computing resources in the inference system can meet the computing resource requirements of the model inference task corresponding to the first request. If the second computing resource does not match the first computing resource, it means that the second computing resource is less than the first computing resource, and the idle computing resources in the inference system cannot meet the computing resource requirements of the model inference task corresponding to the first request.
[0131] In some implementations, step 503 may be specifically implemented by comparing the first computing resource with the second computing resource to determine whether the second computing resource is greater than or equal to the first computing resource.
[0132] It should be noted that after step 503, step 504 or steps 505 to 406 may be executed.
[0133] Step 504: When the first request is the request with the earliest cache time in the message queue and the second computing resource matches the first computing resource, take out the first request from the message queue to process the first request.
[0134] In some implementations, requests for model reasoning in the message queue are processed in the order of cache time. Therefore, when the first request is the request with the earliest cache time in the message queue and the first computing resource matches the second computing resource, the first request is taken out of the message queue to process the model reasoning task corresponding to the first request. Processing the model reasoning task corresponding to the first request can be implemented through the aforementioned steps 401 to 406.
[0135] Step 505: When the first request is the request with the earliest cache time in the message queue, but the second computing resource does not match the first computing resource, monitor a third request for resource release.
[0136] The third request refers to a request for releasing resources.
[0137] The reasoning system in the embodiment of the present disclosure can automatically monitor resource release. In this way, when the first computing resource does not match the second computing resource, the third request for resource release can be automatically monitored to determine in real time the processing timing of the model reasoning task corresponding to the first request.
[0138] Step 506: When the sum of the computing resources released by the third request and the second computing resources matches the first computing resources, take out the first request from the message queue.
[0139] The sum of the computing resources released by the third request and the second computing resources matches the first computing resources, which means that the sum of the computing resources released by the third request and the second computing resources is greater than the first computing resources. At this time, the idle computing resources in the inference system can meet the computing resource requirements of the model inference task corresponding to the first request.
[0140] Based on the above embodiments, the model reasoning method provided by the embodiment of the present disclosure further includes the following steps 601 to 603:
[0141] Step 601, responding to a batch request for model reasoning, based on the first type of model and the second type of model corresponding to the batch request, and the deployment architecture of the first type of model and the second type of model, inferring the to-be-processed data of the batch request using different time sequences.
[0142] When implemented, the inference system can process multiple requests for model inference at the same time to improve the processing efficiency of requests and improve the user experience. For example, the number of requests that can be processed in a single batch can be 5.
[0143] In some embodiments, if the first type of model and the second type of model are deployed on different devices, the first type of model and the second type of model corresponding to the batch request are inferred simultaneously; if the first type of model and the second type of model are deployed on the same device, the first type of model and the second type of model corresponding to the batch request are inferred serially.
[0144] Specifically, if the first type of model and the second type of model are deployed on the same device, the first type of model corresponding to the batch request can be inferred first, and then the second type of model corresponding to the batch request can be inferred; alternatively, the second type of model corresponding to the batch request can be inferred first, and then the first type of model corresponding to the batch request can be inferred.
[0145] Step 602: Based on the association relationship between any second request in the batch requests and the first type of model and the second type of model, determine the inference result of the first type of model and the inference result of the second type of model corresponding to the second request.
[0146] The second request refers to multiple requests for model inference. When the number of second requests exceeds the number of requests that can be processed in a single batch, the second requests can be processed in batches.
[0147] It should be noted that the operation of determining the inference result corresponding to each second request is similar to the operation of determining the inference result corresponding to the first request, and reference may be made to the specific implementation of the aforementioned steps 401 to 406 .
[0148] Step 603: Determine the model inference result of the second request based on the inference result of the first type of model and the inference result of the second type of model corresponding to the second request.
[0149] In the disclosed embodiment, the inference system integrates the resources required by different demand parties for model inference; in this way, the model inference resources of different demand parties are concentrated in the non-local inference system, so that the model inference resources are uniformly allocated through the inference system, without the need for additional deployment and without resource redundancy. In response to the first request for model inference, the inference system can allocate various resources (first type model, second type model, and training data) required for the first request by itself, and complete the deployment and processing of model inference based on the allocated resources. The whole process is intelligent and does not require manual operation, which greatly reduces the deployment difficulty and improves the model deployment efficiency and model inference efficiency. In addition, the first computing resource is allocated to the first request based on the data volume of the training data and the parameter volume of the first type model and the second type model, so that the resource scheduling can be performed according to the actual resource demand of the first request, and the appropriate amount of resources can be allocated to each model inference task, so as to maximize the resource utilization and reduce the waste of resources. The inference system can detect the release of resources by itself, without the need for manual detection by the user, and the detection and scheduling of resources are better. Batch processing of requests for model inference can accelerate the process of model inference and further improve the speed of model inference and resource utilization.
[0150] The following describes the application of the model reasoning method provided by the embodiments of the present disclosure in actual scenarios.
[0151] like Figure 7 As shown, the reasoning system 10 mainly includes three parts: a data management module 11 , a resource scheduling module 12 and a reasoning module 13 .
[0152] (1) The main functions of the data management module 11 include: managing large models, including trained large models and untrained large models; models can also be uploaded and deleted. Managing training data, training data can be uploaded, renamed and deleted; wherein the training data can be stored in a fixed format. Managing Lora models, including trained Lora models and untrained Lora models; Lora models can also be renamed and deleted, and locally trained Lora models can be uploaded to the inference system. Among them, large models include Llama model, Baichuan model, Chatglm model, Gpt3 model, bloom model, musachat model, ... Training data includes voice data, picture data, text data, video data, ... Lora model includes adapter 1, adapter 2, and adapter 3, ...
[0153] (2) The main functions of the resource scheduling module 12 include: On the front-end page (human-computer interaction interface), you can select the fine-tuned model and training data, then set the model parameters, and send a request for model reasoning to the back-end. The back-end caches the request in the message queue first after receiving it. The back-end controller takes out the request from the message queue in the order of cache time, allocates computing resources according to the parameters of the model reasoning task corresponding to the request, to perform model reasoning, and synchronizes the loss function (loss), training time and other model parameters to the front-end in real time. If the computing resources are insufficient and there are still requests in the message queue, you need to wait for the resources to be released. After training, save the lora model.
[0154] For example, the resource scheduling module 12 may allocate computing resources to different model reasoning tasks such as model reasoning task 1, model reasoning task 2, model reasoning task 3, ... . . . . .
[0155] (3) The main function of the inference module 13 is to process model inference tasks. Specifically, when deploying the inference system, the base large model and the LoRa model are not merged and are deployed separately. When a request comes, the base large model and the LoRa model are trained separately, and then the results are merged. Inference can also be accelerated through batch processing technology and KV cache technology.
[0156] like Figure 8 As shown, the model reasoning method using batch processing technology may include the following steps 801 to 803:
[0157] Step 801: Determine the requests for model inference that can be processed in a single batch.
[0158] Step 802: Train the requested large model and Lora model in a single batch simultaneously.
[0159] Step 803: merge the trained large model and the Lora model to obtain the inference results of the requests in a single batch.
[0160] like Fig. 9 As shown, the overall process of the model reasoning method includes the following steps 901 to 907:
[0161] Step 901: Set model parameters.
[0162] Step 902: Determine whether the reasoning system has the required large model.
[0163] If the reasoning system does not have the required large model, then step 903 is executed. If the reasoning system has the required large model, then step 904 is executed.
[0164] Step 903: Upload the large model.
[0165] Step 904: Determine whether the inference system has the required training data.
[0166] If the reasoning system does not have the required training data, then step 905 is executed. If the reasoning system has the required large model, then step 906 is executed.
[0167] Step 905: Upload training data.
[0168] Step 906: Train the large model and the Lora model to obtain inference results.
[0169] Step 907: Output the inference result.
[0170] Based on the foregoing embodiments, the embodiments of the present disclosure provide a model reasoning device, which includes the units included and the modules included in the units, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.
[0171] Fig.10 A schematic diagram of the structure of a model inference device provided in an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the model reasoning device 1000 includes: a deployment unit 1010, an allocation unit 1020 and a reasoning unit 1030, wherein:
[0172] In response to a first request for model inference from any demander, the allocation unit 1010 is used to allocate the first type of model, the second type of model, and the training data matching the first request based on the indication information of the first request; the first type of model is used to control the processing capability of the model for overall features, and the second type of model is used to control the processing capability of the model for local features;
[0173] An allocating unit 1020, configured to allocate a first computing resource to the first request based on the amount of the training data and the amount of parameters of the first type of model and the second type of model;
[0174] The inference unit 1030 is used to determine the model inference result of the first request based on the training data, the first type of model, the second type of model, and the first computing resource.
[0175] In some embodiments, the allocation unit 1020 is specifically used to: determine the first video memory resources and first computing power resources required for processing the first request based on the data volume of the training data and the parameter volume of the first type of model and the second type of model; determine the first computing resource based on the first video memory resources and the first computing power resources; and allocate the first computing resource to the first request.
[0176] In some embodiments, the inference unit 1030 is specifically used to: optimize the second type of model based on the training data, the first type of model, and the first computing resources to obtain an optimized second type of model; infer the to-be-processed data of the first request based on the optimized second type of model and the first type of model to obtain the model inference result.
[0177] In some embodiments, the inference unit 1030 is specifically used to: call the first computing resource, and train the first type of model and the second type of model respectively based on the training data; during the training process, based on the output of the first type of model and the output of the second type of model, continuously optimize the model parameters of the second type of model until the iteration stop condition is reached, thereby obtaining the optimized second type of model.
[0178] In some embodiments, the reasoning unit 1030 is specifically used to: use the first type of model to reason on the data to be processed to obtain a first reasoning result; use the optimized second type of model to reason on the data to be processed to obtain a second reasoning result; determine the model reasoning result based on the first reasoning result and the second reasoning result.
[0179] In some embodiments, the allocation unit 1010 is specifically used to: when receiving the first request, cache the first request to a message queue; determine the first computing resource required for processing the first request based on the amount of data of the training data and the amount of parameters of the first type of model and the second type of model; determine whether the idle second computing resource in the inference system matches the first computing resource; and when the first request is the earliest request cached in the message queue and the second computing resource matches the first computing resource, take out the first request from the message queue to process the first request.
[0180] In some embodiments, the reasoning unit 1030 is specifically used to: use a first optimization algorithm to optimize the operation of using the first type of model to infer the data to be processed; use a second optimization algorithm to optimize the operation of using the optimized second type of model to infer the data to be processed to obtain a second reasoning result.
[0181] In some embodiments, the inference unit 1030 is specifically used to: respond to batch requests for model inference, and based on the first and second models corresponding to the batch requests, and the deployment architecture of the first and second models, use different time sequences to infer the to-be-processed data of the batch requests; determine the inference results of the first and second models corresponding to the second request based on the association between any second request in the batch requests, the first and second models; and determine the model inference results of the second request based on the inference results of the first and second models corresponding to the second request.
[0182] In some embodiments, the inference unit 1030 is specifically used for: if the first type of model and the second type of model are deployed on different devices, then the first type of model and the second type of model corresponding to the request are inferred simultaneously; the request includes a single request and a batch request; if the first type of model and the second type of model are deployed on the same device, then the first type of model and the second type of model corresponding to the request are inferred serially.
[0183] In some embodiments, the allocation unit 1020 is specifically used to: monitor a third request for resource release when the first request is the request with the earliest cache time in the message queue, but the second computing resource does not match the first computing resource; and take out the first request from the message queue when the sum of the computing resources released by the third request and the second computing resource matches the first computing resource.
[0184] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.
[0185] It should be noted that in the embodiments of the present disclosure, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.
[0186] An embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.
[0187] The embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.
[0188] An embodiment of the present disclosure provides a computer program, including a computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.
[0189] The present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.
[0190] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product disclosed in the present invention, please refer to the description of the method embodiment disclosed in the present invention for understanding.
[0191] It should be noted that Fig.11 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure, such as Fig.11 As shown, the hardware entity of the computer device 1100 includes: a processor 1101, a communication interface 1102 and a memory 1103, wherein:
[0192] Processor 1101 generally controls the overall operation of computer device 1100 .
[0193] The communication interface 1102 enables the computer device to communicate with other terminals or servers through a network.
[0194] The memory 1103 is configured to store instructions and applications executable by the processor 1101, and can also cache data to be processed or processed by the processor 1101 and each module in the computer device 1100 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 1101, the communication interface 1102, and the memory 1103 through the bus 1104.
[0195] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution. The execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.
[0196] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0197] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0198] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0199] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0200] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.
[0201] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0202] The above description is only an implementation mode of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure.
Claims
1. A model reasoning method, characterized in that: Applied to the inference system, the inference system is deployed on the server, the inference system integrates the resources required by different demand parties during model inference, and the model inference method includes: In response to a first request for model inference from any demander, based on the indication information of the first request, respectively deploy the first type of model, the second type of model, and the training data that match the first request; the first type of model is used to control the model's processing capability for overall features, and the second type of model is used to control the model's processing capability for local features; Allocate a first computing resource to the first request based on the amount of the training data and the amount of parameters of the first model and the second model; Based on the training data, the first type of model, the second type of model, and the first computing resource, a model inference result of the first request is determined.
2. The model reasoning method according to claim 1, characterized in that: The allocating a first computing resource to the first request based on the amount of the training data and the amount of parameters of the first type of model and the second type of model includes: Determine, based on the amount of the training data and the amount of parameters of the first model and the second model, the first graphics memory resources and the first computing power resources required for processing the first request; Determine the first computing resource based on the first graphics memory resource and the first computing power resource; A first computing resource is allocated to the first request.
3. The model reasoning method according to claim 1, characterized in that: The determining, based on the training data, the first type of model, the second type of model, and the first computing resource, a model inference result of the first request includes: Optimizing the second type of model based on the training data, the first type of model, and the first computing resource to obtain an optimized second type of model; Based on the optimized second-type model and the first-type model, the to-be-processed data of the first request is inferred to obtain the model inference result.
4. The model reasoning method according to claim 3, characterized in that: The optimizing the second type of model based on the training data, the first type of model, and the first computing resource to obtain the optimized second type of model includes: Calling the first computing resource to train the first type of model and the second type of model respectively based on the training data; During the training process, based on the output of the first type of model and the output of the second type of model, the model parameters of the second type of model are continuously optimized until the iteration stop condition is reached, thereby obtaining the optimized second type of model.
5. The model reasoning method according to claim 3, characterized in that: The reasoning on the to-be-processed data of the first request based on the optimized second type model and the first type model to obtain the model reasoning result includes: Using the first type of model to infer the data to be processed to obtain a first inference result; Using the optimized second type of model to infer the data to be processed to obtain a second inference result; The model inference result is determined based on the first inference result and the second inference result.
6. The model reasoning method according to claim 1, characterized in that: The model reasoning method also includes: When receiving the first request, caching the first request in a message queue; Determine, based on the amount of the training data and the amount of parameters of the first type of model and the second type of model, a first computing resource required for processing the first request; Determine whether a second computing resource in the inference system that is idle matches the first computing resource; When the first request is the request with the earliest cache time in the message queue and the second computing resource matches the first computing resource, the first request is taken out from the message queue to process the first request.
7. The model reasoning method according to claim 5, characterized in that: The model reasoning method also includes: Using a first optimization algorithm, optimizing the operation of using the first type of model to infer the data to be processed; The operation of performing reasoning on the data to be processed using the optimized second type model is optimized using a second optimization algorithm.
8. The model inference method according to any one of claims 1 to 7, characterized in that: The model reasoning method also includes: In response to a batch request for model reasoning, based on the first type of model and the second type of model corresponding to the batch request and the deployment architecture of the first type of model and the second type of model, inferring the to-be-processed data of the batch request using different time sequences; Determine, based on an association relationship between any second request in the batch of requests, the first type of model and the second type of model, an inference result of the first type of model and an inference result of the second type of model corresponding to the second request; Based on the inference results of the first type of model and the inference results of the second type of model corresponding to the second request, the model inference result of the second request is determined.
9. The model reasoning method according to claim 1, characterized in that: The model reasoning method also includes: If the first type of model and the second type of model are deployed on different devices, then reasoning is performed on the first type of model and the second type of model corresponding to the request at the same time; the request includes a single request and a batch request; If the first type of model and the second type of model are deployed on the same device, the first type of model and the second type of model corresponding to the request are inferred serially.
10. The model reasoning method according to claim 6, characterized in that: The model reasoning method also includes: When the first request is the request with the earliest cache time in the message queue, but the second computing resource does not match the first computing resource, monitoring a third request for resource release; When the sum of the computing resources released by the third request and the second computing resources matches the first computing resources, the first request is taken out from the message queue.
11. An inference system, characterized in that The inference system is deployed on the server. The inference system integrates the resources required by different demand parties during model inference. The inference system includes: a data management module, a resource scheduling module and an inference module; In response to a first request for model inference from any demander, the data management module is used to allocate the first type of model, the second type of model, and the training data matching the first request based on the indication information of the first request; the first type of model is used to control the processing capability of the model for overall features, and the second type of model is used to control the processing capability of the model for local features; The resource scheduling module is used to allocate a first computing resource to the first request based on the data volume of the training data and the parameter volume of the first type of model and the second type of model; The reasoning module is used to determine the model reasoning result of the first request based on the training data, the first type of model, the second type of model, and the first computing resource.
12. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 10 are implemented.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
14. A computer program product, comprising a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, the steps of the method according to any one of claims 1 to 10 are implemented.