Dynamic batch processing method and device for large model reasoning service

Through reinforcement learning technology, the batch size in the inference process of large-scale model is dynamically adjusted, the problem of latency and throughput contradictions caused by batch size selection is solved, the service response timeliness and maximum utilization of computing resources is realized, and the system's throughput and Processing efficiency.

CN119005345AActive Publication Date: 2024-11-22ZHEJIANG LAB

Patent Information

Application Number
CN202411486703.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-23
Publication Date
2024-11-22
Estimated Expiration
2044-10-23

AI Technical Summary

Technical Problem

In the process of large-scale model inference, the selection of batch size leads to a contradiction between latency and throughput. How to dynamically adjust the batch size to ensure timely service response and maximize the utilization of computing resources is an urgent problem.

Method used

Using reinforcement learning technology to dynamically adjust the batch size in the large model inference process, calculate the reward value based on the current throughput and delay constraint parameters, and select appropriate adjustment actions (increase or reduce the batch size) to optimize the batch size to ensure timely service response maximizing utilization of sexual and computing resources.

Benefits of technology

It realizes that on the basis of ensuring the timely service response, dynamically adjusting batch sizes to optimize the utilization of computing resources, and improves the system's throughput and processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005345A_ABST
    Figure CN119005345A_ABST
Patent Text Reader

Abstract

The invention discloses a dynamic batch processing method and device for a large model reasoning service, and the method comprises the steps: in a process that a server executes a batch processing reasoning task through a target large model, carrying out the batch processing reasoning task through a preset reinforcement learning agent model on the premise of meeting a preset delay constraint; according to the method, optimization of the throughput of the target large model for executing the reasoning task is taken as an optimization target, and the batch size used by the target large model is dynamically adjusted, so that computing resources can be utilized to the maximum extent on the basis of ensuring the service response timeliness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a dynamic batch processing method and device for large model reasoning services. Background Art

[0002] In modern machine learning applications, especially in large-scale model reasoning based on deep learning, batch size is a key parameter that defines the number of data samples processed by the model each time. The choice of batch size directly affects two important indicators for measuring the quality of reasoning service: latency and throughput. Latency refers to the time required from the input data entering the system to the result, while throughput refers to the amount of data that can be processed per unit time.

[0003] NVIDIA's benchmark tests on its V100 GPU show that as the batch size increases, the throughput of the system can be significantly improved, but it also causes the processing time of a single request, that is, the latency, to increase accordingly. This is because a larger batch size allows more efficient use of hardware resources such as parallel computing units, thereby speeding up the overall processing speed. However, for some application scenarios that have strict requirements on response time, excessive latency is unacceptable.

[0004] Therefore, how to dynamically adjust the batch size to maximize the use of computing resources while ensuring timely service response is an urgent problem to be solved. Summary of the invention

[0005] This specification provides a dynamic batch processing method and device for large model reasoning services to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions: This specification provides a method for dynamic batch processing of a large model inference service, including: Receiving an inference task execution request; According to the inference task execution request, a data set to be processed is determined, and the data set to be processed is input into a preset target large model, so as to execute the inference task through the target large model, and in the process of the target large model executing the inference task, a batch size used by the target large model is dynamically adjusted for several rounds, wherein the data contained in the data set to be processed includes: at least one of text data, image data, and video data; wherein, For each round of adjustment, the batch size of the target large model after the previous round of adjustment is determined as the batch size to be adjusted in this round of adjustment, and the reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the previous round of adjustment and the current state parameters of the target large model are determined. According to the adjustment action performed in the previous round of adjustment, the state parameters and the reward value, a target adjustment action is determined from the preset candidate adjustment actions, and the batch size to be adjusted is adjusted according to the target adjustment action to obtain the adjusted batch size, and the target large model is made to perform the reasoning task according to the adjusted batch size. The state parameters include: the current throughput of the target large model, the current task delay of the target large model, and the preset delay constraint parameters. The reward value is negatively correlated with the task delay and the reward value is positively correlated with the current throughput of the target large model. The adjustment action includes: increasing the batch size by a preset step size and reducing the batch size by a preset step size.

[0007] Optionally, according to the adjustment action performed in the previous round of adjustment, the state parameter and the reward value, a target adjustment action is determined from preset candidate adjustment actions, specifically including: For each preset candidate adjustment action, determine the probability value of the candidate adjustment action according to the adjustment action performed in the previous round of adjustment, the state parameter and the reward value; According to the probability value of each candidate adjustment action, a target adjustment action is determined from preset candidate adjustment actions.

[0008] Optionally, determining a reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the last round of adjustment specifically includes: Determine a ratio between a current task delay of the target large model and a preset delay constraint parameter as a target ratio; Based on the target ratio and the current throughput of the target large model, the reward value of the target large model for executing the inference task according to the batch size to be adjusted after the last round of adjustment is determined, wherein the smaller the ratio between the current task delay of the target large model and the preset delay constraint parameter and the greater the current throughput of the target large model, the greater the reward value.

[0009] Optionally, determining a reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the last round of adjustment specifically includes: When it is determined that the ratio between the task delay and the preset delay constraint parameter is greater than 1, the reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the last round of adjustment is determined based on a preset penalty factor.

[0010] This specification provides a device for dynamic batch processing of large model reasoning services, including: A receiving module, used for receiving an inference task execution request; An adjustment module is used to perform the reasoning task according to the reasoning task execution request through a preset target large model, and in the process of the target large model executing the reasoning task, perform several rounds of dynamic adjustment on the batch size used by the target large model; wherein, for each round of adjustment, the batch size of the target large model after the previous round of adjustment is determined as the batch size to be adjusted in this round of adjustment, and the reward value of the target large model executing the reasoning task according to the batch size to be adjusted after the previous round of adjustment and the current state parameters of the target model are determined, and according to the adjustment actions performed in the previous round of adjustment, the state parameters and the reward value, determine the target adjustment action from the preset candidate adjustment actions, and adjust the batch size to be adjusted according to the target adjustment action to obtain the adjusted batch size, and make the target large model perform the inference task according to the adjusted batch size, the state parameters include: the current throughput of the target large model, the current task delay of the target large model, and the preset delay constraint parameters, the reward value is negatively correlated with the task delay and the reward value is positively correlated with the current throughput of the target model, and the adjustment action includes: increasing the batch size according to the preset step size and reducing the batch size according to the preset step size.

[0011] Optionally, the adjustment module is specifically configured to determine, for each preset candidate adjustment action, a probability value of the candidate adjustment action according to the adjustment action performed in the previous round of adjustment, the state parameter and the reward value; According to the probability value of each candidate adjustment action, a target adjustment action is determined from preset candidate adjustment actions.

[0012] Optionally, the adjustment module is specifically used to determine a ratio between a current task delay of the target large model and a preset delay constraint parameter as a target ratio; and determine a reward value for the target model to perform the inference task according to the batch size to be adjusted after the last round of adjustment based on the target ratio and the current throughput of the target large model, wherein the smaller the ratio between the current task delay of the target large model and the preset delay constraint parameter and the greater the current throughput of the target large model, the greater the reward value.

[0013] Optionally, the adjustment module is specifically used to determine, based on a preset penalty factor, a reward value for the target large model to perform the reasoning task according to the batch size to be adjusted after the last round of adjustment when it is determined that the ratio between the task delay and the preset delay constraint parameter is greater than 1.

[0014] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for dynamic batch processing of the above-mentioned large model reasoning service is implemented.

[0015] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for dynamic batch processing of the above-mentioned large model reasoning service when executing the program.

[0016] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: In the method for dynamic batch processing of large model inference service provided in this specification, an inference task execution request is received, and a data set to be processed is determined according to the inference task execution request, and the data set to be processed is input into a preset target large model to execute the inference task through the target large model, and in the process of the target large model executing the inference task, the reinforcement learning agent model performs several rounds of dynamic adjustment on the batch size used by the target model, and the data contained in the data set to be processed includes: at least one of text data, image data, and video data, wherein for each round of adjustment, the batch size of the target large model after the previous round of adjustment is determined as the batch size to be adjusted in the current round of adjustment, and the batch size of the target large model after the previous round of adjustment is determined. According to the reward value of the inference task executed according to the batch size to be adjusted and the current state parameters of the target large model, according to the adjustment action, state parameters and reward value executed in the previous round of adjustment, the target adjustment action is determined from the preset candidate adjustment actions, and the batch size to be adjusted is adjusted according to the target adjustment action to obtain the adjusted batch size, and the target large model is made to execute the inference task according to the adjusted batch size. The state parameters include: the current throughput of the target large model, the current task delay of the target large model, and the preset delay constraint parameters. The reward value is negatively correlated with the task delay and the reward value is positively correlated with the current throughput of the target model. The adjustment actions include: increasing the batch size according to the preset step size and reducing the batch size according to the preset step size.

[0017] It can be seen from the above method that the reasoning task can be performed by the preset target large model, and in the process of the target large model performing the reasoning task, the batch size used by the target large model can be dynamically adjusted by using reinforcement learning technology to maximize the use of computing resources while ensuring the timeliness of service response. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings: Figure 1A flowchart of a method for dynamic batch processing of a large model reasoning service provided in this specification; Figure 2 A schematic diagram of the process of executing the reasoning task for the target large model provided in this specification; Figure 3 A schematic diagram of the batch size adjustment process for the reinforcement learning agent model provided in this specification; Figure 4 A schematic diagram of a device for dynamic batch processing of a large model reasoning service provided in this specification; Figure 5 A method corresponding to the Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0020] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0021] Figure 1 A flowchart of a method for dynamic batch processing of a large model inference service provided in this specification includes the following steps: S101: receiving an inference task execution request.

[0022] In this specification, the business platform can perform reasoning tasks through a preset target large model, and can use reinforcement learning technology to dynamically adjust the batch size used by the target large model during the process of the target large model performing reasoning tasks, so as to maximize the use of computing resources while ensuring the timeliness of service response.

[0023] Prior to this, the business platform can first receive the reasoning task execution request, so as to respond to the reasoning task execution request and determine the data set to be processed that needs to be input into the target big model, so that the target big model can perform corresponding processing on each data set to be processed according to the received reasoning task execution request.

[0024] The above-mentioned reasoning task may refer to a task of automatically performing reasoning services on a large amount of data through a target large model without user interaction.

[0025] The above-mentioned target big model may refer to a deep learning big model, and the above-mentioned data set to be processed may be determined according to the actual application scenario. For example, in an intelligent customer service scenario, the above-mentioned data set to be processed may be question text data sent by users of the business platform.

[0026] For another example, in an image recognition service scenario, the aforementioned data set to be processed may be image data to be recognized collected by a business platform.

[0027] For another example, in a motion recognition service scenario, the aforementioned data set to be processed may be video data to be recognized collected by a business platform.

[0028] In this specification, the execution subject of the method for implementing dynamic batch processing of large model reasoning services may refer to a designated device set up in a business platform such as a server or Internet of Things (IOT) device, or may refer to a terminal device such as a desktop computer or a laptop computer. For the sake of ease of description, the following only takes the server as the execution subject as an example to illustrate the method for dynamic batch processing of large model reasoning services provided in this specification.

[0029] S102: Determine a data set to be processed according to the reasoning task execution request, and input the data set to be processed into a preset target large model to execute the reasoning task through the target large model, and in the process of the target large model executing the reasoning task, dynamically adjust the batch size used by the target large model for several rounds, wherein the data contained in the data set to be processed includes: at least one of text data, image data, and video data; S103: wherein, for each round of adjustment, the batch size of the target large model after the previous round of adjustment is determined as the batch size to be adjusted in this round of adjustment, and the reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the previous round of adjustment and the current state parameters of the target large model are determined; according to the adjustment action performed in the previous round of adjustment, the state parameters and the reward value, a target adjustment action is determined from the preset candidate adjustment actions, and the batch size to be adjusted is adjusted according to the target adjustment action to obtain the adjusted batch size, and the target large model is made to perform the reasoning task according to the adjusted batch size, the state parameters include: the current throughput of the target large model, the current task delay of the target large model, and the preset delay constraint parameters, the reward value is negatively correlated with the task delay and the reward value is positively correlated with the current throughput of the target large model, and the adjustment action includes: increasing the batch size by a preset step size and reducing the batch size by a preset step size.

[0030] In this specification, in the process of the target large model executing the reasoning task, for each task cycle, the server can determine an appropriate number of to-be-processed data from each to-be-processed data set contained in the to-be-processed data set according to the batch size used by the target large model in the current task cycle, and input them into the target large model, so that the target large model can perform reasoning services based on the input data batches.

[0031] The batch size used by the target large model in the current task cycle can be obtained by the server after a round of dynamic adjustment of the batch size used by the target large model in the previous task cycle according to the state of the target model in the previous task cycle, as shown in the following example: Figure 2 shown.

[0032] Figure 2 Schematic diagram of the process of performing reasoning tasks for the target large model provided in this specification.

[0033] Combination Figure 2 It can be seen that the server can perform several rounds of dynamic adjustment on the batch size used by the target large model during the process of the target large model executing the inference task. For each round of adjustment, the batch size of the target large model after the previous round of adjustment is determined as the batch size to be adjusted in this round of adjustment.

[0034] Furthermore, the server can determine the reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the last round of adjustment and the current state parameters of the target large model, and determine the target adjustment action from the preset candidate adjustment actions according to the adjustment action, state parameters and reward value performed in the last round of adjustment, and adjust the batch size to be adjusted according to the target adjustment action to obtain the batch size after this round of adjustment, and make the target large model perform the reasoning task of the next task cycle according to the batch size after this round of adjustment, as shown in the following figure. Figure 3 shown.

[0035] Figure 3 Schematic diagram of the batch size adjustment process for the reinforcement learning agent model provided in this specification.

[0036] Combination Figure 3It can be seen that the server can input the adjustment actions, state parameters and reward values ​​performed in the previous round of adjustment into the preset reinforcement learning agent model, and then the reinforcement learning agent model can be used to determine the probability value of each preset candidate adjustment action according to the adjustment actions, state parameters and reward values ​​performed in the previous round of adjustment. Then, according to the probability value of each candidate adjustment action, the target adjustment action can be determined from the preset candidate adjustment actions, and the batch size to be adjusted can be adjusted according to the target adjustment action to obtain the batch size after this round of adjustment, and the target large model can perform the inference task of the next task cycle according to the batch size after this round of adjustment.

[0037] The above-mentioned state parameters include: the current throughput of the target large model, the current task delay of the target large model, and the preset delay constraint parameters.

[0038] The reward value in the above content is negatively correlated with the above task delay and positively correlated with the current throughput of the target model.

[0039] The adjustment actions in the above content include: increasing the batch size by a preset step size, and decreasing the batch size by a preset step size.

[0040] In the above content, the method for the server to determine the reward value of the target large model for performing the inference task according to the batch size to be adjusted after the last round of adjustment can be to determine the current task delay and throughput of the target large model, and then determine the reward value of the target large model for performing the inference task according to the batch size to be adjusted after the last round of adjustment based on the current task delay and throughput of the target large model. The reward value here is negatively correlated with the task delay and positively correlated with the current throughput of the target model.

[0041] Furthermore, since the target large model has different requirements for task delay when executing different reasoning tasks, in order to accurately adjust the batch size used by the target large model for executing different reasoning tasks, the server can also determine the ratio between the current task delay of the target large model and the preset delay constraint parameter as the target ratio, and then determine the reward value of the target large model for executing the reasoning task according to the batch size to be adjusted after the last round of adjustment based on the functional relationship between the target ratio and the current throughput of the target large model and the preset target ratio, the current throughput of the target large model and the reward value. The smaller the target ratio, the larger the current throughput of the target model, and the larger the determined reward value. The calculation method of the target ratio here can refer to the following formula:

[0042] In the above formula, is the pre-set delay constraint parameter, The current task delay of the target large model, This is the target ratio.

[0043] In addition, when business requirements change, the optimization direction can be adjusted by adjusting the above-mentioned delay constraint parameters without modifying the functional relationship between the target ratio, the current throughput of the target model, and the reward value.

[0044] In actual application scenarios, if you need to reduce the task delay of the target model, you need to set a smaller batch size. However, the smaller the batch size, the better. Although a smaller batch size can reduce the delay of a single piece of data to be processed, it may reduce the overall throughput. A larger batch size allows the target model to process more data to be processed in one forward propagation, thereby increasing the processing volume per unit time.

[0045] Therefore, the server can also determine the reward value of the target large model for performing the inference task according to the batch size to be adjusted after the last round of adjustment based on the preset reward function, the current throughput of the target large model, and the current task delay of the target large model. For details, please refer to the following formula:

[0046] In the above formula, R is the reward value of the target large model to perform the inference task according to the batch size to be adjusted after the last round of adjustment, and t e is the current throughput of the target model, r p is the penalty factor. The penalty factor here can be set according to actual needs. Preferably, the penalty factor here can be set to 0.001×t e .

[0047] From the above formula, we can see that t e The current throughput of the target large model can truly reflect the quality of the target large model's inference service after the last round of batch size adjustment, because it will change significantly with the change of batch size. When the delay satisfies the delay constraint, if t e The higher the value, the better the effect of the last batch size adjustment, and the higher the reward value. At the same time, when the delay exceeds the delay constraint, the reward value is controlled by the above penalty factor. At this time, if t e The higher it is, the worse the adjustment effect of the batch size in the previous round is, and the lower the determined reward value is.

[0048] Furthermore, after determining the batch size after this round of adjustment, the server can use the batch size after this round of adjustment as the batch size used by the target large model in the current task cycle, and then determine an appropriate number of processed data from each processed data contained in the processed data set and input them into the target large model, so that the target model can perform the inference task of the current task cycle based on the input data batch.

[0049] It can be seen from the above method that in the process of the server performing reasoning tasks through the target large model, the preset reinforcement learning agent model can be used to optimize the throughput of the target large model to perform reasoning tasks under the premise of satisfying the preset delay constraints. The batch size used by the target large model can be dynamically adjusted, thereby maximizing the use of computing resources on the basis of ensuring the timeliness of service response.

[0050] The above are one or more methods for implementing dynamic batch processing of large model reasoning services in this specification. Based on the same idea, this specification also provides a corresponding device for dynamic batch processing of large model reasoning services, such as Figure 4 shown.

[0051] Figure 4 A schematic diagram of a device for dynamic batch processing of a large model reasoning service provided in this specification includes: A receiving module 401 is used to receive an inference task execution request; The adjustment module 402 is used to perform the reasoning task according to the reasoning task execution request through the preset target large model, and in the process of the target large model executing the reasoning task, perform several rounds of dynamic adjustment on the batch size used by the target large model; wherein, for each round of adjustment, the batch size of the target large model after the previous round of adjustment is determined as the batch size to be adjusted in this round of adjustment, and the reward value of the target large model executing the reasoning task according to the batch size to be adjusted after the previous round of adjustment and the current state parameters of the target model are determined, and according to the adjustment actions performed in the previous round of adjustment, the state parameters and The reward value determines a target adjustment action from preset candidate adjustment actions, and adjusts the batch size to be adjusted according to the target adjustment action to obtain the adjusted batch size, and enables the target large model to perform the inference task according to the adjusted batch size. The state parameters include: the current throughput of the target large model, the current task delay of the target large model, and the preset delay constraint parameters. The reward value is negatively correlated with the task delay and the reward value is positively correlated with the current throughput of the target model. The adjustment action includes: increasing the batch size according to a preset step size and reducing the batch size according to a preset step size.

[0052] Optionally, the adjustment module 402 is specifically configured to determine, for each preset candidate adjustment action, a probability value of the candidate adjustment action according to the adjustment action performed in the previous round of adjustment, the state parameter, and the reward value; According to the probability value of each candidate adjustment action, a target adjustment action is determined from preset candidate adjustment actions.

[0053] Optionally, the adjustment module 402 is specifically used to determine a ratio between a current task delay of the target large model and a preset delay constraint parameter as a target ratio; based on the target ratio and the current throughput of the target large model, determine a reward value for the target model to perform the inference task according to the batch size to be adjusted after the last round of adjustment, wherein the smaller the ratio between the current task delay of the target large model and the preset delay constraint parameter and the greater the current throughput of the target large model, the greater the reward value.

[0054] Optionally, the adjustment module 402 is specifically used to determine, based on a preset penalty factor, a reward value for the target large model to perform the reasoning task according to the batch size to be adjusted after the last round of adjustment when it is determined that the ratio between the task delay and the preset delay constraint parameter is greater than 1.

[0055] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A dynamic batch processing method for large model inference services is provided.

[0056] This manual also provides Figure 5 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Figure 5 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The method of dynamic batch processing of the large model reasoning service. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0057] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0058] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0059] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0060] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0061] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0065] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0066] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0067] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0068] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0069] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0071] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0072] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for dynamic batch processing of large model inference services, characterized in that: include: Receive an inference task execution request; According to the inference task execution request, a data set to be processed is determined, and the data set to be processed is input into a preset target large model, so as to execute the inference task through the target large model, and in the process of the target large model executing the inference task, a batch size used by the target large model is dynamically adjusted for several rounds, wherein the data contained in the data set to be processed includes: at least one of text data, image data, and video data; wherein, For each round of adjustment, the batch size of the target large model after the previous round of adjustment is determined as the batch size to be adjusted in this round of adjustment, and the reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the previous round of adjustment and the current state parameters of the target large model are determined. According to the adjustment action performed in the previous round of adjustment, the state parameters and the reward value, a target adjustment action is determined from the preset candidate adjustment actions, and the batch size to be adjusted is adjusted according to the target adjustment action to obtain the adjusted batch size, and the target large model is made to perform the reasoning task according to the adjusted batch size. The state parameters include: the current throughput of the target large model, the current task delay of the target large model, and the preset delay constraint parameters. The reward value is negatively correlated with the task delay and the reward value is positively correlated with the current throughput of the target large model. The adjustment action includes: increasing the batch size by a preset step size and reducing the batch size by a preset step size.

2. The method according to claim 1, characterized in that According to the adjustment action performed in the previous round of adjustment, the state parameter and the reward value, a target adjustment action is determined from the preset candidate adjustment actions, specifically including: For each preset candidate adjustment action, determine the probability value of the candidate adjustment action according to the adjustment action performed in the previous round of adjustment, the state parameter and the reward value; According to the probability value of each candidate adjustment action, a target adjustment action is determined from preset candidate adjustment actions.

3. The method according to claim 1, characterized in that Determining the reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the last round of adjustment, specifically includes: Determine a ratio between a current task delay of the target large model and a preset delay constraint parameter as a target ratio; Based on the target ratio and the current throughput of the target large model, the reward value of the target large model for executing the inference task according to the batch size to be adjusted after the last round of adjustment is determined, wherein the smaller the ratio between the current task delay of the target large model and the preset delay constraint parameter and the greater the current throughput of the target large model, the greater the reward value.

4. The method according to claim 3, characterized in that Determining the reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the last round of adjustment, specifically includes: When it is determined that the ratio between the task delay and the preset delay constraint parameter is greater than 1, the reward value of the target large model for performing the reasoning task according to the batch size to be adjusted after the last round of adjustment is determined based on a preset penalty factor.

5. A device for dynamic batch processing of large model reasoning services, characterized in that: include: A receiving module, used for receiving an inference task execution request; An adjustment module is used to perform the reasoning task according to the reasoning task execution request through a preset target large model, and in the process of the target large model executing the reasoning task, perform several rounds of dynamic adjustment on the batch size used by the target large model; wherein, for each round of adjustment, the batch size of the target large model after the previous round of adjustment is determined as the batch size to be adjusted in this round of adjustment, and the reward value of the target large model executing the reasoning task according to the batch size to be adjusted after the previous round of adjustment and the current state parameters of the target model are determined, and according to the adjustment actions performed in the previous round of adjustment, the state parameters and the reward value, determine the target adjustment action from the preset candidate adjustment actions, and adjust the batch size to be adjusted according to the target adjustment action to obtain the adjusted batch size, and make the target large model perform the inference task according to the adjusted batch size, the state parameters include: the current throughput of the target large model, the current task delay of the target large model, and the preset delay constraint parameters, the reward value is negatively correlated with the task delay and the reward value is positively correlated with the current throughput of the target model, and the adjustment action includes: increasing the batch size according to the preset step size and reducing the batch size according to the preset step size.

6. The device according to claim 5, characterized in that The adjustment module is specifically used to determine, for each preset candidate adjustment action, a probability value of the candidate adjustment action according to the adjustment action performed in the previous round of adjustment, the state parameter and the reward value; According to the probability value of each candidate adjustment action, a target adjustment action is determined from preset candidate adjustment actions.

7. The device according to claim 5, characterized in that The adjustment module is specifically used to determine the ratio between the current task delay of the target large model and the preset delay constraint parameter as the target ratio; based on the target ratio and the current throughput of the target large model, determine the reward value of the target model for executing the inference task according to the batch size to be adjusted after the last round of adjustment, wherein the smaller the ratio between the current task delay of the target large model and the preset delay constraint parameter and the larger the current throughput of the target large model, the larger the reward value.

8. The device according to claim 5, characterized in that The adjustment module is specifically used to determine, when it is determined that the ratio between the task delay and the preset delay constraint parameter is greater than 1, a reward value for the target large model to perform the reasoning task according to the batch size to be adjusted after the last round of adjustment based on a preset penalty factor.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 4 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • Batch processing remote sensing image intelligent processing model prediction performance optimization system and method

    CN113434303A

  • Model reasoning scheduling method and system based on batch processing dynamic adjustment

    CN115408163A

  • Inference method of artificial intelligence inference framework, computer equipment and medium

    CN115952866A

  • Data processing method and device, storage medium and electronic equipment

    CN115964181A

  • Network model reasoning method and device, storage medium and electronic equipment

    CN116258205A

Cited By

  • Inference request scheduling method and device based on reinforcement learning, equipment and medium

    CN120950225A

  • Flume-based time sequence database dynamic adaptation method and device and medium

    CN121051164A

  • Video recommendation system and method based on AI large model

    CN122578911A