A request scheduling method and apparatus

Abstract

The application discloses a request scheduling method and device, the method comprises the following steps: a first device receives a first request of a first user, the first request is a request of a first service, and the first request comprises an identifier of the first service; the first device schedules the first request to a second device; the second device sends a second request to the first device, the second request comprises the identifier of the first service and first output information, the second request is associated with the first request; when the first output information does not satisfy a set condition, the first device schedules the second request to a third device, the third device is different from the second device; the third device sends a third request to the first device, the third request comprises the identifier of the first service and second output information, and the third request is associated with the second request; when the second output information does not satisfy the set condition, the first device schedules the third request to the third device, and the throughput of a service system in large-scale language model service deployment can be improved.

Description

Request scheduling method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 24, 2023, with application number 202311386911.8 and application name “A Request Scheduling Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of computer technology, and in particular to a request scheduling method and device. Background Art

[0003] Large Language Models (LLMs) have revolutionized the field of artificial intelligence (AI). The inference process of a Generative Pre-trained Transformer (GPT) using key-value caching (KV Cache) technology consists of two phases: an initial context prefill phase and multiple decoding phases. While dynamic scheduling is employed in large-scale language model service deployments, the throughput of the service system in LLM deployments remains low.

[0004] Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a request scheduling method and device, which can improve the throughput of a service system in LLM service deployment.

[0006] In a first aspect, an embodiment of the present invention provides a request scheduling method, which is applied to a request scheduling system. The method includes:

[0007] The first device receives a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service;

[0008] The first device dispatches the first request to the second device;

[0009] The second device sends a second request to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request;

[0010] When the first output information does not meet a set condition, the first device dispatches the second request to a third device, where the third device is different from the second device;

[0011] The third device sends a third request to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request;

[0012] When the second output information does not meet the set condition, the first device dispatches the third request to the third device.

[0013] The embodiment of the present invention also adopts an independent and separate deployment method in the request scheduling system to deploy the second device and the third device. Through the scheduling of the request by the first device, the second device and the third device respectively execute the reasoning process of their respective stages. At the same time, information can be transmitted between the two devices, so that the reasoning process can proceed normally, thereby improving the throughput of the service system.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0015] The first device receives a fourth request from a second user, where the fourth request is a request for a second service and includes an identifier of the second service;

[0016] The first device dispatches the fourth request to the second device;

[0017] The second device sends a fifth request to the first device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request;

[0018] When the third output information does not meet the set condition, the first device dispatches the fifth request to the third device;

[0019] The third device sends a sixth request to the first device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request;

[0020] When the fourth output information does not meet the set condition, the first device dispatches the sixth request to the third device.

[0021] In an embodiment of the present invention, the second user is different from the first user, and the first device can schedule requests from different users to the second device and the third device for processing. There is no need to wait for the second device and / or the third device to complete processing the request of one user before processing the request of another user, thereby improving the throughput of the service system.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0023] When the second output information meets the set condition, the first device sends the second output information to the first user.

[0024] In this embodiment of the present invention, a first device determines whether the output information of a service meets a set condition. If not, the service continues to be scheduled and inferred. If the condition is met, the service is prematurely stopped and a second output message is sent to the first user. After prematurely stopping the service, the first device can notify a third device to release the resources corresponding to the service. The third device then reports its current resource status to the first device. If the current resource status contains idle computing service resources, the first device continues to schedule the service to the third device to insert the idle computing service resources, thereby achieving dynamic insertion.

[0025] In conjunction with the first aspect, in certain implementations of the first aspect, the request scheduling system includes the first device, at least one second device, and at least one third device. In embodiments of the present invention, when the number of second and third devices in the request scheduling system is large, independent, separate deployment methods are employed to deploy the second and third devices, with the second devices configured in small batches or single batches, and the third devices configured in large batches. The resource ratio of the second and third device resources can be dynamically adjusted based on system load to achieve maximum throughput, thereby significantly improving throughput.

[0026] In a second aspect, an embodiment of the present invention provides a request scheduling method, applied to a first device, the method comprising:

[0027] receiving a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service;

[0028] dispatching the first request to a second device;

[0029] receiving a second request sent by the second device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request;

[0030] When the first output information does not meet a set condition, dispatching the second request to a third device, the third device being different from the second device;

[0031] receiving a third request sent by the third device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request;

[0032] When the second output information does not meet the set condition, dispatching the third request to the third device.

[0033] This embodiment of the present invention dispatches user requests for new services to a second device, the Prefill inference execution container, for inference calculations during the Prefill phase. It also dispatches requests for legacy services that have already passed the Prefill phase to a third device, the Decode inference execution container, for inference calculations during the Decode phase. By adopting a separate deployment approach, the present invention comprehensively considers the significant differences between the computing bottlenecks of the Prefill phase and the memory access bottlenecks of the Decode phase, thereby improving the throughput of the service system.

[0034] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

[0035] receiving a fourth request from a second user, where the fourth request is a request for a second service and includes an identifier of the second service;

[0036] dispatching the fourth request to the second device;

[0037] receiving a fifth request sent by the second device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request;

[0038] When the third output information does not meet the set condition, dispatching the fifth request to the third device;

[0039] receiving a sixth request sent by the third device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request;

[0040] When the fourth output information does not meet the set condition, dispatching the sixth request to the third device.

[0041] In an embodiment of the present invention, the second user is different from the first user, and the first device can schedule requests from different users to the second device and the third device for processing. There is no need to wait for the second device and / or the third device to complete processing the request of one user before processing the request of another user, thereby improving the throughput of the service system.

[0042] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

[0043] When the second output information meets the set condition, the second output information is sent to the first user.

[0044] In this embodiment of the present invention, a first device determines whether the output information of a service meets a set condition. If not, the service continues to be scheduled and inferred. If the condition is met, the service is prematurely stopped and a second output message is sent to the first user. After prematurely stopping the service, the first device can notify a third device to release the resources corresponding to the service. The third device then reports its current resource status to the first device. If the current resource status contains idle computing service resources, the first device continues to schedule the service to the third device to insert the idle computing service resources, thereby achieving dynamic insertion.

[0045] In combination with the second aspect, in some implementations of the second aspect, the identifier includes a service ID.

[0046] In conjunction with the second aspect, in certain implementations of the second aspect, the set condition includes at least one of the following:

[0047] The length of the output information is greater than or equal to the first threshold, or

[0048] The output information includes a terminator.

[0049] In a third aspect, an embodiment of the present invention provides a scheduling request method, applied to a second device, the method comprising:

[0050] receiving a first request from a first user sent by a first device, where the first request is a request for a first service and includes an identifier of the first service;

[0051] A second request is sent to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

[0052] This embodiment of the present invention dispatches user requests for new services to a second device, the Prefill inference execution container, for inference calculations during the Prefill phase. It also dispatches requests for legacy services that have already passed the Prefill phase to a third device, the Decode inference execution container, for inference calculations during the Decode phase. Through separate deployment, this invention comprehensively considers the significant differences between the computing bottlenecks of the Prefill phase and the memory access bottlenecks of the Decode phase, thereby improving the throughput of the service system.

[0053] In conjunction with the third aspect, in certain implementations of the third aspect, the method further includes:

[0054] receiving a fourth request from a second user sent by the first device, where the fourth request is a request for a second service and includes an identifier of the second service;

[0055] A fifth request is sent to the first device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request.

[0056] In an embodiment of the present invention, the second user is different from the first user, and the first device can schedule requests from different users to the second device and the third device for processing. There is no need to wait for the second device and / or the third device to complete processing the request of one user before processing the request of another user, thereby improving the throughput of the service system.

[0057] In conjunction with the third aspect, in some implementations of the third aspect, before sending the second request to the first device, the method further includes:

[0058] grouping the first request and at least one seventh request arriving at the second device simultaneously with the first request into batches according to the first computing resource state and the first condition to obtain a first batch, wherein the seventh request is a request from the first user, the second user, or another user;

[0059] A first inference calculation is performed on the first batch to obtain the second request and first cache information.

[0060] In conjunction with the third aspect, in certain implementations of the third aspect, the method further includes:

[0061] The first cache information is sent to the third device.

[0062] In combination with the third aspect, in some implementations of the third aspect, the cache information includes key-value cache KV information.

[0063] With reference to the third aspect, in certain implementations of the third aspect, batching the first request and at least one seventh request that arrives at the second device simultaneously with the first request to obtain the first batch according to the first computing resource state and the first condition includes:

[0064] Obtaining, according to the first request, input information corresponding to the first request and a first length thereof;

[0065] The first request and the at least one seventh request are batched to obtain a first batch based on the first computing resource state, the first condition, and the first length. The length of the input information corresponding to the seventh request is similar to the first length, which helps reduce waste of idle resources during the first inference computation process.

[0066] In combination with the third aspect, in some implementations of the third aspect, the first computing resource status includes the number of idle resources in the second device.

[0067] In combination with the third aspect, in certain implementations of the third aspect, the first condition includes the maximum number of batches that the second device can group.

[0068] In combination with the third aspect, in certain implementations of the third aspect, the first processing method is to release the computing service resources occupied by the current batch after all requests of the current batch are inferred and calculated, and insert the next batch into the idle computing service resources for the first inference calculation.

[0069] In an embodiment of the present invention, for the inference calculation in the Prefill stage, the inference calculation is performed in a single batch, that is, after the inference calculation of all requests in the current batch is completed, the computing service resources occupied by the current batch are released and the next batch is inserted into the idle computing service resources, and the first inference calculation of the next batch is started.

[0070] In a fourth aspect, an embodiment of the present invention provides a scheduling request method, applied to a third device, the method comprising:

[0071] receiving a second request sent by the first device from the second device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request, where the first request is a request from a first user for the first service;

[0072] A third request is sent to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request.

[0073] This embodiment of the present invention dispatches user requests for new services to a second device, the Prefill inference execution container, for inference calculations during the Prefill phase. It also dispatches requests for legacy services that have already passed the Prefill phase to a third device, the Decode inference execution container, for inference calculations during the Decode phase. Through separate deployment, this invention comprehensively considers the significant differences between the computing bottlenecks of the Prefill phase and the memory access bottlenecks of the Decode phase, thereby improving the throughput of the service system.

[0074] In conjunction with the fourth aspect, in certain implementations of the fourth aspect, the method further includes:

[0075] receiving a fifth request sent by the first device, the fifth request including an identifier of the second service and third output information corresponding to the second service, the fifth request being associated with a fourth request, the fourth request being a request for the second service from a second user;

[0076] A sixth request is sent to the first device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request.

[0077] In an embodiment of the present invention, the second user is different from the first user, and the first device can schedule requests from different users to the second device and the third device for processing. There is no need to wait for the second device and / or the third device to complete processing the request of one user before processing the request of another user, thereby improving the throughput of the service system.

[0078] In conjunction with the fourth aspect, in some implementations of the fourth aspect, before sending the third request to the first device, the method further includes:

[0079] grouping the second request and at least one eighth request arriving at the third device simultaneously with the second request into a second batch according to a second computing resource state and a second condition, wherein the eighth request comes from the second device or the third device;

[0080] receiving first cache information sent by the second device;

[0081] A second inference calculation is performed on the second batch according to the first cache information to obtain the third request and the second cache information.

[0082] In a fifth aspect, an embodiment of the present invention provides a request scheduling system, the system comprising:

[0083] A first device is configured to receive a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service;

[0084] The first device is further configured to dispatch the first request to the second device;

[0085] the second device is configured to send a second request to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request;

[0086] When the first output information does not meet a set condition, the first device is further configured to dispatch the second request to a third device, where the third device is different from the second device;

[0087] the third device being configured to send a third request to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request;

[0088] When the second output information does not meet the set condition, the first device is further configured to dispatch the third request to the third device.

[0089] In a sixth aspect, an embodiment of the present invention provides a device comprising a processor and a memory, wherein the memory is used to store a program, the program comprising program instructions, and when the processor runs the program instructions, the device executes the steps of the method described above.

[0090] In a seventh aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium stores a program, wherein the program includes program instructions, and when the program request is run by a device, the device is caused to execute the method as described above.

[0091] In an eighth aspect, an embodiment of the present invention provides a program product, which includes instructions. When the program product is run on a device or any at least one processor, the device executes the functions / steps in the above method.

[0092] In the technical solution of the request scheduling method and device provided by the embodiment of the present invention, the first device receives a first request from a first user, the first request is a request for a first service, and the first request includes an identifier of the first service; the first device schedules the first request to the second device; the second device sends a second request to the first device, the second request includes an identifier of the first service and first output information, and the second request is associated with the first request; when the first output information does not meet the set conditions, the first device schedules the second request to a third device, and the third device is different from the second device; the third device sends a third request to the first device, the third request includes an identifier of the first service and second output information, and the third request is associated with the second request; when the second output information does not meet the set conditions, the first device schedules the third request to the third device, which can improve the throughput of the service system in large-scale language model service deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] FIG1 is a schematic structural diagram of a device provided by an embodiment of the present invention;

[0094] FIG2 is a block diagram of the software structure of the device 100 according to an embodiment of the present invention;

[0095] FIG3 is an architecture diagram of another request scheduling system provided by an embodiment of the present invention;

[0096] FIG4 is a signaling interaction diagram of a scheduling request method provided by an embodiment of the present invention;

[0097] FIG5 is a schematic diagram of Prefill reasoning and Decode reasoning in an embodiment of the present invention;

[0098] FIG6 is a signaling interaction diagram of another scheduling request method provided by an embodiment of the present invention;

[0099] FIG7 is a flowchart of a request scheduling method provided by an embodiment of the present invention;

[0100] FIG8 is a flowchart of another request scheduling method provided by an embodiment of the present invention

[0101] FIG9 is a schematic structural diagram of a first device provided in an embodiment of the present invention;

[0102] FIG10 is a schematic structural diagram of a second device provided in an embodiment of the present invention;

[0103] FIG11 is a schematic structural diagram of a third device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0104] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0105] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0106] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0107] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0108] FIG1 shows a schematic structural diagram of a device 100 .

[0109] The device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0110] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on device 100. In other embodiments of the present application, device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0111] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0112] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0113] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0114] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0115] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, processor 110 may include multiple I2C busses. Processor 110 may be coupled to touch sensor 180K, a charger, a flash, camera 193, and the like via different I2C bus interfaces. For example, processor 110 may be coupled to touch sensor 180K via an I2C interface, enabling communication between processor 110 and touch sensor 180K via the I2C bus interface, thereby implementing touch functionality for device 100.

[0116] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0117] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0118] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0119] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the device 100.

[0120] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0121] USB interface 130 is an interface that complies with USB standards and specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. USB interface 130 can be used to connect a charger to charge device 100 and to transfer data between device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other devices, such as AR devices.

[0122] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the device 100. In other embodiments of the present application, the device 100 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0123] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the device 100. While charging the battery 142, the charging management module 140 can also provide power to the device via the power management module 141.

[0124] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0125] The wireless communication function of the device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0126] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0127] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0128] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0129] The wireless communication module 160 can provide wireless communication solutions applied to the device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0130] In some embodiments, antenna 1 of device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that device 100 can communicate with a network and other devices via wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0131] Device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0132] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0133] The device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display 194, and an application processor.

[0134] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0135] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0136] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0137] Video codecs are used to compress or decompress digital video. Device 100 may support one or more video codecs. This allows device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0138] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications such as image recognition, face recognition, speech recognition, and text comprehension on device 100.

[0139] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0140] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0141] The device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0142] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0143] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0144] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the device 100 receives a call or voice message, the user can hear the voice by placing the receiver 170B close to the ear.

[0145] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The device 100 can be provided with at least one microphone 170C. In other embodiments, the device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the device 100 can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and identification of sound sources, and realize directional recording function, etc.

[0146] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0147] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, device 100 detects the touch intensity based on pressure sensor 180A. Device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0148] The gyroscope sensor 180B can be used to determine the motion posture of the device 100. In some embodiments, the angular velocity of the device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0149] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the device 100 calculates altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0150] Magnetic sensor 180D includes a Hall effect sensor. Device 100 can use magnetic sensor 180D to detect the opening and closing of a flip case. In some embodiments, when device 100 is a flip phone, device 100 can detect the opening and closing of the flip cover based on magnetic sensor 180D. Based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0151] Accelerometer 180E detects the magnitude of acceleration of device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when device 100 is stationary. It can also be used to identify device posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0152] Distance sensor 180F is used to measure distance. Device 100 can measure distance using infrared or laser. In some embodiments, when photographing a scene, device 100 can use distance sensor 180F to measure distance to achieve fast focusing.

[0153] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The device 100 emits infrared light outward through the light emitting diode. The device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the device 100. When insufficient reflected light is detected, the device 100 can determine that there is no object near the device 100. The device 100 can use the proximity light sensor 180G to detect when the user holds the device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0154] Ambient light sensor 180L is used to sense ambient light brightness. Device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether device 100 is in a pocket to prevent accidental touches.

[0155] The fingerprint sensor 180H is used to collect fingerprints. The device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0156] Temperature sensor 180J is used to detect temperature. In some embodiments, device 100 uses the temperature detected by temperature sensor 180J to implement a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, device 100 reduces the performance of a processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, device 100 heats battery 142 to prevent abnormal shutdown of device 100 due to low temperature. In other embodiments, when the temperature is below yet another threshold, device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0157] The touch sensor 180K is also referred to as a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the device 100, at a location different from that of the display screen 194.

[0158] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0159] Keys 190 include a power button, a volume button, and the like. Keys 190 may be mechanical keys or touch-sensitive keys. Device 100 may receive key inputs and generate key signal inputs related to user settings and function control of device 100.

[0160] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0161] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0162] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the device 100 by inserting it into or removing it from the SIM card interface 195. The device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the device 100 and cannot be separated from the device 100.

[0163] The software system of the device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the device 100.

[0164] FIG2 is a block diagram of the software structure of the device 100 according to an embodiment of the present invention.

[0165] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0166] The application layer can include a series of application packages.

[0167] As shown in FIG2 , the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

[0168] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0169] As shown in FIG2 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0170] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0171] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0172] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0173] The phone manager is used to provide communication functions of the device 100, such as management of call status (including answering, hanging up, etc.).

[0174] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0175] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically, without requiring user interaction. For example, the Notification Manager can be used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, vibrating the device, or flashing indicator lights.

[0176] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0177] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0178] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0179] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0180] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0181] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0182] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0183] A 2D graphics engine is a drawing engine for 2D drawings.

[0184] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0185] The following describes the workflow of the software and hardware of the device 100 in conjunction with the capture and photo shooting scene.

[0186] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture a still image or video through the camera 193.

[0187] First, the relevant technologies of the request scheduling method involved in the embodiment of the present invention are explained to facilitate understanding by those skilled in the art.

[0188] Transformer: A specialized deep learning architecture that combines multi-head self-attention and a forward-feedback mechanism. This mechanism assigns weights to different parts of the input data based on their importance. This model is primarily used in natural language processing (NLP) and computer vision (CV).

[0189] Token: In Chinese natural language processing, a token corresponds to a character or word in a text. Input text is first processed through a word segmenter and vocabulary mapping to obtain the corresponding token sequence, which is then fed into the model for further processing.

[0190] Prefill inference: The input information is usually very long, which plays a decisive role in latency and creates a computational bottleneck.

[0191] Decode reasoning: The input information length is fixed at 1, the latency overhead is small, and there is a memory access bottleneck.

[0192] Throughput: The number of requests processed by the system per unit time. The throughput calculation formula referred to in this invention is:

[0193] Large-scale language models have set off a new wave in the AI ​​community. Generative models, such as Generative Propagation (GPT), have a unique reasoning process. Given an input token sequence of length S, the model outputs an output token sequence of length N. This process involves N inferences. Specifically, GPT models output only one token per inference. This output token is concatenated with the previous input tokens (S+1, S+2, ..., S+N-1) and used as input for the next inference, until a terminator or the maximum output length is encountered.

[0194] This approach introduces a problem: as the number of iterations increases, the computational load for each inference increases. Furthermore, because each iteration includes the input from the previous iteration, there is a significant amount of redundant computation. Therefore, engineers use KV caching technology to maintain a relatively constant computational load for each inference round after the first iteration, conserving computing power and reducing inference latency.

[0195] The GPT reasoning process using KV cache technology is divided into the first stage, namely the first round of context filling (Prefill) stage and the second stage, namely the multi-round decoding (Decode) stage.

[0196] During the Prefill phase, the input data dimension is [B, S], where B is the batch dimension, commonly used in machine learning. The model's intermediate layer data processing and transmission dimensions are [B, S, H], with H being the hidden layer dimension. The output data dimension is [B, 1], representing a batch of tokens. If there is only one batch, there is only one token, and inference is performed only once. During inference, the key-value cache (KV) information from the intermediate layer (i.e., the transformer layer) is cached in the KV cache for use during the Decode phase.

[0197] During the decoding phase, the input data dimension is [B, 1]. The model's intermediate layer data processing and transmission dimension is [B, 1, H]. The output data dimension is [B, 1]. The decoding phase is executed N times until a stopping condition is met. Similarly, the inference process first retrieves the key-value cache KV information stored in the previous round from the KV cache and simultaneously caches the newly generated key-value cache KV information into the KV cache for updating.

[0198] Table 1 GPT large model calculation and memory access relationship

[0199] Through statistical calculations, we can obtain the relationship between the computational effort and memory access during the inference process when B = 1, L = 88, and S = 4K / 16K, as shown in Table 1, where L is the number of Transformers in the network layer. Clearly, computation is the dominant factor in the Prefill phase (high latency, computational bottleneck), while memory access is the dominant factor in the Decode phase (low latency, memory bottleneck). When there are a large number of user service requests, inference is performed in batches of different service request groups. The significant differences between the two inference phases lead to two problems: 1. Because short input statements within a batch are typically padded to the length of the long input statement before inference is performed, a large amount of inefficient computation is incurred during the Prefill phase for short input statements. 2. Short output statements must wait for the longest output statement in a batch to complete inference before being output, which also results in a large amount of inefficient computation. These two issues lead to very low throughput for the entire system.

[0200] Based on the above technical problems, an embodiment of the present invention provides a request scheduling system. The present invention mainly solves the problem of improving the throughput of the service system in the deployment of large-scale language model services, wherein the large-scale language model is a model with a transformer structure.

[0201] The main current issues are: Large model inference processes involve two inference states: the Prefill phase and the Decode phase. Service deployment fails to consider the significant differences and unique characteristics of the two phases in terms of computing power and memory access. Broadly deploying both phases on the same compute node leads to significant service deployment costs when a large number of services request services simultaneously, as providing a single service request exclusively uses the same compute node. Batched inference (i.e., grouping different services into batches and placing them on the same compute node) results in significant redundant computation and latency overhead, resulting in low overall service system throughput.

[0202] FIG3 is an architecture diagram of a request scheduling system provided by an embodiment of the present invention.

[0203] The request scheduling system includes a first device, at least one second device, and at least one third device. The second device and the third device are both connected to the first device. The connection includes a wired connection or a wireless connection.

[0204] The hardware structure and software structure of the device provided by the embodiment of the present invention can refer to the relevant description of the device 100 in Figures 1 and 2.

[0205] Exemplarily, the first device includes a server.

[0206] Exemplarily, the second device and the third device include electronic devices with computing capabilities. The electronic device includes a computing card. Exemplarily, the computing card includes a GPU or an NPU. The second device is used to perform prefill inference, i.e., inference during the prefill phase; the third device is used to perform decode inference, i.e., inference during the decode phase.

[0207] Taking the request scheduling system including the first device, the second device and the third device as an example, as shown in FIG3 , the request scheduling system includes the first device 210 , the second device 220 and the third device 230 .

[0208] The first device 210 includes a large-scale language model (LLM) service module 211 and a service scheduling module 212 .

[0209] The LLM service module 211 is used to connect to the external user service portal.

[0210] The service scheduling module 212 is used to schedule new services to enter the second device 220 and schedule old services that have not met the stop condition to enter the third device 230.

[0211] The second device 220 includes a first reasoning component package 221 , a first AI engine 222 , and a first computing resource module 223 .

[0212] The first reasoning component package 221 and the first AI engine 222 are used for deploying and calling resources of large-scale language models.

[0213] The first computing resource module 223 includes a first batch scheduling module 2231 and a first execution engine 2232 .

[0214] The first batch scheduling module 2231 is used to dynamically batch or dynamically insert new services, wherein new services are services that have not passed the Prefill stage.

[0215] The first execution engine 2232 is used to perform inference calculations on the service requests output by the first batch scheduling module 2231 .

[0216] The third device 230 includes a second reasoning component package 231 , a second AI engine 232 , and a second computing resource module 233 .

[0217] The second reasoning component package 231 and the second AI engine 232 are used for deploying and calling resources of large-scale language models.

[0218] The second computing resource module 233 includes a second batch scheduling module 2331 and a second execution engine 2332 .

[0219] The second batch scheduling module 2331 is used to dynamically batch, dynamically insert, or dynamically early stop old services that have not met the stop conditions. Among them, old services are services that have passed the Prefill stage.

[0220] The second execution engine 2332 is used to perform inference calculations on the service requests output by the second batch scheduling module 2331 .

[0221] 3 , the second device 220 further includes a first storage module 241, and the third device 230 further includes a second storage module 242. The second device 220 and the third device 230 are connected by wire or wirelessly.

[0222] In some possible embodiments, the first storage module 241 is used to cache the first cache information generated by the second device 220 during the reasoning process, and transmit the cached first cache information to the second storage module 242; the second execution engine 2232 is used to obtain the first cache information from the second storage module 242 during the reasoning process.

[0223] In some possible embodiments, the first execution engine 2232 does not send the first cache information generated by the second device 220 during the reasoning process to the first storage module 241, but directly sends it to the second execution engine 2232 for use by the second execution engine 2232 during the reasoning process.

[0224] The second storage module 242 is also used to cache cache information generated by the third device 230 during the reasoning process; the second execution engine 2232 is used to obtain the cache information generated by the previous round of reasoning process from the second storage module 242 during the reasoning process.

[0225] Exemplarily, the cache information is key-value cache KV information.

[0226] The service scheduling module 212 dispatches multiple service requests to the second device 220 or the third device 230. The second device 220 or the third device 230 then calculates and obtains the output information of the current iteration and the corresponding cache information. The output information is then returned to the service scheduling module 212. If the stop condition is not met, the service scheduling module 212 dispatches the service to an idle third device 230 for the next round of inference. After the inference iteration, the information is returned to the service scheduling module 212. If the stop condition is met, the service scheduling module 212 terminates the scheduling of the service and issues an external business response.

[0227] The embodiment of the present invention also adopts an independent and separate deployment method in the request scheduling system, deploying two execution containers: the second device 220, namely the Prefill inference execution container and the third device 230, namely the Decode inference execution container. Through the scheduling of the request by the first device, the second device and the third device respectively execute the inference process of their respective stages; at the same time, communication can be transmitted between the two devices, so that the inference process can proceed normally, thereby improving the throughput of the service system.

[0228] Both execution containers have a batch scheduling module that implements two functions: automatic batching and automatic early stopping. Automatic batching dynamically groups the input information of multiple service requests into batches, including batching and dynamic insertion. Automatic early stopping prematurely stops completed services, freeing up corresponding resources and notifying the system that the current resources are available for insertion and occupation by the next service.

[0229] By adopting a separate deployment approach, the significant differences between the computing power bottleneck in the Prefill phase and the memory access bottleneck in the Decode phase are comprehensively considered, which can improve the throughput of the service system.

[0230] When a large number of secondary and tertiary devices are required in a request scheduling system, independent, separate deployment methods are used. Secondary devices are configured with small or single batches, while tertiary devices are configured with large batches. The resource ratio between secondary and tertiary devices can be dynamically adjusted based on system load to maximize throughput. This invention can significantly improve the throughput of the service system.

[0231] Based on the system architecture shown in Figure 3, an embodiment of the present invention provides a request scheduling method. Figure 4 is a signaling interaction diagram of a request scheduling method provided by an embodiment of the present invention. As shown in Figure 4, the method includes:

[0232] Step 302: The LLM service module receives a first request from a first user. Prior to step 302, i.e., before requesting the scheduling system to respond to the service, the LLM has already been deployed to the computing resource modules in the second device 220 and the third device 230 via the inference component package and the AI ​​engine. The computing resource modules and storage modules are both ready.

[0233] Exemplarily, the first request is a request for a first service.

[0234] Illustratively, the first request includes an identifier of the first service.

[0235] Illustratively, the identifier includes a service ID.

[0236] Exemplarily, the first request does not include requests returned by the second device and the third device, that is, the first request is a request for a new service.

[0237] In this step, as shown in FIG3 , the LLM service module receives a first request from a first user.

[0238] Step 304: The LLM service module sends the first request to the service scheduling module.

[0239] In this step, as shown in FIG3 , the LLM service module sends the first request to the service scheduling module.

[0240] Step 306: The service scheduling module sends the first request to the first reasoning component package.

[0241] In this step, as shown in FIG3 , the service scheduling module schedules the newly arrived service request to the Prefill inference execution container, ie, the second device, and simultaneously establishes a service pool for storing service requests that are waiting due to resource constraints.

[0242] Step 308: The first reasoning component sends the first request to the first AI engine.

[0243] In this step, as shown in FIG3 , the first reasoning component sends the first request to the first AI engine.

[0244] Step 310: The first AI engine sends a first request to the first computing resource module.

[0245] In this step, as shown in FIG3 , the first AI engine sends the first request to the first computing resource module.

[0246] Step 312: The first computing resource module batches the first request to obtain a first batch, performs a first inference calculation on the first batch, and obtains a second request and first cache information.

[0247] In this step, as shown in Figure 3, the first batch scheduling module of the first computing resource module batches the first request and at least one seventh request that arrives at the second device simultaneously with the first request, based on the first computing resource state and the first condition, to obtain a first batch. The first execution engine of the first computing resource module performs a first inference calculation on the first batch to obtain a second request and first cache information. The seventh request is a request from the first user, the second user, or another user.

[0248] Exemplarily, the first request further includes input information;

[0249] Exemplarily, the input information includes an input Token sequence.

[0250] Exemplarily, the first computing resource state includes the number of idle resources in the second device.

[0251] Illustratively, the first condition includes the maximum number of batches that the second device can group.

[0252] Exemplarily, the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

[0253] Exemplarily, the first output information includes an output Token sequence. The first output information is a first output Token sequence.

[0254] Exemplarily, the cache information includes key-value cache KV information. The first cache information is first key-value cache KV information.

[0255] In some possible embodiments, step 312 specifically includes: a first batch scheduling module of the first computing resource module obtains input information corresponding to the first request and its first length based on the first request; based on the first computing resource state, the first condition, and the first length, batches the first request and at least one seventh request that arrives at the second device simultaneously with the first request to obtain a first batch; and a first execution engine of the first computing resource module performs a first inference calculation on the first batch using a single batch processing method to obtain the second request and first cache information of the first service. The length of the input information corresponding to the seventh request is similar to the first length, which helps reduce waste of idle resources during the first inference calculation process.

[0256] Illustratively, the first batch includes multiple requests.

[0257] Exemplarily, the first batch scheduling module groups multiple requests arriving at the same time into a batch.

[0258] Exemplarily, the first inference calculation is an inference calculation in the Prefill stage, ie, a Prefill calculation.

[0259] Exemplarily, the first processing method is to release the computing service resources occupied by the current batch after all requests in the current batch have completed inference calculations, and insert the next batch into the idle computing service resources to perform the first inference calculation. In an embodiment of the present invention, for the inference calculation in the Prefill phase, the inference calculation is performed as a single batch, that is, after all requests in the current batch have completed inference calculations, the computing service resources occupied by the current batch are released, and the next batch is inserted into the idle computing service resources, and the first inference calculation for the next batch begins.

[0260] Exemplarily, the second device releases the storage resources occupied by the current batch while releasing the computing service resources occupied by the current batch.

[0261] Exemplarily, when the second device has idle computing service resources, it notifies the first device of the existence of idle computing service resources on the second device, and the first device schedules a new service to the second device according to the notification.

[0262] Exemplarily, when the second device has no idle computing service resources, the second device notifies the first device that the second device has no idle computing service resources, and the first device suspends scheduling new services to the second device according to the notification.

[0263] Step 314: The first computing resource module sends the first cache information to the second computing resource module.

[0264] In this step, as shown in FIG3 , the first execution engine of the first computing resource module sends the first cache information to the second computing resource module.

[0265] Optionally, as shown in Figure 3, the first computing resource module may send the first cache information to the first storage module, the first storage module may send the first cache information to the second storage module, and then the second storage module may send the first cache information to the second execution engine module. The second storage module may automatically send the first cache information to the second execution engine module, or may send the first cache information to the second execution engine module based on a message sent by the second execution engine module.

[0266] Optionally, as shown in FIG3 , the first computing resource module may send the first cache information directly to the second computing resource module.

[0267] Step 316 : The first computing resource module sends the second request to the service scheduling module, and then continues to execute 318 or 320 .

[0268] In this step, as shown in FIG3 and FIG4 , the first computing resource module sends the second request to the service scheduling module.

[0269] Step 318: When the first output information meets the set conditions, the service scheduling module sends the first output information to the LLM service module. The LLM service module sends the first output information to the first user, and the process ends.

[0270] In an embodiment of the present invention, the first device determines whether the output information of the service meets the set conditions. If the set conditions are not met, the service continues to be scheduled and inferred; if the set conditions are met, the service is stopped early and the second output information is sent to the first user.

[0271] Step 320: When the first output information does not meet the set condition, the service scheduling module schedules the second request to the second reasoning component package.

[0272] In this step, as shown in FIG3 , the service scheduling module sends the second request to the second reasoning component package.

[0273] Exemplarily, the second request is a request for the first service returned by the second device.

[0274] Exemplarily, the set conditions include at least one of the following:

[0275] The length of the output information is greater than or equal to the first threshold, or

[0276] The output includes the terminator.

[0277] Step 322: The second reasoning component sends the second request to the second AI engine.

[0278] In this step, as shown in FIG3 , the second reasoning component sends the second request to the second AI engine.

[0279] Step 324: The second AI engine sends the second request to the second computing resource module.

[0280] In this step, as shown in FIG3 , the second AI engine sends the second request to the second computing resource module.

[0281] Step 326: The second computing resource module batches the second request to obtain a second batch.

[0282] In this step, as shown in FIG3 , the second batch scheduling module of the second computing resource module batches the second requests according to the second computing resource state and the second condition to obtain a second batch.

[0283] In some possible embodiments, step 326 specifically includes: the second batch scheduling module groups the second request and at least one eighth request that arrives at the third device at the same time as the second request into batches according to the second computing resource state and the second condition to obtain a second batch, and the eighth request comes from the second device or the third device.

[0284] Exemplarily, the second computing resource status includes the number of idle resources in the third device.

[0285] Illustratively, the second condition includes the maximum number of batches that the third device can group.

[0286] Illustratively, the second batch includes multiple requests.

[0287] For example, the second batch scheduling module groups at least one request arriving at the same time into a batch. During the decoding phase, the input length of each request is 1. As long as there is an idle space in the current batch, this length-1 token can be placed in the batch for inference.

[0288] Step 328: The second computing resource module performs a second inference calculation on the second batch according to the first cache information to obtain a third request and second cache information.

[0289] In this step, as shown in FIG3 , the second execution engine of the second computing resource module performs a second inference calculation on the second batch according to the first cache information to obtain a third request and second cache information.

[0290] In some possible embodiments, step 328 specifically includes: the second execution engine of the second computing resource module performs a second inference calculation on the second batch using a second processing method based on the first cache information to obtain the third request and the second cache information of the first service.

[0291] Exemplarily, the third request includes the identifier of the first service and the second output information corresponding to the first service, and the third request is associated with the second request.

[0292] Exemplarily, the second cache information is cache information required by the third device when performing the second inference calculation on the third request.

[0293] Exemplarily, the second inference calculation is an inference calculation in the Decode stage, ie, a Decode calculation.

[0294] Exemplarily, the second processing method is that if there is a target request whose output information meets the set conditions during the current batch inference process, the computing service resources of the target request are released and the requests in the next batch are inserted into the idle computing service resources for the second inference calculation. In an embodiment of the present invention, for the inference calculation in the Decode stage, the inference calculation is performed in batches, that is, during the current batch inference process, if a service ends the inference early, that is, there is a target request whose output information meets the set conditions, the computing service resources of the target request can be released in advance, and before the current batch ends, the requests in the next batch can be inserted into the idle computing service resources for the second inference calculation. Therefore, compared to the first processing method, which needs to wait for all requests in the current batch to be reasoned and released before releasing the resources of the current batch, the second processing method does not need to wait for all requests in the current batch to be reasoned and released before releasing the resources of the current batch, and there will be no waste of idle resources, thereby further improving the throughput of the system.

[0295] Exemplarily, the third device releases the storage resources occupied by the target request while releasing the computing service resources occupied by the target request.

[0296] Exemplarily, when the third device has idle computing service resources, it notifies the first device that the third device has idle computing service resources, and the first device schedules the old service to the second device according to the notification.

[0297] Exemplarily, when the third device has no idle computing service resources, the third device notifies the first device that the third device has no idle computing service resources, and the first device suspends scheduling old services to the third device according to the notification.

[0298] Step 330: The second computing resource module sends the second cache information to the second storage module.

[0299] In this step, as shown in FIG3 , the second execution engine of the second computing resource module sends the second cache information to the second storage module.

[0300] Step 332 : The second computing resource module sends the third request to the service scheduling module, and continues to execute step 334 or step 336 .

[0301] In this step, as shown in FIG3 , the second execution engine of the second computing resource module sends the third request to the service scheduling module.

[0302] Exemplarily, the third request is a request for the first service returned by the third device.

[0303] Step 334: When the second output information meets the set conditions, the service scheduling module sends the second output information to the LLM service module. The LLM service module sends the second output information to the first user, and the process ends.

[0304] In this embodiment of the present invention, a first device determines whether the output information of a service meets a set condition. If not, the service continues to be scheduled and inferred. If the condition is met, the service is prematurely stopped and a second output message is sent to the first user. After prematurely stopping the service, the first device can notify a third device to release the resources corresponding to the service. The third device then reports its current resource status to the first device. If the current resource status contains idle computing service resources, the first device continues to schedule the service to the third device to insert the idle computing service resources, thereby achieving dynamic insertion.

[0305] In this step, as shown in FIG3 , when the second output information meets the set conditions, the service scheduling module sends the second output information to the LLM service module. The LLM service module sends the second output information to the first user, and the process ends.

[0306] Step 336: When the second output information does not meet the set condition, the service scheduling module schedules the third request to the second reasoning component package.

[0307] In this step, as shown in Figure 3, when the second output information does not meet the set conditions, the service scheduling module sends a third request to the second inference component package, causing the third device to continue the next round of decoding calculations on the third request, using the second cache information generated in the previous round of decoding calculations in the next round of decoding calculations. The second execution engine sends the cache information generated in the next round of decoding calculations to the second storage module for storage, for use by the second execution engine in the next round of decoding calculations.

[0308] Taking the request scheduling service system including 4 second devices and 1 third device as an example, Figure 5 is a schematic diagram of Prefill reasoning and Decode reasoning in an embodiment of the present invention. As shown in Figure 5, the 4 second devices are device 0, device 1, device 2 and device 3; the third device is device x. Figure 5 shows the reasoning process in which Prefill reasoning adopts the first processing method and Decode reasoning adopts the second processing method, that is: one row in the Prefill reasoning on the left represents the processing of a service request in a group of computing resources, with a total of 4 groups of Prefill computing resources and 4 service requests; the 4 rows in the Decode reasoning on the right represent the processing of a batch of 4 service requests in a group of computing resources, with a total of 1 group of computing resources and 4 service requests. In Figure 5, each square represents a computing delay. The Prefill calculation on the left and the Decode calculation process on the right are carried out in parallel, and there is no timing relationship.

[0309] The process of Prefill reasoning and Decode reasoning shown in Figure 5 is as follows:

[0310] 1) The lengths of the input token sequences for services req1, req2, req3, and req4 are 3, 2, 4, and 3, respectively. Therefore, when the four services arrive at devices 0, 1, 2, and 3 simultaneously, req2 will be completed first, followed by req1, req4, and req3.

[0311] 2) During the Prefill calculation, req2, which completes early, will enter the Decode calculation early. The right side of Figure 5 illustrates the system startup process. For device x to form a batch of four requests, all four services, req1, req2, req3, and req4, must enter the Decode calculation. As shown in Figure 5, in the Decode reasoning on the right side, req2 arrives first, followed by req1 and req4, and finally req3. Therefore, req1 and req4 will generate one unit of free bubble, while req3 will generate two units of free bubble. This bubble will disappear after the system stabilizes.

[0312] 3) During the decoding calculation, the number of output tokens for req1 to req4 is 5, 7, 6, and 4, respectively. Therefore, although req4 is the second to enter the decoding calculation during the prefill calculation, it is the first to complete the decoding calculation. Therefore, the output information of req4 is the first in the batch to meet the set conditions. After the decoding calculation of req4 is completed, the calculation is stopped early (early stopping mechanism) and the corresponding resources are released. The next requests to complete the calculation are req1, req2, and req3.

[0313] 4) During the decode computation of req1-4, req5-8 have already arrived at the Prefill inference execution container for computation and have completed computation before req1-4. Therefore, as req4, req1, req2, and req3 successively cease computation, the service scheduling module automatically schedules req6, req5, req7, and req8 to device x. Device x then inserts req6, req5, req7, and req8 into the resource locations previously occupied by req4, req1, req2, and req3, respectively, and performs subsequent computations after dynamic insertion.

[0314] 5) After the system stabilizes, steps 3) and 4) will be repeated to dynamically insert batches and dynamically stop and give service responses.

[0315] It's important to note that when the service scheduling module dispatches the third request for a first service to a third device, it only dispatches the third request to the third device that stores the cached information generated during the previous round of decoding inference for the first service. Therefore, during the decoding phase, the decoding portion is for the service being iterated on the current third device, and requests for that service are not forwarded to other third devices.

[0316] In summary, the present invention sends the request output after inference calculation to the service scheduling module through the execution engine. The service scheduling module determines that the request meets the set conditions, thereby realizing automatic early stopping of the service; the batch scheduling module groups the requests into batches to obtain batches, thereby realizing batch grouping; when there are idle computing service resources in the execution engine, the batch scheduling module automatically sends the requests in the next batch to the execution engine to realize dynamic insertion.

[0317] This embodiment of the present invention dispatches user requests for new services to a second device, the Prefill inference execution container, for inference calculations during the Prefill phase. It also dispatches requests for legacy services that have already passed the Prefill phase to a third device, the Decode inference execution container, for inference calculations during the Decode phase. Through separate deployment, this invention comprehensively considers the significant differences between the computing bottlenecks of the Prefill phase and the memory access bottlenecks of the Decode phase, thereby improving the throughput of the service system.

[0318] Optionally, as shown in FIG6 , the method further includes: steps 338 to 372 .

[0319] Step 338: The LLM service module receives the fourth request from the second user. Prior to step 338, i.e., before requesting the scheduling system to respond to the service, the LLM has already been deployed to the computing resource modules in the second and third devices using the inference component package and AI engine. The computing resource modules and storage modules are ready.

[0320] Exemplarily, the second user is different from the first user, and the first device is capable of scheduling requests from different users.

[0321] Exemplarily, the fourth request is a request for the second service.

[0322] Illustratively, the fourth request includes an identifier of the second service.

[0323] Illustratively, the identifier includes a service ID.

[0324] Exemplarily, the fourth request does not include requests returned by the second device and the third device, that is, the fourth request is a request for a new service.

[0325] In this step, as shown in FIG3 , the LLM service module receives a fourth request from the second user.

[0326] Step 340: The LLM service module sends the fourth request to the service scheduling module.

[0327] In this step, as shown in FIG3 , the LLM service module sends the fourth request to the service scheduling module.

[0328] Step 342: The service scheduling module sends the fourth request to the first reasoning component package.

[0329] In this step, as shown in FIG3 , the service scheduling module schedules the newly arrived service request to the Prefill inference execution container, ie, the second device, and simultaneously establishes a service pool for storing service requests that are waiting due to resource constraints.

[0330] Step 344: The first reasoning component sends the fourth request to the first AI engine.

[0331] In this step, as shown in FIG3 , the first reasoning component sends the fourth request to the first AI engine.

[0332] Step 346: The first AI engine sends the fourth request to the first computing resource module.

[0333] In this step, as shown in FIG3 , the first AI engine sends the fourth request to the first computing resource module.

[0334] Step 348: The first computing resource module batches the fourth request to obtain a third batch, performs a first inference calculation on the third batch, and obtains a fifth request and third cache information.

[0335] In this step, as shown in Figure 3, the first batch scheduling module of the first computing resource module batches the fourth request according to the first computing resource state and the first condition to obtain the third batch; the first execution engine of the first computing resource module performs a first inference calculation on the third batch to obtain the fifth request and the third cache information.

[0336] Exemplarily, the fourth request further includes input information;

[0337] Exemplarily, the input information includes an input Token sequence.

[0338] Exemplarily, the first computing resource state includes the number of idle resources in the second device.

[0339] Illustratively, the first condition includes the maximum number of batches that the second device can group.

[0340] Exemplarily, the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request.

[0341] Exemplarily, the third output information includes an output Token sequence. The third output information is a third output Token sequence.

[0342] Exemplarily, the cache information includes key-value cache KV information. The third cache information is third key-value cache KV information.

[0343] In some possible embodiments, step 348 specifically includes: the first batch scheduling module of the first computing resource module obtains the input information corresponding to the fourth request and its second length based on the fourth request; batches the fourth request to obtain a third batch based on the first computing resource state, the first condition and the second length; the first execution engine of the first computing resource module performs a first inference calculation on the third batch using the first processing method to obtain the fifth request and the third cache information of the second service.

[0344] Illustratively, the third batch includes multiple requests.

[0345] Exemplarily, the first batch scheduling module groups multiple requests arriving at the same time into a batch.

[0346] Exemplarily, the first inference calculation is an inference calculation in the Prefill stage, ie, a Prefill calculation.

[0347] Exemplarily, the first processing method is to release the computing service resources occupied by the current batch after all requests of the current batch are inferred and calculated, and insert the next batch into the idle computing service resources to perform the first inference calculation.

[0348] Step 350: The first computing resource module sends the third cache information to the second computing resource module.

[0349] In this step, as shown in FIG3 , the first execution engine of the first computing resource module sends the third cache information to the second computing resource module.

[0350] Optionally, as shown in Figure 3, the first computing resource module may send the third cache information to the first storage module, the first storage module may send the third cache information to the second storage module, and then the second storage module may send the third cache information to the second execution engine module. The second storage module may automatically send the third cache information to the second execution engine module, or may send the third cache information to the second execution engine module based on a message sent by the second execution engine module.

[0351] Optionally, as shown in FIG3 , the first computing resource module may send the third cache information directly to the second computing resource module.

[0352] Step 352 : The first computing resource module sends the fifth request to the service scheduling module, and then continues to execute 354 or 356 .

[0353] In this step, as shown in FIG3 and FIG4 , the first computing resource module sends the fifth request to the service scheduling module.

[0354] Step 354: When the third output information meets the set conditions, the service scheduling module sends the third output information to the LLM service module. The LLM service module sends the third output information to the second user, and the process ends.

[0355] Step 356: When the third output information does not meet the set condition, the service scheduling module schedules the fifth request to the second reasoning component package.

[0356] In this step, as shown in FIG3 , the service scheduling module sends the fifth request to the second reasoning component package.

[0357] Exemplarily, the fifth request is a request for the second service returned by the second device.

[0358] Exemplarily, the set conditions include at least one of the following:

[0359] The length of the output information is greater than or equal to the first threshold, or

[0360] The output includes the terminator.

[0361] Step 358: The second reasoning component sends the fifth request to the second AI engine.

[0362] In this step, as shown in FIG3 , the second reasoning component sends the fifth request to the second AI engine.

[0363] Step 360: The second AI engine sends the fifth request to the second computing resource module.

[0364] In this step, as shown in FIG3 , the second AI engine sends the fifth request to the second computing resource module.

[0365] Step 362: The second computing resource module batches the fifth request to obtain a fourth batch.

[0366] In this step, as shown in FIG3 , the second batch scheduling module of the second computing resource module batches the fifth requests according to the second computing resource state and the second condition to obtain a fourth batch.

[0367] In some possible embodiments, step 362 specifically includes: the second batch scheduling module batches the fifth request according to the second computing resource state and the second condition to obtain a fourth batch.

[0368] Exemplarily, the second computing resource status includes the number of idle resources in the third device.

[0369] Illustratively, the second condition includes the maximum number of batches that the third device can group.

[0370] Illustratively, the fourth batch includes multiple requests.

[0371] For example, the second batch scheduling module groups at least one request arriving at the same time into a batch. During the decoding phase, the input length of each request is 1. As long as there is an idle space in the current batch, this length-1 token can be placed in the batch for inference.

[0372] Step 364: The second computing resource module performs a second inference calculation on the fourth batch according to the third cache information to obtain a sixth request and fourth cache information.

[0373] In this step, as shown in FIG3 , the second execution engine of the second computing resource module performs a second inference calculation on the fourth batch according to the third cache information to obtain a sixth request and fourth cache information.

[0374] In some possible embodiments, step 364 specifically includes: the second execution engine of the second computing resource module performs a second inference calculation on the fourth batch using a second processing method based on the third cache information to obtain the sixth request and the fourth cache information of the second service.

[0375] Exemplarily, the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request.

[0376] Exemplarily, the fourth cache information is cache information required by the third device when performing inference calculation on the sixth request.

[0377] Exemplarily, the second inference calculation is an inference calculation in the Decode stage, ie, a Decode calculation.

[0378] Exemplarily, the second processing method is to release the computing service resources of the target request and insert the requests in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the current batch inference process.

[0379] Step 366: The second computing resource module sends the fourth cache information to the second storage module.

[0380] In this step, as shown in FIG3 , the second execution engine of the second computing resource module sends the fourth cache information to the second storage module.

[0381] Step 368 : The second computing resource module sends the sixth request to the service scheduling module, and continues to execute step 370 or step 372 .

[0382] In this step, as shown in FIG3 , the second execution engine of the second computing resource module sends the sixth request to the service scheduling module.

[0383] Exemplarily, the sixth request is a request for the second service returned by the third device.

[0384] Step 370: When the fourth output information meets the set conditions, the service scheduling module sends the fourth output information to the LLM service module. The LLM service module sends the fourth output information to the second user, and the process ends.

[0385] In this step, as shown in FIG3 , when the fourth output information meets the set conditions, the service scheduling module sends the fourth output information to the LLM service module. The LLM service module sends the fourth output information to the second user, and the process ends.

[0386] Step 372: When the fourth output information does not meet the set condition, the service scheduling module schedules the sixth request to the second reasoning component package.

[0387] Based on the system architecture shown in FIG3 , an embodiment of the present invention provides a request scheduling method. FIG7 is a flow chart of a request scheduling method provided by an embodiment of the present invention. As shown in FIG7 , the method includes:

[0388] Step 402: The first device receives a first request from a first user. The first request is a request for a first service and includes an identifier of the first service.

[0389] Illustratively, the identifier includes a service ID.

[0390] Exemplarily, the first request does not include requests returned by the second device and the third device, that is, the first request is a request for a new service.

[0391] Step 404: The first device dispatches the first request to the second device.

[0392] The first device dispatches the newly arrived service request to the Prefill inference execution container, that is, the second device, and simultaneously establishes a service pool for storing service requests that are waiting due to resource constraints.

[0393] Step 406: The second device sends a second request to the first device. The second request includes the identifier of the first service and the first output information corresponding to the first service. The second request is associated with the first request.

[0394] Exemplarily, the first output information includes an output Token sequence. The first output information is a first output Token sequence.

[0395] Step 408: When the first output information does not meet the set condition, the first device dispatches the second request to a third device, which is different from the second device.

[0396] Exemplarily, the second request is a request for the first service returned by the second device.

[0397] Exemplarily, the set conditions include at least one of the following:

[0398] The length of the output information is greater than or equal to the first threshold, or

[0399] The output includes the terminator.

[0400] Step 410: The third device sends a third request to the first device. The third request includes an identifier of the first service and second output information corresponding to the first service. The third request is associated with the second request.

[0401] Step 412: When the second output information does not meet the set condition, the first device dispatches the third request to the third device.

[0402] Exemplarily, the third request is a request for the first service returned by the third device.

[0403] The first device sends the third request to the third device, causing the third device to perform a next round of decoding calculations on the third request and use the second cache information generated in the previous round of decoding calculations in the next round of decoding calculations. The third device stores the cache information generated in the next round of decoding calculations for use in the next round of decoding calculations.

[0404] Optionally, as shown in FIG7 , between step 404 and step 406 , steps 4052 to 4054 are also included.

[0405] Step 4052: The second device batches the first request according to the first computing resource state and the first condition to obtain a first batch.

[0406] Exemplarily, the first request further includes input information.

[0407] Exemplarily, the input information includes an input Token sequence.

[0408] Exemplarily, the first computing resource state includes the number of idle resources in the second device.

[0409] Illustratively, the first condition includes the maximum number of batches that the second device can group.

[0410] In some possible embodiments, step 4052 specifically includes: the second device obtains input information corresponding to the first request and its first length based on the first request; and batches the first request and at least one seventh request that arrives at the second device at the same time as the first request to obtain a first batch based on the first computing resource state, the first condition and the first length.

[0411] The seventh request is a request from the first user, the second user, or another user.

[0412] The length of the input information corresponding to the seventh request is similar to the first length, which is beneficial to reducing the waste of idle resources in the first inference calculation process.

[0413] Step 4054: The second device performs a first inference calculation on the first batch to obtain a second request and first cache information.

[0414] Exemplarily, the cache information includes key-value cache KV information. The first cache information is first key-value cache KV information.

[0415] In some possible embodiments, step 4052 specifically includes: the second device performs a first inference calculation on the first batch using a first processing method.

[0416] Exemplarily, the first inference calculation is an inference calculation in the Prefill stage, ie, a Prefill calculation.

[0417] Exemplarily, the first processing method is to release the computing service resources occupied by the current batch after all requests of the current batch are inferred and calculated, and insert the next batch into the idle computing service resources to perform the first inference calculation.

[0418] In the embodiment of the present invention, after step 4052, the process further includes: the second device sending the first cache information to the third device.

[0419] Optionally, after step 406, the method further includes: when the first output information meets a set condition, the first device sends the first output information to the first user.

[0420] Optionally, as shown in FIG7 , between step 408 and step 410 , steps 4092 to 4094 are also included.

[0421] Step 4092: The third device batches the second requests according to the second computing resource state and the second condition to obtain a second batch.

[0422] Exemplarily, the second computing resource status includes the number of idle resources in the third device.

[0423] Illustratively, the second condition includes the maximum number of batches that the third device can group.

[0424] In some possible embodiments, step 4092 specifically includes: grouping the second request and at least one eighth request that arrives at the third device at the same time as the second request into batches according to the second computing resource state and the second condition to obtain a second batch, wherein the eighth request comes from the second device or the third device.

[0425] Step 4094: The third device performs a second inference calculation on the second batch based on the first cache information to obtain a third request and second cache information.

[0426] For example, the second inference calculation is the inference calculation in the Decode phase, i.e., the Decode calculation. In the Decode phase, the input length of each request is 1. As long as there is an idle position in the current batch, this token of length 1 can be placed in the batch for inference.

[0427] In some possible embodiments, step 4094 specifically includes: the third device performs a second inference calculation on the second batch using a second processing method based on the first cache information to obtain a third request and second cache information.

[0428] Exemplarily, the second processing method is to release the computing service resources of the target request and insert the requests in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the current batch inference process.

[0429] In the embodiment of the present invention, after step 4094, the method further includes: a third device storing the second cache information.

[0430] Exemplarily, the second cache information is cache information required by the third device when performing the second inference calculation on the third request.

[0431] Optionally, as shown in FIG7 , step 412 further includes: steps 414 to 424 .

[0432] Step 414: The first device receives a fourth request from the second user, where the fourth request is a request for the second service and includes an identifier of the second service.

[0433] Step 416: The first device dispatches the fourth request to the second device.

[0434] Step 418: The second device sends a fifth request to the first device. The fifth request includes the identifier of the second service and the third output information corresponding to the second service. The fifth request is associated with the fourth request.

[0435] Step 420: When the third output information does not meet the set condition, the first device dispatches the fifth request to the third device.

[0436] Step 422: The third device sends a sixth request to the first device. The sixth request includes the identifier of the second service and fourth output information corresponding to the second service. The sixth request is associated with the fifth request.

[0437] Step 424: When the fourth output information does not meet the set condition, the first device dispatches the sixth request to the third device.

[0438] Optionally, between step 416 and step 418 are also included: step 4172 to step 4174.

[0439] Step 4172: The second device batches the fourth requests according to the first computing resource state and the first condition to obtain a third batch.

[0440] Exemplarily, the fourth request further includes input information.

[0441] In some possible embodiments, step 4172 specifically includes: the second device obtains input information corresponding to the fourth request and its second length according to the fourth request; and batches the fourth request to obtain a third batch according to the first computing resource state, the first condition and the second length.

[0442] Step 4174: The second device performs the first inference calculation on the third batch to obtain the fifth request and third cache information.

[0443] In some possible embodiments, step 4172 specifically includes: the second device performs a first inference calculation on the third batch using a first processing method.

[0444] Exemplarily, the first processing method is to release the computing service resources occupied by the current batch after all requests of the current batch are inferred and calculated, and insert the next batch into the idle computing service resources to perform the first inference calculation.

[0445] In the embodiment of the present invention, after step 4172, the process further includes: the second device sending the third cache information to the third device.

[0446] Optionally, after step 418, the method further includes: when the third output information meets a set condition, the first device sends the third output information to the first user.

[0447] Optionally, between step 420 and step 422 are also included: step 4212 to step 4214.

[0448] Step 4212: The third device batches the fifth requests according to the second computing resource state and the second condition to obtain a fourth batch.

[0449] Step 4214: The third device performs a second inference calculation on the fourth batch based on the third cache information to obtain a sixth request and fourth cache information.

[0450] In some possible embodiments, step 4214 specifically includes: the third device performs a second inference calculation on the fourth batch using a second processing method according to the third cache information to obtain a sixth request and fourth cache information.

[0451] Exemplarily, the second processing method is to release the computing service resources of the target request and insert the requests in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the current batch inference process.

[0452] In the embodiment of the present invention, after step 4094, the method further includes: the third device storing the fourth cache information.

[0453] The present invention discloses a request scheduling method and device, which includes: a first device receives a first request from a first user, the first request is a request for a first service, and the first request includes an identifier of the first service; the first device schedules the first request to a second device; the second device sends a second request to the first device, the second request includes an identifier of the first service and first output information, and the second request is associated with the first request; when the first output information does not meet a set condition, the first device schedules the second request to a third device, and the third device is different from the second device; the third device sends a third request to the first device, the third request includes an identifier of the first service and second output information, and the third request is associated with the second request; when the second output information does not meet the set condition, the first device schedules the third request to the third device, which can improve the throughput of the service system in large-scale language model service deployment.

[0454] Based on the system architecture shown in FIG3 , an embodiment of the present invention provides a request scheduling method, which is applied to a first device. FIG8 is a flow chart of another request scheduling method provided by an embodiment of the present invention. As shown in FIG8 , the method includes:

[0455] Step 502: The first device receives a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service.

[0456] Step 504: dispatch the first request to the second device.

[0457] Step 506: Receive a second request sent by the second device, where the second request includes the identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

[0458] Step 508: When the first output information does not meet the set condition, dispatch the second request to a third device, which is different from the second device.

[0459] Step 510: Receive a third request sent by a third device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request.

[0460] Step 512: When the second output information does not meet the set condition, dispatch the third request to the third device.

[0461] In the technical solution of the request scheduling method provided by an embodiment of the present invention, the method includes: a first device receives at least one first request, the first request is a service request, and schedules at least one first request to a second device, the first request does not include requests returned by the second device and the third device; receives at least one second request returned by the second device, and schedules at least one second request to a third device; receives at least one third request returned by the third device, and determines whether the third request meets a third condition based on the third request. If it is determined that the third request does not meet the third condition, the third request is scheduled to the third device, which can improve the throughput of the service system in large-scale language model service deployment.

[0462] FIG9 is a schematic diagram of the structure of a first device provided by an embodiment of the present invention. It should be understood that the first device 600 can execute each step of the first device in the above-mentioned request scheduling method. To avoid repetition, detailed description is omitted here. The first device 600 includes: a first transceiver unit 601 .

[0463] The first transceiver unit 601 is used to receive a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service; schedule the first request to a second device; receive a second request sent by the second device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request; when the first output information does not meet the set conditions, schedule the second request to a third device, where the third device is different from the second device; receive a third request sent by the third device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request; when the second output information does not meet the set conditions, schedule the third request to the third device.

[0464] Optionally, the first transceiver unit 601 is also used to receive a fourth request from a second user, the fourth request being a request for a second service, the fourth request including an identifier of the second service; scheduling the fourth request to the second device; receiving a fifth request sent by the second device, the fifth request including an identifier of the second service and third output information corresponding to the second service, the fifth request being associated with the fourth request; when the third output information does not meet the set conditions, scheduling the fifth request to a third device; receiving a sixth request sent by the third device, the sixth request including an identifier of the second service and fourth output information corresponding to the second service, the sixth request being associated with the fifth request; when the fourth output information does not meet the set conditions, scheduling the sixth request to the third device.

[0465] Optionally, the first transceiver unit 601 is further configured to send the second output information to the first user when the second output information meets the set condition.

[0466] Optionally, the identifier includes a service ID.

[0467] Optionally, the set condition includes at least one of the following:

[0468] The length of the output information is greater than or equal to the first threshold, or

[0469] The output information includes a terminator.

[0470] FIG10 is a schematic diagram of the structure of a second device provided by an embodiment of the present invention. It should be understood that the second device 700 can execute each step of the second device in the above-mentioned request scheduling method. To avoid repetition, the second device 700 includes: a second transceiver unit 701 and a second processing unit 702.

[0471] The second transceiver unit 701 is used to receive a first request from a first user sent by a first device, where the first request is a request for a first service and includes an identifier of the first service; and send a second request to the first device, where the second request includes the identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

[0472] Optionally, the second transceiver unit 701 is also used to receive a fourth request from a second user sent by the first device, where the fourth request is a request for a second service and includes an identifier of the second service; and send a fifth request to the first device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request.

[0473] Optionally, before the second transceiver unit 701 sends the second request to the first device, the second processing unit 702 is used to batch the first request and at least one seventh request that arrives at the second device at the same time as the first request according to the first computing resource state and the first condition to obtain a first batch, where the seventh request is a request from the first user or the second user or other users; and perform a first inference calculation on the first batch to obtain the second request and the first cache information.

[0474] Optionally, the second transceiver unit 701 is further configured to send the first cache information to the third device.

[0475] Optionally, the cache information includes key-value cache KV information.

[0476] Optionally, the second processing unit 702 is specifically used to obtain input information corresponding to the first request and its first length based on the first request; and batch the first request and at least one seventh request that arrives at the second device at the same time as the first request to obtain a first batch based on the first computing resource status, the first condition and the first length.

[0477] FIG11 is a schematic diagram of the structure of a third device provided by an embodiment of the present invention. It should be understood that the third device 800 can execute each step of the third device in the above-mentioned request scheduling method. To avoid repetition, the third device 800 is not described in detail here. The third device 800 includes: a third transceiver unit 801 and a third processing unit 802.

[0478] The third transceiver unit 801 is used to receive a second request from a second device sent by a first device, where the second request includes an identifier of a first service and first output information corresponding to the first service, and the second request is associated with the first request, where the first request is a request for the first service from a first user; and send a third request to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request.

[0479] Optionally, the third transceiver unit 801 is also used to receive a fifth request from the second device sent by the first device, the fifth request including the identifier of the second service and the third output information corresponding to the second service, the fifth request is associated with the fourth request, and the fourth request is a request for the second service from the second user; and send a sixth request to the first device, the sixth request including the identifier of the second service and the fourth output information corresponding to the second service, and the sixth request is associated with the fifth request.

[0480] Optionally, before the third transceiver unit 801 sends the third request to the first device, the third processing unit 802 is configured to group the second request and at least one eighth request that arrives at the third device simultaneously with the second request into a batch according to the second computing resource state and the second condition, where the eighth request comes from the second device or the third device;

[0481] The third transceiver unit 801 is further configured to receive the first buffer information sent by the second device;

[0482] The third processing unit 802 is further configured to perform a second inference calculation on the second batch according to the first cache information to obtain the third request and the second cache information.

[0483] It should be understood that the first device 600, the second device 700, and the third device 800 here are embodied in the form of functional units. The term "unit" here can be implemented in the form of software and / or hardware, and is not specifically limited to this. For example, a "unit" can be a software program, a hardware circuit, or a combination of the two that implements the above-mentioned functions. The hardware circuit may include an application-specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functions.

[0484] Therefore, the various exemplary units described in the embodiments of the present invention can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0485] The present application provides a device, which can be a terminal device or a circuit device built into the terminal device, and can be used to perform the functions / steps of the above method embodiments.

[0486] An embodiment of the present application provides a readable storage medium, which stores instructions. When the instructions are executed on a device or a processor, the device executes the functions / steps in the above method embodiment.

[0487] An embodiment of the present application further provides a program product comprising instructions, which, when executed on a device or any at least one processor, enables the device to perform the functions / steps in the above-mentioned method embodiment.

[0488] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0489] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0490] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0491] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0492] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A request scheduling method, characterized in that: Applied to a request scheduling system, the method comprises: The first device receives a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service; The first device dispatches the first request to the second device; The second device sends a second request to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request; When the first output information does not meet a set condition, the first device dispatches the second request to a third device, where the third device is different from the second device; The third device sends a third request to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request; When the second output information does not satisfy the set condition, the first device dispatches the third request to the third device.

2. The method according to claim 1, characterized in that The method further comprises: The first device receives a fourth request from a second user, where the fourth request is a request for a second service and includes an identifier of the second service; The first device dispatches the fourth request to the second device; The second device sends a fifth request to the first device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request; When the third output information does not satisfy the set condition, the first device dispatches the fifth request to the third device; The third device sends a sixth request to the first device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request; When the fourth output information does not satisfy the set condition, the first device dispatches the sixth request to the third device.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: When the second output information meets the set condition, the first device sends the second output information to the first user.

4. The method according to any one of claims 1 to 3, characterized in that The request scheduling system includes the first device, at least one of the second devices, and at least one of the third devices.

5. A request scheduling method, characterized in that: Applied to a first device, the method includes: receiving a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service; dispatching the first request to a second device; receiving a second request sent by the second device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request; When the first output information does not meet the set condition, dispatching the second request to a third device, the third device being different from the second device; receiving a third request sent by the third device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request; When the second output information does not meet the set condition, scheduling the third request to the third device.

6. The method according to claim 5, characterized in that The method further comprises: receiving a fourth request from a second user, where the fourth request is a request for a second service and includes an identifier of the second service; dispatching the fourth request to the second device; receiving a fifth request sent by the second device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request; When the third output information does not satisfy the set condition, dispatching the fifth request to the third device; receiving a sixth request sent by the third device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request; When the fourth output information does not meet the set condition, scheduling the sixth request to the third device.

7. The method according to claim 5, characterized in that The method further comprises: When the second output information meets the set condition, the second output information is sent to the first user.

8. The method according to any one of claims 5 to 7, characterized in that: The identification includes a service ID.

9. The method according to any one of claims 5 to 8, characterized in that: The set conditions include at least one of the following: The length of the output information is greater than or equal to the first threshold, or The output information includes a terminator.

10. A request scheduling method, characterized in that: Applied to the second device, the method includes: receiving a first request from a first user and sent by a first device, where the first request is a request for a first service and includes an identifier of the first service; A second request is sent to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

11. The method according to claim 10, characterized in that The method further comprises: receiving a fourth request from a second user sent by the first device, where the fourth request is a request for a second service and includes an identifier of the second service; A fifth request is sent to the first device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request.

12. The method according to claim 10, characterized in that Before sending the second request to the first device, the method further includes: According to the first computing resource state and the first condition, the first request and at least one seventh request arriving at the second device simultaneously with the first request are grouped into batches to obtain a first batch, wherein the seventh request is a request from the first user or the second user or another user; A first inference calculation is performed on the first batch to obtain the second request and the first cache information.

13. The method according to claim 12, characterized in that The method further comprises: The first cache information is sent to the third device.

14. The method according to claim 12 or 13, characterized in that The cache information includes key-value cache KV information.

15. The method according to any one of claims 12 to 14, characterized in that: The step of grouping the first request and at least one seventh request arriving at the second device simultaneously with the first request into batches according to the first computing resource state and the first condition to obtain a first batch includes: According to the first request, obtaining input information corresponding to the first request and a first length thereof; The first request and the at least one seventh request are batched to obtain a first batch according to the first computing resource state, the first condition, and the first length.

16. A request scheduling method, characterized in that: Applied to a third device, the method includes: receiving a second request from a second device sent by a first device, where the second request includes an identifier of a first service and first output information corresponding to the first service, where the second request is associated with the first request, and where the first request is a request for the first service from a first user; A third request is sent to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request.

17. The method according to claim 16, characterized in that The method further comprises: receiving a fifth request from a second device sent by the first device, the fifth request including an identifier of a second service and third output information corresponding to the second service, the fifth request being associated with a fourth request, and the fourth request being a request for the second service from a second user; A sixth request is sent to the first device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request.

18. The method according to claim 16 or 17, characterized in that Before sending the third request to the first device, the method further includes: According to a second computing resource state and a second condition, the second request and at least one eighth request arriving at the third device simultaneously with the second request are grouped into batches to obtain a second batch, wherein the eighth request comes from the second device or the third device; Receiving first cache information sent by the second device; A second inference calculation is performed on the second batch according to the first cache information to obtain the third request and the second cache information.

19. A device, characterized in that The method comprises a processor and a memory, wherein the memory is used to store a computer program, and the computer program comprises program instructions. When the processor runs the program instructions, the device executes the steps of the method as described in any one of claims 5 to 9, the steps of the method as described in any one of claims 10 to 15, or the steps of the method as described in any one of claims 16 to 18.

20. A readable storage medium, characterized in that: The readable storage medium stores a program, which includes program instructions. When the program request is executed by the device, the device executes the method according to any one of claims 5 to 9, the method according to any one of claims 10 to 15, or the method according to any one of claims 16 to 18.

Images