A request scheduling method and device

Abstract

This invention discloses a request scheduling method and apparatus. The method includes: a first device receiving a first request from a first user, the first request being a request for a first service, the first request including an identifier of the first service; the first device scheduling the first request to a second device; the second device sending a second request to the first device, the second request including an identifier of the first service and first output information, the second request being associated with the first request; when the first output information does not meet the set conditions, the first device scheduling the second request to a third device, the third device being different from the second device; the third device sending a third request to the first device, the third request including an identifier of the first service and second output information, the third request being associated with the second request; when the second output information does not meet the set conditions, the first device scheduling the third request to the third device, which can improve the throughput of the service system in a large-scale language model service deployment.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a request scheduling method and apparatus. Background Technology

[0002] Large-scale language models (LLMs) have sparked a new wave in the field of artificial intelligence (AI). The inference process of generative pre-trained transformers (GPTs) using key-value caching (KV caching) technology consists of two phases: an initial context filling phase and multiple decoding phases. Although dynamic scheduling is employed in the deployment of large-scale language model services, the throughput of LLM service deployments remains relatively low. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a request scheduling method and device that can improve the throughput of service systems in LLM service deployments.

[0004] In a first aspect, embodiments of the present invention provide a request scheduling method applied to a request scheduling system, the method comprising:

[0005] The first device receives a first request from a first user, the first request being a request for a first service, and the first request including the identifier of the first service;

[0006] The first device will schedule the first request to the second device;

[0007] The second device sends a second request to the first device. The second request includes the identifier of the first service and the first output information corresponding to the first service. The second request is associated with the first request.

[0008] When the first output information does not meet the set conditions, the first device will schedule the second request to a third device, which is different from the second device;

[0009] The third device sends a third request to the first device. The third request includes the identifier of the first service and the second output information corresponding to the first service. The third request is associated with the second request.

[0010] When the second output information does not meet the set conditions, the first device will schedule the third request to the third device.

[0011] In this embodiment of the invention, a separate deployment method is adopted in the request scheduling system, deploying a second device and a third device. Through the scheduling of requests by the first device, the second device and the third device respectively execute the inference process of their respective stages. At the same time, the two devices can exchange information, so that the inference process can proceed normally, thereby improving the throughput of the service system.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0013] The first device receives a fourth request from the second user, the fourth request being a request for the second service, the fourth request including the identifier of the second service;

[0014] The first device will schedule the fourth request to the second device;

[0015] The second device sends a fifth request to the first device. The fifth request includes the identifier of the second service and the third output information corresponding to the second service. The fifth request is associated with the fourth request.

[0016] When the third output information does not meet the set conditions, the first device will schedule the fifth request to the third device;

[0017] The third device sends a sixth request to the first device. The sixth request includes the identifier of the second service and the fourth output information corresponding to the second service. The sixth request is associated with the fifth request.

[0018] When the fourth output information does not meet the set conditions, the first device will schedule the sixth request to the third device.

[0019] In this embodiment of the invention, the second user is different from the first user. The first device can schedule requests from different users to the second device and the third device for processing, without having to wait for the second device and / or the third device to finish processing one user's request before processing another user's request, thereby improving the throughput of the service system.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0021] When the second output information meets the set conditions, the first device sends the second output information to the first user.

[0022] In this embodiment of the invention, a first device determines whether the output information of a service meets set conditions. If the conditions are not met, the service continues to be scheduled and inferenced. If the conditions are met, the service is stopped early, and second output information is sent to the first user. After the first device stops the service early, it can notify a third device to release the resources corresponding to the service. The third device reports its current resource status to the first device. If there are idle computing service resources in the current resource status, the first device continues to schedule services to the third device to insert idle computing service resources, thereby achieving dynamic insertion.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the request scheduling system includes the first device, at least one second device, and at least one third device. In this embodiment of the invention, when the number of second and third devices in the request scheduling system is large, an independent, separate deployment method is adopted, deploying the second and third devices. The second devices are configured for small batches or single batches, while the third devices are configured for large batches. The resource ratio of the second and third devices can be dynamically adjusted based on system load to obtain maximum throughput, thereby significantly improving throughput.

[0024] Secondly, embodiments of the present invention provide a request scheduling method applied to a first device, the method comprising:

[0025] Receive a first request from a first user, the first request being a request for a first service, the first request including the identifier of the first service;

[0026] The first request is routed to the second device;

[0027] Receive a second request sent by the second device, the second request including the identifier of the first service and the first output information corresponding to the first service, and the second request is associated with the first request;

[0028] When the first output information does not meet the set conditions, the second request is scheduled to a third device, which is different from the second device;

[0029] Receive a third request sent by the third device, the third request including the identifier of the first service and the second output information corresponding to the first service, the third request being associated with the second request;

[0030] When the second output information does not meet the set conditions, the third request is scheduled to the third device.

[0031] In this embodiment of the invention, requests for new services from users are scheduled to a second device, namely a Prefill inference execution container, for inference computation during the Prefill phase; requests for old services that have passed the Prefill phase are scheduled to a third device, namely a Decode inference execution container, for inference computation during the Decode phase. This invention, through a separate deployment approach, comprehensively considers the significant differences between the computational bottlenecks of the Prefill phase and the memory access bottlenecks of the Decode phase, thereby improving the throughput of the service system.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

[0033] Receive a fourth request from a second user, the fourth request being a request for a second service, the fourth request including the identifier of the second service;

[0034] The fourth request is scheduled to the second device;

[0035] Receive a fifth request sent by the second device, the fifth request including the identifier of the second service and the third output information corresponding to the second service, the fifth request being associated with the fourth request;

[0036] When the third output information does not meet the set conditions, the fifth request is scheduled to the third device;

[0037] The sixth request sent by the third device is received. The sixth request includes the identifier of the second service and the fourth output information corresponding to the second service. The sixth request is associated with the fifth request.

[0038] When the fourth output information does not meet the set conditions, the sixth request is scheduled to the third device.

[0039] In this embodiment of the invention, the second user is different from the first user. The first device can schedule requests from different users to the second and third devices for processing, without having to wait for the second and / or third devices to finish processing one user's request before processing another user's request, thereby improving the throughput of the service system.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

[0041] When the second output information meets the set conditions, the second output information is sent to the first user.

[0042] In this embodiment of the invention, a first device determines whether the output information of a service meets set conditions. If the conditions are not met, the service continues to be scheduled and inferenced. If the conditions are met, the service is stopped early, and second output information is sent to the first user. After the first device stops the service early, it can notify a third device to release the resources corresponding to the service. The third device reports its current resource status to the first device. If there are idle computing service resources in the current resource status, the first device continues to schedule services to the third device to insert idle computing service resources, thereby achieving dynamic insertion.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the identifier includes a service ID.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the set conditions include at least one of the following:

[0045] The length of the output information is greater than or equal to the first threshold, or

[0046] The output information includes a terminator.

[0047] Thirdly, embodiments of the present invention provide a request scheduling method applied to a second device, the method comprising:

[0048] Receive a first request from a first user sent by a first device, wherein the first request is a request for a first service and includes an identifier of the first service;

[0049] A second request is sent to the first device. The second request includes the identifier of the first service and the first output information corresponding to the first service. The second request is associated with the first request.

[0050] In this embodiment of the invention, requests for new services from users are scheduled to a second device, namely a Prefill inference execution container, for inference computation during the Prefill phase; requests for old services that have passed the Prefill phase are scheduled to a third device, namely a Decode inference execution container, for inference computation during the Decode phase. This invention, through a separate deployment approach, comprehensively considers the significant differences between the computational bottlenecks of the Prefill phase and the memory access bottlenecks of the Decode phase, thereby improving the throughput of the service system.

[0051] In conjunction with the third aspect, in some implementations of the third aspect, the method further includes:

[0052] Receive a fourth request from the second user sent by the first device, the fourth request being a request for the second service, the fourth request including the identifier of the second service;

[0053] A fifth request is sent to the first device. The fifth request includes the identifier of the second service and the third output information corresponding to the second service. The fifth request is associated with the fourth request.

[0054] In this embodiment of the invention, the second user is different from the first user. The first device can schedule requests from different users to the second and third devices for processing, without having to wait for the second and / or third devices to finish processing one user's request before processing another user's request, thereby improving the throughput of the service system.

[0055] In conjunction with the third aspect, in some implementations of the third aspect, before sending the second request to the first device, the method further includes:

[0056] Based on the first computing resource status and the first condition, the first request and at least one seventh request that arrives at the second device at the same time as the first request are grouped into a first batch, wherein the seventh request is a request from the first user, the second user, or other users.

[0057] Perform a first inference calculation on the first batch to obtain the second request and the first cache information.

[0058] In conjunction with the third aspect, in some implementations of the third aspect, the method further includes:

[0059] The first cached information is sent to the third device.

[0060] In conjunction with the third aspect, in some implementations of the third aspect, the cached information includes key-value cache (KV) information.

[0061] In conjunction with the third aspect, in certain implementations of the third aspect, the step of batching the first request and at least one seventh request arriving at the second device simultaneously with the first request into a first batch based on the first computing resource status and the first condition includes:

[0062] Based on the first request, obtain the input information corresponding to the first request and its first length;

[0063] Based on the first computing resource status, the first condition, and the first length, the first request and the at least one seventh request are batched to obtain the first batch. The length of the input information corresponding to the seventh request is similar to the first length, which helps to reduce the waste of idle resources during the first inference computation process.

[0064] In conjunction with the third aspect, in some implementations of the third aspect, the first computing resource state includes the number of idle resources in the second device.

[0065] In conjunction with the third aspect, in some implementations of the third aspect, the first condition includes the maximum number of batches that the second device can form.

[0066] In conjunction with the third aspect, in some implementations of the third aspect, the first processing method is to release the computing service resources occupied by the current batch after all requests in the current batch have completed the inference calculation and insert the next batch into the idle computing service resources for the first inference calculation.

[0067] In this embodiment of the invention, the inference calculation in the Prefill stage is performed in a single batch. That is, after all requests in the current batch have been inferred and calculated, the computing service resources occupied by the current batch are released and the next batch is inserted into the idle computing service resources to start the first inference calculation for the next batch.

[0068] Fourthly, embodiments of the present invention provide a request scheduling method applied to a third device, the method comprising:

[0069] Receive a second request from a second device sent by a first device. The second request includes an identifier of a first service and first output information corresponding to the first service. The second request is associated with the first request, which is a request from the first user for the first service.

[0070] A third request is sent to the first device. The third request includes the identifier of the first service and the second output information corresponding to the first service. The third request is associated with the second request.

[0071] In this embodiment of the invention, requests for new services from users are scheduled to a second device, namely a Prefill inference execution container, for inference computation during the Prefill phase; requests for old services that have passed the Prefill phase are scheduled to a third device, namely a Decode inference execution container, for inference computation during the Decode phase. This invention, through a separate deployment approach, comprehensively considers the significant differences between the computational bottlenecks of the Prefill phase and the memory access bottlenecks of the Decode phase, thereby improving the throughput of the service system.

[0072] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the method further includes:

[0073] The device receives a fifth request sent by the first device. The fifth request includes the identifier of the second service and the third output information corresponding to the second service. The fifth request is associated with a fourth request, which is a request from the second user for the second service.

[0074] A sixth request is sent to the first device. The sixth request includes the identifier of the second service and the fourth output information corresponding to the second service. The sixth request is associated with the fifth request.

[0075] In this embodiment of the invention, the second user is different from the first user. The first device can schedule requests from different users to the second and third devices for processing, without having to wait for the second and / or third devices to finish processing one user's request before processing another user's request, thereby improving the throughput of the service system.

[0076] In conjunction with the fourth aspect, in some implementations of the fourth aspect, before sending the third request to the first device, the method further includes:

[0077] Based on a second computing resource status and a second condition, the second request and at least one eighth request that arrives at the third device simultaneously with the second request are grouped together to obtain a second batch, wherein the eighth request comes from the second device or the third device.

[0078] Receive the first buffer information sent by the second device;

[0079] Based on the first cache information, a second inference calculation is performed on the second batch to obtain the third request and the second cache information.

[0080] Fifthly, embodiments of the present invention provide a request scheduling system, the system comprising:

[0081] A first device is configured to receive a first request from a first user, the first request being a request for a first service, the first request including an identifier of the first service;

[0082] The first device is also configured to schedule the first request to the second device;

[0083] The second device is configured to send a second request to the first device. The second request includes the identifier of the first service and the first output information corresponding to the first service. The second request is associated with the first request.

[0084] When the first output information does not meet the set conditions, the first device is also used to schedule the second request to a third device, which is different from the second device;

[0085] The third device is configured to send a third request to the first device. The third request includes the identifier of the first service and second output information corresponding to the first service. The third request is associated with the second request.

[0086] When the second output information does not meet the set conditions, the first device is also used to schedule the third request to the third device.

[0087] In a sixth aspect, embodiments of the present invention provide a device including a processor and a memory, wherein the memory is used to store a program, the program including program instructions, and when the processor executes the program instructions, causes the device to perform the steps of the method described above.

[0088] In a seventh aspect, embodiments of the present invention provide a readable storage medium storing a program, the program including program instructions that, when the program requests to be run by a device, cause the device to perform the method described above.

[0089] Eighthly, embodiments of the present invention provide a program product comprising instructions that, when the program product is run on a device or any at least one processor, cause the device to perform the functions / steps as described above.

[0090] In the technical solution of the request scheduling method and device provided in this embodiment of the invention, a first device receives a first request from a first user, the first request being a request for a first service, and the first request including an identifier of the first service; the first device schedules the first request to a second device; the second device sends a second request to the first device, the second request including an identifier of the first service and first output information, and the second request is associated with the first request; when the first output information does not meet the set conditions, the first device schedules the second request to a third device, the third device being different from the second device; the third device sends a third request to the first device, the third request including an identifier of the first service and second output information, and the third request is associated with the second request; when the second output information does not meet the set conditions, the first device schedules the third request to the third device, which can improve the throughput of the service system in a large-scale language model service deployment. Attached Figure Description

[0091] Figure 1 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention;

[0092] Figure 2 This is a software structure block diagram of the device 100 according to an embodiment of the present invention;

[0093] Figure 3 This is an architecture diagram of another request scheduling system provided by an embodiment of the present invention;

[0094] Figure 4 A signaling interaction diagram of a request scheduling method provided in an embodiment of the present invention;

[0095] Figure 5This is a schematic diagram of Prefill inference and Decode inference in an embodiment of the present invention;

[0096] Figure 6 Signaling interaction diagram of another request scheduling method provided in an embodiment of the present invention;

[0097] Figure 7 A flowchart of a request scheduling method provided in an embodiment of the present invention;

[0098] Figure 8 A flowchart of another request scheduling method provided in an embodiment of the present invention.

[0099] Figure 9 This is a schematic diagram of the structure of a first device provided in an embodiment of the present invention;

[0100] Figure 10 This is a schematic diagram of the structure of a second device provided in an embodiment of the present invention;

[0101] Figure 11 This is a schematic diagram of the structure of a third device provided in an embodiment of the present invention. Detailed Implementation

[0102] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0103] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0104] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0105] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0106] Figure 1 A schematic diagram of the structure of device 100 is shown.

[0107] Device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0108] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on device 100. In other embodiments of this application, device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0109] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0110] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0111] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0112] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0113] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thus realizing the touch function of the device 100.

[0114] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0115] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0116] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0117] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the camera's shooting function. The processor 110 and the display screen 194 communicate via the DSI interface to enable the device 100's display function.

[0118] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0119] USB port 130 is a USB standard compliant interface, which can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge device 100, and can also be used for data transfer between device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other devices, such as AR devices.

[0120] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the device 100. In other embodiments of this application, the device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0121] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the device 100. While charging the battery 142, the charging management module 140 can also supply power to the device via the power management module 141.

[0122] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0123] The wireless communication function of device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0124] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0125] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via the antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0126] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0127] The wireless communication module 160 can provide solutions for wireless communication applications on device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0128] In some embodiments, antenna 1 of device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0129] Device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0130] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0131] Device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0132] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0133] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0134] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0135] Video codecs are used to compress or decompress digital video. Device 100 may support one or more video codecs. Thus, device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0136] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0137] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the device 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0138] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor.

[0139] Device 100 can implement audio functions, such as music playback and recording, through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor.

[0140] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0141] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Device 100 can listen to music or make hands-free calls through the speaker 170A.

[0142] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the device 100 is answering a phone call or voice message, the receiver 170B can be brought close to the listener's ear to receive the voice message.

[0143] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Device 100 may have at least one microphone 170C. In some embodiments, device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, device 100 may have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0144] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0145] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, device 100 detects the intensity of the touch operation based on pressure sensor 180A. Device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0146] The gyroscope sensor 180B can be used to determine the motion attitude of the device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the device 100's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the device 100 through reverse movement, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0147] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0148] The magnetic sensor 180D includes a Hall sensor. The device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the device 100 is a flip phone, the device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0149] The 180E accelerometer sensor can detect the magnitude of acceleration of device 100 in various directions (typically three axes). When device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify device posture and is applicable to screen orientation switching, pedometers, and other applications.

[0150] A distance sensor 180F is used to measure distance. Device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0151] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. Device 100 emits infrared light outward through the LED. Device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that an object is near device 100. When insufficient reflected light is detected, device 100 can determine that no object is near device 100. Device 100 may use the proximity sensor 180G to detect when a user holds device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0152] The ambient light sensor 180L is used to sense the ambient light intensity. Device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light intensity. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos. The ambient light sensor 180L can also work in conjunction with the proximity sensor 180G to detect whether device 100 is in a pocket, preventing accidental activation.

[0153] The fingerprint sensor 180H is used to collect fingerprints. The device 100 can use the characteristics of the collected fingerprints to achieve fingerprint unlocking, access to application locks, fingerprint photography, fingerprint answering of incoming calls, etc.

[0154] Temperature sensor 180J is used to detect temperature. In some embodiments, device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, device 100 performs thermal protection by reducing the performance of a processor located near temperature sensor 180J to reduce power consumption. In other embodiments, when the temperature is below another threshold, device 100 heats battery 142 to prevent abnormal shutdown of device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0155] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of device 100, in a different position than display screen 194.

[0156] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0157] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Device 100 can receive button input and generate key signal inputs related to user settings and function control of device 100.

[0158] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0159] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0160] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the device 100. The device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the device 100 and cannot be separated from the device 100.

[0161] The software system of device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses a layered Android system as an example to illustrate the software structure of device 100.

[0162] Figure 2 This is a software structure block diagram of the device 100 according to an embodiment of the present invention.

[0163] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0164] The application layer can include a series of application packages.

[0165] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0166] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0167] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0168] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0169] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0170] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0171] The phone manager is used to provide communication functions for device 100. This includes managing call status (including connection and disconnection).

[0172] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0173] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating the device, and flashing indicator lights.

[0174] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0175] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0176] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0177] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0178] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0179] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0180] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0181] A 2D graphics engine is a graphics engine for 2D drawing.

[0182] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0183] The following example illustrates the workflow of the software and hardware of device 100 in the context of capturing and photographing scenes.

[0184] When touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a touch click as an example, where the corresponding control is the camera application icon, the camera application calls the application framework layer's interface to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 193.

[0185] First, the relevant technologies of the request scheduling method involved in the embodiments of the present invention will be explained so that those skilled in the art can understand them.

[0186] Transformer: A special architecture in deep learning that consists of multi-head self-attention and feedforward feedback. This mechanism assigns different weights based on the importance of different parts of the input data. This model is primarily used in Natural Language Processing (NLP) and Computer Vision (CV).

[0187] Tokens: In Chinese natural language processing, a token corresponds to a character or word in the text. The input text is first processed by a word segmenter and a word list mapping to obtain the corresponding token sequence, and then fed into the model for further processing.

[0188] Prefill inference typically involves very long input information, with latency being a decisive factor, resulting in a computational bottleneck.

[0189] Decode inference: The input information length is fixed at 1, the latency overhead is small, but there is a memory access bottleneck.

[0190] Throughput: The number of requests a system processes per unit of time. The throughput calculation formula used in this invention is as follows:

[0191]

[0192] Large-scale language models have sparked a new wave in the AI ​​field. Generative language models, particularly those based on the Generative Probabilistic Language Process (GPT) model, have a unique inference process. Given an input token sequence of length S, the model outputs an output token sequence of length N. This process involves N inference steps. Specifically, a GPT model outputs only one token per inference step, which is then concatenated with previous input tokens (S+1, S+2, ..., S+N-1) as input for the next inference step, continuing until a terminator or the maximum output length is encountered.

[0193] This approach introduces a problem: as the number of iterations increases, the computational cost of each inference increases accordingly; furthermore, since each iteration includes the input from the previous iteration, there is a significant amount of redundant computation. Therefore, engineers use KV caching technology to keep the computational cost of each inference round after the first iteration relatively constant, thereby saving computing power and reducing inference latency.

[0194] The GPT inference process using KV cache technology is divided into two phases: the first phase, namely the initial context filling phase, and the second phase, namely the multi-round decoding phase.

[0195] In the Prefill phase, the input data dimension is [B, S], where B represents the batch size, a common feature in machine learning. The data processing and transmission dimension in the intermediate layers of the model is [B, S, H], where H represents the hidden layer dimension. The output data dimension is [B, 1], representing the number of tokens in the batch. If there is only one batch, there is only one token, and inference is performed only once. During inference, the key-value cache (KV) information of the intermediate layer, i.e., the Transformer layer, is cached in the KV cache for later use in the Decode phase.

[0196] In the Decode phase, the input data dimension is [B, 1]. The data processing and transmission dimension in the intermediate layers of the model is [B, 1, H]. The output data dimension is [B, 1]. The Decode phase will be executed N times until a stopping condition is encountered. Similarly, the inference process first retrieves the key-value cached KV information stored in the previous round from the KV cache, and simultaneously updates the KV cache with newly generated key-value cached KV information.

[0197] Table 1. GPT Large Model Computation and Memory Access Relationships

[0198]

[0199] Statistical calculations reveal the computational load and memory access relationship during inference when B=1, L=88, and S=4K / 16K, as shown in Table 1, where L represents the number of Transformers in the network layer. Clearly, computation is the decisive factor (high latency, computational bottleneck) in the Prefill phase, while memory access is the decisive factor (low latency, memory bottleneck) in the Decode phase. When there are a large number of user service requests, different service requests will be batched for inference. The significant differences between the two inference phases lead to two problems: 1. Because padding is typically used to extend the length of short input statements within a batch to the length of the long input statement before inference, short input statements involve a large amount of invalid computation in the Prefill phase; 2. Short output statements will wait for the longest output statement in a batch to finish inference before being output, also resulting in a large amount of invalid computation. These two problems lead to very low overall system throughput.

[0200] To address the aforementioned technical problems, embodiments of the present invention provide a request scheduling system. The present invention primarily solves the problem of improving the throughput of service systems deployed in large-scale language model service deployments, where the large-scale language model refers to a model with a transformer structure.

[0201] The main problems currently exist: Large-scale model inference involves two inference states: the Prefill stage and the Decode stage. Service deployment does not consider the significant differences and unique characteristics of these two stages in terms of computing power, memory access, etc.; deploying both stages indiscriminately on the same computing node results in enormous deployment costs when a large number of services request services simultaneously, as each service request requires a dedicated computing node; batch inference (i.e., grouping different services into batches and placing them on the same computing node) leads to significant redundant computation and latency overhead, resulting in low overall throughput of the service system.

[0202] Figure 3 This is an architecture diagram of a request scheduling system provided in an embodiment of the present invention.

[0203] The request scheduling system includes a first device, at least one second device, and at least one third device. Both the second and third devices are connected to the first device. The connection can be a wired connection or a wireless connection.

[0204] The hardware and software structures of the device provided in the embodiments of the present invention can be found in [reference needed]. Figure 1 and Figure 2 The relevant description of equipment 100 in the text.

[0205] For example, the first device includes a server.

[0206] For example, the second and third devices include electronic devices with computing capabilities. The electronic devices include computing cards. For example, the computing cards include GPUs or NPUs. The second device is used for Prefill inference, i.e., inference during the Prefill phase; the third device is used for Decode inference, i.e., inference during the Decode phase.

[0207] Taking a request scheduling system comprising a first device, a second device, and a third device as an example, such as... Figure 3 The request scheduling system includes a first device 210, a second device 220, and a third device 230.

[0208] The first device 210 includes a large-scale language model (LLM) service module 211 and a service scheduling module 212.

[0209] LLM service module 211 is used to connect to external user service entry points.

[0210] The service scheduling module 212 is used to schedule new services to enter the second device 220 and to schedule old services that have not met the stopping conditions to enter the third device 230.

[0211] The second device 220 includes a first inference component package 221, a first AI engine 222, and a first computing resource module 223.

[0212] The first inference component package 221 and the first AI engine 222 are used for the deployment and resource access of large-scale language models.

[0213] The first computing resource module 223 includes a first batch scheduling module 2231 and a first execution engine 2232.

[0214] The first batch scheduling module 2231 is used to dynamically group or insert new services. These new services are those that have not undergone the Prefill phase.

[0215] The first execution engine 2232 is used to perform inference calculations on the service requests output by the first batch scheduling module 2231.

[0216] The third device 230 includes a second inference component package 231, a second AI engine 232, and a second computing resource module 233.

[0217] The second inference component package 231 and the second AI engine 232 are used for the deployment and resource access of large-scale language models.

[0218] The second computing resource module 233 includes a second batch scheduling module 2331 and a second execution engine 2332.

[0219] The second batch scheduling module 2331 is used to dynamically group, insert, or prematurely stop old services that have not met the stopping conditions. These old services are those that have already passed the Prefill phase.

[0220] The second execution engine 2332 is used to perform inference calculations on the service requests output by the second batch scheduling module 2331.

[0221] like Figure 3 As shown, the second device 220 further includes a first storage module 241, and the third device 230 further includes a second storage module 242. The second device 220 and the third device 230 are connected by wired or wireless means.

[0222] In some possible embodiments, the first storage module 241 is used to cache the first cache information generated by the second device 220 during the inference process and to transmit the cached first cache information to the second storage module 242; the second execution engine 2232 is used to obtain the first cache information from the second storage module 242 during the inference process.

[0223] In some possible embodiments, the first execution engine 2232 does not send the first cache information generated by the second device 220 during the inference process to the first storage module 241, but sends it directly to the second execution engine 2232 for use by the second execution engine 2232 during the inference process.

[0224] The second storage module 242 is also used to cache cached information generated by the third device 230 during the inference process; the second execution engine 2232 is used to retrieve cached information generated in the previous round of inference from the second storage module 242 during the inference process.

[0225] For example, the cached information is key-value cached KV information.

[0226] Requests for multiple services are scheduled by the service scheduling module 212 to the second device 220 or the third device 230. The second device 220 or the third device 230 calculates and obtains the output information and corresponding cache information of the current iteration round, and the output information is returned to the service scheduling module 212. When the stopping condition is not met, the service scheduling module 212 schedules this service to the idle third device 230 for the next round of inference; after the inference iteration, it will continue to return to the service scheduling module 212. When the stopping condition is met, the service scheduling module 212 will end the scheduling of this service and provide an external business response.

[0227] In this embodiment of the invention, the request scheduling system adopts an independent and separate deployment method, deploying a second device 220, namely a Prefill inference execution container, and a third device 230, namely a Decode inference execution container. Through the scheduling of requests by the first device, the second device and the third device execute the inference process of their respective stages. At the same time, the two devices can communicate and transmit data, so that the inference process can proceed normally, thereby improving the throughput of the service system.

[0228] Both execution containers have a batch scheduling module, which performs two functions: automatic batching and automatic early stopping. Automatic batching is used to dynamically group the input information of multiple service requests into batches, including both grouping and dynamic insertion. Automatic early stopping is used to stop completed services in advance, release the corresponding resources, and notify the system that the current resources can be inserted and occupied by the next service.

[0229] By deploying the system separately, the significant differences between the computing power bottleneck in the Prefill phase and the memory access bottleneck in the Decode phase are taken into account, which can improve the throughput of the service system.

[0230] When there are a large number of second and third devices in the request scheduling system, an independent, separate deployment approach is adopted. The second devices are configured for small batches or single batches, while the third devices are configured for large batches. The resource ratio of the second and third devices can be dynamically adjusted based on system load to achieve maximum throughput. This invention can significantly improve the throughput of the service system.

[0231] based on Figure 3The system architecture shown in this invention provides a request scheduling method. Figure 4 This is a signaling interaction diagram of a request scheduling method provided in an embodiment of the present invention. For example... Figure 4 As shown, the method includes:

[0232] Step 302: The LLM service module receives the first request from the first user. Before step 302, i.e. before requesting the scheduling system to respond to the service, the LLM has been deployed to the computing resource modules in the second device 220 and the third device 230 through the inference component package and the AI ​​engine. Both the computing resource modules and the storage modules are ready.

[0233] For example, the first request is a request for the first service.

[0234] For example, the first request includes the identifier of the first service.

[0235] For example, the identifier includes a service ID.

[0236] For example, the first request does not include the requests returned by the second and third devices; that is, the first request is a request for a new service.

[0237] In this step, such as Figure 3 As shown, the LLM service module receives the first request from the first user.

[0238] Step 304: The LLM service module sends the first request to the service scheduling module.

[0239] In this step, such as Figure 3 As shown, the LLM service module sends the first request to the service scheduling module.

[0240] Step 306: The service scheduling module sends the first request to the first inference component package.

[0241] In this step, such as Figure 3 As shown, the service scheduling module schedules newly arriving service requests to the Prefill inference execution container, i.e., the second device, and at the same time establishes a service pool to store service requests that are waiting due to resource constraints.

[0242] Step 308: The first inference component package sends the first request to the first AI engine.

[0243] In this step, such as Figure 3 As shown, the first inference component package sends the first request to the first AI engine.

[0244] Step 310: The first AI engine sends the first request to the first computing resource module.

[0245] In this step, such as Figure 3 As shown, the first AI engine sends the first request to the first computing resource module.

[0246] Step 312: The first computing resource module batches the first request to obtain the first batch, performs the first inference calculation on the first batch, and obtains the second request and the first cache information.

[0247] In this step, such as Figure 3 As shown, the first batch scheduling module of the first computing resource module groups the first request and at least one seventh request arriving at the second device simultaneously with the first request into a first batch based on the first computing resource status and the first condition; the first execution engine of the first computing resource module performs a first inference calculation on the first batch to obtain the second request and the first cache information. The seventh request is a request from the first user, the second user, or other users.

[0248] For example, the first request may also include input information;

[0249] For example, the input information includes an input token sequence.

[0250] For example, the first computing resource status includes the number of idle resources in the second device.

[0251] For example, the first condition includes the maximum number of batches that the second device can be grouped into.

[0252] For example, the second request includes the identifier of the first service and the first output information corresponding to the first service, and the second request is associated with the first request.

[0253] For example, the first output information includes an output token sequence. The first output information is a first output token sequence.

[0254] For example, the cached information includes key-value cached KV information. The first cached information is the first key-value cached KV information.

[0255] In some possible embodiments, step 312 specifically includes: the first batch scheduling module of the first computing resource module obtains the input information corresponding to the first request and its first length according to the first request; based on the first computing resource status, the first condition, and the first length, it groups the first request and at least one seventh request arriving at the second device simultaneously with the first request into a first batch; the first execution engine of the first computing resource module performs a first inference calculation on the first batch using a single-batch processing method to obtain the first cache information of the second request and the first service. The length of the input information corresponding to the seventh request is similar to the first length, which helps to reduce the waste of idle resources during the first inference calculation process.

[0256] For example, the first batch includes multiple requests.

[0257] For example, the first batch scheduling module groups multiple requests that arrive at the same time into a batch.

[0258] For example, the first inference computation is the inference computation of the Prefill stage, i.e., the Prefill computation.

[0259] For example, the first processing method is to release the computing service resources occupied by the current batch after all requests in the current batch have completed the inference calculation, and then insert the next batch into the idle computing service resources for the first inference calculation. In this embodiment of the invention, for the inference calculation in the Prefill stage, inference calculation is performed on a single batch basis. That is, after all requests in the current batch have completed the inference calculation, the computing service resources occupied by the current batch are released and the next batch is inserted into the idle computing service resources to start the first inference calculation for the next batch.

[0260] For example, the second device releases the computing service resources occupied by the current batch, as well as the storage resources occupied by the current batch.

[0261] For example, when the second device has idle computing service resources, it notifies the first device that the second device has idle computing service resources, and the first device schedules new services to the second device according to the notification.

[0262] For example, when the second device does not have any idle computing service resources, it notifies the first device that the second device does not have any idle computing service resources, and the first device suspends scheduling new services to the second device according to the notification.

[0263] Step 314: The first computing resource module sends the first cache information to the second computing resource module.

[0264] In this step, such as Figure 3 As shown, the first execution engine of the first computing resource module sends the first cache information to the second computing resource module.

[0265] Optionally, such as Figure 3 As shown, the first computing resource module can send the first cached information to the first storage module, the first storage module sends the first cached information to the second storage module, and then the second storage module sends the first cached information to the second execution engine module. The second storage module can either send the first cached information to the second execution engine module automatically, or it can send the first cached information to the second execution engine module based on a message sent by the second execution engine module.

[0266] Optionally, such as Figure 3As shown, the first computing resource module can directly send the first cache information to the second computing resource module.

[0267] Step 316: The first computing resource module sends the second request to the service scheduling module and continues to execute 318 or 320.

[0268] In this step, such as Figure 3 and Figure 4 As shown, the first computing resource module sends the second request to the service scheduling module.

[0269] Step 318: When the first output information meets the set conditions, the service scheduling module sends the first output information to the LLM service module. The LLM service module then sends the first output information to the first user, and the process ends.

[0270] In this embodiment of the invention, the first device determines whether the output information of the service meets the set conditions. If the set conditions are not met, the service continues to be scheduled and inferenced. If the set conditions are met, the service is stopped early and the second output information is sent to the first user.

[0271] Step 320: When the first output information does not meet the set conditions, the service scheduling module will schedule the second request to the second inference component package.

[0272] In this step, such as Figure 3 As shown, the service scheduling module sends the second request to the second inference component package.

[0273] For example, the second request is a request for the first service returned by the second device.

[0274] For example, the conditions set include at least one of the following:

[0275] The length of the output information is greater than or equal to the first threshold, or

[0276] The output information includes a terminator.

[0277] Step 322: The second inference component package sends the second request to the second AI engine.

[0278] In this step, such as Figure 3 As shown, the second inference component package sends the second request to the second AI engine.

[0279] Step 324: The second AI engine sends the second request to the second computing resource module.

[0280] In this step, such as Figure 3 As shown, the second AI engine sends the second request to the second computing resource module.

[0281] Step 326: The second computing resource module processes the second request into batches to obtain the second batch.

[0282] In this step, such as Figure 3 As shown, the second batch scheduling module of the second computing resource module groups the second request into a second batch based on the second computing resource status and the second condition.

[0283] In some possible embodiments, step 326 specifically includes: the second batch scheduling module, based on the second computing resource status and the second condition, batches the second request and at least one eighth request that arrives at the third device at the same time as the second request to obtain a second batch, wherein the eighth request comes from the second device or the third device.

[0284] For example, the second computing resource status includes the amount of idle resources in the third device.

[0285] For example, the second condition includes the maximum number of batches that the third device can be grouped into.

[0286] For example, the second batch includes multiple requests.

[0287] For example, the second batch scheduling module groups at least one request arriving at the same time into a batch. During the Decode phase, the input length of each request is 1. As long as there is a free slot in the current batch, this 1-length token can be placed into the batch for inference.

[0288] Step 328: The second computing resource module performs a second inference calculation on the second batch based on the first cache information to obtain the third request and the second cache information.

[0289] In this step, such as Figure 3 As shown, the second execution engine of the second computing resource module performs a second inference calculation on the second batch based on the first cache information to obtain the third request and the second cache information.

[0290] In some possible embodiments, step 328 specifically includes: the second execution engine of the second computing resource module performs a second inference calculation on the second batch according to the first cache information using a second processing method to obtain the second cache information of the third request and the first service.

[0291] For example, the third request includes the identifier of the first service and the second output information corresponding to the first service, and the third request is associated with the second request.

[0292] For example, the second cache information is the cache information required by the third device when performing the second inference calculation on the third request.

[0293] For example, the second inference computation is the inference computation of the Decode stage, i.e., the Decode computation.

[0294] For example, the second processing method involves releasing the computing service resources of the target request if there is a target request whose output information meets the set conditions during the current batch inference process, and inserting the request from the next batch into the idle computing service resources for the second inference calculation. In this embodiment of the invention, for the inference calculation in the Decode stage, inference calculation is performed in multiple batches. That is, if a service ends inference early during the current batch inference process, i.e., there is a target request whose output information meets the set conditions, the computing service resources of the target request can be released early, and before the current batch ends, the request from the next batch is inserted into the idle computing service resources for the second inference calculation. Therefore, compared with the first processing method, which requires waiting for all requests in the current batch to finish inference before releasing the resources of the current batch, the second processing method does not require waiting for all requests in the current batch to finish inference before releasing the resources of the current batch, thus avoiding the waste of idle resources and further improving the system throughput.

[0295] For example, the third device releases the computing service resources occupied by the target request, as well as the storage resources occupied by the target request.

[0296] For example, when the third device has idle computing service resources, it notifies the first device that the third device has idle computing service resources, and the first device schedules old services to the second device according to the notification.

[0297] For example, when the third device does not have any idle computing service resources, it notifies the first device that the third device does not have any idle computing service resources, and the first device suspends scheduling old services to the third device according to the notification.

[0298] Step 330: The second computing resource module sends the second cache information to the second storage module.

[0299] In this step, such as Figure 3 As shown, the second execution engine of the second computing resource module sends the second cache information to the second storage module.

[0300] Step 332: The second computing resource module sends the third request to the service scheduling module and continues to execute step 334 or step 336.

[0301] In this step, such as Figure 3 As shown, the second execution engine of the second computing resource module sends the third request to the service scheduling module.

[0302] For example, the third request is a request for the first service returned by the third device.

[0303] Step 334: When the second output information meets the set conditions, the service scheduling module sends the second output information to the LLM service module. The LLM service module then sends the second output information to the first user, and the process ends.

[0304] In this embodiment of the invention, a first device determines whether the output information of a service meets set conditions. If the conditions are not met, the service continues to be scheduled and inferenced. If the conditions are met, the service is stopped early, and second output information is sent to the first user. After the first device stops the service early, it can notify a third device to release the resources corresponding to the service. The third device reports its current resource status to the first device. If there are idle computing service resources in the current resource status, the first device continues to schedule services to the third device to insert idle computing service resources, thereby achieving dynamic insertion.

[0305] In this step, such as Figure 3 As shown, when the second output information meets the set conditions, the service scheduling module sends the second output information to the LLM service module. The LLM service module then sends the second output information to the first user, and the process ends.

[0306] Step 336: When the second output information does not meet the set conditions, the service scheduling module will schedule the third request to the second inference component package.

[0307] In this step, such as Figure 3 As shown, when the second output information does not meet the set conditions, the service scheduling module sends a third request to the second inference component package, enabling the third device to continue the next round of Decode computation for the third request. In the next round of Decode computation, the second cached information generated in the previous round of Decode computation is used. The second execution engine sends the cached information generated in the next round of Decode computation to the second storage module for storage, so that the second execution engine can use it when performing the next round of Decode computation.

[0308] Taking a request scheduling service system comprising four second devices and one third device as an example, Figure 5 This is a schematic diagram illustrating Prefill inference and Decode inference in an embodiment of the present invention. Figure 5 As shown, the four second devices are device 0, device 1, device 2 and device 3; and the third device is device x. Figure 5The inference process is demonstrated by using the first processing method for Prefill inference and the second processing method for Decode inference. Specifically, in the Prefill inference on the left, one row represents processing a service request in a set of computing resources. There are a total of 4 sets of Prefill computing resources and 4 service requests. In the Decode inference on the right, 4 rows represent processing a batch of 4 service requests in a set of computing resources. There is a total of 1 set of computing resources and 4 service requests. Figure 5 In the diagram, each square represents a computational delay. The Prefill computation on the left and the Decode computation on the right are performed in parallel and have no timing relationship.

[0309] Figure 5 The processes for Prefill inference and Decode inference are shown below:

[0310] 1) The lengths of the input token sequences for the four services, req1, req2, req3 and req4, are 3, 2, 4 and 3 respectively. Therefore, when the four services arrive at device 0, device 1, device 2 and device 3 at the same time, req2 will be completed first, followed by req1, req4 and req3.

[0311] 2) During the Prefill calculation, req2 that is completed in advance will be entered into the Decode calculation ahead of time. Figure 5 The right side illustrates the system startup process; device x needs to form a batch, and a batch includes 4 requests, so services req1, req2, req3, and req4 all need to enter the Decode calculation. For example... Figure 5 As shown, in the Decode inference on the right, req2 arrives first, followed by req1 and req4, and finally req3. Therefore, req1 and req4 will generate 1 unit of free bubble, while req3 will generate 2 units of free bubble. These bubbles will not reappear after the system stabilizes.

[0312] 3) During the Decode computation, the number of output tokens for req1 to req4 are 5, 7, 6, and 4 respectively. Therefore, although req4 is the second to enter the Decode computation during the Prefill computation, it is the first to complete the Decode computation. Thus, the output information of req4 is the first in its batch to meet the set conditions. Req4, having completed the Decode computation, will stop its operation early (early stop mechanism) and release the corresponding resources. The requests that complete the computation next are req1, req2, and req3, respectively.

[0313] 4) During the Decode calculation of req1-4, req5-8 have already reached the Prefill inference execution container for calculation and have completed their calculations before the calculations of req1-4 finish. Therefore, as req4, req1, req2, and req3 successively stop their calculations, the service scheduling module will automatically schedule req6, req5, req7, and req8 to device x. Device x will then insert req6, req5, req7, and req8 into the resource positions previously occupied by req4, req1, req2, and req3, respectively, and perform subsequent calculations after dynamic insertion.

[0314] 5) Once the system is stable, steps 3) and 4) will be repeated to dynamically insert batches and dynamically stop the system and provide service responses.

[0315] It should be noted that when the service scheduling module schedules a third request for the first service to a third device, it will only schedule the third request to the third device that stores the cached information generated during the previous round of Decode inference for the first service. Therefore, during the Decode phase, the Decode part is the service being iterated on in the current third device, and requests for this service will not be forwarded to other third devices.

[0316] In summary, this invention sends the requests output after inference calculation by the execution engine to the service scheduling module. The service scheduling module determines that the request meets the set conditions, thereby realizing automatic early termination of the service. The batch scheduling module groups the requests into batches to achieve batching. When there are idle computing service resources in the execution engine, the batch scheduling module automatically sends the requests in the next batch to the execution engine to achieve dynamic insertion.

[0317] In this embodiment of the invention, requests for new services from users are scheduled to a second device, namely a Prefill inference execution container, for inference computation during the Prefill phase; requests for old services that have passed the Prefill phase are scheduled to a third device, namely a Decode inference execution container, for inference computation during the Decode phase. This invention, through a separate deployment approach, comprehensively considers the significant differences between the computational bottlenecks of the Prefill phase and the memory access bottlenecks of the Decode phase, thereby improving the throughput of the service system.

[0318] Optionally, such as Figure 6 As shown, the method also includes steps 338-372.

[0319] Step 338: The LLM service module receives the fourth request from the second user. Prior to step 338, i.e., before requesting the scheduling system to respond, the LLM has been deployed to the computing resource modules in the second and third devices via the inference component package and AI engine. Both the computing resource modules and the storage modules are ready.

[0320] For example, the second user is different from the first user. The first device is able to schedule requests from different users.

[0321] For example, the fourth request is a request for the second service.

[0322] For example, the fourth request includes the identifier of the second service.

[0323] For example, the identifier includes a service ID.

[0324] For example, the fourth request does not include the requests returned by the second and third devices; that is, the fourth request is a request for a new service.

[0325] In this step, such as Figure 3 As shown, the LLM service module receives a fourth request from the second user.

[0326] Step 340: The LLM service module sends the fourth request to the service scheduling module.

[0327] In this step, such as Figure 3 As shown, the LLM service module sends the fourth request to the service scheduling module.

[0328] Step 342: The service scheduling module sends the fourth request to the first inference component package.

[0329] In this step, such as Figure 3 As shown, the service scheduling module schedules newly arriving service requests to the Prefill inference execution container, i.e., the second device, and at the same time establishes a service pool to store service requests that are waiting due to resource constraints.

[0330] Step 344: The first inference component package sends the fourth request to the first AI engine.

[0331] In this step, such as Figure 3 As shown, the first inference component package sends the fourth request to the first AI engine.

[0332] Step 346: The first AI engine sends the fourth request to the first computing resource module.

[0333] In this step, such as Figure 3 As shown, the first AI engine sends the fourth request to the first computing resource module.

[0334] Step 348: The first computing resource module batches the fourth request to obtain the third batch, performs the first inference calculation on the third batch, and obtains the fifth request and the third cache information.

[0335] In this step, such as Figure 3 As shown, the first batch scheduling module of the first computing resource module groups the fourth request into a third batch according to the first computing resource status and the first condition; the first execution engine of the first computing resource module performs a first inference calculation on the third batch to obtain the fifth request and the third cache information.

[0336] For example, the fourth request may also include input information;

[0337] For example, the input information includes an input token sequence.

[0338] For example, the first computing resource status includes the number of idle resources in the second device.

[0339] For example, the first condition includes the maximum number of batches that the second device can be grouped into.

[0340] For example, the fifth request includes the identifier of the second service and the third output information corresponding to the second service, and the fifth request is associated with the fourth request.

[0341] For example, the third output information includes an output token sequence. The third output information is a third output token sequence.

[0342] For example, the cached information includes key-value cached KV information. The third cached information is third key-value cached KV information.

[0343] In some possible embodiments, step 348 specifically includes: the first batch scheduling module of the first computing resource module obtains the input information corresponding to the fourth request and its second length according to the fourth request; the fourth request is batched according to the first computing resource status, the first condition and the second length to obtain a third batch; the first execution engine of the first computing resource module performs a first inference calculation on the third batch using a first processing method to obtain the third cache information of the fifth request and the second service.

[0344] For example, the third batch includes multiple requests.

[0345] For example, the first batch scheduling module groups multiple requests that arrive at the same time into a batch.

[0346] For example, the first inference computation is the inference computation of the Prefill stage, i.e., the Prefill computation.

[0347] For example, the first processing method is to release the computing service resources occupied by the current batch after all requests in the current batch have completed the inference calculation, and insert the next batch into the idle computing service resources for the first inference calculation.

[0348] Step 350: The first computing resource module sends the third cache information to the second computing resource module.

[0349] In this step, such as Figure 3 As shown, the first execution engine of the first computing resource module sends the third cache information to the second computing resource module.

[0350] Optionally, such as Figure 3 As shown, the first computing resource module can send the third cache information to the first storage module, the first storage module sends the third cache information to the second storage module, and then the second storage module sends the third cache information to the second execution engine module. The second storage module can either send the third cache information to the second execution engine module automatically, or it can send the third cache information to the second execution engine module based on a message sent by the second execution engine module.

[0351] Optionally, such as Figure 3 As shown, the first computing resource module can directly send the third cache information to the second computing resource module.

[0352] Step 352: The first computing resource module sends the fifth request to the service scheduling module and continues to execute steps 354 or 356.

[0353] In this step, such as Figure 3 and Figure 4 As shown, the first computing resource module sends the fifth request to the service scheduling module.

[0354] Step 354: When the third output information meets the set conditions, the service scheduling module sends the third output information to the LLM service module. The LLM service module then sends the third output information to the second user, and the process ends.

[0355] Step 356: When the third output information does not meet the set conditions, the service scheduling module will schedule the fifth request to the second inference component package.

[0356] In this step, such as Figure 3 As shown, the service scheduling module sends the fifth request to the second inference component package.

[0357] For example, the fifth request is a request for the second service returned by the second device.

[0358] For example, the conditions set include at least one of the following:

[0359] The length of the output information is greater than or equal to the first threshold, or

[0360] The output information includes a terminator.

[0361] Step 358: The second inference component package sends the fifth request to the second AI engine.

[0362] In this step, such as Figure 3 As shown, the second inference component package sends the fifth request to the second AI engine.

[0363] Step 360: The second AI engine sends the fifth request to the second computing resource module.

[0364] In this step, such as Figure 3 As shown, the second AI engine sends the fifth request to the second computing resource module.

[0365] Step 362: The second computing resource module batches the fifth request to obtain the fourth batch.

[0366] In this step, such as Figure 3 As shown, the second batch scheduling module of the second computing resource module groups the fifth request into a fourth batch based on the second computing resource status and the second condition.

[0367] In some possible embodiments, step 362 specifically includes: the second batch scheduling module batching the fifth request to obtain the fourth batch based on the second computing resource status and the second condition.

[0368] For example, the second computing resource status includes the amount of idle resources in the third device.

[0369] For example, the second condition includes the maximum number of batches that the third device can be grouped into.

[0370] For example, the fourth batch includes multiple requests.

[0371] For example, the second batch scheduling module groups at least one request arriving at the same time into a batch. During the Decode phase, the input length of each request is 1. As long as there is a free slot in the current batch, this 1-length token can be placed into the batch for inference.

[0372] Step 364: The second computing resource module performs a second inference calculation on the fourth batch based on the third cache information to obtain the sixth request and the fourth cache information.

[0373] In this step, such as Figure 3 As shown, the second execution engine of the second computing resource module performs a second inference calculation on the fourth batch based on the third cache information to obtain the sixth request and the fourth cache information.

[0374] In some possible embodiments, step 364 specifically includes: the second execution engine of the second computing resource module performs a second inference calculation on the fourth batch according to the third cache information using a second processing method to obtain the fourth cache information of the sixth request and the second service.

[0375] For example, the sixth request includes the identifier of the second service and the fourth output information corresponding to the second service, and the sixth request is associated with the fifth request.

[0376] For example, the fourth cache information is the cache information needed by the third device when performing inference calculations on the sixth request.

[0377] For example, the second inference computation is the inference computation of the Decode stage, i.e., the Decode computation.

[0378] For example, the second processing method is to release the computing service resources of the target request and insert the request in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the inference process of the current batch.

[0379] Step 366: The second computing resource module sends the fourth cache information to the second storage module.

[0380] In this step, such as Figure 3 As shown, the second execution engine of the second computing resource module sends the fourth cache information to the second storage module.

[0381] Step 368: The second computing resource module sends the sixth request to the service scheduling module and continues to execute step 370 or step 372.

[0382] In this step, such as Figure 3 As shown, the second execution engine of the second computing resource module sends the sixth request to the service scheduling module.

[0383] For example, the sixth request is a request for the second service returned by the third device.

[0384] Step 370: When the fourth output information meets the set conditions, the service scheduling module sends the fourth output information to the LLM service module. The LLM service module then sends the fourth output information to the second user, and the process ends.

[0385] In this step, such as Figure 3 As shown, when the fourth output information meets the set conditions, the service scheduling module sends the fourth output information to the LLM service module. The LLM service module then sends the fourth output information to the second user, and the process ends.

[0386] Step 372: When the fourth output information does not meet the set conditions, the service scheduling module will schedule the sixth request to the second inference component package.

[0387] based on Figure 3 The system architecture shown in this invention provides a request scheduling method. Figure 7 This is a flowchart illustrating a request scheduling method provided in an embodiment of the present invention. Figure 7 As shown, the method includes:

[0388] Step 402: The first device receives a first request from the first user. The first request is a request for the first service and includes the identifier of the first service.

[0389] For example, the identifier includes a service ID.

[0390] For example, the first request does not include the requests returned by the second and third devices; that is, the first request is a request for a new service.

[0391] Step 404: The first device schedules the first request to the second device.

[0392] The first device schedules newly arriving service requests to the Prefill inference execution container, i.e., the second device, and at the same time establishes a service pool to store service requests that are waiting due to resource constraints.

[0393] Step 406: The second device sends a second request to the first device. The second request includes the identifier of the first service and the first output information corresponding to the first service. The second request is associated with the first request.

[0394] For example, the first output information includes an output token sequence. The first output information is a first output token sequence.

[0395] Step 408: When the first output information does not meet the set conditions, the first device will dispatch the second request to the third device, which is different from the second device.

[0396] For example, the second request is a request for the first service returned by the second device.

[0397] For example, the conditions set include at least one of the following:

[0398] The length of the output information is greater than or equal to the first threshold, or

[0399] The output information includes a terminator.

[0400] Step 410: The third device sends a third request to the first device. The third request includes the identifier of the first service and the second output information corresponding to the first service. The third request is associated with the second request.

[0401] Step 412: When the second output information does not meet the set conditions, the first device will schedule the third request to the third device.

[0402] For example, the third request is a request for the first service returned by the third device.

[0403] The first device sends the third request to the third device, enabling the third device to continue the next round of decoding computation on the third request. In the next round of decoding computation, the second cached information generated in the previous round of decoding computation is used. The third device stores the cached information generated in the next round of decoding computation for use in the next round of decoding computation after that.

[0404] Optionally, such as Figure 7 As shown, steps 4052-4054 ​​are also included between steps 404 and 406.

[0405] Step 4052: The second device processes the first request into batches based on the first computing resource status and the first conditions to obtain the first batch.

[0406] For example, the first request may also include input information.

[0407] For example, the input information includes an input token sequence.

[0408] For example, the first computing resource status includes the number of idle resources in the second device.

[0409] For example, the first condition includes the maximum number of batches that the second device can be grouped into.

[0410] In some possible embodiments, step 4052 specifically includes: the second device obtaining input information corresponding to the first request and its first length according to the first request; and batching the first request and at least one seventh request that arrives at the second device at the same time as the first request to obtain the first batch according to the first computing resource status, the first condition and the first length.

[0411] The seventh request is a request from the first user, the second user, or another user.

[0412] The length of the input information corresponding to the seventh request is similar to that of the first request, which helps to reduce the waste of idle resources during the first inference calculation process.

[0413] Step 4054: The second device performs the first inference calculation on the first batch to obtain the second request and the first cache information.

[0414] For example, the cached information includes key-value cached KV information. The first cached information is the first key-value cached KV information.

[0415] In some possible embodiments, step 4052 specifically includes: the second device performing a first inference calculation on the first batch using a first processing method.

[0416] For example, the first inference computation is the inference computation of the Prefill stage, i.e., the Prefill computation.

[0417] For example, the first processing method is to release the computing service resources occupied by the current batch after all requests in the current batch have completed the inference calculation, and insert the next batch into the idle computing service resources for the first inference calculation.

[0418] In this embodiment of the invention, after step 4052, the method further includes: the second device sending the first cache information to the third device.

[0419] Optionally, after step 406, the method further includes: when the first output information meets the set conditions, the first device sends the first output information to the first user.

[0420] Optionally, such as Figure 7 As shown, steps 4092-4094 are also included between steps 408 and 410.

[0421] Step 4092: The third device batches the second request according to the second computing resource status and the second condition to obtain the second batch.

[0422] For example, the second computing resource status includes the amount of idle resources in the third device.

[0423] For example, the second condition includes the maximum number of batches that the third device can be grouped into.

[0424] In some possible embodiments, step 4092 specifically includes: batching the second request and at least one eighth request that arrives at the third device simultaneously with the second request to obtain a second batch based on a second computing resource status and a second condition, wherein the eighth request comes from the second device or the third device.

[0425] Step 4094: The third device performs a second inference calculation on the second batch based on the first cache information to obtain the third request and the second cache information.

[0426] For example, the second inference computation is the inference computation in the Decode phase, i.e., the Decode computation. In the Decode phase, the input length of each request is 1. As long as there is a free slot in the current batch, this token with a length of 1 can be put into the batch for inference.

[0427] In some possible embodiments, step 4094 specifically includes: the third device performs a second inference calculation on the second batch based on the first cache information using a second processing method to obtain the third request and the second cache information.

[0428] For example, the second processing method is to release the computing service resources of the target request and insert the request in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the inference process of the current batch.

[0429] In this embodiment of the invention, after step 4094, the method further includes: a third device storing second cache information.

[0430] For example, the second cache information is the cache information required by the third device when performing the second inference calculation on the third request.

[0431] Optionally, such as Figure 7 As shown, step 412 is followed by steps 414-424.

[0432] Step 414: The first device receives a fourth request from the second user. The fourth request is a request for the second service and includes the identifier of the second service.

[0433] Step 416: The first device schedules the fourth request to the second device.

[0434] Step 418: The second device sends a fifth request to the first device. The fifth request includes the identifier of the second service and the third output information corresponding to the second service. The fifth request is associated with the fourth request.

[0435] Step 420: When the third output information does not meet the set conditions, the first device will dispatch the fifth request to the third device.

[0436] Step 422: The third device sends a sixth request to the first device. The sixth request includes the identifier of the second service and the fourth output information corresponding to the second service. The sixth request is associated with the fifth request.

[0437] Step 424: When the fourth output information does not meet the set conditions, the first device will schedule the sixth request to the third device.

[0438] Optionally, steps 4172-4174 may be included between steps 416 and 418.

[0439] Step 4172: The second device batches the fourth request according to the first computing resource status and the first condition to obtain the third batch.

[0440] For example, the fourth request may also include input information.

[0441] In some possible embodiments, step 4172 specifically includes: the second device obtaining input information corresponding to the fourth request and its second length according to the fourth request; and batching the fourth request to obtain a third batch according to the first computing resource status, the first condition and the second length.

[0442] Step 4174: The second device performs the first inference calculation on the third batch to obtain the fifth request and the third cache information.

[0443] In some possible embodiments, step 4172 specifically includes: the second device performing a first inference calculation on the third batch using a first processing method.

[0444] For example, the first processing method is to release the computing service resources occupied by the current batch after all requests in the current batch have completed the inference calculation, and insert the next batch into the idle computing service resources for the first inference calculation.

[0445] In this embodiment of the invention, after step 4172, the method further includes: the second device sending the third cache information to the third device.

[0446] Optionally, after step 418, the method further includes: when the third output information meets the set conditions, the first device sends the third output information to the first user.

[0447] Optionally, steps 4212-4214 may be included between steps 420 and 422.

[0448] Step 4212: The third device batches the fifth request according to the second computing resource status and the second condition to obtain the fourth batch.

[0449] Step 4214: The third device performs a second inference calculation on the fourth batch based on the third cache information to obtain the sixth request and the fourth cache information.

[0450] In some possible embodiments, step 4214 specifically includes: the third device performs a second inference calculation on the fourth batch using a second processing method based on the third cache information to obtain the sixth request and the fourth cache information.

[0451] For example, the second processing method is to release the computing service resources of the target request and insert the request in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the inference process of the current batch.

[0452] In this embodiment of the invention, after step 4094, the method further includes: a third device storing fourth cache information.

[0453] This invention discloses a request scheduling method and apparatus. The method includes: a first device receiving a first request from a first user, the first request being a request for a first service, the first request including an identifier of the first service; the first device scheduling the first request to a second device; the second device sending a second request to the first device, the second request including the identifier of the first service and first output information, the second request being associated with the first request; when the first output information does not meet the set conditions, the first device scheduling the second request to a third device, the third device being different from the second device; the third device sending a third request to the first device, the third request including the identifier of the first service and second output information, the third request being associated with the second request; when the second output information does not meet the set conditions, the first device scheduling the third request to the third device, which can improve the throughput of the service system in a large-scale language model service deployment.

[0454] based on Figure 3 The system architecture shown in this invention provides a request scheduling method applied to a first device. Figure 8 A flowchart illustrating another request scheduling method provided in an embodiment of the present invention. Figure 8 As shown, the method includes:

[0455] Step 502: The first device receives a first request from the first user. The first request is a request for the first service and includes the identifier of the first service.

[0456] Step 504: Schedule the first request to the second device.

[0457] Step 506: Receive a second request sent by the second device. The second request includes the identifier of the first service and the first output information corresponding to the first service. The second request is associated with the first request.

[0458] Step 508: When the first output information does not meet the set conditions, the second request is scheduled to the third device, which is different from the second device.

[0459] Step 510: Receive a third request sent by a third device. The third request includes the identifier of the first service and the second output information corresponding to the first service. The third request is associated with the second request.

[0460] Step 512: When the second output information does not meet the set conditions, the third request is scheduled to the third device.

[0461] In the technical solution of the request scheduling method provided in the embodiments of the present invention, the method includes: a first device receiving at least one first request, wherein the first request is a service request, and scheduling the at least one first request to a second device, wherein the first request does not include requests returned by the second device and the third device; receiving at least one second request returned by the second device, and scheduling the at least one second request to the third device; receiving at least one third request returned by the third device, and determining whether the third request meets a third condition based on the third request; if it is determined that the third request does not meet the third condition, scheduling the third request to the third device, which can improve the throughput of the service system in the deployment of large-scale language model services.

[0462] Figure 9 This is a schematic diagram of the structure of a first device provided in an embodiment of the present invention. It should be understood that the first device 600 is capable of executing each step of the first device in the above-described request scheduling method. To avoid repetition, it will not be described in detail here. The first device 600 includes: a first transceiver unit 601.

[0463] A first transceiver unit 601 is configured to receive a first request from a first user, the first request being a request for a first service, the first request including an identifier of the first service; schedule the first request to a second device; receive a second request sent by the second device, the second request including an identifier of the first service and first output information corresponding to the first service, the second request being associated with the first request; when the first output information does not meet a set condition, schedule the second request to a third device, the third device being different from the second device; receive a third request sent by the third device, the third request including an identifier of the first service and second output information corresponding to the first service, the third request being associated with the second request; when the second output information does not meet the set condition, schedule the third request to the third device.

[0464] Optionally, the first transceiver unit 601 is further configured to receive a fourth request from a second user, the fourth request being a request for a second service, the fourth request including an identifier of the second service; schedule the fourth request to the second device; receive a fifth request sent by the second device, the fifth request including an identifier of the second service and third output information corresponding to the second service, the fifth request being associated with the fourth request; when the third output information does not meet the set conditions, schedule the fifth request to the third device; receive a sixth request sent by the third device, the sixth request including an identifier of the second service and fourth output information corresponding to the second service, the sixth request being associated with the fifth request; when the fourth output information does not meet the set conditions, schedule the sixth request to the third device.

[0465] Optionally, the first transceiver unit 601 is further configured to send the second output information to the first user when the second output information meets the set conditions.

[0466] Optionally, the identifier includes a service ID.

[0467] Optionally, the set conditions include at least one of the following:

[0468] The length of the output information is greater than or equal to the first threshold, or

[0469] The output information includes a terminator.

[0470] Figure 10 This is a schematic diagram of the structure of a second device provided in an embodiment of the present invention. It should be understood that the second device 700 is capable of executing each step of the above-described request scheduling method. To avoid repetition, it will not be described in detail here. The second device 700 includes: a second transceiver unit 701 and a second processing unit 702.

[0471] The second transceiver unit 701 is configured to receive a first request from a first user sent by a first device, the first request being a request for a first service, the first request including an identifier of the first service; and to send a second request to the first device, the second request including an identifier of the first service and first output information corresponding to the first service, the second request being associated with the first request.

[0472] Optionally, the second transceiver unit 701 is further configured to receive a fourth request from the second user sent by the first device, the fourth request being a request for the second service, the fourth request including an identifier of the second service; and to send a fifth request to the first device, the fifth request including an identifier of the second service and third output information corresponding to the second service, the fifth request being associated with the fourth request.

[0473] Optionally, before the second transceiver unit 701 sends the second request to the first device, the second processing unit 702 is configured to batch the first request and at least one seventh request that arrives at the second device at the same time as the first request according to the first computing resource status and the first condition to obtain a first batch, wherein the seventh request is a request from the first user, the second user, or other users; and perform a first inference calculation on the first batch to obtain the second request and the first cache information.

[0474] Optionally, the second transceiver unit 701 is further configured to send the first cached information to the third device.

[0475] Optionally, the cache information includes key-value cache (KV) information.

[0476] Optionally, the second processing unit 702 is specifically configured to obtain input information corresponding to the first request and its first length according to the first request; and to batch the first request and at least one seventh request that arrives at the second device at the same time as the first request according to the first computing resource status, the first condition and the first length to obtain the first batch.

[0477] Figure 11 This is a schematic diagram of a third device provided in an embodiment of the present invention. It should be understood that the third device 800 is capable of executing each step of the third device in the above-described request scheduling method. To avoid repetition, it will not be described in detail here. The third device 800 includes: a third transceiver unit 801 and a third processing unit 802.

[0478] The third transceiver unit 801 is configured to receive a second request from the second device sent by the first device, the second request including an identifier of the first service and first output information corresponding to the first service, the second request being associated with the first request, the first request being a request from the first user for the first service; and to send a third request to the first device, the third request including an identifier of the first service and second output information corresponding to the first service, the third request being associated with the second request.

[0479] Optionally, the third transceiver unit 801 is further configured to receive a fifth request from the second device sent by the first device, the fifth request including an identifier of the second service and third output information corresponding to the second service, the fifth request being associated with a fourth request, the fourth request being a request from the second user for the second service; and to send a sixth request to the first device, the sixth request including an identifier of the second service and fourth output information corresponding to the second service, the sixth request being associated with the fifth request.

[0480] Optionally, before the third transceiver unit 801 sends the third request to the first device, the third processing unit 802 is configured to batch the second request and at least one eighth request that arrives at the third device at the same time as the second request according to the second computing resource status and the second condition to obtain a second batch, wherein the eighth request comes from the second device or the third device.

[0481] The third transceiver unit 801 is also used to receive the first buffer information sent by the second device;

[0482] The third processing unit 802 is further configured to perform a second inference calculation on the second batch based on the first cache information to obtain the third request and the second cache information.

[0483] It should be understood that the first device 600, the second device 700, and the third device 800 are embodied in the form of functional units. The term "unit" here can be implemented in software and / or hardware, without specific limitation. For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include application-specific integrated circuits (ASICs), electronic circuits, a processor (e.g., a shared processor, a proprietary processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, combined logic circuitry, and / or other suitable components supporting the described functions.

[0484] Therefore, the units of the various examples described in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the present invention.

[0485] This application provides a device, which can be a terminal device or a circuit device built into the terminal device. This device can be used to perform the functions / steps described in the method embodiments above.

[0486] This application provides a readable storage medium storing instructions that, when executed on a device or processor, cause the device to perform the functions / steps described in the above method embodiments.

[0487] This application also provides a program product containing instructions that, when run on a device or any at least one processor, cause the device to perform the functions / steps described in the method embodiments above.

[0488] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0489] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0490] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0491] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0492] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A request scheduling method, characterized in that, Applied to a request scheduling system, the method includes: The first device receives a first request from a first user, the first request being a request for a first service, and the first request including the identifier of the first service; The first device will schedule the first request to the second device; The second device generates cache information during the inference process, and then sends the cache information to the third device. The second device sends a second request to the first device. The second request includes the identifier of the first service and the first output information corresponding to the first service. The second request is associated with the first request. When the first output information does not meet the set conditions, the first device will schedule the second request to the third device, which is different from the second device; The third device obtains a third request based on the cached information; The third device sends the third request to the first device. The third request includes the identifier of the first service and the second output information corresponding to the first service. The third request is associated with the second request. When the second output information does not meet the set conditions, the first device will schedule the third request to the third device.

2. The method according to claim 1, characterized in that, The method further includes: The first device receives a fourth request from the second user, the fourth request being a request for the second service, the fourth request including the identifier of the second service; The first device will schedule the fourth request to the second device; The second device sends a fifth request to the first device. The fifth request includes the identifier of the second service and the third output information corresponding to the second service. The fifth request is associated with the fourth request. When the third output information does not meet the set conditions, the first device will schedule the fifth request to the third device; The third device sends a sixth request to the first device. The sixth request includes the identifier of the second service and the fourth output information corresponding to the second service. The sixth request is associated with the fifth request. When the fourth output information does not meet the set conditions, the first device will schedule the sixth request to the third device.

3. The method according to claim 2, characterized in that, The method further includes: When the second output information meets the set conditions, the first device sends the second output information to the first user.

4. The method according to claim 3, characterized in that, The request scheduling system includes the first device, at least one second device, and at least one third device.

5. The method according to any one of claims 1-3, characterized in that, The second device includes a first computing resource module, and the third device includes a second computing resource module. The sending of cache information from the second device to the third device includes the first computing resource module sending cache information to the second computing resource module.

6. The method according to claim 5, characterized in that, The second device further includes a first storage module, and the third device further includes a second storage module. The sending of cache information from the first computing resource module to the second computing resource module includes the first computing resource module sending the cache information to the first storage module, the first storage module sending the cache information to the second storage module, and then the second storage module sending the cache information to the second computing resource module.

7. The method according to claim 6, characterized in that, The second storage module automatically sends the cached information to the second computing resource module.

8. The method according to claim 6, characterized in that, The second storage module sends the cached information to the second computing resource module according to the message sent by the second computing resource module.

9. A request scheduling method, characterized in that, Applied to a request scheduling system, the method includes: The first device receives a first request from a first user, the first request being a request for a first service, and the first request including the identifier of the first service; The first device will schedule the first request to the second device; The second device generates the first cache information during the inference process; The second device sends a second request to the first device. The second request includes the identifier of the first service and the first output information corresponding to the first service. The second request is associated with the first request. When the first output information does not meet the set conditions, the first device will dispatch the second request to a third device, which is different from the second device; The third device acquires the first cached information during the inference process; The third device sends a third request to the first device. The third request includes the identifier of the first service and the second output information corresponding to the first service. The third request is associated with the second request. When the second output information does not meet the set conditions, the first device will schedule the third request to the third device.

10. The method according to claim 9, characterized in that, The method further includes: The first device receives a fourth request from the second user, the fourth request being a request for the second service, the fourth request including the identifier of the second service; The first device will schedule the fourth request to the second device; The second device sends a fifth request to the first device. The fifth request includes the identifier of the second service and the third output information corresponding to the second service. The fifth request is associated with the fourth request. When the third output information does not meet the set conditions, the first device will schedule the fifth request to the third device; The third device sends a sixth request to the first device. The sixth request includes the identifier of the second service and the fourth output information corresponding to the second service. The sixth request is associated with the fifth request. When the fourth output information does not meet the set conditions, the first device will schedule the sixth request to the third device.

11. The method according to claim 10, characterized in that, The method further includes: When the second output information meets the set conditions, the first device sends the second output information to the first user.

12. The method according to claim 11, characterized in that, The request scheduling system includes the first device, at least one second device, and at least one third device.

13. A device, characterized in that, The device includes a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions that, when the processor executes the program instructions, cause the device to perform the steps of the method as described in any one of claims 1-12.

14. A readable storage medium, characterized in that, The readable storage medium stores a program, the program including program instructions that, when executed by the device, cause the device to perform the method as described in any one of claims 1-12.

Images