Request scheduling method and device

Abstract

Disclosed are a request scheduling method and device, the method comprising: a first device receiving a first request of a first user, the first request being a request of a first service, the first request comprising an identifier of the first service; the first device schedules the first request to the second device; the second equipment sends a second request to the first equipment, the second request comprises the identifier of the first service and the first output information, and the second request is associated with the first request; when the first output information does not meet the set condition, the first device dispatches the second request to a third device, and the third device is different from the second device; the third equipment sends a third request to the first equipment, the third request comprises the identifier of the first service and the second output information, and the third request is associated with the second request; when the second output information does not meet the set condition, the first device dispatches the third request to the third device, and the throughput of a service system in large-scale language model service deployment can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a request scheduling method and device. Background Art

[0002] Large Language Model (LLM) has set off a new wave in the field of artificial intelligence (AI). The reasoning process of the Generative Pre-trained Transformer (GPT) using the key-value cache (KV Cache) technology consists of two stages, namely the first round of context filling (Prefill) stage and multiple rounds of decoding (Decode) stage. In the large-scale language model service deployment, although the service system adopts a dynamic scheduling method, the throughput of the service system in the LLM service deployment is still low. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a request scheduling method and device, which can improve the throughput of a service system in LLM service deployment.

[0004] In a first aspect, an embodiment of the present invention provides a request scheduling method, which is applied to a request scheduling system, and the method includes:

[0005] The first device receives a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service;

[0006] The first device dispatches the first request to the second device;

[0007] The second device sends a second request to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request;

[0008] When the first output information does not meet a set condition, the first device dispatches the second request to a third device, where the third device is different from the second device;

[0009] The third device sends a third request to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request;

[0010] When the second output information does not satisfy the set condition, the first device dispatches the third request to the third device.

[0011] The embodiment of the present invention also adopts an independent separate deployment method in the request scheduling system to deploy the second device and the third device. Through the scheduling of the request by the first device, the second device and the third device respectively execute the reasoning process of their respective stages. At the same time, information can be transmitted between the two devices, so that the reasoning process can proceed normally, thereby improving the throughput of the service system.

[0012] In combination with the first aspect, in some implementations of the first aspect, the method further includes:

[0013] The first device receives a fourth request from a second user, where the fourth request is a request for a second service and includes an identifier of the second service;

[0014] The first device dispatches the fourth request to the second device;

[0015] The second device sends a fifth request to the first device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request;

[0016] When the third output information does not satisfy the set condition, the first device dispatches the fifth request to the third device;

[0017] The third device sends a sixth request to the first device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request;

[0018] When the fourth output information does not satisfy the set condition, the first device dispatches the sixth request to the third device.

[0019] In the embodiment of the present invention, the second user is different from the first user, and the first device can schedule requests from different users to the second device and the third device for processing. There is no need to wait for the second device and / or the third device to complete processing the request of one user before processing the request of another user, thereby improving the throughput of the service system.

[0020] In combination with the first aspect, in some implementations of the first aspect, the method further includes:

[0021] When the second output information meets the set condition, the first device sends the second output information to the first user.

[0022] In the embodiment of the present invention, the first device determines whether the output information of the service meets the set conditions. If the set conditions are not met, the service continues to be scheduled and inferred; if the set conditions are met, the service is stopped early and the second output information is sent to the first user. After the first device stops the service early, it can notify the third device to release the resources corresponding to the service; the third device reports the current resource status of the third device to the first device. If there are idle computing service resources in the current resource status, the first device continues to schedule the service to the third device to insert the idle computing service resources, thereby realizing dynamic insertion.

[0023] In combination with the first aspect, in some implementations of the first aspect, the request scheduling system includes the first device, at least one of the second devices, and at least one of the third devices. In an embodiment of the present invention, when the number of second devices and third devices in the request scheduling system is large, an independent separation deployment method is adopted to deploy the second devices and the third devices, the second devices are configured in small batches or single batches, and the third devices are configured in large batches. The second device resources and the third device resources can dynamically adjust the resource ratio according to the system load to obtain the maximum throughput, thereby greatly improving the throughput.

[0024] In a second aspect, an embodiment of the present invention provides a request scheduling method, which is applied to a first device, and the method includes:

[0025] receiving a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service;

[0026] dispatching the first request to a second device;

[0027] receiving a second request sent by the second device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request;

[0028] When the first output information does not meet the set condition, dispatching the second request to a third device, the third device being different from the second device;

[0029] receiving a third request sent by the third device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request;

[0030] When the second output information does not meet the set condition, scheduling the third request to the third device.

[0031] The embodiment of the present invention dispatches the request for the new service from the user to the second device, namely the Prefill inference execution container, to perform the inference calculation of the Prefill stage; and dispatches the request for the old service that has passed the Prefill stage to the third device, namely the Decode inference execution container, to perform the inference calculation of the Decode stage. The present invention adopts a separate deployment method, comprehensively considering the significant differences between the computing power bottleneck of the Prefill stage and the memory access bottleneck of the Decode stage, and can improve the throughput of the service system.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

[0033] receiving a fourth request from a second user, where the fourth request is a request for a second service and includes an identifier of the second service;

[0034] dispatching the fourth request to the second device;

[0035] receiving a fifth request sent by the second device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request;

[0036] When the third output information does not satisfy the set condition, dispatching the fifth request to the third device;

[0037] receiving a sixth request sent by the third device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request;

[0038] When the fourth output information does not meet the set condition, scheduling the sixth request to the third device.

[0039] In the embodiment of the present invention, the second user is different from the first user, and the first device can schedule requests from different users to the second device and the third device for processing. There is no need to wait for the second device and / or the third device to complete processing the request of one user before processing the request of another user, thereby improving the throughput of the service system.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

[0041] When the second output information meets the set condition, the second output information is sent to the first user.

[0042] In the embodiment of the present invention, the first device determines whether the output information of the service meets the set conditions. If the set conditions are not met, the service continues to be scheduled and inferred; if the set conditions are met, the service is stopped early and the second output information is sent to the first user. After the first device stops the service early, it can notify the third device to release the resources corresponding to the service; the third device reports the current resource status of the third device to the first device. If there are idle computing service resources in the current resource status, the first device continues to schedule the service to the third device to insert the idle computing service resources, thereby realizing dynamic insertion.

[0043] In combination with the second aspect, in some implementations of the second aspect, the identifier includes a service ID.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the set condition includes at least one of the following:

[0045] The length of the output information is greater than or equal to the first threshold, or

[0046] The output information includes a terminator.

[0047] In a third aspect, an embodiment of the present invention provides a request scheduling method, which is applied to a second device, and the method includes:

[0048] receiving a first request from a first user and sent by a first device, where the first request is a request for a first service and includes an identifier of the first service;

[0049] A second request is sent to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

[0050] The embodiment of the present invention dispatches the request for the new service from the user to the second device, namely the Prefill inference execution container, to perform the inference calculation of the Prefill stage; and dispatches the request for the old service that has passed the Prefill stage to the third device, namely the Decode inference execution container, to perform the inference calculation of the Decode stage. The present invention comprehensively considers the significant difference between the computing power bottleneck of the Prefill stage and the memory access bottleneck of the Decode stage through separate deployment, thereby improving the throughput of the service system.

[0051] In conjunction with the third aspect, in some implementations of the third aspect, the method further includes:

[0052] receiving a fourth request from a second user sent by the first device, where the fourth request is a request for a second service and includes an identifier of the second service;

[0053] A fifth request is sent to the first device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request.

[0054] In the embodiment of the present invention, the second user is different from the first user, and the first device can schedule requests from different users to the second device and the third device for processing. There is no need to wait for the second device and / or the third device to complete processing the request of one user before processing the request of another user, thereby improving the throughput of the service system.

[0055] In combination with the third aspect, in some implementations of the third aspect, before sending the second request to the first device, the method further includes:

[0056] According to the first computing resource state and the first condition, the first request and at least one seventh request arriving at the second device simultaneously with the first request are grouped into batches to obtain a first batch, wherein the seventh request is a request from the first user or the second user or another user;

[0057] A first inference calculation is performed on the first batch to obtain the second request and the first cache information.

[0058] In conjunction with the third aspect, in some implementations of the third aspect, the method further includes:

[0059] The first cache information is sent to the third device.

[0060] In combination with the third aspect, in certain implementations of the third aspect, the cache information includes key-value cache KV information.

[0061] In combination with the third aspect, in some implementations of the third aspect, batching the first request and at least one seventh request that arrives at the second device simultaneously with the first request to obtain a first batch according to the first computing resource state and the first condition includes:

[0062] According to the first request, obtaining input information corresponding to the first request and a first length thereof;

[0063] The first request and the at least one seventh request are grouped into batches according to the first computing resource state, the first condition and the first length to obtain a first batch, wherein the length of the input information corresponding to the seventh request is similar to the first length, which is beneficial to reducing the waste of idle resources in the first reasoning calculation process.

[0064] In combination with the third aspect, in some implementations of the third aspect, the first computing resource status includes the number of idle resources in the second device.

[0065] In combination with the third aspect, in certain implementations of the third aspect, the first condition includes the maximum number of batches that the second device can group.

[0066] In combination with the third aspect, in certain implementations of the third aspect, the first processing method is to release the computing service resources occupied by the current batch and insert the next batch into the idle computing service resources for the first inference calculation after all requests of the current batch are completed.

[0067] In an embodiment of the present invention, for the inference calculation in the Prefill stage, the inference calculation is performed in a single batch, that is, after the inference calculation of all requests of the current batch is completed, the computing service resources occupied by the current batch are released and the next batch is inserted into the idle computing service resources, and the first inference calculation of the next batch is started.

[0068] In a fourth aspect, an embodiment of the present invention provides a request scheduling method, which is applied to a third device, and the method includes:

[0069] receiving a second request from a second device sent by a first device, where the second request includes an identifier of a first service and first output information corresponding to the first service, where the second request is associated with the first request, and where the first request is a request for the first service from a first user;

[0070] A third request is sent to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request.

[0071] The embodiment of the present invention dispatches the request for the new service from the user to the second device, namely the Prefill inference execution container, to perform the inference calculation of the Prefill stage; and dispatches the request for the old service that has passed the Prefill stage to the third device, namely the Decode inference execution container, to perform the inference calculation of the Decode stage. The present invention comprehensively considers the significant difference between the computing power bottleneck of the Prefill stage and the memory access bottleneck of the Decode stage through separate deployment, thereby improving the throughput of the service system.

[0072] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the method further includes:

[0073] receiving a fifth request sent by the first device, the fifth request including an identifier of a second service and third output information corresponding to the second service, the fifth request being associated with a fourth request, and the fourth request being a request for the second service from a second user;

[0074] A sixth request is sent to the first device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request.

[0075] In the embodiment of the present invention, the second user is different from the first user, and the first device can schedule requests from different users to the second device and the third device for processing. There is no need to wait for the second device and / or the third device to complete processing the request of one user before processing the request of another user, thereby improving the throughput of the service system.

[0076] In combination with the fourth aspect, in some implementations of the fourth aspect, before sending the third request to the first device, the method further includes:

[0077] According to a second computing resource state and a second condition, the second request and at least one eighth request arriving at the third device simultaneously with the second request are grouped into batches to obtain a second batch, wherein the eighth request comes from the second device or the third device;

[0078] Receiving first cache information sent by the second device;

[0079] A second inference calculation is performed on the second batch according to the first cache information to obtain the third request and the second cache information.

[0080] In a fifth aspect, an embodiment of the present invention provides a request scheduling system, the system comprising:

[0081] A first device is configured to receive a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service;

[0082] The first device is further configured to dispatch the first request to a second device;

[0083] the second device is used to send a second request to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request;

[0084] When the first output information does not meet a set condition, the first device is further configured to dispatch the second request to a third device, where the third device is different from the second device;

[0085] the third device being configured to send a third request to the first device, the third request including an identifier of the first service and second output information corresponding to the first service, the third request being associated with the second request;

[0086] When the second output information does not meet the set condition, the first device is further configured to schedule the third request to the third device.

[0087] In a sixth aspect, an embodiment of the present invention provides a device, comprising a processor and a memory, wherein the memory is used to store a program, the program comprising program instructions, and when the processor runs the program instructions, the device executes the steps of the method described above.

[0088] In a seventh aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium stores a program, wherein the program includes program instructions, and when the program request is run by a device, the device is caused to execute the method as described above.

[0089] In an eighth aspect, an embodiment of the present invention provides a program product, which includes instructions. When the program product is run on a device or any at least one processor, the device executes the functions / steps in the above method.

[0090] In the technical solution of the request scheduling method and device provided by the embodiment of the present invention, a first device receives a first request from a first user, the first request is a request for a first service, and the first request includes an identifier of the first service; the first device schedules the first request to a second device; the second device sends a second request to the first device, the second request includes an identifier of the first service and first output information, and the second request is associated with the first request; when the first output information does not meet the set conditions, the first device schedules the second request to a third device, and the third device is different from the second device; the third device sends a third request to the first device, the third request includes an identifier of the first service and second output information, and the third request is associated with the second request; when the second output information does not meet the set conditions, the first device schedules the third request to the third device, which can improve the throughput of the service system in large-scale language model service deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 A schematic diagram of the structure of a device provided by an embodiment of the present invention;

[0092] Figure 2 is a software structure block diagram of the device 100 according to an embodiment of the present invention;

[0093] Figure 3 An architecture diagram of another request scheduling system provided by an embodiment of the present invention;

[0094] Figure 4 A signaling interaction diagram of a scheduling request method provided by an embodiment of the present invention;

[0095] Figure 5Schematic diagram of Prefill reasoning and Decode reasoning in an embodiment of the present invention;

[0096] Figure 6 A signaling interaction diagram of another scheduling request method provided by an embodiment of the present invention;

[0097] Figure 7 A flowchart of a request scheduling method provided by an embodiment of the present invention;

[0098] Figure 8 Flow chart of another request scheduling method provided by an embodiment of the present invention

[0099] Fig. 9 A schematic diagram of the structure of a first device provided in an embodiment of the present invention;

[0100] Fig.10 A schematic diagram of the structure of a second device provided by an embodiment of the present invention;

[0101] Fig.11 A schematic structural diagram of a third device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0102] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0103] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0104] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0105] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0106] Figure 1 A schematic structural diagram of the device 100 is shown.

[0107] The device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0108] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the device 100. In other embodiments of the present application, the device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0109] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0110] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0111] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0112] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0113] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface to realize the touch function of the device 100.

[0114] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0115] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0116] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0117] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the device 100.

[0118] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0119] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the device 100, and can also be used to transfer data between the device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other devices, such as AR devices, etc.

[0120] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the device 100. In other embodiments of the present application, the device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0121] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through the wireless charging coil of the device 100. While the charging management module 140 is charging the battery 142, it can also power the device through the power management module 141.

[0122] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0123] The wireless communication function of the device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0124] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0125] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0126] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0127] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0128] In some embodiments, the antenna 1 of the device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0129] The device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0130] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0131] The device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0132] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0133] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0134] The digital signal processor is used to process digital signals, and in addition to processing digital image signals, it can also process other digital signals. For example, when the device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0135] Video codecs are used to compress or decompress digital videos. Device 100 may support one or more video codecs. Thus, device 100 may play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0136] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0137] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0138] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0139] The device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0140] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0141] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0142] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0143] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The device 100 can be provided with at least one microphone 170C. In other embodiments, the device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0144] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0145] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the device 100 detects the touch operation intensity according to the pressure sensor 180A. The device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0146] The gyroscope sensor 180B can be used to determine the motion posture of the device 100. In some embodiments, the angular velocity of the device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0147] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0148] The magnetic sensor 180D includes a Hall sensor. The device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the device 100 is a flip phone, the device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0149] The acceleration sensor 180E can detect the magnitude of acceleration of the device 100 in all directions (generally three axes). When the device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the device posture and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0150] The distance sensor 180F is used to measure the distance. The device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0151] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The device 100 emits infrared light outward through the light emitting diode. The device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the device 100. When insufficient reflected light is detected, the device 100 can determine that there is no object near the device 100. The device 100 can use the proximity light sensor 180G to detect that the user holds the device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0152] The ambient light sensor 180L is used to sense the ambient light brightness. The device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the device 100 is in a pocket to prevent accidental touches.

[0153] The fingerprint sensor 180H is used to collect fingerprints. The device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0154] The temperature sensor 180J is used to detect temperature. In some embodiments, the device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the device 100 heats the battery 142 to avoid abnormal shutdown of the device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown due to low temperature.

[0155] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the device 100, which is different from the position of the display screen 194.

[0156] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0157] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The device 100 may receive key input and generate key signal input related to user settings and function control of the device 100.

[0158] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0159] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0160] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the device 100 and cannot be separated from the device 100.

[0161] The software system of the device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the device 100.

[0162] Figure 2 It is a software structure block diagram of the device 100 according to an embodiment of the present invention.

[0163] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0164] The application layer can include a series of application packages.

[0165] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0166] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0167] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0168] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0169] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0170] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0171] The phone manager is used to provide communication functions of the device 100, such as management of call status (including connection, disconnection, etc.).

[0172] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0173] The notification manager allows applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the device vibrates, the indicator light flashes, etc.

[0174] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0175] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0176] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0177] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0178] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0179] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0180] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0181] A 2D graphics engine is a drawing engine for 2D drawings.

[0182] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0183] The following is an illustrative description of the software and hardware workflow of the device 100 in conjunction with the capture and photo shooting scene.

[0184] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0185] First, the relevant technologies of the request scheduling method involved in the embodiment of the present invention are explained to facilitate understanding by those skilled in the art.

[0186] Transformer: A special structure in the field of deep learning, including multi-head self-attention and forward feedback. This mechanism can assign different weights according to the importance of each part of the input data. This model is mainly used in the fields of natural language processing (NLP) and computer vision (CV).

[0187] Token: In Chinese natural language processing, a token corresponds to a character or a word in the text. The input text will first be processed by the word segmenter and vocabulary mapping, and after obtaining the corresponding token sequence, it will be sent to the model for further processing.

[0188] Prefill reasoning: Usually the input information is very long, latency is a decisive factor, and there is a computing bottleneck.

[0189] Decode reasoning: The input information length is fixed at 1, the latency overhead is small, and there is a memory access bottleneck.

[0190] Throughput: The number of requests processed by the system per unit time. The throughput calculation formula referred to in this invention is:

[0191]

[0192] Large-scale language models have set off a new wave in the AI ​​community. The reasoning process of GPT-type models, represented by generative models, is very unique. Given a set of input token sequences of length S, the model will output an output token sequence of length N. In fact, there are N reasoning processes in this process. That is, the GPT-type model only outputs one token at a time, and the output token will be concatenated with the previous input token (S+1, S+2, ..., S+N-1) as the input for the next reasoning, until the terminator or the maximum output length is encountered.

[0193] This processing method will introduce a problem: as the number of iterations increases, the amount of calculation for each reasoning will increase accordingly; and because each iteration contains the input part of the previous iteration, there is a lot of redundant calculation. Therefore, technicians use KV cache technology to keep the amount of calculation for each round of reasoning after the first round of iteration basically constant, saving computing power and reducing reasoning latency.

[0194] The GPT reasoning process using KV cache technology is divided into the first stage, namely the first round of context filling (Prefill) stage and the second stage, namely the multi-round decoding (Decode) stage.

[0195] In the Prefill phase, the input data dimension is [B, S], where B is the dimension of the group batch, which is the batch commonly used in machine learning. The data processing and transmission dimension of the model middle layer is [B, S, H], where H is the hidden layer dimension. The output data dimension is [B, 1], which is a batch of tokens. If there is only one batch, there is only one token, and reasoning is performed only once. During reasoning, the key-value cache KV information of the middle layer, that is, the transformer layer, is cached in the KV cache for use in the Decode phase.

[0196] In the decode phase, the input data dimension is [B, 1]. The model middle layer data processing and transmission dimension is [B, 1, H]. The output data dimension is [B, 1]. The decode phase will be executed N times until the stop condition is met. Similarly, the reasoning process first obtains the key-value cache KV information stored in the previous round from the KV cache, and at the same time caches the newly generated key-value cache KV information into the KV cache for update.

[0197] Table 1 GPT large model calculation and memory access relationship

[0198]

[0199] Through statistical calculation, we can obtain the relationship between the amount of computation and memory access (access to memory) of the reasoning process when B=1, L=88, and S=4K / 16K, as shown in Table 1, where L is the number of Transformers in the network layer. Obviously, computation is the decisive factor in the Prefill stage (high latency, computational bottleneck), while memory access is the decisive factor in the Decode stage (low latency, memory access bottleneck). When there are a large number of user service requests, different service request groups will be batched for reasoning; and the significant difference between the two reasoning stages will lead to two problems: 1. Since the length is usually padded to the length of the long input sentence in a batch, and then reasoning is performed, there are a lot of invalid calculations for short input sentences in the Prefill stage; 2. Short output statements will wait for the longest output statement in a batch to be reasoned and then output uniformly, and there are also a lot of invalid calculations. These two problems will lead to a very low throughput of the entire system.

[0200] Based on the above technical problems, an embodiment of the present invention provides a request scheduling system. The present invention mainly solves the problem of improving the throughput of a service system in a large-scale language model service deployment, wherein the large-scale language model is a model with a transformer structure.

[0201] The main problems at present are: there are two inference states in the large model inference process, namely the Prefill stage and the Decode stage. The service deployment does not take into account the significant differences and characteristics of the two stages in terms of computing power, memory access, etc.; the two stages are generally deployed in the same computing node. When a large number of services request services at the same time, if a single service request is provided to exclusively use the same computing node, it will lead to huge service deployment costs; batch reasoning (i.e., grouping different services into batches and placing them in the same computing node) will have a lot of redundant calculations and latency overhead, resulting in low overall throughput of the service system.

[0202] Figure 3 An architecture diagram of a request scheduling system provided in an embodiment of the present invention.

[0203] The request scheduling system includes a first device, at least one second device, and at least one third device. The second device and the third device are both connected to the first device. The connection includes a wired connection or a wireless connection.

[0204] The hardware structure and software structure of the device provided in the embodiment of the present invention can be found in Figure 1 and Figure 2 For relevant description about device 100.

[0205] Exemplarily, the first device includes a server.

[0206] Exemplarily, the second device and the third device include electronic devices with computing capabilities. The electronic device includes a computing card. Exemplarily, the computing card includes a GPU or an NPU. The second device is used for Prefill reasoning, that is, reasoning in the Prefill stage; the third device is used for Decode reasoning, that is, reasoning in the Decode stage.

[0207] Take the request scheduling system including the first device, the second device and the third device as an example. Figure 3 , the request scheduling system includes a first device 210, a second device 220, and a third device 230.

[0208] The first device 210 includes a large-scale language model (LLM) service module 211 and a service scheduling module 212 .

[0209] The LLM service module 211 is used to connect to the external user service portal.

[0210] The service scheduling module 212 is used to schedule new services to enter the second device 220 , and schedule old services that have not reached the stop condition to enter the third device 230 .

[0211] The second device 220 includes a first reasoning component package 221 , a first AI engine 222 , and a first computing resource module 223 .

[0212] The first reasoning component package 221 and the first AI engine 222 are used for the deployment and resource calling of large-scale language models.

[0213] The first computing resource module 223 includes a first batch scheduling module 2231 and a first execution engine 2232 .

[0214] The first batch scheduling module 2231 is used to dynamically batch or dynamically insert new services, wherein the new services are services that have not gone through the Prefill stage.

[0215] The first execution engine 2232 is used to perform inference calculations on the service requests output by the first batch scheduling module 2231 .

[0216] The third device 230 includes a second reasoning component package 231 , a second AI engine 232 , and a second computing resource module 233 .

[0217] The second reasoning component package 231 and the second AI engine 232 are used for the deployment and resource calling of large-scale language models.

[0218] The second computing resource module 233 includes a second batch scheduling module 2331 and a second execution engine 2332 .

[0219] The second batch scheduling module 2331 is used to dynamically batch, dynamically insert or dynamically stop old services that have not reached the stop condition. The old services are services that have passed the Prefill stage.

[0220] The second execution engine 2332 is used to perform inference calculations on the service requests output by the second batch scheduling module 2331 .

[0221] like Figure 3 As shown, the second device 220 further includes a first storage module 241, and the third device 230 further includes a second storage module 242. The second device 220 and the third device 230 are connected by wire or wirelessly.

[0222] In some possible embodiments, the first storage module 241 is used to cache the first cache information generated by the second device 220 during the reasoning process, and transmit the cached first cache information to the second storage module 242; the second execution engine 2232 is used to obtain the first cache information from the second storage module 242 during the reasoning process.

[0223] In some possible embodiments, the first execution engine 2232 does not send the first cache information generated by the second device 220 during the reasoning process to the first storage module 241, but directly sends it to the second execution engine 2232 for use by the second execution engine 2232 during the reasoning process.

[0224] The second storage module 242 is also used to cache cache information generated by the third device 230 during the reasoning process; the second execution engine 2232 is used to obtain the cache information generated by the previous round of reasoning process from the second storage module 242 during the reasoning process.

[0225] Exemplarily, the cache information is key-value cache KV information.

[0226] The service scheduling module 212 schedules the requests of multiple services to the second device 220 or the third device 230. The output information of the current iteration round and the corresponding cache information are obtained through the calculation of the second device 220 or the third device 230, and the output information is returned to the service scheduling module 212. When the stop condition is not met, the service scheduling module 212 schedules the service to the idle third device 230 for the next round of reasoning; after the reasoning iteration, it will continue to be returned to the service scheduling module 212. When the stop condition is met, the service scheduling module 212 will end the scheduling of the service and give an external business response.

[0227] The embodiment of the present invention also adopts an independent separate deployment method in the request scheduling system, deploying two execution containers, namely the second device 220, namely the Prefill inference execution container and the third device 230, namely the Decode inference execution container, and scheduling the request by the first device so that the second device and the third device respectively execute the inference process of their respective stages; at the same time, communication can be transmitted between the two devices, so that the inference process can proceed normally, thereby improving the throughput of the service system.

[0228] Both execution containers have a batch scheduling module, which implements two functions: automatic batching and automatic early stopping. Automatic batching is used to dynamically batch the input information of multiple service requests, including batching and dynamic insertion; automatic early stopping is used to stop the completed service in advance, release the corresponding resources and notify the system that the current resources can be inserted and occupied by the next service.

[0229] Through separate deployment, the significant differences between the computing power bottleneck in the Prefill stage and the memory access bottleneck in the Decode stage are taken into consideration, which can improve the throughput of the service system.

[0230] When the number of second devices and third devices in the request scheduling system is large, an independent separation deployment method is adopted to deploy the second devices and the third devices, the second devices are configured in small batches or single batches, and the third devices are configured in large batches. The resource ratio of the second device resources and the third device resources can be dynamically adjusted according to the system load to obtain the maximum throughput. The present invention can greatly improve the throughput of the service system.

[0231] based on Figure 3The system architecture shown in the figure, an embodiment of the present invention provides a request scheduling method. Figure 4 A signaling interaction diagram of a request scheduling method provided by an embodiment of the present invention. Figure 4 As shown, the method includes:

[0232] Step 302: The LLM service module receives a first request from a first user. Before step 302, that is, before requesting the scheduling system to respond to the service, the LLM has been deployed to the computing resource modules in the second device 220 and the third device 230 through the reasoning component package and the AI ​​engine, and the computing resource module and the storage module are ready.

[0233] Exemplarily, the first request is a request for a first service.

[0234] Exemplarily, the first request includes an identifier of the first service.

[0235] Illustratively, the identifier includes a service ID.

[0236] Exemplarily, the first request does not include requests returned by the second device and the third device, that is, the first request is a request for a new service.

[0237] In this step, if Figure 3 As shown, the LLM service module receives a first request from a first user.

[0238] Step 304: The LLM service module sends the first request to the service scheduling module.

[0239] In this step, if Figure 3 As shown, the LLM service module sends the first request to the service scheduling module.

[0240] Step 306: The service scheduling module sends the first request to the first reasoning component package.

[0241] In this step, if Figure 3 As shown, the service scheduling module schedules the newly arrived service request to the Prefill inference execution container, that is, the second device, and at the same time establishes a service pool for storing service requests that are waiting due to resource constraints.

[0242] Step 308: The first reasoning component sends the first request to the first AI engine.

[0243] In this step, if Figure 3 As shown, the first reasoning component sends the first request to the first AI engine.

[0244] Step 310: The first AI engine sends a first request to a first computing resource module.

[0245] In this step, if Figure 3 As shown, the first AI engine sends the first request to the first computing resource module.

[0246] Step 312: The first computing resource module batches the first request to obtain a first batch, performs a first inference calculation on the first batch, and obtains a second request and first cache information.

[0247] In this step, if Figure 3 As shown, the first batch scheduling module of the first computing resource module batches the first request and at least one seventh request that arrives at the second device at the same time as the first request according to the first computing resource state and the first condition to obtain the first batch; the first execution engine of the first computing resource module performs the first reasoning calculation on the first batch to obtain the second request and the first cache information. The seventh request is a request from the first user or the second user or other users.

[0248] Exemplarily, the first request further includes input information;

[0249] Exemplarily, the input information includes an input Token sequence.

[0250] Exemplarily, the first computing resource state includes the number of idle resources in the second device.

[0251] Exemplarily, the first condition includes the maximum number of batches that the second device can group.

[0252] Exemplarily, the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

[0253] Exemplarily, the first output information includes an output Token sequence. The first output information is a first output Token sequence.

[0254] Exemplarily, the cache information includes key-value cache KV information. The first cache information is first key-value cache KV information.

[0255] In some possible embodiments, step 312 specifically includes: the first batch scheduling module of the first computing resource module obtains the input information corresponding to the first request and its first length according to the first request; the first request and at least one seventh request that arrives at the second device at the same time as the first request are batched to obtain the first batch according to the first computing resource state, the first condition and the first length; the first execution engine of the first computing resource module performs the first inference calculation on the first batch in a single batch processing manner to obtain the second request and the first cache information of the first service. Among them, the length of the input information corresponding to the seventh request is similar to the first length, which is conducive to reducing the waste of idle resources in the first inference calculation process.

[0256] Illustratively, the first batch includes a plurality of requests.

[0257] Exemplarily, the first batch scheduling module groups multiple requests arriving at the same time into a batch.

[0258] Exemplarily, the first inference calculation is an inference calculation in the Prefill stage, ie, a Prefill calculation.

[0259] Exemplarily, the first processing method is to release the computing service resources occupied by the current batch and insert the next batch into the idle computing service resources for the first inference calculation after all requests of the current batch are inferred and calculated. In an embodiment of the present invention, for the inference calculation of the Prefill stage, the inference calculation is performed according to a single batch, that is, after all requests of the current batch are inferred and calculated, the computing service resources occupied by the current batch are released and the next batch is inserted into the idle computing service resources to start the first inference calculation for the next batch.

[0260] Exemplarily, the second device releases the storage resources occupied by the current batch while releasing the computing service resources occupied by the current batch.

[0261] Exemplarily, when the second device has idle computing service resources, the second device notifies the first device that the second device has idle computing service resources, and the first device schedules a new service to the second device according to the notification.

[0262] Exemplarily, when the second device has no idle computing service resources, the second device notifies the first device that the second device has no idle computing service resources, and the first device suspends scheduling new services to the second device according to the notification.

[0263] Step 314: The first computing resource module sends the first cache information to the second computing resource module.

[0264] In this step, if Figure 3 As shown, the first execution engine of the first computing resource module sends the first cache information to the second computing resource module.

[0265] Alternatively, if Figure 3 As shown, the first computing resource module can send the first cache information to the first storage module, the first storage module sends the first cache information to the second storage module, and then the second storage module sends the first cache information to the second execution engine module. The second storage module can automatically send the first cache information to the second execution engine module, or send the first cache information to the second execution engine module according to the message sent by the second execution engine module.

[0266] Alternatively, if Figure 3As shown, the first computing resource module can send the first cache information directly to the second computing resource module.

[0267] Step 316 : The first computing resource module sends the second request to the service scheduling module, and continues to execute 318 or 320 .

[0268] In this step, if Figure 3 and Figure 4 As shown, the first computing resource module sends the second request to the service scheduling module.

[0269] Step 318: When the first output information meets the set condition, the service scheduling module sends the first output information to the LLM service module. The LLM service module sends the first output information to the first user, and the process ends.

[0270] In an embodiment of the present invention, the first device determines whether the output information of the service meets the set conditions. If the set conditions are not met, the service continues to be scheduled and inferred. If the set conditions are met, the service is stopped early and the second output information is sent to the first user.

[0271] Step 320: When the first output information does not meet the set condition, the service scheduling module schedules the second request to the second reasoning component package.

[0272] In this step, if Figure 3 As shown, the service scheduling module sends the second request to the second reasoning component package.

[0273] Exemplarily, the second request is a request for the first service returned by the second device.

[0274] Exemplarily, the set conditions include at least one of the following:

[0275] The length of the output information is greater than or equal to the first threshold, or

[0276] The output includes the terminator.

[0277] Step 322: The second reasoning component sends the second request to the second AI engine.

[0278] In this step, if Figure 3 As shown, the second reasoning component sends the second request to the second AI engine.

[0279] Step 324: The second AI engine sends the second request to the second computing resource module.

[0280] In this step, if Figure 3 As shown, the second AI engine sends the second request to the second computing resource module.

[0281] Step 326: The second computing resource module batches the second request to obtain a second batch.

[0282] In this step, if Figure 3 As shown, the second batch scheduling module of the second computing resource module batches the second requests according to the second computing resource state and the second condition to obtain a second batch.

[0283] In some possible embodiments, step 326 specifically includes: the second batch scheduling module batches the second request and at least one eighth request that arrives at the third device simultaneously with the second request to obtain a second batch according to the second computing resource state and the second condition, and the eighth request comes from the second device or the third device.

[0284] Exemplarily, the second computing resource state includes the number of idle resources in the third device.

[0285] Exemplarily, the second condition includes the maximum number of batches that the third device can group.

[0286] Illustratively, the second batch includes a plurality of requests.

[0287] Exemplarily, the second batch scheduling module groups at least one request arriving at the same time into a batch. In the decoding stage, the input length of each request is 1. As long as there is an idle position in the current batch, this token with a length of 1 can be put into the batch for inference.

[0288] Step 328: The second computing resource module performs a second inference calculation on the second batch according to the first cache information to obtain a third request and second cache information.

[0289] In this step, if Figure 3 As shown, the second execution engine of the second computing resource module performs a second inference calculation on the second batch according to the first cache information to obtain a third request and second cache information.

[0290] In some possible embodiments, step 328 specifically includes: a second execution engine of the second computing resource module performs a second inference calculation on the second batch according to the first cache information using a second processing method to obtain the third request and the second cache information of the first service.

[0291] Exemplarily, the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request.

[0292] Exemplarily, the second cache information is cache information required when the third device performs a second inference calculation on the third request.

[0293] Exemplarily, the second inference calculation is an inference calculation in the Decode stage, that is, a Decode calculation.

[0294] Exemplarily, the second processing method is to release the computing service resources of the target request and insert the requests in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the current batch inference process. In an embodiment of the present invention, for the inference calculation in the Decode stage, the inference calculation is performed in multiple batches, that is, during the current batch inference process, if a service ends the inference in advance, that is, there is a target request whose output information meets the set conditions, the computing service resources of the target request can be released in advance, and before the current batch ends, the requests in the next batch can be inserted into the idle computing service resources for the second inference calculation. Therefore, compared with the first processing method, which needs to wait for all requests in the current batch to be inferred before releasing the resources of the current batch, the second processing method does not need to wait for all requests in the current batch to be inferred before releasing the resources of the current batch, and there will be no waste of idle resources, thereby further improving the throughput of the system.

[0295] Exemplarily, the third device releases the storage resources occupied by the target request while releasing the computing service resources occupied by the target request.

[0296] Exemplarily, when the third device has idle computing service resources, the third device notifies the first device that the third device has idle computing service resources, and the first device schedules the old service to the second device according to the notification.

[0297] Exemplarily, when the third device has no idle computing service resources, the first device is notified that the third device has no idle computing service resources, and the first device suspends scheduling old services to the third device according to the notification.

[0298] Step 330: The second computing resource module sends the second cache information to the second storage module.

[0299] In this step, if Figure 3 As shown, the second execution engine of the second computing resource module sends the second cache information to the second storage module.

[0300] Step 332 : The second computing resource module sends the third request to the service scheduling module, and continues to execute step 334 or step 336 .

[0301] In this step, if Figure 3 As shown, the second execution engine of the second computing resource module sends the third request to the service scheduling module.

[0302] Exemplarily, the third request is a request for the first service returned by the third device.

[0303] Step 334: When the second output information meets the set condition, the service scheduling module sends the second output information to the LLM service module. The LLM service module sends the second output information to the first user, and the process ends.

[0304] In the embodiment of the present invention, the first device determines whether the output information of the service meets the set conditions. If the set conditions are not met, the service continues to be scheduled and inferred; if the set conditions are met, the service is stopped early and the second output information is sent to the first user. After the first device stops the service early, it can notify the third device to release the resources corresponding to the service; the third device reports the current resource status of the third device to the first device. If there are idle computing service resources in the current resource status, the first device continues to schedule the service to the third device to insert the idle computing service resources, thereby realizing dynamic insertion.

[0305] In this step, if Figure 3 As shown, when the second output information meets the set condition, the service scheduling module sends the second output information to the LLM service module. The LLM service module sends the second output information to the first user, and the process ends.

[0306] Step 336: When the second output information does not meet the set condition, the service scheduling module schedules the third request to the second reasoning component package.

[0307] In this step, if Figure 3 As shown, when the second output information does not meet the set conditions, the service scheduling module sends the third request to the second reasoning component package, so that the third device continues to perform the next round of Decode calculation on the third request, and uses the second cache information generated by the previous round of Decode calculation in the next round of Decode calculation. The second execution engine sends the cache information generated by the next round of Decode calculation to the second storage module for storage, so that it can be used by the second execution engine when performing the next round of Decode calculation.

[0308] For example, the request scheduling service system includes 4 second devices and 1 third device. Figure 5 Schematic diagram of Prefill reasoning and Decode reasoning in an embodiment of the present invention. Figure 5 As shown, the four second devices are device 0, device 1, device 2 and device 3; and the third device is device x. Figure 5The figure shows the reasoning process in which Prefill reasoning uses the first processing method and Decode reasoning uses the second processing method, namely: one row in the Prefill reasoning on the left represents processing a service request in a group of computing resources, with a total of 4 groups of Prefill computing resources and 4 service requests; the 4 rows in the Decode reasoning on the right represent processing a batch of 4 service requests in a group of computing resources, with a total of 1 group of computing resources and 4 service requests. Figure 5 In the figure, each square represents a computation delay. The Prefill computation on the left and the Decode computation on the right are performed in parallel, and there is no timing relationship.

[0309] Figure 5 The process of Prefill reasoning and Decode reasoning shown is as follows:

[0310] 1) The lengths of the input Token sequences of the four services req1, req2, req3 and req4 are 3, 2, 4 and 3 respectively. Therefore, when the four services arrive at device 0, device 1, device 2 and device 3 at the same time, req2 will be completed in advance, followed by req1, req4 and req3.

[0311] 2) In the Prefill calculation, the req2 that is completed in advance will enter the Decode calculation in advance. Figure 5 The right side of shows the schematic diagram of the system startup process; if device x wants to form a batch, and a batch includes 4 requests, then the 4 services req1, req2, req3 and req4 need to enter the Decode calculation. Figure 5 As shown in the right decode reasoning, req2 arrives first, followed by req1 and req4, and finally req3. Therefore, req1 and req4 will generate 1 unit of free bubble, while req3 will generate 2 units of free bubble. The bubble here will not exist again after the system stabilizes.

[0312] 3) In the Decode calculation, the output token numbers of req1 to 4 are 5, 7, 6, and 4 respectively. Therefore, although req4 is the second to enter the Decode calculation in the Prefill calculation, it is the first to complete the Decode calculation. Therefore, the output information of req4 is the first to meet the set conditions in the batch. After completing the Decode calculation, req4 will stop the calculation in advance (early stopping mechanism) and release the corresponding resource occupation; the requests that complete the calculation are req1, req2, and req3.

[0313] 4) During the Decode calculation process of req1-4, req5-8 have reached the Prefill inference execution container for calculation and have completed calculation before the calculation of req1-4 ends. Therefore, when req4, req1, req2, and req3 stop calculating one after another, the service scheduling module will automatically schedule req6, req5, req7, and req8 to device x one after another, and device x will insert req6, req5, req7, and req8 into the resource positions previously occupied by req4, req1, req2, and req3 one after another, and perform subsequent calculations after dynamic insertion.

[0314] 5) After the system is stable, steps 3) and 4) will be repeated to dynamically insert batches and dynamically stop and give service responses.

[0315] It should be noted that when the service scheduling module schedules the third request of the first service to the third device, it will only schedule the third request to the third device that stores the cache information generated by the previous round of Decode reasoning of the first service. Therefore, in the Decode stage, the Decode part is the service iterated in the current third device, and the request of this service will not be forwarded to other third devices.

[0316] In summary, the present invention sends the request output after reasoning calculation to the service scheduling module through the execution engine, and the service scheduling module determines that the request meets the set conditions, thereby realizing automatic early stopping of the service; the batch scheduling module groups the requests into batches to obtain batches, thereby realizing batch grouping; when there are idle computing service resources in the execution engine, the batch scheduling module automatically sends the requests in the next batch to the execution engine to realize dynamic insertion.

[0317] The embodiment of the present invention dispatches the request for the new service from the user to the second device, namely the Prefill inference execution container, to perform the inference calculation of the Prefill stage; and dispatches the request for the old service that has passed the Prefill stage to the third device, namely the Decode inference execution container, to perform the inference calculation of the Decode stage. The present invention comprehensively considers the significant difference between the computing power bottleneck of the Prefill stage and the memory access bottleneck of the Decode stage through separate deployment, thereby improving the throughput of the service system.

[0318] Alternatively, if Figure 6 As shown, the method also includes: steps 338 to 372.

[0319] Step 338: The LLM service module receives the fourth request from the second user. Before step 338, that is, before requesting the scheduling system to respond to the service, the LLM has been deployed to the computing resource modules in the second device and the third device through the reasoning component package and the AI ​​engine, and the computing resource module and the storage module are ready.

[0320] Exemplarily, the second user is different from the first user. The first device is capable of scheduling requests from different users.

[0321] Exemplarily, the fourth request is a request for the second service.

[0322] Exemplarily, the fourth request includes an identifier of the second service.

[0323] Illustratively, the identifier includes a service ID.

[0324] Exemplarily, the fourth request does not include requests returned by the second device and the third device, that is, the fourth request is a request for a new service.

[0325] In this step, if Figure 3 As shown, the LLM service module receives a fourth request from the second user.

[0326] Step 340: The LLM service module sends the fourth request to the service scheduling module.

[0327] In this step, if Figure 3 As shown, the LLM service module sends the fourth request to the service scheduling module.

[0328] Step 342: The service scheduling module sends the fourth request to the first reasoning component package.

[0329] In this step, if Figure 3 As shown, the service scheduling module schedules the newly arrived service request to the Prefill inference execution container, that is, the second device, and at the same time establishes a service pool for storing service requests that are waiting due to resource constraints.

[0330] Step 344: The first reasoning component sends the fourth request to the first AI engine.

[0331] In this step, if Figure 3 As shown, the first reasoning component sends the fourth request to the first AI engine.

[0332] Step 346: The first AI engine sends the fourth request to the first computing resource module.

[0333] In this step, if Figure 3 As shown, the first AI engine sends the fourth request to the first computing resource module.

[0334] Step 348: The first computing resource module batches the fourth request to obtain a third batch, performs a first inference calculation on the third batch, and obtains a fifth request and third cache information.

[0335] In this step, if Figure 3 As shown, the first batch scheduling module of the first computing resource module batches the fourth request to obtain the third batch according to the first computing resource state and the first condition; the first execution engine of the first computing resource module performs the first inference calculation on the third batch to obtain the fifth request and the third cache information.

[0336] Exemplarily, the fourth request further includes input information;

[0337] Exemplarily, the input information includes an input Token sequence.

[0338] Exemplarily, the first computing resource state includes the number of idle resources in the second device.

[0339] Exemplarily, the first condition includes the maximum number of batches that the second device can group.

[0340] Exemplarily, the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request.

[0341] Exemplarily, the third output information includes an output Token sequence. The third output information is a third output Token sequence.

[0342] Exemplarily, the cache information includes key-value cache KV information. The third cache information is third key-value cache KV information.

[0343] In some possible embodiments, step 348 specifically includes: the first batch scheduling module of the first computing resource module obtains the input information corresponding to the fourth request and its second length according to the fourth request; the fourth request is batched to obtain a third batch according to the first computing resource state, the first condition and the second length; the first execution engine of the first computing resource module performs a first inference calculation on the third batch using the first processing method to obtain the fifth request and the third cache information of the second service.

[0344] Illustratively, the third batch includes a plurality of requests.

[0345] Exemplarily, the first batch scheduling module groups multiple requests arriving at the same time into a batch.

[0346] Exemplarily, the first inference calculation is an inference calculation in the Prefill stage, ie, a Prefill calculation.

[0347] Exemplarily, the first processing method is to release the computing service resources occupied by the current batch and insert the next batch into the idle computing service resources to perform the first inference calculation after all requests of the current batch are inferred and calculated.

[0348] Step 350: The first computing resource module sends the third cache information to the second computing resource module.

[0349] In this step, if Figure 3 As shown, the first execution engine of the first computing resource module sends the third cache information to the second computing resource module.

[0350] Alternatively, if Figure 3 As shown, the first computing resource module can send the third cache information to the first storage module, the first storage module sends the third cache information to the second storage module, and then the second storage module sends the third cache information to the second execution engine module. Among them, the second storage module can automatically send the third cache information to the second execution engine module, or send the third cache information to the second execution engine module according to the message sent by the second execution engine module.

[0351] Alternatively, if Figure 3 As shown, the first computing resource module can send the third cache information directly to the second computing resource module.

[0352] Step 352 : The first computing resource module sends the fifth request to the service scheduling module, and continues to execute 354 or 356 .

[0353] In this step, if Figure 3 and Figure 4 As shown, the first computing resource module sends the fifth request to the service scheduling module.

[0354] Step 354: When the third output information meets the set condition, the service scheduling module sends the third output information to the LLM service module. The LLM service module sends the third output information to the second user, and the process ends.

[0355] Step 356: When the third output information does not meet the set condition, the service scheduling module schedules the fifth request to the second reasoning component package.

[0356] In this step, if Figure 3 As shown, the service scheduling module sends the fifth request to the second reasoning component package.

[0357] Exemplarily, the fifth request is a request for the second service returned by the second device.

[0358] Exemplarily, the set conditions include at least one of the following:

[0359] The length of the output information is greater than or equal to the first threshold, or

[0360] The output includes the terminator.

[0361] Step 358: The second reasoning component sends the fifth request to the second AI engine.

[0362] In this step, if Figure 3 As shown, the second reasoning component sends the fifth request to the second AI engine.

[0363] Step 360: The second AI engine sends the fifth request to the second computing resource module.

[0364] In this step, if Figure 3 As shown, the second AI engine sends the fifth request to the second computing resource module.

[0365] Step 362: The second computing resource module batches the fifth request to obtain a fourth batch.

[0366] In this step, if Figure 3 As shown, the second batch scheduling module of the second computing resource module batches the fifth requests according to the second computing resource state and the second condition to obtain a fourth batch.

[0367] In some possible embodiments, step 362 specifically includes: the second batch scheduling module batches the fifth request according to the second computing resource state and the second condition to obtain a fourth batch.

[0368] Exemplarily, the second computing resource state includes the number of idle resources in the third device.

[0369] Exemplarily, the second condition includes the maximum number of batches that the third device can group.

[0370] Illustratively, the fourth batch includes a plurality of requests.

[0371] Exemplarily, the second batch scheduling module groups at least one request arriving at the same time into a batch. In the decoding stage, the input length of each request is 1. As long as there is an idle position in the current batch, this token with a length of 1 can be put into the batch for inference.

[0372] Step 364: The second computing resource module performs a second inference calculation on the fourth batch according to the third cache information to obtain a sixth request and fourth cache information.

[0373] In this step, if Figure 3 As shown, the second execution engine of the second computing resource module performs a second inference calculation on the fourth batch according to the third cache information to obtain a sixth request and fourth cache information.

[0374] In some possible embodiments, step 364 specifically includes: the second execution engine of the second computing resource module performs a second inference calculation on the fourth batch according to the third cache information using a second processing method to obtain the sixth request and the fourth cache information of the second service.

[0375] Exemplarily, the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request.

[0376] Exemplarily, the fourth cache information is cache information required by the third device when performing inference calculation on the sixth request.

[0377] Exemplarily, the second inference calculation is an inference calculation in the Decode stage, that is, a Decode calculation.

[0378] Exemplarily, the second processing method is to release the computing service resources of the target request and insert the requests in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the current batch inference process.

[0379] Step 366: The second computing resource module sends the fourth cache information to the second storage module.

[0380] In this step, if Figure 3 As shown, the second execution engine of the second computing resource module sends the fourth cache information to the second storage module.

[0381] Step 368 : The second computing resource module sends the sixth request to the service scheduling module, and continues to execute step 370 or step 372 .

[0382] In this step, if Figure 3 As shown, the second execution engine of the second computing resource module sends the sixth request to the service scheduling module.

[0383] Exemplarily, the sixth request is a request for the second service returned by the third device.

[0384] Step 370: When the fourth output information meets the set condition, the service scheduling module sends the fourth output information to the LLM service module. The LLM service module sends the fourth output information to the second user, and the process ends.

[0385] In this step, if Figure 3 As shown, when the fourth output information meets the set condition, the service scheduling module sends the fourth output information to the LLM service module. The LLM service module sends the fourth output information to the second user, and the process ends.

[0386] Step 372: When the fourth output information does not meet the set condition, the service scheduling module schedules the sixth request to the second reasoning component package.

[0387] based on Figure 3 The system architecture shown in the figure, an embodiment of the present invention provides a request scheduling method. Figure 7 A flowchart of a request scheduling method provided by an embodiment of the present invention. Figure 7 As shown, the method includes:

[0388] Step 402: The first device receives a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service.

[0389] Illustratively, the identifier includes a service ID.

[0390] Exemplarily, the first request does not include requests returned by the second device and the third device, that is, the first request is a request for a new service.

[0391] Step 404: The first device dispatches the first request to the second device.

[0392] The first device schedules the newly arrived service request to the Prefill inference execution container, that is, the second device, and simultaneously establishes a service pool for storing service requests that are waiting due to resource constraints.

[0393] Step 406: The second device sends a second request to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

[0394] Exemplarily, the first output information includes an output Token sequence. The first output information is a first output Token sequence.

[0395] Step 408: When the first output information does not meet the set condition, the first device dispatches the second request to a third device, which is different from the second device.

[0396] Exemplarily, the second request is a request for the first service returned by the second device.

[0397] Exemplarily, the set conditions include at least one of the following:

[0398] The length of the output information is greater than or equal to the first threshold, or

[0399] The output includes the terminator.

[0400] Step 410: The third device sends a third request to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request.

[0401] Step 412: When the second output information does not meet the set condition, the first device schedules the third request to the third device.

[0402] Exemplarily, the third request is a request for the first service returned by the third device.

[0403] The first device sends the third request to the third device, so that the third device continues to perform the next round of Decode calculation on the third request, and uses the second cache information generated by the previous round of Decode calculation in the next round of Decode calculation. The third device stores the cache information generated by the next round of Decode calculation for use in the next round of Decode calculation.

[0404] Alternatively, if Figure 7 As shown, between step 404 and step 406 are also included: step 4052-step 4054.

[0405] Step 4052: The second device batches the first request according to the first computing resource state and the first condition to obtain a first batch.

[0406] Exemplarily, the first request also includes input information.

[0407] Exemplarily, the input information includes an input Token sequence.

[0408] Exemplarily, the first computing resource state includes the number of idle resources in the second device.

[0409] Exemplarily, the first condition includes the maximum number of batches that the second device can group.

[0410] In some possible embodiments, step 4052 specifically includes: the second device obtains input information corresponding to the first request and its first length according to the first request; and batches the first request and at least one seventh request that arrives at the second device at the same time as the first request to obtain a first batch according to the first computing resource state, the first condition and the first length.

[0411] The seventh request is a request from the first user or the second user or other users.

[0412] The length of the input information corresponding to the seventh request is similar to the first length, which is beneficial to reducing the waste of idle resources in the first reasoning calculation process.

[0413] Step 4054: The second device performs a first inference calculation on the first batch to obtain a second request and first cache information.

[0414] Exemplarily, the cache information includes key-value cache KV information. The first cache information is first key-value cache KV information.

[0415] In some possible embodiments, step 4052 specifically includes: the second device performs a first inference calculation on the first batch using a first processing method.

[0416] Exemplarily, the first inference calculation is an inference calculation in the Prefill stage, ie, a Prefill calculation.

[0417] Exemplarily, the first processing method is to release the computing service resources occupied by the current batch and insert the next batch into the idle computing service resources to perform the first inference calculation after all requests of the current batch are inferred and calculated.

[0418] In the embodiment of the present invention, after step 4052, it also includes: the second device sends the first cache information to the third device.

[0419] Optionally, after step 406, the method further includes: when the first output information meets a set condition, the first device sends the first output information to the first user.

[0420] Alternatively, if Figure 7 As shown, between step 408 and step 410 are also steps 4092 to 4094.

[0421] Step 4092: The third device batches the second requests according to the second computing resource state and the second condition to obtain a second batch.

[0422] Exemplarily, the second computing resource state includes the number of idle resources in the third device.

[0423] Exemplarily, the second condition includes the maximum number of batches that the third device can group.

[0424] In some possible embodiments, step 4092 specifically includes: grouping the second request and at least one eighth request that arrives at the third device simultaneously with the second request into batches according to a second computing resource state and a second condition to obtain a second batch, wherein the eighth request comes from the second device or the third device.

[0425] Step 4094: The third device performs a second inference calculation on the second batch according to the first cache information to obtain a third request and second cache information.

[0426] Exemplarily, the second inference calculation is the inference calculation in the Decode stage, that is, the Decode calculation. In the Decode stage, the input length of each request is 1. As long as there is an idle position in the current batch, this token with a length of 1 can be put into the batch for inference.

[0427] In some possible embodiments, step 4094 specifically includes: the third device performs a second inference calculation on the second batch using a second processing method according to the first cache information to obtain a third request and second cache information.

[0428] Exemplarily, the second processing method is to release the computing service resources of the target request and insert the requests in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the current batch inference process.

[0429] In the embodiment of the present invention, after step 4094, it also includes: a third device storing the second cache information.

[0430] Exemplarily, the second cache information is cache information required when the third device performs a second inference calculation on the third request.

[0431] Alternatively, if Figure 7 As shown, step 412 also includes: steps 414 to 424.

[0432] Step 414: The first device receives a fourth request from the second user, where the fourth request is a request for the second service and includes an identifier of the second service.

[0433] Step 416: The first device schedules the fourth request to the second device.

[0434] Step 418: The second device sends a fifth request to the first device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request.

[0435] Step 420: When the third output information does not meet the set condition, the first device schedules the fifth request to the third device.

[0436] Step 422: The third device sends a sixth request to the first device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request.

[0437] Step 424: When the fourth output information does not meet the set condition, the first device schedules the sixth request to the third device.

[0438] Optionally, between step 416 and step 418 also include: step 4172-step 4174.

[0439] Step 4172: The second device batches the fourth requests according to the first computing resource state and the first condition to obtain a third batch.

[0440] Exemplarily, the fourth request also includes input information.

[0441] In some possible embodiments, step 4172 specifically includes: the second device obtains input information corresponding to the fourth request and its second length according to the fourth request; and batches the fourth request to obtain a third batch according to the first computing resource state, the first condition and the second length.

[0442] Step 4174: The second device performs the first inference calculation on the third batch to obtain the fifth request and third cache information.

[0443] In some possible embodiments, step 4172 specifically includes: the second device performs a first inference calculation on the third batch using a first processing method.

[0444] Exemplarily, the first processing method is to release the computing service resources occupied by the current batch and insert the next batch into the idle computing service resources to perform the first inference calculation after all requests of the current batch are inferred and calculated.

[0445] In the embodiment of the present invention, after step 4172, it also includes: the second device sends the third cache information to the third device.

[0446] Optionally, after step 418, the method further includes: when the third output information meets a set condition, the first device sends the third output information to the first user.

[0447] Optionally, between step 420 and step 422 are also included: step 4212 to step 4214.

[0448] Step 4212: The third device batches the fifth requests according to the second computing resource state and the second condition to obtain a fourth batch.

[0449] Step 4214: The third device performs a second inference calculation on the fourth batch according to the third cache information to obtain a sixth request and fourth cache information.

[0450] In some possible embodiments, step 4214 specifically includes: the third device performs a second inference calculation on the fourth batch according to the third cache information using a second processing method to obtain a sixth request and fourth cache information.

[0451] Exemplarily, the second processing method is to release the computing service resources of the target request and insert the requests in the next batch into the idle computing service resources for the second inference calculation if there is a target request whose output information meets the set conditions during the current batch inference process.

[0452] In the embodiment of the present invention, after step 4094, it also includes: the third device stores the fourth cache information.

[0453] The present invention discloses a request scheduling method and device, the method comprising: a first device receives a first request from a first user, the first request is a request for a first service, and the first request includes an identifier of the first service; the first device schedules the first request to a second device; the second device sends a second request to the first device, the second request includes an identifier of the first service and first output information, and the second request is associated with the first request; when the first output information does not meet a set condition, the first device schedules the second request to a third device, and the third device is different from the second device; the third device sends a third request to the first device, the third request includes an identifier of the first service and second output information, and the third request is associated with the second request; when the second output information does not meet the set condition, the first device schedules the third request to the third device, which can improve the throughput of the service system in large-scale language model service deployment.

[0454] based on Figure 3 In the system architecture shown, an embodiment of the present invention provides a request scheduling method applied to a first device. Figure 8 A flowchart of another request scheduling method provided by an embodiment of the present invention. Figure 8 As shown, the method includes:

[0455] Step 502: The first device receives a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service.

[0456] Step 504: dispatch the first request to the second device.

[0457] Step 506: Receive a second request sent by the second device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

[0458] Step 508: When the first output information does not meet the set condition, dispatch the second request to a third device, which is different from the second device.

[0459] Step 510: Receive a third request sent by a third device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request.

[0460] Step 512: When the second output information does not meet the set condition, dispatch the third request to the third device.

[0461] In the technical solution of the request scheduling method provided by an embodiment of the present invention, the method includes: a first device receives at least one first request, the first request is a request for a service, and schedules at least one first request to a second device, the first request does not include requests returned by the second device and the third device; receives at least one second request returned by the second device, and schedules at least one second request to a third device; receives at least one third request returned by the third device, and determines whether the third request satisfies a third condition based on the third request. If it is determined that the third request does not satisfy the third condition, scheduling the third request to the third device, which can improve the throughput of the service system in large-scale language model service deployment.

[0462] Fig. 9 This is a schematic diagram of the structure of a first device provided by an embodiment of the present invention. It should be understood that the first device 600 can execute each step of the first device in the above request scheduling method. To avoid repetition, it is not described in detail here. The first device 600 includes: a first transceiver unit 601.

[0463] The first transceiver unit 601 is used to receive a first request from a first user, the first request is a request for a first service, and the first request includes an identifier of the first service; schedule the first request to a second device; receive a second request sent by the second device, the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request; when the first output information does not meet the set conditions, schedule the second request to a third device, and the third device is different from the second device; receive a third request sent by the third device, the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request; when the second output information does not meet the set conditions, schedule the third request to the third device.

[0464] Optionally, the first transceiver unit 601 is also used to receive a fourth request from a second user, the fourth request being a request for a second service, the fourth request including an identifier of the second service; scheduling the fourth request to the second device; receiving a fifth request sent by the second device, the fifth request including an identifier of the second service and third output information corresponding to the second service, the fifth request being associated with the fourth request; when the third output information does not meet the set conditions, scheduling the fifth request to a third device; receiving a sixth request sent by the third device, the sixth request including an identifier of the second service and fourth output information corresponding to the second service, the sixth request being associated with the fifth request; when the fourth output information does not meet the set conditions, scheduling the sixth request to the third device.

[0465] Optionally, the first transceiver unit 601 is further configured to send the second output information to the first user when the second output information meets the set condition.

[0466] Optionally, the identifier includes a service ID.

[0467] Optionally, the set condition includes at least one of the following:

[0468] The length of the output information is greater than or equal to the first threshold, or

[0469] The output information includes a terminator.

[0470] Fig.10 The second device 700 is a schematic diagram of a structure of a second device provided in an embodiment of the present invention. It should be understood that the second device 700 can execute each step of the second device in the above request scheduling method. To avoid repetition, it is not described in detail here. The second device 700 includes: a second transceiver unit 701 and a second processing unit 702.

[0471] The second transceiver unit 701 is used to receive a first request from a first user sent by a first device, where the first request is a request for a first service and includes an identifier of the first service; and send a second request to the first device, where the second request includes the identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request.

[0472] Optionally, the second transceiver unit 701 is also used to receive a fourth request from a second user sent by the first device, where the fourth request is a request for a second service and includes an identifier of the second service; and send a fifth request to the first device, where the fifth request includes the identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request.

[0473] Optionally, before the second transceiver unit 701 sends the second request to the first device, the second processing unit 702 is used to batch the first request and at least one seventh request that arrives at the second device at the same time as the first request according to the first computing resource state and the first condition to obtain a first batch, where the seventh request is a request from the first user or the second user or other users; and perform a first inference calculation on the first batch to obtain the second request and the first cache information.

[0474] Optionally, the second transceiver unit 701 is further configured to send the first cache information to the third device.

[0475] Optionally, the cache information includes key-value cache KV information.

[0476] Optionally, the second processing unit 702 is specifically used to obtain input information corresponding to the first request and its first length according to the first request; and to batch the first request and at least one seventh request that arrives at the second device simultaneously with the first request to obtain a first batch according to the first computing resource state, the first condition and the first length.

[0477] Fig.11 The third device 800 is a schematic diagram of a structure of a third device provided in an embodiment of the present invention. It should be understood that the third device 800 can execute each step of the third device in the above request scheduling method. To avoid repetition, it is not described in detail here. The third device 800 includes: a third transceiver unit 801 and a third processing unit 802.

[0478] The third transceiver unit 801 is used to receive a second request from a second device sent by a first device, where the second request includes an identifier of a first service and first output information corresponding to the first service, and the second request is associated with the first request, where the first request is a request for the first service from a first user; and send a third request to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request.

[0479] Optionally, the third transceiver unit 801 is also used to receive a fifth request from the second device sent by the first device, the fifth request including an identifier of the second service and third output information corresponding to the second service, the fifth request being associated with the fourth request, the fourth request being a request for the second service from a second user; and send a sixth request to the first device, the sixth request including an identifier of the second service and fourth output information corresponding to the second service, the sixth request being associated with the fifth request.

[0480] Optionally, before the third transceiver unit 801 sends the third request to the first device, the third processing unit 802 is configured to group the second request and at least one eighth request that arrives at the third device simultaneously with the second request into batches according to the second computing resource state and the second condition to obtain a second batch, wherein the eighth request comes from the second device or the third device;

[0481] The third transceiver unit 801 is further configured to receive first cache information sent by the second device;

[0482] The third processing unit 802 is further configured to perform a second inference calculation on the second batch according to the first cache information to obtain the third request and the second cache information.

[0483] It should be understood that the first device 600, the second device 700 and the third device 800 here are embodied in the form of functional units. The term "unit" here can be implemented in the form of software and / or hardware, and is not specifically limited to this. For example, a "unit" can be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a proprietary processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a merged logic circuit, and / or other suitable components that support the described functions.

[0484] Therefore, the units of each example described in the embodiments of the present invention can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0485] The embodiment of the present application provides a device, which may be a terminal device or a circuit device built into the terminal device. The device may be used to execute the functions / steps in the above method embodiment.

[0486] An embodiment of the present application provides a readable storage medium, in which instructions are stored. When the instructions are executed on a device or a processor, the device executes the functions / steps in the above method embodiment.

[0487] The embodiment of the present application also provides a program product including instructions, which, when executed on a device or at least one processor, enables the device to execute the functions / steps in the above method embodiment.

[0488] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0489] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0490] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0491] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for enabling a device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0492] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A request scheduling method, characterized in that: Applied to a request scheduling system, the method comprises: The first device receives a first request from a first user, where the first request is a request for a first service and includes an identifier of the first service; The first device dispatches the first request to the second device; The second device sends the cache information to the third device; The second device sends a second request to the first device, where the second request includes an identifier of the first service and first output information corresponding to the first service, and the second request is associated with the first request; When the first output information does not meet the set condition, the first device dispatches the second request to the third device, where the third device is different from the second device; The third device obtains a third request according to the cache information; The third device sends the third request to the first device, where the third request includes an identifier of the first service and second output information corresponding to the first service, and the third request is associated with the second request; When the second output information does not satisfy the set condition, the first device dispatches the third request to the third device.

2. The method according to claim 1, characterized in that The method further comprises: The first device receives a fourth request from a second user, where the fourth request is a request for a second service and includes an identifier of the second service; The first device dispatches the fourth request to the second device; The second device sends a fifth request to the first device, where the fifth request includes an identifier of the second service and third output information corresponding to the second service, and the fifth request is associated with the fourth request; When the third output information does not satisfy the set condition, the first device dispatches the fifth request to the third device; The third device sends a sixth request to the first device, where the sixth request includes an identifier of the second service and fourth output information corresponding to the second service, and the sixth request is associated with the fifth request; When the fourth output information does not satisfy the set condition, the first device dispatches the sixth request to the third device.

3. The method according to claim 2, characterized in that The method further comprises: When the second output information meets the set condition, the first device sends the second output information to the first user.

4. The method according to claim 3, characterized in that The request scheduling system includes the first device, at least one of the second devices, and at least one of the third devices.

5. The method according to any one of claims 1 to 3, characterized in that: The second device includes a first computing resource module, and the third device includes a second computing resource module, wherein the second device sending cache information to the third device includes the first computing resource module sending cache information to the second computing resource module.

6. The method according to claim 5, characterized in that The second device also includes a first storage module, and the third device also includes a second storage module, wherein the first computing resource module sends cache information to the second computing resource module, including: the first computing resource module sends the cache information to the first storage module, the first storage module sends the cache information to the second storage module, and then the second storage module sends the cache information to the second computing resource module.

7. The method according to claim 6, characterized in that The second storage module automatically sends the cache information to the second computing resource module.

8. The method according to claim 6, characterized in that The second storage module sends the cache information to the second computing resource module according to the message sent by the second computing resource module.

9. A device, characterized in that: The device comprises a processor and a memory, wherein the memory is used to store a computer program, and the computer program comprises program instructions. When the processor runs the program instructions, the device executes the steps of the method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium stores a program, wherein the program includes program instructions, and when the program request is executed by a device, the device is caused to execute the method according to any one of claims 1 to 8.

Images