Request processing method and device of large language model, medium, equipment and product
By decoupling the pre-filling and decoding processing stages in the large language model and processing them on two graphics processors respectively, the problems of slow inference speed and high computing cost of large language model are solved, and GPU resource utilization and service throughput are improved quantity.
Patent Information
- Application Number
- CN202411119228.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-08
AI Technical Summary
The huge scale of large language models leads to slow inference and expensive computational cost. How to improve their resource utilization has become a technical problem that needs to be solved urgently.
Through a large language model deployed on two graphics processors, the pre-filling processing and decoding processing stages are decoupled, and performed on the first graphics processor and the second graphics processor respectively, improving the utilization rate of GPU resources and service throughput of the service quantity.
By decoupling the pre-filling and decoding stages, the utilization of GPU resources and the service throughput of the large language model are improved, and the average computing cost per service request is reduced.
Smart Images

Figure CN118916175A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of machine learning technology, and in particular, to a method, apparatus, medium, device and product for processing requests of a large language model. Background Art
[0002] Large Language Models (LLMs) can provide excellent performance in various natural language tasks. However, due to the huge size of large language models, their inference speed is generally slow and the computational cost is expensive. Therefore, how to improve the resource utilization of large language models has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.
[0004] In a first aspect, the present disclosure provides a method for processing a request of a large language model, comprising: Get the service request sent to the large language model; Pre-filling the service request by using a large language model deployed on the first graphics processor, obtaining a pre-filling result corresponding to the service request, and sending the pre-filling result to the second graphics processor; The pre-filled result is decoded by a large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request.
[0005] In a second aspect, the present disclosure provides a request processing device for a large language model, comprising: an acquisition module, configured to acquire a service request sent to the large language model; A first processing module is configured to perform pre-filling processing on the service request through a large language model deployed on the first graphics processor, obtain a pre-filling result corresponding to the service request, and send the pre-filling result to the second graphics processor; The second processing module is configured to decode the pre-filled result through the large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request.
[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.
[0007] In a fourth aspect, the present disclosure provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect.
[0008] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, which implements the steps of the method described in the first aspect when executed by a processor.
[0009] Based on the above technical solution, by obtaining a service request sent to a large language model, and pre-filling the service request through the large language model deployed on the first graphics processor, obtaining a pre-filling result corresponding to the service request, and sending the pre-filling result to the second graphics processor, and decoding the pre-filling result through the large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request, the pre-filling stage and the decoding stage of the large language model can be decoupled, thereby improving the utilization of GPU resources and improving the service throughput of the large language model.
[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 is a flowchart of a method for processing requests for a large language model according to some embodiments.
[0012] Figure 2 is a flowchart of a method for processing requests of a large language model according to some other embodiments.
[0013] Figure 3 is a schematic diagram of a scheduler according to some embodiments.
[0014] Figure 4 It is a schematic diagram of module connections of a request processing device for a large language model according to some embodiments.
[0015] Figure 5 is a schematic diagram of the structure of an electronic device according to some embodiments. DETAILED DESCRIPTION
[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0017] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0018] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0022] Figure 1 FIG. 1 is a flowchart of a method for processing a request for a large language model according to some embodiments. Figure 1 As shown, the embodiment of the present disclosure provides a method for processing a request of a large language model, which can be executed by an electronic device, and specifically can be executed by a device for processing a request of a large language model, which can be implemented by software and / or hardware and configured in the electronic device. Figure 1 As shown, the method may include the following steps.
[0023] In step 110 , a service request sent to a large language model is obtained.
[0024] Here, the service request may be a request sent by a user or application to a large language model deployed on an electronic device to instruct the large language model to perform a specific natural language processing task. For example, the service request may request the large language model to translate text in one language into another language, the service request may request the large language model to generate text based on a given prompt or context, the service request may request the large language model to answer questions about a specific topic, and so on.
[0025] The electronic device may receive a service request initiated by a client, and transmit the received service request to a large language model deployed in the electronic device.
[0026] In step 120, the service request is pre-filled by using the large language model deployed on the first graphics processor, a pre-filled result corresponding to the service request is obtained, and the pre-filled result is sent to the second graphics processor.
[0027] Here, the electronic device can be understood as a computing node of the large language model service, which is used to provide the large language model service. In the electronic device, there are two types of graphics processors (Graphics Processing Unit, GPU), namely a first graphics processor of the first type and a second graphics processor of the second type. Among them, the first graphics processor is used to prefill the service request to obtain the prefill result, and the second graphics processor is used to decode the prefill result to obtain the inference result.
[0028] It should be understood that the electronic device includes a plurality of graphics processors, and the ratio of the number of the second graphics processor to the number of the first graphics processor in the plurality of graphics processors is a preset ratio.
[0029] As some examples, the preset ratio may be 1:(N-1), where N is the total number of graphics processors. That is, one of the multiple graphics processors is used as the second graphics processor, and the remaining graphics processors are used as the first graphics processor.
[0030] As some other examples, the preset ratio may also be 1: 7. For example, assuming that the electronic device includes 8 graphics processors, 1 graphics processor is the second graphics processor, and 7 graphics processors are the first graphics processors.
[0031] Of course, in the embodiment of the present disclosure, the size of the preset ratio can also be divided according to the real-time performance of the electronic device in processing the service request. In other words, the size of the preset ratio can be dynamically adjusted according to the real-time performance of multiple graphics processors.
[0032] The reasoning process of the large language model includes the Prefill stage and the decoding stage. In the Prefill stage, the large language model calculates all inputs (input ids) in the service request and generates corresponding KV caches and tokens. In the decoding stage, based on the input ids and the tokens and KV cache generated in the Prefill stage, it decodes and generates subsequent tokens.
[0033] That is, a portion of all graphics processors deployed in the electronic device can be divided into a first graphics processor, and another portion of the graphics processors can be divided into a second graphics processor. The first graphics processor is used to perform the prefill phase of the large language model for the service request, and the second graphics processor is used to perform the decoding phase of the large language model for the service request.
[0034] The electronic device receives a service request sent by a client, and schedules the service request to a first graphics processor, performs pre-fill processing on the service request through a large language model deployed on the first graphics processor, and obtains a pre-fill result corresponding to the service request.
[0035] The prefill result may include the input ids, KV cache, and token of the service request generated by the large language model in the Prefill phase.
[0036] After the large language model deployed on the first graphics processor obtains the pre-filled result corresponding to the service request, the pre-filled result is sent to the second graphics processor of the electronic device.
[0037] It is worth noting that, since the large language model deployed on the first GPU only performs reasoning in the Prefill stage, the resources of the first GPU are all used by the large language model to perform reasoning in the Prefill stage.
[0038] In step 130 , the pre-filled result is decoded by the large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request.
[0039] Here, the large language model deployed on the second graphics processor receives the pre-filled result output by the large language model deployed on the first graphics processor, performs decoding processing on the pre-filled result in the decoding phase, and obtains the inference result corresponding to the service request.
[0040] The reasoning result corresponding to the service request can obtain different reasoning results according to different reasoning tasks. For example, for a text translation task, the reasoning result of the service request can be to translate a text in one language into another language. For a text generation task, the reasoning result of the service request can be to generate coherent and relevant text content according to the prompt or context given by the service request.
[0041] It is worth noting that the model weight of the large language model deployed on the second graphics processor is the same as the model weight of the large language model deployed on the first graphics processor. That is to say, on each electronic device, the large language model deployed on the first graphics processor and the second graphics processor is a large language model with the same model weight, so as to process the service request input to the large language model.
[0042] It should be understood that in the Prefill stage, the large language model needs to calculate the KV cache and self-attention for all input ids of the service request. Therefore, the Prefill stage is a computationally intensive stage with a large amount of calculation. In the decoding stage, the large language model has already calculated the KV cache in the Prefill stage, and the decoding stage uses an autoregressive method for decoding. The length of each input sequence is 1, so the decoding stage of the large language model has a smaller amount of calculation than the Prefill stage.
[0043] By dividing the graphics processor deployed in the electronic device into a first graphics processor and a second graphics processor, the original serial Prefill stage and decoding stage of the large language model can be separated. In other words, in view of the different dependence of the Prefill stage and the decoding stage on GPU resources, by using a first graphics processor dedicated to performing prefill processing and a second graphics processor dedicated to performing decoding processing, it can be ensured that GPU resources can be fully utilized.
[0044] Therefore, by obtaining a service request sent to a large language model, and pre-filling the service request through the large language model deployed on the first graphics processor, obtaining a pre-filling result corresponding to the service request, and sending the pre-filling result to the second graphics processor, and decoding the pre-filling result through the large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request, the pre-filling stage and the decoding stage of the large language model can be decoupled, thereby improving the utilization of GPU resources and improving the service throughput of the large language model.
[0045] In some feasible implementations, when the pre-filled results accumulated by the second graphics processor meet preset conditions, all pre-filled results accumulated by the second graphics processor can be decoded using a large language model deployed on the second graphics processor to obtain inference results corresponding to each service request.
[0046] Here, after the pre-filled results corresponding to the service requests enter the second graphics processor, they will first be in a waiting state. When the pre-filled results accumulated by the second graphics processor meet the preset conditions, the large language model deployed on the second graphics processor will be used to decode and process the pre-filled results corresponding to all service requests accumulated by the second graphics processor to obtain the inference results corresponding to each service request.
[0047] That is, after the pre-filled result of the received service request meets the preset condition, the second graphics processor uniformly decodes all the accumulated pre-filled results of the service request through the large language model deployed on the second graphics processor.
[0048] Decoding all pre-filled results accumulated by the second GPU is actually batching the pre-filled results of the service request. Batching is to merge multiple service requests into one or several larger service requests to improve the reasoning efficiency of the large language model. By optimizing the batch size, hardware resources can be effectively utilized, the average computing cost of each service request can be reduced, and the overall reasoning efficiency of the large language model can be improved.
[0049] It should be noted that in the decoding phase, the amount of computation in the decoding phase is smaller than that in the Prefill phase, and the model memory IO (memory input / output) in the decoding phase takes up more time. By accumulating a certain number of prefill results of service requests before decoding, the throughput of large language models can be improved.
[0050] In some embodiments, when the number of pre-fill results received by the second graphics processor reaches a target number, it is determined that the pre-fill results accumulated by the second graphics processor meet a preset condition.
[0051] The target number can be understood as the size of the batch, and the size of the target number can be set according to the actual situation. Of course, the target number refers to the maximum batch size, that is, the max batch size. If the pre-filled results of the service requests received by the large language model deployed on the second graphics processor do not reach the target number after the preset time interval, the pre-filled results of all accumulated service requests can also be decoded.
[0052] After the second graphics processor receives the pre-filled results of the service requests, the second graphics processor is in a waiting state. When the number of pre-filled results of the received service requests reaches the target number, the large language model deployed on the second graphics processor decodes and processes the pre-filled results of all received service requests.
[0053] In some other embodiments, when the time interval between two consecutive decoding processes performed by the second graphics processor reaches the target duration, it is determined that the pre-filled result accumulated by the second graphics processor meets a preset condition.
[0054] The time interval between two consecutive decoding processes performed by the second graphics processor refers to the time interval between the current decoding process and the last decoding process. The time interval between two consecutive decoding processes performed by the second graphics processor reaches the target duration, which can be understood as the second graphics processor starting to perform interface processing on the pre-filled results of all service requests accumulated for the target duration after accumulating the pre-filled results of the service requests for the target duration.
[0055] It should be understood that the size of the target duration can be set according to actual conditions. In some examples, the target duration can be determined according to the number of service requests received by the large language model in real time.
[0056] After the second graphics processor receives the pre-filled result of the service request, the second graphics processor is in a waiting state. When the time interval between the current time and the last time the second graphics processor performed decoding processing reaches the target duration, the large language model deployed on the second graphics processor decodes the pre-filled results of all service requests received within the target duration.
[0057] Therefore, through the above implementation, in the decoding stage, the large language model deployed on the second graphics processor can accumulate service requests of a certain batch size, and then decode all the accumulated service requests, thereby improving the service throughput of the large language model.
[0058] For example, assuming that the electronic device has eight graphics processors, the electronic device receives a service request, and the batch size of the decoding stage on each graphics processor is 1, then the CPU resources of the electronic device cannot be effectively utilized.
[0059] Assuming that the Prefill phase takes 1 second and the decoding phase takes 1 second, the electronic device can process 8 service requests within 2 seconds. By using the request processing method of the large language model provided by the embodiment of the present disclosure, 7 of the 8 graphics processors are set as the first graphics processor, and 1 graphics processor is set as the second graphics processor, then all the first graphics processors can process the Prefill phase of 14 service requests within 2 seconds, and the second graphics processor can process 14 decoding phases within 1-2 seconds. Since increasing the batch size in the decoding phase does not bring obvious processing delays, the electronic device can process 14 service requests within 2 seconds, and the throughput of the large language model is increased by nearly 75%.
[0060] Figure 2 FIG. 1 is a flowchart of a method for processing a request of a large language model according to some other embodiments. Figure 2 As shown, in some practicable implementations, the request processing method for a large language model may include the following steps.
[0061] In step 210 , a service request distributed by a scheduler to a first graphics processor is received.
[0062] Here, the service request is a service request sent by the client to the large language model. The scheduler is used to receive the service request sent by the client to request the large language model service, and distribute the service request to the corresponding first graphics processor. It should be noted that the scheduler can actually be understood as a middleware located between the client and the graphics processor.
[0063] The scheduler may distribute the received service request to the corresponding first graphics processor according to a preset rule.
[0064] It should be understood that, in an electronic device, multiple first graphics processors may be deployed, and after receiving a service request sent by a client, the scheduler distributes the service request to the corresponding first graphics processor according to a preset rule.
[0065] Exemplarily, the preset rule may be random distribution, that is, the scheduler randomly distributes the service request to one of the plurality of first graphics processors.
[0066] In some embodiments, the scheduler is used to distribute the service request to the corresponding first graphics processor according to the idle state of each first graphics processor.
[0067] The idle state of the first graphics processor may refer to the number of service requests currently processed by the first graphics processor. Accordingly, distributing the service requests to the corresponding first graphics processors according to the idle state of each first graphics processor may refer to the scheduler distributing the received service requests to the first graphics processor with the least number of service requests currently processed.
[0068] In step 220, the service request is pre-filled using the large language model deployed on the first graphics processor to obtain a pre-filled result corresponding to the service request, and the pre-filled result is sent to the second graphics processor.
[0069] Here, the first graphics processor receives the service request distributed by the scheduler, and performs prefill processing on the service request in the Prefill phase through the large language model deployed on the first graphics processor.
[0070] It is worth noting that the detailed description of step 220 can refer to the relevant description of the above embodiment, which will not be repeated here.
[0071] In step 230, the pre-filled result is decoded by the large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request.
[0072] Here, for the detailed description of step 230, reference may be made to the relevant description of the above embodiment, which will not be repeated here.
[0073] In step 240 , the inference result is sent to the scheduler through the second graphics processor, so that the scheduler receives the inference result and returns the inference result to the corresponding client.
[0074] Here, after obtaining the inference result corresponding to the service request, the second graphics processor sends the inference result output by the large language model to the scheduler. The scheduler receives the inference result sent by the second graphics processor and returns the inference result to the corresponding client.
[0075] It is worth noting that, in the disclosed embodiment, each service request may have its corresponding service identifier (ID) to globally uniquely identify the service request. In the case where the second graphics processor uses batch processing to decode the pre-filled results of the service request, the number of inference results received by the scheduler may be multiple. Through the service identifier, the scheduler may return the inference result corresponding to the service identifier to the client corresponding to the service identifier.
[0076] It should be understood that through the scheduler, the distribution logic of service requests is decoupled from the calculation logic of the large language model, which can not only better distribute service requests, but also enable the request processing method of the large language model provided in the embodiment of the present disclosure to be integrated with other reasoning optimization strategies.
[0077] Therefore, through the above steps 210 to 240, not only can the request processing method for the large language model provided in the embodiment of the present disclosure be integrated with other inference optimization strategies, but the pre-filling stage and the decoding stage of the large language model can also be decoupled, thereby improving the utilization rate of GPU resources and improving the service throughput of the large language model.
[0078] Figure 3 FIG. 1 is a schematic diagram of a scheduler according to some embodiments. Figure 3 As shown, the scheduler receives a service request sent by a client, and distributes the service request to the corresponding first graphics processor according to preset rules. The first graphics processor pre-fills the service request to obtain a pre-fill result. The first graphics processor passes the pre-fill result to the second graphics processor. The second graphics processor decodes the pre-fill result to obtain an inference result of the service request, and sends the inference result of the service request to the scheduler. The scheduler returns the inference result of the service request to the corresponding client.
[0079] It is worth noting that the scheduler can be deployed on the electronic device or outside the electronic device. For example, when a large language model service is deployed in a cluster, a large language model service is deployed on each electronic device, and the scheduler can be used to distribute service requests to the first graphics processor of each electronic device in the cluster.
[0080] Figure 4 FIG. 1 is a schematic diagram showing module connections of a request processing device for a large language model according to some embodiments. Figure 4 As shown, a request processing device 400 for a large language model is provided, and the request processing device 400 for a large language model may include: An acquisition module 401 is configured to acquire a service request sent to a large language model; The first processing module 402 is configured to perform pre-filling processing on the service request through the large language model deployed on the first graphics processor, obtain a pre-filling result corresponding to the service request, and send the pre-filling result to the second graphics processor; The second processing module 403 is configured to decode the pre-filled result through the large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request.
[0081] Optionally, the second processing module 403 is specifically configured to: When the pre-filled results accumulated by the second graphics processor meet preset conditions, all the pre-filled results accumulated by the second graphics processor are decoded by a large language model deployed on the second graphics processor to obtain inference results corresponding to each of the service requests.
[0082] Optionally, the large language model request processing device 400 further includes: A first determining module is configured to determine that the pre-filling results accumulated by the second graphics processor meet the preset condition when the number of the pre-filling results received by the second graphics processor reaches a target number; or The second determination module is configured to determine that the pre-filling result accumulated by the second graphics processor meets the preset condition when a time interval between two consecutive decoding processes performed by the second graphics processor reaches a target duration.
[0083] Optionally, the acquisition module 401 is specifically configured to: receiving a service request distributed by a scheduler to the first graphics processor, wherein the service request is a service request sent by a client to the large language model; The large language model request processing device 400 further includes: The sending module is configured to send the inference result to the scheduler through the second graphics processor, so as to receive the inference result through the scheduler and return the inference result to the corresponding client.
[0084] Optionally, the scheduler is used to distribute the service request to the corresponding first graphics processor according to the idle state of each first graphics processor.
[0085] Optionally, the large language model request processing device 400 is configured in an electronic device, and the electronic device includes a plurality of graphics processors, and the ratio of the number of the second graphics processor to the first graphics processor in the plurality of graphics processors is a preset ratio.
[0086] Regarding the request processing device 400 for a large language model in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0087] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device (eg, a server) 500 suitable for implementing the embodiments of the present disclosure. It should be noted that, Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0088] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0089] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0090] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0091] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0092] In some embodiments, the client, electronic device can communicate using any currently known or future developed network protocol such as HTTP (HyperText TransferProtocol), and can be interconnected with any form or medium of digital data communication (e.g., communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
[0093] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0094] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a service request sent to a large language model; pre-fills the service request through the large language model deployed on the first graphics processor, obtains a pre-fill result corresponding to the service request, and sends the pre-fill result to the second graphics processor; decodes the pre-fill result through the large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request.
[0095] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0096] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0097] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a module does not, in some cases, limit the module itself.
[0098] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0099] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0100] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0101] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0102] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Claims
1. A method for processing requests for a large language model, characterized in that: include: Get the service request sent to the large language model; Pre-filling the service request by using a large language model deployed on the first graphics processor, obtaining a pre-filling result corresponding to the service request, and sending the pre-filling result to the second graphics processor; The pre-filled result is decoded by a large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request.
2. The method according to claim 1, characterized in that: The decoding process of the pre-filled result by the large language model deployed on the second graphics processor to obtain the inference result corresponding to the service request includes: When the pre-filled results accumulated by the second graphics processor meet preset conditions, all the pre-filled results accumulated by the second graphics processor are decoded by a large language model deployed on the second graphics processor to obtain inference results corresponding to each of the service requests.
3. The method according to claim 2, characterized in that The method further comprises: When the number of the pre-filling results received by the second graphics processor reaches the target number, determining that the pre-filling results accumulated by the second graphics processor meet the preset condition; or When the time interval between two consecutive decoding processes performed by the second graphics processor reaches the target time length, it is determined that the pre-filling result accumulated by the second graphics processor meets the preset condition.
4. The method according to any one of claims 1 to 3, characterized in that The obtaining of the service request sent to the large language model includes: receiving a service request distributed by a scheduler to the first graphics processor, wherein the service request is a service request sent by a client to the large language model; After decoding the pre-filled result by the large language model deployed on the second graphics processor to obtain the inference result corresponding to the service request, the method further includes: The inference result is sent to the scheduler through the second graphics processor, so that the scheduler receives the inference result and returns the inference result to the corresponding client.
5. The method according to claim 4, characterized in that The scheduler is used to distribute the service request to the corresponding first graphics processor according to the idle state of each first graphics processor.
6. The method according to any one of claims 1 to 3, characterized in that The large language model request processing method is executed by an electronic device, wherein the electronic device includes a plurality of graphics processors, and the ratio of the number of the second graphics processor to the number of the first graphics processor in the plurality of graphics processors is a preset ratio.
7. A request processing device for a large language model, characterized in that: include: an acquisition module, configured to acquire a service request sent to the large language model; A first processing module is configured to perform pre-filling processing on the service request through a large language model deployed on the first graphics processor, obtain a pre-filling result corresponding to the service request, and send the pre-filling result to the second graphics processor; The second processing module is configured to decode the pre-filled result through the large language model deployed on the second graphics processor to obtain an inference result corresponding to the service request.
8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 6 are implemented.
9. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Cited By
Storage and calculation integration-based big language model reasoning device, reasoning system and electronic equipment
CN119337953A
Large language model inference device, inference system and electronic device based on in-memory computing
CN119337953B
Inference system and method for large model
CN120144300A
Scheduling method and device for large language model reasoning system with multiple processing units
CN120450049A
Reply content prediction method, and apparatus, device, medium, product and chip
WO2025236856A1
