Inference method and system, computer equipment and storage medium

Abstract

The invention discloses an inference method and system, computer equipment and a storage medium. The method is applied to a target system comprising a central processing unit (CPU), a graphic processing unit (GPU) and a field programmable gate array (FPGA), and comprises the following steps: in response to a request of a target reasoning service, the CPU transmits a target input of the target reasoning service to the GPU; the GPU performs calculation of a pre-filling stage on target input based on a network structure and weight parameters of the large language model to obtain first data including an intermediate calculation result and a final calculation result of the pre-filling stage, and transmits the first data to the CPU; the CPU transmits the first data to the FPGA; the FPGA performs calculation in a decoding stage based on a network structure of a large language model, a weight parameter and the first data to obtain second data and transmits the second data to the CPU; and the CPU takes the second data as the output of the target reasoning service. According to the method, the target reasoning service can be deployed to the GPU and the FPGA in stages, the target reasoning service is completed in a cooperative operation mode, and hardware resources of the GPU and the FPGA are fully utilized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a reasoning method, system, computer equipment and storage medium. Background Art

[0002] In the reasoning service of a large language model based on the Transformer algorithm, the Transformer algorithm is mainly divided into the calculation of the prefill stage and the calculation of the decode stage. Among them, the calculation of the prefill stage is used to extract information from the input text of the reasoning service, and the calculation of the decoding stage is used to generate the output text of the reasoning service. The calculation characteristics of the prefill stage and the decoding stage are different. The prefill stage requires greater computing power and high computing throughput, and the decoding stage requires high storage. Therefore, it is difficult to fully utilize the computing resources and storage resources of the existing platform (such as a graphics processing unit GPU or a field programmable gate array FPGA) when deploying the reasoning service of the large language model to the existing platform (such as a graphics processing unit GPU or a field programmable gate array FPGA) for calculation. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention provides an inference method, system, computer device and storage medium, which can deploy the target inference service to the GPU and FPGA in stages, complete the target inference service through collaborative operation, and make full use of the hardware resources of the GPU and FPGA.

[0004] In a first aspect, an embodiment of the present invention provides an inference method, which is applied to a target system, wherein the target system includes a central processing unit CPU, a graphics processing unit GPU, and a field programmable gate array FPGA, wherein the CPU and the GPU communicate via a first interface, and the CPU and the FPGA communicate via a second interface, comprising the following steps:

[0005] S1: In response to a request for a target reasoning service of a large language model, the CPU transmits a target input of the target reasoning service to the GPU through a first interface;

[0006] S2: The GPU performs a pre-filling phase calculation on the target input based on the network structure and weight parameters of the large language model to obtain first data, where the first data includes an intermediate calculation result of the pre-filling phase and a final calculation result of the pre-filling phase, and the GPU transmits the first data to the CPU through the first interface;

[0007] S3: The CPU transmits the first data to the FPGA through the second interface;

[0008] S4: FPGA performs calculations in the decoding stage based on the network structure, weight parameters and first data of the large language model to obtain second data, which is the final calculation result of the decoding stage. FPGA transmits the second data to the CPU through the second interface.

[0009] S5: The CPU uses the second data as the output of the target reasoning service.

[0010] Furthermore, the target system includes multiple GPUs and multiple FPGAs, and the inference method also includes:

[0011] The CPU obtains a mapping relationship, where the mapping relationship includes a plurality of input-output pairs, wherein each input-output pair includes sample input data and sample output data of any sample inference service;

[0012] The CPU predicts the task load and inference time of the target inference service based on the mapping relationship;

[0013] The CPU determines, based on the task load, the inference time, and the preset scheduling strategy, to schedule the computing tasks of the pre-filling stage to at least one GPU among the multiple GPUs, and to schedule the computing tasks of the decoding stage to at least one FPGA among the multiple FPGAs.

[0014] Furthermore, the CPU obtains the mapping relationship including:

[0015] The CPU obtains sample input data and sample output data of multiple sample reasoning services;

[0016] The CPU determines an input-output pair for sample input data and sample output data of each sample inference service;

[0017] The CPU determines a mapping relationship based on the input-output pairs corresponding to each of the multiple sample inference services.

[0018] Furthermore, the CPU predicts the task load and inference time of the target inference service based on the mapping relationship, including:

[0019] The CPU searches for a target input-output pair corresponding to the target reasoning service in the mapping relationship based on the target input of the target reasoning service;

[0020] The CPU determines the length of output data of the target reasoning service based on the target input-output pair;

[0021] The CPU predicts the task load and inference time based on the length of the output data.

[0022] Furthermore, the calculation of the pre-filling stage is completed by N first decoder blocks, and the intermediate calculation results include N groups of intermediate calculation results. After a first decoder block completes the calculation, one group of the N groups of intermediate calculation results is obtained, and N is an integer greater than or equal to 1.

[0023] Furthermore, the calculation of the decoding stage is completed by the N second decoder blocks, and the FPGA performs the calculation of the decoding stage based on the network structure, weight parameters and first data of the large language model, including:

[0024] Based on the network structure and weight parameters of the large language model, the FPGA uses the final calculation result of the pre-filling stage as the input for the calculation of the first decoder block in the decoding stage;

[0025] For the i-th second decoder block among the N second decoder blocks, the i-th second decoder block is calculated based on the i-th group of intermediate calculation results, and the i-th group of intermediate calculation results is the calculation result of the i-th first decoder block corresponding to the i-th second decoder block, 2≤i≤N.

[0026] Further, each of the N groups of intermediate calculation results includes a key and a value matrix.

[0027] Furthermore, the first interface is a PCIe interface.

[0028] Furthermore, the second interface is a PCIe interface or an interface capable of connecting to a network cable.

[0029] In an embodiment of the present invention, in a second aspect, the embodiment of the present invention provides an inference system, including:

[0030] The CPU is configured to respond to a request of a target reasoning service of the large language model and transmit a target input of the target reasoning service to the GPU through a first interface;

[0031] The GPU is used to perform a pre-filling phase calculation on the target input based on the network structure and weight parameters of the large language model to obtain first data, where the first data includes an intermediate calculation result of the pre-filling phase and a final calculation result of the pre-filling phase, and transmit the first data to the CPU through a first interface;

[0032] The CPU is further used to transmit the first data to the FPGA through the second interface;

[0033] The FPGA is used to perform calculations in the decoding stage based on the network structure of the large language model, the weight parameters and the first data to obtain second data, which is the final calculation result of the decoding stage, and transmit the second data to the CPU through the second interface;

[0034] The CPU is further configured to use the second data as an output of the target reasoning service.

[0035] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned reasoning method when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned reasoning method are implemented.

[0037] Other optional features and technical effects of the embodiments of the present invention are partially described below, and partially can be understood by reading this document.

[0038] Compared with the prior art, the present invention has the following beneficial technical effects:

[0039] The present invention provides a reasoning method, system, computer device and storage medium, which are applied to a target system including a GPU and an FPGA, and the pre-filling stage and the decoding stage of the target reasoning service of a large language model are respectively deployed to the GPU and the FPGA for collaborative computing through the CPU. During the collaborative computing process, the GPU can first store the intermediate calculation results and the final calculation results of the pre-filling stage locally on the GPU, and then transmit the intermediate calculation results and the final calculation results to the FPGA through the CPU, so that the FPGA can continue to perform the calculation of the decoding stage. This collaborative computing process utilizes the centralized scheduling capability of the CPU, and deploys the computing stages with different requirements for computing resources and storage resources on demand to a processing platform with corresponding computing power and storage capacity, thereby improving the reasoning performance while making full use of the hardware resources of the GPU and FPGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The elements shown are not limited to the proportions shown in the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements, wherein:

[0041] Figure 1 is a flow chart of the reasoning method according to the present invention;

[0042] Figure 2 is a schematic diagram of a target system according to the present invention;

[0043] Figure 3 A schematic diagram of another target system according to the present invention;

[0044] Figure 4 A schematic diagram of another target system according to the present invention;

[0045] Figure 5A schematic diagram of a data flow for performing target reasoning services for a large language model according to the present invention;

[0046] Figure 6 is an inference device according to the present invention.

[0047] In the figure: 1000, electronic device; 1001, processor; 1002, read-only memory; 1003, memory; 1004, bus; 1005, I / O interface; 1006, input part; 1007, output part; 1008, storage part; 1009, communication part; 1010, drive; 1011, removable medium. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and drawings. Here, the exemplary implementation methods of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0049] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0050] Currently, in the process of performing target reasoning services on a large language model, all computing stages (such as the pre-filling stage and the decoding stage) of the target reasoning service can be completed on a GPU (graphics processing unit). The hardware structure of the GPU can provide more computing resources. The main computing tasks in the pre-filling stage are composed of matrix-matrix multiplication (MM), and the computing density is relatively high. Therefore, the high computing throughput on the GPU hardware can be used when the GPU completes the pre-filling stage. However, the main computing tasks in the decoding stage are composed of matrix-vector multiplication (MV), and the computing density is relatively low. Therefore, the computing resource utilization of the GPU decreases in the decoding stage. In addition, there is a relatively large power consumption overhead when the GPU completes the target reasoning service.

[0051] If all the computing stages (such as pre-filling and decoding stages) of the target reasoning service are completed in FPGA (field programmable gate array), the hardware architecture of FPGA, such as high bandwidth memory (HBM), has abundant storage resources, which can guarantee the high memory access volume in the decoding stage. However, the computing resources and frequency of FPGA are limited, so there will be a computing bottleneck in the pre-filling stage, which limits the overall reasoning speed of the target reasoning service.

[0052] In summary, when the inference service of a large language model is deployed on only a single hardware (GPU or FPGA), the calculation and storage of the pre-filling stage and the decoding stage are performed by the corresponding resources on the same hardware. Since the calculations in the pre-filling stage and the decoding stage have their own characteristics, it is difficult to fully utilize the computing and memory access capabilities of a single hardware, and it is impossible to further improve the performance of the inference service.

[0053] In view of this, in an embodiment of the present invention, a reasoning method P100 is provided, such as Figure 1 As shown, the following steps are included:

[0054] S1: In response to a request for a target reasoning service of a large language model, the CPU transmits a target input of the target reasoning service to the GPU through a first interface.

[0055] It can be understood that the reasoning method P100 can be applied to the target system, the target reasoning service is, for example, any reasoning service initiated by the user, and the target input is text data, image data or other format data input by the user to the large language model in the target reasoning service. The first interface is, for example, a PCIe (Peripheral Component Interconnect express) interface.

[0056] For example, the target system is Figure 2 As shown, it includes a CPU, a GPU and an FPGA, the CPU and the GPU communicate through a first interface, and the CPU and the FPGA communicate through a second interface. The CPU can assign the pre-filling stage of the target reasoning task to the GPU and the decoding stage to the FPGA.

[0057] For example, the target system is Figure 3 As shown, it includes a CPU, a GPU and n FPGAs, where n is an integer greater than 1, the CPU and the GPU communicate through a first interface, and the CPU and the FPGA communicate through a second interface. The CPU can assign the pre-filling stage of the target inference task to the GPU and the decoding stage to the n FPGAs.

[0058] For example, the target system is Figure 4 As shown, it includes a CPU, m GPUs and n FPGAs, where n and m are integers greater than 1, the CPU and the GPU communicate through a first interface, and the CPU and the FPGA communicate through a second interface. The CPU can assign the pre-filling stage of the target inference task to the m GPUs and the decoding stage to the n FPGAs.

[0059] S2: The GPU performs a pre-filling phase calculation on the target input based on the network structure and weight parameters of the large language model to obtain first data, where the first data includes intermediate calculation results of the pre-filling phase and final calculation results of the pre-filling phase. The GPU transmits the first data to the CPU through the first interface.

[0060] For example, the network structure (e.g., computational graph, etc.) and weight parameters of the large language model can be pre-stored in the GPU, and the network structure can be understood as an instruction to instruct the GPU to use the weight parameters in a certain order when performing calculations in the pre-filling phase. Figure 2 As shown, after the GPU completes the calculation in the pre-filling phase, the first data is transmitted to the CPU. Alternatively, if there is a communication interface between the GPU and the FPGA, after the GPU completes the calculation in the pre-filling phase, the first data can be directly transmitted to the FPGA.

[0061] Exemplarily, calculations in the pre-filling phase are performed by N first decoder blocks, and intermediate calculation results include N groups of intermediate calculation results. After a first decoder block completes the calculation, one group of the N groups of intermediate calculation results is obtained, where N is an integer greater than or equal to 1.

[0062] For example, Figure 2 Taking the target system of the example as an example, Figure 5The data flow in the calculation process of GPU and FPGA is illustrated. The calculation of the pre-filling stage is completed by N first decoder blocks, and the N first decoder blocks include: first decoder block 0, first decoder block 1, ..., first decoder block N-1. Each first decoder block includes three submodules, namely self-attention, linear, and feed-forward network. Taking the first decoder block 0 as an example, during the calculation process, the input of the first decoder block 0 is the target input in S1. After the calculation is completed, the first group of intermediate calculation results will be generated. The first group of intermediate calculation results includes Key and Value matrices. After the first decoder block 0 completes the calculation, the generated Key and Value matrices are stored in the GPU in the form of KV cache. The calculation output of the first decoder block 0 is used as the input of the first decoder block 1. After the calculation of the first decoder block 1 is completed, the second group of intermediate calculation results will be generated. Similarly, after the calculation of the first decoder block N-1 is completed, the Nth group of intermediate calculation results will be generated. The contents and storage forms of the second to Nth groups of intermediate calculation results are as described above for the first group of intermediate calculation results, and will not be described in detail again.

[0063] That is, in the pre-filling stage, the first data that needs to be stored and transmitted to the CPU includes N groups of intermediate calculation results and the final calculation result of the pre-filling stage calculated by the first decoder block N-1.

[0064] S3: The CPU transmits the first data to the FPGA through the second interface.

[0065] Exemplarily, the second interface is a PCIe interface or an interface capable of connecting to a network cable.

[0066] S4: FPGA performs calculations in the decoding stage based on the network structure, weight parameters and first data of the large language model to obtain second data, which is the final calculation result of the decoding stage. FPGA transmits the second data to the CPU through the second interface.

[0067] Exemplarily, the network structure and weight parameters of the large language model can be pre-stored in the FPGA, and the network structure can be understood as an instruction that instructs the FPGA to use the weight parameters in a certain order when performing calculations in the decoding stage.

[0068] Exemplarily, the calculation of the decoding stage is completed by N second decoder blocks, wherein the N second decoder blocks correspond one to one to the N first decoder blocks. Based on the network structure and weight parameters of the large language model, the FPGA uses the final calculation result of the pre-filling stage as the input of the calculation of the first decoder block in the decoding stage; for the i-th second decoder block among the N second decoder blocks, the calculation of the i-th second decoder block is performed based on the i-th group of intermediate calculation results, and the i-th group of intermediate calculation results is the calculation result of the i-th first decoder block corresponding to the i-th second decoder block, 2≤i≤N.

[0069] For example, after the CPU transmits the first data to the FPGA, the first data is first stored locally in the FPGA. After the FPGA receives the first data, it triggers the FPGA to start the calculation in the decoding stage. Figure 5 As shown, the calculation of the decoding stage is completed by N second decoder blocks, and the N second decoder blocks include: second decoder block 0, second decoder block 1, ..., second decoder block N-1. Each second decoder block may include 3 submodules, namely, self-attention, linear, and feed-forward network. After the second decoder block 0 starts to calculate, the final calculation result of the pre-filling stage is obtained from the stored first data as the input of the second decoder block 0, and in the calculation process, it is also necessary to obtain the first group of intermediate calculation results generated by the first decoder block 0 in the pre-filling stage from the first data to participate in the calculation of the second decoder block 0. After the calculation of the second decoder block 0 is completed, the calculation output of the second decoder block 0 is used as the input of the second decoder block 1, and in the calculation process of the second decoder block 1, it is also necessary to obtain the second group of intermediate calculation results generated by the first decoder block 1 in the pre-filling stage from the first data to participate in the calculation of the second decoder block 1. By analogy, in the calculation process of the second decoder block N-1, it is also necessary to obtain the Nth group of intermediate calculation results generated by the first decoder block N-1 in the pre-filling stage from the first data to participate in the calculation of the second decoder block N-1. After the calculation of the second decoder block N-1 is completed, the final calculation result of the decoding stage, that is, the second data, is obtained.

[0070] S5: The CPU uses the second data as the output of the target reasoning service.

[0071] Exemplarily, the output of the target reasoning service is the conclusion (in the form of text, document, code, etc.) given by the large language model to the text data, image data, or data in other formats input by the user. It should be understood that the decoding stage can be iterated multiple times to obtain an output including multiple tokens. Before outputting the final result to the user, the large language model can also perform various necessary processing on the second data, such as combination, format adjustment, screening, etc.

[0072] In some embodiments, the target system, for example Figure 4 As shown, multiple GPUs and multiple FPGAs are included, and the reasoning method also includes:

[0073] The CPU obtains a mapping relationship, where the mapping relationship includes a plurality of input-output pairs, wherein each input-output pair includes sample input data and sample output data of any sample inference service;

[0074] Exemplarily, the CPU obtains sample input data and sample output data of multiple sample reasoning services, determines an input-output pair for the sample input data and sample output data of each sample reasoning service, and determines a mapping relationship based on the input-output pairs corresponding to each of the multiple sample reasoning services.

[0075] For example, a service module is set up on the CPU host side. The service module can record the sample input data and sample output data of the completed sample reasoning service. The sample reasoning service is any reasoning service initiated by the user. The sample input data is the data input by the user for the sample reasoning service. The sample output data is the output data calculated by the large language model based on the sample input data. The service module can also record the task load of the GPU and FPGA during the calculation process of the sample reasoning service or the target reasoning service.

[0076] A load predictor can be set in the service module. The load predictor can obtain sample input data and sample output data corresponding to multiple sample reasoning services, and use the sample input data and sample output data corresponding to one of the sample reasoning services as a training sample. Based on the multiple training samples formed in this way, the load predictor can predict the number of tokens of the output data through the input data of any reasoning service. After training, a mapping relationship table can be generated, and the mapping relationship table includes multiple input-output pairs. The load predictor can be implemented through algorithms such as regression analysis and machine learning.

[0077] In some embodiments, the CPU predicts the task load and reasoning time of the target reasoning service based on the mapping relationship. It should be understood that the prediction can be based on a deterministic algorithm or a non-deterministic algorithm (eg, an algorithm based on probability statistics, etc.).

[0078] Exemplarily, the CPU searches for a target input-output pair corresponding to the target reasoning service in a mapping relationship based on the target input of the target reasoning service, the CPU determines the length of the output data of the target reasoning service based on the target input-output pair, and the CPU predicts the task load and reasoning time based on the length of the output data.

[0079] For example, the load predictor can predict the number of tokens output by the target reasoning service based on the target input of the target reasoning service, thereby predicting the task load and reasoning time corresponding to the target reasoning service based on the number of tokens output. The task load can be understood as the workload, that is, the load (e.g., the amount of computation) required to complete the target reasoning service. The reasoning time can be understood as the time required to complete the target reasoning service.

[0080] Specifically, the load predictor can predict the load #1 and time #1 of the pre-filling phase of the target reasoning service, as well as the load #2 and time #2 of the decoding phase based on the target input. The sum of load #1 and load #2 is the task load corresponding to the target reasoning service, and the sum of time #1 and time #2 is the reasoning time corresponding to the target reasoning service.

[0081] In some embodiments, the CPU determines to schedule the computing tasks of the pre-filling stage to at least one GPU among multiple GPUs and to schedule the computing tasks of the decoding stage to at least one FPGA among multiple FPGAs based on the task load, the inference time, and a preset scheduling strategy.

[0082] Exemplarily, the preset scheduling strategy is such as giving priority to the shortest task (e.g., the shortest required reasoning time), or giving priority to the task with the shortest remaining time (e.g., the shortest required reasoning time for the remaining calculation). For example, if there are multiple reasoning services running at the same time, the reasoning service with the shortest reasoning time can be preferentially allocated to the reasoning calculation in the GPU and FPGA. For another example, the reasoning calculation of reasoning service A has already started, then when the CPU allocates other reasoning services and the remaining computing tasks of reasoning service A, it needs to give priority to allocating resources to the remaining computing tasks of reasoning service A, because the reasoning time required for the remaining computing tasks of reasoning service A is shorter than that of other reasoning services.

[0083] For example, Figure 3 As shown, the CPU allocates the pre-filling stage to the GPU, and can determine to allocate the decoding stage to the FPGA that meets the QoS (quality of service) index based on the current load conditions of FPGA 0 to FPGAn-1, as well as load #2 and time #2, such as allocating the decoding stage to a certain FPGA, or dividing the decoding stage into multiple stages, which are respectively allocated to multiple FPGAs in FPGA0 to FPGA n-1. During the calculation process of the decoding stage coordinated by the multiple FPGAs, the data required by each other can be transmitted to each other, or a certain FPGA transmits the data required by other FPGAs for calculation to the CPU, and the CPU is responsible for transmitting it to other FPGAs.

[0084] For example, Figure 4As shown, the CPU determines to allocate the pre-filling stage to the GPU that meets the QoS index based on the current load conditions of GPU 0 to GPU m-1, load #1 and time #1, for example, to allocate the pre-filling stage to a certain GPU, or to divide the pre-filling stage into multiple stages, which are respectively allocated to multiple GPUs in GPU 0 to GPU m-1. And it can determine to allocate the decoding stage to the FPGA that meets the QoS index based on the current load conditions of FPGA 0 to FPGA n-1, as well as load #2 and time #2, for example, to allocate the decoding stage to a certain FPGA, or to divide the decoding stage into several stages, which are respectively allocated to multiple FPGAs in FPGA 0 to FPGA n-1. In the process of the multiple GPUs coordinating to complete the calculation of the pre-filling stage, the data required by each other can be transmitted to each other, or a GPU transmits the data required for the calculation of other GPUs to the CPU, and the CPU is responsible for transmitting it to other GPUs.

[0085] In summary, the reasoning method provided by the present invention is applied to a target system including a GPU and an FPGA, and the pre-filling stage and the decoding stage of the target reasoning service of the large language model are respectively deployed to the GPU and the FPGA for collaborative computing through the CPU. During the collaborative computing process, the GPU can first store the intermediate calculation results and the final calculation results of the pre-filling stage locally on the GPU, and then transmit the intermediate calculation results and the final calculation results to the FPGA through the CPU, so that the FPGA can continue to perform the calculation of the decoding stage. This collaborative computing process utilizes the centralized scheduling capability of the CPU, and deploys the computing stages with different requirements for computing resources and storage resources on demand to the processing platform with corresponding computing power and storage capacity, thereby improving the reasoning performance while making full use of the hardware resources of the GPU and FPGA.

[0086] The present invention also provides an inference system, comprising:

[0087] The CPU is configured to respond to a request of a target reasoning service of the large language model and transmit a target input of the target reasoning service to the GPU through a first interface;

[0088] The GPU is used to perform a pre-filling phase calculation on the target input based on the network structure and weight parameters of the large language model to obtain first data, where the first data includes an intermediate calculation result of the pre-filling phase and a final calculation result of the pre-filling phase, and transmit the first data to the CPU through a first interface;

[0089] The CPU is further used to transmit the first data to the FPGA through the second interface;

[0090] The FPGA is used to perform calculations in the decoding stage based on the network structure of the large language model, the weight parameters and the first data to obtain second data, which is the final calculation result of the decoding stage, and transmit the second data to the CPU through the second interface;

[0091] The CPU is also used to use the second data as the output of the target reasoning service.

[0092] In an embodiment of the present invention, a computer device is provided, including: a processor and a memory storing a computer program, wherein the processor is configured to execute the reasoning method of any embodiment of the present invention when running the computer program.

[0093] Figure 6 A schematic diagram of an electronic device 1000 that can implement a method or implement an embodiment of the present invention is shown, and in some embodiments, more or fewer electronic devices than shown may be included. In some embodiments, it can be implemented using a single or multiple electronic devices. In some embodiments, it can be implemented using cloud or distributed electronic devices.

[0094] like Figure 6 As shown, the electronic device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) 1002 or the programs and / or data loaded from the storage part 1008 to the random access memory (RAM) 1003. Processor 1001 can be a multi-core processor, or it can include multiple processors. In some embodiments, processor 1001 can include a general main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1003, various programs and data required for the operation of electronic device 1000 are also stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0095] Exemplarily, processor 1001 includes a CPU, a GPU, and an FPGA.

[0096] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0097] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed. Figure 6 Only some components are schematically shown in the figure, which does not mean that the computer system 1000 only includes Figure 6 Components shown.

[0098] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0099] Although not shown, in an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores a computer program, and the computer program is configured to execute any reasoning method of the embodiments of the present invention when executed.

[0100] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0101] The methods, programs, systems, devices, etc. of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0102] The present application also relates to a computer program product, which includes executable code, and when the executable code is executed by a computer, the steps of the reasoning method described in the embodiment of the present application are implemented.

[0103] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.

[0104] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0105] In this article, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" are meant to be applicable to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0106] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A reasoning method, characterized in that: The method is applied to a target system, the target system includes a central processing unit (CPU), a graphics processing unit (GPU) and a field programmable gate array (FPGA), the CPU and the GPU communicate via a first interface, and the CPU and the FPGA communicate via a second interface, and the method includes the following steps: S1: In response to a request for a target reasoning service of a large language model, the CPU transmits a target input of the target reasoning service to the GPU through the first interface; S2: the GPU performs a pre-filling phase calculation on the target input based on the network structure and weight parameters of the large language model to obtain first data, where the first data includes an intermediate calculation result of the pre-filling phase and a final calculation result of the pre-filling phase, and the GPU transmits the first data to the CPU through the first interface; S3: the CPU transmits the first data to the FPGA through the second interface; S4: the FPGA performs calculations in a decoding phase based on the network structure, weight parameters and the first data of the large language model to obtain second data, where the second data is the final calculation result of the decoding phase, and the FPGA transmits the second data to the CPU through the second interface; S5: The CPU uses the second data as output of the target reasoning service.

2. The inference method according to claim 1, characterized in that: The target system includes multiple GPUs and multiple FPGAs, and the method further includes: The CPU obtains a mapping relationship, where the mapping relationship includes a plurality of input-output pairs, wherein each input-output pair includes sample input data and sample output data of any sample inference service; The CPU predicts the task load and reasoning time of the target reasoning service based on the mapping relationship; The CPU determines, based on the task load, the inference time, and a preset scheduling strategy, to schedule the computing tasks of the pre-filling stage to at least one GPU among the multiple GPUs, and to schedule the computing tasks of the decoding stage to at least one FPGA among the multiple FPGAs.

3. The inference method according to claim 2, characterized in that: The CPU obtaining the mapping relationship includes: The CPU obtains sample input data and sample output data of multiple sample reasoning services; The CPU determines an input-output pair for sample input data and sample output data of each sample reasoning service; The CPU determines the mapping relationship based on the input-output pairs corresponding to each of the plurality of sample inference services.

4. The inference method according to claim 2, characterized in that: The CPU predicts the task load and reasoning time of the target reasoning service based on the mapping relationship, including: The CPU searches for a target input-output pair corresponding to the target reasoning service in the mapping relationship based on the target input of the target reasoning service; The CPU determines the length of output data of the target reasoning service based on the target input-output pair; The CPU predicts the task load and the inference time based on the length of the output data.

5. The inference method according to claim 1, characterized in that: The calculation of the pre-filling stage is completed by N first decoder blocks, and the intermediate calculation results include N groups of intermediate calculation results. After one of the first decoder blocks completes the calculation, one group of the N groups of intermediate calculation results is obtained, and N is an integer greater than or equal to 1.

6. The inference method according to claim 5, characterized in that: The calculation of the decoding stage is completed by N second decoder blocks, and the FPGA performs the calculation of the decoding stage based on the network structure, weight parameters and the first data of the large language model, including: The FPGA uses the final calculation result of the pre-filling stage as the input of the first decoder block calculation in the decoding stage based on the network structure and weight parameters of the large language model; For the i-th second decoder block among the N second decoder blocks, the i-th second decoder block is calculated based on the i-th group of intermediate calculation results, and the i-th group of intermediate calculation results is the calculation result of the i-th first decoder block corresponding to the i-th second decoder block, where 2≤i≤N.

7. The inference method according to claim 5 or 6, characterized in that: Each of the N groups of intermediate calculation results includes a key and a value matrix.

8. An inference system, characterized in that include: A central processing unit CPU, configured to respond to a request of a target reasoning service of a large language model and transmit a target input of the target reasoning service to the GPU through the first interface; A graphics processing unit GPU is used to perform a pre-filling phase calculation on the target input based on the network structure and weight parameters of the large language model to obtain first data, where the first data includes an intermediate calculation result of the pre-filling phase and a final calculation result of the pre-filling phase, and transmit the first data to the CPU through the first interface; The CPU is further configured to transmit the first data to a field programmable gate array FPGA through the second interface; The FPGA is used to perform calculations in a decoding phase based on the network structure, weight parameters and the first data of the large language model to obtain second data, where the second data is a final calculation result of the decoding phase, and transmit the second data to the CPU through the second interface; The CPU is further configured to use the second data as output of the target reasoning service.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the reasoning method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the reasoning method according to any one of claims 1 to 7 are implemented.

Images