Reasoning method, system, computer device and storage medium

Abstract

The application discloses a reasoning method, system, computer device and storage medium. The method is applied to a target system including a central processing unit (CPU), a graphics processing unit (GPU) and a field programmable gate array (FPGA), and includes the following steps: in response to a request of a target reasoning service, the CPU transmits a target input of the target reasoning service to the GPU; the GPU performs calculation in a pre-padding stage on the target input based on a network structure and weight parameters of a large language model, obtains first data including intermediate calculation results and final calculation results of the pre-padding stage, and transmits the first data to the CPU; the CPU transmits the first data to the FPGA; the FPGA performs calculation in a decoding stage based on the network structure, the weight parameters and the first data of the large language model, obtains second data, and transmits the second data to the CPU; and the CPU takes the second data as output of the target reasoning service. The application can deploy the target reasoning service to the GPU and the FPGA in stages, complete the target reasoning service in a cooperative running mode, and fully utilize hardware resources of the GPU and the FPGA.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an inference method, system, computer equipment and storage medium. Background Art

[0002] In the inference service of a large language model based on the Transformer algorithm, the Transformer algorithm is mainly divided into the calculation of the prefill phase and the calculation of the decode phase. The calculation of the prefill phase is used to extract information from the input text of the inference service, and the calculation of the decode phase is used to generate the output text of the inference service. The calculation characteristics of the prefill phase and the decoding phase are different. The prefill phase requires greater computing power and high computing throughput, while the decoding phase requires high storage. Therefore, it is difficult to fully utilize the computing and storage resources of the existing platform (such as graphics processing units (GPUs) or field-programmable gate arrays (FPGAs)) when deploying the inference service of large language models for calculation. Summary of the Invention

[0003] In response to the problems existing in the prior art, the present invention provides an inference method, system, computer device and storage medium, which can deploy the target inference service to the GPU and FPGA in stages, complete the target inference service through collaborative operation, and make full use of the hardware resources of the GPU and FPGA.

[0004] In a first aspect, an embodiment of the present invention provides an inference method applied to a target system, the target system including a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA), wherein the CPU and the GPU communicate via a first interface, and the CPU and the FPGA communicate via a second interface, including the following steps:

[0005] S1: In response to a request for a target inference service of a large language model, the CPU transmits a target input of the target inference service to the GPU through a first interface;

[0006] S2: The GPU performs a pre-filling calculation on the target input based on the network structure and weight parameters of the large language model to obtain first data. The first data includes an intermediate calculation result of the pre-filling calculation and a final calculation result of the pre-filling calculation. The GPU transmits the first data to the CPU through the first interface.

[0007] S3: The CPU transmits the first data to the FPGA through the second interface;

[0008] S4: The FPGA performs decoding calculations based on the network structure, weight parameters, and first data of the large language model to obtain second data, which is the final calculation result of the decoding stage. The FPGA transmits the second data to the CPU through the second interface.

[0009] S5: The CPU uses the second data as the output of the target reasoning service.

[0010] Furthermore, the target system includes multiple GPUs and multiple FPGAs, and the inference method also includes:

[0011] The CPU obtains a mapping relationship, where the mapping relationship includes multiple input-output pairs, wherein each input-output pair includes sample input data and sample output data of any sample inference service;

[0012] The CPU predicts the task load and inference time of the target inference service based on the mapping relationship;

[0013] Based on the task load, inference time and preset scheduling strategy, the CPU determines to schedule the computing tasks of the pre-filling stage to at least one GPU among the multiple GPUs, and to schedule the computing tasks of the decoding stage to at least one FPGA among the multiple FPGAs.

[0014] Furthermore, the CPU obtains the mapping relationship including:

[0015] The CPU obtains sample input data and sample output data of multiple sample inference services;

[0016] The CPU determines an input-output pair for the sample input data and sample output data of each sample inference service;

[0017] The CPU determines a mapping relationship based on the input and output pairs corresponding to the multiple sample inference services.

[0018] Furthermore, the CPU predicts the task load and inference time of the target inference service based on the mapping relationship, including:

[0019] The CPU searches for a target input-output pair corresponding to the target inference service in the mapping relationship based on the target input of the target inference service;

[0020] The CPU determines the length of output data of the target inference service based on the target input and output pairs;

[0021] The CPU predicts the task load and inference time based on the length of the output data.

[0022] Furthermore, the calculation of the pre-filling stage is completed by N first decoder blocks, and the intermediate calculation results include N groups of intermediate calculation results. After a first decoder block completes the calculation, one group of the N groups of intermediate calculation results is obtained, where N is an integer greater than or equal to 1.

[0023] Furthermore, the calculations in the decoding stage are performed by the N second decoder blocks. The FPGA performs the calculations in the decoding stage based on the network structure, weight parameters, and first data of the large language model, including:

[0024] Based on the network structure and weight parameters of the large language model, the FPGA uses the final calculation results of the pre-filling stage as the input for the calculation of the first decoder block in the decoding stage;

[0025] For the i-th second decoder block among the N second decoder blocks, the i-th second decoder block is calculated based on the i-th group of intermediate calculation results, and the i-th group of intermediate calculation results is the calculation result of the i-th first decoder block corresponding to the i-th second decoder block, 2≤i≤N.

[0026] Furthermore, each of the N groups of intermediate calculation results includes a key and value matrix.

[0027] Furthermore, the first interface is a PCIe interface.

[0028] Furthermore, the second interface is a PCIe interface or an interface capable of connecting to a network cable.

[0029] In a second aspect, an embodiment of the present invention provides an inference system, including:

[0030] The CPU is configured to respond to a request of a target inference service of the large language model and transmit a target input of the target inference service to the GPU through the first interface;

[0031] The GPU is configured to perform a pre-filling calculation on a target input based on the network structure and weight parameters of the large language model to obtain first data, the first data including an intermediate calculation result of the pre-filling calculation and a final calculation result of the pre-filling calculation, and transmit the first data to the CPU via a first interface;

[0032] The CPU is further configured to transmit the first data to the FPGA via the second interface;

[0033] The FPGA is configured to perform calculations in a decoding phase based on the network structure, weight parameters, and first data of the large language model to obtain second data, which is a final calculation result of the decoding phase, and transmit the second data to the CPU via a second interface;

[0034] The CPU is further configured to use the second data as an output of the target inference service.

[0035] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned reasoning method when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned reasoning method are implemented.

[0037] Other optional features and technical effects of the embodiments of the present invention are partially described below, and partially can be understood by reading this document.

[0038] Compared with the prior art, the present invention has the following beneficial technical effects:

[0039] The present invention provides an inference method, system, computer device, and storage medium, which are applied to a target system including a GPU and an FPGA. The pre-filling and decoding stages of a target inference service for a large language model are deployed to the GPU and FPGA, respectively, via a CPU, for collaborative computing. During the collaborative computing process, the GPU can first store the intermediate and final calculation results of the pre-filling stage locally on the GPU. The CPU then transmits these intermediate and final calculation results to the FPGA, allowing the FPGA to continue computing the decoding stage. This collaborative computing process utilizes the CPU's centralized scheduling capabilities to deploy computing stages with different requirements for computing and storage resources on demand to processing platforms with corresponding computing and storage capabilities, thereby improving inference performance while fully utilizing the hardware resources of the GPU and FPGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The elements shown are not limited to the scale shown in the drawings. The same or similar reference numerals in the drawings represent the same or similar elements, wherein:

[0041] Figure 1 is a flow chart of the reasoning method according to the present invention;

[0042] Figure 2 is a schematic diagram of a target system according to the present invention;

[0043] Figure 3 is a schematic diagram of another target system according to the present invention;

[0044] Figure 4 is a schematic diagram of another target system according to the present invention;

[0045] Figure 5A schematic diagram of the data flow for performing target reasoning services on a large language model according to the present invention;

[0046] Figure 6 is an inference device according to the present invention.

[0047] In the figure: 1000, electronic device; 1001, processor; 1002, read-only memory; 1003, memory; 1004, bus; 1005, I / O interface; 1006, input part; 1007, output part; 1008, storage part; 1009, communication part; 1010, drive; 1011, removable medium. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0049] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0050] Currently, in the process of performing target reasoning services on a large language model, all computing stages of the target reasoning service (such as the pre-filling stage and the decoding stage) can be completed on a GPU (graphics processing unit). The hardware structure of the GPU can provide more computing resources. The main computing tasks in the pre-filling stage are composed of matrix-matrix multiplication (MM), and the computing density is relatively high. Therefore, the high computing throughput on the GPU hardware can be utilized when the GPU completes the pre-filling stage. However, the main computing tasks in the decoding stage are composed of matrix-vector multiplication (MV), and the computing density is relatively low. Therefore, the utilization rate of the GPU's computing resources decreases in the decoding stage. In addition, there is a problem of relatively high power consumption overhead when the GPU completes the target reasoning service.

[0051] If all computational stages of the target inference service (such as the pre-population stage and the decoding stage) are completed on an FPGA (field programmable gate array), the FPGA's hardware architecture, including high bandwidth memory (HBM), provides abundant storage resources that can guarantee high memory access in the decoding stage. However, due to limited FPGA computing resources and frequency, a computational bottleneck will be encountered in the pre-population stage, limiting the overall inference speed of the target inference service.

[0052] In summary, when the inference service of a large language model is deployed on a single piece of hardware (GPU or FPGA), the computation and storage of the pre-filling and decoding stages are performed by the corresponding resources on the same hardware. Since the computations in the pre-filling and decoding stages have their own characteristics, it is difficult to fully utilize the computation and memory access capabilities of a single piece of hardware, and thus it is impossible to further improve the performance of the inference service.

[0053] In view of this, in an embodiment of the present invention, a reasoning method P100 is provided, such as Figure 1 As shown, the following steps are included:

[0054] S1: In response to a request for a target inference service of a large language model, the CPU transmits a target input of the target inference service to the GPU through a first interface.

[0055] It is understood that the inference method P100 can be applied to a target system, a target inference service, such as any inference service initiated by a user, and a target input, such as text data, image data, or data in other formats, input by the user to the large language model in the target inference service. The first interface is, for example, a PCIe (Peripheral Component Interconnect Express) interface.

[0056] For example, the target system is Figure 2 As shown, the system includes a CPU, a GPU, and an FPGA. The CPU and GPU communicate via a first interface, and the CPU and FPGA communicate via a second interface. The CPU can assign the pre-filling phase of the target inference task to the GPU and the decoding phase to the FPGA.

[0057] For example, the target system is Figure 3 As shown, the system includes a CPU, a GPU, and n FPGAs, where n is an integer greater than 1. The CPU and GPU communicate via a first interface, and the CPU and FPGA communicate via a second interface. The CPU can assign the pre-filling phase of the target inference task to the GPU and the decoding phase to the n FPGAs.

[0058] For example, the target system is Figure 4 As shown, the system includes a CPU, m GPUs, and n FPGAs, where n and m are integers greater than 1. The CPU and GPUs communicate via a first interface, and the CPU and FPGAs communicate via a second interface. The CPU can assign the pre-population phase of the target inference task to the m GPUs and the decoding phase to the n FPGAs.

[0059] S2: The GPU performs pre-filling phase calculations on the target input based on the network structure and weight parameters of the large language model to obtain first data. The first data includes intermediate calculation results of the pre-filling phase and final calculation results of the pre-filling phase. The GPU transmits the first data to the CPU through the first interface.

[0060] For example, the network structure (e.g., computational graph, etc.) and weight parameters of the large language model can be pre-stored in the GPU. The network structure can be understood as an instruction to instruct the GPU to use the weight parameters in a certain order when performing calculations in the pre-filling phase. Figure 2 As shown, after the GPU completes the calculation in the pre-filling phase, it transmits the first data to the CPU. Alternatively, if there is a communication interface between the GPU and the FPGA, after the GPU completes the calculation in the pre-filling phase, it can directly transmit the first data to the FPGA.

[0061] Exemplarily, the calculations in the pre-filling stage are performed by N first decoder blocks, and the intermediate calculation results include N groups of intermediate calculation results. After a first decoder block completes the calculation, one group of the N groups of intermediate calculation results is obtained, where N is an integer greater than or equal to 1.

[0062] For example, Figure 2 Taking the target system of the example as an example, Figure 5This example illustrates the data flow during GPU and FPGA computation. The pre-population phase is performed by N first decoder blocks, which include: first decoder block 0, first decoder block 1, ..., first decoder block N-1. Each first decoder block includes three submodules: self-attention, linear, and feed-forward networks. Taking first decoder block 0 as an example, during the computation, the input of first decoder block 0 is the target input in S1. After the computation is complete, the first set of intermediate results is generated, consisting of key and value matrices. After first decoder block 0 completes the computation, the generated key and value matrices are stored on the GPU as a KV cache. The output of first decoder block 0 serves as the input to first decoder block 1. After the computation of first decoder block 1 is completed, the second set of intermediate results is generated. This continues in this order until the Nth set of intermediate results is generated after the computation of first decoder block N-1 is completed. The contents and storage formats of the second to Nth groups of intermediate calculation results are as described above for the first group of intermediate calculation results, and will not be repeated here.

[0063] That is, in the pre-filling stage, the first data that needs to be stored and transmitted to the CPU includes N groups of intermediate calculation results and the final calculation result of the pre-filling stage calculated by the first decoder block N-1.

[0064] S3: The CPU transmits the first data to the FPGA through the second interface.

[0065] Exemplarily, the second interface is a PCIe interface or an interface capable of connecting to a network cable.

[0066] S4: FPGA performs calculations in the decoding stage based on the network structure, weight parameters, and first data of the large language model to obtain second data, which is the final calculation result of the decoding stage. FPGA transmits the second data to the CPU through the second interface.

[0067] Exemplarily, the network structure and weight parameters of the large language model can be pre-stored in the FPGA. The network structure can be understood as an instruction instructing the FPGA to use the weight parameters in a certain order when performing calculations in the decoding stage.

[0068] Exemplarily, the decoding phase is performed by N second decoder blocks, where each of the N second decoder blocks corresponds one-to-one to each of the N first decoder blocks. Based on the network structure and weight parameters of the large language model, the FPGA uses the final calculation results of the pre-filling phase as the input for the calculation of the first decoder block in the decoding phase. For the i-th second decoder block among the N second decoder blocks, the i-th second decoder block is calculated based on the i-th group of intermediate calculation results. The i-th group of intermediate calculation results is the calculation result of the i-th first decoder block corresponding to the i-th second decoder block, where 2≤i≤N.

[0069] For example, after the CPU transmits the first data to the FPGA, the first data is first stored in the FPGA. After the FPGA receives the first data, it triggers the FPGA to start the calculation in the decoding stage. Figure 5 As shown, the decoding phase is performed by N second decoder blocks, including: second decoder block 0, second decoder block 1, ..., second decoder block N-1. Each second decoder block can include three submodules: self-attention, linear, and feed-forward network. After second decoder block 0 begins calculation, it obtains the final calculation result of the pre-filling phase from the stored first data as the input of second decoder block 0. During the calculation process, it also needs to obtain the first set of intermediate calculation results generated by first decoder block 0 in the pre-filling phase from the first data to participate in the calculation of second decoder block 0. After the calculation of second decoder block 0 is completed, the calculation output of second decoder block 0 is used as the input of second decoder block 1. During the calculation process of second decoder block 1, it also needs to obtain the second set of intermediate calculation results generated by first decoder block 1 in the pre-filling phase from the first data to participate in the calculation of second decoder block 1. Similarly, during the calculation process of the second decoder block N-1, it is also necessary to obtain the Nth group of intermediate calculation results generated by the first decoder block N-1 in the pre-filling stage from the first data to participate in the calculation of the second decoder block N-1. After the calculation of the second decoder block N-1 is completed, the final calculation result of the decoding stage, that is, the second data, is obtained.

[0070] S5: The CPU uses the second data as the output of the target reasoning service.

[0071] Exemplarily, the output of the target inference service is the conclusion (in the form of text, documents, code, etc.) given by the large language model to the text data, image data, or other data formats input by the user. It should be understood that the decoding stage can be iterated multiple times to obtain an output including multiple tokens. Before outputting the final result to the user, the large language model can also perform various necessary processing on the second data, such as combining, formatting, filtering, etc.

[0072] In some embodiments, the target system, for example, Figure 4 As shown, multiple GPUs and multiple FPGAs are included. In this case, the inference method also includes:

[0073] The CPU obtains a mapping relationship, where the mapping relationship includes multiple input-output pairs, wherein each input-output pair includes sample input data and sample output data of any sample inference service;

[0074] Exemplarily, the CPU obtains sample input data and sample output data of multiple sample reasoning services, determines an input-output pair for the sample input data and sample output data of each sample reasoning service, and determines a mapping relationship based on the input-output pairs corresponding to each of the multiple sample reasoning services.

[0075] For example, a service module can be set up on the CPU host to record the sample input data and sample output data of completed sample inference services. A sample inference service is any completed user-initiated inference service. The sample input data is the user's input data for the sample inference service, and the sample output data is the output data calculated by the large language model based on the sample input data. The service module can also record the task load on the GPU and FPGA during the calculation process of the sample inference service or the target inference service.

[0076] A load predictor can be set up in the service module. This load predictor can obtain sample input and output data corresponding to multiple sample inference services. The sample input and output data corresponding to one of the sample inference services is used as a training sample. Based on these multiple training samples, the load predictor can be trained to predict the number of tokens in the output data of any inference service using the input data. After training, a mapping relationship table containing multiple input-output pairs is generated. The load predictor can be implemented using algorithms such as regression analysis and machine learning.

[0077] In some embodiments, the CPU predicts the task load and reasoning time of the target reasoning service based on the mapping relationship. It should be understood that the prediction can be based on a deterministic algorithm or a non-deterministic algorithm (e.g., an algorithm based on probability statistics, etc.).

[0078] Exemplarily, the CPU searches for a target input-output pair corresponding to the target reasoning service in a mapping relationship based on the target input of the target reasoning service, determines the length of the output data of the target reasoning service based on the target input-output pair, and predicts the task load and reasoning time based on the length of the output data.

[0079] For example, the load predictor can predict the number of tokens output by the target inference service based on the target input of the target inference service, thereby predicting the task load and inference time corresponding to the target inference service based on the number of tokens output. Task load can be understood as workload, that is, the load (e.g., computational load) required to complete the target inference service. Inference time can be understood as the time required to complete the target inference service.

[0080] Specifically, the load predictor can predict the load #1 and time #1 of the pre-filling stage of the target inference service, as well as the load #2 and time #2 of the decoding stage based on the target input. The sum of load #1 and load #2 is the task load corresponding to the target inference service, and the sum of time #1 and time #2 is the inference time corresponding to the target inference service.

[0081] In some embodiments, the CPU determines to schedule the computing tasks of the pre-filling stage to at least one GPU among multiple GPUs and to schedule the computing tasks of the decoding stage to at least one FPGA among multiple FPGAs based on task load, inference time, and a preset scheduling policy.

[0082] For example, the preset scheduling strategy is such as giving priority to the shortest task (e.g., the shortest required inference time) or the task with the shortest remaining time (e.g., the shortest required inference time for the remaining calculation). For example, if there are multiple inference services running at the same time, the inference service with the shortest inference time can be preferentially allocated to the inference calculation in the GPU and FPGA. For another example, if the inference calculation of inference service A has already started, then when the CPU allocates other inference services and the remaining computing tasks of inference service A, it needs to give priority to allocating resources to the remaining computing tasks of inference service A, because the inference time required for the remaining computing tasks of inference service A is shorter than that of other inference services.

[0083] For example, Figure 3 As shown, the CPU assigns the pre-population phase to the GPU and, based on the current load of FPGAs 0 to FPGAn-1, as well as Load #2 and Time #2, determines which FPGA to allocate the decoding phase to that meets QoS (Quality of Service) requirements. For example, the decoding phase can be assigned to a specific FPGA, or the decoding phase can be divided into multiple phases and assigned to multiple FPGAs from FPGA0 to FPGAn-1. As the multiple FPGAs coordinate to complete the decoding phase, they can transmit data required by each other. Alternatively, one FPGA can transmit data required by other FPGAs to the CPU, which then transmits the data to the other FPGAs.

[0084] For example, Figure 4As shown, the CPU determines, based on the current load conditions of GPUs 0 to m-1, load #1, and time #1, to allocate the pre-fill phase to a GPU that meets QoS requirements. For example, the CPU may allocate the pre-fill phase to a specific GPU, or divide the pre-fill phase into multiple phases and allocate them to multiple GPUs from GPUs 0 to m-1. Furthermore, based on the current load conditions of FPGAs 0 to n-1, load #2, and time #2, the CPU may determine to allocate the decoding phase to an FPGA that meets QoS requirements. For example, the CPU may allocate the decoding phase to a specific FPGA, or divide the decoding phase into multiple phases and allocate them to multiple FPGAs from FPGAs 0 to n-1. While the multiple GPUs coordinate to complete the computations in the pre-fill phase, they can transmit data required by each other. Alternatively, a GPU can transmit data required by other GPUs to the CPU, which then transmits the data to the other GPUs.

[0085] In summary, the inference method provided by the present invention is applied to a target system including a GPU and an FPGA. The pre-filling and decoding stages of the target inference service for a large language model are deployed to the GPU and FPGA, respectively, via the CPU, for collaborative computing. During the collaborative computing process, the GPU can first store the intermediate and final computational results of the pre-filling stage locally on the GPU. These intermediate and final computational results are then transmitted to the FPGA via the CPU, allowing the FPGA to continue computing the decoding stage. This collaborative computing process leverages the centralized scheduling capabilities of the CPU, deploying computational stages with varying requirements for computing and storage resources on demand to processing platforms with corresponding computing and storage capabilities. This improves inference performance while fully utilizing the hardware resources of the GPU and FPGA.

[0086] The present invention also provides an inference system, comprising:

[0087] The CPU is configured to respond to a request of a target inference service of the large language model and transmit a target input of the target inference service to the GPU through the first interface;

[0088] The GPU is configured to perform a pre-filling calculation on a target input based on the network structure and weight parameters of the large language model to obtain first data, the first data including an intermediate calculation result of the pre-filling calculation and a final calculation result of the pre-filling calculation, and transmit the first data to the CPU via a first interface;

[0089] The CPU is further configured to transmit the first data to the FPGA via the second interface;

[0090] The FPGA is configured to perform calculations in a decoding phase based on the network structure, weight parameters, and first data of the large language model to obtain second data, which is a final calculation result of the decoding phase, and transmit the second data to the CPU via a second interface;

[0091] The CPU is further configured to use the second data as an output of the target reasoning service.

[0092] In an embodiment of the present invention, a computer device is provided, comprising: a processor and a memory storing a computer program, wherein the processor is configured to execute the reasoning method of any embodiment of the present invention when running the computer program.

[0093] Figure 6 A schematic diagram of an electronic device 1000 that can implement a method or implement an embodiment of the present invention is shown. In some embodiments, the method may include more or fewer electronic devices than shown. In some embodiments, the method may be implemented using a single electronic device or multiple electronic devices. In some embodiments, the method may be implemented using cloud-based or distributed electronic devices.

[0094] like Figure 6 As shown, the electronic device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) 1002 or the programs and / or data loaded from the storage part 1008 into the random access memory (RAM) 1003. The processor 1001 can be a multi-core processor or can include multiple processors. In some embodiments, the processor 1001 can include a general-purpose main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are also stored. The processor 1001, ROM 1002 and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0095] Exemplarily, the processor 1001 includes a CPU, a GPU, and an FPGA.

[0096] The processor and memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0097] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed. Figure 6 Only some components are shown schematically, which does not mean that the computer system 1000 only includes Figure 6 Components shown.

[0098] The systems, devices, modules, or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smartphone, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server, or a combination thereof.

[0099] Although not shown, in an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores a computer program, and the computer program is configured to execute any reasoning method of the embodiment of the present invention when executed.

[0100] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information using any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0101] The methods, programs, systems, and apparatuses of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0102] The present application also relates to a computer program product, which includes executable code, and when the executable code is executed by a computer, it implements the steps of the reasoning method described in the embodiment of the present application.

[0103] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, those skilled in the art will appreciate that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented using software, hardware, or a combination of software / hardware.

[0104] Unless explicitly stated, the actions or steps of the methods, procedures, and methods described in accordance with the embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0105] In this document, multiple embodiments of the present invention are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this document, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to apply to at least one embodiment or example according to the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. Those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are mutually contradictory.

[0106] While the exemplary systems and methods of the present invention have been specifically shown and described with reference to the foregoing embodiments, these are merely examples of the best modes for implementing the present systems and methods. Those skilled in the art will appreciate that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A reasoning method, characterized in that: The method is applied to a target system, the target system including a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA), wherein the CPU and the GPU communicate via a first interface, and the CPU and the FPGA communicate via a second interface. The method comprises the following steps: S1: In response to a request for a target inference service of a large language model, the CPU transmits a target input of the target inference service to the GPU through the first interface; S2: The GPU performs a pre-filling calculation on the target input based on the network structure and weight parameters of the large language model to obtain first data, where the first data includes an intermediate calculation result of the pre-filling calculation and a final calculation result of the pre-filling calculation. The GPU transmits the first data to the CPU through the first interface. S3: The CPU transmits the first data to the FPGA through the second interface; S4: The FPGA performs calculations in a decoding phase based on the network structure, weight parameters, and the first data of the large language model to obtain second data, where the second data is a final calculation result of the decoding phase. The FPGA transmits the second data to the CPU through the second interface. S5: The CPU uses the second data as the output of the target reasoning service; The target system includes multiple GPUs and multiple FPGAs, and the method further includes: The CPU obtains a mapping relationship, where the mapping relationship includes a plurality of input-output pairs, wherein each input-output pair includes sample input data and sample output data of any sample inference service; The CPU predicts the task load and reasoning time of the target reasoning service based on the mapping relationship; The CPU determines, based on the task load, the inference time, and a preset scheduling policy, to schedule the computing task of the pre-filling stage to at least one of the multiple GPUs, and to schedule the computing task of the decoding stage to at least one of the multiple FPGAs; The CPU predicts the task load and inference time of the target inference service based on the mapping relationship, including: The CPU searches for a target input-output pair corresponding to the target reasoning service in the mapping relationship based on the target input of the target reasoning service; The CPU determines the length of output data of the target inference service based on the target input-output pair; The CPU predicts the task load and the inference time based on the length of the output data.

2. The inference method according to claim 1, characterized in that The CPU obtaining the mapping relationship includes: The CPU obtains sample input data and sample output data of a plurality of sample inference services; The CPU determines an input-output pair for sample input data and sample output data of each sample inference service; The CPU determines the mapping relationship based on input-output pairs corresponding to each of the plurality of sample inference services.

3. The inference method according to claim 1, characterized in that The calculation of the pre-filling stage is completed by N first decoder blocks, and the intermediate calculation results include N groups of intermediate calculation results. After one of the first decoder blocks completes the calculation, one group of the N groups of intermediate calculation results is obtained, and N is an integer greater than or equal to 1.

4. The inference method according to claim 3, characterized in that The calculations in the decoding stage are performed by N second decoder blocks, and the FPGA performs the calculations in the decoding stage based on the network structure, weight parameters and the first data of the large language model, including: The FPGA uses the final calculation result of the pre-filling stage as the input for calculation of the first decoder block in the decoding stage based on the network structure and weight parameters of the large language model; For the i-th second decoder block among the N second decoder blocks, the i-th second decoder block is calculated based on the i-th group of intermediate calculation results, and the i-th group of intermediate calculation results is the calculation result of the i-th first decoder block corresponding to the i-th second decoder block, where 2≤i≤N.

5. The inference method according to claim 3 or 4, characterized in that Each of the N groups of intermediate calculation results includes a key and a value matrix.

6. An inference system, adapted to employ the inference method according to any one of claims 1 to 5, characterized in that: include: a central processing unit (CPU) configured to respond to a request of a target inference service of a large language model and transmit a target input of the target inference service to the GPU via a first interface; a graphics processing unit (GPU), configured to perform a pre-filling phase calculation on the target input based on the network structure and weight parameters of the large language model, obtain first data, the first data including an intermediate calculation result of the pre-filling phase and a final calculation result of the pre-filling phase, and transmit the first data to the CPU via the first interface; The CPU is further configured to transmit the first data to a field programmable gate array (FPGA) via a second interface; The FPGA is configured to perform calculations in a decoding phase based on the network structure, weight parameters, and the first data of the large language model to obtain second data, where the second data is a final calculation result of the decoding phase, and transmit the second data to the CPU via the second interface; The CPU is further configured to use the second data as output of the target reasoning service.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the reasoning method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the reasoning method according to any one of claims 1 to 5 are implemented.

Images