Method, medium, apparatus, and program product for computing attention scores based on cpu

Abstract

A method, medium, device, and program product for CPU-based attention score calculation, the method comprising: obtaining a user query for input into a large language model; retrieving information related to the user query and obtaining a key value corresponding to the information from a storage medium; the storage medium being distinct from GPU video memory, and the key value being a key value pre-calculated based on the large language model; the CPU performing attention calculation based on the key value to obtain an attention score, and sending the attention score to the GPU, so that the GPU uses the attention score to generate response information for the user query.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, and particularly relates to a method for calculating attention scores based on CPUs, a medium, an equipment and a program product. BACKGROUND

[0002] Retrieval-Augmented Generation (RAG) is a framework combining retrieval and generation techniques, which retrieves relevant information from an external knowledge base and combines it with a user query to enhance the generation capabilities of a large language model (LLM). In the RAG process, the user input query and the retrieved information are used as prompt information for the large language model, which can process the obtained prompt information based on an attention mechanism and output response information. To optimize this process, the TurboRAG scheme proposes to pre-compute and store the key values corresponding to each text block. In this way, after retrieving information related to the user query, the corresponding key values can be obtained directly, and attention calculation can be performed based on the obtained key values, avoiding the step of online computing key values, significantly reducing the computational overhead and improving the response efficiency of the large language model.

[0003] Due to the powerful computing performance of the Graphics Processing Unit (GPU), large language models are usually deployed on GPUs. However, RAG needs to retrieve information related to the user query from a large number of text blocks, and due to the limited memory capacity of the GPU, the key values corresponding to these text blocks are usually stored in the memory of the Central Processing Unit (CPU) or other storage media. When performing attention calculation, the CPU needs to first obtain the key values from the storage medium and then send the obtained key values to the GPU. In the case of low data transmission bandwidth or large data volume, the data transmission time is long, which will result in low efficiency of the attention calculation process. SUMMARY

[0004] In a first aspect, an embodiment of the present application provides a method for calculating attention scores based on CPUs, which comprises:

[0005] obtaining a user query for inputting a large language model;

[0006] retrieving information related to the user query and obtaining the key values corresponding to the information from a storage medium; the storage medium is different from the GPU memory, and the key values are key values pre-computed based on the large language model;

[0007] The CPU performs attention calculation based on the key value to obtain an attention score, and sends the attention score to the GPU, so that the GPU uses the attention score to generate response information for the user query.

[0008] In a second aspect, an embodiment of the present application provides a method for calculating an attention score based on a CPU, the method comprising:

[0009] The GPU receives an attention score sent by the CPU, where the attention score is calculated by the CPU based on a key value, where the key value is retrieved by the CPU from a storage medium, and the key value corresponds to information retrieved by the CPU; the information is related to a user query input into the large language model, the storage medium is distinct from the GPU memory, and the key value is a key value pre-calculated based on the large language model;

[0010] The GPU generates response information for the user query based on the attention score.

[0011] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present application.

[0012] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present application when executing the computer program.

[0013] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any embodiment of the present application when executed by a processor.

[0014] In an embodiment of the present application, different processors are used to execute the process of calculating the attention score in the large language model and other processing processes of the large language model. Specifically, the CPU executes the process of calculating the attention score, and the GPU executes other processing processes other than calculating the attention score to obtain response information. In this way, the CPU does not need to send the retrieved key value to the GPU, but directly sends the attention score to the GPU after calculating the attention score. Since the amount of data of the attention score is much smaller than the amount of data of the key value, the method of the embodiment of the present application can save the transmission time of the key value between different processors, thereby improving the efficiency of the attention calculation process.

[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate implementations of the application and, together with the description, further serve to explain the principles of the application.

[0017] Figure 1 is a schematic diagram of the overall flow of RAG and TurboRAG in the related art.

[0018] Figure 2 is a flowchart of a method for calculating an attention score based on a CPU according to an embodiment of the present application.

[0019] Figure 3 is a flowchart of a method for calculating an attention score based on a CPU according to another embodiment of the present application.

[0020] Figure 4 is a block diagram of an apparatus for calculating an attention score based on a CPU according to an embodiment of the present application.

[0021] Figure 5 is a block diagram of an apparatus for calculating an attention score based on a CPU according to another embodiment of the present application.

[0022] Figure 6 is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The exemplary embodiments will be described in detail herein below with reference to the accompanying drawings. The following description is with reference to the drawings, in which like numerals represent like elements, throughout the several views. The following description is not to be understood as limiting the true scope of the present application. In addition, the exemplary embodiments described in the detailed description section are not meant to be limiting in terms of the scope of the present application. That is, the present application is deemed to encompass any and all techniques within the scope of the appended claims.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Additionally, use of the term "and / or" is meant to include any and all possible combinations of one or more of the associated listed items.

[0025] It should be understood that, although the terms first, second, third, etc. can be employed in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another type of information. For example, without departing from the scope of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0026] In order to better understand the technical solutions in the embodiments of the present application by those skilled in the art, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more apparent and easy to understand, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the drawings.

[0027] Figure 1 The overall flow of standard RAG and TurboRAG in related art is shown.

[0028] In the standard RAG, a knowledge base can be established in advance, each text block in the knowledge base is embedded, and the vector of each text block is stored. In the reasoning process, first, K (K is a positive integer) text blocks (i.e. context information) related to the user query are retrieved, then the K text blocks and the user query are used to construct a prompt information (prompt) together, and the prompt information is passed to a large language model. The large language model can generate response information based on an attention mechanism. Specifically, the query vector (Q), the key vector (K) and the value vector (V) can be calculated based on the vectors of the user query and the K text blocks, the attention score can be obtained based on the calculated Q, K and V, and the response information can be generated based on the attention score.

[0029] In order to speed up the above process, the TurboRAG scheme proposes to pre-calculate and store the corresponding key-value of each text block, and the stored key-value is called KV cache. In this way, after retrieving the information, the corresponding key-value can be directly obtained, and the attention calculation is performed based on the obtained key-value, avoiding the step of online calculating the key-value, significantly reducing the calculation overhead and improving the response efficiency of the large language model.

[0030] Since the GPU has strong computing performance, a large language model is usually deployed on the GPU. However, the RAG needs to retrieve information from a large number of text blocks, and since the GPU has limited video memory capacity, the key values corresponding to the text blocks are usually stored in the memory or the storage medium accessible by the CPU, such as a disk. When attention calculation is performed, the CPU needs to first obtain the key values from the storage medium and then send the obtained key values to the GPU. In the case of low data transmission bandwidth or large data volume, the data transmission time is relatively long, which reduces the efficiency of the attention calculation process.

[0031] Based on this, the embodiment of the present application proposes a method for calculating attention scores based on a CPU. The CPU replaces the GPU to perform the calculation process of the attention scores. In this way, the CPU does not need to send the retrieved key values to the GPU, but directly sends the attention scores to the GPU after calculating the attention scores. Since the data volume of the attention scores is much smaller than that of the key values, the transmission time of the key values between different processors can be saved by the method of the embodiment of the present application, thereby improving the efficiency of the attention calculation process. Referring to Figure 2 , the method comprises:

[0032] Step S12: obtaining a user query for inputting a large language model;

[0033] Step S14: retrieving information related to the user query and obtaining key values corresponding to the information from a storage medium; the storage medium is different from the GPU video memory, and the key values are key values calculated in advance based on the large language model;

[0034] Step S16: the CPU performs attention calculation based on the key values to obtain attention scores, and sends the attention scores to the GPU, so that the GPU uses the attention scores to generate response information for the user query.

[0035] In the embodiment of the present application, the step of performing attention calculation is performed by the CPU, and other steps can be performed by the CPU or other processing units. For the sake of brevity, the scheme of the embodiment of the present application will be described below by taking an example in which each step is performed by the CPU.

[0036] In step S12, the user can input a user query (query) in a client (such as a browser, a mobile application, etc.) and send the query to the server of the large model service through an HTTP request. The HTTP request is transmitted to the server through the network, and the server receives the HTTP request, parses the HTTP request through the CPU, and extracts the query input by the user. At this time, the query data is stored in a storage medium. The storage medium can be a medium other than the GPU memory, for example, a medium that can be directly or indirectly accessed by the CPU of the server, such as a memory or a disk, etc. Due to the limitation of the hardware architecture, the GPU usually cannot directly access the memory, the disk, etc.

[0037] In step S14, the CPU can retrieve information related to the user query. Specifically, the CPU can perform vectorization processing on the user query to obtain a vector corresponding to the user query, and based on the similarity between the vector corresponding to the user query and the vectors of the pre-stored text blocks, retrieve a plurality of text blocks related to the user query from the pre-stored text blocks. These retrieved text blocks are the information related to the user query. The information related to the user query can be stored in the above-mentioned storage medium, or in the cloud, or in other storage space.

[0038] After retrieving the information related to the user query, the CPU can obtain the key value corresponding to the above-mentioned information. Specifically, after generating the key value, the source information of the key value (such as the text block corresponding to the key value and the document to which the text block belongs) and the storage address of the key value can be recorded. After retrieving the information related to the user query, the CPU can determine the storage address of the key value bound to the retrieved information based on the recorded source information, and read the key value from the corresponding storage address.

[0039] In step S16, the CPU can obtain the query vector (Q), the key vector (K), and the value vector (V) calculated based on the user query. In some embodiments, the above-mentioned Q, K, and V can be calculated by the GPU. Specifically, after parsing the query input by the user, the CPU can send the query to the GPU, and the GPU can calculate the above-mentioned Q, K, and V based on the received query and return them to the CPU. The data transmission between the CPU and the GPU can be realized based on the PCIe bus. Since the data volume of the query and Q, K, and V is usually small, the time consumption of transmitting the query and Q, K, and V is usually short. Moreover, the computing power of the GPU is relatively strong, so the process of calculating Q, K, and V based on the query consumes less time, and therefore the overall time consumption of the above-mentioned process is usually short.

[0040] In other examples, the CPU can also calculate Q, K, and V directly based on the query after parsing the query. Although the computing power of the CPU is weaker than that of the GPU, this can save the time for the CPU to transmit the query to the GPU and for the GPU to return Q, K, and V to the CPU.

[0041] After obtaining Q, K, and V, the CPU can perform attention calculation based on Q, K, and V and the key value obtained in step S14 to obtain an attention score. Specifically, the CPU can fuse the key vector in the key value with the key vector calculated based on the user query to obtain a fused key vector (denoted as K'), and fuse the value vector in the key value with the value vector calculated based on the user query to obtain a fused value vector (denoted as V'), and then perform attention calculation based on Q, K', and V' to obtain the attention score.

[0042] After obtaining the attention score, the CPU can send the attention score to the GPU through the PCIe bus. After receiving the attention score, the GPU can use the attention score to generate response information for the user query.

[0043] In some embodiments, the GPU and the CPU can obtain the response information through multiple rounds of iterative interaction. In each iteration, the CPU can perform attention calculation based on the query vector, the key vector, and the value vector sent by the GPU in the current iteration to obtain an attention score in the current iteration. The query vector, the key vector, and the value vector sent by the GPU in the first iteration are generated based on the user query or based on the historical output token (such as the previous output token) of the large language model. The query vector, the key vector, and the value vector sent by the GPU in the i-th iteration are generated based on the attention score obtained in the (i-1)-th iteration; 1 < i < N, N is a preset maximum number of iterations. The attention score obtained in the N-th iteration is used to generate the response information for the user query. The following will illustrate the iteration process through a specific example. In this embodiment, the CPU and the GPU perform N times of iterative interaction to generate an output token in the response information each time. The process of generating the first output token is as follows:

[0044] In the first iteration, the GPU generates a query vector, a key vector, and a value vector based on the user query and sends them to the CPU. The CPU fuses the key vector in the key value with the key vector to obtain a fused key vector, fuses the value vector in the key value with the value vector to obtain a fused value vector, and sends the attention score generated based on the query vector, the fused key vector, and the fused value vector to the GPU. The GPU calculates a hidden state vector based on the attention score.

[0045] At the second iteration, the GPU obtains the hidden state vector obtained by the CPU at the first iteration, calculates the query vector, the key vector and the value vector based on the hidden state vector, and sends the calculated query vector, the key vector and the value vector to the CPU, so that the CPU recalculates the attention score. Then, the CPU sends the recalculated attention score to the GPU, so that the GPU recalculates the hidden state vector.

[0046] The subsequent iteration process is similar to the foregoing embodiment, and will not be described here. At the Nth iteration, the GPU calculates the probability corresponding to each token based on the Nth obtained hidden state vector, and determines the token with the maximum probability as the first output token.

[0047] The process of generating other output tokens is similar to the foregoing process, except that in the process of generating the second output token, the GPU generates the query vector, the key vector and the value vector based on the first output token at the first iteration. In the process of generating the third output token, the GPU generates the query vector, the key vector and the value vector based on the second output token at the first iteration. And so on.

[0048] In some embodiments, the large language model can include a plurality of processing modules, each processing module including a sub-module deployed on a CPU and a sub-module deployed on a GPU. The i th iteration process can be performed by the i th processing module, and N is the total number of processing modules. The operations performed by the CPU at the i th iteration are performed by the sub-module deployed on the CPU in the i th processing module, and the operations performed by the GPU at the i th iteration are performed by the sub-module deployed on the GPU in the i th processing module. For details of the specific processing process, please refer to the foregoing embodiments, which will not be described here.

[0049] In some embodiments, the CPU is used instead of the GPU to calculate the attention score, which can not improve the calculation efficiency in all cases. Since the CPU has weak computing power, the time spent by the GPU in calculating the attention score is mainly affected by the computing power of the CPU. When the GPU is used to calculate the attention score, the key value needs to be transmitted between the CPU and the GPU, so the time spent by the GPU in calculating the attention score is mainly affected by the transmission time of the key value. In order to improve the calculation efficiency of the attention score, the preset condition can be determined based on the estimated calculation time of the CPU in calculating the attention and the estimated transmission time of the CPU in transmitting the key value to the GPU. Only when the preset condition is met, step S16 is performed.

[0050] In some embodiments, the preset condition comprises that the estimated calculation duration of the CPU for attention calculation is less than the estimated transmission duration of the CPU for transmitting the key-value to the GPU. When the preset condition is met, the attention score is calculated by the CPU instead of the GPU, which can reduce the time consumption caused by the transmission of the key-value and improve the calculation efficiency of the attention score.

[0051] The estimated calculation duration (hereinafter referred to as calculation duration) of the CPU for attention calculation can be determined to be less than the estimated transmission duration (hereinafter referred to as transmission duration) of the CPU for transmitting the key-value to the GPU based on at least one of the following conditions:

[0052] (1) The length of the information related to the user query. The length of the information can be represented by the number of characters included in the information. The length of the key-value is usually positively correlated with the length of the information, and the length of the information will affect the amount of data to be processed during attention calculation and the amount of data transmission between the CPU and the GPU. Therefore, whether the calculation duration is less than the transmission duration can be determined according to the length of the information. In some embodiments, if the length of the information is within a preset length range (such as between a preset lower length limit and a preset upper length limit), it can be determined that the calculation duration is less than the transmission duration. Conversely, if the length of the information is not within the preset length range, it can be determined that the calculation duration is greater than or equal to the transmission duration.

[0053] (2) The resource utilization rate of the CPU. The resource utilization rate of the CPU is the ratio between the non-idle resources in the CPU and the total resources of the CPU, for example, the resource utilization rate of the CPU can be represented by the ratio between the number of non-idle threads in the CPU and the total number of threads of the CPU. The resource utilization rate of the CPU can reflect the load state of the CPU. If the resource utilization rate of the CPU is too high, it means that the CPU is busy processing various tasks and may not have enough resources to calculate the attention score. Therefore, if the resource utilization rate of the CPU is less than a preset resource utilization rate threshold, it can be determined that the calculation duration is less than the transmission duration. Conversely, if the resource utilization rate of the CPU is greater than or equal to the preset resource utilization rate threshold, it can be determined that the calculation duration is greater than or equal to the transmission duration.

[0054] (3) The data transmission bandwidth between the CPU and the GPU. The greater the data transmission bandwidth, the shorter the duration of the CPU transmitting the key-value cache to the GPU. Therefore, if the data transmission bandwidth between the CPU and the GPU is less than a preset bandwidth threshold, it can be determined that the calculation duration is less than the transmission duration. Conversely, if the data transmission bandwidth between the CPU and the GPU is greater than or equal to the preset bandwidth threshold, it can be determined that the calculation duration is greater than or equal to the transmission duration.

[0055] In actual applications, the size relationship between the calculation duration and the transmission duration can be determined in combination with one or more of the above parameters, or the size relationship between the calculation duration and the transmission duration can be determined based on other factors.

[0056] If it is determined that the preset condition is met, step S16 can be performed, that is, the attention score is calculated by the CPU. Further, if the preset condition is not met, the key value can be sent to the GPU to perform attention calculation based on the query vector, the key vector, the value vector, and the key value to obtain the attention score. That is, in the case where the preset condition is not met, the GPU can perform the calculation process of the attention score. Based on this, the embodiments of the present application realize flexible selection of the processor for performing the attention score calculation between the CPU and the GPU according to the actual situation, which can meet the needs of various application scenarios.

[0057] In some embodiments, when the CPU performs the calculation process of the attention score, the CPU can call multiple threads to perform attention calculation based on the key value to obtain the attention score. The number of called threads can be pre-set or determined according to the actual situation. Alternatively, the number of threads can be determined according to the current available number of threads of the CPU. For example, a certain proportion of threads can be selected from the current available threads to calculate the attention score.

[0058] In natural language processing, the order of tokens is crucial to semantic understanding. Therefore, in related technologies, position encoding information of each token is generated based on the position of the token in the original document, such as rotary position encoding (RoPE), and the key value is generated and stored based on the key vector fused with the position encoding information. However, when retrieving, several text blocks are retrieved from the original document, and the position of the token in the text block can be different from the position in the original document. For example, the original document includes 100 tokens, and the retrieved text block includes the 51st token in the original document to the 100th token in the original document. Then, the 51st token in the original document is the 1st token in the retrieved text block, which is different from its position information in the original document, thereby causing the position encoding information of the token to change.

[0059] To solve the above problem, the key value obtained by the embodiments of the present application is a key value without fused position encoding information, and the position encoding information of the token in the information is obtained after the information is retrieved, and the above position encoding information is fused into the key value. The above method separates the key value acquisition process from the position encoding information acquisition process, avoiding the case where the position encoding information of the token changes before and after retrieval, resulting in errors in the attention calculation result.

[0060] See also Figure 3 , an embodiment of the present application further provides a method for calculating an attention score based on a CPU, the method comprising:

[0061] Step S22: The GPU receives an attention score sent by the CPU, where the attention score is calculated by the CPU based on a key value, where the key value is retrieved by the CPU from a storage medium, and the key value corresponds to information retrieved by the CPU; the information is related to the user query used to input the large language model, the storage medium is different from the GPU memory, and the key value is a key value pre-calculated based on the large language model;

[0062] Step S24: The GPU generates response information for the user query based on the attention score.

[0063] In some embodiments, the GPU receives the attention score sent by the CPU, including: the GPU receives the attention score sent by the CPU when preset conditions are met; the preset conditions include: the estimated calculation time for the CPU to perform attention calculation is less than the estimated transmission time for the CPU to transmit the key value to the GPU.

[0064] In some embodiments, the method further includes: the GPU receiving the key value sent by the CPU when the preset condition is not met; the GPU performing attention calculation based on the key value to obtain the attention score.

[0065] The specific implementation details of the embodiments of the present application are detailed in the aforementioned method embodiment executed by the CPU and will not be repeated here.

[0066] See also Figure 4 , an embodiment of the present application further provides a device for calculating an attention score based on a CPU, the device comprising:

[0067] An acquisition module 102 is used to acquire a user query for input into a large language model;

[0068] A retrieval module 104 is configured to retrieve information related to the user query and obtain a key value corresponding to the information from a storage medium, wherein the storage medium is different from the GPU memory and the key value is a key value pre-calculated based on a large language model;

[0069] The calculation module 106 is used to perform attention calculation based on the key value to obtain an attention score, and send the attention score to the GPU so that the GPU uses the attention score to generate response information for the user query. The calculation module 106 is a module in the CPU.

[0070] The specific implementation details of the embodiments of the present application are described in the foregoing method embodiments executed by the CPU, which will not be repeated here.

[0071] Referring to Figure 5 The embodiments of the present application also provide a device for calculating attention scores based on a CPU, and the method comprises the following steps:

[0072] The receiving module 202 is configured to receive an attention score sent by the CPU, wherein the attention score is obtained by the CPU based on a key value, the key value is obtained by the CPU from a storage medium, and the key value corresponds to information retrieved by the CPU; the information is related to a user query for inputting a large language model, the storage medium is different from GPU memory, and the key value is a key value calculated in advance based on a large language model;

[0073] The response generation module 204 is configured to generate response information for the user query based on the attention score.

[0074] The receiving module 202 and the response generation module 204 are modules in the GPU.

[0075] The specific implementation details of the embodiments of the present application are described in the foregoing method embodiments executed by the GPU, which will not be repeated here.

[0076] The embodiments of the present application also provide a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any of the foregoing embodiments.

[0077] Figure 6 A more specific hardware structure of a computer device provided by the embodiments of the present application is shown, which can include a processor 302, a memory 304, an input / output interface 306, a communication interface 308, and a bus 310. The processor 302, the memory 304, the input / output interface 306, and the communication interface 308 are connected to each other through the bus 310 for communication within the device.

[0078] The processor 302 can include a CPU and a GPU for executing related programs to implement the technical solutions provided by the embodiments of the present application. The processor 302 can also include a graphics card, which can be an Nvidia titan X graphics card or a 1080Ti graphics card, etc.

[0079] The memory 304 can be implemented in the form of a Read Only Memory (ROM), a Random Access Memory (RAM), a static storage device, a dynamic storage device, etc. In the case where the processor 302 includes a CPU and a GPU, the memory 304 can include a memory (such as a memory and a disk) corresponding to the CPU and a memory (such as a video memory) corresponding to the GPU. The memory corresponding to the CPU can be accessed by the CPU, and the CPU can obtain data therein and send it to the GPU. Similarly, the memory corresponding to the GPU can be accessed by the GPU, and the GPU can obtain data therein and send it to the CPU. The memory 304 can store an operating system and other application programs, and when the technical solutions provided in the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 304 and called and executed by the processor 302.

[0080] The input / output interface 306 is configured to connect an input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0081] The communication interface 308 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, Wi-Fi, Bluetooth, etc.).

[0082] The bus 310 includes a channel for transmitting information between various components (such as the processor 302, the memory 304, the input / output interface 306, and the communication interface 308) of the device.

[0083] It should be noted that although the above device only shows the processor 302, the memory 304, the input / output interface 306, the communication interface 308, and the bus 310, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include components necessary for implementing the embodiments of the present application, and does not necessarily include all the components shown in the figure.

[0084] The embodiments of the present application provide a computer program product, including a computer program, which is executed by a processor to implement the method described in any of the embodiments of the present application.

[0085] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method in any of the foregoing embodiments.

[0086] Computer readable media includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computer device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carriers.

[0087] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, the device embodiments are described more simply because they are basically similar to the method embodiments, and the relevant parts can be referred to the part of the method embodiments. The device embodiments described above are only illustrative, and the modules described as separate components can be or can not be physically separated, and the functions of the modules can be realized in one or more software and / or hardware in the implementation of the embodiments of the present application. Part or all of the modules can be selected to achieve the purpose of the embodiments of the present application according to the actual needs. Those skilled in the art can understand and implement without creative labor.

[0088] The above is only a specific implementation of the embodiments of the present application. It should be noted that for those skilled in the art, without departing from the principles of the embodiments of the present application, several improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the embodiments of the present application.

Claims

1. A method for calculating an attention score based on a CPU, the method comprising: Obtain user queries for input into the large language model; Retrieving information related to the user query and obtaining a key value corresponding to the information from a storage medium; The storage medium is different from the GPU memory, and the key value is a key value pre-calculated based on the large language model; When a preset condition is met, the CPU performs attention calculation based on the key value to obtain an attention score, and sends the attention score to the GPU, so that the GPU uses the attention score to generate response information for the user query; The preset condition includes: an estimated calculation time of the CPU performing attention calculation is less than an estimated transmission time of the CPU transmitting the key value to the GPU.

2. The method according to claim 1, further comprising: Determining whether the estimated computation time of the CPU performing the attention computation is less than the estimated transmission time of the CPU transmitting the key value to the GPU based on at least one of the following conditions: the length of the information; resource utilization of the CPU; The data transmission bandwidth between the CPU and the GPU.

3. The method according to claim 1, further comprising: If the estimated calculation time of the CPU for attention calculation is greater than or equal to the estimated transmission time of the CPU for transmitting the key value to the GPU, the CPU sends the key value to the GPU so that the GPU performs attention calculation based on the key value to obtain the attention score.

4. The method according to claim 1, wherein the CPU performs attention calculation based on the key value to obtain an attention score, comprising: The CPU calls multiple threads to perform attention calculation based on the key value to obtain an attention score.

5. The method according to claim 1, wherein the CPU performs attention calculation based on the key value to obtain an attention score, comprising: The CPU obtains the query vector, key vector, and value vector sent by the GPU; The CPU fuses the key vector sent by the GPU with the key vector in the key value to obtain a fused key vector, and fuses the value vector sent by the GPU with the value vector in the key value to obtain a fused value vector; The CPU performs attention calculation based on the query vector, the fusion key vector, and the fusion value vector to obtain an attention score.

6. The method according to claim 5, wherein the GPU and the CPU interact to obtain the response information through multiple rounds of iterations, and during each iteration, the CPU performs an attention calculation based on the query vector, key vector, and value vector sent by the GPU during the current iteration to obtain an attention score for the current iteration; in, The query vector, key vector, and value vector sent by the GPU during the first iteration are generated based on the user query, or based on historical output word units of the large language model; The query vector, key vector, and value vector sent by the GPU during the i-th iteration are generated based on the attention scores obtained during the i-1-th iteration; 1<i<N, N is the preset maximum number of iterations; The attention score obtained in the Nth iteration is used to generate response information for the user query.

7. The method according to claim 1, before the CPU performs attention calculation based on the key value to obtain the attention score, the method further comprises: Obtaining position encoding information of each word in the information; The position encoding information is integrated into the key value.

8. A method for calculating an attention score based on a CPU, the method comprising: When a preset condition is met, the GPU receives an attention score sent by the CPU, where the attention score is obtained by the CPU performing an attention calculation based on a key value, the key value is obtained by the CPU from a storage medium, and the key value corresponds to information retrieved by the CPU; the information is related to a user query used to input a large language model, the storage medium is different from the GPU video memory, and the key value is a key value pre-calculated based on the large language model; the preset condition includes: an estimated calculation time for the CPU to perform the attention calculation is less than an estimated transmission time for the CPU to transmit the key value to the GPU; The GPU generates response information for the user query based on the attention score.

9. The method according to claim 8, further comprising: The GPU receives the key value sent by the CPU when the preset condition is not met; The GPU performs attention calculation based on the key value to obtain the attention score.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

12. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images