Method and device for generating reasoning result, storage medium and electronic device

Abstract

The application discloses a reasoning result generation method and device, a storage medium and an electronic device, relates to the technical field of natural language processing, and the method is applied to a reasoning engine and includes the following steps: determining a first word element and a second word element included in a first reasoning prompt word in a reasoning request; obtaining a first reasoning vector corresponding to the first word element from a cache space, the first reasoning vector being a reasoning vector corresponding to the first word element in a second reasoning prompt word; converting the first reasoning vector into a second reasoning vector according to a first position of the first word element in the first reasoning prompt word and a second position of the first word element in the second reasoning prompt word; calculating a reasoning vector of the second word element in the first reasoning prompt word to obtain a third reasoning vector; and generating a first reasoning result according to the second reasoning vector and the third reasoning vector and sending the first reasoning result to a target object. The technical problem of low reasoning efficiency of the reasoning engine is solved, and the technical effect of improving the reasoning efficiency of the reasoning engine is achieved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method and apparatus for generating reasoning results, a storage medium, and an electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models are widely used in various scenarios such as text generation, dialogue systems, and content understanding. However, large language models need to be pre-trained on massive amounts of text data to obtain powerful language understanding and generation capabilities. Due to the huge number of model parameters, ranging from billions to trillions, the inference process of the model consumes a lot of computational resources and has high computational complexity.

[0003] To improve the inference efficiency of large models, Key-Value Cache (KV Cache) technology has been proposed and applied to the inference engine of large language models to optimize the computation process of the self-attention mechanism. KV Cache avoids redundant computation by storing intermediate results (K and V vectors, key and value vectors) of the attention layer, thus significantly improving the efficiency of long sequence generation. However, when the same token (lexical) is in different positions in different requests, existing KV Cache technology cannot reuse the cached KV vector for that token. This necessitates recalculating the KV vector for that token in new requests, resulting in lower inference efficiency for the inference engine.

[0004] This demonstrates that the inference engine in the relevant technology suffers from low inference efficiency. Summary of the Invention

[0005] This application provides a method and apparatus for generating inference results, a storage medium and an electronic device, to at least solve the problem of low inference efficiency of inference engines in related technologies.

[0006] This application provides a method for generating inference results, applied to an inference engine, comprising: receiving an inference request initiated by a target object, wherein the inference request includes a first inference prompt word input by the target object, the first inference prompt word including a first word element and a second word element, the first word element representing a word element of a cached inference vector in a cache space, and the second word element representing a word element of a cached inference vector in the cache space; obtaining a first inference vector corresponding to the first word element from the cache space, wherein the first inference vector is the inference vector corresponding to the first word element in a second inference prompt word; transforming the first inference vector according to a first position and a second position to obtain a second inference vector, wherein the first position represents the position of the first word element in the first inference prompt word, and the second position represents the position of the first word element in the second inference prompt word; calculating the inference vector of the second word element in the first inference prompt word to obtain a third inference vector; generating a first inference result according to the second inference vector and the third inference vector, and sending the first inference result to the target object.

[0007] This application also provides an apparatus for generating inference results, comprising: a receiving module for receiving an inference request initiated by a target object, wherein the inference request includes a first inference prompt word input by the target object, the first inference prompt word including a first word element and a second word element, the first word element representing a word element of a cached inference vector in a cache space, and the second word element representing a word element of a cached inference vector in the cache space; an obtaining module for obtaining a first inference vector corresponding to the first word element from the cache space, wherein the first inference vector is an inference vector corresponding to the first word element in a second inference prompt word; a conversion module for converting the first inference vector according to a first position and a second position to obtain a second inference vector, wherein the first position represents the position of the first word element in the first inference prompt word, and the second position represents the position of the first word element in the second inference prompt word; a calculation module for calculating the inference vector of the second word element in the first inference prompt word to obtain a third inference vector; and a generation module for generating a first inference result according to the second inference vector and the third inference vector, and sending the first inference result to the target object.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of generating any of the above-described reasoning results.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method for generating any of the above-described inference results.

[0010] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of generating any of the above-described reasoning results.

[0011] This application determines a first inference hint word from the inference request initiated by the target object. The first inference hint word includes a first word element that exists in the cache and a second word element that does not exist in the cache. A first inference vector corresponding to the first word element is obtained from the cache space. The first inference vector is the inference vector corresponding to the first word element in the second inference hint word. The first inference vector is converted into a second inference vector based on the first position of the first word element in the first inference hint word and the second position of the first word element in the second inference hint word. The inference vector of the second word element in the first inference hint word is calculated to obtain a third inference vector. A first inference result is generated based on the second and third inference vectors and sent to the target object. Using this scheme, the reuse rate of cached inference vectors in the inference engine can be improved, thereby reducing the computational load of the inference engine. This solves the technical problem of low inference efficiency in the inference engine and achieves the technical effect of improving the inference efficiency of the inference engine. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of a method for generating reasoning results according to an embodiment of this application;

[0014] Figure 2 This is a flowchart illustrating an optional method for generating reasoning results according to an embodiment of this application;

[0015] Figure 3 This is a schematic diagram of an optional inference system according to an embodiment of this application;

[0016] Figure 4 This is a structural block diagram of an optional inference result generation device according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] According to one aspect of the embodiments of this application, a method for generating inference results is provided. Optionally, in this embodiment, the above-described method for generating inference results may be applied, but is not limited to, to applications such as... Figure 1 The hardware environment shown includes terminal device 102 and server 104. Server 104 can be connected to terminal device 102 via a network and can be used to provide services (e.g., application services, etc.) to terminal device 102 or clients installed on terminal device 102. A database can be set up on server 104 or independently of server 104 to provide data storage services for server 104.

[0021] The aforementioned network may include, but is not limited to, at least one of the following: wired network and wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network (WAN), metropolitan area network (MAN), and local area network (LAN). The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity) and Bluetooth. Terminal device 102 may be, but is not limited to, PC (Personal Computer), mobile phone, tablet computer, etc. Server 104 may be, but is not limited to, cloud server, server cluster, or other server types.

[0022] The method for generating the reasoning result in this embodiment can be executed by server 104, terminal device 102, or jointly by server 104 and terminal device 102. Alternatively, the method for generating the reasoning result in this embodiment can be executed by a client installed on terminal device 102.

[0023] Taking the generation method of inference results in this embodiment executed by terminal device 102 as an example, here, terminal device 102 can be a physical host, the generation method of inference results in this embodiment is applied to the physical host, the memory that the physical host can call is divided into multiple memory levels, one of the multiple memory levels contains at least one type of memory, and the multiple memory levels include the first memory level corresponding to the physical memory of the physical host. Here, physical hosts can be enterprise-level servers, cluster servers, office computers, embedded devices, or other physical devices that serve as underlying hardware support in a virtualization environment. Physical memory, on the other hand, is a physical memory module directly connected to the host hardware. It is the core and foundation of the memory architecture. Physical memory is usually composed of Dynamic Random Access Memory (DRAM), which has extremely fast read and write speeds and can respond to the processor's memory access requests with nanosecond-level response times. This makes physical memory suitable for carrying the core code of the virtual machine operating system, frequently called system function libraries, and critical process data that are running at high speeds. For example, at the beginning of virtual machine startup, the operating system kernel needs to quickly load and initialize various hardware drivers and establish a basic system operating environment. At this time, physical memory can complete data read and write operations with extremely high efficiency, ensuring that the virtual machine can start quickly and stably. During the operation of the virtual machine, application parts that have extremely demanding requirements for memory read and write performance, such as the transaction processing module of the database management system and the real-time rendering engine, also rely on physical memory to ensure their efficient operation, thereby maintaining the smoothness and responsiveness of the entire virtual machine system.

[0024] Figure 2 This is a flowchart illustrating an optional method for generating inference results according to an embodiment of this application, as shown below. Figure 2 As shown, the process of this method may include the following steps:

[0025] Step S202: Receive a reasoning request initiated by the target object, wherein the reasoning request includes a first reasoning prompt word input by the target object. The first reasoning prompt word includes a first word element and a second word element. The first word element represents a word element of a cached reasoning vector that exists in the cache space, and the second word element represents a word element of a reasoning vector that does not exist in the cache space.

[0026] Optionally, in step S202 above, for example, the first prompt word is "give me a joke", which includes multiple tokens such as "give", "me", "tell", "a", and "joke".

[0027] Step S204: Obtain the first inference vector corresponding to the first word element from the cache space, wherein the first inference vector is the inference vector corresponding to the first word element in the second inference prompt word;

[0028] Optionally, in step S204 above, for example, if the historical inference request contains the inference prompt word "What are some jokes about pandas", the inference vector of the word "joke" has already been cached in the cache space.

[0029] Step S206: The first inference vector is transformed according to the first position and the second position to obtain the second inference vector, wherein the first position represents the position of the first word element in the first inference prompt word, and the second position represents the position of the first word element in the second inference prompt word;

[0030] Optionally, in step S206 above, when the same word "joke" appears in different positions in the prompt words, the calculated inference vector is also different. The position of the word cannot be ignored and the inference vector of the word cannot be directly reused. Therefore, the cached inference vector needs to be converted according to the position of the word in different prompt words before it can be reused.

[0031] Step S208: Calculate the inference vector of the second word in the first inference prompt to obtain the third inference vector;

[0032] Optionally, in step S208 above, if there is no cached word in the first inference prompt, then the inference vector of that word needs to be fully calculated.

[0033] Step S210: Generate a first inference result based on the second inference vector and the third inference vector, and send the first inference result to the target object.

[0034] Optionally, in step S210 above, after obtaining the inference vectors of all words in the first inference prompt, the inference result of the first inference prompt can be determined and sent to the user to respond to the inference request.

[0035] Through the above steps, a first inference hint word is determined from the inference request initiated by the target object. This first inference hint word includes a cached first term and a cached second term. A first inference vector corresponding to the first term is retrieved from the cache space; this first inference vector is the inference vector corresponding to the first term in the second inference hint word. The first inference vector is converted into a second inference vector based on the first position of the first term in the first inference hint word and the second position of the first term in the second inference hint word. The inference vector of the second term in the first inference hint word is calculated to obtain a third inference vector. A first inference result is generated based on the second and third inference vectors and sent to the target object. This scheme improves the reuse rate of cached inference vectors in the inference engine, thereby reducing the computational load. It solves the technical problem of low inference efficiency in the inference engine and achieves the technical effect of improving the inference efficiency of the inference engine.

[0036] In an exemplary embodiment, the first inference vector includes a first key vector and a first value vector. Transforming the first inference vector according to a first position and a second position to obtain a second inference vector includes: comparing the first position and the second position to obtain a comparison result; and determining the first key vector and the first value vector as the second inference vector if the comparison result indicates that the first position and the second position are the same.

[0037] Optionally, in the above embodiments, the inference vector, i.e., the KV Cache, includes a K vector (key vector) and a V vector (value vector). For example, if user A asks, "Tell me a joke," and user B immediately follows with, "Tell me a joke, about pandas," the inference engine finds that the beginning of user B's prompt ("Tell me a joke") perfectly matches user A's prompt. Therefore, during inference, user B can directly reuse the K and V values ​​cached by user A for the "Tell me a joke" part of the prompt, only needing to additionally calculate the K and V values ​​for the "about pandas" part.

[0038] In one exemplary embodiment, the method further includes: determining a position offset matrix based on the first position and the second position when it is determined that the comparison result indicates that the first position and the second position are different; determining a second key vector based on the first key vector and the position offset matrix; determining a second value vector based on the first value vector and the position offset matrix; and determining the second key vector and the second value vector as a second inference vector.

[0039] Optionally, in the above embodiments, for example, after users A and B ask questions, user C asks, "What are some jokes about pandas?" Although both "jokes" and "pandas" have appeared in the cache (in users A and B's requests), they appear in different positions and in different orders, so the KV caches for "jokes" and "pandas" cannot be directly reused. Therefore, the KV caches for "jokes" and "pandas" need to be converted.

[0040] Optionally, in the above embodiments, the main reason for limiting KV cache reuse is the different position embeddings in the KV vectors. The most mainstream position embedding currently is RoPE (Rotary Position Embedding). RoPE does not assign a fixed encoding value to each position, but rather incorporates position information by applying a rotation matrix to the token vector itself.

[0041] RoPE has a mathematical property: the attention score between a word at position m with its K-vector infused with RoPE position information and the K-vector of the same word at position n depends only on the relative position difference (nm) of the word, not on the absolute positions m or n. That is, if the K-vector is calculated when a word is at position m... m Then, through mathematical transformations, we can calculate k′ for the same word at position n. n This transformation is achieved by multiplying by a rotation matrix R representing the position difference (nm). n-m The same applies to vector V. The specific conversion formula is as follows:

[0042] Due to the rotation matrix R k R v It is an approximate diagonal matrix, and its computational complexity is O(d), while based on the original input x m The computational complexity from scratch is at least O(ld). It should be noted that this embodiment uses RoPE as an example to illustrate the location conversion calculation process of KV Cache, but the above conversion process can also be applied to other location vector algorithms.

[0043] Through the above embodiments, by performing position transformation calculations on local KV Cache blocks, this application can reuse existing K and V vectors in different requests, significantly reducing redundant calculations, thereby improving the inference efficiency of the inference engine and the response speed of large language models.

[0044] In an exemplary embodiment, calculating the inference vector of the second word element in the first inference prompt word to obtain the third inference vector includes: calculating the key vector corresponding to the second word element and the value vector corresponding to the second word element using the following formula: Km =R m ×W k ×x m V m =R m ×W v ×x m Among them, K m V represents the key vector corresponding to the second word element. m R represents the value vector corresponding to the second word element. m W represents the position matrix corresponding to the position m of the second word in the first inference prompt. k W represents the weight matrix that generates the key vectors. v The weight matrix x represents the generated value vector. m This represents the embedding vector corresponding to the second word element; the key vector and the value vector corresponding to the second word element are determined as the inference vector of the second word element.

[0045] Optionally, in the above embodiments, R m This is the RoPE rotation position vector corresponding to position m, specifically expressed as:

[0046]

[0047] Where d represents the dimension of vectors K and V, θ0, θ1, ..., θ d / 2-1 These are constants for the inference model.

[0048] In an exemplary embodiment, generating a first inference result based on a second inference vector and a third inference vector includes: determining the vector concatenation order of the second and third inference vectors based on the word combination order in the first inference prompt; determining the positional order of the second and third inference vectors based on the positions of the first and second words in the inference prompt; concatenating the second and third inference vectors according to the vector concatenation order to obtain a concatenated vector; and decoding the concatenated vector to obtain the first inference result.

[0049] Optionally, in the above embodiment, taking the prompt "What are some jokes about pandas?" as an example, assuming that "panda" and "jokes" are cached, and the cache positions are 5 and 3 respectively, the target position of "panda" is 0, the original position is 5, the target position of "of" is 1, the target position of "jokes" is 2, the original position is 3, the target position of "have" is 3, the target position of "which" is 4, and the target position of "?" is 5. The inference engine converts or calculates the K-vectors and V-vectors of all tokens at the target positions, and then concatenates them into the complete KV Cache corresponding to the prompt. The inference result can be obtained by decoding the complete KV Cache corresponding to the prompt.

[0050] In an exemplary embodiment, after generating a first inference result based on a second inference vector and a third inference vector, the method further includes: determining a first index based on the hash value of a first word, and saving the first index, the second inference vector, and the first position to a cache space; and / or, determining a second index based on the hash value of a second word, and saving the second index, the third inference vector, and the third position to a cache space, wherein the third position represents the position of the second word in the first inference prompt.

[0051] Optionally, in the above embodiments, the caching and retrieval of the KV Cache can be implemented using a hash table. For the newly generated KV Cache, a hash value can be calculated for each token in the prompt term as the key of the hash table index structure (unrelated to the key in the KV Cache), and the position of the token can be used as the value of the hash table index structure (unrelated to the value in the KV Cache). During retrieval, each token in the prompt term is hash-retrieved separately.

[0052] The above embodiments demonstrate that searching with a single token results in a high hit rate and effectively improves cache utilization.

[0053] Optionally, in the above embodiments, the hash value of the input string between the start and end points can be calculated using the position of any token in the prompt word as the start and end point (the end point must be after the start point) as the key of the hash table index structure, and the token positions of the start and end points can be used as the value of the hash table index structure. During retrieval, hash retrieval is performed using the complete input prompt word.

[0054] Through the above embodiments, the entire input request prompt can be retrieved in a single hash search, resulting in low retrieval time complexity and effectively improving retrieval speed.

[0055] In practical applications, you can choose to combine two retrieval methods depending on the reasoning environment and needs. First, search the entire input prompt. If no match is found, then search for each token. Alternatively, you can search for a certain number of tokens.

[0056] In an exemplary embodiment, after receiving the inference request initiated by the target object, the method further includes: matching cached inference prompts from the cache space according to the first inference prompt to obtain a matching result; if it is determined that the matching result indicates that the first inference prompt is matched in the cached inference prompts, generating a second inference result according to the cached content corresponding to the first inference prompt, wherein the cached content includes inference vectors corresponding to all tokens in the first inference prompt; and sending the second inference result to the target object.

[0057] Through the above embodiments, it is possible to reuse cached KV caches of any position and length (single token, multiple tokens, or the entire prompt word) to any position in a new request, thereby greatly improving the overall reasoning efficiency of large language models.

[0058] The following describes an optional method for generating inference results according to an embodiment of this application, with reference to optional embodiments. In one optional embodiment, the method for generating inference results can be achieved through methods such as... Figure 3 The inference system implementation shown is as follows: Figure 3 As shown, the inference system includes an online inference service, a KV Cache management center, a backend inference decoding engine, and a backend inference pre-filling engine.

[0059] The online inference service provides users with an API interface to retrieve or save data from the KV Cache based on user requests, call the backend inference engine to perform inference based on the retrieval results, and encapsulate the inference results returned by the inference engine to respond to the user.

[0060] The KV Cache Management Center is responsible for the management and use of the KV Cache, and needs to maintain KV Cache cache block data and KV Cache cache index data. The KV Cache cache block data stores the KV vector results. In a model inference service, the dimension of the KV vector is fixed, so its address is aligned. The specific KV vector cache can be retrieved through the global counting address. In addition, the KV Cache cache block data also stores the position value of the KV vector to provide the relative position for transformation in KV Cache transformation calculations. The KV Cache cache index data stores the mapping between the hash value of the input prompt and the corresponding global counting address (including the starting address) of the KV vector cache. It can return the corresponding KV cache vector in O(1) constant time complexity after hash calculation based on the input.

[0061] The KV Cache management center can use cache eviction algorithms such as LRU (Least Recently Used) for updates, and can also be extended to use other algorithms. In this embodiment, when a retrieval miss occurs, the KV Cache management center calls the backend inference pre-filling engine to calculate the KV vector result from scratch through the pre-filling process. This result is then stored in the KV Cache according to the cache management strategy. The system also iterates through the (0, n) positions to calculate the hash value of the corresponding block for the input prompt word and stores it in the index data. For each new KV cache block containing an input prompt word, (n+1)n / 2 new indices are added, including blocks with the same starting position, i.e., blocks with a single token. It should be noted that the pre-filling calculation process for a missed input is synchronous and serial, but the calculation of the index data can be asynchronous and parallel, without affecting the overall time complexity of the inference process.

[0062] The backend inference pre-filling engine is responsible for executing the computational logic of the pre-filling stage of large language model inference.

[0063] The backend inference and decoding engine is responsible for executing the computational logic of the inference and decoding phase of the large language model.

[0064] In an optional embodiment, the specific process of the above-described inference system in executing the method for generating inference results includes the following steps:

[0065] 1. A user initiates a reasoning request with a set of prompts, and the online reasoning service receives the user's request.

[0066] 2. The online inference service calls the KV Cache management center to retrieve complete prompt words.

[0067] 3. The KV Cache management center receives the request and performs a cache retrieval in the index.

[0068] 3.1 The KV Cache management center first performs a full-text search, which involves hashing the entire request prompt and performing a hit check. If a hit is found, the hit KV cache block and its original location are recorded, and step 4 is executed. If no hit is found, step 3.2 is executed.

[0069] 3.2 The KV Cache management center performs a hit check on each token for the request prompt. For tokens that do not hit, the backend pre-filling engine is called to calculate the corresponding K and V vectors from scratch. The position vector calculation is not superimposed, but the corresponding original position is recorded as 0. The KV cache block and the corresponding original position are recorded for the hit tokens. At present, all tokens have corresponding K and V vectors and original positions.

[0070] 4. Based on the KV vector and position returned in step 3, calculate the KV vector result corresponding to the position of the token in the current prompt word using the conversion formula in the above embodiment, and return it to the online inference service.

[0071] 4.1 Asynchronously, perform KV Cache update according to the cache update strategy.

[0072] 5. The online inference service retrieves the pre-filled key-value (KV) cache corresponding to the prompt words and sends it to the backend inference engine for decoding and inference. The KV vectors generated during the inference process can also be updated to the KV cache management center according to a certain strategy.

[0073] 6. The backend inference engine decodes and infers token by token according to its position, and returns the inference results to the online inference service.

[0074] 7. The online reasoning service returns the reasoning results to the user.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (e.g., read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0076] According to another aspect of the embodiments of this application, an apparatus for generating inference results is also provided. This apparatus can be used to implement the method for generating inference results provided in the above embodiments, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0077] Figure 4 This is a structural block diagram of an optional inference result generation device according to an embodiment of this application, such as... Figure 4 As shown, the apparatus for generating the reasoning result includes:

[0078] The receiving module 40 is used to receive a reasoning request initiated by the target object, wherein the reasoning request includes a first reasoning prompt word input by the target object, the first reasoning prompt word includes a first word element and a second word element, the first word element represents a word element of a cached reasoning vector in the cache space, and the second word element represents a word element of a cached reasoning vector in the cache space.

[0079] The acquisition module 42 is used to acquire the first inference vector corresponding to the first word element from the cache space, wherein the first inference vector is the inference vector corresponding to the first word element in the second inference prompt word;

[0080] The conversion module 44 is used to convert the first inference vector according to the first position and the second position to obtain the second inference vector, wherein the first position represents the position of the first word in the first inference prompt word, and the second position represents the position of the first word in the second inference prompt word;

[0081] Calculation module 46 is used to calculate the inference vector of the second word element in the first inference prompt word to obtain the third inference vector;

[0082] The generation module 48 is used to generate a first inference result based on the second inference vector and the third inference vector, and send the first inference result to the target object.

[0083] The inference result generation apparatus provided in this application determines a first inference prompt word from the inference request initiated by the target object. The first inference prompt word includes a first term that exists in the cache and a second term that does not exist in the cache. A first inference vector corresponding to the first term is obtained from the cache space; this first inference vector is the inference vector corresponding to the first term in the second inference prompt word. The first inference vector is converted into a second inference vector based on the first position of the first term in the first inference prompt word and the second position of the first term in the second inference prompt word. The inference vector of the second term in the first inference prompt word is calculated to obtain a third inference vector. A first inference result is generated based on the second and third inference vectors and sent to the target object. By adopting the above scheme, the reuse rate of cached inference vectors by the inference engine can be improved, thereby reducing the computational load of the inference engine. This solves the technical problem of low inference efficiency of the inference engine and achieves the technical effect of improving the inference efficiency of the inference engine.

[0084] In an exemplary embodiment, the first inference vector includes a first key vector and a first value vector. The conversion module 44 is further configured to compare the first position and the second position to obtain a comparison result; if the comparison result indicates that the first position and the second position are the same, the first key vector and the first value vector are determined as the second inference vector.

[0085] In an exemplary embodiment, the conversion module 44 is further configured to, when determining that the comparison result indicates that the first position and the second position are different, determine a position offset matrix based on the first position and the second position; determine a second key vector based on the first key vector and the position offset matrix; determine a second value vector based on the first value vector and the position offset matrix; and determine the second key vector and the second value vector as a second inference vector.

[0086] In an exemplary embodiment, the calculation module 46 is configured to calculate the key vector corresponding to the second word and the value vector corresponding to the second word using the following formula: K m =R m ×W k ×x m V m =R m ×W v ×x m Among them, K m V represents the key vector corresponding to the second word element. m R represents the value vector corresponding to the second word element. m W represents the position matrix corresponding to the position m of the second word in the first inference prompt. k W represents the weight matrix that generates the key vectors. v The weight matrix x represents the generated value vector. m This represents the embedding vector corresponding to the second word element; the key vector and the value vector corresponding to the second word element are determined as the inference vector of the second word element.

[0087] In an exemplary embodiment, the generation module 48 is configured to determine the vector concatenation order of the second inference vector and the third inference vector based on the word combination order in the first inference prompt word; determine the positional order of the second inference vector and the inference vector based on the positions of the first word and the second word in the inference prompt word; concatenate the second inference vector and the third inference vector according to the vector concatenation order to obtain a concatenated vector; and decode the concatenated vector to obtain the first inference result.

[0088] In an exemplary embodiment, the generation module 48 is configured to determine a first index based on the hash value of a first word, and save the first index, a second inference vector, and a first position to a cache space; and / or, determine a second index based on the hash value of a second word, and save the second index, a third inference vector, and a third position to a cache space, wherein the third position represents the position of the second word in the first inference prompt.

[0089] In an exemplary embodiment, the apparatus is further configured to match cached inference prompts from a cache space based on a first inference prompt to obtain a matching result; if it is determined that the matching result indicates that the first inference prompt is matched in the cached inference prompts, generate a second inference result based on the cached content corresponding to the first inference prompt, wherein the cached content includes inference vectors corresponding to all lexical units in the first inference prompt; and send the second inference result to a target object.

[0090] For a description of the features in the embodiment corresponding to the above-mentioned device for generating the reasoning result, please refer to the relevant description of the embodiment corresponding to the method for generating the reasoning result, which will not be repeated here.

[0091] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described methods for generating inference results.

[0092] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described methods for generating inference results when it is run.

[0093] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0094] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described methods for generating inference results.

[0095] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described methods for generating inference results.

[0096] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] The foregoing has provided a detailed description of the method, apparatus, storage medium, and electronic device of a distributed storage system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for generating reasoning results, characterized in that, Applied to inference engines, including: Receive a reasoning request initiated by a target object, wherein the reasoning request includes a first reasoning prompt word input by the target object, the first reasoning prompt word includes a first word element and a second word element, the first word element represents a word element of a cached reasoning vector in the cache space, and the second word element represents a word element of a cached reasoning vector in the cache space. Obtain the first inference vector corresponding to the first word in the cache space, wherein the first inference vector is the inference vector corresponding to the first word in the second inference prompt word; The first inference vector is transformed based on the first position and the second position to obtain the second inference vector, wherein the first position represents the position of the first word in the first inference prompt word, and the second position represents the position of the first word in the second inference prompt word; Calculate the inference vector of the second word element in the first inference prompt to obtain the third inference vector; A first inference result is generated based on the second inference vector and the third inference vector, and the first inference result is sent to the target object; The first inference vector includes a first key vector and a first value vector. The second inference vector is obtained by transforming the first inference vector based on the first position and the second position, including: The first position and the second position are compared to obtain the comparison result; If the comparison result indicates that the first position and the second position are the same, the first key vector and the first value vector are determined as the second inference vector; If the comparison result indicates that the first position and the second position are not the same, a position offset matrix is ​​determined based on the first position and the second position; The second key vector is determined based on the first key vector and the position offset matrix, and the second value vector is determined based on the first value vector and the position offset matrix; The second key vector and the second value vector are determined as the second inference vector; Specifically, the inference vector of the second word element in the first inference prompt is calculated to obtain the third inference vector, which includes: The key vector and value vector corresponding to the second word element are calculated using the following formulas: ； ； in, This represents the key vector corresponding to the second word element. This represents the value vector corresponding to the second word element. Indicates the position of the second word in the first inference prompt. The corresponding position matrix, The weight matrix represents the generated key vector. The weight matrix represents the generated value vector. This represents the embedding vector corresponding to the second word element; The key vector and value vector corresponding to the second word are determined as the inference vector of the second word.

2. The method for generating reasoning results according to claim 1, characterized in that, Generate a first inference result based on the second inference vector and the third inference vector, including: The vector concatenation order of the second inference vector and the third inference vector is determined based on the word combination order in the first inference prompt word; The positional order of the second inference vector and the inference vector is determined based on the positions of the first lexical unit and the second lexical unit in the inference prompt word; The second inference vector and the third inference vector are concatenated according to the vector concatenation order to obtain the concatenated vector; The concatenated vector is decoded to obtain the first inference result.

3. The method for generating reasoning results according to claim 1, characterized in that, After generating the first inference result based on the second inference vector and the third inference vector, the method further includes: The first index is determined based on the hash value of the first word, and the first index, the second inference vector, and the first position are saved to the cache space. And / or, determine a second index based on the hash value of the second term, and save the second index, the third inference vector, and the third position to the cache space, wherein the third position represents the position of the second term in the first inference prompt word.

4. The method for generating reasoning results according to claim 1, characterized in that, After receiving the inference request initiated by the target object, the method further includes: Based on the first inference prompt, the cached inference prompt is matched with the cached inference prompt in the cache space to obtain the matching result; If the matching result indicates that the first inference prompt is matched in the cached inference prompt, a second inference result is generated based on the cached content corresponding to the first inference prompt, wherein the cached content includes the inference vectors corresponding to all lexical units in the first inference prompt; The second reasoning result is sent to the target object.

5. A device for generating reasoning results, characterized in that, include: A receiving module is used to receive a reasoning request initiated by a target object, wherein the reasoning request includes a first reasoning prompt word input by the target object, the first reasoning prompt word includes a first word element and a second word element, the first word element represents a word element of a cached reasoning vector in the cache space, and the second word element represents a word element of a cached reasoning vector in the cache space. The acquisition module is used to acquire the first inference vector corresponding to the first word element from the cache space, wherein the first inference vector is the inference vector corresponding to the first word element in the second inference prompt word; The conversion module is used to convert the first inference vector according to the first position and the second position to obtain the second inference vector, wherein the first position represents the position of the first word in the first inference prompt word, and the second position represents the position of the first word in the second inference prompt word; The calculation module is used to calculate the inference vector of the second word element in the first inference prompt word to obtain the third inference vector; A generation module is configured to generate a first inference result based on the second inference vector and the third inference vector, and send the first inference result to the target object; Wherein, the first inference vector includes a first key vector and a first value vector, and the conversion module is further used to compare the first position and the second position to obtain a comparison result; If the comparison result indicates that the first position and the second position are the same, the first key vector and the first value vector are determined as the second inference vector; If the comparison result indicates that the first position and the second position are not the same, a position offset matrix is ​​determined based on the first position and the second position; The second key vector is determined based on the first key vector and the position offset matrix, and the second value vector is determined based on the first value vector and the position offset matrix; The second key vector and the second value vector are determined as the second inference vector; The calculation module is further configured to calculate the key vector and the value vector corresponding to the second word element using the following formulas: ； ； in, This represents the key vector corresponding to the second word element. This represents the value vector corresponding to the second word element. Indicates the position of the second word in the first inference prompt. The corresponding position matrix, The weight matrix represents the generated key vector. The weight matrix represents the generated value vector. This represents the embedding vector corresponding to the second word element; The key vector and value vector corresponding to the second word are determined as the inference vector of the second word.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the method for generating the reasoning result as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the method for generating the reasoning result as described in any one of claims 1 to 4.

Images