Context vector generation method, electronic device, storage medium, and program product

Abstract

The application discloses a context vector generation method, an electronic device, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: firstly, obtaining a word merging rule and target input data, dividing the target input data into a plurality of initial words, and obtaining context vectors corresponding to the plurality of initial words respectively and word probability distribution information corresponding to the positions of the initial words respectively. Then, at least one merged word is generated by merging the plurality of initial words based on the word merging rule, thereby reducing the number of words. Finally, the context vector of the target merged word is generated according to the context vectors of the candidate initial words included in the target merged word and the word probability distribution information of the positions of the candidate initial words. In this way, the merged word is obtained after the initial words are merged, the context vector of the merged word is calculated, and the inference operation is performed by using the calculated context vector, and the words are not discarded, so that the inference quality can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to methods for generating context vectors, electronic devices, storage media, and program products. Background Technology

[0002] In the field of artificial intelligence technology, when reasoning models perform contextual reasoning, they usually need to generate context vectors to execute reasoning tasks. In this process, context vectors will consume a lot of GPU memory resources.

[0003] Currently, to reduce the consumption of video memory resources, context vectors are generally adjusted from high precision to low precision, or some context vectors are discarded. For example, by evaluating the importance of each context vector, less important context vectors are discarded, and only the context vectors of the most recent N words are retained. This leads to a deterioration in inference quality. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, storage medium, and program product for generating context vectors to address the problem of deteriorating inference quality.

[0005] This application provides a method for generating context vectors, including:

[0006] Obtain the word merging rules and the target input data corresponding to the target inference task;

[0007] The target input data is divided into multiple initial words, and the context vectors corresponding to the multiple initial words and the word probability distribution information corresponding to the word positions of the multiple initial words are obtained respectively. The word positions are used to indicate the position of the initial words in the target input data.

[0008] Based on the word merging rules, multiple initial words are merged to generate at least one merged word.

[0009] Based on the context vectors corresponding to the multiple candidate initial words included in the target merging word element, and the word element probability distribution information corresponding to the candidate word element positions of the multiple candidate initial words, a context vector of the target merging word element is generated to perform inference operations in the target inference task. The target merging word element is any one of the at least one merging word elements.

[0010] This application also provides a context vector generation apparatus, comprising:

[0011] The acquisition module is used to acquire word merging rules and target input data corresponding to the target inference task; it divides the target input data into multiple initial words and acquires the context vectors corresponding to the multiple initial words and the word probability distribution information corresponding to the word positions of the multiple initial words, wherein the word positions are used to indicate the position of the initial words in the target input data;

[0012] The merging module is used to merge multiple initial lexical units based on lexical unit merging rules to generate at least one merged lexical unit.

[0013] The generation module is used to generate the context vector of the target merging word element based on the context vectors corresponding to the multiple candidate initial word elements included in the target merging word element, and the word element probability distribution information corresponding to the candidate word element positions of the multiple candidate initial word elements, so as to perform inference operations in the target inference task. The target merging word element is any one of the at least one merging word elements.

[0014] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing any of the above-described methods for generating context vectors when executing the computer program.

[0015] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described methods for generating context vectors.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described methods for generating context vectors.

[0017] This application first obtains the word merging rules and target input data. Then, the target input data can be divided into multiple initial words, and the context vectors corresponding to each initial word and the word probability distribution information corresponding to their respective word positions can be obtained. Since the number of initial words is large, the number of corresponding context vectors is also large. Therefore, this scheme designs a word merging rule, which can merge multiple initial words to generate at least one merged word, reducing the number of words. Furthermore, based on the context vectors of the multiple candidate initial words included in the target merged word and the word probability distribution information corresponding to their respective candidate word positions, the context vector of the target merged word is generated. In this way, by merging the initial words to obtain the merged word and calculating its context vector, the number of context vectors that need to be stored can be greatly reduced, thus reducing the consumption of GPU memory resources. Moreover, this scheme does not discard any context vectors or reduce parameter precision; that is, this scheme can ensure inference quality while saving GPU memory resources. In addition, this solution can significantly reduce the amount of self-attention computation during the reasoning process, greatly improve the reasoning speed, and reduce the response time to reasoning requests. Attached Figure Description

[0018] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for generating a context vector provided in an embodiment of this application;

[0020] Figure 2 This application provides a schematic diagram of a data flow for calculating information entropy in an embodiment of the present application.

[0021] Figure 3 A data flow diagram of a predicted word segment is provided for an embodiment of this application;

[0022] Figure 4 This is another data flow diagram of predicted terms provided in an embodiment of this application;

[0023] Figure 5 A schematic diagram of a trie provided for an embodiment of this application;

[0024] Figure 6 A schematic flowchart of a context vector generation apparatus provided in an embodiment of this application;

[0025] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0027] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0028] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] With the rapid evolution of artificial intelligence technology, Retrieval-Augmented Generation (RAG) and Agent technologies have become key paradigms driving the deployment of reasoning models in dialogue systems. For example, Large Language Models (LLMs) are deep learning models trained on large amounts of text data, enabling them to generate natural language text or understand the meaning of language text. RAG technology, by combining external knowledge bases with LLMs, significantly improves the accuracy of reasoning tasks. Specifically, current RAG technologies can construct long contexts through historical dialogues, allowing LLMs to process more complete structured information at once, thus improving accuracy. Furthermore, Agent technology, through components such as task planning, long-term memory, and tool invocation, improves the reliability of dialogue systems in completing complex tasks. However, in agent-based question answering, the more complex the task, the longer the context that LLMs need to process. Based on these two technologies, LLMs need to handle long contexts, which places higher demands on memory capacity, bandwidth, and inference speed.

[0030] For example, when LLM performs long context inference, caching context vectors consumes a significant amount of GPU memory. Generally, the GPU memory occupied by context vectors is approximately 2•L•H•D•T•Dtype, where T is the length of the input LLM sequence, L is the number of LLM layers, H is the number of attention heads, D is the hidden layer dimension, and Dtype is the parameter precision. For a 7B parameter model (e.g., Llama-7B), using half-precision floating-point numbers (FP16), a 4K context requires approximately 2GB of GPU memory, and a 32K context requires approximately 16GB. Furthermore, in actual deployments, some redundant GPU memory needs to be reserved for logits and activations, which are also generally referred to as raw prediction scores. As concurrency increases, the GPU memory requirement also increases exponentially.

[0031] To address the issue of high memory requirements, current methods for processing context vectors include sliding windowing, hierarchical / grouping discarding, and quantization. In the sliding window approach, the context vectors of the most recent N terms are typically retained, while the remaining terms are discarded. In the hierarchical / grouping discarding approach, terms are stratified or grouped, and a pre-defined evaluation method is used to calculate the relevance of terms in each level or group to the current inference task. Terms in highly relevant groups are retained in their entirety, while terms in low-relevance groups undergo partial retention, summarization, quantization, or discarding. Quantization converts the context vector from high precision to low precision; for example, converting half-precision floating-point numbers to 8-bit integers (INT8), 4-bit integers (INT4), or binarizing them. However, this leads to the loss of a significant amount of useful information, affecting the accuracy of the inference results and thus reducing the quality of inference.

[0032] To address the aforementioned issue of reduced inference quality, embodiments of this application provide a method for generating context vectors, which can be executed by an inference platform. For example, the inference platform can be a server, a server cluster, etc. Figure 1 As shown, the specific processing steps of the context vector generation method may include:

[0033] Step S101: Obtain the word merging rules and the target input data corresponding to the target reasoning task.

[0034] The target input data may include system prompts from the dialogue system, as well as the user's latest and historical input data. System prompts can be used to instruct the inference model on how to perform inference tasks, such as "You are a calculus tutor."

[0035] Specifically, users can input questions on the inference platform, and system prompts can be pre-set and stored in a preset storage location on the inference platform by the platform's technical staff. When the inference platform receives the user's input data, it can read the system prompts from the preset storage location and generate the target input data for the target inference task based on the system prompts, the latest input data, and the previously entered historical input data.

[0036] Step S102: Divide the target input data into multiple initial word units, and obtain the context vectors corresponding to the multiple initial word units and the word probability distribution information corresponding to the word positions of the multiple initial word units.

[0037] The lexical position indicates the location of the initial lexical in the target input data. The lexical position can be a positional code, such as a number like "1" or "2".

[0038] Specifically, the target inference task can include a prefill stage and a decoding stage. In the prefill stage, the inference platform can identify multiple initial words in the target input data through a pre-built word segmenter and an initial word library. Then, the generated multiple initial words are input into the Large Language Model (LLM). The embedding layer in the LLM can map the multiple initial words into the input data to obtain the embedding vectors corresponding to the multiple initial words. Then, after linear transformation of the embedding vectors, the context vector is obtained. The context vectors of the multiple initial words are then input into the network layer and the normalization layer (e.g., the Softmax layer) in sequence to obtain the word probability distribution information of the word position of each initial word in the target input data.

[0039] Step S103: Based on the word merging rules, multiple initial words are merged to generate at least one merged word.

[0040] Specifically, since there are a large number of initial lexical units, directly using the context vectors of the initial lexical units for subsequent reasoning tasks would consume a lot of GPU memory resources. Therefore, this scheme designs a lexical unit merging rule, which can merge initial lexical units that conform to the lexical unit merging rule into a merged lexical unit.

[0041] Step S104: Based on the context vectors corresponding to the multiple candidate initial words included in the target merging word unit, and the word probability distribution information corresponding to the candidate word positions of the multiple candidate initial words, generate the context vector of the target merging word unit, which is used to perform reasoning operations in the target reasoning task.

[0042] The target merging term is any one of at least one merging term.

[0043] Specifically, after identifying the merged tokens, the inference platform can calculate the context vector for each merged token. Taking the target merged token as an example, the inference platform can generate the context vector for the target merged token by calculating the context vectors of multiple candidate initial tokens included in the target merged token, as well as the token probability distribution information at multiple token positions. In this way, the inference platform can delete the context vectors of the initial tokens and retain the context vectors of the merged tokens. In the subsequent decoding stage, the context vectors of the merged tokens can be directly used for subsequent target inference tasks. During this process, the number of merged tokens is less than the number of initial tokens, meaning that the memory resources occupied by the context vectors of the merged tokens are much less than those occupied by the initial tokens.

[0044] The context vector generation method of this application first obtains a word merging rule and target input data. Then, the target input data can be divided into multiple initial words, and the context vectors corresponding to each initial word and the word probability distribution information corresponding to their respective word positions are obtained. Since the number of initial words is large, the number of context vectors corresponding to them is also large. Therefore, this solution designs a word merging rule, which can merge multiple initial words to generate at least one merged word, reducing the number of words. Further, based on the context vectors of the multiple candidate initial words included in the target merged word and the word probability distribution information corresponding to their respective candidate word positions, the context vector of the target merged word is generated. In this way, by merging the initial words to obtain the merged word and calculating its context vector, the number of context vectors that need to be saved can be greatly reduced, thus reducing the occupation of GPU memory resources. Furthermore, this solution does not discard any context vectors or reduce parameter precision; that is, this solution can ensure inference quality while saving GPU memory resources. In addition, this solution can significantly reduce the amount of self-attention computation during the reasoning process, greatly improve the reasoning speed, and reduce the response time to reasoning requests.

[0045] In some optional implementations, step S104 above may specifically include:

[0046] Step 1: Calculate the information entropy of the target candidate word position based on the word probability distribution information corresponding to the target candidate word position.

[0047] The target candidate lexical position is the lexical position of any one of the multiple candidate initial lexical positions.

[0048] Step 2: After determining the information entropy corresponding to the positions of multiple candidate word units, generate the context vector of the target merged word unit based on the context vectors corresponding to the positions of multiple candidate initial word units and the information entropy corresponding to the positions of multiple candidate word units.

[0049] Specifically, since information entropy can be used to indicate the degree of uncertainty or disorder of a random event or information source, and this scheme has already obtained the probability distribution information of each word position, which reflects the probability distribution of each word at that position (i.e., the probability that the word position is any word), taking the target candidate word position as an example, the inference platform can calculate the information entropy of the candidate word position based on its probability distribution information. Similarly, after calculating the information entropy of all candidate word positions included in the target merged word, the context vector of the merged word can be calculated based on the context vectors of all initial candidate words included in the target merged word and the information entropy of all candidate word positions.

[0050] In this way, the information entropy of the target candidate word position can be accurately determined by using the word probability distribution information. For the target merged word, the context vector of its included candidate initial word provides the semantic information of the target merged word. Information entropy can indicate the clarity of the semantic information of the candidate word position. For example, the larger the information entropy, the stronger the uncertainty of the candidate word position, that is, the more unclear the specific information it indicates. The smaller the information entropy, the weaker the uncertainty of the candidate word position, that is, the more clear the specific information it indicates. Therefore, this scheme can accurately determine the context vector of the target merged word by using the context vector of the candidate initial word and the information entropy of the candidate word position.

[0051] In some optional implementations, the lexical probability distribution information of the target candidate lexical position includes multiple lexical probability values. The target lexical probability value can be used to indicate the probability that the target candidate lexical position is the target initial lexical. The target lexical probability value is any one of the multiple lexical probability values, and the target initial lexical is an initial lexical in a pre-constructed initial lexical library. The initial lexical library can be the lexical library used to divide the target input data into initial lexical units in step S102. Accordingly, step one above may specifically include:

[0052] Step 1: Perform a logarithmic operation on the probability value of the target word to obtain the logarithmic value corresponding to the probability value of the target word.

[0053] Step 2: Based on the target word probability value and the corresponding logarithmic value, determine the information contribution of the target initial word at the target candidate word position.

[0054] Step 3: After determining the information contribution items corresponding to the multiple word probability values, determine the information entropy of the target candidate word position based on the information contribution items corresponding to the multiple word probability values.

[0055] Specifically, for each word probability value in the word probability distribution information of the target candidate word position, the inference platform can calculate the logarithm of each word probability value. Taking the target word probability value as an example, after calculating the logarithm of the target word probability value, the inference platform can calculate the product of the target word probability value and its logarithm, and determine this product as the information contribution term of the target initial word at the target candidate word position. Furthermore, the inference platform can sum the information contribution terms of all candidate word positions of the target merged word, and determine the negative value of the sum as the information entropy of the target candidate word position.

[0056] For example, steps 1 to 3 above can be represented by the following expression:

[0057] (1)

[0058] in, The information entropy represents the position of the target candidate lexical, and v represents the number of initial lexical units included in the initial lexical unit library. This can be represented as the lexical unit at position t (i.e., the target candidate lexical unit position) based on the first t-1 contexts. The probability of a word, i.e. , Let v represent the i-th information contribution item, i = 1, 2, ..., v.

[0059] Alternatively, steps 1 to 3 above can also be expressed as:

[0060] (2)

[0061] refer to Figure 2 After processing by the embedding layer, the tokens, initial words, and delimiters can be transformed into embedding vectors corresponding to the tokens, initial words, and delimiters, respectively. For example, Figure 2 In the context of E_[cls], E_1, E_t, E_[sep], etc. Figure 2 In this context, C, T_1, T_t, and T_sep can represent the positions of the token, each initial word, and the delimiter. After processing the embedding vectors, the LLM outputs the activation values ​​(Logits) corresponding to the positions of each initial word. After normalization using the "Softmax" normalization layer, the probability values ​​of each word position are obtained. For example, Figure 2The token probability value that the next token position of t is "day" is 0.12, the token probability value that it is "of" is 0.02, the token probability value that it is "per" is 0.07, and the token probability value that it is "day" is 0.31. These token probability values constitute the token probability distribution information of the next token position of t. After calculation according to the above formula (1) or formula (2), the information entropy of the next token position of t can be obtained.

[0062] In some optional implementation manners, the above step two may specifically include:

[0063] Step 1, according to the information entropy of the target candidate token position and the information entropies respectively corresponding to multiple candidate token positions, determine the weight value of the target candidate initial token of the target candidate token position in the target merged token.

[0064] Step 2, after determining the weight values of multiple candidate initial tokens in the target merged token, according to the weight values respectively corresponding to multiple candidate initial tokens and the context vector, determine the context vector of the target merged token.

[0065] Specifically, since the information entropy of each candidate token position can represent the uncertainty of the token at this candidate token position, therefore, in order to determine the importance of each candidate token position in the target merged token, the inference platform can compare the information entropy of the target candidate token position with the information entropies of all candidate token positions of the target merged token, and determine the weight value of the target candidate initial token of the target candidate token position in the target merged token. Then, after the inference platform determines the weight value of each candidate initial token in the target merged token in a similar manner, it can perform a weighted sum on the weight values respectively corresponding to multiple candidate initial tokens and the context vector to determine the context vector of the target merged token.

[0066] In some optional implementation manners, step 1 of the above step two may include the following specific steps:

[0067] Step a, according to the information entropies respectively corresponding to multiple candidate token positions, determine the total information entropy.

[0068] Step b, according to the information entropy of the target candidate token position and the total information entropy, determine the weight value of the target candidate initial token in the target merged token.

[0069] Specifically, the inference platform can sum the information entropies of all candidate token positions of the target merged token to obtain the total information entropy, and then determine the ratio of the information entropy of the target candidate token position to the total information entropy as the weight value of the target candidate initial token in the target merged token.

[0070] For example, the above step a to step b can adopt the following expression:

[0071] (3)

[0072] in, This represents the weight value of the initial candidate word at position j in the target merged word set. Let m represent the information entropy of the j-th candidate word position, and m represent the number of candidate initial words included in the target merged word group. This represents the total information entropy.

[0073] In some optional implementations, step 2 of step two above may include the following specific steps:

[0074] Step c: Determine the sub-vector of the target candidate word position based on the weight value of the initial target candidate word and the context vector.

[0075] Step d: After determining the sub-vectors of multiple candidate word positions, determine the context vector of the target merge word based on the sub-vectors of multiple candidate word positions.

[0076] Specifically, the inference platform can determine the sub-vector of the target candidate word position by multiplying the weight value of the initial word position of the target candidate word by the context vector. For all candidate word positions in the target merged word, the inference platform can calculate the sub-vector in a similar way. In this way, after determining the sub-vectors of all candidate word positions in the target merged word, the sub-vectors of all candidate word positions can be summed to obtain the context vector of the target merged word.

[0077] In some optional implementations, the context vector of the candidate initial lexical unit mentioned above includes a key vector and a value vector, wherein the key vector can be used to represent the feature identifier of the lexical unit, and the value vector can be used to represent the actual semantic information carried by the lexical unit. Accordingly, step c may include the following specific steps:

[0078] Step c1: Determine the sub-key vector of the target candidate word position based on the weight value and key vector of the initial target candidate word.

[0079] Step c2: Determine the sub-value vector of the target candidate word position based on the weight value and value vector of the initial target candidate word.

[0080] Among them, the sub-key vector and sub-value vector of the target candidate word position constitute the sub-vector of the target candidate word position.

[0081] Specifically, the inference platform can determine the sub-key vector at the target candidate token position by multiplying the weight value of the target candidate initial token by the key vector. Similarly, the inference platform can determine the sub-value vector at the target candidate token position by multiplying the weight value of the target candidate initial token by the value vector.

[0082] In some alternative embodiments, step d described above may include the following specific steps:

[0083] Step d1: Sum the sub-key vectors at multiple candidate token positions to obtain the total key vector of the target merged token.

[0084] Step d2: Sum the sub-value vectors at multiple candidate token positions to obtain the total value vector of the target merged token.

[0085] Wherein, the total key vector and the total value vector of the target merged token constitute the context vector of the target merged token.

[0086] For example, "今" and "天" are two initial tokens, and "今天" is the merged token. The total key vector of "今天" can be calculated using the following expression:

[0087] (4)

[0088] The total value vector of "今天" can be calculated using the following expression:

[0089] (5)

[0090] Wherein, represents the weight value of "今" in "今天", represents the weight value of "天" in "今天", is the total key vector of the merged token "今天", is the total value vector of the merged token "今天", [[ID=3৮]] is the key vector of "今", is the key vector of "天", is the value vector of "今", is the value vector of "天".

[0091] In this way, the context vector of the target merged token can be determined through the weighted summation operation, which is relatively simple. Compared with the related art solutions that evaluate the importance of context vectors and discard the context vectors with lower importance, it occupies less computing resources. Moreover, it also takes into account the comprehensiveness of context vectors. That is, based on ensuring no information loss, this solution only needs to occupy less computing resources to calculate the context vector of the merged token.

[0092] Reference Figure 3, assume the input sentence is "The weather is nice today", and the first 6 tokens "The weather is not" have been generated. Now, predict the token at the 7th token position. During the prediction process, each token needs to pass through an embedding layer to obtain an embedding vector (e.g., x1, x2, x n ), and then, after a linear transformation of the embedding vector, a query vector (e.g., q1, q2, q n ) can be generated. The key vector and value vector will be cached to obtain a key-value cache (Key Value Cache, KV Cache). Further, under the processing of the attention mechanism, the similarity between the query vector and the historical key vectors can be calculated to obtain attention scores (Score), e.g., 112 and 96. Then, through the first normalization (Normalization, NORM) layer, the attention scores are normalized, and then through the second normalization layer Softmax, a normalization operation is performed to obtain the weight values at each token position (e.g., 0.4 and 0.2), e.g., Figure 2 the weight 0.5 indicated in n . Finally, a weighted sum can be performed according to each value vector and its corresponding weight (e.g., x

[0093] Reference Figure 4 , and Figure 3 different from the steps shown, after determining the key-value vectors of each token, a token merging operation can be performed. For example, for "今" and "天", they can be merged into "今天", and the key-value vector of "今天" can be calculated. That is, Figure 4 the total key vector shown in and the total value vector . In this way, only the key-value vector of "今天" needs to be cached.

[0094] In some optional embodiments, for each merged token, the inference platform can also determine the context vector of each merged token in the following manner:

[0095] For the target merged word unit, the inference platform can concatenate the context vectors corresponding to the multiple candidate initial word units included in the target merged word unit according to their positions in the target merged word unit from front to back, to obtain the target concatenated vector. Then, the inference platform can input the target concatenated vector into a pre-built multi-layer perceptron (MLP) to obtain a target vector with the same dimension as the context vectors corresponding to the candidate initial word units output by the MLP, and determine the target vector as the context vector of the target merged word unit.

[0096] When the context vector includes both key and value vectors, for each candidate initial word, its key and value vectors can be concatenated to obtain a concatenated vector. Then, the concatenated vectors of all candidate initial words are concatenated to obtain the target concatenated vector. Correspondingly, after determining the target vector, the first d elements can be designated as the key vector, and the last d elements as the value vector.

[0097] In this way, by using a multilayer perceptron to nonlinearly fuse the context vectors of each candidate initial word, the interaction information between each initial word can be captured, generating a more accurate context vector. Furthermore, in reasoning scenarios, this can greatly improve the accuracy of reasoning results.

[0098] In some optional implementations, the inference platform can first obtain its own load indicators (e.g., the number of inference tasks currently being executed in parallel, or the utilization rate of computing resources, etc.), and then determine whether the load indicators are less than a preset load indicator value. If not, the context vector of each merged word can be determined in the manner described in step S104 above. If so, the context vector of each merged word can be determined using a multilayer perceptron.

[0099] This allows for greater flexibility in generating context vectors by selecting different methods based on the actual load.

[0100] In some optional implementations, the above-mentioned word merging rules can be merging word libraries, merging word trie trees, etc. Accordingly, the word merging rules in step S101 can be generated using the following specific steps:

[0101] Step 1: Obtain the target corpus file and the merging stop condition.

[0102] Step 2: Generate word merging rules based on the target corpus file, the initial word library, and the merging stopping conditions.

[0103] Specifically, a technician can upload a target corpus file that is common in various fields to the inference platform and input the required merging stop conditions, such as the number of merged tokens, the number of merging operations, etc. In this way, the inference platform can obtain the target corpus file and the merging stop conditions. Then, based on the initial token library, the inference platform can identify the initial tokens in the target corpus file and perform a merging operation on the initial tokens in the target corpus file with the merging stop conditions as the goal to obtain the final token merging rule.

[0104] After determining the merged token library, the merged token library can be directly determined as the token merging rule, or the merged token library can be converted into a trie tree, and the trie tree can be directly determined as the token merging rule.

[0105] For example, the trie tree can be as Figure 5 shown. The root node in the trie tree includes "今" and "天", and the new phrase formed by the root node to the leaf node is the merged token. Correspondingly, in the above step S103, the multiple initial tokens included in the target input data can be merged through the trie tree. For the given sentence "今日金价上涨", the multiple initial tokens it includes are "今", "日", "金", "价", "上", "涨". After merging through the trie tree, "今", "日", "金", "价" can be merged into "今日金价".

[0106] In this way, by pre-generating the token merging rule, a benchmark for the merged token operation can be obtained, and the token merging operation can be performed more accurately to obtain accurate merged tokens. In addition, by performing the merged token operation through the trie tree in step S103, the efficiency is relatively high.

[0107] In some optional implementation manners, step two of the above step S101 can specifically include:

[0108] Step 1, in the current round, obtain multiple tokens to be processed in the target corpus file in the current round.

[0109] Step 2, according to the token positions of the multiple tokens to be processed in the current round in the target corpus file respectively, identify at least one group of tokens to be processed with adjacent token positions.

[0110] Step 3, count the occurrence frequency of at least one group of tokens to be processed in the target corpus file.

[0111] Step 4, perform a merging operation on the multiple initial tokens included in the group of tokens to be processed with the highest occurrence frequency in the target corpus file to generate a merged token corresponding to the current round. [[ID=Z7]]

[0112] Step 5: After determining that the current round meets the merging stop condition, generate word merging rules based on the word elements generated in the current round and all rounds before the current round.

[0113] Alternatively, in step 6, after determining that the current round does not meet the merging stop condition, proceed to the next round until a target round that meets the merging stop condition is determined. Then, generate word merging rules based on the merging words generated in the target round and all rounds before the target round.

[0114] When the current round is the first round, all the words to be processed are initial words in the initial word library; or when the current round is not the first round, the words to be processed include merged words and initial words generated in previous rounds.

[0115] Specifically, in step 1, since the merging operation can be an iterative process, the current round can be either the first round or a non-first round. Accordingly, the inference platform can perform the operation of obtaining the unprocessed lexical units of the current round based on the indication information of whether the current round is the first round.

[0116] In step 2, the inference platform can identify all groups of words to be processed that are adjacent to each word in the target corpus file based on the word position of each word to be processed (the method of obtaining word position can be referred to the processing in step S102, which will not be repeated here).

[0117] In step 3, the inference platform can count the frequency of each word tuple to be processed in the target corpus file.

[0118] In step 4, the inference platform can sort each lexical group to be processed according to its frequency of occurrence from high to low or from low to high, determine the target lexical group to be processed with the highest frequency, and then merge all the lexical groups to be processed included in the target lexical group to obtain the merged lexical group generated in the current round.

[0119] In step 5, the inference platform can determine whether the number of currently generated merge tokens is equal to the number of merge tokens indicated in the merge stop condition. If so, it can be determined that the current round meets the merge stop condition. Alternatively, the inference platform can determine whether the number of merge operations currently performed is equal to the number of merge operations indicated in the merge stop condition. If so, it can be determined that the current round meets the merge stop condition.

[0120] In step 6, when it is determined that the number of currently generated merged tokens has not reached the number of merged tokens indicated in the merge stop condition, and the number of merge operations currently performed has not reached the number of merge operations indicated in the merge stop condition, it can be determined that the current round does not meet the merge stop condition, and the inference platform can enter the next round to continue the merge processing.

[0121] This process continues until the merging stopping condition is met. At this point, the inference platform can stop iterating and construct a word merging rule based on the target round that meets the merging stopping condition and the merging words generated in all rounds prior to the target round. The specific processing of steps 1 to 6 above can be implemented by the inference platform using the Byte Pair Encoding (BPE) algorithm.

[0122] Since the initial lexical unit is generally a single character, this scheme can gradually construct merged lexical units by continuously merging high-frequency adjacent lexical units, significantly reducing the number of lexical units and obtaining lexical unit merging rules. Furthermore, during the execution of the inference task, it is unnecessary to repeatedly perform the above process of merging operations. Instead, the pre-generated lexical unit merging rules can be directly used to merge the initial lexical units, which can greatly reduce the computing resources occupied during the execution of the inference task and improve the execution efficiency of the inference task.

[0123] In some optional implementations, in step 1 above, the inference platform performs the operation of obtaining the unprocessed lexical units of the current round according to the indication information of whether the current round is the first round. This can be specifically divided into the following two cases:

[0124] Scenario 1: The current round is the first round. Based on the initial lexicon, identify multiple initial lexicons included in the target corpus. All of these initial lexicons in the target corpus are determined as the lexicons to be processed in the first round of the target corpus.

[0125] Specifically, when the current round is the first round, the inference platform can use the initial word library as a benchmark to identify the initial words included in the target corpus file, and determine the initial words included in the target corpus file as the words to be processed in the target corpus file in the first round.

[0126] In the second scenario, the candidate lexical units included in the target corpus are replaced with the merged lexical units generated in the previous round before the current round, resulting in multiple unprocessed lexical units in the target corpus in the current round.

[0127] Among them, candidate lexical units are those included in the merged lexical units generated in the previous round.

[0128] Specifically, in the previous round, the inference platform has generated merged tokens (hereinafter referred to as the first merged tokens). Correspondingly, the inference platform can re-determine the tokens included in the target corpus file, and can replace all candidate tokens that can be merged into the first merged tokens that appear in the target corpus file with the first merged tokens. For example, replace each group of candidate tokens "今" and "天" with the merged token "今天". In this way, multiple tokens to be processed in the current round of the target corpus file can be obtained.

[0129] In some optional embodiments, in the decoding stage, the inference platform will continuously generate new tokens and context vectors of the new tokens during the execution of the task, which also requires a large amount of video memory resources. Therefore, the inference platform can also merge the new tokens generated in the decoding stage. Correspondingly, the inference platform can set a preset quantity threshold. Whenever it is detected that the number of newly generated tokens reaches the preset quantity threshold, a merging operation is performed on the unmerged new tokens, and the context vector of the merged tokens is calculated (specifically, refer to the processing from step S103 to step S104 above, which will not be elaborated here). In this way, the inference platform can delete the context vectors of the new tokens and only retain the context vectors of the merged tokens, and can also reduce the occupation of video memory resources in the decoding stage.

[0130] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0131] An embodiment of the present application also provides a device for generating context vectors, as Figure 6 shown, including:

[0132] An acquisition module 610, configured to acquire a token merging rule and target input data corresponding to a target inference task; divide the target input data into multiple initial tokens, and acquire context vectors corresponding to the multiple initial tokens respectively and token probability distribution information corresponding to the token positions of the multiple initial tokens respectively, where the token position is used to indicate the position of the initial token in the target input data;

[0133] A merging module 620, configured to generate at least one merged token after merging the multiple initial tokens based on the token merging rule;

[0134] The generation module 630 is used to generate a context vector of the target merging word element based on the context vectors corresponding to the multiple candidate initial word elements included in the target merging word element, and the word element probability distribution information corresponding to the candidate word element positions of the multiple candidate initial word elements, so as to perform inference operations in the target inference task. The target merging word element is any one of the at least one merging word elements.

[0135] In some alternative implementations, the generation module 630 is specifically used for:

[0136] Based on the probability distribution information of the lexical units corresponding to the position of the target candidate lexical unit, the information entropy of the position of the target candidate lexical unit is calculated, where the position of the target candidate lexical unit is the position of any one of the multiple candidate initial lexical units;

[0137] After determining the information entropy corresponding to the positions of multiple candidate lexical units, the context vector of the target merged lexical unit is generated based on the context vectors corresponding to the positions of multiple candidate initial lexical units and the information entropy corresponding to the positions of multiple candidate lexical units.

[0138] In some optional implementations, the lexical probability distribution information of the target candidate lexical position includes multiple lexical probability values. The target lexical probability value is used to indicate the probability that the target candidate lexical position is the target initial lexical. The target lexical probability value is any one of the multiple lexical probability values, and the target initial lexical is an initial lexical in a pre-built initial lexical library. The generation module 630 is specifically used for:

[0139] Perform a logarithmic operation on the probability value of the target word to obtain the logarithmic value corresponding to the probability value of the target word.

[0140] Based on the target word probability value and the corresponding logarithm value, determine the information contribution of the target initial word at the position of the target candidate word;

[0141] After determining the information contribution items corresponding to the probability values ​​of multiple word elements, the information entropy of the target candidate word element position is determined based on the information contribution items corresponding to the probability values ​​of multiple word elements.

[0142] In some alternative implementations, the generation module 630 is specifically used for:

[0143] Based on the information entropy of the target candidate word position and the information entropy corresponding to the positions of multiple candidate word positions, determine the weight value of the target candidate initial word at the target candidate word position in the target merged word;

[0144] After determining the weight values ​​of multiple candidate initial lexical units in the target merged lexical unit, the context vector of the target merged lexical unit is determined based on the weight values ​​and context vectors corresponding to the multiple candidate initial lexical units.

[0145] In some alternative implementations, the generation module 630 is specifically used for:

[0146] The total information entropy is determined based on the information entropy corresponding to the positions of multiple candidate lexical units.

[0147] The weight values ​​of the initial target candidate word units in the target merged word units are determined based on the information entropy of the target candidate word unit positions and the total information entropy.

[0148] In some alternative implementations, the generation module 630 is specifically used for:

[0149] Based on the weight values ​​of the initial target candidate lexical units and the context vector, determine the sub-vectors of the target candidate lexical unit positions;

[0150] After determining the sub-vectors of multiple candidate word positions, the context vector of the target merge word is determined based on the sub-vectors of multiple candidate word positions.

[0151] In some optional implementations, the context vector of the candidate initial lexical includes a key vector and a value vector. The generation module 630 is specifically used for:

[0152] Based on the weight values ​​and key vectors of the initial target candidate lexical units, determine the sub-key vectors at the positions of the target candidate lexical units;

[0153] Based on the weight values ​​and value vectors of the initial target candidate lexical units, determine the sub-value vectors of the target candidate lexical unit positions;

[0154] Among them, the sub-key vector and sub-value vector of the target candidate word position constitute the sub-vector of the target candidate word position.

[0155] In some alternative implementations, the generation module 630 is specifically used for:

[0156] The sub-key vectors of multiple candidate word positions are summed to obtain the total key vector of the target merged word.

[0157] The sub-value vectors of multiple candidate word positions are summed to obtain the total value vector of the target merged word;

[0158] Among them, the total key vector and total value vector of the target merged word constitute the context vector of the target merged word.

[0159] In some optional implementations, the acquisition module 610 is specifically used for:

[0160] Obtain the target corpus file and the merging stopping condition;

[0161] Based on the target corpus file, the initial lexicon, and the merging stopping condition, lexicon merging rules are generated.

[0162] For a description of the features in the embodiment corresponding to the context vector generation device, please refer to the relevant description in the embodiment corresponding to the context vector generation method, which will not be repeated here.

[0163] Embodiments of this application also provide an electronic device, such as... Figure 7 As shown, it includes a memory 10 and a processor 20. The memory 10 stores a computer program, and the processor 20 is configured to run the computer program to perform the steps in any of the above-described embodiments of the context vector generation method.

[0164] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the context vector generation method at runtime.

[0165] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0166] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the context vector generation method.

[0167] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above-described context vector generation method embodiments.

[0168] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] The foregoing has provided a detailed description of a method, apparatus, electronic device, storage medium, and program product for generating context vectors. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for generating context vectors, characterized in that, include: Obtain the word merging rules and the target input data corresponding to the target inference task; The target input data is divided into multiple initial word units, and the context vectors corresponding to the multiple initial word units and the word unit probability distribution information corresponding to the word unit positions of the multiple initial word units are obtained respectively, wherein the word unit positions are used to indicate the positions of the initial word units in the target input data; Based on the lexical merging rules, multiple initial lexical units are merged to generate at least one merged lexical unit; Based on the probability distribution information of the word elements corresponding to the position of the target candidate word element, the information entropy of the position of the target candidate word element is calculated, wherein the position of the target candidate word element is the word element position of any one of the multiple candidate initial word elements included in the target merged word element, and the target merged word element is any one of the merged word elements. After determining the information entropy corresponding to the positions of the multiple candidate lexical units, a context vector of the target merged lexical unit is generated based on the context vectors corresponding to the positions of the multiple candidate initial lexical units and the information entropy corresponding to the positions of the multiple candidate lexical units, so as to perform reasoning operations in the target reasoning task.

2. The method for generating context vectors according to claim 1, characterized in that, The lexical probability distribution information of the target candidate lexical position includes multiple lexical probability values. The target lexical probability value is used to indicate the probability that the target candidate lexical position is the target initial lexical. The target lexical probability value is any one of the multiple lexical probability values. The target initial lexical is an initial lexical in a pre-constructed initial lexical library. The step of calculating the information entropy of the target candidate word position based on the word probability distribution information corresponding to the target candidate word position includes: Perform a logarithmic operation on the probability value of the target word to obtain the logarithmic value corresponding to the probability value of the target word. Based on the target word probability value and the logarithm corresponding to the target word probability value, determine the information contribution item of the target initial word at the position of the target candidate word; After determining the information contribution items corresponding to the multiple word probability values, the information entropy of the target candidate word position is determined based on the information contribution items corresponding to the multiple word probability values.

3. The method for generating context vectors according to claim 1 or 2, characterized in that, After determining the information entropy corresponding to the positions of the multiple candidate lexical units, the step of generating the context vector of the target merged lexical unit based on the context vectors corresponding to the positions of the multiple candidate initial lexical units and the information entropy corresponding to the positions of the multiple candidate lexical units includes: Based on the information entropy of the target candidate word position and the information entropy corresponding to the multiple candidate word positions respectively, the weight value of the target candidate initial word at the target candidate word position in the target merged word is determined; After determining the weight values ​​of the multiple candidate initial lexical units in the target merged lexical unit, the context vector of the target merged lexical unit is determined based on the weight values ​​and context vectors corresponding to the multiple candidate initial lexical units respectively.

4. The method for generating context vectors according to claim 3, characterized in that, The step of determining the weight value of the initial target candidate word at the target candidate word position in the target merged word group based on the information entropy of the target candidate word position and the information entropy corresponding to the multiple candidate word positions respectively includes: The total information entropy is determined based on the information entropy corresponding to the positions of the multiple candidate word elements; The weight value of the initial target candidate word in the target merged word is determined based on the information entropy of the target candidate word position and the total information entropy.

5. The method for generating context vectors according to claim 3, characterized in that, After determining the weight values ​​of the multiple candidate initial lexical units in the target merged lexical unit, the step of determining the context vector of the target merged lexical unit based on the weight values ​​and context vectors corresponding to the multiple candidate initial lexical units includes: Based on the weight values ​​and context vectors of the initial target candidate lexical units, a sub-vector representing the position of the target candidate lexical unit is determined; After determining the sub-vectors of the positions of multiple candidate lexical units, the context vector of the target merged lexical unit is determined based on the sub-vectors of the positions of multiple candidate lexical units.

6. The method for generating context vectors according to claim 5, characterized in that, The context vector of the candidate initial word includes a key vector and a value vector. The step of determining the sub-vector representing the position of the target candidate word based on its weight value and context vector includes: Based on the weight value and key vector of the initial target candidate word, determine the sub-key vector of the target candidate word position; Based on the weight value and value vector of the initial target candidate word, determine the sub-value vector of the target candidate word position; Wherein, the sub-key vector and sub-value vector of the target candidate word position constitute the sub-vector of the target candidate word position.

7. The method for generating context vectors according to claim 6, characterized in that, After determining the sub-vectors of the positions of multiple candidate lexical units, the step of determining the context vector of the target merged lexical unit based on the sub-vectors of the positions of multiple candidate lexical units includes: The sub-key vectors of multiple candidate word positions are summed to obtain the total key vector of the target merged word; The sub-value vectors of the positions of multiple candidate word elements are summed to obtain the total value vector of the target merged word element; The total key vector and total value vector of the target merged word constitute the context vector of the target merged word.

8. The method for generating context vectors according to claim 2, characterized in that, Obtaining the word merging rules includes: Obtain the target corpus file and the merging stopping condition; Based on the target corpus file, the initial lexicon, and the merging stop condition, the lexicon merging rule is generated.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the context vector generation method as described in any one of claims 1 to 8 when executing the computer program.

Images