Text processing method and device, computer equipment and storage medium

Abstract

The invention discloses a text processing method and device, computer equipment and a storage medium, and relates to the technical field of AI. In the application, dependency analysis is performed on a plurality of semantic elements in an input text to obtain at least one root semantic element, and the at least one root semantic element is a semantic element on which other semantic elements in the input text depend, namely a core semantic element in the input text. And on the basis, selecting a plurality of target semantic elements with relatively high relevancy with the at least one root semantic element from a plurality of semantic elements contained in the input text, and performing attention sparse processing by utilizing the plurality of target semantic elements. Therefore, in the process of selecting the target semantic element, the dependency relationship among the semantic elements in the input text is reserved, so that attention sparse processing is performed by utilizing the target semantic element, and the influence on the model prediction precision can be reduced as much as possible while the calculation amount of the model is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI), and more particularly to a text processing method, apparatus, computer device, and storage medium. Background Technology

[0002] Currently, AI models such as large language models (LLM) and multimodal models are all based on the transformer architecture. The transformer architecture introduces a self-attention mechanism, which enables the modeling and efficient computation of the relationship between any two semantic elements in the input text. This allows the AI ​​model to fully consider the context of the input text when generating prediction results, thus improving the quality of the prediction results.

[0003] When using an AI model based on self-attention to generate prediction results, the input text can be divided into multiple semantic elements. The query (Q) vector, key (K) vector, and value (V) vector of each semantic element are then obtained. The key vectors of these semantic elements are combined to form a key matrix, and the value vectors are combined to form a value matrix. Finally, prediction results are generated based on the query vector, key matrix, and value matrix of each semantic element. Since the key matrix and value matrix are composed of the key and value vectors of each semantic element in the input text, respectively, they will be large when the input text contains many semantic elements. Consequently, the computational cost of generating prediction results based on these matrices is also high. Therefore, related technologies select a subset of semantic elements from the multiple semantic elements and use the key and value vectors of the selected semantic elements to determine sparse key and sparse value matrices. This achieves attention sparsity processing, thereby reducing the computational cost of the model. However, since selecting some semantic elements from multiple semantic elements will affect the quality of the prediction results generated by the model, how to select some semantic elements from multiple semantic elements for attention sparsity processing is a key issue. Summary of the Invention

[0004] This application provides a text processing method, apparatus, computer device, and storage medium that can perform attention sparsity processing while minimizing the impact on the model's prediction accuracy, thereby reducing the computational load of the AI ​​model in the process of generating prediction results.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, a text processing method is provided, the method comprising: performing dependency analysis on multiple semantic elements in a first input text to obtain at least one root semantic element; selecting multiple target semantic elements from the multiple semantic elements, wherein the correlation between the multiple target semantic elements and the at least one root semantic element is greater than the correlation between other semantic elements among the multiple semantic elements and the at least one root semantic element; performing attention sparsity processing based on the multiple target semantic elements to obtain a first processing result; and generating a first prediction result based on the first processing result.

[0007] In this application, dependency analysis is performed on multiple semantic elements contained in the first input text to obtain at least one root semantic element, wherein the at least one root semantic element is a semantic element on which other semantic elements in the first input text depend. In other words, the at least one root semantic element is the core semantic element in the first input text. Based on this, multiple target semantic elements with high relevance to the at least one root semantic element are selected from the multiple semantic elements contained in the first input text, and attention sparsity processing is performed using these multiple target semantic elements. Therefore, this application preserves the dependencies between semantic elements in the input text during the selection of target semantic elements. Thus, using target semantic elements for attention sparsity processing reduces the computational load of the model while minimizing the impact on the model's prediction accuracy, that is, reducing the impact on the quality of the prediction results generated by the model.

[0008] It should be noted that in this application, semantic elements can also be called lexical units or tokens, and root semantic elements can also be called root tokens.

[0009] Optionally, the first input text includes at least one statement, and the process of performing dependency analysis on multiple semantic elements in the first input text to obtain at least one root semantic element may include: performing dependency analysis on the semantic elements in the at least one statement to obtain the at least one root semantic element.

[0010] In this application, dependency analysis is performed on the input text at the sentence level, and a root semantic element can be determined for each sentence. For example, state-of-the-art neural parsing models or transition-based dependency parsing can be used to perform dependency analysis on each sentence, thereby obtaining a dependency tree corresponding to each sentence. The token indicated by the root label in the dependency tree is determined as the root token. Among these, the state-of-the-art neural parsing model is a single-layer network, which has a faster processing speed. Alternatively, the dependency analysis module can also use a single-layer parsing network to perform dependency analysis on each sentence. In this case, the parsing network does not need to output a complete dependency tree; it can directly output the root token corresponding to each sentence, thus saving time.

[0011] Optionally, the process of selecting multiple target semantic elements from the plurality of semantic elements may include: obtaining the key vector of each semantic element and the query vector of each root semantic element; determining the similarity between the key vector of each semantic element and the query vector of each root semantic element; determining multiple target similarities from the determined multiple similarities, wherein the multiple target similarities are greater than other similarities among the multiple similarities; and determining the semantic element corresponding to the key vector of the multiple target similarities as the multiple target semantic elements.

[0012] In this application, the similarity between the query vector of the root token and the key vector of any token can be used to characterize the relevance or association between the token and the root token. The higher the similarity between the query vector of the root token and the key vector of the token, the greater the relevance or association between the token and the root token.

[0013] Optionally, the first input text is a prompt text in a reasoning scenario, and the first prediction result includes a first prediction statement. After generating the first prediction result, dependency analysis can be performed on the first prediction statement to obtain a new root semantic element. Multiple update target semantic elements are determined from multiple candidate semantic elements. The multiple candidate semantic elements include semantic elements in the first prediction statement and the multiple target semantic elements. The relevance between the multiple update target semantic elements and the new root semantic element is greater than the relevance between other semantic elements in the candidate semantic elements and the new root semantic element. Based on the multiple update target semantic elements, attention sparsity processing is performed to obtain a second processing result. Based on the second input text and the second processing result, a second prediction result is generated. The second input text includes the first prediction result.

[0014] In this application, during the inference scenario, whenever a new predicted statement is generated, dependency analysis is performed on that statement to obtain its root token. This root token is then used as a new root token. Based on this new root token, at least one previous root token, and multiple target tokens, attention sparsity processing is performed again to obtain a second processing result. This second processing result contains the contextual information of the newly generated predicted statement. Therefore, continuing subsequent inference based on this second processing result can improve inference accuracy.

[0015] Optionally, the first processing result is stored in a key-value (KV) cache; the method further includes replacing the first processing result in the KV cache with the second processing result.

[0016] In this application, the sparse processing results can be stored in a KV cache for inference, thereby improving inference speed.

[0017] Optionally, the first processing result includes information on multiple target tokens and information on at least one root token. The second processing result includes information on multiple updated target tokens and information on multiple root tokens. The number of updated target tokens and the number of target tokens can be the same. Furthermore, compared to the root token information included in the first processing result, the root token information included in the second processing result adds information on the root token from the prediction statement. That is, in this application, for each prediction statement generated, the sparse processing result stored in the KV cache will add information on one root token. In related technologies, for each token generated, the KV cache will add information corresponding to that token; if the prediction statement includes j tokens, for each prediction statement generated, the KV cache will add information on j tokens. Therefore, compared to related technologies, in incremental inference scenarios, this application can reduce the KV cache occupancy rate and also reduce the amount of data read when calling the sparse processing result in the KV cache.

[0018] Optionally, the process of performing attention sparsity processing based on the plurality of target semantic elements to obtain a first processing result may include: determining a global sparse key matrix and a global sparse value matrix based on the plurality of target semantic elements, wherein the first processing result includes the global sparse key matrix and the global sparse value matrix, wherein the global sparse key matrix includes the key vectors of the plurality of target semantic elements and the global sparse value matrix includes the value vectors of the plurality of target semantic elements.

[0019] Optionally, the process of generating the first prediction result based on the first processing result may include: determining the sparse key matrix corresponding to each of the plurality of semantic elements, wherein the sparse key matrix corresponding to each semantic element includes the key vector in the global sparse key matrix and the key vectors of the adjacent semantic elements of the corresponding semantic element; determining the sparse value matrix corresponding to each of the plurality of semantic elements, wherein the sparse value matrix corresponding to each semantic element includes the value vector in the global sparse value matrix and the value vectors of the adjacent semantic elements of the corresponding semantic element; determining the weighted value vector of each semantic element based on the query vector of each semantic element, the corresponding sparse key matrix, and the sparse value matrix; and generating the first prediction result based on the weighted value vector of each semantic element.

[0020] This application can not only use the root token and target tokens that are highly related to the root token to construct sparse key matrices and sparse value matrices, but also add the local context information of the token to the sparse key matrix and sparse value matrix when calculating the weighted value vector of each token, thereby improving the accuracy of inference or training.

[0021] Optionally, in an encoder-based model, the neighboring semantic elements of each semantic element include the preceding N semantic elements and the following N semantic elements of the corresponding semantic element in the first input text, where N is an integer greater than 0.

[0022] Optionally, in a decoder-based model, the neighboring semantic elements of each semantic element include the first N semantic elements of the corresponding semantic element in the first input text, where N is an integer greater than 0.

[0023] Optionally, the process of determining the weighted value vector of each semantic element based on the query vector of each semantic element, the corresponding sparse key matrix, and the sparse value matrix may include: determining the attention weight vector of the first semantic element based on the query vector of the first semantic element and the corresponding sparse key matrix, wherein the first semantic element is one of the plurality of semantic elements; and determining the weighted value vector of the first semantic element based on the attention weight vector of the first semantic element and the sparse value matrix corresponding to the first semantic element.

[0024] In a second aspect, a text processing apparatus is provided, the text processing apparatus comprising at least one module for performing the text processing method described in the first aspect above.

[0025] Thirdly, a computer device is provided, the computer device including a processor, the processor being configured to execute at least one program instruction or code stored in a memory to implement the text processing method described in the first aspect above.

[0026] Optionally, the computer device further includes a memory, the memory including a KV cache for storing sparse processing results involved in the text processing method described in the first aspect above.

[0027] Fourthly, a computer device cluster is provided, the computer device cluster including at least one computer device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the text processing method described in the first aspect above.

[0028] Fifthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer device, cause the computer device to perform the text processing method described in the first aspect.

[0029] In a sixth aspect, a computer program product containing instructions is provided, which, when run on a computer device, causes the computer device to perform the text processing method described in the first aspect.

[0030] The technical effects achieved by the second to sixth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the structure of an encoder;

[0032] Figure 2 This is a schematic diagram of the structure of a decoder;

[0033] Figure 3 A flowchart illustrating a text processing method provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram illustrating the attention weights contained in the attention weight vector corresponding to each token in a text, as provided in an embodiment of this application.

[0035] Figure 5 A flowchart illustrating another text processing method provided in this application embodiment;

[0036] Figure 6 This is a schematic diagram illustrating how, in a training scenario, a model with only an encoder (encoder-only) uses text processing methods for prediction.

[0037] Figure 7 This is a schematic diagram illustrating how a decoder-only model uses text processing methods for prediction in an inference scenario, as provided in an embodiment of this application.

[0038] Figure 8 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application;

[0039] Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0040] Figure 10 This application provides a schematic diagram of the structure of a computer device cluster.

[0041] Figure 11 This is a schematic diagram of another computer device cluster structure provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0043] Before providing a detailed explanation of the embodiments of this application, let's first introduce the application scenarios involved in the embodiments of this application.

[0044] Currently, AI models such as LLM and multimodal models are all based on the transformer architecture. The transformer architecture introduces a self-attention mechanism, which enables the modeling and efficient computation of the relationship between any two semantic elements in the input text. This allows the AI ​​model to fully consider the context of the input text when generating prediction results, thus improving the quality of the prediction results.

[0045] When making predictions, AI models based on the Transformer architecture typically determine key and value matrices based on the semantic elements contained in the input text, and then use these matrices to calculate a weighted value vector. The size of the key and value matrices depends on the size of the input text. Generally, the more semantic elements the input text contains, the larger the key and value matrices become, resulting in higher computational and memory usage for the model. Currently, Transformer-based AI models have high requirements for the size of the input text in both training and inference scenarios. For example, in training scenarios, the number of semantic elements in the input text can reach 8,000, while in inference scenarios, it can reach 100,000. Therefore, input text containing a large number of semantic elements presents a significant challenge to the computational demands of AI models in both training and inference scenarios.

[0046] To address the aforementioned issues, this application provides a text processing method applicable to AI model training and / or inference scenarios. In this method, dependency analysis is performed on multiple semantic elements contained in a first input text to obtain at least one root semantic element. This root semantic element is a semantic element upon which other semantic elements in the first input text depend. In other words, the at least one root semantic element is the core semantic element in the first input text. Based on this, multiple target semantic elements with high relevance to the at least one root semantic element are selected from the multiple semantic elements contained in the first input text. These target semantic elements are then used for attention sparsity processing to reduce the size of the key matrix and value matrix. The reduced key matrix and value matrix are then used to generate prediction results. Therefore, this application embodiment reduces the size of the key matrix and value matrix while preserving the dependencies between semantic elements in the input text. This reduces the computational load of the model in training and / or inference scenarios while minimizing the impact on the model's training and / or inference accuracy.

[0047] Before providing a detailed explanation of the text processing methods provided in the embodiments of this application, we will first explain some of the terms involved in the embodiments of this application.

[0048] 1. Transformer architecture

[0049] The transformer architecture is a machine learning model architecture used to model sequential data, widely applied in large models such as LLM and multimodal models. The transformer architecture consists of an encoder and a decoder. Currently, there are three main types of models based on the transformer architecture: encoder-only models, decoder-only models, and models that include both encoder and decoder. The structures of the encoder and decoder will be described below.

[0050] For example, Figure 1 A schematic diagram of an encoder structure is shown. Figure 1 As shown, encoder 10 may include embedding and position encoding modules 11 and multiple transformer layers ( Figure 1 The image shows two transformer layers, 12 and 13 respectively.

[0051] For models containing an encoder, a tokenizer can be set before the embedding and positional encoding module 11. The tokenizer can segment the input text to obtain multiple semantic elements. A semantic element is the smallest linguistic unit for dividing the text; for example, a semantic element can be a character or a word. Then, the tokenizer can represent each semantic element with a numerical value, thereby obtaining the input sequence corresponding to the input text. Thus, the input sequence corresponding to the input text is a numerical representation of multiple semantic elements in the input text. In this embodiment, the semantic elements obtained by the tokenizer and the numerical values ​​used to represent the semantic elements can both be called tokens.

[0052] The embedding and positional encoding module 11 receives the input sequence corresponding to the input text and performs word embedding and positional encoding on each token in the input sequence to obtain an initial encoding vector for each token. This initial encoding vector contains both the semantic information of the corresponding token and its positional information within the input sequence. Then, the embedding and positional encoding module 11 outputs the initial encoding vectors for each token in the input sequence.

[0053] After obtaining the initial encoding vector of each token through the embedding and position encoding module 11, the subsequent transformer layers can process the received encoding vector of each token based on the self-attention mechanism to obtain the final encoding vector of each token.

[0054] For example, transformer layer 12 can receive the initial encoding vectors of each token output by embedding and position encoding module 11. Based on the initial encoding vectors of each token, it generates a query vector, key vector, and value vector for each token. It then forms a value matrix from the value vectors of each token and a key matrix from the key vectors of each token. Next, using the query vector, key matrix, and value matrix of each token, it calculates the weighted value vector of each token and performs a linear transformation on the weighted value vector of each token to output the intermediate encoding vector of each token. Afterward, transformer layer 12 can output the intermediate encoding vector of each token.

[0055] Transformer layer 13 receives the intermediate encoded vectors of each token output from transformer layer 12 and further processes these vectors to obtain a weighted vector for each token. Based on this weighted vector, it outputs the encoded vector for each token. The implementation details can be found in the description of transformer layer 12 above. Since transformer layer 13 is the last transformer layer, the encoded vectors of each token obtained by transformer layer 13 are the final encoded vectors of each token obtained by the encoder. Based on the final encoded vectors of each token output by transformer layer 13, subsequent training or inference can be performed.

[0056] Figure 2 A schematic diagram of a decoder structure is shown. Figure 2 As shown, the decoder 20 may include an embedding and position encoding module 21 and multiple transformer layers ( Figure 2 The image shows two transformer layers, 22 and 23 respectively.

[0057] The embedding and position encoding module 21 functions similarly to the embedding and position encoding module 11, except that it receives the target sequence. In the training scenario, this target sequence includes the numeric sequence corresponding to the prompt text and the numeric sequence corresponding to the complete training text generated by the decoder 20. In the inference scenario, such as... Figure 2 As shown, the target sequence may include the number sequence corresponding to the prompt text and the number sequence corresponding to the predicted text that the decoder 20 has generated based on the prompt text before the current moment.

[0058] The functions of transformer layer 22 and transformer layer 23 are similar to those of the transformer layer in the encoder described above. The difference is that, during training, when transformer layer 22 and transformer layer 23 calculate the weighted vector of each token in the target sequence, they can only consider the influence of tokens preceding that token in the target sequence on that token, without considering the influence of tokens following that token.

[0059] Furthermore, for models that simultaneously include encoders and decoders, transformer layers 22 and 23 can also be used to receive the final encoded vectors of each token output by the encoder. Based on this, after calculating the weighted vectors of each token in the target sequence based on the self-attention mechanism, transformer layers 22 and 23 can also calculate the encoded vectors of each token in the target sequence based on the weighted vectors of each token in the target sequence, the key vectors and value vectors corresponding to the encoded vectors of each token output by the encoder, that is, transformer layers 22 and 23 consider the influence of the tokens represented by the encoded vectors output by the encoder on each token in the target sequence of the decoder 20.

[0060] 2. Self-attention mechanism

[0061] Self-attention refers to the process of considering the relationship between each token in a sequence and other tokens in the sequence when processing a sequence, or in other words, the influence of other tokens on the token. This helps the model better understand the contextual information in the sequence and obtain more accurate processing results.

[0062] The self-attention mechanism is implemented as follows: Based on the encoding vector of each token in the sequence, a query vector, key vector, and value vector are generated for each token. The key vectors of each token are combined into a key matrix, and the value vectors of each token are combined into a value matrix. Then, based on the query vector and key matrix of each token, an attention weight vector is calculated for each token. Each attention weight in this vector represents the relevance between the value vector of that token and a certain key vector. Finally, based on the attention weight vector and value matrix of each token, a weighted value vector for that token is calculated.

[0063] Therefore, the more tokens a sequence contains, the larger the key and value matrices become during the implementation of the self-attention mechanism, resulting in greater computational pressure on the model. Furthermore, during the self-attention mechanism, the key and value matrices can be cached in a KV cache for easy access later, reducing computational load. However, the larger the key and value matrices become, the more KV cache is used, leading to greater data reads when retrieving them from the KV cache, thus increasing memory access pressure.

[0064] 3. Attention sparse processing

[0065] As discussed earlier regarding self-attention mechanisms, the more tokens the sequence contains, the larger the key and value matrices become, leading to increased computational pressure on the model. Sparse attention processing involves selecting a subset of tokens from the sequence and using their key and value vectors to construct sparse key and value matrices, respectively, thereby reducing their size. Reducing the size of these matrices decreases the computational load during self-attention implementation. Furthermore, caching the key and value matrices in a key-value cache further reduces cache usage and memory access pressure during subsequent calls to these matrices.

[0066] 4. Dependency parsing

[0067] Dependency analysis refers to the analysis of dependency relationships between semantic elements in a sentence. A sentence contains a central semantic element, and all other semantic elements in the sentence depend directly or indirectly on this central semantic element. The dependency relationships between semantic elements can be represented using a dependency tree. A dependency tree is a directed graph in which, for two semantic elements with a direct dependency relationship, an arrow points from the dependent semantic element to the dependent semantic element. Thus, in the dependency tree, the semantic element to which no arrow points is the central semantic element of the sentence; this central semantic element can also be called the root semantic element or root token.

[0068] The text processing method provided in the embodiments of this application will be described in detail below.

[0069] Figure 3 A flowchart illustrating a text processing method provided in this application embodiment. This method can be applied to a text processing device (hereinafter referred to as the processing device). The processing device includes a target model, which is an AI model based on the transformer architecture. See [link to relevant documentation]. Figure 3 The method includes the following steps S301 to S304, which will be described in detail below.

[0070] S301: Perform dependency analysis on multiple semantic elements in the first input text to obtain at least one root semantic element.

[0071] In this embodiment of the application, the processing device can acquire a first input text, divide the first input text to obtain at least one statement, perform dependency analysis on the semantic elements in the at least one statement, and thereby obtain at least one root semantic element.

[0072] In an inference scenario, when the target model in the processing device is an encoder-only model, the first input text can be the source text to be processed. When the target model in the processing device is a decoder-only model, the first input text can include a prompt text entered by the user; optionally, the first input text can also include target text that the processing device has generated based on the prompt text before the current moment. When the target model in the processing device is an encoder-decoder model, the first input text can be the source text to be processed by the encoder in the target model, or the first input text can be the target text generated before the current moment by the decoder in the target model.

[0073] In a training scenario, when the target model in the processing device is an encoder-only model, the first input text can be the source text to be processed. When the target model in the processing device is a decoder-only model, the first input text can include training prompt text and the target text that the processing device is expected to generate based on the prompt text. When the target model in the processing device is an encoder-decoder model, the first input text can be the source text to be processed by the encoder in the target model, or the first input text can be the target text corresponding to the source text to be processed by the decoder in the target model.

[0074] In some embodiments, after acquiring the first input text, the processing device can input the first input text into a word segmenter to segment the first input text into words, thereby obtaining an input sequence representing the first input text. This input sequence includes numerical values ​​representing each semantic element in the first input text. The processing device can then input the input sequence into a target model, which can perform initial encoding on the input sequence to obtain an initial encoding matrix representing the first input text. Based on this initial encoding matrix, the first input text is segmented to obtain at least one sentence in the first input text. Dependency analysis is performed on the semantic elements in the at least one sentence to obtain at least one root semantic element.

[0075] For example, the processing device may include a tokenizer. The first input text is input to the tokenizer. The tokenizer segments the first input text into words, thereby obtaining multiple semantic elements in the first input text. Then, each semantic element is represented by a numerical value, thereby obtaining the input sequence corresponding to the first input text. The semantic elements in the text and the numerical values ​​used to represent the semantic elements in the input sequence can both be referred to as tokens.

[0076] For example, the first input text includes an English sentence: "A little girl is looking at a black dog." This English sentence is segmented into words, with each word treated as a token. Including the sentence terminator, the English sentence contains 10 tokens. Each token is represented by a numerical value, resulting in an input sequence containing 10 tokens. For example, this input sequence is [6,61,33,10,56,20,6,26,34,5], where 6 represents the numerical representation of the word "a," 61 represents the numerical representation of the word "little," and so on. It should be noted that the numerical values ​​used to represent each word can be determined using a preset dictionary. The numerical values ​​used to represent each word in the input sequence corresponding to the English sentence above are merely an example and do not constitute any limitation on the embodiments of this application.

[0077] After obtaining the input sequence, the tokenizer can output the obtained input sequence to the target model. The target model includes an embedding and positional encoding module and a dependency parsing module. The tokenizer outputs the input sequence to the embedding and positional encoding module. This module encodes each token in the input sequence to obtain an initial encoding matrix corresponding to the input sequence. This initial encoding matrix includes an initial encoding vector for each token in the input sequence. Then, the embedding and positional encoding module outputs the initial encoding matrix corresponding to the input sequence to the dependency parsing module.

[0078] For example, suppose the first input text contains n tokens, where the initial encoding vector a of the i-th token is... i If the initial encoding matrix A output by the embedding and positional encoding modules is an m-dimensional vector, then it can be represented by Equation 1 below.

[0079]

[0080] The dependency analysis module determines the target encoding vector based on the initial encoding vectors of each token in the initial encoding matrix. The token represented by this target encoding vector is the sentence terminator. Then, based on the determined target encoding vector, the dependency analysis module divides the initial encoding vector matrix into at least one submatrix. Each submatrix includes the initial encoding vectors of multiple tokens in a sentence. Dependency analysis is performed on each submatrix to obtain the root semantic element, or root token, of the sentence represented by that submatrix. Thus, by performing dependency analysis on each of the at least one submatrix, at least one root token corresponding to at least one sentence represented by each submatrix can be obtained.

[0081] The dependency parsing module can employ state-of-the-art neural parsing models or transition-based dependency parsing to perform dependency analysis on at least one submatrix, thereby obtaining the dependency tree corresponding to the statement represented by each submatrix. The token indicated by the root label in the dependency tree is then determined as the root token. Alternatively, the dependency parsing module can use a single-layer parsing network to perform dependency analysis on each submatrix. In this case, the parsing network does not need to output the complete dependency tree; it can directly output the root token corresponding to each submatrix.

[0082] In other embodiments, after acquiring the first input text, the processing device can input the first input text into a word segmenter to segment the first input text, thereby obtaining an input sequence representing the first input text. This input sequence includes numerical values ​​representing semantic elements in the first input text. Based on this input sequence, the first input text is divided to obtain at least one sentence. Dependency analysis is performed on the semantic elements in the at least one sentence to obtain at least one root token. Then, the processing device can input the input sequence and at least one root token into a target model, where the target model performs initial encoding on the input sequence to obtain an initial encoding matrix corresponding to the input sequence.

[0083] It should be noted that in this implementation, the processing device may include a word segmenter and a dependency analysis module. Based on this, the processing device can perform word segmentation on the first input text using the word segmenter to obtain the corresponding input sequence; the relevant implementation methods can also be referred to the foregoing description.

[0084] After obtaining the input sequence, the processing device can determine the token used to represent sentence terminators in the input sequence through the dependency analysis module. Based on the token used to represent sentence terminators, the input sequence is divided into at least one subsequence, each of which is a numerical representation of a sentence. Furthermore, the processing device can perform dependency analysis on each subsequence through the dependency analysis module to obtain the root token corresponding to each subsequence. The dependency analysis module can employ the models, modules, or parsing networks described above for dependency analysis to perform dependency analysis on each subsequence; no specific limitations are specified here.

[0085] After determining at least one root token, the processing device can input the input sequence corresponding to the first input text and the at least one root token into the target model. The target model includes an embedding and positional encoding module. The embedding and positional encoding module can perform initial encoding on the input sequence as described above, thereby obtaining an initial encoding matrix representing the first input text. From this initial encoding matrix, the initial encoding vector corresponding to each root token can be obtained.

[0086] S302: Select multiple target semantic elements from multiple semantic elements, wherein the relevance of the multiple target semantic elements to at least one root semantic element is greater than the relevance of other semantic elements among the multiple semantic elements to at least one root semantic element.

[0087] After determining at least one root token in the first input text, the processing device can select multiple target tokens that are highly correlated with the at least one root token from the multiple tokens included in the first input text.

[0088] For example, the processing device may obtain the key vector of each token and the query vector of each root token in the first input text; determine the similarity between the key vector of each token and the query vector of each root token; determine multiple target similarities from the determined multiple similarities, wherein the multiple target similarities are greater than other similarities among the multiple similarities; and determine multiple target tokens based on the tokens corresponding to the key vectors corresponding to the multiple target similarities.

[0089] In some embodiments, the target model in the processing apparatus may further include a sparsity processing module. The embedding and positional encoding module of the target model may output an initial encoding matrix representing the first input text and initial encoding vectors for each root token, wherein the initial encoding matrix includes the initial encoding vectors of each token in the first input text. Alternatively, the embedding and positional encoding module of the target model may output an initial encoding matrix representing the first input text; the dependency analysis module may output an initial encoding vector for at least one root token. Based on this, the sparsity processing module may receive the initial encoding matrix and the initial encoding vectors of at least one root token, obtain the key vector of each token based on the initial encoding vectors of each token in the initial encoding matrix, and determine the query vector for each root token based on the initial encoding vectors of the at least one root token.

[0090] The sparse processing module can multiply the initial encoding vector of each token included in the initial encoding matrix by the key parameter matrix to perform a linear transformation on the initial encoding vector of each token, thereby obtaining the key vector of each token.

[0091] For example, the key parameter matrix is ​​W k If the initial encoding matrix is ​​A as shown in Equation 1 above, then the key vector k of the i-th token is... i As shown in Equation 2 below, i can be equal to 1, 2, 3, ..., n.

[0092]

[0093] Additionally, the sparse processing module can multiply the initial encoding vectors of each token included in the initial encoding matrix by the query parameter matrix to perform a linear transformation on the initial encoding vectors of each token, thereby obtaining the query vector for each token. From the obtained query vectors of multiple tokens, at least one query vector for the root token is obtained.

[0094] For example, the query parameter matrix is ​​W q If the initial encoding matrix is ​​A as shown in Equation 1 above, then the query vector q of the i-th token is... i As shown in Equation 3 below. Based on this, assuming that at least one root token includes the second token, then q2 can be obtained.

[0095]

[0096] Optionally, since the value vectors of the tokens will be used in subsequent attention sparsity processing, the sparsity processing module can also multiply the initial encoding vectors of each token included in the initial encoding matrix by the value parameter matrix to perform a linear transformation on the initial encoding vectors of each token, thereby obtaining the value vectors of each token.

[0097] For example, the value parameter matrix is ​​W v If the initial encoding matrix is ​​A as shown in Equation 1 above, then the query vector v of the i-th token is... i As shown in Equation 4 below.

[0098]

[0099] It should be noted that in the training scenario, the key parameter matrix, query parameter matrix, and value parameter matrix mentioned above are all parameters that the target model needs to learn; in the inference scenario, the key parameter matrix, query parameter matrix, and value parameter matrix are all parameters that the target model has already learned.

[0100] After obtaining the key vectors of each token and the query vector of at least one root token from the first input text, the sparse processing module can calculate the similarity between the query vector of each root token and the key vectors of each token. The similarity between the query vector of the root token and the key vector of any token can be used to characterize the relevance or association between the two tokens. A higher similarity between the query vector of the root token and the key vector of the token indicates a greater relevance or association between the two tokens.

[0101] Taking the calculation of the similarity between a root token and a token among multiple tokens as an example, in one example, when the query vector of the root token and the key vector of the token have the same dimension, the sparse processing module can calculate the dot product between the query vector of the root token and the key vector of the token, thereby obtaining the similarity between the root token and the token.

[0102] In another example, when the query vector of the root token and the key vector of the token have different dimensions, the sparse processing module can concatenate the query vector of the root token and the key vector of the token, and then process them through a multilayer perceptron to obtain the similarity between the root token and the token.

[0103] Based on the above method, for each root token, the sparse processing module can calculate the similarity between the root token and each other token. Therefore, for each root token, the similarity between the root token and multiple tokens in the first input text includes the similarity between the root token and itself, as well as the similarity with other root tokens. Based on this, the sparse processing module can remove the similarity between root tokens from the calculated similarity scores, and then sort the remaining similarities, selecting the tokens with the highest similarity to the root token (ranked in the top M positions) as target tokens. Thus, the determined target tokens do not include at least one root token.

[0104] Where M is a preset value, and M is an integer greater than 1. In this embodiment, the size of M can be set by the user. The smaller the value of M, the fewer target tokens are selected, and subsequently, the smaller the sparse bond matrix and sparse value matrix obtained based on the target tokens, resulting in a more significant reduction in model computation. The larger the value of M, the more target tokens are selected, and the larger the sparse bond matrix and sparse value matrix become. Correspondingly, the impact on the model's prediction accuracy is smaller. Therefore, the user can set M based on their expectations for model computation and prediction accuracy.

[0105] S303: Based on multiple target semantic elements, perform attention sparsity processing to obtain the first processing result.

[0106] After identifying multiple target tokens, the processing device can perform attention sparsity processing based on these multiple target tokens to obtain the first processing result.

[0107] As described above, attention sparsity processing refers to constructing a sparse key matrix and a sparse value matrix by selecting a subset of tokens, thereby reducing the size of the key matrix and value matrix. Based on this, in this embodiment, the processing device can determine a global sparse key matrix and a global sparse value matrix based on the multiple target tokens, thus implementing attention sparsity processing. The first processing result includes the global sparse key matrix and the global sparse value matrix.

[0108] For example, after determining multiple target tokens, the sparse processing module in the target model can obtain the key vectors of multiple target tokens from the key vectors of multiple tokens in the first input text calculated by Equation 2. Based on the key vectors of these multiple target tokens, a global sparse key matrix is ​​generated.

[0109] In one example, the sparse processing module can directly combine the key vectors of multiple target tokens into a global sparse key matrix.

[0110] In another example, as described above, the multiple target tokens do not include the root token. Based on this, the sparse processing module can use the key vectors of the multiple target tokens and at least one root token in the first input text to form a global sparse key matrix.

[0111] For example, the key vectors of multiple target tokens are k2 and k9, and the key vectors of the root token contained in the first input text are k3 and k9. 11 Each key vector is an m-dimensional vector. Based on the key vectors of the multiple target tokens and the key vector of the root token, a global sparse key matrix as shown in Equation 5 can be formed.

[0112]

[0113] In addition, the sparse processing module can obtain the value vectors of multiple target tokens and generate a global sparse value matrix based on these value vectors.

[0114] If the sparse processing module has generated value vectors for each token in S302, then in this step, the sparse processing module can obtain the value vectors of multiple target tokens from these value vectors. If the sparse processing module has not generated value vectors for each token in S302, then in this step, the sparse processing module can calculate the value vectors of multiple target tokens based on the initial encoding vectors and value parameter matrices of the multiple target tokens, referring to the calculation method in Equation 4.

[0115] After obtaining the value vectors of multiple target tokens, the sparse processing module can refer to the aforementioned method for generating a global sparse key matrix to generate a global sparse value matrix based on the value vectors of these multiple target tokens. This global sparse value matrix includes the value vectors of the tokens corresponding to each key vector in the global sparse key matrix.

[0116] For example, when the global sparse key matrix is ​​as shown in Equation 5, and the value vector of each token is also an m-dimensional vector, the global sparse value matrix can be as shown in Equation 6.

[0117]

[0118] It should be noted that if the global sparse key matrix and the global sparse value matrix do not include information about at least one root token, the first processing result will also include the key vector and value vector of at least one root token.

[0119] After obtaining the first processing result, the sparse processing module can output the first processing result.

[0120] It should be noted that in the inference scenario, the first processing result output by the sparse processing module can not only be entered into the prediction module of the target model for subsequent prediction, but the processing device can also store the first processing result in the cache so that the subsequent prediction module can call it when performing multiple self-attention calculations, thereby reducing the computational load of the model.

[0121] For example, the processing device in this application embodiment can be deployed in a computer device, which can allocate a key-value (KV) cache for the target model in the processing device, and the processing device can store the first processing result in the KV cache.

[0122] It is worth noting that this application embodiment mainly uses single-head attention as an example to illustrate the process of attention sparsity processing. In some possible implementations, the processing device can perform multi-head attention sparsity processing. In this case, there will be multiple sets of parameter matrices, each set of parameter matrices including a key parameter matrix, a query parameter matrix, and a value parameter matrix. Accordingly, based on each set of parameter matrices, the query vector, key vector, and value vector of each token in the first input text under the corresponding parameter matrix can be obtained through the aforementioned equations 2 to 4. Then, attention sparsity processing is performed based on the query vector, key vector, and value vector of each token under the corresponding parameter matrix to obtain the sparsity processing result under the corresponding parameter matrix. That is, in this case, the first processing result can include the multi-head attention sparsity processing result. Subsequently, for any processing result in the multi-head attention sparsity processing result, the weighted value vector of each token corresponding to the corresponding sparsity processing result can be determined by referring to the implementation of S304. Finally, the multiple weighted value vectors of each token obtained using the multi-head attention sparsity processing result are fused to obtain the weighted value vector of each token. The following text will still use the first processing result, which is the result of single-head attention sparse processing, as an example for illustration.

[0123] S304: Generate the first prediction result based on the first processing result.

[0124] After obtaining the first processing result, the processing device can determine the weighted value vector of each token in the first input text based on the first processing result, and then generate a first prediction result based on the weighted value vector of each token.

[0125] For example, the target model may also include a prediction module that can receive a first processing result output by the sparse processing module and generate a first prediction result based on the first processing result.

[0126] In one possible implementation, the prediction module can determine the sparse key matrix and sparse value matrix corresponding to each of the multiple tokens in the first input text, based on the global sparse key matrix and global sparse value matrix included in the first processing result. The sparse key matrix corresponding to each token includes the key vector in the global sparse key matrix and the key vectors of the corresponding token's neighboring tokens; the sparse value matrix corresponding to each token includes the value vector in the global sparse value matrix and the value vectors of the corresponding token's neighboring tokens. Then, based on the query vector of each token, the corresponding sparse key matrix, and the sparse value matrix, a weighted value vector for each token is determined; and based on the weighted value vector of each token, a first prediction result is generated.

[0127] It should be noted that the prediction module may include one or more transformer layers. The first transformer layer in the prediction module receives the first processing result from the sparse processing module. Then, starting from the first token in the first input text, this transformer layer sequentially calculates the weighted vector for each token based on the first processing result and each token's neighboring tokens.

[0128] Taking a token from the first input text as an example (referred to as the first token for ease of description), the transformer layer can obtain the query vector of the first token and the key and value vectors of its neighboring tokens. The key vectors of the neighboring tokens are added to the global sparse key matrix. If the global sparse key matrix does not include the key vector of at least one root token, then the key vector of at least one root token is also added to the global sparse key matrix, thus obtaining the sparse key matrix corresponding to the first token. Similarly, the value vectors of the neighboring tokens are added to the global sparse value matrix. If the global sparse value matrix does not include the value vector of at least one root token, then the value vector of at least one root token is also added to the global sparse key matrix, thus obtaining the sparse value matrix corresponding to the first token.

[0129] It should be noted that the global sparse key matrix and global sparse value matrix contain information about tokens in the first input text that are highly relevant to the root token; that is, they contain the global semantic information of the first input text. Based on this, for a given token, the key vectors and value vectors of its neighboring tokens are added to the global sparse key matrix and global sparse value matrix, respectively. This means that the local semantic information of that token is also added to the sparse key matrix and sparse value matrix. Thus, the final sparse key matrix and sparse value matrix corresponding to that token will simultaneously contain both global and local semantic information. This achieves sparse attention processing while better ensuring the accuracy of subsequent predictions.

[0130] For example, in one possible scenario, the prediction module could be a prediction module within an encoder. Accordingly, the adjacent tokens of the first token could include the N tokens preceding and following the first token in the first input text. Specifically, when the number of tokens preceding or following the first token in the first input text is less than N, the adjacent tokens of the first token will include S tokens preceding or following the first token, where S is less than N. For instance, when the first token is the first token in the first input text, since there are no other tokens preceding it, the adjacent tokens of the first token include the N tokens following it.

[0131] In another possible scenario, the prediction module could be a prediction module within a decoder. Accordingly, the adjacent tokens of the first token could include the N tokens preceding the first token in the first input text. Similarly, when there are fewer than N tokens preceding the first token, only the tokens preceding the first token (less than N) need to be retrieved. When the first token is the first token in the first input text, since there are no other tokens preceding it, it is impossible to retrieve the adjacent tokens of the first token.

[0132] Furthermore, N can be a statically configured value or a user-defined value. For example, N can be equal to 2. It's worth noting that the larger N is, the more neighboring tokens of the first token are likely to be obtained. Correspondingly, the sparse key matrix (containing the key vectors of the first token's neighboring tokens) and the sparse value matrix (containing the value vectors of the first token's neighboring tokens) will be larger, resulting in higher accuracy of the weighted value vector of the first token calculated based on these sparse key and value matrices. Conversely, the smaller N is, the fewer neighboring tokens of the first token are likely to be obtained. Correspondingly, the increment of the sparse key matrix (containing the key vectors of the first token's neighboring tokens) and the sparse value matrix (containing the value vectors of the first token's neighboring tokens) will be smaller, resulting in better sparsity.

[0133] Optionally, if the information of the neighboring tokens of the first token is already included in the global sparse key matrix and the global sparse value matrix, that is, if the global sparse key matrix and the global sparse value matrix include the key vector and value vector of a certain neighboring token of the first token, then the transformer layer does not need to add the key vector and value vector of the neighboring token to the global sparse key matrix and the global sparse value matrix.

[0134] Optionally, the transformer layer can also add the key vector and value vector of the first token to the global sparse key matrix and the global sparse value matrix, respectively.

[0135] After determining the sparse key matrix and sparse value matrix corresponding to the first token based on the neighboring tokens of the first token, the transformer layer can determine the attention weight vector of the first token based on the query vector of the first token and the corresponding sparse value matrix; then, based on the attention weight vector of the first token and the sparse value matrix corresponding to the first token, the weighted value vector of the first token is determined.

[0136] The transformer layer multiplies the query vector of the first token by the transpose of the sparse key matrix corresponding to the first token to obtain the attention weight vector of the first token. This attention weight vector includes the attention weights between the first token and each target token, each root token, and each neighboring token. The transformer layer then normalizes each attention weight in this vector to obtain an attention score for each weight. Finally, the processed attention weight vector is multiplied by the sparse value matrix corresponding to the first token to obtain the weighted vector of the first token. This weighted vector of the first token is then the encoded vector representing the first token, considering the influence of multiple target tokens, at least one root token, and neighboring tokens.

[0137] For example, taking the key vector, query vector, and value vector in Equations 2, 3, and 4 above as examples, assuming that the first token is the sixth token in the first input text, the corresponding query vector, key vector, and value vector are q6, k6, and v6, respectively. The adjacent tokens of the first token are the one before and one after the first token, that is, the adjacent tokens of the first token include the fifth token and the seventh token, with corresponding key vectors k5 and k7, and corresponding value vectors v5 and v7, respectively. Thus, the sparse key matrix K6 and sparse value matrix V6 corresponding to the first token are shown in Equations 7 and 8, respectively.

[0138]

[0139] Based on the query vector of the first token and the corresponding sparse key matrix, the attention weight vector W6 corresponding to the first token can be calculated, as shown in Equation 9 below.

[0140]

[0141] For example, a text containing 10 tokens, Figure 4 This shows the attention weights contained in the attention weight vectors corresponding to each token in the text. See also... Figure 4 In Figures (a) to (d), the square in the i-th row and j-th column represents the attention weight between the i-th token and the j-th token.

[0142] For example, Figure 4Figure (a) illustrates the attention weights between the root token and each token in the text. The root token in the text is the fourth token; therefore, the attention weight vector for each token includes the attention weights between the corresponding token and the fourth token.

[0143] Figure 4 Figure (b) illustrates the attention weights between each token in the text and itself, as well as between each token and its neighboring tokens. Each token's neighboring tokens can include the previous and next tokens. For example, the neighboring token of the first token is the second token; therefore, the attention weight vector of the first token includes the attention weight between the first token and itself, and the attention weight between the first token and the second token.

[0144] Figure 4 Figure (c) illustrates the attention weights between the target tokens selected from the text based on the root token and each token. The target tokens in the text include the 1st token, the 6th token, and the 9th token; therefore, the attention weight vector corresponding to each token can include the attention weights between the corresponding token and each target token.

[0145] comprehensive Figure 4 As shown in Figures (a) to (c), for each token in the text, the attention weight vector corresponding to that token will include the attention weights of that token with the root token, itself, neighboring tokens, and the target token. Figure 4 Figure (d) illustrates the attention weights included in the attention weight vector for each token. For example, for the 5th token in the text, the attention weight vector for the 5th token includes the attention weights between this token and the 1st token (target token), the 4th token (which is both the root token and an adjacent token), the 5th token (itself), the 6th token (which is both the target token and an adjacent token), and the 9th token (target token).

[0146] After obtaining the attention weight vector, the attention weight vector W6 is normalized, and the weighted value vector B6 of the first token is calculated based on the processed attention weight vector W6' and the sparse value matrix corresponding to the first token, as shown in Equation 10 below.

[0147]

[0148] For each token in the first input text, the transformer layer can refer to the method described above to calculate the weighted vector corresponding to the token.

[0149] After obtaining the weighted vectors of each token, the first transformer layer of the prediction module can further process these vectors using a feedforward network to output the encoded vectors of each token. If the prediction module includes only one transformer layer, the encoded vectors of each token output by that transformer layer are the final encoded vectors of each token. If the prediction module includes multiple transformer layers, the first transformer layer can input the encoded vectors of each token it has obtained into the next transformer layer. The next transformer layer can refer to the method described above to re-obtain the query vector, key vector, and value vector of each token based on the received encoded vectors. Then, based on the multiple target tokens, the key vectors, and value vectors of each token, the global sparse value matrix and the global sparse key matrix are updated. At this point, the updated global sparse key matrix and global sparse value matrix include the updated key vectors and value vectors of the multiple target tokens. Next, referring to the method described above, based on the re-obtained query vectors of each token, the updated global sparse key matrix, and the global sparse value matrix, the encoding vector of each token is calculated, and the encoding vector of each token is output to the next transformer layer. This process continues until the last transformer layer outputs the encoding vector of each token, thus obtaining the final encoding vector of each token.

[0150] It should be noted that in the training scenario, when the prediction module includes multiple transformer layers, in one possible implementation, in addition to the sparse processing module before the first transformer layer of the prediction module, a sparse processing module can be set every L transformer layers. In this case, the sparse processing module located between two transformer layers can receive the encoded vectors of each token output by the preceding transformer layer, and, referring to the method described in S302 above, reselect multiple target tokens with high relevance to the at least one root token determined by the dependency analysis module, thereby obtaining multiple updated target tokens. Then, referring to the method described in S303 above, attention sparse processing is performed again based on these multiple updated target tokens, and the new sparse processing result and the encoded vectors of each token are input to the subsequent transformer layer, so that the subsequent transformer layer can calculate the weighted vector of each token based on the new sparse processing result and the encoded vectors of each token.

[0151] The value L mentioned above can be a default value or a value set by the user. L can be set according to the number of transformer layers included in the prediction module. For example, when the number of transformer layers is T and T is greater than 4, L can be (0.25*T-1). When 0.25*T is not an integer, it can be rounded down.

[0152] In the training scenario, after obtaining the final encoded vectors of each token, the prediction module can perform subsequent prediction tasks based on these vectors to obtain a first prediction result. Then, the processing device can calculate a loss function based on this first prediction result and the label corresponding to the first input text, and update the parameters of the target model based on this loss function.

[0153] In the inference scenario, if the target model is an encoder-only model, the prediction module can generate a first prediction result based on the final encoded vector of each token, which is the final inference result.

[0154] If the target model is a decoder-only model, the processing device can generate a predicted token based on the final encoded vector of each token, and input the predicted token into the target model. The target model performs embedding and positional encoding on the predicted token to obtain an initial encoded vector of the predicted token. Then, based on the initial encoded vector of the predicted token and the first processing result, a weighted vector of the predicted token is calculated. Based on the weighted vector of the predicted token and the weighted vectors of the previous multiple tokens, the next token is predicted. This process continues until the obtained predicted token is a sentence terminator. At this point, the generated tokens, including the sentence terminator, excluding the tokens included in the first input text, are used as the first prediction result. Afterward, the processing device can continue to execute S305 to S308 based on the first prediction result, thereby updating the first processing result to a second processing result, and then using the second processing result and the input text for subsequent inference.

[0155] If the target model is an encoder-decoder model, and the prediction module is the prediction module in the encoder, then the prediction module can input the final encoded vector of each token in the first input text into the decoder so that the decoder can perform subsequent processing and generate the first prediction result.

[0156] The decoder may include an embedding and positional encoding module, a sparse processing module, and a prediction module, and the prediction module of the decoder may also include one or more transformer layers. Optionally, the decoder may also include a dependency parsing module. Based on this, the final encoded vectors of each token in the first input text output by the encoder's prediction module can be used as inputs to each transformer layer in the decoder's prediction module. Furthermore, the decoder first performs attention sparse processing on the tokens in its corresponding third input text using the embedding and positional encoding module, dependency parsing module, and sparse processing module, referring to the methods described in S301 to S303 above, to obtain a third processing result. Then, each transformer layer in the decoder's prediction module can calculate a first-order weighted vector of each token in the third input text based on the third processing result. A key matrix and a value matrix are generated based on the encoded vectors of each token in the first input text. Then, based on these key and value matrices and the first-order weighted value vectors of each token in the third input text, a second-order weighted value vector is calculated for each token in the third input text. The corresponding encoded vector is then output based on these second-order weighted value vectors. The second-order weighted value vectors of each token in the third input text contain not only the contextual information of the third input text but also the semantic information of each token in the first input text. After the final transformer layer in the decoder outputs the final encoded vectors of each token, the decoder can perform inference based on these final encoded vectors to obtain a predicted token. This predicted token is then used as the input for the next inference process, and the decoder generates the next predicted token based on the aforementioned method for generating predicted tokens. This process continues until a predicted token is a sentence terminator, at which point the first predicted statement corresponding to that sentence terminator is taken as the first prediction result. In this case, the decoder can refer to the methods described in S305 to S308 below to update the third processing result based on the first prediction statement, and continue to perform subsequent inference based on the updated third processing result and the input text.

[0157] If the target model is an encoder-decoder model, and the prediction module is the prediction module within the decoder, then each transformer layer in the prediction module can also receive the final encoded vectors of each token in the fourth input text output by the encoder. Based on this, after each transformer layer calculates the first-order weighted vector of each token in the first input text using the method described above, it can also generate a key matrix and a value matrix based on the final encoded vectors of each token in the fourth input text output by the encoder. Based on the key matrix, value matrix, and the first-order weighted vectors of each token in the first input text, it calculates the second-order weighted vector of each token in the first input text. Then, referring to the decoder operations in the encoder-decoder model described above, a first prediction result is generated. In this case, the decoder can continue to execute steps S305 to S308 below, thereby updating the first processing result to a second processing result, which facilitates subsequent inference using the second processing result and the input text.

[0158] In this embodiment, dependency analysis is performed on multiple tokens contained in the first input text to obtain at least one root token, wherein the at least one root token is a token on which other tokens in the first input text depend. In other words, the at least one root token is the core token in the first input text. Based on this, multiple target tokens with high relevance to the at least one root token are selected from the multiple tokens contained in the first input text, and attention sparsity processing is performed using these multiple target tokens. Therefore, this embodiment preserves the semantic dependencies between tokens in the input text during the selection of target tokens. Thus, by using target tokens to construct sparse key matrices and sparse value matrices, the computational cost of the model is reduced while minimizing the impact on the model's prediction accuracy, that is, reducing the impact on the quality of the prediction results generated by the model.

[0159] In addition, the sparse key matrix and sparse value matrix constructed using the target token can be stored in the KV cache for subsequent calls. Since the size of the sparse key matrix and sparse value matrix is ​​smaller than that of the full key matrix and value matrix, the occupancy rate of the KV cache is reduced. Furthermore, when the processing device reads the sparse key matrix and sparse value matrix from the KV cache, the amount of data read is also reduced accordingly, alleviating the memory access pressure.

[0160] Finally, in this embodiment, not only can the root token and target tokens with high relevance to the root token be used to construct sparse key matrices and sparse value matrices, but the local context information of the token can also be added to the sparse key matrix and sparse value matrix when calculating the weighted value vector of each token, thereby improving the inference or training accuracy.

[0161] Based on the above embodiments, in the inference scenario, when the decoder in the target model processes the first input text through S301 to S304 to obtain the first prediction result, the first prediction result will include a latest prediction statement generated by the target model token-by-token based on the first input text. In this embodiment, this prediction statement can be referred to as the first prediction statement. See also... Figure 5 The processing device can then continue to execute S305 to S308 to update the first processing result and generate a prediction result based on the updated processing result. The detailed implementation process of S305 to S308 will be described below.

[0162] S305: Perform dependency analysis on the first prediction statement included in the first prediction result to obtain the newly added root semantic element.

[0163] The processing device can refer to the method described in S301 above to perform dependency analysis on the first prediction statement to obtain the root token of the first prediction statement, which is the newly added root token.

[0164] S306: Determine multiple update target semantic elements from multiple candidate semantic elements, wherein the relevance of these multiple update target semantic elements to the newly added root semantic element is greater than the relevance of other semantic elements to the newly added root semantic element.

[0165] After obtaining a new root token, the processing device can add the new root token to at least one root token obtained in S301, thereby obtaining multiple root tokens.

[0166] The processing device can use the multiple target tokens selected in S302 and the tokens included in the first prediction statement as multiple candidate tokens. Using multiple root tokens, it reselects multiple target tokens from these candidate tokens that have a high correlation with the root tokens. These reselected target tokens are called multiple updated target tokens. The multiple updated target tokens do not contain the root tokens.

[0167] For example, the processing device can read a global sparse key matrix containing key vectors of multiple target tokens and at least one root token from the KV cache. Using the query vectors of the multiple root tokens, the key vectors in the read global sparse key matrix, and the key vector of the token in the first prediction statement, the similarity between each root token and the aforementioned key vectors is calculated, referring to the implementation method in S302. Then, based on the calculated similarity, M tokens corresponding to key vectors with high similarity are selected as multiple update target tokens. Detailed implementation methods can be found in the description in S302, and will not be repeated here. The selected multiple update target tokens have a higher correlation with the newly added root token than other tokens have a higher correlation with the newly added root token. Of course, the correlation between these multiple update target tokens and other root tokens is also higher than the correlation between other tokens and other root tokens.

[0168] S307: Based on multiple updated target semantic elements, attention sparsity processing is performed to obtain the second processing result.

[0169] After obtaining multiple target tokens for updating, the processing device can, referring to the method described in S303 above, update the global sparse key matrix and the global sparse value matrix based on these multiple target tokens to obtain a second processing result. This second processing result includes the updated global sparse key matrix and the updated global sparse value matrix. The updated global sparse key matrix includes key vectors of multiple target tokens for updating, and optionally, key vectors of multiple root tokens. If the updated global sparse key matrix does not include key vectors of multiple root tokens, the second processing result may include key vectors of multiple root tokens. The updated global sparse value matrix includes value vectors of multiple target tokens for updating, and optionally, value vectors of multiple root tokens. If the updated global sparse value matrix does not include value vectors of multiple root tokens, the second processing result may include value vectors of multiple root tokens.

[0170] After obtaining the second processing result, the processing device can replace the first processing result in the KV cache with the second processing result.

[0171] Based on the acquisition process of the first and second processing results, it can be seen that both the first and second processing results include information on M target tokens and a root token. The difference lies in that the second processing result not only includes the root token information already present in the first processing result but also includes information on newly added root tokens. Therefore, in the inference scenario, whenever a prediction statement is generated, the sparse processing results stored in the KV cache will have a new key vector and value vector for the root token added; that is, two vectors will be added to the KV cache. In related technologies, each time a token is generated, the KV cache will have its corresponding key vector and value vector added; that is, each time a token is generated, the KV cache will have two vectors added. When the prediction statement includes j tokens, each time a prediction statement is generated, the KV cache will have 2*j vectors added. Therefore, compared to related technologies, in the incremental inference scenario, the embodiments of this application can reduce the KV cache occupancy rate and also reduce the amount of data read by the processing device when calling the sparse processing results in the KV cache.

[0172] S308: Generate a second prediction result based on the second input text and the second processing result, wherein the second input text includes the first prediction result.

[0173] After obtaining the second processing result, the processing device can subsequently calculate the weighted vector of each token in the second input text based on the second processing result, and then generate a second prediction result based on the weighted vector of each token. The second input text may include the first prediction result. Optionally, the second input text may also include the first input text.

[0174] In this embodiment, whenever a new predicted statement is generated, dependency analysis can be performed on the predicted statement to obtain its root token. Based on this, and using the newly added root token and the first processing result, a second processing result is obtained, including the updated global sparse value matrix and global sparse key matrix. This second processing result contains the context information of the newly generated predicted statement. Thus, continuing subsequent inference based on the second processing result can improve inference accuracy.

[0175] Based on the text processing method described in the above embodiments, the following will provide illustrative examples of how the model uses this text processing method to make predictions in both training and inference scenarios.

[0176] In the first exampleTaking an encoder-only model as an example, this paper describes the process of using this text processing method to make predictions using this target model in a training scenario. In this case, the processing device can also be called a model training device. For example, see [link to example]. Figure 6 The target model may include a word segmenter, an embedding and position encoding module, a dependency parsing module, a sparse processing module, a prediction module, and an output layer. The sparse processing module may have one or more components. Figure 6 The diagram illustrates two sparse processing modules: sparse processing module 1 and sparse processing module 2. The prediction module may include multiple transformer layers. Figure 6 The diagram shows four transformer layers, numbered 1 to 4. Sparse processing module 1 is located before the first transformer layer, and sparse processing module 2 is located before the third transformer layer. In other words, a sparse processing module is set up every two transformer layers.

[0177] like Figure 6 As shown, text A is input into a word segmenter. The word segmenter performs word segmentation and digitization on text A, thereby outputting the corresponding numerical sequence of text A. Here, text A is a training sample, and text A may correspond to a label.

[0178] The embedding and positional encoding module receives the numerical sequence output by the word segmenter. It performs word embedding and positional encoding on this numerical sequence to output the initial encoding vector of text A. This initial encoding vector of text A can be used as input to the dependency parsing module, sparse processing module 1, and the first transformer layer (transformer layer 1) in the prediction module.

[0179] The dependency analysis module performs dependency analysis on text A based on the initial encoding vector of text A, thereby outputting the root token in text A.

[0180] Sparse processing module 1 performs attention sparsity processing based on the root token and initial encoding vector in text A, thereby outputting a global sparse key matrix K. globa1l and the global sparse value matrix V globa1l The K globa1l and V global1 This is the input to the first two transformer layers in the prediction module.

[0181] The transformer layer 1 in the prediction module is based on the initial encoding vector of text A, K globa1l V global1Calculate the weighted vector of each token in text A, and output the intermediate encoding vector of each token in text A based on the weighted vector of each token.

[0182] Transformer layer 2 is based on the intermediate encoding vector output by transformer layer 1, K globa1l V global1 The algorithm calculates the weighted vector of each token in text A and outputs the intermediate encoding vector of each token in text A based on the weighted vector of each token. The intermediate encoding vector of each token in text A output by transformer layer 2 can be used as the input to sparse processing module 2 and transformer layer 3.

[0183] It should be noted that before calculating the weighted vector of each token in text A, transformer layer 2 first calculates the weighted vector of K based on the intermediate encoding vector output by transformer layer 1. globa1l V global1 The key and value vectors of each token contained within are updated. Then, based on the updated K... globa1l V global1 To calculate the weighted vector of each token in text A.

[0184] Based on the root token in text A output by the dependency parsing module and the intermediate encoding vector of each token in text A output by transformer layer 2, sparse processing module 2 performs attention sparsity processing again, thereby outputting the updated global sparse key matrix K. globa12 and V global2 The updated global sparse bond matrix and global sparse value matrix can be used as inputs to the last two transformer layers in the prediction module.

[0185] Transformer layer 3 is based on the intermediate encoded vector output by transformer layer 2, K globa12 and V global2 Calculate the weighted vector of each token in text A, and output the intermediate encoding vector of each token in text A based on the weighted vector of each token.

[0186] Transformer layer 4 is based on the intermediate encoding vector output by transformer layer 3, K globa12 and V global2The first layer calculates the weighted vector of each token in text A and outputs the final encoded vector of each token in text A based on the weighted vector of each token. The output layer processes the final encoded vectors of each token output by transformer layer 4 to output the prediction result.

[0187] Before calculating the weighted vector of each token in text A, transformer layer 4 first calculates the weighted vector of K based on the intermediate encoding vector output by transformer layer 3. globa12 V global2 The key and value vectors of each token contained within are updated. Then, based on the updated K... globa12 V global2 To calculate the weighted vector of each token in text A.

[0188] After obtaining the prediction result using the target model through the process described above, the processing device can calculate the loss function value based on the prediction result and the label corresponding to text A, and then update the parameters in the target model based on the loss function value. For example, it can update the query parameter matrix, key parameter matrix and value parameter matrix of each transformer layer of the prediction module of the target model, thereby realizing the training of the target model.

[0189] In the second example Taking a decoder-only model as an example, this paper describes the process of using this text processing method to make predictions using the target model in a reasoning scenario. In this case, the processing device can also be called a model inference device. For example, see [link to example]. Figure 7 The target model may include a tokenizer, embedding and position encoding modules, dependency parsing modules, sparse processing modules, prediction modules, and an output layer. The prediction module may include multiple transformer layers. Figure 7 The diagram shows two transformer layers, referred to as transformer layer 1 and layer 2.

[0190] like Figure 7 As shown, text B is input into a word segmenter. The word segmenter performs word segmentation and digitization on text B, thereby outputting a numerical sequence corresponding to text B. Here, text B is a prompt text, which may include the prompt text entered by the user and supplementary prompt text based on the user's input.

[0191] The embedding and positional encoding module receives the numeric sequence output by the word segmenter. It performs word embedding and positional encoding on this numeric sequence to output the initial encoding vector of text B. This initial encoding vector of text B can be used as input to transformer layer 1 in the dependency parsing module, sparse processing module, and prediction module, respectively.

[0192] The dependency analysis module performs dependency analysis on text B based on the initial encoding vector of text B, thereby outputting the root token in text B.

[0193] The sparsity processing module performs attention sparsity processing based on the root token and initial encoding vector in text B, thereby outputting a global sparse key matrix K. globa1 and the global sparse value matrix V globa1 The global sparse bond matrix K globa1 and the global sparse value matrix V globa1 This is the input to the transformer layer in the prediction module.

[0194] The transformer layer 1 in the prediction module is based on the initial encoding vector of text B, K globa1 V globa1 Calculate the weighted vector of each token in text B, and output the intermediate encoding vector of each token in text B based on the weighted vector of each token.

[0195] Transformer layer 2 is based on the intermediate encoding vector output by transformer layer 1, K globa1 K globa1 Calculate the weighted vector of each token in text B, and output the final encoded vector of each token in text B based on the weighted vector of each token.

[0196] Before calculating the weighted vector of each token in text B, transformer layer 2 first calculates the weighted vector of K based on the intermediate encoding vector output by transformer layer 1. globa1 V global The key and value vectors of each token contained within are updated. Then, based on the updated K... globa1 V global To calculate the weighted vector of each token in text B.

[0197] The output layer outputs a predicted token based on the final encoding vector output by transformer layer 2, at which point one inference process ends. This predicted token is added to text B and used as input to the tokenizer in the next round of inference, thus repeating the above process for the next inference. This continues until the predicted token output by the output layer is a sentence terminator. At this point, the initial encoding vector of sentence C containing that sentence terminator, output by the embedding and position encoding module, is used as input to the dependency parsing module and the sparse processing module.

[0198] The dependency parsing module performs dependency parsing on sentence C based on the initial encoding vector of sentence C, thereby obtaining the root token of sentence C. The root token of sentence C is the newly added root token, and the newly added root token is output.

[0199] The sparse processing module is based on the root token of text B, the newly added root token, and the token pairs K in sentence C. globa1 and V globa1 Perform an update and output the updated K. globa1 and V globa1 .

[0200] The transformer layer 1 in the prediction module is based on the initial encoding vectors of text B and sentence C, and the updated K. globa1 and V globa1 Calculate the weighted vector of each token in text B and sentence C, and output the intermediate encoded vector of each token in text B and sentence C based on the weighted vector of each token.

[0201] Transformer layer 2 is based on the intermediate encoding vector output by transformer layer 1 and the updated K. globa1 and V globa1 Calculate the weighted vector of each token in text B and sentence C, and output the final encoded vector of each token in text B and sentence C based on the weighted vector of each token.

[0202] In this process, before calculating the weighted vector of each token in text B and sentence C, transformer layer 2 first updates the K based on the intermediate encoding vector output by transformer layer 1. globa1 and V global The key and value vectors of each token contained within are updated. Then, based on the updated K... globa1 and V global To calculate the weighted vector of each token in text B and sentence C.

[0203] The output layer, based on the final encoding vector output by transformer layer 2, outputs the next predicted token of sentence terminator C and adds this predicted token to the input text to continue the next inference.

[0204] It should be noted that the detailed implementation methods of the operations performed by the word segmenter, dependency analysis module, sparse processing module and prediction module in the two examples above can be found in the relevant implementation methods in the previous embodiments, and will not be repeated here.

[0205] The text processing apparatus provided in the embodiments of this application will be described next.

[0206] Figure 8 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application. See also... Figure 8 The text processing device 800 includes a dependency analysis module 801, a sparse processing module 802, and a prediction module 803.

[0207] The dependency analysis module 801 can be used to execute S301 in the aforementioned embodiments; the sparse processing module 802 can be used to execute S302 and S303 in the aforementioned embodiments; and the prediction module 803 can be used to execute S304 in the aforementioned embodiments.

[0208] Optionally, the first input text includes at least one statement, and the dependency analysis module 801 is specifically used to: perform dependency analysis on the semantic elements in the at least one statement to obtain at least one root semantic element.

[0209] Optionally, the sparse processing module 802 is specifically used to: obtain the key vector of each semantic element and the query vector of each root semantic element; determine the similarity between the key vector of each semantic element and the query vector of each root semantic element; determine multiple target similarities from the determined multiple similarities, wherein the multiple target similarities are greater than other similarities among the multiple similarities; and determine the semantic elements corresponding to the key vectors of the multiple target similarities as the multiple target semantic elements.

[0210] Optionally, the first input text is a prompt text in a reasoning scenario, and the first prediction result includes the first prediction statement; the dependency analysis module 801 is further used to perform dependency analysis on the first prediction statement to obtain a new root semantic element; the sparse processing module 802 is further used to determine multiple updated target semantic elements from multiple candidate semantic elements, and perform attention sparse processing based on the multiple updated target semantic elements to obtain a second processing result, wherein the multiple candidate semantic elements include semantic elements in the first prediction statement and multiple target semantic elements, and the correlation between the multiple updated target semantic elements and the new root semantic element is greater than the correlation between other semantic elements in the candidate semantic elements and the new root semantic element; the prediction module 803 is further used to generate a second prediction result based on the second input text and the second processing result, wherein the second input text includes the first prediction result.

[0211] Optionally, the first processing result is stored in a key-value (KV) cache; the text processing device is also used to replace the first processing result in the KV cache with a second processing result.

[0212] Optionally, the sparse processing module 802 is specifically used to: determine a global sparse key matrix and a global sparse value matrix based on multiple target semantic elements, wherein the first processing result includes a global sparse key matrix and a global sparse value matrix, the global sparse key matrix including key vectors of multiple target semantic elements, and the global sparse value matrix including value vectors of multiple target semantic elements.

[0213] Optionally, the prediction module 803 is specifically used to: determine the sparse key matrix corresponding to each semantic element among multiple semantic elements, wherein the sparse key matrix corresponding to each semantic element includes the key vector in the global sparse key matrix and the key vectors of the adjacent semantic elements of the corresponding semantic element; determine the sparse value matrix corresponding to each semantic element among multiple semantic elements, wherein the sparse value matrix corresponding to each semantic element includes the value vector in the global sparse value matrix and the value vectors of the adjacent semantic elements of the corresponding semantic element; determine the weighted value vector of each semantic element based on the query vector of each semantic element, the corresponding sparse key matrix and the sparse value matrix; and generate a first prediction result based on the weighted value vector of each semantic element.

[0214] Optionally, the adjacent semantic elements of each semantic element include the preceding N semantic elements and the following N semantic elements of the corresponding semantic element in the first input text, where N is an integer greater than 0.

[0215] Optionally, the adjacent semantic elements of each semantic element include the first N semantic elements of the corresponding semantic element in the first input text.

[0216] Optionally, the prediction module 803 is specifically used to: determine the attention vector of the first semantic element based on the query vector of the first semantic element and the corresponding sparse key matrix, wherein the first semantic element is one of multiple semantic elements; and determine the weighted value vector of the first semantic element based on the attention vector of the first semantic element and the sparse value matrix corresponding to the first semantic element.

[0217] In this embodiment, the text processing device can perform dependency analysis on multiple tokens contained in the first input text to obtain at least one root token, wherein the at least one root token is a token on which other tokens in the first input text depend. In other words, the at least one root token is the core token in the first input text. Based on this, multiple target tokens with high relevance to the at least one root token are selected from the multiple tokens contained in the first input text, and attention sparsity processing is performed using these multiple target tokens. Therefore, this embodiment preserves the semantic dependencies between tokens in the input text during the selection of target tokens. Thus, by using target tokens to construct sparse key matrices and sparse value matrices, the computational load of the model can be reduced while minimizing the impact on the model's prediction accuracy, that is, reducing the impact on the quality of the prediction results generated by the model.

[0218] It should be noted that the module division in the text processing device provided in the above embodiments is illustrative and only represents one logical functional division. In actual implementation, other division methods may also be used. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0219] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a cluster of computer devices (including at least one computer device) or a computer device (which may be a terminal device or a server) or a processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0220] Furthermore, the text processing apparatus and text processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0221] This application also provides a computer device 900. For example... Figure 9 As shown, the computer device 900 includes a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The computer device 900 can be a server or a terminal device. It should be understood that the embodiments of this application do not limit the number of processors and memories in the computer device 900.

[0222] The 902 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 902 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 902 may include a path for transmitting information between various components of the computer device 900 (e.g., memory 906, processor 904, communication interface 908).

[0223] Processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0224] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0225] The memory 906 stores executable program code, and the processor 904 executes the executable program code to implement the functions of the aforementioned text processing device, thereby implementing the text processing method in the aforementioned embodiment. That is, the memory 906 stores instructions for executing the aforementioned text processing method.

[0226] For example, the processor 904 can execute the executable program code to implement the functions of the aforementioned dependency analysis module 801, sparse processing module 802, and prediction module 803, respectively.

[0227] Optionally, memory 906 can also be used to store data generated during the execution of the executable program code by processor 904. For example, memory 906 may include a KV cache 910 for storing sparse processing results obtained by processor 904 during sparse processing.

[0228] The communication interface 908 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computer device 900 and other devices or communication networks.

[0229] This application also provides a computing device cluster. The computing device cluster includes at least one computer device. The computer device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computer device can also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0230] like Figure 10As shown, the computer device cluster includes at least one computer device 900. The memory 906 of one or more computer devices 900 in the computer device cluster may store the same instructions for performing text processing methods.

[0231] In some possible implementations, the memory 906 of one or more computer devices 900 in the computer device cluster may also store partial instructions for executing the text processing method in the foregoing embodiments. In other words, a combination of one or more computer devices 900 can jointly execute instructions for executing the text processing method in the foregoing embodiments.

[0232] It should be noted that the memory 906 in different computer devices 900 within the computer device cluster can store different instructions, each used to execute a portion of the functions of the text processing device. For example, the instructions stored in the memory 906 of different computer devices 900 can implement the functions of one or more modules among the dependency analysis module 801, the sparse processing module 802, and the prediction module 803.

[0233] In some possible implementations, one or more computer devices in a computer cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 11 One possible implementation is shown. For example... Figure 11 As shown, two computer devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computer device. In this possible implementation, as an example, the memory 906 in computer device 900A may store instructions for executing the functions of the dependency analysis module 801 and the sparse processing module 802. Simultaneously, the memory 906 in computer device 900B stores instructions for executing the function of the prediction module 803. Based on this, computer device 900A can obtain the sparse processing result through the dependency analysis module 801 and the sparse processing module 802, and then send the sparse processing result to computer device 900B via the network. Afterwards, computer device 900B can use the sparse processing result through the prediction module 803 to perform subsequent predictions, thereby obtaining the prediction result.

[0234] It should be understood that Figure 11 The functions of computer device 900A shown can also be performed by multiple computer devices 900. Similarly, the functions of computer device 900B can also be performed by multiple computer devices 900.

[0235] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0236] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. In the textual description of the embodiments of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. In this application, "first," "second," and various numerical designations are only for ease of description and are not used to limit the scope of the embodiments of this application. For example, they are used to distinguish different messages, rather than to describe a specific order or sequence.

[0237] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0238] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A text processing method, characterized in that, The method includes: Dependency analysis is performed on multiple semantic elements in the first input text to obtain at least one root semantic element; Multiple target semantic elements are selected from the plurality of semantic elements, wherein the relevance between the plurality of target semantic elements and the at least one root semantic element is greater than the relevance between the other semantic elements among the plurality of semantic elements and the at least one root semantic element; Based on the multiple target semantic elements, attention sparsity processing is performed to obtain the first processing result; Based on the first processing result, a first prediction result is generated.

2. The method according to claim 1, characterized in that, The first input text includes at least one statement, and the dependency analysis of multiple semantic elements in the first input text to obtain at least one root semantic element includes: Dependency analysis is performed on the semantic elements in the at least one statement to obtain the at least one root semantic element.

3. The method according to claim 1 or 2, characterized in that, The step of selecting multiple target semantic elements from the plurality of semantic elements includes: Obtain the key vector of each semantic element and the query vector of each root semantic element from the plurality of semantic elements; Determine the similarity between the key vector of each semantic element and the query vector of each root semantic element; Multiple target similarities are determined from a plurality of identified similarities, wherein the plurality of target similarities are greater than the other similarities among the plurality of similarities; The semantic elements corresponding to the key vectors of the multiple target similarities are determined as the multiple target semantic elements.

4. The method according to any one of claims 1 to 3, characterized in that, The first input text is a prompt text in a reasoning scenario, the first prediction result includes a first prediction statement, and after generating the first prediction result, it further includes: Dependency analysis is performed on the first predicted statement to obtain the new root semantic element; Multiple target semantic elements are determined from multiple candidate semantic elements. The multiple candidate semantic elements include semantic elements in the first prediction statement and the multiple target semantic elements. The correlation between the multiple target semantic elements and the newly added root semantic element is greater than the correlation between other semantic elements in the candidate semantic elements and the newly added root semantic element. Based on the multiple updated target semantic elements, attention sparsity processing is performed to obtain the second processing result; A second prediction result is generated based on the second input text and the second processing result, wherein the second input text includes the first prediction result.

5. The method according to claim 4, characterized in that, The first processing result is stored in a key-value cache; The method further includes: Replace the first processing result in the KV cache with the second processing result.

6. The method according to claim 5, characterized in that, The first processing result includes information about the plurality of target semantic elements and the at least one root semantic element, and the second processing result includes information about the plurality of updated target semantic elements, the at least one root semantic element, and the newly added root semantic element, wherein the number of target semantic elements and the number of updated target semantic elements are equal.

7. The method according to any one of claims 1 to 6, characterized in that, The first processing result obtained by performing attention sparsity processing based on the multiple target semantic elements includes: Based on the multiple target semantic elements, a global sparse key matrix and a global sparse value matrix are determined. The first processing result includes the global sparse key matrix and the global sparse value matrix. The global sparse key matrix includes the key vectors of the multiple target semantic elements, and the global sparse value matrix includes the value vectors of the multiple target semantic elements.

8. The method according to claim 7, characterized in that, The step of generating a first prediction result based on the first processing result includes: Determine the sparse key matrix corresponding to each of the plurality of semantic elements. The sparse key matrix corresponding to each semantic element includes the key vector in the global sparse key matrix and the key vectors of the adjacent semantic elements of the corresponding semantic element. Determine the sparse value matrix corresponding to each of the plurality of semantic elements. The sparse value matrix corresponding to each semantic element includes the value vector in the global sparse value matrix and the value vectors of the adjacent semantic elements of the corresponding semantic element. Based on the query vector, the corresponding sparse key matrix, and the sparse value matrix of each semantic element, determine the weighted value vector of each semantic element; The first prediction result is generated based on the weighted vector of each semantic element.

9. The method according to claim 8, characterized in that, The adjacent semantic elements of each semantic element include the preceding N semantic elements and the following N semantic elements of the corresponding semantic element in the first input text, where N is an integer greater than 0.

10. The method according to claim 8, characterized in that, The adjacent semantic elements of each semantic element include the first N semantic elements of the corresponding semantic element in the first input text, where N is an integer greater than 0.

11. The method according to any one of claims 8 to 10, characterized in that, The determination of the weighted value vector for each semantic element based on the query vector, the corresponding sparse key matrix, and the sparse value matrix of each semantic element includes: Based on the query vector of the first semantic element and the corresponding sparse key matrix, the attention weight vector of the first semantic element is determined, wherein the first semantic element is one of the plurality of semantic elements; Based on the attention weight vector of the first semantic element and the sparse value matrix corresponding to the first semantic element, the weighted value vector of the first semantic element is determined.

12. A text processing device, characterized in that, The text processing device includes: The dependency analysis module is used to perform dependency analysis on multiple semantic elements in the first input text to obtain at least one root semantic element. A sparse processing module is used to select multiple target semantic elements from the multiple semantic elements, and perform attention sparse processing based on the multiple target semantic elements to obtain a first processing result, wherein the correlation between the multiple target semantic elements and the at least one root semantic element is greater than the correlation between other semantic elements among the multiple semantic elements and the at least one root semantic element. The prediction module is used to generate a first prediction result based on the first processing result.

13. The apparatus according to claim 12, characterized in that, The first input text includes at least one statement, and the dependency analysis module is specifically used for: Dependency analysis is performed on the semantic elements in the at least one statement to obtain the at least one root semantic element.

14. The apparatus according to claim 12 or 13, characterized in that, The sparse processing module is specifically used for: Obtain the key vector of each semantic element and the query vector of each root semantic element from the plurality of semantic elements; Determine the similarity between the key vector of each semantic element and the query vector of each root semantic element; Multiple target similarities are determined from a plurality of identified similarities, wherein the plurality of target similarities are greater than the other similarities among the plurality of similarities; The semantic elements corresponding to the key vectors of the multiple target similarities are determined as the multiple target semantic elements.

15. The apparatus according to any one of claims 12 to 14, characterized in that, The first input text is a prompt text in a reasoning scenario, and the first prediction result includes a first prediction statement; The dependency analysis module is also used to perform dependency analysis on the first predicted statement to obtain the new root semantic element. The sparse processing module is further configured to determine multiple updated target semantic elements from multiple candidate semantic elements, and perform attention sparse processing based on the multiple updated target semantic elements to obtain a second processing result. The multiple candidate semantic elements include semantic elements in the first prediction statement and the multiple target semantic elements. The correlation between the multiple updated target semantic elements and the newly added root semantic element is greater than the correlation between other semantic elements in the candidate semantic elements and the newly added root semantic element. The prediction module is further configured to generate a second prediction result based on the second input text and the second processing result, wherein the second input text includes the first prediction result.

16. The apparatus according to claim 15, characterized in that, The first processing result is stored in a key-value cache; the device is also used for: Replace the first processing result in the KV cache with the second processing result.

17. The apparatus according to claim 16, characterized in that, The first processing result includes information about the plurality of target semantic elements and the at least one root semantic element, and the second processing result includes information about the plurality of updated target semantic elements, the at least one root semantic element, and the newly added root semantic element, wherein the number of target semantic elements and the number of updated target semantic elements are equal.

18. The apparatus according to any one of claims 12 to 17, characterized in that, The sparse processing module is specifically used for: Based on the multiple target semantic elements, a global sparse key matrix and a global sparse value matrix are determined. The first processing result includes the global sparse key matrix and the global sparse value matrix. The global sparse key matrix includes the key vectors of the multiple target semantic elements, and the global sparse value matrix includes the value vectors of the multiple target semantic elements.

19. The apparatus according to claim 18, characterized in that, The prediction module is specifically used for: Determine the sparse key matrix corresponding to each of the plurality of semantic elements. The sparse key matrix corresponding to each semantic element includes the key vector in the global sparse key matrix and the key vectors of the adjacent semantic elements of the corresponding semantic element. Determine the sparse value matrix corresponding to each of the plurality of semantic elements. The sparse value matrix corresponding to each semantic element includes the value vector in the global sparse value matrix and the value vectors of the adjacent semantic elements of the corresponding semantic element. Based on the query vector, the corresponding sparse key matrix, and the sparse value matrix of each semantic element, determine the weighted value vector of each semantic element; The first prediction result is generated based on the weighted vector of each semantic element.

20. The apparatus according to claim 19, characterized in that, The adjacent semantic elements of each semantic element include the preceding N semantic elements and the following N semantic elements of the corresponding semantic element in the first input text, where N is an integer greater than 0.

21. The apparatus according to claim 19, characterized in that, The adjacent semantic elements of each semantic element include the first N semantic elements of the corresponding semantic element in the first input text, where N is an integer greater than 0.

22. The apparatus according to any one of claims 19 to 21, characterized in that, The prediction module is specifically used for: Based on the query vector of the first semantic element and the corresponding sparse key matrix, the attention vector of the first semantic element is determined, wherein the first semantic element is one of the plurality of semantic elements; Based on the attention vector of the first semantic element and the sparse value matrix corresponding to the first semantic element, the weighted value vector of the first semantic element is determined.

23. A computer device, characterized in that, The computer device includes a processor for executing a computer program stored in a memory to implement the text processing method according to any one of claims 1 to 11.

24. The computer device according to claim 23, characterized in that, The computer device further includes the memory, which includes a key-value (KV) cache for storing sparse processing results.

25. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer device, cause the computer device to perform the text processing method according to any one of claims 1 to 11.

26. A computer program product containing instructions, characterized in that, When the instructions are executed by a computer device, the computer device performs the text processing method as described in any one of claims 1 to 11.

Images