Task execution method, model training method and related products

Abstract

The embodiment of the invention discloses a task execution method, a model training method and a related product. The method comprises the steps that elements in an input sequence are grouped, a first element group and a second element group are obtained, and the correlation between the elements in the first element group and a natural language processing task is higher than the correlation between the elements in the second element group; performing first attention processing on elements in the first element group to obtain a first processing result; performing second attention processing on the elements in the second element group to obtain a second processing result; and executing the natural language processing task based on the first processing result and the second processing result to obtain an output sequence. According to the embodiment of the invention, the processing precision of the natural language processing task can be ensured while the execution efficiency of the natural language processing task is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and specifically to a task execution method, a model training method, and related products. Background Art

[0002] With the rapid development of artificial intelligence (AI), the field of natural language processing (NLP) has undergone tremendous changes. Today, many NLP tasks, such as text classification, sentiment analysis, machine translation, and question-answering systems, are primarily performed by large language models (LLMs). These models are typically built on the Transformer architecture, which is highly favored for its powerful parallel processing capabilities and ability to capture long-range dependencies. At the heart of the Transformer architecture lies the self-attention mechanism, which enables the model to consider the context of the entire input sequence when processing each element in the sequence.

[0003] Currently, the two most commonly used attention mechanisms for NLP tasks are Multi-Head Attention (MHA) and Grouped-Query Attention (GQA). MHA has more semantic subspaces and focuses more on global information, extracting rich semantic information and achieving high processing accuracy. However, its excessive focus on global information can slow inference speed, resulting in relatively slow NLP task execution efficiency. GQA has fewer heads and faster inference speed, but its smaller semantic subspaces lack the richness of semantic information, resulting in relatively low processing accuracy for NLP tasks.

[0004] Therefore, how to improve the execution efficiency of natural language processing tasks while ensuring the processing accuracy of natural language processing tasks is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The embodiments of the present application provide a task execution method, a model training method, and related products. By grouping the elements in the input sequence and dynamically allocating attention to the elements, the execution efficiency of the natural language processing task is improved while ensuring the processing accuracy of the natural language processing task.

[0006] In a first aspect, an embodiment of the present application provides a task execution method, including:

[0007] Grouping elements in the input sequence to obtain a first element group and a second element group, wherein the elements in the first element group are more relevant to the natural language processing task than the elements in the second element group;

[0008] performing a first attention process on the elements in the first element group to obtain a first processing result;

[0009] performing a second attention process on the elements in the second element group to obtain a second processing result;

[0010] The natural language processing task is performed based on the first processing result and the second processing result to obtain an output sequence.

[0011] Among them, the number of semantic subspaces processed by the first attention is greater than the number of semantic subspaces processed by the second attention. Accordingly, the semantic information extraction ability of the first attention processing is higher than that of the second attention processing; the reasoning speed of the second attention processing is higher than that of the first attention processing.

[0012] In a second aspect, an embodiment of the present application provides a model training method, wherein the model includes a first attention module and a second attention module; the method includes:

[0013] Obtaining a sample sequence corresponding to a natural language processing task and a label sequence of the sample sequence;

[0014] Grouping the sample elements in the sample sequence to obtain a first sample element group and a second sample element group, wherein the elements in the first sample element group are more highly correlated with the natural language processing task than the elements in the second sample element group;

[0015] Inputting the first sample element group into the first attention module for first attention processing to obtain a first sample processing result;

[0016] Inputting the second sample element group into the second attention module for second attention processing to obtain a second sample processing result;

[0017] Performing the natural language processing task based on the first sample processing result and the second sample processing result to obtain a prediction sequence;

[0018] Based on the prediction sequence and the label sequence, the first attention module and the second attention module are trained to obtain the model.

[0019] In a third aspect, an embodiment of the present application provides a task execution device, the device comprising a grouping unit and a processing unit;

[0020] The grouping unit is used to obtain an input sequence;

[0021] The processing unit is used to group the elements in the input sequence to obtain a first element group and a second element group, wherein the elements in the first element group are more relevant to the natural language processing task than the elements in the second element group; perform a first attention processing on the elements in the first element group to obtain a first processing result; perform a second attention processing on the elements in the second element group to obtain a second processing result; and execute the natural language processing task based on the first processing result and the second processing result to obtain an output sequence.

[0022] In a fourth aspect, an embodiment of the present application provides a model training device, the device comprising: an acquisition unit and a processing unit;

[0023] The acquisition unit is configured to acquire a sample sequence corresponding to a natural language processing task and a label sequence of the sample sequence;

[0024] The processing unit is used to group the sample elements in the sample sequence to obtain a first sample element group and a second sample element group, wherein the elements in the first sample element group are more correlated with the natural language processing task than the elements in the second sample element group; input the first sample element group into the first attention module for first attention processing to obtain a first sample processing result; input the second sample element group into the second attention module for second attention processing to obtain a second sample processing result; perform the natural language processing task based on the first sample processing result and the second sample processing result to obtain a prediction sequence; and train the first attention module and the second attention module based on the prediction sequence and the label sequence to obtain the model.

[0025] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the processor being connected to the memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the electronic device performs the method described in the first aspect or the second aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0027] In a seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the method described in the first aspect or the second aspect.

[0028] The implementation of the embodiments of the present application has the following beneficial effects:

[0029] It can be seen that in the embodiment of the present application, in the process of performing the natural language processing task, the input sequence is first grouped, and the elements with higher relevance to the natural language processing task, that is, the elements that are relatively important to the natural language processing task, are divided into the first element group, and the elements with lower relevance to the natural language processing task, that is, the elements that are relatively unimportant to the natural language processing task, are divided into the second element group. In addition, the first attention processing with a strong semantic information extraction capability is performed on the elements in the first element group. By performing semantic information extraction on the elements in the first element group through the first attention, richer semantic information can be extracted from the elements in the first element group, ensuring the integrity and richness of the semantic information of the entire input sequence, thereby ensuring the processing accuracy of the natural language processing task. The second attention processing with a relatively fast reasoning speed is performed on the elements in the second element group, which can quickly extract semantic information from the elements in the second element group, thereby improving the execution efficiency of the natural language processing task. Therefore, for the entire input sequence, by grouping the elements in the input sequence and dynamically allocating attention, it is achieved that while ensuring the processing accuracy of the natural language processing task, the execution efficiency of the natural language processing task can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 Schematic diagram of the processing mechanism of MHA, GQA, and MQA provided in the embodiments of the present application;

[0032] Figure 2 A schematic diagram of an attention mechanism provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of a model provided in an embodiment of the present application;

[0034] Figure 4 A schematic diagram of another model provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of a first attention module and a second attention module provided in an embodiment of the present application;

[0036] Figure 6A schematic diagram of an intelligent customer service scenario provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of an agent marketing scenario provided in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of a labeling task provided in an embodiment of the present application;

[0039] Figure 9 A flowchart of a task execution method provided in an embodiment of the present application;

[0040] Figure 10 A schematic diagram of splicing the third processing result and the fourth processing result provided in an embodiment of the present application;

[0041] Figure 11 A schematic diagram of superimposing a third processing result and a fourth processing result provided in an embodiment of the present application;

[0042] Figure 12 A flowchart of a model training method provided in an embodiment of the present application;

[0043] Figure 13 A schematic diagram of a task execution device provided in an embodiment of the present application;

[0044] Figure 14 A schematic diagram of a model training device provided in an embodiment of the present application;

[0045] Figure 15 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0047] The terms "first," "second," "third," and "fourth," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, rather than to describe a specific order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0048] References herein to "embodiments" mean that a particular feature, result, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] In order to facilitate understanding of the technical solution of this application, the relevant technical terms involved in this application are first introduced.

[0050] Token: A word can be understood as the smallest meaningful unit in natural language processing, or the smallest semantic unit in the input sequence of natural language processing. These units can be words, numbers, symbols, punctuation marks, etc.

[0051] Elements: The elements mentioned in this application can be understood as the aforementioned tokens, that is, the smallest semantic unit in the input sequence. Therefore, the tokens and elements mentioned in this application are essentially similar and can be distinguished.

[0052] Tokenization: The process of dividing the input sequence into tokens.

[0053] Tokenizer Encoder: The process of segmenting the input sequence and encoding the tokens obtained by segmentation. Token encoding refers to the process of encoding the token into a token ID.

[0054] Tokenizer Decoder: The process of converting the token ID sequence generated by the model back to the corresponding token, which is the inverse process of the Tokenizer Encoder.

[0055] Query: The query vector for each element in the input sequence when calculating the attention weight, which can be abbreviated as Q.

[0056] Key: The key vector of each element in the input sequence, which can be abbreviated as K.

[0057] Value: Value corresponds one-to-one with Key, storing the actual information of each element in the input sequence. It is the implicit vector of each element in the input sequence, also known as the value vector of the element, and can be abbreviated as V.

[0058] Multi-Head Attention (MHA): MHA includes multiple attention heads, each of which corresponds to its own set of Key matrix, Query matrix and Value matrix. For the convenience of description, the Key matrix, Query matrix and Value matrix mentioned in this application can be referred to as K matrix, Q matrix and V matrix or simply referred to as KQV matrix.

[0059] Grouped-Query Attention (GQA): GQA divides multiple attention heads into G groups. Each attention head in each group has its own Q matrix, but all attention heads in each group share a key matrix and a value matrix. GQA-G refers to grouped query attention with G groups of attention heads.

[0060] Multi-Query Attention (MQA): MQA groups all attention heads into a single group, so that all heads share a common key and value matrix. MQA is a special case of GQA with G=1, so GQA and MQA are the same type of attention mechanism. This application uses GQA as an example.

[0061] Root Mean Square Layer Normalization (RMSNorm): A layer normalization technique used to normalize activation values ​​in deep neural networks. Its purpose is to adjust the norm (i.e., modulus) of the input vector without changing its direction, thereby accelerating training and improving network stability.

[0062] Linear layer: The Linear layer in this application can be understood as the fully connected layer in the model. It is one of the most basic neural network layers. The function of the Linear layer is to map the input to the output through a linear transformation.

[0063] Rotary Position Embedding (RoPE): A position encoding method that can integrate relative position information dependencies into the attention mechanism and improve the performance of the Transformer architecture. It mainly adds the position information of the elements in the input sequence to the elements in the input sequence.

[0064] Mask: A masking mechanism that is mainly used to mask the attention weight matrix between the query vector and the key vector of an element to avoid seeing the token that comes after the current token when predicting the current token.

[0065] For example, the input sequence includes the elements A, B, and C. After attention processing, the attention weight matrix between the elements is as follows:

[0066]

[0067] Among them, the first row of the above attention weights represents the attention weights between A and A, B, and C, the second row represents the attention weights between B and A, B, and C, and the third row represents the attention weights between C and A, B, and C. However, when the model predicts each token one-by-one, it can only use the output token that is located before the token to predict the token. Therefore, when predicting A, B and C cannot be seen. A only has weight with itself, so the attention weights between A and B and C need to be masked. For example, the attention weights between A and B and C can be replaced with zero or negative infinity. When predicting B, C cannot be seen, and only A and B can be seen, so the attention weights between B and C need to be masked.

[0068] In general, Masking is to mask out the elements above the diagonal of the attention weight matrix between elements. For example, you can use zero or negative infinity to replace the elements above the diagonal of the attention weight matrix. This application mainly uses zero replacement as an example. Therefore, after masking the above attention weights, the obtained attention weights are as follows:

[0069]

[0070] First Attention: This application's first attention is an attention mechanism with strong semantic information extraction capabilities. It has multiple semantic subspaces and can capture the feature representations of tokens in multiple different semantic subspaces, thus extracting rich semantic information. Compared with the second attention, the first attention has a higher semantic information extraction capability.

[0071] Second Attention: Second Attention is an attention mechanism with faster reasoning speed, which can quickly extract semantic information and complete the decoding process. Compared with the first attention, the second attention has a relatively small semantic subspace, but the reasoning speed of the second attention is higher than that of the first attention.

[0072] In general, the number of semantic subspaces of the first attention is greater than that of the second attention, so that from the perspective of each attention, the number of semantic subspaces of the first attention is greater than the threshold, while the number of semantic subspaces of the second attention is less than the threshold. The threshold can be determined based on the ability to extract semantic information. The attention greater than the threshold has a stronger ability to extract semantic information, but a relatively slow reasoning speed. The attention less than the threshold has a weaker ability to extract semantic information, but a relatively fast reasoning speed. For ease of description, this application mainly takes the first attention as MHA as an example, and the second attention as GQA as an example for explanation, but does not limit the types of the first attention and the second attention. As long as it meets the above-mentioned characteristics of the first attention and any other attention that meets the characteristics of the second attention, it is within the scope of protection of this application.

[0073] The following describes the shortcomings of MHA and GQA with reference to the accompanying drawings and the processing mechanisms of MHA and GQA.

[0074] like Figure 1 As shown, multiple attention heads are designed for MHA, each with a corresponding set of KQV matrices. Therefore, each attention head needs to store the corresponding KQV matrix, which results in a large number of matrix parameters and high memory and access overhead. Furthermore, each attention head must independently calculate the query vector, key vector, and value vector for each element, resulting in high computational overhead. This computational complexity increases linearly with the number of attention heads, increasing GPU memory usage during training and inference. This demand is particularly pronounced when reasoning about long sequences. Furthermore, to capture rich semantic information about elements in different semantic subspaces and improve inference accuracy, more attention heads are typically designed in practical applications. However, the number of these attention heads is fixed and cannot be dynamically adjusted during inference. Consequently, in complex tasks, some attention heads may not capture sufficient semantic information, while others may become redundant. Furthermore, as the length of the input sequence increases, the attention process performed by multiple attention heads significantly slows inference speed, resulting in relatively low efficiency in natural language processing tasks.

[0075] like Figure 1 As shown in Figure 2, in GQA, a group of attention heads share the same K and V matrices. While this sharing reduces the number of matrix parameters required, lowers storage and access overhead, and improves inference speed, since multiple attention heads share the same K and V matrices, GQA reduces the number of learned semantic subspaces, risks information loss, and reduces the richness of semantic information. For complex tasks, a single K and V matrix may not capture sufficient semantic details, reducing the accuracy of natural language processing tasks.

[0076] like Figure 1 As shown in Figure 1, MQA is a special GQA-G with G=1. Compared with GQA, it has fewer semantic subspaces and a greater risk of information loss, which further reduces the processing accuracy of natural language processing tasks.

[0077] Therefore, how to improve the execution efficiency of natural language processing tasks while ensuring the processing accuracy of natural language processing tasks is a technical problem that needs to be solved urgently.

[0078] See Figure 2 , Figure 2 This is a schematic diagram of an attention mechanism provided by this application.

[0079] like Figure 2 As shown, before performing attention processing on the elements in the input sequence, the present application first groups the elements in the input sequence to obtain a first element group and a second element group, wherein the elements in the first element group have a higher correlation with the natural language processing task than the elements in the second element group. Then, a first attention processing is performed on the elements in the first element group through MHA to obtain a first processing result; and a second attention processing is performed on the elements in the second element group through GQA to obtain a second processing result. Finally, the natural language processing task is performed based on the first processing result and the second processing result to obtain an output sequence, that is, an output result corresponding to the natural language processing task.

[0080] It can be seen that in the embodiment of the present application, for the elements in the input sequence that are relatively highly correlated with the natural language processing task, that is, the important elements, attention processing is performed through MHA, so that rich semantic information can be extracted for the important elements, ensuring the richness of the semantic information without reducing the processing accuracy of the natural language processing task; for the elements in the input sequence that are relatively less correlated with the natural language processing task, that is, the secondary elements, attention processing is performed through GQA, which can improve the speed and efficiency of extracting semantic information and improve the execution efficiency of the natural language task. For the entire input sequence, it is achieved that the execution efficiency of the natural language processing task is improved while ensuring the processing accuracy of the natural language processing task.

[0081] First, let's explain that the model used in this application is a Large Language Model (LLM), which performs various natural language processing tasks in a generative manner. Furthermore, the LLM is based on the Transform architecture. The following describes the process of performing natural language processing tasks in this application, combining the structure of the model.

[0082] See Figure 3 , Figure 3 This is a schematic diagram of a model provided in an embodiment of the present application. Figure 3 As shown, the model of the present application includes: a word segmentation encoding (Tokenizer Encoder) layer, a word embedding (Embedding) layer, a Transformer Layer, a normalization layer (ie, RMS Norm), a linear mapping layer (ie, Linear), an activation layer (ie, SoftMax), and a word segmentation decoding (Tokenizer Decoder) layer.

[0083] It should be noted that the linear mapping layers mentioned in this application are all illustrated by the Linear layer, the normalization layers are all illustrated by the RMS Norm layer, and the activation layers are all illustrated by the SoftMax. The Linear layer, RMS Norm layer and SoftMax mentioned subsequently can all be understood as linear mapping layers, normalization layers and activation layers, and there is no need to distinguish them. However, this application does not limit the types of normalization layers, linear mapping layers and activation layers.

[0084] like Figure 3 As shown, the input sequence is first input into the Tokenizer Encoder layer, the input sequence is first segmented to obtain the elements in the input sequence, and then each element in the input sequence is encoded, and each element in the input sequence is encoded into a corresponding Token ID. Then, the Token ID of each element of the input sequence is input into the Embedding layer for word embedding to obtain the first feature vector corresponding to each element. Then, the first feature vector corresponding to each element is input into the Transformer Layer to extract features from the elements in the input sequence to obtain the superimposed feature extraction results. Among them, in the process of extracting features from the elements in the input sequence, the Transformer Layer will use the following Figure 2 The attention mechanism shown in FIG5 performs attention processing on the elements in the input sequence to obtain the superimposed feature extraction results.

[0085] Finally, the superimposed feature extraction results are input into the RMS Norm layer for normalization to obtain the normalized results. The normalized results are input into the SoftMax layer for activation to obtain the activation results, and the activation layer results are input into the Tokenizer Decoder layer for decoding to obtain the output sequence.

[0086] See Figure 4 , Figure 4 A schematic diagram of another model provided in an embodiment of the present application. Figure 4 The model structure shown is similar to Figure 3 The model structure shown mainly describes the model structure of the Transformer Layer in detail.

[0087] For example, Figure 4 As shown in , the Transformer Layer includes: Dynamic Multi-query Attention Layer and Feedforward Neural Network (FFN). Figure 4 As shown, DMA includes: a heuristic query allocation mechanism (HQAM) module, a scoring module, a scoring normalization module, multiple RMS Norm layers, multiple Linear layers, a first attention module, and a second attention module. Optionally, this application mainly uses a lightweight FFN as an example for illustration, but does not limit the scoring module. Any model network that can implement the scoring function can be used as the scoring module of this application.

[0088] The following combination Figure 4 The model structure shown in FIG. 4 describes in detail the process by which the Transformer Layer extracts features from elements in the input sequence and obtains the superimposed feature extraction results.

[0089] like Figure 4 As shown in the figure, the input sequence is first input to the Tokenizer Encoder layer, which first segmented the input sequence to obtain the elements in the input sequence. Then, each element in the input sequence is encoded into a corresponding token ID. The token ID of each element in the input sequence is then input to the Embedding layer for word embedding, obtaining the first feature vector corresponding to each element.

[0090] Then, for the elements in the input sequence, the HQAM module is first used to divide the elements in the input sequence into a third element group and a fourth element group, wherein the elements in the third element group have a higher relevance to the natural language processing task than the elements in the fourth element group, i.e., the elements in the fourth element group are minor elements. Then, for each element in the third element group, the first eigenvector of each element is input into the scoring module for relevance scoring, and the predicted score is normalized by the score normalization module to obtain a score for each element, wherein the score of each element is used to indicate the relevance of the element to the natural language processing task, and the larger the score, the higher the relevance. Then, a decision is made based on the score of each element, for example, elements with a score greater than a scoring threshold are divided into a first element group, and elements with a score less than or equal to the scoring threshold are divided into a fifth element group, wherein the elements in the first element group are important elements and the elements in the fifth element group are minor elements; finally, the fourth element group and the fifth element group are combined into a second element group, i.e., the elements in the input sequence are divided into important elements and minor elements.

[0091] Then, for each element in the input sequence, the first eigenvector of each element is normalized and linearly mapped through the RMS Norm layer and the Linear layer respectively to obtain the fourth eigenvector corresponding to each element.

[0092] Then, the fourth eigenvector corresponding to each element in the first element group is input into the first attention module for first attention processing to obtain the first attention processing result. The fourth eigenvector corresponding to each element in the second element group is input into the second attention module for second attention processing to obtain the second attention processing result.

[0093] Then, the first processing result is input into the Linear layer for linear transformation to obtain a third processing result; the second processing result is input into the Linear layer for linear transformation to obtain a fourth processing result. And, the third processing result and the fourth processing result are fused to obtain a fifth processing result.

[0094] Furthermore, the fifth processing result is superimposed with the first feature vector of each element in the input sequence to obtain a superimposed processing result. The superimposed processing result is input into the FFN for feature extraction to obtain a feature extraction result, and the feature extraction result is superimposed with the superimposed processing result to obtain a superimposed feature extraction result.

[0095] Finally, the superimposed feature extraction results are input into the RMS Norm layer for normalization to obtain the normalized results. The normalized results are input into the SoftMax layer for activation to obtain the activation results, and the activation layer results are input into the Tokenizer Decoder layer for decoding to obtain the output sequence.

[0096] It can be seen that in the embodiment of the present application, when the model is used to perform natural language processing tasks, the model divides the elements in the input sequence into important elements and minor elements through the HQAM module and the scoring module, and for the important elements, the first attention module (MHA) performs the first attention processing, so that rich semantic information can be extracted for the important elements, ensuring the richness of the semantic information and not reducing the processing accuracy of the natural language processing task; for the minor elements, the second attention module (GQA) performs the second attention processing, which can improve the extraction speed and efficiency of the semantic information, thereby improving the execution efficiency of the natural language task. Therefore, for the entire input sequence, it is achieved while improving the execution efficiency of the natural language processing task and ensuring the processing accuracy of the natural language processing task.

[0097] See Figure 5 , Figure 5 A schematic diagram of a first attention module and a second attention module provided in an embodiment of the present application.

[0098] For example, Figure 5 As shown, for the first attention module, the first attention module includes multiple attention heads, Figure 5Only the process of performing the first attention processing on the elements in the first element group through one attention head is shown. Therefore, the fourth eigenvector of each element in the first element group can be input into each attention head of the first attention module, and the fourth eigenvector of each element is respectively operated with the Q matrix, K matrix, and V matrix corresponding to the attention head to obtain the query vector, key vector, and value vector of each element in the first element group under each attention head. Then, for each attention head, the query vector and key vector of each element in the first element group under each attention head are respectively rope-encoded to obtain the fifth eigenvector and sixth eigenvector of each element under each attention head. The similarity of the fifth eigenvector and the sixth eigenvector of each element under each attention head is calculated to obtain the first attention weight matrix of the first element group under each attention head, and the first attention weight matrix is ​​masked to obtain the second attention weight matrix of the first element group under each attention head. Based on the second attention weight matrix of the first element group under each attention head, the value vector of each element of the first element group under each attention head is weighted to obtain the attention processing result of the first element group under each attention head. Finally, the multiple attention processing results of the first element group under the multiple attention heads of the first attention module are fused to obtain the above-mentioned first processing result.

[0099] For example, Figure 5 As shown, for the second attention module, the second attention module includes multiple groups of attention heads. Figure 5 Only one set of attention heads is shown, where each set of attention heads includes multiple attention heads. The following uses one set of attention heads as an example to illustrate the process of performing the second attention processing on the second element group.

[0100] Exemplarily, the fourth eigenvector of each element in the second element group is input into the group of attention heads, and is respectively operated with the Q matrix of each attention head in the group of attention heads, as well as with the K matrix and V matrix of the group of attention heads, to obtain the query vector of each element under each attention head in the group of attention heads, as well as the key vector and value vector of each element under the group of attention heads. Then, for each attention head in the group of attention heads, the query vector of each element under each attention head is rope-encoded to obtain the seventh eigenvector of each element under each attention head, and the key vector of each element under the group of attention heads is rope-encoded to obtain the eighth eigenvector of each element under the group of attention heads. Finally, the seventh eigenvector of each element under each attention head is similarly calculated with the eighth eigenvector of each element under the group of attention heads to obtain the third attention weight matrix of the second element group under each attention head in the group of attention heads, and the third attention weight matrix of the second element group under each attention head is masked to obtain the fourth attention weight matrix of the second element group under each attention head. Finally, based on the fourth attention weight matrix of each attention head in the second element group, the value vector of each element in the second element group under the group of attention heads is weighted to obtain the attention processing result corresponding to each attention head in the group of attention heads. Finally, the attention processing results corresponding to each attention head in the group of attention heads are concatenated to obtain the attention processing result corresponding to the group of attention heads, and the multiple attention processing results corresponding to multiple groups of attention heads are fused to obtain the second processing result.

[0101] The following describes the application scenarios corresponding to each natural language processing task involved in this application in conjunction with the accompanying drawings.

[0102] 1. Dialogue tasks:

[0103] For example, Figure 6As shown, when the natural language processing task is a dialogue task, the corresponding application scenario can be an intelligent customer service scenario. Accordingly, a user can access the dialogue interface of the intelligent customer service system through a user device, and then the user can enter a corresponding question (Question) in the dialogue interface. The dialogue device of the intelligent customer service system then outputs an answer (Answer) corresponding to the question for the user. For the current round of dialogue, the dialogue device can obtain the question Question m:xxx entered by the user in the current round of dialogue. The dialogue device then uses the question entered by the user in the current round of dialogue and the dialogue content of the first round of dialogue as input data to obtain an input sequence corresponding to the current round. The dialogue content of the first round of dialogue includes the question asked by the user in the first round of dialogue and the answer output by the dialogue device to the question. The first round of dialogue includes part or all of all previous rounds of dialogue. For ease of description, this application mainly uses the example of the first round of dialogue including all previous rounds of dialogue as an example for illustration.

[0104] Therefore, if Figure 6 As shown, the dialogue device can take the question of the first historical round of dialogue (Question1: xxx), the answer of the first historical round of dialogue (Answer1: xxx), the question of the second historical round of dialogue (Question2: xxx), the answer of the second historical round of dialogue (Answer2: xxx), ..., and the question of the current round of dialogue (Question m: xxx) as an input sequence. Then, the dialogue device performs the natural language processing task of this application on the input sequence to obtain an output sequence, and returns the output sequence as the answer of the current round of dialogue to the user device and displays the output sequence on the dialogue interface, thereby providing the user with the answer corresponding to the question of the current round of dialogue.

[0105] For example, Figure 7 As shown, if the language processing task is a conversational task, the corresponding application scenario can also be an agent marketing scenario. Accordingly, the agent marketing system's outbound call device will conduct intelligent outbound calls to market products to users. During the marketing process, the outbound call device first establishes a call connection with the user's user device. Then, for the current conversation turn, the outbound call device outputs the corresponding output sequence based on the input sequence and places an outbound call to the user.

[0106] Specifically, in an agent marketing scenario, for the current round of conversation, the outbound call device will obtain an input sequence corresponding to the current round of conversation, where the input sequence is the conversation content of the first round of conversation. The first round of conversation includes part or all of all historical rounds of conversation. In the agent marketing scenario, after the outbound call device establishes a call connection with the user's device, it will first introduce the opening remarks and the product, and then start the conversation between the user and the outbound call device. For example, after the outbound call device is introduced, the user starts asking questions, and only then does the first round of conversation begin. Therefore, in the agent marketing scenario, in addition to the conversation between the user and the outbound call device, the historical round of conversation can also include the opening remarks and the product introduction.

[0107] Therefore, if Figure 7 As shown, the outbound call device can take the opening remarks: xxx, product introduction: xxx, the question of the first round of dialogue (Question1: xxx), the answer to the first round of dialogue (Answer1: xxx), the question of the second round of dialogue (Question2: xxx), the answer to the second round of dialogue (Answer2: xxx), ..., the question of the current round of dialogue (Question m: xxx) as an input sequence. The outbound call device then performs the natural language processing task of this application on this input sequence to obtain an output sequence, and calls the output sequence out to the user to answer the user's question in the current round of dialogue.

[0108] It should be noted that in conversation tasks, as the number of conversation turns increases, the input sequence becomes very long, and the length of the input sequence is likely to exceed the maximum length allowed by the model. When the length of the input sequence exceeds the maximum length allowed by the model, because some of the intermediate conversation content is relatively weakly related to the current conversation turn, this application will use a sliding window method to select the conversation content of the first k conversation turns, the conversation content of the last y conversation turns, and the question of the current conversation turn as the input sequence, that is, filtering out the conversation content of the intermediate turns, thereby obtaining an input sequence of the required length.

[0109] It can be seen that in the dialogue task, the input sequence includes the dialogue content of the first round of dialogue, the output sequence includes the reply content of the current round of dialogue, that is, the answer to the question of the current round of dialogue, and in the dialogue task, the attribute features of the element include: the position of the element in the input sequence.

[0110] 2. Labeling task:

[0111] Exemplarily, in the case where the natural language processing task is a labeling task, the natural language processing task is mainly to give a corresponding plurality of label instances, and then instruct the task execution device to use the given plurality of label instances to label the text that needs to be labeled with corresponding labels. The text can be any type of text, such as call text, text downloaded from a website, or text edited online, etc. This application mainly uses the text as a call text, and takes the call text under agent marketing as an example for illustration. Labeling the call text can facilitate the classification and archiving of all call texts of agent marketing in the later stage, thereby improving the convenience of agent marketing management.

[0112] like Figure 8 As shown, in the case where the text is a call text, the input sequence corresponding to the natural language processing task includes the following:

[0113] #Role Definition

[0114] You are a call monitoring expert who specializes in analyzing and monitoring call texts between agents and customers.

[0115] #Tag Definition

[0116] You need to master and understand the following tags:

[0117] Lable_1 / / Explanation and description of Lable_1 / /

[0118] Lable_2 / / Explanation and description of Lable_2 / /

[0119] …

[0120] Lable_m / / Explanation and description of Lable_m / /

[0121] #Task definition

[0122] Your task is to learn analysis <data>< / data> Output the tag to which the call text belongs.

[0123] #Quest Requirements

[0124] The output tag must be the tag given in the #tag definition; the output format must be in xx format.

[0125] # <data>

[0126] Call text that needs to be tagged

[0127] < / data> .

[0128] Exemplarily, in the tagging task, the user can send an input sequence to the tag recognition device through the user device. Then, the tag recognition device identifies the tag to which the call text to be tagged belongs through the execution logic of the natural language processing task of the present application, and outputs the tag to which the call text belongs in a specified format to obtain an output sequence.

[0129] For example, if the specified output format is:

[0130] Output_Lable:

[0131] [<Name of Lable, Explanation of Lable>, ……, <Name of Lable, Explanation of Lable>].

[0132] When the tags to which the call text to be tagged belongs include Lable_1 and Lable_2, then as Figure 8 shown, the output sequence corresponding to this natural language processing task is:

[0133] Output_Label:

[0134] [<Lable_1, Explanation and description of Lable_1>, <Lable_2, Explanation and description of Lable_2>].

[0135] It can be seen that in the tagging task scenario, the input sequence includes the call text to be tagged, as well as a given number of tags, and the explanation and description of each tag, and some prompt texts with prompt functions, such as the text corresponding to the role definition, and the format requirements for the output sequence. And the output sequence includes the tags to which the call text belongs, and this output sequence can be output according to the format requirements in the input sequence. Correspondingly, in the tagging task scenario, the attribute characteristics of the element include: the position of the element in the input sequence.

[0136] It should be noted that the above-mentioned dialogue device, outbound call device, and tag recognition device can all be the task execution devices of the present application, and they can be the same in actual applications.

[0137] In addition, in addition to the above two natural language processing tasks, the natural language processing tasks of the present application can also include at least one of the following: entity recognition task, emotion recognition task, place name recognition task, and text translation task.

[0138] Exemplarily, when the natural language processing task is an entity recognition task, the input sequence includes text that needs to be recognized as an entity, and the output sequence is the entity in the text. Of course, the input sequence can also include other prompt texts with prompt functions, such as text for defining the role of the model, text for formatting the output sequence, etc. This application does not limit the content of the output sequence, and it can be set and combined according to actual needs. When the natural language processing task is an emotion recognition task, the input sequence includes text that needs to be recognized as an emotion, and the output sequence is the emotion of the text; when the natural language processing task is the place name recognition task, the input sequence includes text that needs to be recognized as a place name, and the output sequence is the place name in the text; when the natural language processing task is the text translation task, the input sequence includes text that needs to be translated, and the output sequence includes the translated text.

[0139] It should be noted that this application includes but is not limited to the above-mentioned scenarios. As long as the attention processing mechanism of this application is used in the process of performing natural language processing tasks, it is within the scope of protection of this application.

[0140] The following combination Figure 3-Figure 5 The structure of the model shown describes in detail the execution process of the natural language processing task of this application.

[0141] See Figure 9 , Figure 9 This is a flowchart of a task execution method provided in an embodiment of the present application. The method is applied to a task execution device. The task execution device can execute the task by calling the model of the present application. The method includes but is not limited to the following steps:

[0142] 901: Group elements in an input sequence to obtain a first element group and a second element group, wherein the elements in the first element group have a higher relevance to the natural language processing task than the elements in the second element group.

[0143] Among them, the relevance of an element to a natural language processing task can also be understood as the importance of the element in the natural language processing task, and the higher the relevance of an element to a natural language processing task, the higher the importance of the element in the natural language processing task. Therefore, the relevance and importance in this application are similar, and there is no need to distinguish between relevance and importance. In addition, for ease of description, the elements in the first element group can also be referred to as important elements in the input sequence, and the elements in the second element group can be referred to as secondary elements in the input sequence.

[0144] The natural language processing task includes, but is not limited to, one of: a conversation task, a tagging task, an entity recognition task, a sentiment recognition task, a place name recognition task, and a text translation task. It is understood that, as described above in the application scenarios of this application, the input sequence has different representations for different natural language processing tasks.

[0145] First, before grouping the elements in the input sequence, you need to segment the input sequence to obtain the individual elements in the input sequence, that is, the individual tokens. Then, you need to encode each element to obtain its TokenID. For example, you can input the input sequence into the Tokenizer Encoder layer of the model to segment and encode the input sequence to obtain the Token ID of each element. Then, you can embed the Token ID of each element into a word to obtain the first feature vector of each element. For example, you can input the Token ID of each element into the Embedding layer to embed the word to obtain the first feature vector of each element.

[0146] Optionally, in one embodiment of the present application, attribute features of each element in the input sequence are first obtained, wherein the attribute features of the element are features related to the attributes of the element. Optionally, the attribute features of the element include, but are not limited to, one or more of: the element's part of speech, the element's semantics, the element's position in the input sequence, and the element's word frequency.

[0147] It should be explained that the semantics of an element can be expressed in two forms: explicit and implicit. Explicit representation uses the word corresponding to the semantics of the element. For example, the word corresponding to the element "Mr." is "gender." Implicit representation can be expressed using the element's feature vector, that is, the feature vector obtained after the element passes through the Embedding layer. The position of an element in the input sequence can be represented by the element's number in the input sequence. For example, for an input sequence of length 4096, the element numbers are 0, 1, 2, ..., 4095, and the positions of the elements are 0, 1, 2, ..., 4095, respectively. The word frequency of an element is pre-determined through big data statistical analysis. For example, by counting user queries within a preset time period, and then performing word frequency statistics on the elements in the user queries within the preset time period, the word frequency of each element is obtained.

[0148] Different natural language processing tasks may require different attribute features, and different attribute features have different grouping rules. Therefore, we first determine the task type of the natural language processing task. Then, based on the task type, we obtain the attribute features corresponding to each element in the input sequence. For example, if the natural language processing task is a conversational task, the attribute feature of the element is the element's position in the input sequence. Alternatively, if the natural language processing task is a sentiment recognition task, the attribute feature of the element is the element's semantic meaning.

[0149] Then, based on the attribute characteristics of each element in the input sequence, the elements in the input sequence are grouped to obtain a third element group and a fourth element group, wherein the elements in the third element group have a higher correlation with the natural language processing task than the elements in the fourth element group, and the fourth element group is part of the second element group, that is, the elements in the fourth element group are secondary elements. It should be noted that since the screening of important elements using attribute characteristics is relatively rough, many unimportant elements will be screened as important elements. If an element is considered to be a secondary element using attribute characteristics, then the element is largely a secondary element, and no secondary discrimination and screening is performed. Exemplarily, the attribute characteristics of each element in the input sequence can be input into the HQAM module of the model to group the elements to obtain a third element group and a fourth element group.

[0150] The following describes in detail the process of dividing the elements in the input sequence into the third element group and the fourth element group based on the attribute characteristics of each element in the input sequence in combination with specific attribute characteristics.

[0151] For example, when the attribute features include the part of speech of the element, for natural language processing tasks, nouns and verbs often carry more semantic information, and these elements have a relatively high correlation with natural language processing tasks. Therefore, elements with nouns or verbs and these parts of speech can be regarded as important elements, while other auxiliary words, such as articles, conjunctions, pause words, etc., have less impact on the global semantics, and elements with these parts of speech can be regarded as secondary elements. Therefore, first, a first part of speech set is constructed using prior knowledge, wherein the first part of speech set includes but is not limited to nouns, verbs, and other parts of speech that carry more semantic information. Therefore, for the input sequence, the elements in the input sequence whose part of speech hits the first part of speech set are formed into a third element group, and the elements in the input sequence whose part of speech does not hit the first part of speech set are formed into a fourth element group.

[0152] For example, the input sequence is "The new phone is amazing, but the price is too high". In this input sequence, the nouns include: phone, price. Therefore, phone, is, price can be classified into the third element group, and The, new, amazing, but, the, too, high, can be classified into the fourth element group.

[0153] Of course, in practical applications, a priori knowledge can also be used to construct a second preset part-of-speech set, where the parts of speech entered in this second preset part-of-speech set include, but are not limited to, articles, conjunctions, pause words, and other parts of speech with less impact on global semantics. Therefore, for this input sequence, the elements in the input sequence that match the second preset part-of-speech set can be formed into a fourth element group, and the elements in the input sequence that do not match the second preset part-of-speech set can be formed into a third element group.

[0154] Exemplarily, when the attribute features include the semantics of the element, and the semantics are explicitly expressed, that is, when the semantics of the element are expressed by words corresponding to the semantics, then the semantics of each element in the input sequence obtained are essentially the words corresponding to the semantics of the element. Therefore, the task type of the natural language processing task can be obtained, and the task type is used to indicate which natural language processing task the natural language processing task belongs to. Then, a first dictionary corresponding to the task type is obtained, and the semantics recorded in the first dictionary are important semantics, and the elements containing the semantics can be regarded as important elements. Therefore, the elements in the input sequence whose semantics hit the first dictionary are formed into a third element group, and the elements in the input sequence whose semantics do not hit the first dictionary are formed into a fourth element group.

[0155] For example, for a conversation task, the semantics recorded in the first dictionary configured for the task include, but are not limited to, gender, name, age, and surname. If the input sequence includes: "I am Mr. Wang, and I would like to inquire about microloans," the semantic corresponding to the element "I" is "speaker," the semantic corresponding to the element "is" is "equivalence," the semantic corresponding to the element "Wang" is "surname," the semantic corresponding to the element "Mr." is "gender," the semantic corresponding to the element "want" is "will," the semantic corresponding to the element "consult" is "inquiry," and the semantic corresponding to "microloan" is "financial product." Therefore, if the elements "Wang" and "Mr." match the first dictionary, they are classified into the third element group. If the elements "I," "Is," "I want," "consult," and "microloan" do not match the first dictionary, they are classified into the fourth element group.

[0156] For example, if the attribute features include the semantics of the element, and the semantics of the element are implicitly represented, for example, by the first feature vector of the element, the task type of the natural language task can be obtained. Then, based on the task type and the first feature vector of each element in the input sequence, each element in the input sequence is classified to obtain the correlation probability between each element in the input sequence and the natural language processing task.

[0157] Exemplarily, the task type is encoded to obtain a task vector corresponding to the task type. This task vector is then superimposed on the first eigenvector of each element of the input sequence to obtain a ninth eigenvector for each element, wherein the ninth eigenvector contains the semantic information of each element and the task type. The ninth eigenvector of each element is then input into a pre-trained classification network for classification, obtaining the associated probability corresponding to each element.

[0158] Finally, the elements in the input sequence whose correlation probability is greater than the first threshold are grouped into the third element group. That is, if the correlation probability is greater than the first threshold, the element is preliminarily considered to be an important element and is classified into the third element group. The elements in the input sequence whose correlation probability is less than or equal to the first threshold are grouped into the fourth element group. That is, if the correlation probability is less than or equal to the first threshold, the element is directly determined to be a minor element and is classified into the fourth element group.

[0159] Exemplarily, when the attribute feature includes the position of the element in the input sequence, the natural language task type is obtained, and the position determination rule corresponding to the task type is obtained; based on the position determination rule and the position of each element in the input sequence, the elements in the input sequence are grouped to obtain a third element group and a fourth element group.

[0160] It should be noted that the pre-configured position determination rules are different for different natural language processing tasks. For example, for some natural language processing tasks, the beginning and end of the input sequence usually contain key information. In this case, the position determination rule corresponding to the natural language processing task can be set as follows: the elements at the front and the elements at the back of the input sequence are important elements, and the elements at other positions are secondary elements. For some natural language processing tasks, specific positions in the input sequence contain key information. In this case, the position determination rule corresponding to the natural language processing task can be set as follows: the elements at specific positions are important elements, and the elements at other positions are secondary elements.

[0161] In a specific example, when the natural language processing task is a conversational task, the opening remarks generally contain the user's personal information. Therefore, the content of the opening remarks is relatively important. Furthermore, the content of several historical conversation rounds that are relatively close to the current conversation round is highly relevant to the current conversation round. These historical conversation rounds are also relatively important and highly relevant to the conversation task. However, the content of some intermediate conversation rounds is less relevant to the current conversation round, making them less important and less relevant to the conversation task. Therefore, in the context of a conversational task, the elements at the beginning and end of the input sequence are important elements. Therefore, elements in the input sequence with a position less than a second threshold (i.e., at the beginning of the input sequence) and / or greater than a third threshold (i.e., at the end of the input sequence) can be classified as important elements, forming the aforementioned third element group. Accordingly, elements in the input sequence with a position greater than or equal to the second threshold and less than or equal to the third threshold (i.e., in the middle of the input sequence) can be classified as minor elements, forming the aforementioned fourth element group.

[0162] In another specific example, when the natural language processing task is a labeling task, the input sequence contains multiple given labels and explanations of each label, so that the model can use these multiple given labels to label the call text that needs to be labeled. Therefore, in the scenario of the labeling task, the labels in the input sequence have a relatively high correlation with the labeling task, so the labels can be regarded as important elements; the explanations of the labels in the input sequence have a relatively low correlation with the labeling task, so the explanations of the labels in the input sequence can be regarded as secondary elements. Generally speaking, the labels in the input sequence are located between preset elements (i.e., special symbols). Therefore, the position determination rule set for the labeling task is: the elements in the input sequence located between the preset elements are classified as important elements, and the elements located outside the preset elements are classified as secondary elements. Exemplarily, multiple preset elements in the input sequence are obtained, wherein the multiple preset elements are multiple special symbols, and the multiple special symbols include but are not limited to: ":", ";", " / / ", etc. Then, starting from the first preset element, two adjacent preset elements are selected in sequence from the multiple preset elements to obtain multiple preset element groups, wherein no repeated preset elements exist in any two adjacent preset elements selected. Finally, the elements in the input sequence located between each preset element group are formed into the third element group; and the elements in the input sequence other than the third element group are formed into the fourth element group.

[0163] For example, for labeling tasks, the "Lable" given in the input sequence is a relatively important element, while other elements are relatively minor elements. Figure 8, it can be recognized that the preset elements in the input sequence are ":" and " / / ", and all ":" and " / / " in the input sequence are detected. Then, ":" and the first " / / " are formed into an element group, and the elements between the preset element group, that is, Lable_1, are divided into the third element group, the second " / / " and the third " / / " are formed into another preset element group, and the elements between the preset element groups, that is, Lable_2, are divided into the third element group. The division is sequentially divided, and "Lable_1", "Lable_2", ..., "Lable_m" are divided into the third element group, and the minor elements in the input sequence except "Lable_1", "Lable_2", ..., "Lable_m" are divided into the fourth element group.

[0164] For example, when the attribute characteristics of an element include the word frequency of the element, the higher the word frequency of the element, the higher the probability of the element appearing, which also means that the element is an ordinary word. Such elements contain relatively little contextual semantic information. For example, the common stop words "oh" and "la" have a relatively high word frequency and contain relatively little semantic information. Such words have a relatively low relevance to natural language processing tasks. Elements with lower word frequencies appear uniquely in natural language processing tasks and have a closer relationship with specific semantics. Such words have a relatively high relevance to natural language processing tasks. Therefore, the elements in the input sequence whose word frequency is greater than the fourth threshold can be formed into a fourth element group, and the elements in the input sequence whose word frequency is less than or equal to the fourth threshold can be formed into a third element group.

[0165] Furthermore, regarding word frequency, a low word frequency does not necessarily mean that the element is unique. It may also be caused by user input error, or the element is a newly emerging word. Therefore, to ensure the accuracy of the third element group, this application also pre-configures a second dictionary with unique words pre-entered in it. These words are generally words with rich semantic information. In this way, the elements can be grouped by combining the dictionary and word frequency.

[0166] Exemplarily, the elements in the input sequence whose word frequency is less than or equal to the fourth threshold and that hit the second dictionary are formed into a third element group, that is, the elements with low word frequency and are unique are classified as important elements, thereby ensuring the accuracy of the division of the third element group. In addition, the elements in the input sequence whose word frequency is greater than the fourth threshold (i.e., relatively common words) and the elements in the input sequence whose word frequency is less than or equal to the fourth threshold and that do not hit the second dictionary (i.e., elements with low word frequency but not unique) are formed into the fourth element group.

[0167] By combining the second dictionary, it can be ensured that the elements classified into the third element group are important elements, and the elements classified into the fourth element group are minor elements, thereby ensuring the accuracy of grouping and further improving the accuracy of natural language processing tasks.

[0168] It should be noted that in actual applications, for each natural language processing task, multiple attribute features of elements can be used to group elements. For example, for a dialogue task, the position of the element in the input sequence and the semantics of the element can be obtained, and the elements can be grouped using the two attribute features of position and semantics. Exemplarily, when using multiple attribute features of elements to group elements, for each attribute feature, a third element group can be divided except for the third element group corresponding to each attribute feature. Finally, the multiple third element groups corresponding to the multiple attribute features are merged to obtain the above-mentioned third element group. Then, the remaining elements in the input sequence other than the elements in the third element group are divided into the above-mentioned fourth element group. Of course, the fourth element group corresponding to each attribute feature can also be obtained, and then the multiple fourth element groups corresponding to the multiple attribute features are merged to obtain the above-mentioned fourth element group, and then the remaining elements in the input sequence other than the elements in the fourth element group are divided into the above-mentioned third element group. Of course, it is also possible to obtain the third element group and the fourth element group corresponding to each attribute feature separately, and then merge the multiple third element groups corresponding to the multiple attribute features to obtain a candidate third element group, and merge the multiple fourth element groups corresponding to the multiple attribute features to obtain a candidate fourth element group. It should be noted that since different attribute features are grouped in different ways, for a certain element, it is classified as an important element under a certain attribute feature, and is classified as a secondary element under a certain attribute feature, resulting in the candidate third element group and the candidate fourth element group possibly having the same elements. Exemplarily, if the candidate third element group and the candidate fourth element group may have the same elements, the same elements in the candidate fourth elements can be deleted, and the same elements in the candidate third element group can be retained to obtain the above-mentioned third element group and the above-mentioned fourth element group; or, the same elements in the candidate third elements can be deleted, and the same elements in the candidate fourth element group can be retained to obtain the above-mentioned third element group and the above-mentioned fourth element group.

[0169] For ease of description, this application mainly uses an attribute feature of an element to group elements as an example for explanation.

[0170] It should be noted that since the attribute feature grouping is based on the surface characteristics of the elements, it is relatively rough. Especially for important elements, a large number of important elements will be screened out. Therefore, for the third element group, the elements in the third element group will be grouped again to screen out the truly important elements from the third element group.

[0171] Exemplarily, the first and fifth element groups are determined based on the first eigenvector of each element in the third element group, i.e., the elements in the third element group are reclassified into important elements and minor elements. For example, the first eigenvector of each element in the third element group is input into the scoring module of the above model to obtain the first and fifth element groups.

[0172] Specifically, a relevance score is performed on each element in the third element group by the scoring module to obtain a score for each element. More specifically, a score is performed on each element in the third element group by the scoring module to obtain a first score for each element, and then the first score of each element is normalized by the scoring normalization module to obtain a score for each element in the third element group. Further, based on the score of each element in the third element group, the elements in the third element group are grouped to obtain the above-mentioned first element group and fifth element group. Exemplarily, the elements in the third element group whose scores are greater than the scoring threshold are divided into the first element group, and the elements whose scores are less than or equal to the scoring threshold are divided into the fifth element group, then the elements in the first element group are important elements, and the elements in the fifth element group are secondary elements. Finally, the fifth element group and the fourth element group are merged to obtain the second element group, that is, the second element group includes the fourth element group and the fifth element group.

[0173] It can be seen that in the embodiment of the present application, when grouping elements, the elements can first be preliminarily grouped according to their attribute characteristics, and then the important elements preliminarily grouped can be further grouped using the scoring module, thereby ensuring that the important elements finally grouped are truly important elements and the secondary elements are truly secondary elements, ensuring the accuracy of element grouping, and thus improving the accuracy of the natural language processing task. Moreover, when scoring the elements, the scoring module only needs to score the elements in the third element group selected by the HQAM module, without having to score every element in the input sequence, thereby improving the efficiency of element grouping.

[0174] It is understood that when grouping elements using both their attribute characteristics and their scores, the attributes are first used to perform a preliminary grouping of the elements, and then the scores are used to further group the resulting third element group. However, in actual applications, both attribute characteristics and scores can be used for grouping simultaneously, i.e., in a parallel grouping approach. The element groups generated by the two grouping approaches can then be merged to obtain the first and second element groups described above.

[0175] Exemplarily, the attribute features of each element in the input sequence are obtained. Then, based on the attribute features of each element in the input sequence, the elements in the input sequence are grouped to obtain a third element group and a fourth element group, that is, the important elements are preliminarily divided into the third element group, and the minor elements are preliminarily divided into the fourth element group. Furthermore, based on the first feature vector of each element in the input sequence, the elements in the input sequence are scored for relevance to obtain a score for the elements in the input sequence; based on the score of each element in the input sequence, the elements in the input sequence are grouped to obtain a sixth element group and a seventh element group, wherein the elements in the sixth element group have a higher relevance to the natural language processing task than the elements in the seventh element group, that is, the important elements are preliminarily divided into the sixth element group, and the minor elements are preliminarily divided into the seventh element group.

[0176] Then, based on the third element group, the fourth element group, the sixth element group, and the seventh element group, the first element group and the second element group are obtained.

[0177] Optionally, the intersection of the third element group and the sixth element group is divided into the above-mentioned first element group, and the union of the fourth element group and the seventh element group is divided into the above-mentioned second element group. In this division method, the element is only divided into the first element group when both the HQAM module and the scoring module consider the element to be an important element, thereby ensuring that the selected important elements are truly important elements, avoiding grouping too many important elements, thereby slowing down the reasoning speed, and thus improving the accuracy of grouping and the reasoning speed. Alternatively, the union of the third element group and the sixth element group is divided into the above-mentioned first element group, and the intersection of the fourth element group and the seventh element group is divided into the above-mentioned second element group. Under this division method, if one of the HQAM module and the scoring module considers that the element is an important element, it will divide the element into the first element group. The reason for selecting the union is that the heuristic rule module is mainly a pre-established grouping rule, which may not be adaptable to some newly emerging words. The scoring module is an artificial intelligence module with autonomous learning ability, which can dig out the potential semantic information of these newly emerging words, thereby screening out important elements missed by the heuristic rule module. Therefore, taking the union as the first element group can obtain more comprehensive and rich important elements, avoid missing important elements, thereby ensuring the accuracy of grouping and improving the accuracy of natural language processing tasks.

[0178] The above grouping of sample elements is a comprehensive grouping of elements by attribute characteristics of the elements and scoring of the sample elements. However, in actual applications, the sample elements can also be grouped by attribute characteristics or scoring alone.

[0179] Optionally, in one embodiment of the present application, when attribute characteristics are used alone to group elements, the attribute characteristics of each element in the input sequence can be obtained. Then, based on the attribute characteristics of each element in the input sequence, the first element group and the second element group are determined, that is, the attribute characteristics of each element are used to directly determine which are important elements and which are secondary elements. Among them, the method of grouping the first element group and the second element group using the attribute characteristics of the elements is similar to the method of grouping the third element group and the fourth element group using the attribute characteristics of the elements described above, that is, the first element group is equivalent to the third element group described above, and the second element group is equivalent to the fourth element group described above, and no further description will be given.

[0180] Optionally, in one embodiment of the present application, when a scoring method is used alone to group elements, a correlation score is performed on each element in the input sequence based on the first eigenvector of each element in the input sequence to obtain a score for each element in the input sequence; for example, the first eigenvector of each element in the input sequence is input into a scoring module to obtain a score for each element in the input sequence. Based on the score of each element in the input sequence, the first element group and the second element group are determined. For example, elements in the input sequence with a score greater than a scoring threshold are divided into the first element group, and elements with a score less than or equal to the scoring threshold are divided into the second element group.

[0181] 902: Perform a first attention process on the elements in the first element group to obtain a first processing result.

[0182] Exemplarily, before the attention processing is performed on the elements, the first eigenvector of each element in the input sequence is first normalized and linearly transformed to obtain the fourth eigenvector of each element.

[0183] Then, if Figure 4 and Figure 5 As shown, the fourth eigenvector of each element in the first element group can be input into the first attention module for first attention processing to obtain a first processing result. The process of performing the first attention processing based on the fourth eigenvector of each element can refer to the above Figure 5 Part of the description of the first attention will not be repeated.

[0184] 903: Perform a second attention process on the elements in the second element group to obtain a second processing result.

[0185] For example, Figure 4 and Figure 5As shown, the fourth eigenvector of each element in the second element group is input into the second attention module for second attention processing to obtain a second processing result. The process of performing the second attention processing based on the fourth eigenvector of each element can refer to the above Figure 5 Part of the description of the first attention will not be repeated.

[0186] 904: Execute the natural language processing task based on the first processing result and the second processing result to obtain an output sequence.

[0187] Exemplarily, a linear transformation is performed on the first processing result to obtain a third processing result. Figure 4 As shown, the first processing result can be input into the Linear layer corresponding to the first attention module, and a linear transformation can be performed on the first processing result to obtain a third processing result; and the second processing result can be linearly transformed to obtain a fourth processing result. Similarly, the second processing result can be input into the Linear layer corresponding to the second attention module, and a linear transformation can be performed on the second processing result to obtain a fourth processing result.

[0188] Furthermore, the third processing result and the fourth processing result are fused to obtain a fifth processing result.

[0189] Optionally, the third processing result and the fourth processing result are concatenated to obtain the fifth processing result. Figure 10 As shown, assuming that the order of elements in the input sequence is "x1, y1, x2, y2, ..., xn, yn", after grouping the elements, the elements in the first element group include "x1, x2, ..., xn", and the elements in the second element group include "y1, y2, ..., yn". Figure 10 As shown, the third processing result and the fourth processing result are directly spliced ​​together to obtain Figure 10 The fifth processing result, shown on the right, includes the attention-processed results for each element in the input sequence and contains the complete semantic information of the entire input sequence. Therefore, directly concatenating the third and fourth processing results can quickly fuse the processing results of important elements with those of less important elements, improving fusion efficiency and, in turn, the execution efficiency of natural language processing tasks.

[0190] Alternatively, directly concatenating the third processing result with the fourth processing result can quickly merge them, but the position of the element in the input sequence is not considered. For example, Figure 10The fifth processing result shown is the processing result corresponding to the sequence "x1, x2, ..., xn, y1, y2, ..., yn". Although the fifth processing result includes the processing results of each element in the input sequence after attention processing, the positions of the elements in the fifth processing result are not arranged according to the order of the elements in the input sequence. The relative position information of the elements in the input sequence may be lost during subsequent decoding.

[0191] Therefore, in view of this, the present application will also map the third processing result based on the position of each element in the input sequence to obtain a sixth processing result. It can be understood that the third processing result only contains processing results corresponding to the elements in the first element group, wherein mapping the third processing result is to map the second eigenvector corresponding to the element at the position corresponding to the element in the second element group in the third processing result. For example, the mapped second eigenvector can be represented by a preset vector (for example, a zero vector) to obtain the sixth processing result. Therefore, the sixth processing result includes the second eigenvector corresponding to each element in the input sequence, and the second eigenvector corresponding to each element in the input sequence is arranged according to the position of each element in the input sequence, and the second eigenvector corresponding to the element of the second element group in the sixth processing result is a preset vector. Among them, the second eigenvector corresponding to the element in the first element group is the eigenvector corresponding to the element in the first processing result, and the new eigenvector obtained after Linear layer mapping.

[0192] For example, if Figure 11 As shown, for the third processing result corresponding to the first element group "x1, x2, ..., xn", the second eigenvector (zero vector, gray represents the vector as a zero vector) corresponding to the element "y1, y2, ..., yn" can be mapped to the position corresponding to the element "y1, y2, ..., yn" in the input sequence to obtain the sixth processing result. The sixth processing result includes the second eigenvector corresponding to each element in the input sequence.

[0193] Likewise, if Figure 11As shown, based on the position of each element in the input sequence, the fourth processing result is mapped to obtain a seventh processing result, wherein the mapping of the fourth processing result is to map the third eigenvector corresponding to the element at the position corresponding to the element in the first element group in the fourth processing result. For example, the mapped third eigenvector can be represented by a preset vector (for example, a zero vector) to obtain the seventh processing result. Therefore, the seventh processing result includes the third eigenvector corresponding to each element in the input sequence, and the third eigenvector corresponding to each element in the input sequence is arranged according to the position of each element in the input sequence, and the third eigenvector corresponding to the element of the first element group in the seventh processing result is a preset vector. Among them, the third eigenvector corresponding to the element in the second element group is essentially the eigenvector corresponding to the element in the second processing result, and the new eigenvector obtained after the Linear layer mapping.

[0194] Finally, if Figure 11 As shown, the sixth processing result and the seventh processing result are superimposed to obtain the fifth processing result.

[0195] It can be seen that in the embodiment of the present application, in the process of performing the natural language processing task, the input sequence is first grouped, and the elements with higher relevance to the natural language processing task, that is, the elements that are relatively important to the natural language processing task, are divided into the first element group, and the elements with lower relevance to the natural language processing task, that is, the elements that are relatively unimportant to the natural language processing task, are divided into the second element group. In addition, the first attention processing with a strong semantic information extraction capability is performed on the elements in the first element group. By performing semantic information extraction on the elements in the first element group through the first attention, richer semantic information can be extracted from the elements in the first element group, ensuring the integrity and richness of the semantic information of the entire input sequence, thereby ensuring the processing accuracy of the natural language processing task. The second attention processing with a relatively fast reasoning speed is performed on the elements in the second element group, which can quickly extract semantic information from the elements in the second element group, thereby improving the execution efficiency of the natural language processing task. Therefore, for the entire input sequence, by grouping the elements in the input sequence and dynamically allocating attention, it is achieved that while ensuring the processing accuracy of the natural language processing task, the execution efficiency of the natural language processing task can be improved.

[0196] Furthermore, after obtaining the fifth processing result, a natural language processing task is performed based on the fifth processing result to obtain the output sequence.

[0197] Exemplarily, the fifth processing result is superimposed with the first feature vector corresponding to each element in the input sequence to obtain a superimposed processing result. Then, the superimposed processing result is input into the FFN network for feature extraction to obtain a feature extraction result, and the feature extraction result is superimposed with the superimposed processing result to obtain a superimposed feature processing result. Finally, the superimposed feature processing result is subjected to normalization processing, linear mapping processing, activation processing, and word segmentation decoding processing to obtain an output sequence. Exemplarily, based on Figure 4 The model structure shown in FIG5 is used to process the superimposed feature processing results through RMSnorm, Linear, SoftMax, and Tokenizer decoder to obtain an output sequence.

[0198] It should be noted that, by grouping the elements in the input sequence, compared with the existing MHA, the number of elements that the first attention processing module needs to pay attention to will be reduced, and the semantic subspace that needs to be deployed will also be reduced. Correspondingly, the number of attention heads that need to be designed for the first attention processing module will also be reduced; similarly, compared with the existing GQA, the number of elements that the second attention processing module needs to pay attention to will also be reduced, and the number of groups of attention heads will also be reduced. For the model, the matrix parameters that need to be stored will be reduced, and the storage overhead will be reduced in the process of performing attention processing on the elements in the input sequence, as well as the number of accesses to the storage, thereby improving the execution efficiency of natural language tasks.

[0199] For example, if elements are not grouped and all attention processing is performed using MHA, then eight attention heads need to be designed for MHA. The dimensions of the Q matrix, K matrix, and V matrix corresponding to each attention head are all 512*512. Therefore, the dimensions of the Q matrix, K matrix, and V matrix corresponding to the eight attention heads for the entire MHA are all 8*512*512. After the application groups the elements, only four attention heads need to be designed for the MHA part, and the dimensions of the Q matrix, K matrix, and V matrix corresponding to the MHA part are all: 4*512*512. Then, after the elements are grouped, the GQA part of this application only needs to design 4 attention heads, and the four attention heads are divided into two groups. Then the dimension of the Q matrix corresponding to the GQA part is: 4*512*512, the dimension of the K matrix is ​​2*512*512, and the dimension of the V matrix is ​​2*512*512. From the overall perspective of the model, although the dimension of the Q matrix has not been reduced, the dimension of the K matrix and the V matrix has been reduced, so the parameters of the K matrix and the V matrix that need to be stored in the entire model are reduced. Therefore, the model of this application can store relatively fewer matrix parameters, reducing storage overhead and the number of memory accesses, thereby reducing access overhead.

[0200] It can be seen that in an embodiment of the present application, the elements in the input sequence are first grouped, important elements are divided into the first element group, and minor elements are divided into the second element group, and the first attention with relatively high semantic information extraction ability is adopted for the important elements, that is, attention processing is performed through MHA, so that relatively rich semantic information can be extracted from the important elements, and the loss of important semantic information is not caused, thereby ensuring the execution accuracy of the natural language processing task, and being able to output an accurate output sequence for the user to ensure the user experience; the second attention with relatively high reasoning speed is adopted for the minor elements, that is, attention processing is performed through GQA, so that the reasoning speed of the minor elements can be accelerated, thereby improving the execution efficiency of the natural language processing task. Therefore, the present application achieves the goal of improving the execution efficiency of the natural language processing task while ensuring the processing accuracy of the natural language processing task by dynamically allocating attention to the elements in the input sequence.

[0201] Furthermore, compared with the current pure MHA mechanism, the number of attention heads in the MHA part of the present application will also be reduced, and the number of K matrices and V matrices will also be relatively reduced. When MHA is used to perform the first attention processing on the elements in the first element group, the bandwidth occupancy and computational complexity of the kv cache can be effectively reduced. During the reasoning process, frequent access to the memory can be reduced, which will accelerate the reasoning process and further improve the execution efficiency of the natural language processing task. Furthermore, compared with the current pure GQA mechanism, the number of attention heads in the GQA part of the present application will be reduced, and the number of K matrices and V matrices will also be relatively reduced. When GQA is used to perform the second attention processing on the elements in the second element group, the bandwidth occupancy and computational complexity of the kv cache can be effectively reduced. During the reasoning process, frequent access to the memory can be reduced, which will accelerate the reasoning process and further improve the execution efficiency of the natural language processing task.

[0202] Furthermore, MHA can extract rich semantic information due to its global attention mechanism, while GQA has a relatively faster reasoning speed due to its reduced head size. Therefore, when processing long texts or long sequences, this application uses a mixture of the two types of attention. Most of the minor elements in the long text or long sequence can be processed by the more efficient GQA, while the important elements are processed in a refined manner using MHA. Therefore, this application's use of a mixture of the two types of attention can well adapt to long texts or long sequences and can improve the processing accuracy of long texts or long sequences.

[0203] It should be noted that for Figure 3-Figure 5The model shown in the figure has the Tokenizer Encoder, word embedding layer, normalization layer, linear mapping layer, activation layer, and Tokenizer Decoder in the model all pre-trained and do not participate in the model training process. The HQAM module is a pre-configured allocation rule and will not participate in the model training. Therefore, the model training of this application mainly trains the first attention module, the second attention module, and the scoring module in the model. Therefore, the following will be combined with Figure 3-Figure 5 The structure of the model shown details the model training process of this application.

[0204] See Figure 12 , Figure 12 This is a flow chart of a model training method provided in an embodiment of the present application. The method is applied to a model training device. The method includes but is not limited to the following steps:

[0205] 1201: Obtain a sample sequence corresponding to a natural language processing task and a label sequence of the sample sequence.

[0206] The model in this application is an LLM, and the LLM can perform any type of natural language processing task. Therefore, the above natural language processing tasks include, but are not limited to, one or more of the following: dialogue tasks, labeling tasks, entity recognition tasks, sentiment recognition tasks, place name recognition tasks, and text translation tasks.

[0207] It should be noted that because different natural language processing tasks require different input and / or output content, a separate sample sequence and label sequence corresponding to each natural language processing task must be constructed. For example, for a conversation task, the sample sequence constructed is the content of the previous N rounds of conversation, and the label sequence is the response content of the next round. For a labeling task, the sample sequence constructed is the text to be labeled, and the label sequence is the labels contained in the text.

[0208] It should also be noted that in this application, the sample sequence corresponding to each natural language processing task and the label sequence of the sample sequence can be merged together to form a large training set, and then the model can be trained using the training set as a whole to make the model converge on all natural language processing tasks; or for each natural language processing task, the sample sequence and sample label corresponding to the natural language processing task can be used alone to train the model until the model converges on the natural language processing task, and then the sample sequence and sample label corresponding to another natural language processing task can be used to train the model again to make the model converge on the natural language processing task, until the model converges on all natural language processing tasks. Therefore, this application does not limit the order of training models.

[0209] 1202: Group the sample elements in the sample sequence to obtain a first sample element group and a second sample element group.

[0210] The elements in the first sample element group have a higher relevance to the natural language processing task than the elements in the second sample element group. Accordingly, the elements in the first sample element group can be called important sample elements, and the elements in the second sample element group can be called secondary sample elements.

[0211] First, before grouping the sample elements in the sample sequence, it is necessary to segment the sample sequence to obtain the individual sample elements in the sample sequence, and then perform word encoding on each sample element to obtain the Token ID of each sample element. For example, the sample sequence can be input into the model's Tokenizer Encoder to perform word segmentation encoding on the sample sequence to obtain the Token ID of each sample element. Then, the Token ID of each sample element is word embedded to obtain the first feature vector of each sample element. For example, the Token ID of each sample element can be input into the Embedding function to perform word embedding to obtain the first feature vector of each sample element.

[0212] Optionally, in one embodiment of the present application, attribute features of each sample element in the sample sequence are first obtained. The attribute features include, but are not limited to, the part of speech of the sample element, the semantics of the sample element, the position of the sample element in the sample sequence, and the word frequency of the sample element. The properties of the attribute features of the sample elements are similar to those of the elements in the input sequence described above and are not described again. The method for obtaining the attribute features of the sample elements is similar to that for obtaining the attribute features of the elements in the input sequence described above and is not described again.

[0213] Then, based on the attribute characteristics of each sample element, the sample elements in the sample sequence are grouped to obtain a third sample element group and a fourth sample element group, wherein the elements in the third sample element group have a higher correlation with the natural language processing task than the elements in the fourth sample element group, and the fourth sample element group is part of the second sample element group. Exemplarily, the attribute characteristics of the sample elements in the sample sequence can be input into the HQAM module to group the sample elements to obtain a third sample element group and a fourth sample element group. The specific implementation of grouping the sample elements in the sample sequence to obtain the third sample element group and the fourth sample element group is similar to the above-mentioned method of grouping the elements in the input sequence to obtain the third element group and the fourth element group, and will not be described again.

[0214] Similarly, by initially grouping sample elements using attribute characteristics, we can initially screen out important sample elements and some minor sample elements. However, because the attribute characteristics screening is relatively coarse, a large number of important sample elements will be screened out. Therefore, for the third sample element group, the first feature vector of each sample element in the third sample element group is also input into the scoring module to obtain the first sample element group and the fifth sample element group.

[0215] Exemplarily, a scoring module performs a relevance score on each sample element in the third sample element group to obtain a score for each sample element. Specifically, the scoring module performs a score on each sample element in the third sample element group to obtain a first score for each sample element. The score normalization module then normalizes the first score for each sample element to obtain a score for each sample element. Furthermore, based on the score of each sample element in the third sample element group, the elements in the third sample element group are grouped to obtain a first sample element group and a fifth sample element group.

[0216] Exemplarily, sample elements in the third sample element group with scores greater than a scoring threshold are divided into a first sample element group, and sample elements with scores less than or equal to the scoring threshold are divided into a fifth sample element group. The elements in the first sample element group are important elements, and the elements in the fifth sample element group are less important elements. Finally, the fifth sample element group and the second sample element group are merged to obtain a second sample element group, i.e., the second sample element group includes the fourth sample element group and the fifth sample element group.

[0217] It can be seen that in the embodiment of the present application, when grouping sample elements, the sample elements can be preliminarily grouped according to the attribute characteristics of the sample elements, and then the important sample elements preliminarily grouped can be further grouped through the scoring module, so as to ensure that the important elements finally grouped are truly important elements, and the secondary elements are truly secondary elements, thereby ensuring the accuracy of the grouping of sample elements and thus improving the accuracy of the natural language processing task.

[0218] Optionally, in one embodiment of the present application, when grouping elements using the attribute characteristics and scores of the sample elements, the attribute characteristics are first used to perform a preliminary grouping of the elements, and then the scores are used to further group the sample element groups obtained by the preliminary grouping. However, in actual applications, the attribute characteristics and scores can be used simultaneously for grouping, that is, a parallel grouping method. The method of synchronously using attribute characteristics and scores to group the sample elements in the sample sequence in parallel is similar to the method of parallel grouping of elements in the input sequence described above, and will not be repeated here.

[0219] Optionally, in one embodiment of the present application, the attribute characteristics of the sample elements can be used alone to group the sample elements. First, the attribute characteristics of each sample element in the sample sequence are obtained; then, based on the attribute characteristics of each sample element in the sample sequence, the first sample element group and the second sample element group are determined. The method of grouping based on the attribute characteristics of each sample element to obtain the first sample element group and the second sample element group is similar to the method of grouping based on the attribute characteristics of each element in the input sequence to obtain the first element group and the second element group, and will not be repeated here.

[0220] Optionally, in one embodiment of the present application, a scoring method can be used alone to group the sample elements to obtain a first sample element group and a second sample element group. Exemplarily, the first feature vector of each sample element is input into a scoring module for correlation scoring to obtain a score for each sample element. Then, based on the score of each sample element, the first sample element group and the second sample element group are determined. For example, sample elements in the sample sequence with a score greater than a scoring threshold are grouped into the first sample element group, and sample elements with a score less than or equal to the scoring threshold are grouped into the second sample element group.

[0221] 1203: Input the first sample element group into the first attention module for first attention processing to obtain a first sample processing result.

[0222] Exemplarily, before performing attention processing on the sample elements, the first eigenvector of each sample element in the sample sequence is first normalized and linearly transformed to obtain the fourth eigenvector of each sample element.

[0223] For example, for the sample elements in the first sample element group, the sample elements in the first sample element group can be input into the first attention module for first attention processing to obtain a first sample processing result. The process of performing the first attention processing on the sample elements in the first sample element group can refer to the above-mentioned process of performing the first attention processing on the elements in the first element group, and will not be repeated here.

[0224] 1204: Input the second sample element group into the second attention module for second attention processing to obtain a second sample processing result.

[0225] For example, for the sample elements in the second sample element group, the sample elements in the second sample element group can be input into the second attention module for second attention processing to obtain a second sample processing result. The process of performing the second attention processing on the sample elements in the second sample element group can refer to the above-mentioned process of performing the second attention processing on the elements in the second element group, and will not be repeated here.

[0226] 1205: Execute the natural language processing task based on the first sample processing result and the second sample processing result to obtain a prediction sequence.

[0227] Exemplarily, a natural language processing task is performed based on the first sample processing result and the second sample processing result to obtain a prediction sequence, which is similar to the above-mentioned method of performing the natural language processing task based on the first processing result and the second processing result, and will not be repeated.

[0228] 1206: Based on the prediction sequence and the label sequence, train the first attention module and the second attention module to obtain the model.

[0229] Exemplarily, a loss is calculated based on the prediction sequence and the label sequence; the first attention module and the second attention module are trained based on the loss to obtain a model. Specifically, the prediction sequence includes multiple predicted elements, and the label sequence includes a real element corresponding to each predicted element. The cross-entropy loss between each predicted element and the real element corresponding to the predicted element can be calculated, and then the cross-entropy losses corresponding to the multiple predicted elements are averaged to obtain the above-mentioned loss.

[0230] It should be noted that, when only the HQAM module is used to group the sample elements, the scoring module can be omitted from the model, and the scoring module does not need to be trained. The first attention module and the second attention module can be directly used for training to obtain the above-mentioned model. When the scoring module is used to group the sample elements, for example, only the scoring module is used or the scoring module and the HQAM module are used at the same time to group the sample elements, then in the process of model training, in addition to training the first attention module and the second attention module, the scoring module also needs to be trained. Therefore, after calculating the loss, the first attention module, the second attention module and the scoring module can be trained based on the loss to obtain the model.

[0231] See Figure 13 , Figure 13 A schematic diagram of a task execution device provided in an embodiment of the present application. The task execution device 1300 includes: a grouping unit 1301 and a processing unit 1302, wherein:

[0232] a grouping unit 1301 configured to group elements in an input sequence to obtain a first element group and a second element group, wherein the elements in the first element group are more relevant to the natural language processing task than the elements in the second element group;

[0233] Processing unit 1302 is used to perform a first attention processing on the elements in the first element group to obtain a first processing result; perform a second attention processing on the elements in the second element group to obtain a second processing result; and perform the natural language processing task based on the first processing result and the second processing result to obtain an output sequence.

[0234] In one embodiment of the present application, in grouping elements in an input sequence to obtain a first element group and a second element group, the grouping unit 1301 is specifically configured to:

[0235] determining a third element group and a fourth element group based on attribute features of each element in the input sequence, wherein the elements in the third element group have a higher relevance to the natural language processing task than the elements in the fourth element group;

[0236] determining the first element group and the fifth element group based on the first eigenvector of each element in the third element group;

[0237] Based on the fourth element group and the fifth element group, the second element group is determined, wherein the second element group includes the fourth element group and the fifth element group.

[0238] In one embodiment of the present application, when the attribute feature includes the part of speech of an element, in determining the third element group and the fourth element group based on the attribute feature of each element in the input sequence, the grouping unit 1301 is specifically configured to:

[0239] The elements of the input sequence whose part of speech matches the first part of speech set are formed into the third element group;

[0240] The elements in the input sequence whose parts of speech do not match the first part of speech set are formed into the fourth element group.

[0241] In one embodiment of the present application, when the attribute feature includes the semantics of the element, in determining the third element group and the fourth element group based on the attribute feature of each element in the input sequence, the grouping unit 1301 is specifically configured to:

[0242] Obtaining a task type for the natural language processing task;

[0243] Obtaining a first dictionary corresponding to the task type;

[0244] The elements in the input sequence that semantically match the first dictionary form the third element group;

[0245] Elements in the input sequence that are semantically not matched in the first dictionary are grouped into the fourth element group.

[0246] In one embodiment of the present application, when the attribute feature includes the semantics of an element, in determining the third element group and the fourth element group based on the attribute feature of each element in the input sequence, the grouping unit 1301 is specifically configured to:

[0247] Obtaining a task type for the natural language processing task;

[0248] classifying each element of the input sequence based on the task type and a first feature vector of each element of the input sequence to obtain a correlation probability between each element of the input sequence and the natural language processing task;

[0249] The elements in the input sequence whose correlation probability is greater than a first threshold are formed into the third element group;

[0250] Elements in the input sequence whose correlation probability is less than or equal to the first threshold are formed into the fourth element group.

[0251] In one embodiment of the present application, when the attribute feature includes the position of the element in the input sequence; in determining the third element group and the fourth element group based on the attribute feature of each element in the input sequence, the grouping unit 1301 is specifically configured to:

[0252] The elements whose positions in the input sequence are smaller than a second threshold and / or larger than a third threshold form the third element group;

[0253] Elements in the input sequence whose positions are greater than or equal to the second threshold and less than or equal to the third threshold are formed into the fourth element group.

[0254] In one embodiment of the present application, when the attribute feature includes the position of the element in the input sequence; in determining the third element group and the fourth element group based on the attribute feature of each element in the input sequence, the grouping unit 1301 is specifically configured to:

[0255] Obtaining multiple preset elements in the input sequence;

[0256] Starting from the first preset element, two adjacent preset elements are selected in sequence from the plurality of preset elements to obtain a plurality of preset element groups, wherein no repeated preset elements exist between any two adjacent preset elements selected;

[0257] The elements in the input sequence that are located between each preset element group form the third element group;

[0258] The elements in the input sequence except the third element group are grouped into the fourth element group.

[0259] In one embodiment of the present application, when the attribute feature includes the word frequency of an element, in determining the third element group and the fourth element group based on the attribute feature of each element in the input sequence, the grouping unit 1301 is specifically configured to:

[0260] The elements in the input sequence whose word frequency is less than or equal to the fourth threshold and the elements in the input sequence that hit the second dictionary are formed into the third element group;

[0261] The elements in the input sequence whose word frequency is greater than a fourth threshold and the elements in the input sequence whose word frequency is less than or equal to the fourth threshold and which do not hit the second dictionary are formed into the fourth element group.

[0262] In one embodiment of the present application, before determining the third element group and the fourth element group based on the attribute characteristics of each element of the input sequence, the grouping unit 1301 is further configured to:

[0263] Obtaining a task type for the natural language processing task;

[0264] Based on the type of the natural language processing task, the attribute feature corresponding to the task type for each element in the input sequence is obtained.

[0265] In one embodiment of the present application, in determining the first element group and the fifth element group based on the first eigenvector of each element in the third element group, the processing unit 1302 is specifically configured to:

[0266] performing a relevance score on each element in the third element group based on the first eigenvector of each element in the third element group to obtain a score for each element in the third element group;

[0267] The first element group and the fifth element group are determined based on the score of each element in the third element group.

[0268] In one embodiment of the present application, in performing the natural language processing task based on the first processing result and the second processing result to obtain an output sequence, the processing unit 1302 is specifically configured to:

[0269] performing a linear transformation on the first processing result to obtain a third processing result;

[0270] performing a linear transformation on the second processing result to obtain a fourth processing result;

[0271] fusing the third processing result and the fourth processing result to obtain a fifth processing result;

[0272] A natural language processing task is performed based on the fifth processing result to obtain the output sequence.

[0273] In one embodiment of the present application, in terms of fusing the third processing result and the fourth processing result to obtain the fifth processing result, the processing unit 1302 is specifically configured to:

[0274] The third processing result and the fourth processing result are concatenated to obtain the fifth processing result.

[0275] In one embodiment of the present application, in terms of fusing the third processing result and the fourth processing result to obtain the fifth processing result, the processing unit 1302 is specifically configured to:

[0276] Mapping the third processing result based on the position of each element in the input sequence to obtain a sixth processing result, wherein the sixth processing result includes a second eigenvector corresponding to each element in the input sequence, and the second eigenvectors corresponding to each element in the input sequence are arranged according to the position of each element in the input sequence, and the second eigenvectors corresponding to the elements of the second element group in the sixth processing result are preset vectors;

[0277] Mapping the fourth processing result based on the position of each element in the input sequence to obtain a seventh processing result, wherein the seventh processing result includes a third eigenvector corresponding to each element in the input sequence, and the third eigenvectors corresponding to each element in the input sequence are arranged according to the position of each element in the input sequence, and the third eigenvectors corresponding to the elements of the first element group in the seventh processing result are the preset vectors;

[0278] The sixth processing result and the seventh processing result are superimposed to obtain the fifth processing result.

[0279] In one embodiment of the present application, the first attention is multi-head attention, and the second attention is group query attention.

[0280] In one embodiment of the present application, the natural language processing task includes at least one of the following: a dialogue task, a labeling task, an entity recognition task, a sentiment recognition task, a place name recognition task, and a text translation task;

[0281] In the case where the natural language processing task is the dialogue task, the input sequence includes the dialogue content of the first round of dialogue, and the output sequence includes the reply content of the current round of dialogue;

[0282] In the case where the natural language processing task is the labeling task, the input sequence includes the conversation text to be labeled, and the output sequence includes the label to which the conversation text belongs;

[0283] In the case where the natural language processing task is an entity recognition task, the input sequence includes text requiring entity recognition, and the output sequence is entities in the text;

[0284] In the case where the natural language processing task is an emotion recognition task, the input sequence includes text requiring emotion recognition, and the output sequence is the emotion of the text;

[0285] In the case where the natural language processing task is the place name recognition task, the input sequence includes text requiring place name recognition, and the output sequence is the place name in the text;

[0286] In the case where the natural language processing task is the text translation task, the input sequence includes the text to be translated, and the output sequence includes the translated text.

[0287] In one embodiment of the present application, in identifying elements in an input sequence to obtain a first element group and a second element group, the grouping unit 1301 is specifically configured to:

[0288] Obtaining attribute characteristics of each element in the input sequence;

[0289] The first element group and the second element group are determined based on attribute characteristics of each element in the input sequence.

[0290] In one embodiment of the present application, in identifying elements in an input sequence to obtain a first element group and a second element group, the grouping unit 1301 is specifically configured to:

[0291] Performing a relevance score on each element in the input sequence based on the first eigenvector of each element in the input sequence to obtain a score for each element in the input sequence;

[0292] The first element group and the second element group are determined based on the score of each element in the input sequence.

[0293] See Figure 14 , Figure 14 This is a schematic diagram of a model training device provided in an embodiment of the present application. Figure 14 As shown, the model training device 1400 includes: an acquisition unit 1401 and a processing unit 1402, wherein:

[0294] An acquisition unit 1401 is configured to acquire a sample sequence corresponding to a natural language processing task and a label sequence of the sample sequence;

[0295] a processing unit 1402 configured to group the sample elements in the sample sequence to obtain a first sample element group and a second sample element group, wherein the elements in the first sample element group are more highly correlated with the natural language processing task than the elements in the second sample element group;

[0296] Inputting the first sample element group into the first attention module for first attention processing to obtain a first sample processing result;

[0297] Inputting the second sample element group into the second attention module for second attention processing to obtain a second sample processing result;

[0298] Performing the natural language processing task based on the first sample processing result and the second sample processing result to obtain a prediction sequence;

[0299] Based on the prediction sequence and the label sequence, the first attention module and the second attention module are trained to obtain the model.

[0300] In one embodiment of the present application, the model further includes a scoring module; in terms of identifying the sample elements in the sample sequence to obtain the first sample element group and the second sample element group, the processing unit 1402 is specifically configured to:

[0301] determining, based on the attribute characteristics of each sample element in the sample sequence, a third sample element group and a fourth sample element group, wherein the elements in the third sample element group have a higher relevance to the natural language processing task than the elements in the fourth sample element group;

[0302] Inputting the first feature vector of each sample element in the third sample element group into the scoring module to obtain the first sample element group and the fifth sample element group;

[0303] determining the second sample element group based on the fourth sample element group and the fifth sample element group, wherein the second sample element group includes the fourth sample element group and the fifth sample element group;

[0304] In terms of training the first attention module and the second attention module based on the prediction sequence and the label sequence to obtain the model, the processing unit 1402 is specifically configured to:

[0305] Based on the prediction sequence and the label sequence, the first attention module, the second attention module, and the scoring module are trained to obtain the model.

[0306] In one embodiment of the present application, in inputting the first feature vector of each sample element in the third sample element group into the scoring module to obtain the first sample element group and the fifth sample element group, the processing unit 1402 is specifically configured to:

[0307] Inputting the first feature vector of each sample element in the third sample element group into the scoring module to obtain a score for each sample element in the third sample element group;

[0308] The first sample element group and the fifth sample element group are determined based on the score of each sample element in the third sample element group.

[0309] See Figure 15 , Figure 15This is a schematic diagram of an electronic device provided in an embodiment of the present application. Figure 15 As shown, electronic device 1500 includes a transceiver 1501, a processor 1502, and a memory 1503. These are connected via a bus 1504. Memory 1503 is used to store computer programs and data, and can transmit data stored in memory 1503 to processor 1502. Alternatively, electronic device 1500 can be the task execution device 1300 or model training device 1400 described above.

[0310] Exemplarily, when the electronic device 1500 is the task execution device 1300, the processor 1502 is configured to read the computer program in the memory 1503 and perform the following operations:

[0311] Grouping elements in the input sequence to obtain a first element group and a second element group, wherein the elements in the first element group are more relevant to the natural language processing task than the elements in the second element group;

[0312] Perform a first attention processing on the elements in the first element group to obtain a first processing result; perform a second attention processing on the elements in the second element group to obtain a second processing result; perform the natural language processing task based on the first processing result and the second processing result to obtain an output sequence.

[0313] Optionally, the specific functions of the processor 1502 may refer to the specific functions of the grouping unit 1301 and the processing unit 1302 of the task execution device 1300, and will not be described again.

[0314] Exemplarily, when the electronic device 1500 is the above-mentioned model training device 1400, the processor 1502 is configured to read the computer program in the memory 1503 and perform the following operations:

[0315] Obtaining a sample sequence corresponding to a natural language processing task and a label sequence of the sample sequence;

[0316] The sample elements in the sample sequence are grouped to obtain a first sample element group and a second sample element group, wherein the elements in the first sample element group have a higher correlation with the natural language processing task than the elements in the second sample element group; the first sample element group is input into the first attention module for first attention processing to obtain a first sample processing result; the second sample element group is input into the second attention module for second attention processing to obtain a second sample processing result; the natural language processing task is performed based on the first sample processing result and the second sample processing result to obtain a prediction sequence; the first attention module and the second attention module are trained based on the prediction sequence and the label sequence to obtain the model.

[0317] Optionally, the specific functions of the processor 1502 can refer to the specific functions of the acquisition unit 1401 and the processing unit 1402 of the above-mentioned task model training device 1400, and will not be described again.

[0318] It should be understood that the user devices in this application may include mobile phones (such as Android phones, iOS phones, etc.), tablet computers, PDAs, laptops or wearable devices, etc. The above electronic devices are examples only, not exhaustive, and include but are not limited to the above electronic devices. The task execution device or model training transposition of this application can be a server, wherein the server can be a cloud server, a content delivery network (Content Delivery Network, CDN) server, a network time protocol (Network Time Protocol, NTP), a domain name resolution system (Domain Name System, DNS) server, and other various types of servers. The above servers are examples only, not exhaustive, and include but are not limited to the above servers.

[0319] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement part or all of the steps of any task execution method or model training method recorded in the above method embodiments.

[0320] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps of any task execution method or model training method recorded in the above method embodiments.

[0321] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0322] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0323] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0324] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0325] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0326] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0327] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0328] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A task execution method, characterized in that: include: Grouping elements in the input sequence to obtain a first element group and a second element group, wherein the elements in the first element group are more relevant to the natural language processing task than the elements in the second element group; performing a first attention process on the elements in the first element group to obtain a first processing result; performing a second attention process on the elements in the second element group to obtain a second processing result; The natural language processing task is performed based on the first processing result and the second processing result to obtain an output sequence.

2. The method according to claim 1, characterized in that The grouping of elements in the input sequence to obtain a first element group and a second element group includes: determining a third element group and a fourth element group based on attribute features of each element in the input sequence, wherein the elements in the third element group have a higher relevance to the natural language processing task than the elements in the fourth element group; determining the first element group and the fifth element group based on the first eigenvector of each element in the third element group; Based on the fourth element group and the fifth element group, the second element group is determined, wherein the second element group includes the fourth element group and the fifth element group.

3. The method according to claim 2, characterized in that The determining of the third element group and the fourth element group based on the attribute characteristics of each element in the input sequence includes: In the case where the attribute feature includes the part of speech of the element, the elements in the input sequence whose part of speech matches the first part of speech set are formed into the third element group; the elements in the input sequence whose part of speech does not match the first part of speech set are formed into the fourth element group; Alternatively, in the case where the attribute feature includes the semantics of the element, obtaining a task type of the natural language processing task; obtaining a first dictionary corresponding to the task type; forming the third element group from the elements in the input sequence whose semantics match the first dictionary; and forming the fourth element group from the elements in the input sequence whose semantics do not match the first dictionary; Alternatively, in a case where the attribute feature includes the semantics of the element, obtaining a task type of the natural language processing task; classifying each element of the input sequence based on the task type and the first feature vector of each element in the input sequence to obtain a correlation probability between each element in the input sequence and the natural language processing task; forming the third element group from the elements in the input sequence having a correlation probability greater than a first threshold; and forming the fourth element group from the elements in the input sequence having a correlation probability less than or equal to the first threshold; Alternatively, in the case where the attribute feature includes the position of the element in the input sequence, the elements whose positions in the input sequence are less than a second threshold and / or greater than a third threshold are grouped into the third element group; the elements whose positions in the input sequence are greater than or equal to the second threshold and less than or equal to the third threshold are grouped into the fourth element group; Alternatively, in a case where the attribute feature includes the position of an element in the input sequence, a plurality of preset elements in the input sequence are obtained; starting from the first preset element, two adjacent preset elements are sequentially selected from the plurality of preset elements to obtain a plurality of preset element groups, wherein no preset elements are repeated between any two selected adjacent preset elements; elements in the input sequence located between each preset element group are formed into the third element group; and elements in the input sequence other than the third element group are formed into the fourth element group; Alternatively, in the case where the attribute feature includes the word frequency of the element, the elements in the input sequence whose word frequency is less than or equal to the fourth threshold and hits the second dictionary are formed into the third element group; the elements in the input sequence whose word frequency is greater than the fourth threshold, and the elements in the input sequence whose word frequency is less than or equal to the fourth threshold and does not hit the second dictionary are formed into the fourth element group.

4. The method according to claim 2 or 3, characterized in that Before determining the third element group and the fourth element group based on the attribute characteristics of each element of the input sequence, the method further includes: Obtaining a task type for the natural language processing task; Based on the type of the natural language processing task, the attribute feature corresponding to the task type for each element in the input sequence is obtained.

5. The method according to any one of claims 2 to 4, characterized in that: The determining the first element group and the fifth element group based on the first eigenvector of each element in the third element group includes: performing a relevance score on each element in the third element group based on the first eigenvector of each element in the third element group to obtain a score for each element in the third element group; The first element group and the fifth element group are determined based on the score of each element in the third element group.

6. The method according to any one of claims 1 to 5, characterized in that The performing the natural language processing task based on the first processing result and the second processing result to obtain an output sequence includes: performing a linear transformation on the first processing result to obtain a third processing result; performing a linear transformation on the second processing result to obtain a fourth processing result; fusing the third processing result and the fourth processing result to obtain a fifth processing result; A natural language processing task is performed based on the fifth processing result to obtain the output sequence.

7. The method according to claim 6, characterized in that The fusing the third processing result and the fourth processing result to obtain a fifth processing result includes: Mapping the third processing result based on the position of each element in the input sequence to obtain a sixth processing result, wherein the sixth processing result includes a second eigenvector corresponding to each element in the input sequence, and the second eigenvectors corresponding to each element in the input sequence are arranged according to the position of each element in the input sequence, and the second eigenvectors corresponding to the elements of the second element group in the sixth processing result are preset vectors; Mapping the fourth processing result based on the position of each element in the input sequence to obtain a seventh processing result, wherein the seventh processing result includes a third eigenvector corresponding to each element in the input sequence, and the third eigenvectors corresponding to each element in the input sequence are arranged according to the position of each element in the input sequence, and the third eigenvectors corresponding to the elements of the first element group in the seventh processing result are the preset vectors; The sixth processing result and the seventh processing result are superimposed to obtain the fifth processing result.

8. The method according to any one of claims 1 to 7, characterized in that The natural language processing task includes at least one of the following: a dialogue task, a labeling task, an entity recognition task, a sentiment recognition task, a place name recognition task, and a text translation task; In the case where the natural language processing task is the dialogue task, the input sequence includes the dialogue content of the first round of dialogue, and the output sequence includes the reply content of the current round of dialogue; In the case where the natural language processing task is the labeling task, the input sequence includes the conversation text to be labeled, and the output sequence includes the label to which the conversation text belongs; In the case where the natural language processing task is an entity recognition task, the input sequence includes text requiring entity recognition, and the output sequence is entities in the text; In the case where the natural language processing task is an emotion recognition task, the input sequence includes text requiring emotion recognition, and the output sequence is the emotion of the text; In the case where the natural language processing task is the place name recognition task, the input sequence includes text requiring place name recognition, and the output sequence is the place name in the text; In the case where the natural language processing task is the text translation task, the input sequence includes the text to be translated, and the output sequence includes the translated text.

9. The method according to claim 1, characterized in that The grouping of elements in the input sequence to obtain a first element group and a second element group includes: Acquire attribute characteristics of each element in the input sequence; determine the first element group and the second element group based on the attribute characteristics of each element in the input sequence; Alternatively, based on the first eigenvector of each element in the input sequence, a correlation score is performed on each element in the input sequence to obtain a score for each element in the input sequence; and based on the score of each element in the input sequence, the first element group and the second element group are determined.

10. A model training method, characterized in that: The model includes a first attention module and a second attention module; the method includes: Obtaining a sample sequence corresponding to a natural language processing task and a label sequence of the sample sequence; Grouping the sample elements in the sample sequence to obtain a first sample element group and a second sample element group, wherein the elements in the first sample element group are more highly correlated with the natural language processing task than the elements in the second sample element group; Inputting the first sample element group into the first attention module for first attention processing to obtain a first sample processing result; Inputting the second sample element group into the second attention module for second attention processing to obtain a second sample processing result; Performing the natural language processing task based on the first sample processing result and the second sample processing result to obtain a prediction sequence; Based on the prediction sequence and the label sequence, the first attention module and the second attention module are trained to obtain the model.

11. The method according to claim 10, characterized in that The model further includes a scoring module; the identifying of sample elements in the sample sequence to obtain a first sample element group and a second sample element group includes: determining a third sample element group and a fourth sample element group based on attribute characteristics of each sample element in the sample sequence, wherein the elements in the third sample element group have a higher correlation with the natural language processing task than the elements in the fourth sample element group; Inputting the first feature vector of each sample element in the third sample element group into the scoring module to obtain the first sample element group and the fifth sample element group; determining the second sample element group based on the fourth sample element group and the fifth sample element group, wherein the second sample element group includes the fourth sample element group and the fifth sample element group; The training of the first attention module and the second attention module based on the prediction sequence and the label sequence to obtain the model includes: Based on the prediction sequence and the label sequence, the first attention module, the second attention module, and the scoring module are trained to obtain the model.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 11.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images