Language model training, language processing methods, apparatuses, devices, and storage media

Abstract

This invention discloses a language model training, language processing method, apparatus, device, and storage medium. The method includes: inputting a tokenized representation of sample text into a language processing model to be trained; the model includes an input layer, an encoder layer, and an output layer; obtaining an encoder input hidden vector based on the tokenized representation of the sample text through the input layer; obtaining an encoder output hidden vector based on the encoder input hidden vector, the relative position embedding vector of the encoder layer, a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter through the encoder layer; obtaining a training result based on the encoder output hidden vector through the output layer; and determining the function value of a training loss function based on the training result, and adjusting the model parameters of the language processing model according to the function value of the training loss function. This invention can improve the adaptability, processing accuracy, and transfer efficiency of language processing models.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a language model training, language processing method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, natural language processing models such as the Bidirectional Encoder Representations from Transformers (BERT) model and the Decoding-enhanced BERT with Disentangled Attention (DeBERTa) model have received widespread attention and citations. The DeBERTa model, through its innovative decoupling attention mechanism, separates and then fuses the content and positional information of words, enabling the model to more accurately understand language structure and grammatical relationships. Simultaneously, by incorporating absolute positional information through an enhanced mask decoder, it effectively improves the model's ability to model context. However, during the pre-training of the DeBERTa model, the attention allocation ratio between content and positional information cannot be adjusted and optimized. Therefore, the resulting language processing model exhibits poor task adaptability, limited processing accuracy, and low knowledge transfer efficiency. Summary of the Invention

[0003] This invention provides a language model training, language processing method, apparatus, device, and storage medium that can improve the adaptability, processing accuracy, and transfer efficiency of language processing models.

[0004] In a first aspect, embodiments of the present invention provide a language processing model training method, comprising:

[0005] The tokenized representation of the sample text is input into the language processing model to be trained. The language processing model is a DeBERTa-based model, which includes an input layer, an encoder layer, and an output layer. The encoder layer includes an adaptive decoupling attention mechanism module and an encoder output module. The adaptive decoupling attention mechanism module includes a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter that correspond one-to-one with the semantic content interaction weight, the content-to-position interaction weight, and the position-to-content interaction weight.

[0006] The tokenized representation based on sample text obtains the encoder input hidden vector through the input layer;

[0007] The attention score matrix is ​​determined by the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter and third fusion weight parameter through the adaptive decoupling attention mechanism module, and the encoder output hidden vector is obtained by the encoder output module based on the attention score matrix.

[0008] The training results are obtained through the output layer based on the encoder's output hidden vectors; and

[0009] The value of the training loss function is determined based on the training results, and the model parameters of the device language processing model are adjusted according to the value of the training loss function.

[0010] Secondly, embodiments of the present invention provide a language processing model training apparatus, comprising:

[0011] The input module is used to input the tokenized representation of the sample text into the language processing model to be trained. The language processing model is a DeBERTa-based model, which includes an input layer, an encoder layer, and an output layer. The encoder layer includes an adaptive decoupling attention mechanism module and an encoder output module. The adaptive decoupling attention mechanism module includes a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter that correspond one-to-one with the semantic content interaction weight, the content-to-position interaction weight, and the position-to-content interaction weight.

[0012] The input hidden vector acquisition module is used to obtain the encoder input hidden vector through the input layer based on the tokenized representation of the sample text;

[0013] The attention score matrix acquisition module is used to determine the attention score matrix through the adaptive decoupling attention mechanism module based on the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter and third fusion weight parameter, and obtain the encoder output hidden vector through the encoder output module based on the attention score matrix;

[0014] The training result acquisition module is used to obtain the training results through the output layer based on the encoder's output hidden vectors; and

[0015] The parameter adjustment module is used to determine the function value of the training loss function based on the training processing results, and adjust the model parameters of the device language processing model according to the function value of the training loss function.

[0016] Thirdly, a language processing method is also provided, comprising: obtaining a tokenized representation of the language text to be processed, and obtaining the processing result corresponding to the language text to be processed by training a language processing model using the language processing model training method provided in any embodiment of the present invention based on the tokenized representation of the language text to be processed.

[0017] Fourthly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the language processing model training method or language processing method as described in any of the embodiments of the present invention.

[0018] Fifthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the language processing model training method or language processing method as described in any of the embodiments of the present invention.

[0019] This invention provides a language model training, language processing method, apparatus, device, and storage medium. By inputting the tokenized representation of sample text into an adaptive decoupled attention mechanism module, a language processing model including a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter is constructed. Based on the tokenized representation of the sample text, an encoder input hidden vector is obtained through an input layer. Then, based on the encoder input hidden vector, relative position embedding vector, the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter, an attention score matrix is ​​determined through the adaptive decoupled attention mechanism module. Because the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight are set, the semantic content interaction weight, content-to-position interaction weight, and position-to-content interaction weight can be adaptively adjusted. This invention further obtains the encoder output matrix based on the attention score matrix and the training processing result based on the encoder output hidden vector. Then, based on the training processing result, the function value of the training loss and the model function are determined and adjusted. This improves the adaptability, processing accuracy, and transfer efficiency of the language processing model, thereby significantly improving the processing accuracy of the model when processing different downstream tasks. Attached Figure Description

[0020] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a language processing model training method provided in an embodiment of the present invention;

[0022] Figure 2 This is another flowchart illustrating the language processing model training method provided in this embodiment of the invention;

[0023] Figure 3This is another flowchart illustrating the language processing model training method provided in this embodiment of the invention;

[0024] Figure 4 This is a schematic diagram of a language processing model training device provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Figure 1 This is a flowchart illustrating a language processing model training method provided in an embodiment of the present invention. This method can be executed by a language processing model training device provided in this embodiment, which can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, such as a computer or server. The following embodiments will illustrate this using the integration of the device into an electronic device as an example. (Reference) Figure 1 The method may specifically include the following steps:

[0029] Step 101: Input the tokenized representation of the sample text into the language processing model to be trained. The language processing model is based on the DeBERTa architecture and includes an input layer, an encoder layer, and an output layer. The encoder layer includes an adaptive decoupled attention mechanism module and an encoder output module. The adaptive decoupled attention mechanism module includes a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter that correspond one-to-one with the semantic content interaction weights, content-to-position interaction weights, and position-to-content interaction weights. This step facilitates the acquisition of the relative position embedding vector and the encoder input hidden vector based on the tokenized representation of the sample text.

[0030] Specifically, the language processing model in the embodiments of the present invention can be an untrained original model or a pre-trained model.

[0031] Specifically, the sample text mentioned above can be a single sentence or a pair of sentences.

[0032] Specifically, the sample texts mentioned above can be unlabeled general texts, which can be obtained from the Internet, data, news corpora, or multilingual corpora.

[0033] Specifically, the sample text mentioned above can also be labeled task text, which can be determined based on a specific language processing task, such as text classification, named entity recognition, question answering, or text matching.

[0034] Optionally, the sample text is the task text corresponding to the language processing task.

[0035] Optionally, before step 101, a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter can be added to the adaptive decoupling attention mechanism module of the pre-trained DeBERTa model to obtain the language processing model to be trained.

[0036] Specifically, before step 101, the original text can be obtained and normalized, such as cleaning up garbled and redundant characters and unifying capitalization to obtain normalized text; the normalized text can be segmented into tokens that the model can understand using a token segmenter and necessary special tags can be added to obtain the token sequence corresponding to the sample text; the token sequence can be converted into the corresponding numerical identifier and the padding and attention masking problems of sequences of different lengths can be handled to obtain the tokenized representation of the sample text.

[0037] Specifically, the tokenization representation mentioned above includes a token identifier sequence, an attention mask, and a position index sequence.

[0038] Specifically, the encoder layer mentioned above may include a multi-layer decoder. The specific number of layers can be set based on empirical data or after multiple experiments and based on the experimental results. For example, it can be set to 12 or 24 layers.

[0039] Specifically, the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter can be set as matrix parameters, and the dimensions of the three matrix parameters can be the same or different.

[0040] Specifically, the first, second, and third fusion weight parameters can be determined based on the features of the sample text, or based on the functional requirements of the trained language processing model, or based on the computing resources, data scale, or training strategy used to train the language processing model.

[0041] Step 102: Based on the tokenized representation of the sample text, the encoder input hidden vector is obtained through the input layer. This step facilitates the determination of content interaction weights, content-position interaction weights, and position-content interaction weights based on the encoder input hidden vector, and further obtains the encoder output hidden vector based on the encoder input hidden vector, content interaction weights, content-position interaction weights, and position-content interaction weights.

[0042] Specifically, the above-mentioned tokenized representation based on sample text can be used to obtain the encoder input hidden vector through the input layer, which may include: obtaining the encoder input hidden vector of the first layer encoder through the input layer based on the tokenized representation based on sample text.

[0043] Specifically, the process of obtaining the encoder input hidden vector of the first layer encoder through the input layer using the tokenized representation based on the sample text may include: converting the input token identifier sequence into the corresponding content vector representation through a lookup table; generating the absolute position vector representation of each token according to its position index in the sequence; and adding the content embedding vector and the absolute position embedding vector element by element to generate the encoder input hidden vector of the first layer encoder.

[0044] Specifically, the above-mentioned tokenized representation based on sample text to obtain the encoder input hidden vector through the input layer may also include: obtaining the encoder input hidden vector of the Nth layer encoder through the input layer and the encoder from the first layer to the (N-1)th layer based on the tokenized representation of sample text, where N is an integer not less than 2 and not greater than the total number of encoder layers.

[0045] Specifically, the process of obtaining the encoder input hidden vector of the Nth layer encoder through the input layer and the encoders from the first layer to the (N-1)th layer based on the tokenized representation of the sample text includes: obtaining the output hidden vector of the (N-1)th layer encoder through the input layer and the encoders from the first layer to the (N-1)th layer based on the tokenized representation of the sample text, and obtaining the output hidden vector of the (N-1)th layer encoder as the encoder input hidden vector of the Nth layer encoder.

[0046] Step 103: Based on the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter, and third fusion weight parameter, the attention score matrix is ​​determined through the adaptive decoupling attention mechanism module. Then, based on the attention score matrix, the encoder output hidden vector is obtained through the encoder output module. This step, by setting the first fusion weight parameter, second fusion weight parameter, and third fusion weight, can adaptively adjust the semantic content interaction weight, content-to-position interaction weight, and position-to-content interaction weight.

[0047] Specifically, before step 103, the relative position embedding vector can be obtained from the shared relative position embedding matrix of the language processing model based on the relative distance between the query and key pairs.

[0048] Optionally, the process of determining the attention score matrix through the adaptive decoupling attention mechanism module based on the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter, and third fusion weight parameter includes: calculating the product of the encoder input hidden vector with the content query projection matrix and the content key projection matrix of the adaptive decoupling attention mechanism module to obtain the content query matrix and content key matrix respectively; calculating the product of the relative position embedding vector with the relative position query projection matrix and the relative position key projection matrix of the adaptive decoupling attention mechanism module to obtain the relative position query matrix and relative position key matrix respectively; calculating the product of the content query matrix and the transpose of the content key matrix to obtain the content interaction weight branch matrix to be fused; calculating the product of the content query matrix and the transpose of the relative position key matrix to obtain the content-to-position weight branch matrix to be fused; calculating the product of the content key matrix and the transpose of the relative position query matrix to obtain the position-to-content weight branch matrix to be fused; and weighting and summing the content interaction weight branch matrix, the content-to-position weight branch matrix, and the position-to-content weight branch matrix to be fused based on the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter to obtain the attention score matrix.

[0049] Specifically, the above-mentioned content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix can also be obtained through other existing technologies. For example, matrix block operation algorithm can be used to obtain the above-mentioned content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix.

[0050] Specifically, the content interaction weight branch matrix, the content-to-position weight branch matrix, and the position-to-content weight branch matrix can be represented as follows: , and ,in , , , .

[0051] Where H represents the encoder input hidden vector, P represents the relative position embedding vector, and the content query projection matrix and content key projection matrix are also present. and The corresponding representations are the content query projection matrix and the content key projection matrix. and The corresponding representations are the relative position query projection matrix and the relative position key projection matrix.

[0052] Optionally, the process of obtaining the attention score matrix by weighted summation of the content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix based on the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter includes:

[0053] Based on all or part of the elements of the scaling content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix corresponding to the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter, the semantic content interaction weight, content-to-position interaction weight, and position-to-content mutual weight are obtained. The element-wise summation of the semantic content interaction weight, content-to-position interaction weight, and position-to-content mutual weight is then used to obtain the determined attention score matrix.

[0054] Specifically, the process of obtaining the encoder output hidden vector based on the attention score matrix through the encoder output module can be performed using the following formula:

[0055]

[0056] in, This indicates that the encoder outputs the hidden vector. Represents the attention score matrix, , This is the projection matrix of the content values.

[0057] Specifically, when the differences between the first, second, and third fusion weight parameters are large, the attention score matrix can be obtained by weighted summation of the content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix to be fused based on the first, second, and third fusion weight parameters; otherwise, the attention score matrix can be obtained by element-wise summation of the content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix to be fused.

[0058] Specifically, the element-wise absolute error sum of each pair of the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter can be calculated. When any element-wise absolute error sum is greater than a preset error sum threshold, it can be determined that there is a large difference between each pair of the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter.

[0059] Step 104: Obtain the training results through the output layer based on the encoder's output hidden vectors. This step facilitates determining the value of the training loss function based on the training results, and consequently, adjusting the model parameters based on the value of the training loss function.

[0060] Specifically, the encoder output hidden vector mentioned above may include the output vectors of one or more encoders in the encoder layer.

[0061] Specifically, when the encoder output hidden vector includes only the output vector of one encoder, the above encoder output vector can be understood as the output vector of the last encoder in the encoder layer.

[0062] Step 105: Determine the value of the training loss function based on the training results, and adjust the model parameters of the device language processing model according to the value of the training loss function. Building upon steps 101 to 104, this step improves the adaptability, processing accuracy, and transfer efficiency of the language processing model, thereby significantly improving the model's processing accuracy when handling different downstream tasks.

[0063] Specifically, the training loss function mentioned above can be either the cross-entropy loss function or the mean squared error loss function.

[0064] The following further describes the language processing model training method provided by the embodiments of the present invention. The embodiments of the present invention are applicable to scenarios where the sample text is the task text corresponding to a language processing task, and the language processing model to be trained is a pre-trained DeBERTa model.

[0065] Optionally, the language processing model training method provided in this embodiment of the invention includes:

[0066] Step 201: Based on the length of the token identifier sequence in the tokenized representation of the sample text and the length threshold, set the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter.

[0067] Specifically, the aforementioned length threshold may include one length threshold or multiple length thresholds.

[0068] Optionally, the length threshold includes a first length threshold and a second length threshold, wherein the first length threshold is less than the second length threshold.

[0069] Specifically, the first and second length thresholds mentioned above can be set based on empirical data or computing resources.

[0070] Optionally, the process of setting the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter based on the length of the token identifier sequence and the length threshold in the tokenized representation based on the sample text includes: when the length of the token identifier sequence is less than the first length threshold, generating three square matrices with equal initial values, a dimension equal to the length of the token identifier sequence, and a sum of 1 for corresponding elements, to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter.

[0071] Optionally, the process of setting the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter based on the length of the token identifier sequence and the length threshold in the tokenized representation based on the sample text includes: when the length of the token identifier sequence is not less than the first length threshold and less than the second length threshold, generating three single-column matrices with equal initial values, column length equal to the length of the token identifier sequence, and the sum of the corresponding elements being 1, to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter.

[0072] Optionally, the process of setting the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter based on the length of the token identifier sequence and the length threshold in the tokenized representation based on the sample text includes: when the length of the token identifier sequence is not less than the second length threshold, generating three scalar values ​​with equal initial values ​​and a sum of 1 to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter.

[0073] Step 202: Input the tokenized representation of the sample text into the language processing model to be trained;

[0074] Step 203: Obtain the encoder input hidden vector through the input layer based on the tokenized representation of the sample text.

[0075] Step 204: Based on the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter and third fusion weight parameter, the attention score matrix is ​​determined by the adaptive decoupling attention mechanism module, and the encoder output hidden vector is obtained by the encoder output module based on the attention score matrix.

[0076] Optional, such as Figure 3 As shown, the process described above, which determines the attention score matrix through an adaptive decoupling attention mechanism module based on the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter, and third fusion weight parameter, and then obtains the encoder output hidden vector through the encoder output module based on the attention score matrix, includes:

[0077] Step 2041A: When the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter are scalar values, semantic content interaction weight, content-to-position interaction weight, and position-to-content interaction weight are obtained based on each element of the scaling content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix corresponding to the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter, respectively.

[0078] Step 2041B: When the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter are single-column matrices, the corresponding rows of the content interaction weight branch matrix, the content-to-position weight branch matrix, and the position-to-content weight branch matrix are scaled based on the row elements of the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter, respectively, to obtain the semantic content interaction weight, the content-to-position interaction weight, and the position-to-content interaction weight.

[0079] Step 2041C: When the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter are square matrices, calculate the element-wise multiplication results of the first fusion weight parameter with the content interaction weight branch matrix, the element-wise multiplication results of the second fusion weight parameter with the content-to-position weight branch matrix, and the element-wise multiplication results of the third fusion weight parameter with the position-to-content weight branch matrix, respectively, to obtain the semantic content interaction weight, the content-to-position interaction weight, and the position-to-content interaction weight.

[0080] Step 2042: Obtain the element-wise sum of the semantic content interaction weights, content-to-position interaction weights, and position-to-content interaction weights to obtain the attention score matrix.

[0081] Specifically, the attention score matrix can be obtained using the following formula:

[0082]

[0083] in, Represents the attention score matrix, and The corresponding parameters are the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter.

[0084] Specifically, when calculating the semantic content interaction weight matrix, one can... Assign to When assigning values, use "⊙" for element-wise multiplication instead of matrix multiplication; if the first, second, and third fusion weight parameters are scalar values, it is equivalent to scaling the entire matrix. If the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter are single-column matrices, Each row is multiplied by the corresponding row element of this vector; if the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter are square matrices, then... Each element multiplied by The corresponding element in.

[0085] Specifically, when calculating the content-to-location interaction weight matrix, because What is retrieved is the row of the content-to-position interaction weight matrix. Therefore, if the first, second, and third fusion weight parameters are single-column matrices or square matrices, an index needs to be used. To find the corresponding element in a single-column matrix or policy of the weight parameters; if the first, second, and third fusion weight parameters are scalar values, no index is needed.

[0086] Specifically, when calculating the position-content cross-weight matrix, because What we get are the columns of the position-to-content cross-weight matrix. Therefore, if the first, second, and third fusion weight parameters are square matrices, we need to use indexes. To find the corresponding element in a single-column matrix or policy of the weight parameters; if the first, second, and third fusion weight parameters are scalar values ​​or single-column matrices, no index is needed.

[0087] Step 205: Obtain the training results through the output layer based on the encoder's output hidden vector.

[0088] Step 206: Determine the value of the training loss function based on the training results, and adjust the model parameters of the device language processing model according to the value of the training loss function.

[0089] It is understood that if the embodiments of the present invention introduce a matrix matching the token identifier sequence as the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter, assuming the sequence length is N, since the content or relative position of each token must interact with the content or relative position of every other token, the magnitudes of the three weights are respectively This will increase The increase in the number of parameters, especially quadratic parameters, can lead to an excessively large number of model parameters, which can negatively impact model training and deployment.

[0090] This invention determines the first, second, and third fusion weight parameters by using the length and length threshold of the token identifier sequence in the tokenized representation of the sample text. This allows for the allocation of learnable weights line-by-line when the token identifier sequence is long. Since each row in the attention matrix represents the relationship between the content of a token and the content or relative position of all tokens, weight sharing is achieved at the token level, reducing the number of parameters to 3N. Furthermore, when the token identifier sequence is even longer, each decoupled attention matrix shares a learnable weight, reducing the number of new parameters to only 3. This enables adaptive adjustment of each decoupled attention matrix without adding a large number of parameters, avoiding a significant increase in model complexity and impacting computational efficiency and deployment feasibility.

[0091] Figure 4 This is a structural diagram of a language processing model training apparatus provided in an embodiment of the present invention. This apparatus is suitable for executing the language processing model training method provided in an embodiment of the present invention. Figure 4 As shown, the device may specifically include:

[0092] Input module 401 is used to input the tokenized representation of sample text into the language processing model to be trained. The language processing model is based on the DeBERTa architecture and includes an input layer, an encoder layer, and an output layer. The encoder layer includes an adaptive decoupling attention mechanism module and an encoder output module. The adaptive decoupling attention mechanism module includes a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter that correspond one-to-one with the semantic content interaction weight, the content-to-position interaction weight, and the position-to-content interaction weight. This module can facilitate the acquisition of relative position embedding vectors and encoder input hidden vectors based on the tokenized representation of sample text.

[0093] Optionally, the sample text is the task text corresponding to the language processing task.

[0094] The input hidden vector acquisition module 402 is used to obtain the encoder input hidden vector through the input layer based on the tokenized representation of the sample text. This module can determine the content interaction weights, content-position interaction weights, and position-content interaction weights based on the encoder input hidden vector, and further obtain the encoder output hidden vector based on the encoder input hidden vector, content interaction weights, content-position interaction weights, and position-content interaction weights.

[0095] The attention score matrix acquisition module 403 is used to determine the attention score matrix through an adaptive decoupled attention mechanism module based on the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter, and third fusion weight parameter, and to obtain the encoder output hidden vector based on the attention score matrix through the encoder output module. This module can adaptively adjust the semantic content interaction weight, content-to-position interaction weight, and position-to-content interaction weight by setting the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight.

[0096] Optionally, the attention score matrix acquisition module 403 can be specifically used to: calculate the content query matrix and content key matrix by multiplying the encoder input hidden vector with the content query projection matrix and content key projection matrix of the adaptive decoupling attention mechanism module, respectively; calculate the relative position query matrix and relative position key matrix by multiplying the relative position embedding vector with the relative position query projection matrix and relative position key projection matrix of the adaptive decoupling attention mechanism module, respectively; calculate the content interaction weight branch matrix to be fused by multiplying the content query matrix with the transpose of the content key matrix; calculate the content pair position weight branch matrix to be fused by multiplying the content query matrix with the transpose of the relative position key matrix; calculate the position pair content weight branch matrix to be fused by multiplying the content key matrix with the transpose of the relative position query matrix; and perform a weighted summation of the content interaction weight branch matrix, the content pair position weight branch matrix, and the position pair content weight branch matrix to be fused based on the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter to obtain the attention score matrix.

[0097] The training result acquisition module 404 is used to obtain the training results through the output layer based on the encoder's output hidden vectors. This module can help determine the value of the training loss function based on the training results, and thus help adjust the model parameters based on the value of the training loss function.

[0098] The parameter adjustment module 405 is used to determine the value of the training loss function based on the training results, and adjust the model parameters of the device language processing model according to the value of the training loss function. This module, together with modules 401 to 405, can improve the adaptability, processing accuracy, and transfer efficiency of the trained language processing model, thereby significantly improving the processing accuracy of the model when processing different downstream tasks.

[0099] Optionally, the language processing model training device provided in this embodiment of the invention further includes: a parameter addition module, which is used to add a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter to the adaptive decoupling attention mechanism module of the pre-trained DeBERTa model before inputting the tokenized representation of the sample text into the language processing model to be trained, so as to obtain the language processing model to be trained.

[0100] Optionally, the above parameter addition module can also be used to set the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter based on the length and length threshold of the token identifier sequence in the tokenized representation of the sample text.

[0101] Optionally, the length threshold includes a first length threshold and a second length threshold, wherein the first length threshold is less than the second length threshold.

[0102] Optionally, the above parameter addition module can also be used to generate three square matrices with equal initial values, dimensions equal to the length of the token identifier sequence, and sums of corresponding elements equal to 1 when the length of the token identifier sequence is less than the first length threshold, to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter.

[0103] When the length of the token identifier sequence is not less than the first length threshold and less than the second length threshold, generate three single-column matrices with equal initial values, column length equal to the length of the token identifier sequence, and sum of corresponding elements to 1, to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter;

[0104] When the length of the token identifier sequence is not less than the second length threshold, three scalar values ​​with equal initial values ​​and a sum of 1 are generated to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter.

[0105] Optionally, the attention score matrix acquisition module 403 can be specifically used to obtain semantic content interaction weights, content-to-position interaction weights, and position-to-content interaction weights based on each element of the scaling content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix corresponding to the first, second, and third fusion weight parameters when the first, second, and third fusion weight parameters are scalar values; and when the first, second, and third fusion weight parameters are single-column matrices, to obtain the scaling content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix corresponding to each row of the first, second, and third fusion weight parameters when the first, second, and third fusion weight parameters are single-column matrices. The corresponding rows of the matrix and the position-to-content weight branch matrix yield the semantic content interaction weight, content-to-position interaction weight, and position-to-content mutual weight. When the first, second, and third fusion weight parameters are square matrices, the element-wise multiplication results of the first fusion weight parameter with the content interaction weight branch matrix, the element-wise multiplication results of the second fusion weight parameter with the content-to-position weight branch matrix, and the element-wise multiplication results of the third fusion weight parameter with the position-to-content weight branch matrix are calculated respectively, yielding the semantic content interaction weight, content-to-position interaction weight, and position-to-content mutual weight. Finally, the element-wise summation of the semantic content interaction weight, content-to-position interaction weight, and position-to-content mutual weight is obtained to determine the attention score matrix.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] This invention also provides a language processing method, which includes: obtaining a tokenized representation of a language text to be processed, and obtaining a processing result corresponding to the language text to be processed by training a language processing model using the language processing model training method provided in any embodiment of this invention based on the tokenized representation of the language text to be processed.

[0108] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the language processing model training method or language processing method provided in any of the above embodiments.

[0109] This invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the language processing model training method or language processing method provided in any of the above embodiments.

[0110] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the language processing model training method or language processing method as described in any of the embodiments of this invention.

[0111] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing an electronic device according to embodiments of the present invention. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0112] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0113] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0114] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0115] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The modules and / or units described in the embodiments of this invention can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, a processor can be described as including an input module, an input hidden vector acquisition module, an attention score matrix acquisition module, a training result acquisition module, and a parameter tuning module. The names of these modules do not necessarily limit the module itself.

[0118] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist alone and not assembled into the device. The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the device to include: inputting a tokenized representation of sample text into a language processing model to be trained; the language processing model is a DeBERTa-based model, comprising an input layer, an encoder layer, and an output layer, wherein the encoder layer includes an adaptive decoupling attention mechanism module and an encoder output module, and the adaptive decoupling attention mechanism module includes a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter; obtaining an encoder input hidden vector based on the tokenized representation of the sample text through the input layer; determining an attention score matrix based on the encoder input hidden vector, the relative position embedding vector, the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter through the adaptive decoupling attention mechanism module, and obtaining an encoder output hidden vector based on the attention score matrix through the encoder output module; obtaining a training result based on the encoder output hidden vector through the output layer; and determining the function value of a training loss function based on the training result, and adjusting the model parameters of the device's language processing model according to the function value of the training loss function.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for training a language processing model, characterized in that, include: Based on the length of the token identifier sequence in the tokenized representation of the sample text, the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter are set. The length threshold includes a first length threshold and a second length threshold, wherein the first length threshold is less than the second length threshold. The specific setting method includes: when the length of the token identifier sequence is less than the first length threshold, generating three square matrices with equal initial values, a dimension equal to the length of the token identifier sequence, and a sum of 1 for corresponding elements, to obtain a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter; when the length of the token identifier sequence is not less than the first length threshold and is less than the second length threshold, generating three single-column matrices with equal initial values, a column length equal to the length of the token identifier sequence, and a sum of 1 for corresponding elements, to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter; when the length of the token identifier sequence is not less than the second length threshold, generating three scalar values ​​with equal initial values ​​and a sum of 1 to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter. The tokenized representation of the sample text is input into the language processing model to be trained. The language processing model is a DeBERTa-based model, which includes an input layer, an encoder layer, and an output layer. The encoder layer includes an adaptive decoupling attention mechanism module and an encoder output module. The adaptive decoupling attention mechanism module includes a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter that correspond one-to-one with the semantic content interaction weight, the content-to-position interaction weight, and the position-to-content interaction weight. The tokenized representation based on sample text obtains the encoder input hidden vector through the input layer; The attention score matrix is ​​determined by the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter and third fusion weight parameter through the adaptive decoupled attention mechanism module, and the encoder output hidden vector is obtained by the encoder output module based on the attention score matrix. The training results are obtained through the output layer based on the encoder's output hidden vectors; and The value of the training loss function is determined based on the training results, and the model parameters of the language processing model are adjusted according to the value of the training loss function.

2. The language processing model training method according to claim 1, characterized in that, The process of determining the attention score matrix through an adaptive decoupling attention mechanism module based on the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter, and third fusion weight parameter includes: The content query matrix and content key matrix are obtained by calculating the product of the encoder input hidden vector and the content query projection matrix and the content key projection matrix of the adaptive decoupled attention mechanism module, respectively. The relative position query matrix and the relative position key matrix are obtained by multiplying the relative position embedding vector with the relative position query projection matrix and the relative position key projection matrix of the adaptive decoupling attention mechanism module, respectively. The product of the content query matrix and the transpose of the content key matrix is ​​used to obtain the content interaction weight branch matrix to be fused. The product of the content query matrix and the transpose of the relative position key matrix is ​​used to obtain the position weight branch matrix of the content pairs to be merged. The product of the content key matrix and the transpose of the relative position query matrix is ​​used to obtain the position-to-content weight branch matrix to be fused; and The attention score matrix is ​​obtained by weighting and summing the content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix based on the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter.

3. The language processing model training method according to claim 1, characterized in that, The sample text is the task text corresponding to the language processing task; Before inputting the tokenized representation of the sample text into the language processing model to be trained, the method further includes: By adding a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter to the adaptive decoupling attention mechanism module of the pre-trained DeBERTa model, the language processing model to be trained is obtained.

4. The language processing model training method according to claim 2, characterized in that, The attention score matrix is ​​obtained by weighted summation of the content interaction weight branch matrix, content-to-position weight branch matrix, and position-to-content weight branch matrix based on the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter, including: When the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter are scalar values, the semantic content interaction weight, the content-to-position interaction weight, and the position-to-content interaction weight are obtained based on the elements of the scaling content interaction weight branch matrix, the content-to-position weight branch matrix, and the position-to-content weight branch matrix corresponding to the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter, respectively. When the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter are single-column matrices, the corresponding rows of the content interaction weight branch matrix, the content-to-position weight branch matrix, and the position-to-content weight branch matrix are scaled based on the row elements of the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter, respectively, to obtain the semantic content interaction weight, the content-to-position interaction weight, and the position-to-content interaction weight. When the first, second, and third fusion weight parameters are square matrices, the element-wise multiplication results of the first fusion weight parameter with the content interaction weight branch matrix, the element-wise multiplication results of the second fusion weight parameter with the content-to-position weight branch matrix, and the element-wise multiplication results of the third fusion weight parameter with the position-to-content weight branch matrix are calculated respectively, yielding the semantic content interaction weight, content-to-position interaction weight, and position-to-content interaction weight; and The attention score matrix is ​​determined by summing the element-wise weights of semantic content interaction, content-to-position interaction, and position-to-content interaction.

5. A language processing method, characterized in that, include: Obtain the tokenized representation of the language text to be processed; Based on the tokenized representation of the language text to be processed, the processing result corresponding to the language text to be processed is obtained by training the language processing model using any one of the language processing model training methods described in claims 1 to 4.

6. A language processing model training device, characterized in that, include: The parameter addition module is used to set the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter based on the length of the token identifier sequence in the tokenized representation of the sample text and the length threshold. The length threshold includes a first length threshold and a second length threshold, wherein the first length threshold is less than the second length threshold. The specific setting method includes: when the length of the token identifier sequence is less than the first length threshold, generating three square matrices with equal initial values, a dimension equal to the length of the token identifier sequence, and a sum of 1 for corresponding elements, to obtain a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter; when the length of the token identifier sequence is not less than the first length threshold and is less than the second length threshold, generating three single-column matrices with equal initial values, a column length equal to the length of the token identifier sequence, and a sum of 1 for corresponding elements, to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter; when the length of the token identifier sequence is not less than the second length threshold, generating three scalar values ​​with equal initial values ​​and a sum of 1 to obtain the first fusion weight parameter, the second fusion weight parameter, and the third fusion weight parameter. The input module is used to input the tokenized representation of the sample text into the language processing model to be trained. The language processing model is a DeBERTa-based model, which includes an input layer, an encoder layer, and an output layer. The encoder layer includes an adaptive decoupling attention mechanism module and an encoder output module. The adaptive decoupling attention mechanism module includes a first fusion weight parameter, a second fusion weight parameter, and a third fusion weight parameter that correspond one-to-one with the semantic content interaction weight, the content-to-position interaction weight, and the position-to-content interaction weight. The input hidden vector acquisition module is used to obtain the encoder input hidden vector through the input layer based on the tokenized representation of the sample text; The attention score matrix acquisition module is used to determine the attention score matrix through the adaptive decoupling attention mechanism module based on the encoder input hidden vector, relative position embedding vector, first fusion weight parameter, second fusion weight parameter and third fusion weight parameter, and obtain the encoder output hidden vector through the encoder output module based on the attention score matrix; The training result acquisition module is used to obtain the training results through the output layer based on the encoder's output hidden vectors; and The parameter adjustment module is used to determine the function value of the training loss function based on the training processing results, and to adjust the model parameters of the language processing model according to the function value of the training loss function.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the language processing model training method as described in any one of claims 1 to 4, or the language processing method as described in claim 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the language processing model training method as described in any one of claims 1 to 4, or the language processing method as described in claim 5.

Images