Cross-modal non-autoregressive decoding method and system combined with pre-trained language model

Abstract

A cross-modal non-autoregressive decoding method and system, combined with a pre-trained language model, involves cross-modal translation. The decoding system sequentially includes target permutation sampling, pre-trained text encoding, pre-trained text decoding, pre-training loss calculation, source-side feature extraction, downstream encoding, CTC enhancement, downstream decoding, downstream task loss calculation, and output. The decoding method involves: sampling pre-trained training samples and target permutations; encoding the source text into latent vectors; decoding the target text; calculating the loss in the pre-training phase; initializing the parameters of the downstream cross-modal non-autoregressive translation model; extracting source features; encoding source features; generating input to the downstream decoding module; generating translation results from the decoding module; calculating the loss of the downstream cross-modal non-autoregressive translation model; and outputting the downstream cross-modal non-autoregressive translation results. Random Drop and CTC are used to mitigate modality differences in decoder inputs during the pre-training and fine-tuning phases.

Description

Technical Field

[0001] The present invention relates to the fields of cross-modal translation, non-autoregressive text generation, and pre-trained language models, and in particular to a cross-modal non-autoregressive decoding method and system combined with a pre-trained language model. Background Art

[0002] In recent years, non-autoregressive (NAR) decoding mechanisms have been applied to cross-modal translation tasks such as sign language translation, effectively reducing model inference latency. Currently, cross-modal non-autoregressive translation models using a curriculum-based non-autoregressive decoder (CND) outperform autoregressive baseline models in both speed and performance.

[0003] The research paradigm of combining pre-training and fine-tuning has attracted widespread attention in academia. Existing work in academia has demonstrated that the capabilities of cross-modal autoregressive translation models can be significantly improved by combining them with autoregressive pre-trained language models. However, directly combining existing non-autoregressive pre-trained language models with cross-modal non-autoregressive translation models presents two problems: First, existing non-autoregressive pre-trained language models cannot explicitly model the dependencies between target words, which is crucial for cross-modal non-autoregressive translation models. Second, there is a modality difference between the representation spaces of the decoder inputs of existing non-autoregressive pre-trained language models and those of cross-modal non-autoregressive translation models. This is because the inputs to non-autoregressive decoders are typically copies of the word embeddings input to the encoder. For non-autoregressive pre-trained language models, the decoder inputs are in the same textual representation space as the encoder inputs. However, for the downstream cross-modal non-autoregressive translation model, the encoder inputs are in a non-textual representation space. This results in the decoder inputs also being in the same non-textual representation space as the encoder during fine-tuning, resulting in a modality difference between the two stages. Summary of the Invention

[0004] One purpose of the present invention is to better integrate a pre-trained language model into a non-autoregressive cross-modal translation model, and to provide a cross-modal non-autoregressive decoding method combined with a pre-trained language model by utilizing the ideas of Random Drop and CTC.

[0005] A second object of the present invention is to implement a cross-modal non-autoregressive translation system combined with a pre-trained language model through a cross-modal non-autoregressive decoding method.

[0006] To achieve the above objectives, the present invention provides the following technical solutions.

[0007] The present invention provides a cross-modal non-autoregressive decoding method combined with a pre-trained language model, comprising the following steps:

[0008] 1) Sampling pre-training training samples and target permutations: Sampling a data pair from the corpus used for pre-training and a permutation of the target sentence ,in, Each element in is the subscript of the target word in the target sequence;

[0009] 2) Encode the source text into a latent vector: Use the text encoding module of the pre-trained model to encode the source text into a latent vector. Encoded as a latent vector;

[0010] 3) Target-side text decoding: The pre-trained source text word embeddings are converted into input for a non-autoregressive decoder through a copying mechanism and then fed into the text decoding module. After receiving the output vector of the pre-trained text encoding module, the text decoding module generates the translation result based on the feature vector of the encoding module.

[0011] 4) Pre-training loss calculation: Use cross-entropy loss to calculate the translation loss of each decoder layer, and add up the losses of all layers to get the total loss;

[0012] 5) Initialize the parameters of the downstream cross-modal non-autoregressive translation model:

[0013] After the pre-trained language model is trained, its parameters can be used to initialize the encoder and decoder parameters of the downstream cross-modal non-autoregressive translation model;

[0014] 6) Source feature extraction: The feature extraction module extracts features from each frame of source data to obtain a feature sequence containing source information.

[0015] 7) Source feature encoding: The source feature vector extracted in step 6) is sent to the downstream encoding module to obtain a hidden representation;

[0016] 8) Downstream decoding module input generation: Combining the predicted probabilities of the CTC layer and the weight matrix of the target-side word embedding layer, the attention score matrix is ​​calculated using positional encoding and learnable weights, and used to generate the input of the decoding module to incorporate the target-side context information;

[0017] 9) The decoding module generates the translation result: The downstream decoding module generates the translation result by simultaneously obtaining the feature vectors of the decoding module input and the encoding module output;

[0018] 10) Calculate the loss of the downstream cross-modal non-autoregressive translation model: In the downstream loss calculation module, the CTC loss function is used to calculate the loss of the downstream encoding module, and the cross-entropy loss function is used to calculate the translation loss of the generated words in the downstream decoding module;

[0019] 11) Downstream cross-modal non-autoregressive translation result output: The output module outputs the spoken sentence obtained by the downstream cross-modal non-autoregressive translation model.

[0020] In step 3), the specific steps of target-side text decoding may be:

[0021] The source word embedding is converted into the input of the non-autoregressive decoder through the copy mechanism, requiring each layer of the decoder to make predictions, and each layer only retains the subscripts belonging to Before The decoder first extracts the words in the elements and replaces the remaining words with a special symbol [mask]. It then concatenates the word embeddings predicted by this layer with the hidden state of this layer, and uses an additional linear layer to reduce the dimensionality to obtain the output of the decoder at the current layer. In this way, the upper layers of the decoder can receive the predictions of the lower layers of the decoder, thereby predicting more words, and the entire sentence will be gradually predicted by the decoder from the bottom up.

[0022] In step 4), the specific steps of the loss calculation in the pre-training stage can be: in the loss calculation module of the pre-training stage, the translation loss of the non-autoregressive decoding module is calculated using the cross entropy loss; since the decoder performs predictions at each layer, the loss function of the pre-training stage is This is the sum of the cross entropy losses of each layer of the decoder:

[0023]

[0024] in, Indicates the decoder Layer The probability distribution of words; Indicates the number of decoder layers; Indicates the number of words predicted in the i-th layer, , Indicates the total number of words, represents the t-th target word;

[0025] In step 8), the specific steps of generating the input of the downstream decoding module can be: first obtain the predicted probability of the top CTC layer of the encoder, and then multiply it with the weight matrix of the pre-trained target end word embedding layer to obtain a representation containing the target end context In order to convert its length to the length of the target sequence, position encoding and a learnable weight are used to calculate the attention score matrix. The input of the CTC-enhanced decoding module is obtained by multiplying the attention score matrix and the representation. In this way, the target-side context information can be incorporated into the decoder input of the fine-tuning stage, thereby mapping its representation space from the source to the target, alleviating the modality difference problem.

[0026] In step 9), the specific steps of generating the translation result by the decoding module can be as follows: the downstream decoding module generates the translation result by simultaneously obtaining the input of the decoding module and the feature vector output by the encoding module; specifically, the present invention requires that each layer of the decoder performs prediction, and each layer only retains the highest confidence in the prediction. words, and then replace the remaining words with a special symbol [mask]; then, the word embedding predicted by this layer is concatenated with the hidden state of this layer, and an additional linear layer is used for dimensionality reduction to obtain the output of the decoder at the current layer; in this way, the lower layers of the decoder can provide the most confident prediction results for the higher layers, thereby promoting the prediction of complex words by the higher layers; in addition, the number of words to be predicted at each layer gradually increases linearly with the increase in the number of layers, and the top layer will predict the entire target sentence, so the decoder can predict the entire sentence from bottom to top and from easy to difficult.

[0027] In step 10), the downstream cross-modal non-autoregressive translation model loss is calculated, and during the fine-tuning experiment of the downstream task, the model is optimized by jointly minimizing the encoder loss and the decoder loss:

[0028]

[0029] in, represents the downstream cross-modal non-autoregressive translation model loss; represents the translation loss of a word, represents the loss of the downstream encoding module; α is a hyperparameter used to balance the two training losses.

[0030] The present invention provides a cross-modal non-autoregressive decoding system combined with a pre-trained language model, which includes: a target permutation sampling module, a pre-trained text encoding module, a pre-trained text decoding module, a pre-trained loss calculation module, a source feature extraction module, a downstream encoding module, a CTC enhancement module, a downstream decoding module, a downstream task loss calculation module, and an output module;

[0031] The target permutation sampling module is used to randomly select a source text and a corresponding target text from the pre-training corpus, and perform permutation transformation on the target text;

[0032] The pre-trained text encoding module is used to convert the source text into a series of latent vector representations;

[0033] The pre-trained text decoding module is used to convert the word embedding of the source text into the input of the non-autoregressive decoder through a copy mechanism, and input it into the text decoding module; the decoding module uses the output of the encoding module to generate a translation result;

[0034] The pre-training loss calculation module is used to calculate the cross entropy loss of each decoding process, and accumulate the losses of all steps to obtain the total loss to guide model optimization;

[0035] The source feature extraction module is used to extract features from source data to obtain a feature sequence containing source information. The extracted source feature vector is sent to the downstream encoding module.

[0036] The downstream encoding module is used to further encode the feature vector obtained by the source feature extraction module to obtain a higher level of abstract representation;

[0037] The CTC enhancement module is used to improve the model's ability to process sequence data. It combines the predicted probability of the CTC (Connectionist Temporal Classification) layer and the weight matrix of the target-side word embedding layer, calculates the attention score matrix through position encoding and learnable weights, and uses it to generate the input of the decoding module to integrate the target-side context information.

[0038] The downstream decoding module generates a translation result by simultaneously obtaining the input of the decoding module and the feature vector output of the encoding module. The parallel processing method is a typical non-autoregressive method.

[0039] The downstream task loss calculation module uses the CTC loss function to calculate the loss of the encoding module and the cross entropy loss function to calculate the translation loss of the generated words in the decoding module. These losses are used to guide the optimization of the entire downstream model.

[0040] The output module is used to output the translation results obtained by the downstream cross-modal non-autoregressive translation model, such as spoken sentences.

[0041] The advantages of the present invention are:

[0042] 1. This paper introduces the idea of ​​Random Drop into the pre-training task of non-autoregressive language models, effectively modeling the dependencies between target-side words.

[0043] 2. This paper introduces the idea of ​​CTC into the downstream cross-modal non-autoregressive translation task. By incorporating text modality information into the decoder input, it effectively alleviates the modality difference problem of the decoder input in the pre-training and fine-tuning stages.

[0044] 3. The performance of the cross-modal non-autoregressive decoding method combined with a pre-trained language model proposed in this invention is consistently better than all strong baselines, and can be applied to different types of cross-modal translation tasks, with broad application prospects and good application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A framework diagram of a cross-modal non-autoregressive decoding system combined with a pre-trained language model.

[0046] Figure 2 Flowchart of a cross-modal non-autoregressive decoding method combined with a pre-trained language model.

[0047] Figure 3 It is a cross-modal non-autoregressive translation system combined with a pre-trained language model. DETAILED DESCRIPTION

[0048] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific embodiments. The following describes the invention of the present invention in detail, and examples of the invention are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be understood as limiting the present invention.

[0049] To address the two issues with the prior art, this invention, inspired by Random Drop Training, proposes that during pre-training, each decoder layer can predict different parts of the entire target sequence, while allowing the higher layers of the decoder to perceive the predictions of the lower layers, thereby learning the inter-layer dependencies of the target vocabulary. The specific vocabulary predicted by each layer can be determined by random permutation and sampling. In this way, the decoder generates the entire sequence from the bottom up in a different order in each training iteration, thereby learning diverse dependencies and alleviating the first issue.

[0050] To mitigate the modality discrepancy problem in decoder input during pre-training and fine-tuning, this paper is inspired by speech translation and automatic speech recognition models based on CTC (Connectionist Temporal Classification). These methods typically add a CTC prediction layer on top of the encoder and use the target text as a supervisory signal, thereby obtaining a rough translation of the target text. This paper considers that incorporating this rough translation into the decoder input can, on the one hand, map the decoder input from the source to the target, thereby alleviating the modality discrepancy problem; on the other hand, it can provide the decoder with certain target-side information, making the model easier to learn.

[0051] Based on the above considerations, the present invention first uses a random permutation progressive prediction pre-training task to learn diverse dependencies in a non-autoregressive decoder on large-scale pre-trained text pairs; then uses a CTC-enhanced soft copy method to incorporate target-side information into the decoder of a downstream cross-modal non-autoregressive model to alleviate the modality difference problem.

[0052] By introducing the ideas of Random Drop and CTC, this paper first uses a random permutation progressive prediction pre-training task to model the diverse dependencies in a non-autoregressive decoder on large-scale pre-trained text pairs. Then, a CTC-enhanced soft copy method is used to incorporate target-side information into the decoder of the downstream cross-modal non-autoregressive model to alleviate the modality difference problem.

[0053] like Figure 1 As shown, an embodiment of the present invention provides a cross-modal non-autoregressive decoding system combined with a pre-trained language model, the system sequentially includes a target arrangement sampling module, a pre-trained text encoding module, a pre-trained text decoding module, a pre-training loss calculation module, a source feature extraction module, a downstream encoding module, a CTC enhancement module, a downstream decoding module, a downstream task loss calculation module, and an output module;

[0054] Target permutation sampling module: used to randomly select a source text and the corresponding target text from the pre-training corpus, and perform permutation transformation on the target text;

[0055] Pre-trained text encoding module: used to convert source text into a series of latent vector representations;

[0056] Pre-trained text decoding module: This module is used to convert the word embeddings of the source text into the input of the non-autoregressive decoder through a copy mechanism, and then input it into the text decoding module. The decoding module uses the output of the encoding module to generate the translation result.

[0057] Pre-training loss calculation module: used to calculate the cross entropy loss of each decoding step, accumulate the losses of all steps to obtain the total loss, and use it to guide model optimization;

[0058] Source feature extraction module: used to extract features from source data to obtain a feature sequence containing source information. The extracted source feature vector is sent to the downstream encoding module.

[0059] Downstream encoding module: used to further encode the feature vectors obtained by the source feature extraction module to obtain a higher level of abstract representation;

[0060] CTC Enhancement Module: This module is used to improve the model's ability to process sequence data. It combines the predicted probabilities of the CTC (Connectionist Temporal Classification) layer and the weight matrix of the target-side word embedding layer, calculates the attention score matrix through positional encoding and learnable weights, and uses it to generate the input of the decoding module to integrate the target-side context information.

[0061] Downstream decoding module: The downstream decoding module generates the translation result by simultaneously obtaining the input of the decoding module and the feature vector output of the encoding module. The parallel processing method is a typical non-autoregressive method;

[0062] Downstream task loss calculation module: In the downstream loss calculation module, the CTC loss function is used to calculate the loss of the encoding module, and the cross-entropy loss function is used to calculate the translation loss of the generated words in the decoding module. These losses are used to guide the optimization of the entire downstream model;

[0063] Output module: The output module outputs the translation results obtained by the downstream cross-modal non-autoregressive translation model, such as spoken sentences.

[0064] like Figure 2 , an embodiment of the present invention provides a cross-modal non-autoregressive decoding method combined with a pre-trained language model, comprising the following steps:

[0065] 1) Sample pre-training samples and target permutations: Sample text data pairs used for pre-training and a permutation of target sentences, with each element using the subscript of the target word in the target sequence. Specifically:

[0066] First, sample a piece of data from the corpus used for pre-training , and use the target sequence sampling module to sample a permutation of the target sequence ,Notice Each element in is the subscript of the target word in the target sequence rather than the word itself. For example, {1, 2, 4, 0, 3} can be a permutation of a target sequence of length 5.

[0067] 2) Input to the pre-trained text encoding module: The pre-trained text encoding module using the pre-trained model will The code is converted into a latent vector and then into a hidden state. The pre-trained text encoding module consists of a word embedding layer, a self-attention layer, a residual connection, a layer normalization layer, and a feedforward neural network layer stacked multiple times.

[0068] 3) Input to the pre-trained text decoding module: First use the copy mechanism to The word embedding is converted into the input of the non-autoregressive decoder and input into the pre-trained text decoding module. After receiving the output vector of the text encoding module, the decoding module generates the translation result through the feature vector of the encoding module.

[0069] Specifically, consider the total The first layer of the pre-trained text decoding module Layer, first get the hidden vector of this layer , and pass it to the output layer to obtain the prediction of this layer These operations are expressed as:

[0070]

[0071]

[0072] in, Represents the weight matrix of the linear output layer.

[0073] At the same time, only retaining The subscript belongs to Before The words in the elements are then replaced with the special symbol [mask] to get the updated .

[0074] Finally, splicing Word embedding and , and use additional linear layers for dimensionality reduction to obtain the decoder Output of the layer :

[0075]

[0076] in Represents the weight matrix of the linear layer. The output of the layer is also the In this way, the high-level decoder can receive the predictions of the low-level decoder, thereby predicting more words, and the entire sentence will be gradually predicted by the decoder from bottom to top.

[0077] 4) Calculate the loss in the pre-training stage: In the loss calculation module of the pre-training stage, the cross entropy loss function is used to calculate the loss of the non-autoregressive decoding module. Since the decoder makes predictions at each layer, the loss function in the pre-training stage is It is the sum of the cross entropy losses of each layer of the decoder. Specifically:

[0078] In the loss calculation module of the pre-training stage, the cross entropy loss is used to calculate the translation loss of the decoding module. Since the decoder makes predictions at each layer, the loss function of the pre-training stage is This is the sum of the cross entropy losses of each layer of the decoder:

[0079]

[0080] in Indicates the decoder Layer The probability distribution of words, and . Indicates the number of decoder layers; Indicates the number of words predicted in the i-th layer, , Indicates the total number of words, represents the t-th target word.

[0081] 5) Initialize the parameters of the downstream cross-modal translation model: After training the non-autoregressive pre-trained language model, use the pre-trained model and its parameters to initialize the encoder and decoder parameters of the downstream cross-modal non-autoregressive translation model.

[0082] 6) Input source data: Prepare source data (such as source language text and corresponding multimodal data such as images or audio). The feature extraction module extracts features from each frame of the source data (such as sign language or speech) to obtain features containing source information.

[0083] 7) Source-side feature extraction: The feature extraction module extracts features from each frame of the source non-text data to obtain its frame-level features and obtain a feature sequence containing source-side information.

[0084] 8) Input to the downstream encoding module: The frame-level features extracted in step 7) are sent to the downstream encoding module, encoded by the encoder to obtain the hidden representation, and the target-side prediction probability after the CTC layer is obtained. Specifically:

[0085] The extracted source feature vector is sent to the downstream encoding module, where it is transformed into a hidden representation through the encoder-decoder. CTC (Connectionist Temporal Classification) is used as the encoding module's loss function, and the supervisory signal is the target text. This ensures that the encoder provides a probability distribution containing target text information during training. The encoding module incorporates an attention mechanism, which overcomes the limitations of input sequence length and enables the model to focus on important information and fully absorb it.

[0086] 9) Input to the CTC enhancement module: Combining the target-side prediction probability of the CTC layer and the weight matrix of the target-side word embedding layer, the attention score matrix is ​​calculated through position encoding and learnable weights, and used to generate the input of the decoding module to integrate the target-side context information.

[0087] In the CTC enhancement module, first obtain the predicted probability of the top CTC layer of the encoder , and then the weight matrix of the pre-trained target end word embedding layer and Multiply to obtain a representation containing the target side context However, at this time Still the length of the source sequence, in order to convert its length into the length of the target sequence, the present invention uses position encoding and a learnable weight To calculate the attention score matrix ,Finally, the decoding module input By the attention matrix and Multiply them together to get .

[0088] These operations are expressed as:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] in Represents the weight matrix of the CTC prediction layer. In this way, the target-side context information can be incorporated into the decoder input during fine-tuning, thereby mapping its representation space from the source to the target, alleviating the modality difference problem.

[0095] 10) Input to the downstream decoding module: The downstream decoding module generates the translation result by simultaneously obtaining the input of the decoding module and the feature vector output by the encoding module. Specifically:

[0096] The downstream decoding module obtains the input of the decoding module at the same time And the feature vector output by the encoding module to generate the translation result.

[0097] The decoding module here adopts the curriculum-based non-autoregressive decoder (Curriculum-basedNon-autoregressive Decoder, CND). Specifically, considering the total The first layer of the non-autoregressive decoder Layer, first get the hidden vector of this layer , and pass it to the output layer to obtain the prediction of this layer And the corresponding confidence These operations are expressed as:

[0098]

[0099]

[0100]

[0101] in, Represents the weight matrix of the linear output layer. It is worth noting that the weight matrix of the prediction layer is shared among all layers of the decoder.

[0102] At the same time, only retaining The most confident elements, and then replace the remaining words with the special symbol [mask] to get the updated Finally, splicing Word embedding and , and use additional linear layers for dimensionality reduction to obtain the decoder Output of the layer :

[0103]

[0104] In this way, the lower layers of the decoder can provide the most confident predictions to the higher layers, thereby facilitating the higher layers’ predictions of complex words.

[0105] In addition, the number of words to be predicted at each layer increases linearly with the number of layers, and the top layer will predict the entire target sentence, so the decoder can predict the entire sentence from bottom to top and from easy to difficult.

[0106] 11) Calculate the loss of the downstream fine-tuning stage: In the downstream loss calculation module, use the CTC loss function to calculate the loss of the downstream encoding module , the cross entropy loss function is used to calculate the translation loss of the generated words in the downstream decoding module .

[0107] During fine-tuning experiments on downstream tasks, the model is optimized by jointly minimizing the encoder loss and the decoder loss.

[0108]

[0109] Here, α is a hyperparameter used to balance the two training losses.

[0110] 12) End: Finally, the output module outputs the spoken sentence obtained by the downstream cross-modal non-autoregressive translation model.

[0111] Our proposed method consistently outperforms all strong baselines and achieves competitive results with an autoregressive cross-modal translation model equipped with an autoregressive pre-trained language model.

[0112] The overall method flow is as follows:

[0113] Step 1: Sample pre-training training samples and target permutations.

[0114] Sample a data pair from the corpus used for pre-training and sample the target sentence.

[0115] Step 2: Encode the source text into a latent vector.

[0116] The text encoding module using the pre-trained model will Encoded as a latent vector.

[0117] Step 3: Target-side text decoding.

[0118] First, a copy mechanism is used to obtain the input of the non-autoregressive decoder and input it into the text decoding module. The text decoding module generates the translation result based on the feature vector of the encoding module.

[0119] Step 4: Pre-training loss calculation.

[0120] The translation loss of the non-autoregressive decoding module is calculated using cross-entropy loss during the pre-training stage.

[0121] Step 5: Initialize the parameters of the downstream cross-modal non-autoregressive translation model.

[0122] The trained non-autoregressive pre-trained language model is used to initialize the encoder and decoder parameters of the downstream cross-modal non-autoregressive translation model.

[0123] Step 6: Source-side feature extraction.

[0124] The feature extraction module is used to extract features from each frame of source data to obtain a feature sequence containing source information.

[0125] Step 7: Source-side feature encoding.

[0126] The source feature sequence is sent to the downstream encoding module to obtain the hidden representation.

[0127] Step 8: Downstream decoding module input generation.

[0128] Get the predicted probability of the top CTC layer of the encoder and multiply it with the weight matrix of the pre-trained target-side word embedding layer to obtain a representation containing the target-side context.

[0129] Step 9: The decoding module generates the translation result.

[0130] The downstream decoding module generates the translation result by simultaneously obtaining the feature vectors of the decoding module input and the encoding module output.

[0131] Step 10: Downstream cross-modal non-autoregressive translation model loss calculation.

[0132] In the downstream loss calculation module, the CTC loss function is used to calculate the loss of the downstream encoding module, and the cross entropy loss function is used to calculate the translation loss of the generated words in the downstream decoding module. At the same time, a hyperparameter is used to balance the two losses.

[0133] Step 11: Output downstream cross-modal non-autoregressive translation results.

[0134] Finally, the spoken sentence obtained by the downstream cross-modal non-autoregressive translation model is output through the output module.

[0135] The translation system primarily consists of an encoding module and a decoding module. This invention addresses an improved decoding method that needs to be integrated with the overall translation system. The core of the invention lies in constructing and optimizing the decoding framework, while the translation system provides an application vehicle for the decoding method. This structural design ensures the integrity of the entire system.

[0136] An embodiment of the present invention provides a cross-modal non-autoregressive translation system combined with a pre-trained language model. Figure 3 The specific instructions are as follows:

[0137] The system comprises, in sequence:

[0138] Video input unit: used for cross-modal translation data selection modeling, inputting source data into the system in the form of frames; the video input unit serves as the input end of the entire system;

[0139] CTC Enhancement Module: This module uses the probability distribution output by the encoder's CTC layer and the pre-trained decoder-side text embedding layer to obtain the CTC-enhanced decoder module input, which contains the target sequence information. The CTC Enhancement Module enhances the decoder module's ability to better generate the target sequence.

[0140] Video feature extraction module: used to extract features from each frame of source data to obtain sequence features containing source information.

[0141] Downstream Encoding Module: The extracted visual feature vector is sent to the downstream encoding module to obtain the hidden representation. The encoder parameters are initialized from the encoder parameters of the non-autoregressive pre-trained language model.

[0142] Downstream decoding module: After receiving the output vector from the downstream encoding module, the downstream decoding module generates the translation result based on the encoding module's feature vector. This decoder uses a non-autoregressive decoder based on curriculum learning. The bottom layer predicts simple words, and the upper layers predict more complex words based on the bottom layer's output, thus achieving a bottom-up, progressively more complex prediction of the entire sequence. The decoder parameters are initialized from the encoder parameters in the non-autoregressive pre-trained language model.

[0143] Storage unit: used to store cross-modal translation data.

[0144] Loss calculation module: used to calculate the loss of the system; in the loss calculation module, the CTC loss function is used to calculate the loss of the downstream encoding module, and the cross entropy loss function is used to calculate the translation loss of the generated words in the downstream decoding module.

[0145] Text output unit: outputs the spoken sentence output obtained through the cross-modal translation model.

[0146] The design of the cross-modal non-autoregressive translation system combined with a pre-trained language model fully considers the characteristics of video data and the advantages of the pre-trained language model. Through the above-mentioned module design and function allocation, a complete translation process from video input to text output can be realized.

[0147] In summary, the present invention proposes a cross-modal non-autoregressive decoding method combined with a pre-trained language model, introduces the ideas of Random Drop and CTC, designs a non-autoregressive pre-training task and a CTC-enhanced soft copy method, and respectively solves two problems existing in the combination of non-autoregressive pre-trained language model and non-autoregressive cross-modal translation model. A cross-modal non-autoregressive decoding algorithm combined with a pre-trained language model provided in one of the invention contents of the present invention includes: first sampling data pairs for pre-training, and sampling a permutation of the target sentence; then using a text encoder to convert the pre-trained source text into a hidden state, and using a decoding module to predict the entire sequence bottom-up according to the sampled permutation, and then training the non-autoregressive pre-trained language model, in the pre-training loss calculation module, using the cross-entropy loss function to calculate the loss of the non-autoregressive decoding module; then using the pre-trained model to initialize the parameters of the downstream cross-modal non-autoregressive translation model encoder and decoder; after the parameters are initialized, the source data input module performs translation data selection modeling, and inputs the source data into the model in the form of frames. ; Then, the source-side feature extraction module extracts features from each frame of the source-side data to obtain sequence features containing source-side information; the extracted feature vector is sent to the downstream encoding module, and the hidden representation is obtained through the encoder, and the probability predicted by the encoder CTC layer and the pre-trained text-side word embedding are used to generate the input of the downstream decoding module; after receiving the output vector of the downstream encoding module and its input, the downstream decoding module generates the translation result through the feature vector of the encoding module; in the downstream task loss calculation module, the CTC loss function is used to calculate the loss of the downstream encoding module, and the cross-entropy loss function is used to calculate the translation loss of the generated words in the downstream decoding module; finally, the spoken sentence output obtained by the cross-modal translation model of the source-side data is output through the output module. The method proposed in the present invention consistently outperforms all strong baselines, and achieves competitive results with the cross-modal autoregressive translation model equipped with an autoregressive pre-trained language model, as shown in Table 1.

[0148] Table 1

[0149]

[0150] As shown in Table 1, our method (ROPre) was experimentally validated on the classic cross-modal translation task of sign language translation. The effectiveness of our method was evaluated using ROUGE and BLEU4 metrics, as well as decoding speed, on the PHOENIX-2014T German Sign Language data. First, using an autoregressive (AR) model combined with an ARPLM as a speed baseline, our method ran 7.92 times faster and outperformed both evaluation metrics. Compared to two iterative NAR models (CMLM and CMLMC), although their speed was 2.47 times faster than the baseline, they lagged behind our method in both speed and translation quality. Second, compared to three purely non-autoregressive (NAR) models (GLEAT, DSLP, and CND), our method, while lagging behind in speed, outperformed these baseline methods in terms of quality.

[0151] Table 2

[0152]

[0153] As shown in Table 2, the effectiveness of our proposed method (ROPre) was also evaluated on the CSL-Daily Chinese Sign Language data using ROUGE, BLEU4, and decoding speed. First, compared to two iterative NAR models (CMLM and CMLMC), our method achieved improvements of 2.33 and 1.79 BLEU4 points, respectively, and significantly lower decoding latency on the CSL-Daily test set. Second, compared to three fully NAR SLT models (GLEAT, DSLP, and CND), our method achieved an average improvement of 2.34 BLEU4 points and comparable decoding speed. Furthermore, our method slightly outperformed CND with a mutual learning framework. Third, compared to the AR base model, our method achieved a performance improvement of 0.92 BLEU4 points while accelerating by approximately 8.02 times. Our method even surpassed the AR SLT model with AR-PLM by 0.94 BLEU4 points, ultimately demonstrating the effectiveness and efficiency of our method.

[0154] The above embodiments are only preferred embodiments of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A cross-modal non-autoregressive decoding method combined with a pre-trained language model, characterized by The following steps are involved: 1) Sampling pre-training training samples and target permutations: Sampling a data pair from the corpus used for pre-training and a permutation of the target sentence ,in, Each element in is the subscript of the target word in the target sequence; 2) Encode the source text into a latent vector: Use the text encoding module of the pre-trained model to encode the source text into a latent vector. Encoded as a latent vector; 3) Target-side text decoding: The pre-trained source text word embeddings are converted into input for a non-autoregressive decoder through a copying mechanism and then fed into the text decoding module. After receiving the output vector of the pre-trained text encoding module, the text decoding module generates the translation result based on the feature vector of the encoding module. 4) Pre-training loss calculation: Use cross-entropy loss to calculate the translation loss of each decoder layer, and add up the losses of all layers to get the total loss; 5) Initialize the parameters of the downstream cross-modal non-autoregressive translation model: After the pre-trained language model is trained, its parameters can be used to initialize the encoder and decoder parameters of the downstream cross-modal non-autoregressive translation model; 6) Source feature extraction: The feature extraction module extracts features from each frame of source data to obtain a feature sequence containing source information. 7) Source feature encoding: The source feature vector extracted in step 6) is sent to the downstream encoding module to obtain a hidden representation; 8) Downstream decoding module input generation: Combining the predicted probabilities of the CTC layer and the weight matrix of the target-side word embedding layer, the attention score matrix is ​​calculated using positional encoding and learnable weights, and used to generate the input of the decoding module to incorporate the target-side context information; 9) The decoding module generates the translation result: The downstream decoding module generates the translation result by simultaneously obtaining the feature vectors of the decoding module input and the encoding module output; 10) Calculate the loss of the downstream cross-modal non-autoregressive translation model: In the downstream loss calculation module, the CTC loss function is used to calculate the loss of the downstream encoding module, and the cross-entropy loss function is used to calculate the translation loss of the generated words in the downstream decoding module; 11) Downstream cross-modal non-autoregressive translation result output: The output module outputs the spoken sentence obtained by the downstream cross-modal non-autoregressive translation model.

2. The cross-modal non-autoregressive decoding method in combination with a pre-trained language model as claimed in claim 1, characterized in that In step 3), the specific steps of the target end text decoding are: Consider the total The first layer of the pre-trained text decoding module Layer, first get the hidden vector of this layer , and pass it to the output layer to obtain the prediction of this layer ; These operations are expressed as: in, Represents the weight matrix of the linear output layer; At the same time, only retaining The subscript belongs to Before The words in the elements are then replaced with the special symbol [mask] to get the updated ; Finally, splicing Word embedding and , and use additional linear layers for dimensionality reduction to obtain the decoder Output of the layer : in Represents the weight matrix of the linear layer; since the decoder is The output of the layer is also the In this way, the high-level decoder can receive the predictions of the low-level decoder, thereby predicting more words, and the entire sentence will be gradually predicted by the decoder from bottom to top.

3. The cross-modal non-autoregressive decoding method in combination with a pre-trained language model as claimed in claim 1, characterized in that In step 4), the specific steps of the loss calculation in the pre-training stage are as follows: in the loss calculation module of the pre-training stage, the translation loss of the non-autoregressive decoding module is calculated using the cross entropy loss; since the decoder makes predictions at each layer, the loss function in the pre-training stage is This is the sum of the cross entropy losses of each layer of the decoder: in, Indicates the decoder Layer The probability distribution of words; Indicates the number of decoder layers; Indicates the number of words predicted in the i-th layer, represents the t-th target word.

4. The cross-modal non-autoregressive decoding method in combination with a pre-trained language model as claimed in claim 1, characterized in that In step 8), the specific steps of generating the input of the downstream decoding module are as follows: first, obtain the predicted probability of the top CTC layer of the encoder , and then the weight matrix of the pre-trained target end word embedding layer and Multiply to obtain a representation containing the target side context However, at this time It is still the length of the source sequence. In order to convert its length to the length of the target sequence, position encoding is used and a learnable weight To calculate the attention score matrix ,Finally, the decoding module input By the attention matrix and Multiply to get; These operations are expressed as: in, Represents the weight matrix of the CTC prediction layer; in this way, the target-side context information can be incorporated into the decoder input of the fine-tuning stage, thereby mapping its representation space from the source side to the target side and alleviating the modality difference problem.

5. The cross-modal non-autoregressive decoding method in combination with a pre-trained language model as claimed in claim 1, characterized in that In step 9), the specific steps of the decoding module generating the translation result are as follows: the downstream decoding module generates the translation result by simultaneously obtaining the input of the decoding module and the feature vector output by the encoding module; specifically, each layer of the decoder makes a prediction, and each layer only retains the top prediction with the highest confidence. words, and then replace the remaining words with the special symbol [mask]; Then, the word embedding predicted by this layer is concatenated with the hidden state of this layer, and an additional linear layer is used for dimensionality reduction to obtain the output of the decoder at the current layer. In this way, the lower layers of the decoder can provide the most confident predictions to the higher layers, thereby facilitating the prediction of complex words by the higher layers. In addition, the number of words to be predicted at each layer increases linearly with the number of layers, and the top layer will predict the entire target sentence, so the decoder can predict the entire sentence from bottom to top and from easy to difficult.

6. The cross-modal non-autoregressive decoding method in combination with a pre-trained language model according to claim 5, characterized in that: The downstream decoding module adopts a non-autoregressive decoder CND based on curriculum learning. Specifically, considering a total of The first layer of the non-autoregressive decoder Layer, first get the hidden vector of this layer , and pass it to the output layer to obtain the prediction of this layer And the corresponding confidence ; These operations are expressed as: in, Represents the weight matrix of the linear output layer; the weight matrix of the prediction layer is shared among all layers of the decoder; At the same time, only retaining The most confident elements, and then replace the remaining words with the special symbol [mask] to get the updated ; Finally, splicing Word embedding and , and use additional linear layers for dimensionality reduction to obtain the decoder Output of the layer : In this way, the lower layers of the decoder can provide the most confident predictions to the higher layers, thereby promoting the prediction of complex words by the higher layers. In addition, the number of words to be predicted at each layer increases linearly with the number of layers, and the top layer will predict the entire target sentence, so the decoder can predict the entire sentence from bottom to top and from easy to difficult.

7. The cross-modal non-autoregressive decoding method in combination with a pre-trained language model as claimed in claim 1, characterized in that In step 10), the downstream cross-modal non-autoregressive translation model loss is calculated, and during the fine-tuning experiment of the downstream task, the model is optimized by jointly minimizing the encoder loss and the decoder loss: in, represents the downstream cross-modal non-autoregressive translation model loss; represents the translation loss of a word, represents the loss of the downstream encoding module; α is a hyperparameter used to balance the two training losses.

8. A cross-modal non-autoregressive decoding system combined with a pre-trained language model, characterized by Including in order: Target permutation sampling module, pre-trained text encoding module, pre-trained text decoding module, pre-training loss calculation module, source feature extraction module, downstream encoding module, CTC enhancement module, downstream decoding module, downstream task loss calculation module, output module; The target permutation sampling module is used to randomly select a source text and a corresponding target text from the pre-training corpus, and perform permutation transformation on the target text; The pre-trained text encoding module is used to convert the source text into a series of latent vector representations; The pre-trained text decoding module is used to convert the word embedding of the source text into the input of the non-autoregressive decoder through a copy mechanism, and input it into the text decoding module; the decoding module uses the output of the encoding module to generate a translation result; The pre-training loss calculation module is used to calculate the cross entropy loss of each decoding process, and accumulate the losses of all steps to obtain the total loss to guide model optimization; The source feature extraction module is used to extract features from source data to obtain a feature sequence containing source information. The extracted source feature vector is sent to the downstream encoding module. The downstream encoding module is used to further encode the feature vector obtained by the source feature extraction module to obtain a higher level of abstract representation; The CTC enhancement module is used to improve the model's ability to process sequence data. It combines the predicted probability of the CTC layer and the weight matrix of the target-side word embedding layer, calculates the attention score matrix through position encoding and learnable weights, and uses it to generate the input of the decoding module to integrate the target-side context information. The downstream decoding module generates a translation result by simultaneously obtaining the input of the decoding module and the feature vector output of the encoding module. The parallel processing method is a typical non-autoregressive method. The downstream task loss calculation module uses the CTC loss function to calculate the loss of the encoding module and the cross entropy loss function to calculate the translation loss of the generated words in the decoding module. These losses are used to guide the optimization of the entire downstream model. The output module is used to output the translation result obtained by the downstream cross-modal non-autoregressive translation model.

Images