Domain multi-modal neural machine translation method based on asymmetric twin distillation coding

Abstract

The invention provides a domain multi-mode neural machine translation method based on asymmetric twin distillation coding. Sparse domain visual details and texts are integrated with superiority of a mode-independent domain hub. The method comprises the steps of firstly obtaining a source sentence and an image most relevant to the source sentence; then, constructing an asymmetric twin text encoder, respectively capturing text and domain-oriented multi-modal semantics, reducing a modal gap between image-text data pairs, and meanwhile, integrating domain-oriented visual details into the text; using a cross-modal domain mask mechanism to obtain domain specific knowledge; and finally, gradually distilling visual details of a specific field into a text through a field adaptive distillation strategy based on field probability sampling. According to the method, the problem of machine translation in a specific field is solved, the difference between modals is drawn, and the problem of cross-modal semantic sparse mapping is solved to promote the machine translation performance of a field text.

Description

Technical Field

[0001] The present invention provides a domain multi-modal neural machine translation method based on asymmetric twin distillation coding, which uses visual information to enhance the performance of domain machine translation and belongs to the technical field of natural language processing. Background Art

[0002] Domain-specific multi-modal neural machine translation aims to translate domain-specific sentences from a source language into a target language by integrating multi-modal data (such as visual input). In the past few decades, these specialized machine translation tasks have attracted extensive research attention in key application areas (such as cross-border e-commerce documents, legal contract localization, and technical travel guide adaptation).

[0003] Compared with general-domain multi-modal neural machine translation, domain-specific translation faces dual challenges: 1) Term density - Domain-specific sentences usually contain a large number of domain terms, which are crucial for semantic nuances; 2) Low frequency - These terms appear less frequently in the training corpus. Therefore, the core challenge of domain-specific multi-modal neural machine translation lies in both understanding the contextual semantics of these key terms and generating translations that are linguistically accurate and conform to domain norms within the target language framework.

[0004] Although there have been some attempts at domain-specific multi-modal neural machine translation, existing methods mainly adopt general-domain multi-modal fusion paradigms and systematically ignore the semantic anchoring role of low-frequency domain terms. This methodological gap leads to the persistent problem of suboptimal term translation - a key obstacle stemming from the sparsity of term distribution - which fundamentally limits the operational effectiveness achieved by domain-adaptive machine translation. Summary of the Invention

[0005] The present invention proposes a domain multi-modal neural machine translation method based on asymmetric twin distillation coding to solve the technical problems in the above technologies, and systematically solves the problem of quasi-equivalent term translation through a two-branch coding architecture. The framework uses domain virtual hubs generated by multi-focus sparse distillation to effectively bridge the inherent asymmetric sparsity between spatially localized visual features and term-dense text semantics.

[0006] The technical solution of the present invention is: a domain multi-modal neural machine translation method based on asymmetric twin distillation coding, and the specific steps of the domain multi-modal neural machine translation method based on asymmetric twin distillation coding are as follows:

[0007] Step1. Use CLIP to obtain the image most relevant to the source language sentence and perform BPE processing on the source language sentence;

[0008] Step2. Obtain text and image features through Word Embedding and Resnet-101 respectively;

[0009] Step3. Input the text features into the constructed asymmetric siamese encoder for full representation;

[0010] Step4. Adopt a text-image interaction masking mechanism to mask the visual virtual representation and enhance the text-image representation ability of the model;

[0011] Step5. Perform domain feature probability sampling on the visual virtual features, and then perform cross-modal distillation on the samples and Resnet-101;

[0012] Step6. Perform hierarchical gating operations on the text representation output by the text encoder and the visual virtual features generated by the visual virtual encoder.

[0013] Step7. Feed the text-image fusion multi-modal features obtained in Step6 into the cross-lingual attention mechanism of the decoder, and the generation of the target sentence follows the traditional Transformer decoder framework.

[0014] The specific steps of the said Step1 are as follows:

[0015] Step1.1. Use a crawler to obtain a multi-modal dataset for a specific domain;

[0016] Step1.2. Use the CLIP model to obtain the image most relevant to the source language text;

[0017] Step1.3. Perform BPE (byte pair encoding) preprocessing on the source language text, and then divide the dataset into a training set, a validation set, and a test set.

[0018] The specific steps of the said Step2 are as follows:

[0019] Step2.1. Use Resnet-101 to encode the image to obtain the corresponding image features;

[0020] Step2.2. Use Word Embedding to encode the text, and then use the text encoder to obtain the text representation;

[0021] The specific steps of the said Step3 are as follows:

[0022] Step3.1. Construct an asymmetric siamese encoder based on the open-source framework Fairseq;

[0023] Step3.2. Input the text features into the text encoder in the asymmetric Siamese encoder for characterization;

[0024] Step3.3. Input the output of the text encoder into the visual virtual encoder of the same layer for characterization.

[0025] The specific steps of Step3.1 are as follows:

[0026] Step3.1.1. Add a visual virtual encoder on the basis of the Transformer of the open-source framework Fairseq;

[0027] Step3.1.2. The input of the visual encoder is the output of the text encoder of the same level, and the output is the visual virtual features:

[0028]

[0029] Among them, represents the visual virtual encoder with a self-attention module and a feed-forward neural network module at the l-th layer.

[0030] The specific steps of Step4 are as follows:

[0031] Step4.1. Perform text-vision and vision-text interactions on the image features obtained in Step2 and the characterized text features obtained in Step3;

[0032] Step4.2. Interact the interaction results between text-vision and vision-text again;

[0033] Step4.3. Use the result after interaction as the mask of the visual virtual encoder.

[0034] The specific steps of Step4.1 are as follows:

[0035] Step4.1.1. Perform the following operations on the fully characterized text representation and visual representation:

[0036]

[0037] Among them, represents the correlation between each token level in the source language sentence and 49 image regions, represents the correlation between 49 image regions and each token level in the source language sentence, represents the text representation of the (l - 1)-th layer, F v represents the visual representation, softmax represents the activation function in the neural network, and d is the embedding dimension of the image.

[0038] The specific steps of Step 4.2 are as follows:

[0039] Step 4.2.1. Perform the following operations on the result obtained in Step 4.1:

[0040]

[0041] Step 4.2.2. Then, binarize the graphic-text interaction matrix C l as follows:

[0042]

[0043] C l C(i,j) represents the element in the i-th row and j-th column of the matrix C, and M l represents the masked matrix after binarization of C l and l represents the binarization threshold.

[0044] The specific steps of Step 4.3 are as follows:

[0045] Step 4.3.1. The method for constructing the visual virtual encoder module is as follows:

[0046]

[0047] where represents the multi-head attention mechanism, and c l represents the binary matrix for modality interaction;

[0048] Step 4.3.2. Perform a mask filling operation on the multi-head attention mechanism of the visual virtual encoder.

[0049] The specific steps of Step 5 are as follows:

[0050] Step 5.1. Perform average pooling on the visual virtual features, and then perform a Repeat operation on the pooling dimension:

[0051]

[0052] where represents the virtual visual representation after pooling and Repeat operations, Pooling represents the pooling operation, and Repeat represents the replication operation, whose function is to restore the global virtual representation after pooling to the original dimension.

[0053] Step 5.2. For the result obtained in Step 5.1 ​Concatenate them, then pass through a linear layer, and finally perform a softmax operation:

[0054]

[0055] Among them, represents the finally obtained probability sampling matrix, Concat represents the concatenation operation, which concatenates the virtual visual representation and the representation after pooling and Repeat on the last dimension. FFN represents the feed-forward neural network, SoftMax represents the activation function, and the finally obtained domain probability matrix

[0056] Step5.3. Use the domain probability sampling matrix obtained in Step5.2 to sample the virtual visual representation:

[0057]

[0058] Among them, represents the sampled virtual visual representation, and MulSam represents performing repeated probability sampling with replacement on according to the probability matrix for

[0059] Step5.4. Calculate the KL loss for the real visual representation F v and the sampled virtual visual representation :

[0060]

[0061] Among them, D KL represents calculating the KL loss value.

[0062] The specific steps of the said Step6 are as follows:

[0063] Step6.1. Perform gating calculation on the output of the visual virtual encoder of the l-th layer and the output of the text encoder:

[0064]

[0065] Among them, W t and W d represent trainable parameters, and Sigmoid represents the activation function;

[0066] Step6.2. Perform hierarchical gating operation on the obtained gating coefficient Ω l :

[0067]

[0068] Among them, the hyperparameter k satisfies 0 ≤ k ≤ L.

[0069] The beneficial effects and advantages of the present invention are as follows:

[0070] 1. The present invention proposes a novel asymmetric twin hierarchical network, which systematically solves the problem of quasi-equivalent term translation through a dual-branch encoding architecture. The framework uses domain virtual hubs generated by multi-focus sparse distillation to effectively bridge the inherent asymmetric sparsity between spatially localized visual features and term-dense text semantics;

[0071] 2. The present invention implements a hierarchical fusion strategy driven by domain virtual hubs, gradually injecting domain-specific semantics into text representations in an asymmetric dual-branch architecture. At each encoder layer, the domain virtual generation module first extracts modality-independent semantic anchors from visual inputs through multi-focus sparse distillation. Subsequently, a dual-branch process is performed: the text branch encodes term-dense content through Transformer layers, while the parallel virtual visual branch isolates sparsely refined domain features through domain probability masks;

[0072] 3. The method of the present invention is implemented based on the Fairseq open-source system, which is relatively stable and has strong text processing capabilities, providing a strong guarantee for multi-modal machine translation in specific domains;

[0073] 4. The method proposed by the present invention has achieved SOTA results on the test sets of multiple public datasets, and visualization experiments and case analyses have both proved the effectiveness and superiority of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flowchart in the present invention;

[0075] Figure 2 is a detailed flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0076] Embodiment 1: In view of problems such as semantic sparse mapping faced by multi-modal machine translation in specific domains, the present invention proposes a domain multi-modal neural machine translation method based on asymmetric twin distillation encoding, adopting a dual-branch asymmetric encoder architecture, including parallel virtual visual and text encoders, to simultaneously generate domain-specific virtual visual and text representations. The domain virtual visual information serves as a modality-independent hub, gradually guiding the learning of domain-consistent representations through hierarchical cross-modal interactions between encoder layers, improving the overall performance of translation.

[0077] Such as Figure 1 - Figure 2As shown in the figure, a domain multi-modal neural machine translation method based on asymmetric twin distillation coding, the method includes:

[0078] Step1. Crawl the open-source public multi-modal dataset through a web crawler, then use the pre-trained model CLIP to encode the text and images, and select the image most relevant to the text; perform BPE (byte pair encoding) operation on the text, and directly obtain the original data of the multi-modal machine translation data Multi30K in the publicly available download link of the Multi30K dataset, with a total of 30,000 items; download the images and texts of the Fashion-MMT dataset in the e-commerce field, and divide the obtained data into a training set, a validation set, and a test set.

[0079] Further, in the step Step1, it includes:

[0080] Step1.1. Use the crawler to obtain a multi-modal dataset for a specific domain;

[0081] Step1.2. Use the CLIP model to obtain the image most relevant to the source language text;

[0082] Step1.3. Perform BPE (byte pair encoding) preprocessing operation on the source language text.

[0083] Step2. Obtain text and image features through Word Embedding and Resnet-101 respectively;

[0084] Further, in the step Step2, it includes:

[0085] Step2.1. Use Resnet-101 to encode the image to obtain the corresponding image features;

[0086] Step2.2. Use Word Embedding to encode the text, and then use the text encoder to obtain the representation of the text;

[0087] Step3. Input the text features into the constructed asymmetric twin encoder for full representation;

[0088] Further, in the step Step3, it includes:

[0089] Step3.1. Construct an asymmetric twin encoder based on the open-source framework Fairseq;

[0090] Step3.2. Input the text features into the text encoder in the asymmetric twin encoder for representation;

[0091] Step 3.3. Input the output of the text encoder into the visual virtual encoder of the same layer for characterization.

[0092] The specific steps of the above Step 3.1 are as follows:

[0093] Step 3.1.1. Add a visual virtual encoder on the basis of the Transformer of the open-source framework Fairseq.

[0094] Step 3.1.2. The input of the visual encoder is the output of the text encoder of the same level, and the output is the visual virtual feature:

[0095]

[0096] where represents the visual virtual encoder with a self-attention module and a feed-forward neural network module at the l-th layer.

[0097] Step 4. Adopt a text-image interaction masking mechanism to mask the visual virtual representation and improve the text-image representation ability of the model.

[0098] Furthermore, in the above Step 4, it includes:

[0099] Step 4.1. Perform text-vision and vision-text interactions on the image features obtained in Step 2 and the characterized text features obtained in Step 3.

[0100] Step 4.2. Perform interactions on the interaction results between text-vision and vision-text again.

[0101] Step 4.3. Use the result after the interaction as the mask of the visual virtual encoder.

[0102] Furthermore, in the above Step 4.1, it includes:

[0103] Step 4.1.1. Perform the following operations on the fully characterized text representation and visual representation:

[0104]

[0105] where represents the relevance between each token level in the source language sentence and 49 image regions, represents the relevance between 49 image regions and each token level in the source language sentence, represents the text representation of the (l - 1)-th layer, F v represents the visual representation, softmax represents the activation function in the neural network, and d is the embedding dimension of the image.

[0106] Furthermore, in the step Step4.2, it includes:

[0107] Step4.2.1. Perform the following operations on the result obtained in Step4.1:

[0108]

[0109] Step4.2.2. Then, binarize the obtained graphic-text interaction matrix C l The operation is as follows:

[0110]

[0111] C l C(i,j) represents the element in the i-th row and j-th column of the matrix C, and M l represents the masked matrix after binarization of C l and represents the binarization threshold. l

[0112] Furthermore, in the step Step4.3, it includes:

[0113] Step4.3.1. The construction method of the visual virtual encoder module is as follows:

[0114]

[0115] Among them, represents the multi-head attention mechanism, and C l represents the binary matrix of modality interaction;

[0116] Step4.3.2. Perform a mask filling operation on the multi-head attention mechanism of the visual virtual encoder.

[0117] Step5. Perform domain feature probability sampling on the visual virtual features, and then perform cross-modal distillation on the samples and Resnet-101;

[0118] Furthermore, in the step Step5, it includes:

[0119] Step5.1. Perform average pooling operation on the visual virtual features, and then perform a Repeat operation on the pooling dimension:

[0120]

[0121] Among them, Denote the virtual visual representation after pooling and Repeat operations. Pooling represents the pooling operation, and Repeat represents the replication operation, whose function is to restore the global virtual representation after pooling to the original dimension.

[0122] Step5.2. Concatenate the result obtained in Step5.1, then pass through a linear layer, and finally perform a softmax operation:

[0123]

[0124] where denotes the finally obtained probability sampling matrix. Concat represents the concatenation operation, which concatenates the virtual visual representation and the representation after pooling and Repeat on the last dimension. FFN represents the feed-forward neural network, and SoftMax represents the activation function. The finally obtained domain probability matrix

[0125] Step5.3. Sample the virtual visual representation using the domain probability sampling matrix obtained in Step5.2:

[0126]

[0127] where denotes the sampled virtual visual representation, and MulSam represents performing repeated probability sampling with replacement on according to the probability matrix .

[0128] Step5.4. Calculate the KL loss between the real visual representation F v and the sampled virtual visual representation :

[0129]

[0130] where D KL denotes calculating the KL loss value.

[0131] Step6. Perform a hierarchical gating operation on the text representation output by the text encoder and the visual virtual features generated by the visual virtual encoder.

[0132] Furthermore, in the step Step6, it includes:

[0133] Step6.1. Perform a gating calculation on the output of the visual virtual encoder of the l-th layer and the output of the text encoder:

[0134]

[0135] Among them, W t and W d represent trainable parameters, and Sigmoid represents an activation function;

[0136] Step 6.2. Perform hierarchical gating operation on the obtained gating coefficient Ω l :

[0137]

[0138] Among them, the hyperparameter k satisfies 0 ≤ k ≤ L.

[0139] Step 7. Feed the multi-modal features of image-text fusion obtained in Step 6 into the cross-lingual attention mechanism of the decoder, and the generation of the target sentence follows the traditional Transformer decoder framework.

[0140] To prove the effectiveness of the method proposed in the present invention, a large number of experiments were carried out on the Fashion-MMT, EMMT, and Multi-30k datasets, and the details of these three datasets are shown in Table 1. In the experiments, in order to accurately evaluate the method performance, BLUE and METEOR were used as evaluation metrics, and these two metrics can objectively and accurately reflect the effect of machine translation. The experimental results are shown in Tables 2, 3, and 4.

[0141] Table 1 Details of Fashion-MMT and Multi-30k Datasets

[0142]

[0143] 1. Fashion-MMT

[0144] Table 2 Performance of the multi-modal machine translation model for specific domains on the Fashion-MMT test set

[0145]

[0146] 2. Multi-30k

[0147] Table 3 Comparison between the model corresponding to the method of the present invention and the existing SOTA model in the English-German task

[0148]

[0149] The results in both Table 2 and Table 3 show that the domain multi-modal neural machine translation method based on asymmetric twin distillation coding of the present invention achieves SOTA results on both the Fashion-MMT and Multi-30k datasets; the results in Table 2 show that when facing the general domain, this method still has strong robustness and exceeds the existing SOTA methods; the results in Table 3 show that when facing a specific domain, this method shows a great improvement, which proves the significant improvement of our method on the performance of domain multi-modal machine translation.

[0150] The domain multi-modal neural machine translation method based on asymmetric twin distillation coding proposed by the present invention, this method adopts a two-branch asymmetric encoder architecture, including parallel virtual vision and text encoders, to simultaneously generate domain-specific virtual vision and text representations. The domain virtual vision information serves as a modality-independent hub, and gradually guides the learning of domain-consistent representations through hierarchical cross-modal interactions between encoder layers.

[0151] The specific implementation manners of the present invention have been described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the above implementation manners, and various changes can be made without departing from the spirit of the present invention within the scope of knowledge possessed by those of ordinary skill in the art.

Claims

1. A domain multi-modal neural machine translation method based on asymmetric twin distillation coding, characterized in that It includes the following steps: Step 1. Use CLIP to obtain the image most relevant to the source language sentence and perform BPE processing on the source language sentence; Step 2. Obtain text and image features through Word Embedding and Resnet-101 respectively; Step 3. Input the text features into the constructed asymmetric twin encoder for full representation; Step 4. Adopt a text-image interaction masking mechanism to mask the visual virtual representation and enhance the model's text-image representation ability; Step 5. Perform domain feature probability sampling on the visual virtual features, and then perform cross-modal distillation on the samples and Resnet-101; Step 6. Perform a hierarchical gating operation on the text representation output by the text encoder and the visual virtual features generated by the visual virtual encoder; Step 7. Feed the text-image fusion multimodal features obtained in Step 6 into the cross-lingual attention mechanism of the decoder, and the generation of the target sentence follows the traditional Transformer decoder framework.

2. The domain multi-modal neural machine translation method based on asymmetric twin distillation coding according to claim 1, characterized in that: The specific steps of Step 1 are as follows: Step 1.

1. Use a crawler to obtain a multi-modal dataset for a specific domain; Step 1.

2. Use the CLIP model to obtain the image most relevant to the source language text; Step 1.

3. Perform BPE preprocessing on the source language text.

3. The domain multi-modal neural machine translation method based on asymmetric twin distillation coding according to claim 1, characterized in that: The specific steps of Step 2 are as follows: Step 2.

1. Use Resnet-101 to encode the image to obtain the corresponding image features; Step 2.

2. Use Word Embedding to encode the text, and then use the text encoder to obtain the text representation; 4. The domain multi-modal neural machine translation method based on asymmetric twin distillation coding according to claim 1, wherein: The specific steps of Step 3 are as follows: Step 3.

1. Build an asymmetric twin encoder based on the open-source framework Fairseq; Step 3.

2. Input the text features into the text encoder in the asymmetric twin encoder for representation; Step 3.

3. Input the output of the text encoder into the visual virtual encoder of the same layer for representation.

5. The domain multi-modal neural machine translation method based on asymmetric twin distillation coding according to claim 1, characterized in that: The specific steps of Step 4 are as follows: Step 4.

1. Perform text-vision and vision-text interactions on the image features obtained in Step 2 and the text features after representation in Step 3; Step 4.

2. Perform interactions on the interaction results between text-vision and vision-text again; Step 4.

3. Use the result after interaction as the mask for the visual virtual encoder.

6. The domain multi-modal neural machine translation method based on asymmetric twin distillation coding according to claim 4, characterized in that: The specific steps of Step 3.1 are as follows: Step 3.1.

1. Add a visual virtual encoder based on the Transformer of the open-source framework Fairseq; Step 3.1.

2. The input of the visual virtual encoder is the output of the text encoder of the same level, and the output is visual virtual features: Among them, denotes a visual virtual encoder with a self-attention module and a feed-forward neural network module at the l-th layer, denotes the text representation at the l-th layer.

7. The domain multi-modal neural machine translation method based on asymmetric twin distillation coding according to claim 5, characterized in that: The specific steps of Step 4.1 are as follows: Step 4.1.

1. Perform the following operations on the text representation and visual representation after sufficient representation: Among them, represents the relevance between each token level in the source language sentence and 49 image regions, represents the relevance between 49 image regions and each token level in the source language sentence, represents the text representation of the (l-1)-th layer, F v represents the visual representation, softmax represents the activation function in the neural network, and d is the embedding dimension of the image.

8. The domain multi-modal neural machine translation method based on asymmetric twin distillation coding according to claim 7, characterized in that: The specific steps of Step 4.2 are as follows: Step 4.2.

1. Perform the following operations on the result obtained in Step 4.1: Step4.2.

2. Next, binarize the graphic-text interaction matrix C obtained from the above operations l The operation is as follows: C l (i, j) represents the matrix C l The element in the i-th row and j-th column, M l represents C l The masked matrix after binarization, represents the binarization threshold.

9. The domain multi-modal neural machine translation method based on asymmetric twin distillation coding according to claim 5, characterized in that: The specific steps of Step 4.3 are as follows: Step 4.3.

1. The method for constructing the visual virtual encoder module is as follows: Among them, represents the multi-head attention mechanism, and C l represents the binary matrix for modality interaction; Step 4.3.

2. Perform mask filling operations on the multi-head attention mechanism of the visual virtual encoder.

10. The domain multi-modal neural machine translation method based on asymmetric twin distillation coding according to claim 1, characterized in that: The specific steps of Step 5 are as follows: Step 5.

1. Perform average pooling operations on the visual virtual features, and then perform Repeat operations in the pooling dimension: Among them, represents the virtual visual representation after the pooling and Repeat operations. Pooling represents the pooling operation, and Repeat represents the replication operation, whose function is to restore the global virtual representation after pooling to the original dimension; Step5.

2. Concatenate the result obtained in Step5.1 , then pass it through a linear layer and finally perform a softmax operation: Among them, represents the finally obtained probability sampling matrix, Concat represents the concatenation operation, which concatenates the virtual visual features and the representations after pooling and Repeat on the last dimension. FFN represents the feed-forward neural network, SoftMax represents the activation function, and the finally obtained domain probability matrix Step 5.

3. Use the domain probability sampling matrix obtained in Step 5.2 to sample the virtual visual representation as follows: Among them, represents the virtual visual representation after sampling, and MulSam represents the probability sampling with replacement according to the probability matrix for to perform repeated probability sampling with replacement. Step5.

4. Perform KL loss calculation on the real visual representation F v and the sampled virtual visual representation : Among them, D KL represents calculating the KL loss value; The specific steps of Step 6 are as follows: Step 6.

1. Perform gating calculations on the outputs of the visual virtual encoder and the text encoder in the l-th layer: Among them, W t and W d represent trainable parameters, and Sigmoid represents the activation function; Step6.

2. Perform hierarchical gating operation on the obtained gating coefficient Ω l as follows: Among them, the hyperparameter k satisfies 0 ≤ k ≤ L.

Images