A multimodal neural machine translation method based on incremental distillation and regularized semantic consistency

Abstract

The present invention relates to a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, and belongs to the field of natural language processing technology. The present invention includes: encoding text and images to obtain text features and image features; partially masking the image features; mixing the image and masked image representations with the text representation respectively to achieve cross-modal fusion; sending the text features into a multi-layer Transformer module with frozen parameters for representation; performing cross-modal aggregation enhancement to increase the complementarity of cross-modal features; performing local point alignment distillation to distill the prior knowledge of the pre-trained pure text model into the multimodal model; constructing a regularized semantic alignment module to further capture more comprehensive visual-text alignment details and achieve semantic consistency constraints. The present invention solves the problems of scarcity of multimodal data pairs, as well as inefficient utilization of computing resources and inaccurate cross-modal alignment caused by training multimodal models from scratch.

Description

Technical Field

[0001] The present invention relates to a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, and belongs to the technical field of natural language processing. Background Art

[0002] Multimodal neural machine translation (MMT) aims to address the challenges of cross-language translation by introducing additional modalities such as images. However, since images and text are two distinct modalities, there is a significant modality gap between them. Collecting high-quality and semantically consistent image and source-target sentence pairs becomes difficult because evaluating the quality of cross-modal alignment is challenging. Therefore, MMT often faces the problems of data scarcity or noise robustness.

[0003] Most existing MMT research focuses on achieving feature-level cross-modal fusion by introducing visual details into a sequence-to-sequence framework. These visually enhanced machine translation models are usually trained from scratch and processed separately from the text-only machine translation task. Therefore, existing MMT models face two major challenges: 1. The training process of MMT usually repeats the training process of plain text machine translation, resulting in inefficient use of computing resources, because MMT and plain text machine translation are similar tasks with the same objectives and model frameworks. 2. Existing vision-to-text fusion strategies tend to treat coarse-grained or fine-grained visual and textual representations as discrete points and fuse them in a point-to-point manner, while the problem of modal space alignment is often ignored.

[0004] Integrating visual information into text in MMT must address two key challenges: learning the intra-modal semantics of the text and capturing the semantics of inter-modal interactions. Together, these intra-modal and inter-modal semantics form a multimodal representation, thereby improving machine translation performance. Existing research in plain-text machine translation has made significant progress in text modality alignment. Therefore, MMT can be viewed as an incremental learning task, building on pre-trained plain-text translation models and focusing primarily on learning inter-modal representations from vision to text. Summary of the Invention

[0005] The present invention proposes a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency to address the problems in the above-mentioned technologies of scarcity of aligned image and text data and the repeated training process of plain text machine translation by training MMT from scratch, which leads to inefficient use of computing resources and inaccurate cross-modal alignment. The present invention improves translation performance and noise robustness.

[0006] The technical solution of the present invention is: a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, the method comprising:

[0007] Step 1: Obtain a multimodal machine translation dataset and encode the text and image to obtain text features and image features;

[0008] Step 2: Partially mask the image features so that they can be subsequently completed based on the complete text semantic information, reducing the modality gap between the image and text pairs.

[0009] Step 3: Feed the text, image, and mask image features into a module with multiple layers of Transformers for full representation.

[0010] Step 4: Mix the image and mask image representations with the text representation to initially achieve cross-modal fusion;

[0011] Step 5: Send the text features into the multi-layer Transformer module with frozen parameters for full representation;

[0012] Step 6: Perform cross-modal aggregation enhancement to increase the complementarity of cross-modal features and input them into the decoder to participate in the generation of the target sentence;

[0013] Step 7: Perform local point alignment distillation to distill the prior knowledge of the pre-trained plain text model into the multimodal model in a feature-level point-to-point manner, while achieving cross-scene and cross-modal alignment.

[0014] Step 8: Build a regularized semantic alignment module to further capture more comprehensive visual-text alignment details and implement semantic consistency constraints;

[0015] The distillation loss obtained in Step 9 and Step 7, the semantic consistency loss obtained in Step 8, and the traditional cross entropy loss are combined as the final training target;

[0016] Step 10: The cross-modal aggregated multimodal features of images and texts obtained in Step 6 are fed into the cross-language attention mechanism of the decoder. The generation of the target sentence follows the traditional Transformer decoder framework.

[0017] Furthermore, the step 1 includes:

[0018] Step 1.1: Use web crawlers to obtain a multimodal machine translation dataset.

[0019] Step 1.2, perform byte pair encoding (BPE) preprocessing on the text;

[0020] Step 1.3: Use Resnet-101 to encode the image and obtain the corresponding image feature E. v , image feature E vThe calculation process is as follows:

[0021] E v =ResNet(v)

[0022] Where v represents the image;

[0023] Step 1.4: Use the Embed & encoding module in Transformer to encode the text to obtain text features E x , the calculation process is as follows:

[0024] E x =Emb(x)

[0025] Where x represents the text.

[0026] Furthermore, in the step Step 2, the image feature E v Perform partial masking and the mask image feature calculation process is as follows:

[0027] E vs =F(E v ,m (1,2,…,k) )

[0028] where m (1,2,…,k) Indicates the position of the selection mask, k represents E v The number of masks in , F represents the mask operation.

[0029] Furthermore, the step 3 includes:

[0030] Step 3.1: Send the text features into the stacked multi-layer Transformer to fully represent the text features and obtain the text representation C x ;

[0031] Step 3.2: Send the image features into the stacked multi-layer Transformer to fully characterize the image features and obtain the image representation C v ;

[0032] Step 3.3, send the mask image features to the stacked multi-layer Transformer, add a visual encoder to the encoder part of the Transformer, and use the mask image features as the query, key and value for attention mechanism calculation, so that the mask image features E vs Perform full characterization to obtain the mask image representation C vs , the calculation process is as follows:

[0033] C vs =TransEnc v (E vs )

[0034] Among them, TransEnc v Represents a stacked image Transformer with self-attention and FFN modules.

[0035] Furthermore, a representation of part of the text is selected from the length dimension of the text representation to replace the corresponding position of the image to perform modal mixing. In the step 4, the calculation process of the cross-modal fusion is as follows:

[0036] C v_mix =aC x +(1-a)C v

[0037] C vs_mix =aC x +(1-a)C vs

[0038] Among them, C x Represents text representation, C v represents the image representation, C vs represents the mask image representation, a represents the coefficient vector for modal mixing, C v_mix represents the mixed image features, C vs_mix Represents mixed mask image features.

[0039] Furthermore, in the step 5, the text features are fed into the multi-layer Transformer with frozen parameters, and the encoder part of the Transformer is added with the text encoder with frozen parameters to convert the text features E x As Query, Key and Value, the attention mechanism is calculated. The calculation process is as follows:

[0040] C f =TransEnc f (E x )

[0041] Among them, TransEnc f Text Transformer with self-attention and FFN modules with frozen parameters, C f Represents the fully characterized text features.

[0042] Furthermore, in the step 6, the cross-modal aggregation enhancement is to combine the fully characterized text and mixed image features C obtained in step 3 and step 4 v_mix Perform aggregation enhancement to obtain the multimodal features of images and texts; the multimodal features of images and texts C s The calculation process is as follows:

[0043] C s =C v_mix +C x

[0044] Among them, C s Represents the multimodal features of images and text, which are subsequently input into the cross-lingual attention mechanism of the Transformer decoder.

[0045] Furthermore, in the step 7, the local point alignment distillation is to perform local point alignment distillation on the results obtained in step 5 and step 6, and the distillation loss in the process of local point alignment distillation adopts mean square error loss, and the mean square error loss loss MSE The calculation is as follows:

[0046]

[0047] Among them, N represents the number of samples, loss MSE represents the mean squared error loss.

[0048] Furthermore, the step 8 includes:

[0049] Step 8.1, perform gated screening and fusion on the results obtained in Step 4 and Step 5 respectively;

[0050] Step 8.2: Perform spatial regularization alignment.

[0051] Furthermore, the step 8.1 includes:

[0052] Step 8.1.1, the mixed image feature C v_mix and the fully characterized text representation C f Perform the following operations to obtain complete semantic features The calculation process is as follows:

[0053]

[0054]

[0055] Among them, μ - represents the cross-modal gating weight, M v_mix , is a trainable parameter, Sigmoid represents the Sigmoid function;

[0056] Step 8.1.2, the mixed mask image feature C vs_mix and the fully characterized text representation C f Perform the following operations to obtain the restored semantic features The calculation process is as follows:

[0057]

[0058]

[0059] Among them, μ + represents the cross-modal gating weight, M vs_mix , is a trainable parameter, Sigmoid represents the Sigmoid function;

[0060] Step 8.1.3, semantically align the complete semantic features and the restored semantic features, and use the information noise contrast estimation loss for semantic consistency loss. con The calculation process is as follows:

[0061]

[0062] Among them, log represents the logits function, exp represents the exponential function, τ represents the temperature coefficient used to adjust the similarity score distribution, and Loss con represents the information-noise contrast estimation loss.

[0063] The present invention transfers the prior knowledge of the pre-trained plain text translation model to the multimodal translation scenario, while adding a regularized semantic alignment strategy to further improve translation performance. First, the embedded representation of the image is partially masked so that it can be subsequently completed based on the complete text semantic information, reducing the modal gap between the image and text pairs. Then, preliminary cross-modal fusion is achieved through modal mixing. Then, through cross-modal aggregation enhancement, the cross-modal features are made more complementary and more discriminative. After that, through local point alignment distillation, the knowledge of the pre-trained plain text model is gradually distilled into the multimodal model, and the text knowledge of the pre-trained model is aligned and fused with the image. Finally, regularized semantic alignment based on contrastive learning further captures more comprehensive visual-text alignment details and implements semantic consistency constraints. The present invention mainly solves the problems of scarcity of multimodal data pairs, inefficient utilization of computing resources and inaccurate cross-modal alignment caused by training multimodal models from scratch. It aims to retain the translation ability of the pre-trained plain text translation model while further improving translation performance through a regularized alignment strategy.

[0064] The beneficial effects of the present invention are:

[0065] 1. This paper proposes a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, which extends machine translation from text-only to image-text scenarios through incremental learning;

[0066] 2. This method inherits the intra-modality semantic alignment capability of the original pre-trained plain text translation model while expanding the model's ability to understand visual modalities in visual perception scenarios. It effectively reduces the use of computing resources and demonstrates significant noise robustness improvements in low-resource and semantically noisy scenarios during the inference phase.

[0067] 3. This paper applies a regularized semantic consistency strategy to capture aligned image details, achieve image-to-text enhancement, and improve overall machine translation performance;

[0068] 4. The method of the present invention is implemented based on the Fairseq open source system, which is relatively stable and has strong text processing capabilities, providing a strong guarantee for multimodal machine translation for specific fields;

[0069] 5. The method proposed in this invention has achieved SOTA results on the test sets of multiple public datasets. Visual experiments and case analysis have demonstrated the effectiveness and superiority of the method of this invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the present invention;

[0071] Figure 2 It is a detailed process framework diagram of the present invention. DETAILED DESCRIPTION

[0072] Example 1: In response to the problems of data scarcity and repeated training faced by multimodal machine translation, the present invention proposes a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency. Starting from a pre-trained plain text translation model, the prior knowledge of the model is transferred to multimodal machine translation. At the same time, the regularized semantic consistency mechanism is used to more effectively utilize visual information, thereby improving the overall performance of translation.

[0073] like Figure 1-Figure 2 As shown, a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, the method comprising:

[0074] Step 1: Obtain a multimodal machine translation dataset and encode the text and images to obtain text and image features. Directly obtain the raw data of Multi30k and Flickr30kEn-Jp, totaling 59,000 entries, from the publicly available download links for the datasets. Download the images and text from the Fashion-MMT dataset for e-commerce and divide the obtained data into training, validation, and test sets. Perform byte pair encoding (BPE) preprocessing on the text.

[0075] Furthermore, the step 1 includes:

[0076] Step 1.1: Use web crawlers to obtain a multimodal machine translation dataset.

[0077] Step 1.2, perform byte pair encoding (BPE) preprocessing on the text;

[0078] Step 1.3: Use Resnet-101 to encode the image and obtain the corresponding image feature E. v , image feature E v The calculation process is as follows:

[0079] E v =ResNet(v)

[0080] Where v represents the image;

[0081] Step 1.4: Use the Embed & encoding module in Transformer to encode the text to obtain text features E x , the calculation process is as follows:

[0082] E x =Emb(x)

[0083] Where x represents the text.

[0084] Step 2: Partially mask the image features so that they can be subsequently completed based on the complete text semantic information, reducing the modality gap between the image and text pairs.

[0085] Furthermore, in the step Step 2, the image feature E v Perform partial masking and the mask image feature calculation process is as follows:

[0086] E vs =F(E v ,m (1,2,…,k) )

[0087] where m (1,2,…,k) Indicates the position of the selection mask, k represents E v The number of masks in , F represents the mask operation.

[0088] Step 3: Feed the text, image, and mask image features into a module with multiple layers of Transformers for full representation.

[0089] Furthermore, the step 3 includes:

[0090] Step 3.1: Send the text features into the stacked multi-layer Transformer, and fully characterize the text features through the Self-attention mechanism, Add&Norm mechanism, and Feed Forward to obtain the text representation C x ;

[0091] Step 3.2: Send the image features to the stacked multi-layer Transformer, and fully characterize the image features through the Self-attention mechanism, Add&Norm mechanism, and Feed Forward to obtain the image representation C v ;

[0092] Step 3.3, the mask image features are fed into the stacked multi-layer Transformer. The encoder part of the Transformer is added with the visual encoder. The mask image features are used as the query, key and value for attention mechanism calculation, so that the mask image features E vs Perform full characterization to obtain the mask image representation C vs , the calculation process is as follows:

[0093] C vs =TransEnc v (E vs )

[0094] Among them, TransEnc v Represents a stacked image transformer with self-attention and FFN modules. Step 4: Mix the image and mask image representations with the text representation to achieve preliminary cross-modal fusion.

[0095] Furthermore, in Step 4, a representation of part of the text is selected from the length dimension of the text representation to replace the corresponding position of the image to perform modal mixing; the calculation process of the cross-modal fusion is as follows:

[0096] C v_mix =aC x +(1-a)C v

[0097] C vs_mix =aC x +(1-a)C vs

[0098] Among them, C x Represents text representation, C v represents the image representation, C vs represents the mask image representation, a represents the coefficient vector for modal mixing, C v_mixrepresents the mixed image features, C vs_mix Represents mixed mask image features.

[0099] Step 5: Send the text features into the multi-layer Transformer module with frozen parameters for full representation;

[0100] Furthermore, in the step 5, the text features are fed into the multi-layer Transformer with frozen parameters, and the encoder part of the Transformer is added with the text encoder with frozen parameters to convert the text features E x As Query, Key and Value, the attention mechanism is calculated. The calculation process is as follows:

[0101] C f =TransEnc f (E x )

[0102] Among them, TransEnc f Text Transformer with self-attention and FFN modules with frozen parameters, C f Represents the fully characterized text features.

[0103] Step 6: Perform cross-modal aggregation enhancement to increase the complementarity of cross-modal features and input them into the decoder to participate in the generation of the target sentence;

[0104] Furthermore, in the step 6, the cross-modal aggregation enhancement is to combine the fully characterized text and mixed image features C obtained in step 3 and step 4 v_mix Perform aggregation enhancement to obtain the multimodal features of images and texts; the multimodal features of images and texts C s The calculation process is as follows:

[0105] C s =C v_mix +C x

[0106] Among them, C s Represents the multimodal features of images and text, which are subsequently input into the cross-lingual attention mechanism of the Transformer decoder.

[0107] Step 7: Perform local point alignment distillation to distill the prior knowledge of the pre-trained plain text model into the multimodal model in a feature-level point-to-point manner, while achieving cross-scene and cross-modal alignment.

[0108] Furthermore, in the step 7, the local point alignment distillation is to perform local point alignment distillation on the results obtained in step 5 and step 6, and the distillation loss in the process of local point alignment distillation adopts mean square error loss, and the mean square error loss loss MSE The calculation is as follows:

[0109]

[0110] Among them, N represents the number of samples, loss MSE represents the mean squared error loss.

[0111] Step 8: Build a regularized semantic alignment module to further capture more comprehensive visual-text alignment details and implement semantic consistency constraints;

[0112] Furthermore, the step 8 includes:

[0113] Step 8.1, perform gated screening and fusion on the results obtained in Step 4 and Step 5 respectively;

[0114] Step 8.2: Perform spatial regularization alignment.

[0115] Furthermore, the step 8.1 includes:

[0116] Step 8.1.1, the mixed image feature C v_mix and the fully characterized text representation C f Perform the following operations to obtain complete semantic features The calculation process is as follows:

[0117]

[0118]

[0119] Among them, μ - represents the cross-modal gating weight, M v_mix , is a trainable parameter, Sigmoid represents the Sigmoid function;

[0120] Step 8.1.2, the mixed mask image feature C vs_mix and the fully characterized text representation C f Perform the following operations to obtain the restored semantic features The calculation process is as follows:

[0121]

[0122]

[0123] Among them, μ + represents the cross-modal gating weight, M vs_mix , is a trainable parameter, Sigmoid represents the Sigmoid function;

[0124] Step 8.1.3, semantically align the complete semantic features and the restored semantic features, and use the information noise contrast estimation loss for semantic consistency loss. con The calculation process is as follows:

[0125]

[0126] Among them, log represents the logits function, exp represents the exponential function, τ represents the temperature coefficient used to adjust the similarity score distribution, and Loss con represents the information-noise contrast estimation loss.

[0127] The distillation loss obtained in Step 9 and Step 7, the semantic consistency loss obtained in Step 8, and the traditional cross entropy loss are combined as the final training target;

[0128] Step 10: The cross-modal aggregated multimodal features of images and texts obtained in Step 6 are fed into the cross-language attention mechanism of the decoder. The generation of the target sentence follows the traditional Transformer decoder framework.

[0129] To demonstrate the effectiveness of our proposed method, we conducted extensive experiments on the Multi-30k, Flickr30kEn-Jp, and Fashion-MMT datasets. The dataset details are shown in Table 1. To accurately evaluate the performance of our method, we used BLUE and METEOR as evaluation metrics. These two metrics can objectively and accurately reflect the effectiveness of machine translation. The experimental results are shown in Tables 2, 3, and 4.

[0130] Table 1 Dataset details

[0131]

[0132] Table 2 Comparison of the model corresponding to the proposed method and the existing SOTA model in Multi-30k

[0133]

[0134] Table 3 is a comparison of the model corresponding to the proposed method and the existing SOTA model in Flickr30kEn-Jp

[0135]

[0136] Table 4 is a comparison of the model corresponding to the method of the present invention and the existing SOTA model in Fashion-MMT

[0137]

[0138] The above results all show that the multimodal neural machine translation method based on incremental distillation and regularized semantic consistency proposed in the present invention has achieved SOTA results on all datasets. Tables 2 and 3 show that the proposed method improves translation performance in general fields and shows significant improvements on the ambiguous MSCOCO test set, which shows that the method is robust; the results in Table 4 show that when facing specific fields, the method still has strong robustness and exceeds the existing SOTA methods. Through the performance comparison on the test set, it can be found that the multimodal Transformer with incremental distillation and regularized semantic consistency can greatly improve the translation performance of the model.

[0139] This paper proposes a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency. Starting from a pre-trained plain text translation model, this method transfers the prior knowledge of the model to multimodal machine translation. At the same time, it uses the regularized semantic consistency mechanism to more effectively utilize visual information and improve the overall translation performance.

[0140] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, characterized by: The method comprises: Step 1: Obtain a multimodal machine translation dataset and encode the text and image to obtain text features and image features; Step 2: Partially mask the image features; Step 3: Send the text, image, and mask image features into a module with multiple layers of Transformer for representation; Step 4: Mix the image and mask image representations with the text representation to initially achieve cross-modal fusion; Step 5: Send the text features into the multi-layer Transformer module with frozen parameters for full representation; Step 6: Perform cross-modal aggregation enhancement to increase the complementarity of cross-modal features and input them into the decoder to participate in the generation of the target sentence; Step 7: Perform local point alignment distillation to distill the prior knowledge of the pre-trained plain text model into the multimodal model through a feature-level point-to-point approach. Step 8: Build a regularized semantic alignment module to further capture more comprehensive visual-text alignment details; The distillation loss obtained in Step 9 and Step 7, the semantic consistency loss obtained in Step 8, and the traditional cross entropy loss are combined as the final training target; Step 10: The cross-modal aggregated image-text multimodal features obtained in Step 6 are fed into the decoder’s cross-language attention mechanism. The target sentence is generated following the traditional Transformer decoder framework. Step 8 includes: Step 8.1, perform gated screening and fusion on the results obtained in Step 4 and Step 5 respectively; Step 8.2, perform spatial regularization alignment; Step 8.1 includes: Step 8.1.1, Mix the image features and fully characterized text representation Perform the following operations to obtain complete semantic features , the calculation process is as follows: ； ； in, represents the cross-modal gating weight, , is a trainable parameter, Represents the Sigmoid function; Step 8.1.2, Mix the mask image features and fully characterized text representation Perform the following operations to obtain the restored semantic features , the calculation process is as follows: ； ； in, represents the cross-modal gating weight, , is a trainable parameter, Represents the Sigmoid function.

2. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: Step 1 includes: Step 1.1: Use web crawlers to obtain a multimodal machine translation dataset. Step 1.2, perform BPE preprocessing on the text; Step 1.3: Use Resnet-101 to encode the image and obtain the corresponding image features , image features The calculation process is as follows: ； in Represents an image; Step 1.4: Use the Embed & encoding module in Transformer to encode the text to obtain text features , the calculation process is as follows: ； in Represents text.

3. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In Step 2, the image features Perform partial masking and the mask image feature calculation process is as follows: ； in Indicates the location of the selection mask, k indicates The number of masks in Indicates a mask operation.

4. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: Step 3 includes: Step 3.1: Send the text features into the stacked multi-layer Transformer to fully represent the text features and obtain the text representation ; Step 3.2: Send the image features into the stacked multi-layer Transformer to fully characterize the image features and obtain the image representation ; Step 3.3, send the mask image features into the stacked multi-layer Transformer, and use the mask image features as Query, Key and Value to calculate the attention mechanism, so that the mask image features Perform full characterization to obtain mask image representation , the calculation process is as follows: ; in, Represents a stacked image Transformer with self-attention and FFN modules.

5. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In Step 4, the calculation process of the cross-modal fusion is as follows: ； ； in, represents text representation, represents the image representation, represents the mask image representation, represents the coefficient vector for mode mixing, Represents mixed image features, Represents mixed mask image features.

6. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In Step 5, the text features are fed into the multi-layer Transformer with frozen parameters. As Query, Key and Value, the attention mechanism is calculated. The calculation process is as follows: ； in, Text Transformer representing self-attention and FFN modules with frozen parameters, Represents the fully characterized text features.

7. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In Step 6, the cross-modal aggregation enhancement is to combine the fully characterized text and mixed image features obtained in Step 3 and Step 4 Perform aggregation enhancement to obtain multimodal features of images and texts; Multimodal features of images and texts The calculation process is as follows: ； in, Represents the multimodal features of images and texts, which are then input into the cross-language attention mechanism of the Transformer decoder. Represents text representation.

8. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In Step 7, the local point alignment distillation is to perform local point alignment distillation on the results obtained in Step 5 and Step 6. The distillation loss in the process of local point alignment distillation adopts mean square error loss. The calculation is as follows: ； in, represents the number of samples, represents the mean square error loss, Represents multimodal features of images and text.

9. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: Step 8.1 also includes: Step 8.1.3, semantically align the complete semantic features and the restored semantic features, and use the information noise contrast estimation loss for semantic consistency loss. The calculation process is as follows: ； in, represents the logits function, represents the exponential function, represents the temperature coefficient used to adjust the similarity score distribution, represents the information-noise contrast estimation loss.

Images