Multimodal neural machine translation method based on incremental distillation and regularization semantic consistency

Abstract

The invention relates to a multi-modal neural machine translation method based on incremental distillation and regularization semantic consistency, and belongs to the technical field of natural language processing. The method comprises the following steps: encoding a text and an image to obtain text features and image features; partially masking the image features; respectively mixing the image representation and the mask image representation with the text representation to realize cross-modal fusion; the text features are sent to a multi-layer Transform module of the freezing parameters to be represented; cross-modal aggregation enhancement is carried out, and complementarity of cross-modal features is increased; carrying out local point alignment distillation, and distilling priori knowledge of the pre-trained plain text model into a multi-modal model; a regularization semantic alignment module is constructed, more comprehensive vision-text alignment details are further captured, and semantic consistency constraint is achieved. According to the method, the problems that multi-modal data pairs are scarce, and due to the fact that a multi-modal model is trained from the beginning, computing resource utilization is low in efficiency and cross-modal alignment is not accurate are solved.

Description

Technical Field

[0001] The invention relates to a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, and belongs to the technical field of natural language processing. Background Art

[0002] Multimodal neural machine translation (MMT) aims to address the challenges of cross-language translation by introducing additional modalities such as images. However, there is a significant modality gap between images and texts, as they are two completely different modalities. Collecting high-quality and semantically consistent image and source-target sentence pairs becomes difficult because evaluating the quality of cross-modal alignment is challenging. Therefore, MMT often faces the problem of data scarcity or noise robustness.

[0003] Most existing MMT research focuses on achieving feature-level cross-modal fusion by introducing visual details into a sequence-to-sequence framework. These visually enhanced machine translation models are usually trained from scratch and treated separately from the text-only machine translation task. Therefore, existing MMT models face two major challenges: 1. The training process of MMT usually repeats the training process of plain text machine translation, resulting in inefficient use of computing resources, because MMT and plain text machine translation are similar tasks with the same objectives and model frameworks. 2. Existing vision-to-text fusion strategies tend to treat coarse-grained or fine-grained visual and textual representations as discrete points and fuse them in a point-to-point manner, while the problem of modal space alignment is often ignored.

[0004] Integrating visual information into text in MMT must address two main challenges: learning text intra-modal semantics and capturing inter-modal interaction semantics. Intra-modal semantics and inter-modal semantics together constitute multimodal representations, thereby improving the performance of machine translation. Existing plain text machine translation research has made significant progress in text modality alignment. Therefore, MMT can be viewed as an incremental learning task, which builds on the pre-trained plain text translation model and mainly focuses on learning inter-modal representations from vision to text. Summary of the invention

[0005] The present invention proposes a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency to solve the problems of scarcity of aligned image and text data in the above-mentioned technology and the training process of repeating plain text machine translation by training MMT from scratch, which leads to inefficient use of computing resources and inaccurate cross-modal alignment. The present invention improves translation performance and noise robustness.

[0006] The technical solution of the present invention is: a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, the method comprising:

[0007] Step 1: Obtain a multimodal machine translation dataset and encode text and images to obtain text features and image features;

[0008] Step 2: Partially mask the image features so that they can be completed based on the complete text semantic information later, reducing the modality gap between the image and text pairs.

[0009] Step 3: Send the text, image and mask image features into a module with multiple layers of Transformer for full representation;

[0010] Step 4: Mix the image and mask image representations with the text representations respectively to initially achieve cross-modal fusion;

[0011] Step 5: Send the text features into the multi-layer Transformer module with frozen parameters for full representation;

[0012] Step 6: Perform cross-modal aggregation enhancement to increase the complementarity of cross-modal features and input them into the decoder to participate in the generation of the target sentence;

[0013] Step 7: Perform local point alignment distillation to distill the prior knowledge of the pre-trained plain text model into the multimodal model in a feature-level point-to-point manner, and achieve cross-scene and cross-modal alignment at the same time;

[0014] Step 8: Build a regularized semantic alignment module to further capture more comprehensive visual-text alignment details and implement semantic consistency constraints;

[0015] The distillation loss obtained in Step 9 and Step 7, the semantic consistency loss obtained in Step 8 and the traditional cross entropy loss are combined as the final training target;

[0016] Step 10: The cross-modal aggregated multimodal features of images and texts obtained in Step 6 are fed into the cross-language attention mechanism of the decoder. The generation of the target sentence follows the traditional Transformer decoder framework.

[0017] Furthermore, the step Step 1 includes:

[0018] Step 1.1, web crawler, obtain multimodal machine translation dataset;

[0019] Step 1.2, perform byte pair encoding BPE preprocessing on the text;

[0020] Step 1.3: Use Resnet-101 to encode the image and obtain the corresponding image feature E v , image feature E vThe calculation process is as follows:

[0021] E v =ResNet(v)

[0022] Where v represents the image;

[0023] Step 1.4: Use the Embed & encoding module in Transformer to encode the text to obtain the text feature E x , the calculation process is as follows:

[0024] E x =Emb(x)

[0025] Where x represents the text.

[0026] Furthermore, in the step Step2, the image feature E v Perform partial masking and the mask image feature calculation process is as follows:

[0027] E vs =F(E v ,m (1,2,…,k) )

[0028] Where m (1,2,…,k) Indicates the location of the selection mask, k represents E v The number of masks in , and F represents the mask operation.

[0029] Furthermore, the step Step 3 includes:

[0030] Step 3.1: Send the text features into the stacked multi-layer Transformer to fully represent the text features and obtain the text representation C x ;

[0031] Step 3.2: Send the image features to the stacked multi-layer Transformer to fully characterize the image features and obtain the image representation C v ;

[0032] Step 3.3, send the mask image features to the stacked multi-layer Transformer, add a visual encoder to the encoder part of the Transformer, and use the mask image features as Query, Key and Value for attention mechanism calculation, so that the mask image features E vs Perform full characterization to obtain the mask image representation C vs , the calculation process is as follows:

[0033] C vs =TransEnc v (E vs )

[0034] Among them, TransEnc v Represents a stacked image Transformer with self-attention and FFN modules.

[0035] Furthermore, a representation of part of the text is selected from the length dimension of the text representation to replace the corresponding position of the image, and modal mixing is performed; in the step Step 4, the calculation process of the cross-modal fusion is as follows:

[0036] C v_mix =aC x +(1-a)C v

[0037] C vs_mix =aC x +(1-a)C vs

[0038] Among them, C x Represents text representation, C v represents the image representation, C vs represents the mask image representation, a represents the coefficient vector for modal mixing, C v_mix represents the mixed image features, C vs_mix Represents mixed mask image features.

[0039] Furthermore, in the step 5, the text features are fed into a multi-layer Transformer with frozen parameters, and the encoder part of the Transformer is added with a text encoder with frozen parameters to convert the text features E x As Query, Key and Value, the attention mechanism is calculated. The calculation process is as follows:

[0040] C f =TransEnc f (E x )

[0041] Among them, TransEnc f Text Transformer with self-attention and FFN modules showing frozen parameters, C f Represents the fully characterized text features.

[0042] Furthermore, in the step Step 6, the cross-modal aggregation enhancement is to combine the fully characterized text and mixed image features C obtained in Step 3 and Step 4 v_mix Aggregation enhancement is performed to obtain the multimodal features of images and texts; the multimodal features of images and texts C s The calculation process is as follows:

[0043] C s =C v_mix +C x

[0044] Among them, C s Represents the multimodal features of images and texts, which are subsequently input into the cross-language attention mechanism of the Transformer decoder.

[0045] Furthermore, in the step 7, the local point alignment distillation is to perform local point alignment distillation on the results obtained in step 5 and step 6, and the distillation loss in the local point alignment distillation process adopts mean square error loss, and the mean square error loss loss MSE The calculation is as follows:

[0046]

[0047] Among them, N represents the number of samples, loss MSE represents the mean square error loss.

[0048] Furthermore, the step Step 8 includes:

[0049] Step8.1, perform gated screening and fusion on the results obtained in Step 4 and Step 5 respectively;

[0050] Step 8.2: Perform spatial regularization alignment.

[0051] Furthermore, the step 8.1 includes:

[0052] Step 8.1.1. Mix the image features C v_mix and the fully characterized text representation C f Perform the following operations to obtain complete semantic features The calculation process is as follows:

[0053]

[0054]

[0055] Among them, μ - represents the cross-modal gating weight, M v_mix , is a trainable parameter, Sigmoid represents the Sigmoid function;

[0056] Step 8.1.2, the mixed mask image feature C vs_mix and the fully characterized text representation C f Perform the following operations to obtain the restored semantic features The calculation process is as follows:

[0057]

[0058]

[0059] Among them, μ + represents the cross-modal gating weight, M vs_mix , is a trainable parameter, Sigmoid represents the Sigmoid function;

[0060] Step 8.1.3: Semantically align the complete semantic features and the restored semantic features. The semantic consistency loss uses the information noise contrast estimation loss. con The calculation process is as follows:

[0061]

[0062] Where log represents the logits function, exp represents the exponential function, τ represents the temperature coefficient used to adjust the similarity score distribution, and Loss con Denotes the information-noise contrast estimation loss.

[0063] The present invention transfers the prior knowledge of the pre-trained plain text translation model to the multimodal translation scenario, and adds a regularized semantic alignment strategy to further improve the translation performance. First, the embedded representation of the image is partially masked so that it can be completed according to the complete text semantic information in the future, reducing the modal gap between the image and text pairs. Then, preliminary cross-modal fusion is achieved through modal mixing. Then, cross-modal aggregation enhancement is performed to make cross-modal features more complementary and more discriminative. After that, the knowledge of the pre-trained plain text model is gradually distilled into the multimodal model through local point alignment distillation, and the text knowledge of the pre-trained model is aligned and fused with the picture. Finally, the regularized semantic alignment based on contrastive learning further captures more comprehensive visual-text alignment details and realizes semantic consistency constraints. The present invention mainly solves the problems of scarcity of multimodal data pairs, inefficient utilization of computing resources and inaccurate cross-modal alignment caused by training multimodal models from scratch, aiming to retain the translation ability of the pre-trained plain text translation model, while further improving the translation performance through a regularized alignment strategy.

[0064] The beneficial effects of the present invention are:

[0065] 1. This paper proposes a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, which extends machine translation from text-only to image-text scenarios through incremental learning;

[0066] 2. This method inherits the intra-modal semantic alignment capability of the original pre-trained plain text translation model, while expanding the model's ability to understand the visual modality in visual perception scenarios; it effectively reduces the use of computing resources and shows significant noise robustness improvement in low-resource and semantically noisy scenarios in the inference phase;

[0067] 3. The present invention applies a regularized semantic consistency strategy to capture aligned image details, achieve image-to-text enhancement, and improve the overall machine translation performance;

[0068] 4. The method of the present invention is implemented based on the Fairseq open source system, which is relatively stable and has strong text processing capabilities, which provides a strong guarantee for multimodal machine translation for specific fields;

[0069] 5. The method proposed in the present invention has achieved SOTA results on the test sets of multiple public data sets. Visual experiments and case analysis have demonstrated the effectiveness and superiority of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the present invention;

[0071] Figure 2 It is a detailed process framework diagram of the present invention. DETAILED DESCRIPTION

[0072] Example 1: In response to the problems of data scarcity and repeated training faced by multimodal machine translation, the present invention proposes a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency. Starting from a pre-trained plain text translation model, the prior knowledge of the model is transferred to multimodal machine translation. At the same time, the regularized semantic consistency mechanism is used to more effectively utilize visual information to improve the overall performance of translation.

[0073] like Figure 1-Figure 2 As shown, a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, the method comprising:

[0074] Step 1. Obtain a multimodal machine translation dataset and encode text and images to obtain text features and image features; directly obtain the original data of the multimodal machine translation data Multi30k and Flickr30kEn-Jp from the public download links of the Multi30k and Flickr30kEn-Jp datasets, a total of 59,000 records; download the pictures and texts of the Fashion-MMT dataset in the e-commerce field, and divide the obtained data into training sets, validation sets, and test sets. Perform byte pair encoding (BPE) preprocessing on the text.

[0075] Furthermore, the step Step 1 includes:

[0076] Step 1.1, web crawler, obtain multimodal machine translation dataset;

[0077] Step 1.2, perform byte pair encoding BPE preprocessing on the text;

[0078] Step 1.3: Use Resnet-101 to encode the image and obtain the corresponding image feature E v , image feature E v The calculation process is as follows:

[0079] E v =ResNet(v)

[0080] Where v represents the image;

[0081] Step 1.4: Use the Embed & encoding module in Transformer to encode the text to obtain the text feature E x , the calculation process is as follows:

[0082] E x =Emb(x)

[0083] Where x represents the text.

[0084] Step 2: Partially mask the image features so that they can be completed based on the complete text semantic information later, reducing the modality gap between the image and text pairs.

[0085] Furthermore, in the step Step2, the image feature E v Perform partial masking and the mask image feature calculation process is as follows:

[0086] E vs =F(E v ,m (1,2,…,k) )

[0087] Where m (1,2,…,k) Indicates the location of the selection mask, k represents E v The number of masks in , and F represents the mask operation.

[0088] Step 3: Send the text, image and mask image features into a module with multiple layers of Transformer for full representation;

[0089] Furthermore, the step Step 3 includes:

[0090] Step 3.1: Send the text features to the stacked multi-layer Transformer, and fully represent the text features through the Self-attention mechanism, Add&Norm mechanism, and Feed Forward to obtain the text representation C x ;

[0091] Step 3.2: Send the image features to the stacked multi-layer Transformer, and fully characterize the image features through the Self-attention mechanism, Add&Norm mechanism, and Feed Forward to obtain the image representation C v ;

[0092] Step 3.3, the mask image features are sent to the stacked multi-layer Transformer, the encoder part of the Transformer is added with the visual encoder, and the mask image features are used as the Query, Key and Value for attention mechanism calculation, so that the mask image features E vs Perform full characterization to obtain the mask image representation C vs , the calculation process is as follows:

[0093] C vs =TransEnc v (E vs )

[0094] Among them, TransEnc v Represents a stacked image Transformer with self-attention and FFN modules. Step 4: Mix the image and mask image representations with the text representations to achieve cross-modal fusion.

[0095] Furthermore, in the step 4, a representation of part of the text is selected from the length dimension of the text representation to replace the corresponding position of the image, and modal mixing is performed; the calculation process of the cross-modal fusion is as follows:

[0096] C v_mix =aC x +(1-a)C v

[0097] C vs_mix =aC x +(1-a)C vs

[0098] Among them, C x Represents text representation, C v represents the image representation, C vs represents the mask image representation, a represents the coefficient vector for modal mixing, C v_mixrepresents the mixed image features, C vs_mix Represents mixed mask image features.

[0099] Step 5: Send the text features into the multi-layer Transformer module with frozen parameters for full representation;

[0100] Furthermore, in the step 5, the text features are fed into a multi-layer Transformer with frozen parameters, and the encoder part of the Transformer is added with a text encoder with frozen parameters to convert the text features E x As Query, Key and Value, the attention mechanism is calculated. The calculation process is as follows:

[0101] C f =TransEnc f (E x )

[0102] Among them, TransEnc f Text Transformer with self-attention and FFN modules showing frozen parameters, C f Represents the fully characterized text features.

[0103] Step 6: Perform cross-modal aggregation enhancement to increase the complementarity of cross-modal features and input them into the decoder to participate in the generation of the target sentence;

[0104] Furthermore, in the step Step 6, the cross-modal aggregation enhancement is to combine the fully characterized text and mixed image features C obtained in Step 3 and Step 4 v_mix Aggregation enhancement is performed to obtain multimodal features of images and texts; multimodal features of images and texts C s The calculation process is as follows:

[0105] C s =C v_mix +C x

[0106] Among them, C s Represents the multimodal features of images and texts, which are subsequently input into the cross-language attention mechanism of the Transformer decoder.

[0107] Step 7: Perform local point alignment distillation to distill the prior knowledge of the pre-trained plain text model into the multimodal model in a feature-level point-to-point manner, and achieve cross-scene and cross-modal alignment at the same time;

[0108] Furthermore, in the step 7, the local point alignment distillation is to perform local point alignment distillation on the results obtained in step 5 and step 6, and the distillation loss in the local point alignment distillation process adopts mean square error loss, and the mean square error loss loss MSE The calculation is as follows:

[0109]

[0110] Among them, N represents the number of samples, loss MSE represents the mean square error loss.

[0111] Step 8: Build a regularized semantic alignment module to further capture more comprehensive visual-text alignment details and implement semantic consistency constraints;

[0112] Furthermore, the step Step 8 includes:

[0113] Step8.1, perform gated screening and fusion on the results obtained in Step 4 and Step 5 respectively;

[0114] Step 8.2: Perform spatial regularization alignment.

[0115] Furthermore, the step 8.1 includes:

[0116] Step 8.1.1. Mix the image features C v_mix and the fully characterized text representation C f Perform the following operations to obtain complete semantic features The calculation process is as follows:

[0117]

[0118]

[0119] Among them, μ - represents the cross-modal gating weight, M v_mix , is a trainable parameter, Sigmoid represents the Sigmoid function;

[0120] Step 8.1.2, the mixed mask image feature C vs_mix and the fully characterized text representation C f Perform the following operations to obtain the restored semantic features The calculation process is as follows:

[0121]

[0122]

[0123] Among them, μ + represents the cross-modal gating weight, M vs_mix , is a trainable parameter, Sigmoid represents the Sigmoid function;

[0124] Step 8.1.3: Semantically align the complete semantic features and the restored semantic features. The semantic consistency loss uses the information noise contrast estimation loss. con The calculation process is as follows:

[0125]

[0126] Where log represents the logits function, exp represents the exponential function, τ represents the temperature coefficient used to adjust the similarity score distribution, and Loss con Denotes the information-noise contrast estimation loss.

[0127] The distillation loss obtained in Step 9 and Step 7, the semantic consistency loss obtained in Step 8 and the traditional cross entropy loss are combined as the final training target;

[0128] Step 10: The cross-modal aggregated multimodal features of images and texts obtained in Step 6 are fed into the cross-language attention mechanism of the decoder. The generation of the target sentence follows the traditional Transformer decoder framework.

[0129] In order to prove the effectiveness of the proposed method, a large number of experiments were conducted on the Multi-30k, Flickr30kEn-Jp and Fashion-MMT datasets, where the details of the datasets are shown in Table 1. In order to accurately evaluate the performance of the method in the experiment, BLUE and METEOR are used as evaluation indicators. These two indicators can objectively and accurately reflect the effect of machine translation. The experimental results are shown in Tables 2, 3 and 4.

[0130] Table 1 Dataset details

[0131]

[0132] Table 2 Comparison of the model corresponding to the proposed method and the existing SOTA model in Multi-30k

[0133]

[0134] Table 3 is a comparison of the model corresponding to the method of the present invention and the existing SOTA model in Flickr30kEn-Jp

[0135]

[0136] Table 4 is a comparison of the model corresponding to the method of the present invention and the existing SOTA model in Fashion-MMT

[0137]

[0138] The above results all show that the multimodal neural machine translation method based on incremental distillation and regularized semantic consistency proposed in the present invention has achieved SOTA results on all data sets. Tables 2 and 3 show that the proposed method improves translation performance in general fields and shows significant improvements on the ambiguous MSCOCO test set, which shows that the method is robust; the results in Table 4 show that when facing specific fields, the method is still very robust and exceeds the existing SOTA methods. Through the performance comparison on the test set, it can be found that the multimodal Transformer with incremental distillation and regularized semantic consistency can greatly improve the translation performance of the model.

[0139] The present invention proposes a multimodal neural machine translation method based on incremental distillation and regularized semantic consistency. Starting from a pre-trained plain text translation model, the method transfers the prior knowledge of the model to multimodal machine translation. At the same time, it utilizes visual information more effectively through a regularized semantic consistency mechanism to improve the overall performance of translation.

[0140] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A multimodal neural machine translation method based on incremental distillation and regularized semantic consistency, characterized by: The method comprises: Step 1: Obtain a multimodal machine translation dataset and encode text and images to obtain text features and image features; Step 2: Partially mask the image features; Step 3: Send the text, image and mask image features into a module with multiple layers of Transformer for representation; Step 4: Mix the image and mask image representations with the text representations respectively to initially achieve cross-modal fusion; Step 5: Send the text features into the multi-layer Transformer module with frozen parameters for full representation; Step 6: Perform cross-modal aggregation enhancement to increase the complementarity of cross-modal features and input them into the decoder to participate in the generation of the target sentence; Step 7: Perform local point alignment distillation to distill the prior knowledge of the pre-trained plain text model into the multimodal model in a feature-level point-to-point manner; Step 8: Build a regularized semantic alignment module to further capture more comprehensive visual-text alignment details; The distillation loss obtained in Step 9 and Step 7, the semantic consistency loss obtained in Step 8 and the traditional cross entropy loss are combined as the final training target; Step 10: The cross-modal aggregated multimodal features of images and texts obtained in Step 6 are fed into the cross-language attention mechanism of the decoder. The generation of the target sentence follows the traditional Transformer decoder framework.

2. A multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: The step 1 includes: Step 1.1, web crawler, obtain multimodal machine translation dataset; Step 1.2, perform byte pair encoding BPE preprocessing on the text; Step 1.3: Use Resnet-101 to encode the image and obtain the corresponding image feature E v , image feature E v The calculation process is as follows: E v =ResNet(v) Where v represents the image; Step 1.4: Use the Embed & encoding module in Transformer to encode the text to obtain the text feature E x , the calculation process is as follows: E x =Emb(x) Where x represents the text.

3. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In the step Step 2, the image feature E v Perform partial masking and the mask image feature calculation process is as follows: HAVE BEEN vs =F(E v m (1,2,…,k) ) Where m (1,2,…,k) Indicates the position of the selection mask, k represents E v The number of masks in , and F represents the mask operation.

4. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: The step 3 includes: Step 3.1: Send the text features into the stacked multi-layer Transformer to fully represent the text features and obtain the text representation C x ; Step 3.2: Send the image features to the stacked multi-layer Transformer to fully characterize the image features and obtain the image representation C v ; Step 3.3, send the mask image features to the stacked multi-layer Transformer, and use the mask image features as Query, Key and Value for attention mechanism calculation, so that the mask image features E vs Perform full characterization to obtain the mask image representation C vs , the calculation process is as follows: C vs =TransEnc v (AND vs ) Among them, TransEnc v Represents a stacked image Transformer with self-attention and FFN modules.

5. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In the step 4, the calculation process of the cross-modal fusion is as follows: C v_mix =aC x +(1-a)C v C vs_mix =aC x +(1-a)C vs Among them, C x Represents text representation, C v represents the image representation, C vs represents the mask image representation, a represents the coefficient vector for modal mixing, C v_mix represents the mixed image features, C vs_mix Represents mixed mask image features.

6. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In the step 5, the text features are fed into the multi-layer Transformer with frozen parameters, and the text features E x As Query, Key and Value, the attention mechanism is calculated. The calculation process is as follows: C f =TransEnc f (AND x ) Among them, TransEnc f Text Transformer with self-attention and FFN modules showing frozen parameters, C f Represents the fully characterized text features.

7. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In the step 6, the cross-modal aggregation enhancement is to combine the fully characterized text and mixed image features C obtained in step 3 and step 4 v_mix Aggregation enhancement is performed to obtain the multimodal features of images and texts; the multimodal features of images and texts C s The calculation process is as follows: C s =C v_mix +C x Among them, C s Represents the multimodal features of images and texts, which are subsequently input into the cross-language attention mechanism of the Transformer decoder.

8. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: In the step 7, the local point alignment distillation is to perform local point alignment distillation on the results obtained in step 5 and step 6. The distillation loss in the local point alignment distillation process adopts the mean square error loss. The mean square error loss loss MSE The calculation is as follows: Among them, N represents the number of samples, loss MSE represents the mean square error loss.

9. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: The step Step 8 includes: Step8.1, perform gated screening and fusion on the results obtained in Step 4 and Step 5 respectively; Step 8.2: Perform spatial regularization alignment.

10. The multimodal neural machine translation method based on incremental distillation and regularized semantic consistency according to claim 1, characterized in that: The step 8.1 includes: Step 8.1.

1. Mix the image features C v_mix and the fully characterized text representation C f Perform the following operations to obtain complete semantic features The calculation process is as follows: Among them, μ - represents the cross-modal gating weight, M v_mix , is a trainable parameter, Sigmoid represents the Sigmoid function; Step 8.1.2, the mixed mask image feature C vs_mix and the fully characterized text representation C f Perform the following operations to obtain the restored semantic features The calculation process is as follows: Among them, μ + represents the cross-modal gating weight, M vs_mix , is a trainable parameter, Sigmoid represents the Sigmoid function; Step 8.1.3: Semantically align the complete semantic features and the restored semantic features. The semantic consistency loss uses the information noise contrast estimation loss. con The calculation process is as follows: Where log represents the logits function, exp represents the exponential function, τ represents the temperature coefficient used to adjust the similarity score distribution, and Loss con Denotes the information-noise contrast estimation loss.

Images