Two-stage semantic aggregation method for multi-modal neural machine translation in specific field

Abstract

The invention relates to a two-stage semantic aggregation method for specific field multi-modal neural machine translation, and belongs to the technical field of natural language processing. The method comprises the following steps of: obtaining embedding characterization of a text and vision; text embedded representation is sent to a traditional Transform model for a first stage, and representation and translation capabilities of general semantics are integrally learned to lay a foundation; carrying out second-stage incremental learning by adopting a comparison decoupling strategy and a multi-level collaborative decoding mechanism, and focusing on specific representation and translation capabilities in the learning field; carrying out consistency constraint on the text representation of the first stage and the text representation of the second stage through a dual-stage representation mild optimization strategy; and the KL divergence and the traditional cross entropy loss are combined as a final training target. According to the method, the problem of fine-grained imbalance of domain graphic and text information in the aspect of domain specific characteristics caused by high domain term specialty, low occurrence frequency and sparse distribution is solved.

Description

Technical Field

[0001] The present invention relates to a two-stage semantic aggregation method for multimodal neural machine translation in a specific field, and belongs to the technical field of natural language processing. Background Art

[0002] Domain-specific multimodal neural machine translation (DMMT) aims to improve translation quality within a specific domain by introducing auxiliary modalities, such as images. Many existing studies treat domain-specific text translation and general-domain text translation as equivalent tasks, still using general-domain image-text fusion methods to address domain representation and adapt to domain data.

[0003] However, due to the high degree of specialization, low frequency, and sparse distribution of domain terms, domain-specific features often exhibit fine-grained imbalances in domain-specific graph-text information. Existing models struggle to fully capture these key domain-specific features, leading to biased domain representations and thus affecting the accuracy of domain term translation. Therefore, the problem of loss of domain specificity persists in the process of generating domain-specific terms.

[0004] DMNMT consists of two subtasks: one is relatively simple and the other is more complex. General domain words are generally easier to understand and translate, while domain-specific words face greater challenges. However, domain-specific words are closely connected to the semantics of their context. Therefore, a word-level translation strategy that proceeds step by step in sequence: transitioning from early non-domain words to more challenging domain-specific words, can effectively address domain-specific challenges. Through a cross-domain incremental learning strategy, the model first handles the simpler subtasks and then shifts its focus to domain-specific content, achieving a balance between the representation and translation capabilities of general and domain words. Summary of the Invention

[0005] This paper proposes a two-stage semantic aggregation method for domain-specific multimodal neural machine translation, addressing the imbalance in fine-grained domain characteristics between domain-specific image and text information in the aforementioned technologies. This approach aims to enhance multimodal domain representation, prevent loss of domain-specificity, and improve the model's ability to perceive fine-grained domain image and text details.

[0006] The technical solution of the present invention is a two-stage semantic aggregation method for multimodal neural machine translation in a specific field, the method comprising:

[0007] Step 1: Obtain a multimodal machine translation dataset and obtain text and visual embedding representations;

[0008] Step 2: Feed the text embedding representation into the traditional Transformer model for the first stage, and learn the overall general semantic representation and translation capabilities to lay the foundation;

[0009] Step 3: Based on the first phase, a second phase of incremental learning is conducted using a contrastive decoupling strategy and a multi-level collaborative decoding mechanism, focusing on learning domain-specific representation and translation capabilities.

[0010] Step 4: Consistency constraints are imposed on the text representations from the first and second stages through the dual-stage representation balance optimization strategy loss, so that the model achieves a balance between the representation and translation capabilities of general terms and domain terms.

[0011] Step 5: Combine the two-stage representation balance optimization strategy loss obtained in Step 4 with the traditional cross entropy loss as the final training target.

[0012] Furthermore, the Step 1 includes:

[0013] Step 1.1: Use web crawlers to obtain a multimodal machine translation dataset.

[0014] Step 1.2, perform bytepair encoding (BPE) preprocessing on the text;

[0015] Step 1.3: Use the Embed & encoding module in Transformer to encode the text to obtain the source language embedding representation E x ∈R n×d , the embedding representation E of the target language y ∈R m×d , the calculation process is as follows:

[0016] E x =Emb x (x k )

[0017] E y =Emb y (y k )

[0018] where x k Indicates the source language text, y k Represents the target language text;

[0019] Step 1.4: Use Resnet-101 and multi-layer perceptron (MLP) to project the visual features into the hidden space to obtain the visual embedding representation E v ∈R 49×d , the calculation process is as follows:

[0020] E v =MLP(ResNet(v k ))

[0021] where vk Represents visual features.

[0022] Furthermore, in Step 2, the traditional Transformer text translation model is used to learn the representation and translation capabilities of general text. The encoding and decoding process is shown as follows:

[0023]

[0024] in, and represents the l-th layer Transformer encoder and masked self-attention layer, represents the l-th layer cross-language attention module; l = 1, ..., L represents the layer index. Each Transformer encoder and cross-language attention module consists of two sublayers: the multi-head attention layer (MHA) and the feedforward neural network layer (FFN); when l = 1, we set Represents the hidden states of the source and target texts; then, It is applied to a fully connected feed-forward network (FFN) and a softmax layer to generate probability distributions.

[0025] Furthermore, the Step 3 includes:

[0026] Step 3.1, modal feature extraction;

[0027] Step 3.2: Aggregate domain-specific cross-modal interactions and calculate the correspondence between fine-grained images and texts.

[0028] Step 3.3: Contrast decoupling to enhance domain-specific representation;

[0029] Step 3.4, multi-level collaborative decoding;

[0030] Furthermore, the Step 3.1 includes:

[0031] Step 3.1.1. Use stacked Transformer encoders to extract domain text representations as follows:

[0032]

[0033] in, represents the lth layer of the text Transformer encoder layer, l=1,…,L represents the layer index, when l=1, we set Source language text representation

[0034] Step 3.1.2: Extract relevant visual representations through a cross-modal attention mechanism, as shown below:

[0035]

[0036] Among them, CrsAtt l (·) represents the cross-modal attention mechanism, Represents a weighted visual representation.

[0037] Furthermore, Step 3.2 includes:

[0038] Step 3.2.1. Represent the source language text and weighted visual representation Through cross-modal interaction through the gated network, fine-grained image-text correspondence is calculated; as shown below:

[0039]

[0040] in, is a cross-modal feature representation; Ω is calculated as follows:

[0041]

[0042] Among them, Sigmoid(·) represents the sigmoid function, are trainable model parameters;

[0043] Step 3.2.2: Obtain the aggregated domain-specific multimodal representation through concatenation, as shown below:

[0044]

[0045] in, represents domain-specific multimodal features, and || represents a concatenation operation.

[0046] Furthermore, the Step 3.3 includes:

[0047] Step 3.3.1, before calculating the interaction between graphics and text ( and )back The cosine similarity is used to measure the distribution of domain-related and domain-independent information retained during the image-text interaction process; the similarity matrix is calculated as follows:

[0048]

[0049]

[0050] in, R n×dRepresent text and visual similarity vectors respectively; high similarity represents the domain-related part of the image-text interaction, while low similarity is the opposite. express Length of the module;

[0051] Step 3.3.2: Use a Gaussian filter to remove local noise, enhance the contrast of the image-text similarity matrix by strengthening the differences in local areas, and achieve decoupling of domain-related features, as shown below:

[0052]

[0053] Among them, G σ (·) represents a Gaussian filter, Respectively represent the similarity matrix of text image contrast enhancement;

[0054] Step 3.3.3: Perform weighted representation on text and images to enhance domain-related features and suppress domain-irrelevant information, as shown below:

[0055]

[0056] Among them, Norm(·) represents the normalization operation, R n×d Domain-enhanced representations for text and vision, respectively.

[0057] Furthermore, the Step 3.4 includes:

[0058] Step 3.4.1. Use the Transformer decoder layer with an additional mask mechanism to obtain the representation of the target sentence as follows:

[0059]

[0060] in, represents the l-th layer of the text Transformer decoder, l = 1, ..., L represents the layer index; when l = 1, we set Target language text representation

[0061] Step 3.4.2, Adaptive Attention Guided by Domain-Enhanced Representation: Use the adaptive attention mechanism as a hub between different modalities, while preserving the unique characteristics of each modality's domain-specific deep representations while avoiding interference between different information densities, as shown below:

[0062]

[0063]

[0064] Among them, Softmax(·) represents the softmax operation, is a learnable projection matrix, Represents the context features of the lth layer;

[0065] Step 3.4.3, Cross-attention guided by modal interaction features: For domain-specific multimodal features The synergy of multi-level domain representations is achieved through a standard cross-lingual attention module as follows:

[0066]

[0067] in, represents the cross-lingual attention module, Represents the lth layer context features that coordinate multi-level image and text domain features;

[0068] Step 3.4.4. Use an FFN to generate the lth layer intermediate semantic representation F l :

[0069]

[0070] Step 3.4.5. Generate the probability distribution of the target sentence through the softmax layer, as shown below:

[0071] P(y|x,v)=Softmax(W p F L +b)

[0072] Where x represents the source language sentence, y represents the target sentence, and v represents the visual input. p ,b is the parameter of the Softmax layer;

[0073] Furthermore, in Step 4, the calculation process of the two-stage characterization balance optimization strategy loss is as follows:

[0074]

[0075] Where KL(·) represents the KL divergence;

[0076] Furthermore, in Step 5, the KL divergence obtained in Step 4 is combined with the traditional cross entropy loss as the final training target. The calculation process is as follows:

[0077] loss=loss ce +α·loss kl

[0078] Among them, loss cerepresents the traditional cross-entropy translation loss, and α is the loss hyperparameter.

[0079] The present invention enhances multimodal domain representation through a two-stage incremental training mechanism, avoids the loss of domain specificity, and improves the model's perception of fine-grained domain image and text details. First, the traditional Transformer text translation model is used as the first stage, so that the model as a whole learns the representation and translation capabilities of general semantics, laying the foundation for the second stage. Then, the second stage is implemented through incremental learning: first, the domain image and text representations are aggregated across modal interactions to calculate fine-grained image-text correspondences; then, the contrast decoupling module is used to measure the distribution of domain-related and domain-irrelevant information retained during the image-text interaction process; then, local contrast enhancement is used to achieve the enhancement of domain-related features and the suppression of domain-irrelevant information; then, the adaptive attention module guided by the domain enhancement representation is used to retain the unique characteristics of the specific deep-level representation of each modal domain while avoiding interference between different information densities, and at the same time, the cross-attention module guided by the modal interaction features achieves the synergy of multi-level domain representations; finally, a two-stage representation balance optimization strategy is used to enable the model to achieve a balance between the representation and translation capabilities of general words and domain words. This invention mainly solves the problem of fine-grained imbalance in domain-specific features of domain graphic information due to the high professionalism, low frequency of occurrence and sparse distribution of domain terms, which leads to deviation in domain representation. It aims to enhance multimodal domain representation, avoid the loss of domain specificity, and improve the model's perception of fine-grained domain graphic details.

[0080] The beneficial effects of the present invention are:

[0081] 1. This paper proposes a two-stage semantic aggregation method for domain-specific multimodal neural machine translation, which solves the key problems faced by domain translation through a two-stage incremental training mechanism;

[0082] 2. This paper designs a contrast decoupling module guided by cross-modal interaction to enhance the domain context representation capability. At the same time, it introduces a multi-level collaborative decoding mechanism to dynamically integrate multi-level domain representations to jointly guide target sentence generation.

[0083] 3. The present invention applies a two-stage representation balance optimization strategy to coordinate the translation performance of domain terms and general vocabulary, thereby improving the overall translation quality;

[0084] 4. The method of the present invention is implemented based on the Fairseq open source system, which is relatively stable and has strong text processing capabilities, providing a strong guarantee for multimodal machine translation for specific fields;

[0085] 5. The method proposed in this invention has achieved SOTA results on the test sets of multiple public datasets. Visual experiments and case analysis have demonstrated the effectiveness and superiority of the method of this invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 is a flow chart of the present invention;

[0087] Figure 2 It is a detailed process framework diagram of the present invention. DETAILED DESCRIPTION

[0088] Example 1: In response to the problems faced by multimodal machine translation in specific domains, such as the imbalance of fine-grained domain features and the deviation of domain representation, the present invention proposes a two-stage semantic aggregation method for multimodal neural machine translation in specific domains. Through staged learning, the model's perception and modeling capabilities of fine-grained multimodal features in the domain are enhanced, thereby improving the model's translation performance for domain terms.

[0089] like Figure 1-Figure 2 As shown, a two-stage semantic aggregation method for domain-specific multimodal neural machine translation includes:

[0090] Step 1: Obtain a multimodal machine translation dataset and obtain text and visual embedding representations: Download the images and text from the Fashion-MMT dataset for e-commerce and divide the data into training, validation, and test sets. Obtain the raw data of Multi30k, a total of 59,000 entries, directly from the public download link for the Multi30k dataset. Perform bytepair encoding (BPE) preprocessing on the text.

[0091] Furthermore, the Step 1 includes:

[0092] Step 1.1: Use web crawlers to obtain a multimodal machine translation dataset.

[0093] Step 1.2, perform bytepair encoding (BPE) preprocessing on the text;

[0094] Step 1.3: Use the Embed & encoding module in Transformer to encode the text to obtain the source language embedding representation E x ∈R n×d , the embedding representation E of the target language y ∈R m×d , the calculation process is as follows:

[0095] E x =Emb x (xk )

[0096] E y =Emb y (y k )

[0097] where x k Indicates the source language text, y k Represents the target language text.

[0098] Step 1.4: Use Resnet-101 and multi-layer perceptron (MLP) to project the visual features into the hidden space to obtain the visual embedding representation E v ∈R 49×d , the calculation process is as follows:

[0099] E v =MLP(ResNet(v k ))

[0100] where v k Represents visual features;

[0101] Step 2: Feed the text embedding representation into the traditional Transformer model for the first stage, and learn the overall general semantic representation and translation capabilities to lay the foundation;

[0102] Furthermore, in Step 2, the traditional Transformer text translation model is used to learn the representation and translation capabilities of general text. The encoding and decoding process is shown as follows:

[0103]

[0104] in, and represents the l-th layer Transformer encoder and masked self-attention layer, Represents the l-th layer cross-language attention module. l=1,…,L represents the layer index. Each Transformer encoder and cross-language attention module consists of two sublayers: multi-head attention layer (MHA) and feedforward neural network layer (FFN). When l=1, we set Represents the hidden state of the source text and the target text. It is applied to a fully connected feed-forward network (FFN) and a softmax layer to generate probability distributions.

[0105] Step 3: Based on the first phase, a second phase of incremental learning is conducted using a contrastive decoupling strategy and a multi-level collaborative decoding mechanism, focusing on learning domain-specific representation and translation capabilities.

[0106] Furthermore, the Step 3 includes:

[0107] Step 3.1, modal feature extraction;

[0108] Step 3.2: Aggregate domain-specific cross-modal interactions and calculate the correspondence between fine-grained images and texts.

[0109] Step 3.3: Contrast decoupling to enhance domain-specific representation;

[0110] Step 3.4, multi-level collaborative decoding;

[0111] Furthermore, the Step 3.1 includes:

[0112] Step 3.1.1. Use stacked Transformer encoders to extract domain text representations as follows:

[0113]

[0114] in, represents the lth layer of the text Transformer encoder layer, l=1,…,l represents the layer index. When l=1, we set Source language text representation

[0115] Step 3.1.2: Extract relevant visual representations through a cross-modal attention mechanism, as shown below:

[0116]

[0117] Among them, CrsAtt l (·) represents the cross-modal attention mechanism, represents weighted visual representation;

[0118] Furthermore, Step 3.2 includes:

[0119] Step 3.2.1. Represent the source language text and weighted visual representation Through the gated network, cross-modal interaction is performed to calculate fine-grained image-text correspondences, as shown below:

[0120]

[0121] in, is a cross-modal feature representation. Ω is calculated as follows:

[0122]

[0123] Among them, Sigmoid(·) represents the sigmoid function, are trainable model parameters;

[0124] Step 3.2.2: Obtain the aggregated domain-specific multimodal representation through concatenation, as shown below:

[0125]

[0126] in, represents domain-specific multimodal features, and || represents a connection operation;

[0127] Furthermore, the Step 3.3 includes:

[0128] Step 3.3.1, before calculating the interaction between graphics and text ( and )back The cosine similarity is used to measure the distribution of domain-related and domain-independent information retained during the image-text interaction process. The similarity matrix is calculated as follows:

[0129]

[0130] in, Represent text and visual similarity vectors respectively. High similarity represents the domain-related parts in the image-text interaction, while low similarity is the opposite. express Length of the module;

[0131] Step 3.3.2: Use a Gaussian filter to remove local noise, enhance the contrast of the image-text similarity matrix by strengthening the differences in local areas, and achieve decoupling of domain-related features, as shown below:

[0132]

[0133] Among them, G σ (·) represents a Gaussian filter, R n×d Respectively represent the similarity matrix of text image contrast enhancement;

[0134] Step 3.3.3: Perform weighted representation on text and images to enhance domain-related features and suppress domain-irrelevant information, as shown below:

[0135]

[0136] Among them, Norm(·) represents the normalization operation, R n×d Domain-enhanced representations for text and vision, respectively;

[0137] Furthermore, the Step 3.4 includes:

[0138] Step 3.4.1. Use the Transformer decoder layer with an additional mask mechanism to obtain the representation of the target sentence as follows:

[0139]

[0140] in, represents the lth layer of the text Transformer decoder, l=1,…,L represents the layer index. When l=1, we set Target language text representation

[0141] Step 3.4.2, Adaptive Attention Guided by Domain-Enhanced Representation: Use the adaptive attention mechanism as a hub between different modalities, while preserving the unique characteristics of each modality's domain-specific deep representations while avoiding interference between different information densities, as shown below:

[0142]

[0143] Among them, Softmax(·) represents the softmax operation, is a learnable projection matrix, Represents the context features of the lth layer;

[0144] Step 3.4.3, Cross-attention guided by modal interaction features: For domain-specific multimodal features The synergy of multi-level domain representations is achieved through a standard cross-lingual attention module as follows:

[0145]

[0146] in, represents the cross-lingual attention module, Represents the lth layer context features that coordinate multi-level image and text domain features;

[0147] Step 3.4.4. Use an FFN to generate the lth layer intermediate semantic representation F l :

[0148]

[0149] Step 3.4.5. Generate the probability distribution of the target sentence through the softmax layer, as shown below:

[0150] P(y|x,v)=Softmax(W p FL +b)

[0151] Where x represents the source language sentence, y represents the target sentence, and v represents the visual input. p ,b is the parameter of the Softmax layer;

[0152] Step 4: Consistency constraints are imposed on the text representations from the first and second stages through the dual-stage representation balance optimization strategy loss, so that the model achieves a balance between the representation and translation capabilities of general terms and domain terms.

[0153] Furthermore, in Step 4, the calculation process of the two-stage characterization balance optimization strategy loss is as follows:

[0154]

[0155] Where KL(·) represents the KL divergence;

[0156] Step 5: Combine the Kullback-Leibler (KL) divergence and the traditional cross entropy loss obtained in Step 4 as the final training target.

[0157] Furthermore, in Step 5, the KL divergence obtained in Step 4 is combined with the traditional cross entropy loss as the final training target. The calculation process is as follows:

[0158] loss=loss ce +α·loss kl

[0159] Among them, loss ce represents the traditional cross-entropy translation loss, and α is the loss hyperparameter.

[0160] To demonstrate the effectiveness of our proposed method, we conducted extensive experiments on the Multi-30k, ENNT, and Fashion-MMT datasets. The dataset details are shown in Table 1. To accurately evaluate the performance of our method, we used BLUE and METEOR as evaluation metrics. These two metrics can objectively and accurately reflect the effectiveness of machine translation. The experimental results are shown in Tables 2, 3, and 4.

[0161] Table 1 shows the details of the dataset

[0162]

[0163]

[0164] Table 2 is a comparison of the model corresponding to the proposed method and the existing SOTA model in Multi-30k

[0165]

[0166] Table 3 is a comparison of the model corresponding to the method of the present invention and the existing SOTA model in EMMT

[0167]

[0168] Table 4 is a comparison of the model corresponding to the method of the present invention and the existing SOTA model in Fashion-MMT

[0169]

[0170]

[0171] The above results all show that the two-stage semantic aggregation method for multimodal neural machine translation for specific domains proposed in the present invention has achieved SOTA results on all datasets. The results in Tables 3 and 4 show that when oriented to specific domains, this method surpasses the existing SOTA methods on specific domain datasets, verifying the effectiveness of this method. Table 2 shows that the proposed method also improves translation performance in general domains, verifying that the method has strong generalization; at the same time, it shows significant improvements on the ambiguous MSCOCO test set, which shows that the method is robust. Through performance comparison on the test set, it can be found that the multimodal Transformer with two-stage incremental semantic aggregation can greatly improve the translation performance of the model.

[0172] This paper proposes a two-stage semantic aggregation method for domain-specific multimodal neural machine translation. This method enhances the model's perception and modeling capabilities of domain-specific fine-grained multimodal features through staged learning.

[0173] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A two-stage semantic aggregation method for domain-specific multimodal neural machine translation, characterized by: The method comprises: Step 1: Obtain a multimodal machine translation dataset and obtain text and visual embedding representations; Step 2: Send the text embedding representation into the traditional Transformer model for the first stage, which is used to lay the foundation for overall learning of general semantic representation and translation capabilities; Step 3: Based on the first stage, a second stage of incremental learning is conducted using a contrastive decoupling strategy and a multi-level collaborative decoding mechanism to focus on learning domain-specific representation and translation capabilities. Step 4: Consistency constraints are applied to the text representations from the first and second stages using a two-stage representation balance optimization strategy loss to achieve a balance between the representation and translation capabilities of the model for general and domain terms. Step 5: Combine the two-stage representation balance optimization strategy loss obtained in Step 4 with the traditional cross entropy loss as the final training target.

2. The two-stage semantic aggregation method for domain-specific multimodal neural machine translation according to claim 1 is characterized by: Step 1 includes: Step 1.1: Use web crawlers to obtain a multimodal machine translation dataset. Step 1.2, perform bytepair encoding (BPE) preprocessing on the text; Step 1.3: Use the Embed & encoding module in Transformer to encode the text to obtain the source language embedding representation E x ∈R n×d , the embedding representation E of the target language y ∈R m×d , the calculation process is as follows: E x =Emb x (x k ) AND y =Emb y (and k ) where x k Indicates the source language text, y k Represents the target language text; Step 1.4: Use Resnet-101 and multi-layer perceptron MLP to project the visual features into the hidden space and obtain the visual embedding representation E. v ∈R 49×d , the calculation process is as follows: E v =MLP(ResNet(v k )) where v k Represents visual features.

3. The two-stage semantic aggregation method for domain-specific multimodal neural machine translation according to claim 1, characterized in that: In Step 2, the traditional Transformer text translation model is used to learn the representation and translation capabilities of general text. The encoding and decoding process is expressed as follows: in, and represents the l-th layer Transformer encoder and masked self-attention layer, Represents the l-th layer cross-language attention module, l = 1, ..., L represents the layer index, each Transformer encoder and cross-language attention module consists of two sublayers: the multi-head attention layer MHA and the feed-forward neural network layer FFN; when l = 1, set Represent the hidden states of the source text and target text respectively, and then Applied to a fully connected feedforward network FFN and a softmax layer to generate probability distribution.

4. The two-stage semantic aggregation method for domain-specific multimodal neural machine translation according to claim 1, characterized in that: Step 3 includes: Step 3.1, modal feature extraction; Step 3.2: Aggregate domain-specific cross-modal interactions and calculate the correspondence between fine-grained images and texts. Step 3.3: Contrast decoupling to enhance domain-specific representation; Step 3.4, multi-level collaborative decoding.

5. The two-stage semantic aggregation method for domain-specific multimodal neural machine translation according to claim 1, characterized in that: Step 3.1 includes: Step 3.1.

1. Use stacked Transformer encoders to extract domain text representations as follows: in, Represents the l-th layer of the text Transformer encoder layer, l=1,…,L represents the layer index, when l=1, set Source language text representation Step 3.1.2: Extract relevant visual representations through a cross-modal attention mechanism, as shown below: Among them, CrsAtt l (·) represents the cross-modal attention mechanism, represents a weighted visual representation.

6. The two-stage semantic aggregation method for domain-specific multimodal neural machine translation according to claim 1, characterized in that: Step 3.2 includes: Step 3.2.

1. Represent the source language text and weighted visual representation Through cross-modal interaction through the gated network, fine-grained image-text correspondence is calculated; as shown below: in, is a cross-modal feature representation, and Ω is calculated as follows: Among them, Sigmoid(·) represents the sigmoid function, are trainable model parameters; Step 3.2.2: Obtain the aggregated domain-specific multimodal representation through concatenation, as shown below: in, represents domain-specific multimodal features, and || represents a concatenation operation.

7. The two-stage semantic aggregation method for domain-specific multimodal neural machine translation according to claim 1, characterized in that: Step 3.3 includes: Step 3.3.

1. Before calculating the interaction between graphics and text and After interacting with the image and text The cosine similarity is used to measure the distribution of domain-related and domain-independent information retained during the image-text interaction process; the similarity matrix is calculated as follows: in, R n×d Represent text and visual similarity vectors respectively. High similarity represents the domain-related part of the image-text interaction, while low similarity is the opposite. express Length of the module; Step 3.3.2: Use a Gaussian filter to remove local noise, enhance the contrast of the image-text similarity matrix by strengthening the differences in local areas, and achieve decoupling of domain-related features, as shown below: Among them, G σ (·) represents a Gaussian filter, R n×d Respectively represent the similarity matrix of text image contrast enhancement; Step 3.3.3: Perform weighted representation on text and images to enhance domain-related features and suppress domain-irrelevant information, as shown below: Among them, Norm(·) represents the normalization operation, R n×d Domain-enhanced representations for text and vision, respectively.

8. The two-stage semantic aggregation method for domain-specific multimodal neural machine translation according to claim 1, characterized in that: Step 3.4 includes: Step 3.4.

1. Use the Transformer decoder layer with an additional mask mechanism to obtain the representation of the target sentence as follows: in, Represents the l-th layer of the text Transformer decoder, l=1,…,L represents the layer index, when l=1, set Target language text representation Step 3.4.2, Adaptive Attention Guided by Domain-Enhanced Representation: Use the adaptive attention mechanism as a hub between different modalities, while preserving the unique characteristics of each modality's domain-specific deep representations while avoiding interference between different information densities, as shown below: Among them, Softmax(·) represents the softmax operation, is a learnable projection matrix, Represents the context features of the lth layer; Step 3.4.3, Cross-attention guided by modal interaction features: For domain-specific multimodal features The synergy of multi-level domain representations is achieved through a standard cross-lingual attention module as follows: in, represents the cross-lingual attention module, Represents the lth layer context features that coordinate multi-level image and text domain features; Step 3.4.

4. Use an FFN to generate the lth layer intermediate semantic representation F l : Step 3.4.

5. Generate the probability distribution of the target sentence through the softmax layer, as shown below: P(y|x,v)=Softmax(W p F L +b) Among them, x represents the source language sentence, y represents the target sentence, and v represents the visual input; W p ,b is the parameter of the Softmax layer.

9. The two-stage semantic aggregation method for domain-specific multimodal neural machine translation according to claim 1, characterized in that: In Step 4, the calculation process of the two-stage characterization balance optimization strategy loss is as follows: Here, KL(·) represents the KL divergence.

10. The two-stage semantic aggregation method for domain-specific multimodal neural machine translation according to claim 1, characterized in that: In Step 5, the two-stage representation balance optimization strategy loss obtained in Step 4, namely the KL divergence, is combined with the traditional cross entropy loss as the final training target. The calculation process is as follows: loss=loss ce +α·loss kl Among them, loss ce represents the traditional cross-entropy translation loss, and α is the loss hyperparameter.

Images