English translation training method based on multi-modal corpus

Abstract

The invention relates to an English translation training method based on a multi-modal corpus, and the method comprises the steps: collecting text, voice, image and context scene information data, constructing a multi-modal parallel corpus, and generating enhanced training data through a cross-modal alignment technology; a dynamic sampling strategy is adopted, related corpora are automatically weighted according to a field input by a user, and field adaptability is optimized; designing a mixed loss function, and performing joint training in combination with semantic similarity, syntactic structure and cross-modal consistency; an online feedback mechanism is introduced, and a corpus and model parameters are updated in real time through user error correction data. Through the multi-modal corpus and the dynamic training strategy, the problems that a traditional translation model is poor in field adaptability and high in data dependence are solved, and the accuracy, robustness and cross-field adaptability of the translation model are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation and natural language processing, and specifically relates to an English translation training method based on a multimodal corpus, which is suitable for improving the accuracy, robustness and cross-domain adaptability of translation models. Background Art

[0002] Natural language processing is a key research area within computer science and artificial intelligence. It studies how to enable effective communication between humans and computers using natural language, and is a discipline that integrates linguistics, computer science, and mathematics. Within this area, neural machine translation is a crucial task that cannot be ignored. In recent years, neural machine translation has attracted widespread attention from both academia and industry. The excellent performance of neural network machine translation models is due to the availability of large-scale, high-quality, bilingual parallel training corpora.

[0003] However, traditional statistical machine translation relies on a single text corpus, making it difficult to handle complex semantics and cross-language alignment. Neural machine translation models, in particular, rely heavily on high-quality parallel corpora and perform poorly for low-resource languages ​​or specialized domains. Existing corpora are mostly static data, lacking dynamic updates and contextualization capabilities. Therefore, improving the accuracy, robustness, and cross-domain adaptability of translation models is a pressing challenge for those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to overcome the deficiencies of the prior art and provide an English translation training method based on a multimodal corpus.

[0005] To achieve the above objectives, the present invention adopts the following technical solutions: An English translation training method based on a multimodal corpus, the specific steps are as follows: Step 1: Build a multimodal parallel corpus by collecting text, speech, image, and contextual scene information data, and generate enhanced training data through cross-modal alignment technology; Furthermore, the text collection includes obtaining high-quality English-target language aligned text from a public parallel corpus; Furthermore, the speech collection includes using a speech dataset with transcribed text, or generating sentence-by-sentence text from the audio using an automatic speech recognition tool; Furthermore, the paralinguistic information of the speech is retained during the speech collection process for subsequent emotion or intention enhanced translation; Furthermore, the image acquisition includes extracting images and their bilingual description texts from an image-text alignment dataset; Furthermore, professional fields use OCR technology to parse mixed text and image content in scanned documents; Furthermore, the contextual scene information collection includes extracting contextual metadata of the text; Furthermore, after the data is collected, noise is removed and spelling is standardized; Furthermore, the generation of enhanced training data through cross-modal alignment technology includes aligning speech segments and text using a dynamic time warping algorithm to ensure phoneme-level synchronization; Furthermore, a visual model is used to generate a cross-lingual description vector of the image, which is aligned with the text description embedding space; Furthermore, the mapping relationship between speech segments and words is established through phoneme-phoneme alignment; Furthermore, the relevance score between the image region and the text word is calculated based on the attention mechanism; Furthermore, graph neural networks are used to model conversation history or document structure and generate context-aware embedding representations. Furthermore, for the same semantic unit, its text description, pronunciation audio, and physical image are associated to construct a triple; Furthermore, background noise is added to the speech data and synonyms are randomly replaced in the text data to improve robustness.

[0006] Step 2: Adopt a dynamic sampling strategy to automatically weight relevant corpus according to the user input domain to optimize domain adaptability; Furthermore, the dynamic sampling includes using a lightweight domain classifier to perform real-time domain identification on user input text, pre-labeling each data in the corpus with a domain label, and constructing a domain-corpus index; Furthermore, based on the term frequency, syntactic patterns, and contextual keywords of the user input text, the domain probability distribution is output; Furthermore, the automatic weighting of relevant corpora includes pre-calculating the domain label and term density corresponding to each data in the corpus, defining a domain relevance score, and adjusting the corpus sampling probability according to the input domain probability; Furthermore, when the user input text is identified as a high-frequency domain, the following process is automatically triggered: Corpus retrieval: crawl relevant bilingual data from external databases and add them to the corpus after quality filtering; Model fine-tuning: Perform mini-batch gradient updates on the newly added corpus, adjusting only the parameters of the last two layers of the model to avoid catastrophic forgetting; Furthermore, when the user corrects the translation results, such as correcting "plaintiff" from "prosecutor" to "plaintiff", the error type and field label are automatically recorded, and the weight of similar error samples in the corpus of the field is dynamically reduced.

[0007] Step 3: Design a hybrid loss function that combines semantic similarity, syntactic structure, and cross-modal consistency for joint training.

[0008] Furthermore, the semantic similarity loss can use multilingual BERT or SimCSE to enhance cross-language semantic matching; Furthermore, sentence-level embedding can be used to calculate cosine similarity.

[0009] Furthermore, the syntactic structure loss can be implemented through dependency tree edit distance or graph attention matching; Furthermore, the cross-modal consistency loss is a comparison loss between speech features and text.

[0010] Step 4: Introduce an online feedback mechanism to update the corpus and model parameters in real time based on user correction data; Furthermore, we detect user edits to the translation and construct error correction triples: original output, user correction, and input context; Furthermore, the error correction data is inserted into the corpus after quality filtering, and the domain labels are updated synchronously. For high-frequency error patterns, a specific domain fine-tuning mode is triggered; Furthermore, elastic weight consolidation is used to prevent catastrophic forgetting and constrain the update amplitude of important parameters; Furthermore, we receive user edits, ratings, and preferences for generated results through the API, use Kafka or Apache Flink to implement streaming data processing, retrieval, and deduplication, and remove extreme edits where the number of modified characters is less than 3 or greater than 50% to avoid situations where users may accidentally touch them. New feedback data is stored in the Redis cache pool and retained for 24-72 hours.

[0011] Technical effects of the technical solution of the present invention: 1. Improve translation accuracy: Multimodal data enhances contextual understanding and resolves ambiguity.

[0012] 2. Reduce data dependence: Dynamic sampling strategies reduce the need for large-scale corpus in a single field.

[0013] 3. Support continuous learning: Online feedback mechanism enables model self-optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1This is a framework diagram of an English translation training method based on a multimodal corpus of the present invention. DETAILED DESCRIPTION

[0016] To make the objectives, technical solutions, and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other implementations obtained by those of ordinary skill in the art without inventive effort are within the scope of protection of the present invention.

[0017] A multimodal corpus-based English translation training method, characterized in that the method comprises: Step 1: Build a multimodal parallel corpus by collecting text, speech, image, and contextual scene information data, and generate enhanced training data through cross-modal alignment technology; Furthermore, the text collection includes obtaining high-quality English-target language aligned text from a public parallel corpus; Furthermore, the speech collection includes using a speech dataset with transcribed text, or generating sentence-by-sentence text from the audio using an automatic speech recognition tool; Furthermore, the paralinguistic information of the speech is retained during the speech collection process for subsequent emotion or intention enhanced translation; Furthermore, the image acquisition includes extracting images and their bilingual description texts from an image-text alignment dataset; Furthermore, professional fields use OCR technology to parse mixed text and image content in scanned documents; Furthermore, the contextual scene information collection includes extracting contextual metadata of the text; Furthermore, after the data is collected, noise is removed and spelling is standardized; Furthermore, the generation of enhanced training data through cross-modal alignment technology includes aligning speech segments and text using a dynamic time warping algorithm to ensure phoneme-level synchronization; Furthermore, a visual model is used to generate a cross-lingual description vector of the image, which is aligned with the text description embedding space; Furthermore, the mapping relationship between speech segments and words is established through phoneme-phoneme alignment; Furthermore, the relevance score between the image region and the text word is calculated based on the attention mechanism; Furthermore, graph neural networks are used to model conversation history or document structure and generate context-aware embedding representations. Furthermore, for the same semantic unit, its text description, pronunciation audio, and physical image are associated to construct a triple; Furthermore, background noise is added to the speech data and synonyms are randomly replaced in the text data to improve robustness.

[0018] Step 2: Adopt a dynamic sampling strategy to automatically weight relevant corpus according to the user input domain to optimize domain adaptability; Furthermore, the dynamic sampling includes using a lightweight domain classifier to perform real-time domain identification on user input text, pre-labeling each data in the corpus with a domain label, and constructing a domain-corpus index; Furthermore, based on the term frequency, syntactic patterns, and contextual keywords of the user input text, the domain probability distribution is output; Furthermore, automatic weighting of relevant corpora includes pre-calculating the domain label and term density corresponding to each data in the corpus, defining the domain relevance score, and adjusting the corpus sampling probability according to the input domain probability; Furthermore, when the user input text is identified as a high-frequency domain, the following process is automatically triggered: Corpus retrieval: crawl relevant bilingual data from external databases and add them to the corpus after quality filtering; Model fine-tuning: Perform mini-batch gradient updates on the newly added corpus, adjusting only the parameters of the last two layers of the model to avoid catastrophic forgetting; Furthermore, when the user corrects the translation results, such as correcting "plaintiff" from "prosecutor" to "plaintiff", the error type and field label are automatically recorded, and the weight of similar error samples in the corpus of the field is dynamically reduced.

[0019] Step 3: Design a hybrid loss function that combines semantic similarity, syntactic structure, and cross-modal consistency for joint training.

[0020] Furthermore, the semantic similarity loss can use multilingual BERT or SimCSE to enhance cross-language semantic matching; Furthermore, sentence-level embedding can be used to calculate cosine similarity.

[0021] Furthermore, the syntactic structure loss can be implemented through dependency tree edit distance or graph attention matching; Furthermore, the cross-modal consistency loss is a comparison loss between speech features and text.

[0022] Step 4: Introduce an online feedback mechanism to update the corpus and model parameters in real time based on user correction data; Furthermore, we measure the user’s editing of the translation and construct the error correction triples: original output, user correction, and input context; Furthermore, the error correction data is inserted into the corpus after quality filtering, and the domain labels are updated synchronously. For high-frequency error patterns, a specific domain fine-tuning mode is triggered; Furthermore, elastic weight consolidation is used to prevent catastrophic forgetting and constrain the update amplitude of important parameters.

[0023] Furthermore, we receive user edits, ratings, and preferences for generated results through the API, use Kafka or Apache Flink to implement streaming data processing, retrieval, and deduplication, and remove extreme edits where the number of modified characters is less than 3 or greater than 50% to avoid situations where users may accidentally touch them. New feedback data is stored in the Redis cache pool and retained for 24-72 hours.

[0024] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. An English translation training method based on a multimodal corpus, characterized in that: The method comprises: Step 1: Collect text, speech, image, and contextual scene information data, build a multimodal parallel corpus, and generate enhanced training data through cross-modal alignment technology; Step 2: Adopt a dynamic sampling strategy to automatically weight relevant corpus according to the user input domain to optimize domain adaptability; Step 3: Design a hybrid loss function that combines semantic similarity, syntactic structure, and cross-modal consistency for joint training; Step 4: Introduce an online feedback mechanism to update the corpus and model parameters in real time through user correction data.

2. The method according to claim 1, wherein The step 1 includes collecting text, voice, image and contextual scene information data, including: Obtain high-quality English-target language aligned texts from public parallel corpora; Use speech datasets with transcripts, or generate sentence-by-sentence text from audio using automatic speech recognition tools; Preserve the paralinguistic information of speech for subsequent emotion or intention-enhanced translation; Extract images and their bilingual description texts from the image-text alignment dataset; Professional fields use OCR technology to analyze mixed text and image content in scanned documents; Extract contextual metadata from text.

3. The method according to claim 1, wherein In step 1, the enhanced training data is generated by the cross-modal alignment technology, including: Use dynamic time warping algorithm to align speech segments and text to ensure phoneme-level synchronization; Use a vision model to generate cross-lingual description vectors of images, aligned with the text description embedding space; Establishing the mapping relationship between speech segments and words through phoneme-phoneme alignment; Calculate the relevance score between image regions and text words based on the attention mechanism; Use graph neural networks to model conversation history or document structure and generate context-aware embedding representations; For the same semantic unit, associate its text description, pronunciation audio, and physical image to construct a triple; Add background noise to speech data and randomly replace synonyms in text data to improve robustness.

4. The method according to claim 1, wherein The dynamic sampling specifically includes: Use a lightweight domain classifier to perform real-time domain identification on user input text, pre-label each data item in the corpus with a domain label, and build a domain-corpus index; Output domain probability distribution based on term frequency, syntactic patterns, and contextual keywords of the user input text.

5. The method according to claim 1, wherein The automatically weighted related corpus includes: Pre-calculate the domain label and term density corresponding to each data in the corpus, define the domain relevance score, and adjust the corpus sampling probability according to the input domain probability; When the user input text is recognized as a high-frequency domain, the following process is automatically triggered: Corpus retrieval: crawl relevant bilingual data from external databases and add them to the corpus after quality filtering; Model fine-tuning: Perform mini-batch gradient updates on the newly added corpus, adjusting only the parameters of the last two layers of the model to avoid catastrophic forgetting; When users correct translation results, the error type and domain label are automatically recorded, and the weight of similar error samples in the corpus of that domain is dynamically reduced.

6. The method according to claim 1, wherein The design of the hybrid loss function combines semantic similarity, syntactic structure and cross-modal consistency for joint training, including: The semantic similarity loss can use multilingual BERT or SimCSE to enhance cross-lingual semantic matching; The syntactic structure loss can be implemented through dependency tree edit distance or graph attention matching; The cross-modal consistency loss is the comparison loss between speech features and text.

7. The method according to claim 1, wherein The introduction of an online feedback mechanism to update the corpus and model parameters in real time through user correction data includes: Detect user edits to the translation and construct error correction triples: original output, user correction, and input context; The error correction data is inserted into the corpus after quality filtering, and the domain labels are updated synchronously. For high-frequency error patterns, a specific domain fine-tuning mode is triggered; Use elastic weight consolidation to prevent catastrophic forgetting and constrain the update range of important parameters; Receive user edits, ratings, and preferences for generated results through the API. Use Kafka or Apache Flink for streaming data processing, retrieval, and deduplication. Remove extreme edits where the number of modified characters is less than 3 or greater than 50% to avoid accidental user touches. New feedback data is stored in the Redis cache pool and retained for 24-72 hours.

Images