Multilingual word alignment method based on attention mechanism

Abstract

The invention provides a multilingual word alignment method based on an attention mechanism, and belongs to the technical field of natural language processing. The multilingual word alignment method based on the attention mechanism comprises the following steps: data collection: collecting a large amount of multilingual text data of Cantonese, Chinese and English; text representation: modeling the text; constructing a large model: adjusting and optimizing the large model; multilingual pre-training: pre-training the large model by using multilingual data; an attention mechanism: introducing a multilingual attention mechanism, and calculating attention scores among different language texts; according to the attention score, the alignment relation between the words is determined; by introducing a multilingual attention mechanism, the semantic relationship between different language texts can be better captured, so that the alignment accuracy is improved, a pre-trained large model capable of processing multiple languages of Cantonese, Chinese and English is realized, and multiple languages are processed at the same time.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and more specifically, to a multilingual word alignment method based on an attention mechanism. Background Art

[0002] At present, with the development of economy, the demand for multilingual information exchange is increasing, and multilingual information processing is becoming more and more important. Cross-language unified semantic representation pre-training technology has become a research hotspot in the field of natural language processing. In the field of natural language processing, pre-training large models has become a very effective method to improve the performance of various downstream tasks; The above-mentioned existing technical solutions have the following defects: most of the existing pre-trained large models only support a single language and cannot fully utilize the advantages of multilingual data. Summary of the invention

[0003] In order to make up for the above shortcomings, this application provides a multilingual word alignment method based on an attention mechanism, aiming to improve the problem of not being able to fully utilize multilingual data.

[0004] The present application embodiment provides a multilingual word alignment method based on an attention mechanism, comprising the following steps: S1. Data collection: Collect a large amount of multilingual text data in Cantonese, Chinese and English; S2. Text representation: Model the text and obtain the feature representation of the text; specifically, any one of the convolutional neural network (CNN), recurrent neural network (RNN) or long short-term memory network (LSTM) is used to model the input text. When the convolutional neural network (CNN) is used, the input text is converted into a sequence of word vectors, and then the word vector sequence is input into the CNN for feature extraction to obtain the feature representation of the text; the convolutional neural network (CNN) is used to model the text, and the accuracy and robustness of the alignment are improved by introducing a multilingual attention mechanism.

[0005] S3. Large model architecture: Adjust and optimize the large model to adapt to the processing of multilingual data; S4. Multilingual pre-training: Use multilingual data to pre-train the large model to learn the common features and rules of the language. During the pre-training process, a variety of techniques are used to improve the performance of the large model. S5. Attention mechanism: A multilingual attention mechanism is introduced to calculate the attention scores between texts in different languages, so as to determine the alignment relationship between words. Specifically, the feature representation of the source language text and the feature representation of the target language text are input into the attention mechanism, the attention score is calculated, and then the alignment relationship between words is determined based on the attention score.

[0006] S6, alignment result: According to the attention score, the alignment relationship between words is determined, and the alignment result is output; specifically, the attention score is converted into an alignment matrix, and then the alignment relationship between words is determined according to the alignment matrix, and the alignment result is output.

[0007] S7. Fine-tuning and optimization: Evaluate and compare large models, verify the performance and effect of large models, and optimize and improve them; S8. Large model evaluation: Use a variety of evaluation indicators to evaluate and compare large models to verify the performance and effectiveness of large models.

[0008] In a preferred embodiment of the present invention, the multilingual text data in S1 includes news, novels, papers and social media; completing data collection and organization to construct a multilingual data set; Verify data quality using methods including data cleaning, screening and verification, as well as cross-validation using multiple data sources to address issues such as inaccurate, incomplete or noisy data.

[0009] In a preferred embodiment of the present invention, the large model described in S3 adopts a Transformer-based architecture, which includes BERT and GPT; the architecture of the pre-trained large model is designed and implemented, and preliminary experiments and verifications are performed; Preprocessing of overfitting of large models involves increasing the amount of data, using regularization techniques, and early stopping methods to address overfitting that may be caused by insufficient data or excessive complexity of the large model.

[0010] In a preferred embodiment of the present invention, the multiple technologies described in S4 include multi-task learning and adversarial training; multi-language pre-training of the large model, and fine-tuning and optimization to improve the performance of the large model on downstream tasks; this method can automatically learn the grammar and semantic relations of the language, has better processing capabilities for complex language structures and semantic relations, and can process multiple languages ​​​​at the same time.

[0011] This method models text by introducing an attention mechanism and determines the alignment relationship by calculating the attention scores between words.

[0012] Multilingual data processing uses data augmentation, balanced datasets, or multi-task learning methods to address the possible differences in the amount of data for different languages, which may result in large models having poor learning effects on certain languages.

[0013] In a preferred embodiment of the present invention, on the basis of pre-training, fine-tuning and optimization are performed for downstream tasks, using intent recognition and text classification; using transfer learning and adaptive learning techniques to improve the performance of large models on specific tasks; without a large amount of manual annotation and prior knowledge, the grammar and semantic relationships of the language can be automatically learned. It has better processing capabilities for complex language structures and semantic relationships, and can improve the accuracy of word alignment.

[0014] The interpretability of large models can be improved by using visualization techniques and feature importance analysis methods. This is because large pre-trained models are usually very complex and their decision-making process is difficult to explain.

[0015] In a preferred embodiment of the present invention, the multiple evaluation indicators in S8 include accuracy, recall rate and F1 value.

[0016] In a preferred embodiment of the present invention, the following steps are also included: Grammatical word formation analysis: Analyze the grammatical word formation of the source language and the target language to understand their vocabulary, part of speech, and syntactic characteristics; Self-supervised word alignment: Design a self-supervised word alignment method based on the attention mechanism to achieve word-level alignment by calculating the similarity between the source language and the target language word vectors. By strengthening the word-level semantic alignment between different languages, the translation quality of the neural machine translation model is improved. Multilingual representations are more robust and can better cope with language variation and noise.

[0017] In a preferred embodiment of the present invention, the following steps are also included: Multilingual representation: Use word alignment information to build more informative and robust multilingual representations to improve the performance of cross-language tasks.

[0018] Word alignment attention layer: A word alignment attention layer is added to the encoder and decoder of the neural machine translation model to obtain high-quality word alignment matrices between the source language and the target language, providing word alignment information for context modeling training. The self-supervised word alignment method does not require manual annotation of data, reducing the cost of data preparation. This technology can be applied to a variety of cross-language tasks and has strong scalability.

[0019] This method improves translation quality and provides better support for cross-language communication. It uses multilingual representations to classify cross-language texts and improves classification accuracy. It helps users quickly and accurately retrieve the required content from multilingual information. It aims to form multilingual representations with richer information and stronger robustness, and provide better support for cross-language tasks such as neural machine translation.

[0020] Improve language understanding ability: By utilizing the language understanding ability of the big model, we can better understand the user's intentions and needs, and provide more accurate and useful answers; the big model has strong language understanding and generation capabilities, and can learn and train a large amount of text data. By utilizing the information processing and fusion capabilities of the big model, different information sources can be integrated and analyzed to better understand the user's intentions and needs.

[0021] Improve the coherence of the conversation: By utilizing the information processing and fusion capabilities of the large model, different information sources can be better integrated and analyzed, thereby improving the coherence and logic of the conversation. The integrated processing method of the large model is adopted to process the conversation task as a whole, thus avoiding the problem of error accumulation.

[0022] Improve generation capabilities: By leveraging the generation capabilities of large models, more natural and fluent responses can be generated, thereby improving user satisfaction. Improve the robustness of the system: By avoiding the problem of error accumulation, the robustness of the system is improved so that it can better cope with various complex dialogue scenarios. By making full use of the information processing and fusion capabilities of the large model and avoiding the problem of error accumulation, the overall performance of the intelligent dialogue system is significantly improved. Specifically, this is achieved through the following aspects.

[0023] By making full use of the powerful information processing and fusion capabilities of the large model, and effectively avoiding the error accumulation problem caused by the modular method, the overall performance of the intelligent dialogue system is significantly improved. It has broad application prospects and brings new opportunities and challenges to the development of intelligent dialogue systems.

[0024] In a preferred embodiment of the present invention, the attention mechanism can dynamically assign weights based on the semantics, context and other information of the words to capture the alignment relationship between words.

[0025] Beneficial Effects

[0026] 1. By introducing a multilingual attention mechanism, the semantic relationship between texts in different languages ​​can be better captured, thereby improving the accuracy of alignment and realizing a pre-trained large model that can handle multiple languages, including Cantonese, Chinese, and English. It can also handle multiple languages ​​at the same time with better versatility and scalability.

[0027] 2. It is robust to noise and ambiguity, and can accurately align words in complex language environments, improving the model's ability to understand multilingual data and forming a richer and more accurate multilingual representation.

[0028] 3. It does not require a large amount of labeled data, has good generalization ability, and can be applied to word alignment tasks in multiple language pairs; it can effectively improve the accuracy and robustness of alignment, enhance the robustness of the model, and enable it to better cope with noise and interference. DETAILED DESCRIPTION

[0029] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0030] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0031] Improve language understanding ability: By utilizing the language understanding ability of the big model, we can better understand the user's intentions and needs, and provide more accurate and useful answers; the big model has strong language understanding and generation capabilities, and can learn and train a large amount of text data. By utilizing the information processing and fusion capabilities of the big model, different information sources can be integrated and analyzed to better understand the user's intentions and needs.

[0032] The present invention provides a multilingual word alignment method based on an attention mechanism, comprising the following steps: S1. Data collection: Collect a large amount of multilingual text data in Cantonese, Chinese and English; S2. Text representation: Model the text and obtain the feature representation of the text; specifically, any one of the convolutional neural network (CNN), recurrent neural network (RNN) or long short-term memory network (LSTM) is used to model the input text. When the convolutional neural network (CNN) is used, the input text is converted into a sequence of word vectors, and then the word vector sequence is input into the CNN for feature extraction to obtain the feature representation of the text; the convolutional neural network (CNN) is used to model the text, and the accuracy and robustness of the alignment are improved by introducing a multilingual attention mechanism.

[0033] S3. Large model architecture: Adjust and optimize the large model to adapt to the processing of multilingual data; S4. Multilingual pre-training: Use multilingual data to pre-train the large model to learn the common features and rules of the language. During the pre-training process, a variety of techniques are used to improve the performance of the large model. S5. Attention mechanism: A multilingual attention mechanism is introduced to calculate the attention scores between texts in different languages, so as to determine the alignment relationship between words. Specifically, the feature representation of the source language text and the feature representation of the target language text are input into the attention mechanism, the attention score is calculated, and then the alignment relationship between words is determined based on the attention score.

[0034] S6, alignment result: According to the attention score, the alignment relationship between words is determined, and the alignment result is output; specifically, the attention score is converted into an alignment matrix, and then the alignment relationship between words is determined according to the alignment matrix, and the alignment result is output.

[0035] S7. Fine-tuning and optimization: Evaluate and compare large models, verify the performance and effect of large models, and optimize and improve them; S8. Large model evaluation: Use a variety of evaluation indicators to evaluate and compare large models to verify the performance and effectiveness of large models.

[0036] In a specific embodiment of the present invention, the multilingual text data described in S1 includes news, novels, papers, and social media; completing data collection and organization to construct a multilingual data set; Verify data quality using methods including data cleaning, screening and verification, as well as cross-validation using multiple data sources to address issues such as inaccurate, incomplete or noisy data.

[0037] In a specific embodiment of the present invention, the large model described in S3 adopts a Transformer-based architecture, which includes BERT and GPT; the architecture of the pre-trained large model is designed and implemented, and preliminary experiments and verifications are performed; Preprocessing of overfitting of large models involves increasing the amount of data, using regularization techniques, and early stopping methods to address overfitting that may be caused by insufficient data or excessive complexity of the large model.

[0038] In a specific embodiment of the present invention, the various technologies described in S4 include multi-task learning and adversarial training; multi-language pre-training of the large model, and fine-tuning and optimization to improve the performance of the large model on downstream tasks; this method can automatically learn the grammar and semantic relations of the language, has better processing capabilities for complex language structures and semantic relations, and can process multiple languages ​​​​at the same time.

[0039] This method models text by introducing an attention mechanism and determines the alignment relationship by calculating the attention scores between words.

[0040] Multilingual data processing uses data augmentation, balanced datasets, or multi-task learning methods to address the possible differences in the amount of data for different languages, which may result in large models having poor learning effects on certain languages.

[0041] In the specific implementation of the present invention, on the basis of pre-training, fine-tuning and optimization are performed for downstream tasks, using intent recognition and text classification; using transfer learning and adaptive learning techniques to improve the performance of large models on specific tasks; without a large amount of manual annotation and prior knowledge, the grammar and semantic relations of the language can be automatically learned. It has better processing capabilities for complex language structures and semantic relations, and can improve the accuracy of word alignment.

[0042] The interpretability of large models can be improved by using visualization techniques and feature importance analysis methods. This is because large pre-trained models are usually very complex and their decision-making process is difficult to explain.

[0043] In a specific embodiment of the present invention, the multiple evaluation indicators described in S8 include precision, recall and F1 value.

[0044] In a specific embodiment of the present invention, the following steps are also included: Grammatical word formation analysis: Analyze the grammatical word formation of the source language and the target language to understand their vocabulary, part of speech, and syntactic characteristics; Self-supervised word alignment: Design a self-supervised word alignment method based on the attention mechanism to achieve word-level alignment by calculating the similarity between the source language and the target language word vectors. By strengthening the word-level semantic alignment between different languages, the translation quality of the neural machine translation model is improved. Multilingual representations are more robust and can better cope with language variation and noise.

[0045] In a specific embodiment of the present invention, the following steps are also included: Multilingual representation: Use word alignment information to build more informative and robust multilingual representations to improve the performance of cross-language tasks.

[0046] Word alignment attention layer: A word alignment attention layer is added to the encoder and decoder of the neural machine translation model to obtain high-quality word alignment matrices between the source language and the target language, providing word alignment information for context modeling training. The self-supervised word alignment method does not require manual annotation of data, reducing the cost of data preparation. This technology can be applied to a variety of cross-language tasks and has strong scalability.

[0047] This method improves translation quality and provides better support for cross-language communication. It uses multilingual representations to classify cross-language texts and improves classification accuracy. It helps users quickly and accurately retrieve the required content from multilingual information. It aims to form multilingual representations with richer information and stronger robustness, and provide better support for cross-language tasks such as neural machine translation.

[0048] Improve language understanding ability: By utilizing the language understanding ability of the big model, we can better understand the user's intentions and needs, and provide more accurate and useful answers; the big model has strong language understanding and generation capabilities, and can learn and train a large amount of text data. By utilizing the information processing and fusion capabilities of the big model, different information sources can be integrated and analyzed to better understand the user's intentions and needs.

[0049] Improve the coherence of the conversation: By utilizing the information processing and fusion capabilities of the large model, different information sources can be better integrated and analyzed, thereby improving the coherence and logic of the conversation. The integrated processing method of the large model is adopted to process the conversation task as a whole, thus avoiding the problem of error accumulation.

[0050] Improve generation capabilities: By leveraging the generation capabilities of large models, more natural and fluent responses can be generated, thereby improving user satisfaction. Improve the robustness of the system: By avoiding the problem of error accumulation, the robustness of the system is improved so that it can better cope with various complex dialogue scenarios. By making full use of the information processing and fusion capabilities of the large model and avoiding the problem of error accumulation, the overall performance of the intelligent dialogue system is significantly improved. Specifically, this is achieved through the following aspects.

[0051] By making full use of the powerful information processing and fusion capabilities of the large model, and effectively avoiding the error accumulation problem caused by the modular method, the overall performance of the intelligent dialogue system is significantly improved. It has broad application prospects and brings new opportunities and challenges to the development of intelligent dialogue systems.

[0052] In a specific embodiment of the present invention, the attention mechanism can dynamically assign weights according to the semantics, context and other information of the words to capture the alignment relationship between words.

[0053] This multilingual word alignment method based on the attention mechanism studies the cross-language unified semantic representation pre-training technology. By analyzing the grammatical word formation characteristics of the source language and the target language, a self-supervised word alignment method based on the attention mechanism is designed to form a multilingual representation with richer information and stronger robustness. Based on the encoder and decoder of the neural machine translation model, a word alignment attention layer is added to obtain a high-quality word alignment matrix between the source language and the target language, providing word alignment information for context modeling training to strengthen word-level semantic alignment between different languages; The use of large-scale pre-trained language models unifies the traditional modular implementation methods of intelligent dialogue into an end-to-end processing paradigm, fully utilizing the powerful information processing and fusion capabilities of large models while effectively avoiding the error accumulation problem caused by modular methods, significantly improving the overall performance of the intelligent dialogue system. In addition, in response to the multilingual problems in actual needs, a large pre-trained model that integrates multiple languages ​​such as Cantonese, Chinese, and English is implemented to form a multilingual representation with richer information and stronger robustness, improving the performance of downstream tasks such as intent recognition.

[0054] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multilingual word alignment method based on attention mechanism, characterized in that: include Follow these steps: S1. Data collection: Collect a large amount of multilingual text data in Cantonese, Chinese and English; S2, text representation: model the text and obtain the feature representation of the text; S3. Large model architecture: Adjust and optimize the large model to adapt to the processing of multilingual data; S4. Multilingual pre-training: Use multilingual data to pre-train the large model to learn the common features and rules of the language. During the pre-training process, a variety of techniques are used to improve the performance of the large model. S5. Attention mechanism: Introduce a multilingual attention mechanism to calculate the attention scores between texts in different languages, thereby determining the alignment relationship between words; S6, alignment result: determine the alignment relationship between words according to the attention score, and output the alignment result; S7. Fine-tuning and optimization: Evaluate and compare large models, verify the performance and effect of large models, and optimize and improve them; S8. Large model evaluation: Use a variety of evaluation indicators to evaluate and compare large models to verify the performance and effectiveness of large models.

2. According to the multilingual word alignment method based on attention mechanism in claim 1, it is characterized in that: The multilingual text data described in S1 include news, novels, papers, and social media; complete data collection and organization to build a multilingual dataset; Verify data quality using methods including data cleaning, screening and verification, as well as cross-validation using multiple data sources to address issues such as inaccurate, incomplete or noisy data.

3. The multilingual word alignment method based on attention mechanism according to claim 1, characterized in that: The large model described in S3 uses a Transformer-based architecture that includes BERT and GPT. The architecture of the pre-trained large model is designed and implemented, and preliminary experiments and verifications are performed. Preprocessing of overfitting of large models involves increasing the amount of data, using regularization techniques, and early stopping methods to address overfitting that may be caused by insufficient data or excessive complexity of the large model.

4. The multilingual word alignment method based on attention mechanism according to claim 1, characterized in that: The various techniques described in S4 include multi-task learning, adversarial training; Perform multi-language pre-training on large models, and perform fine-tuning and optimization to improve the performance of large models on downstream tasks; This includes multilingual data processing, data augmentation, the use of balanced data sets or multi-task learning methods to cope with the fact that the amount of data in different languages ​​may be different, resulting in large models not being able to learn certain languages ​​well.

5. The multilingual word alignment method based on attention mechanism according to claim 1, characterized in that: Based on pre-training, fine-tune and optimize downstream tasks, using intent recognition and text classification; use transfer learning and adaptive learning techniques to improve the performance of large models on specific tasks; The interpretability of large models can be improved by using visualization techniques and feature importance analysis methods. This is because large pre-trained models are usually very complex and their decision-making process is difficult to explain.

6. The multilingual word alignment method based on attention mechanism according to claim 1, characterized in that: The various evaluation indicators described in S8 include precision, recall, and F1 value.

7. The multilingual word alignment method based on attention mechanism according to claim 1, characterized in that: The following steps are also included: Grammatical word formation analysis: Analyze the grammatical word formation of the source language and the target language to understand their vocabulary, part of speech, and syntactic characteristics; Self-supervised word alignment: Design a self-supervised word alignment method based on the attention mechanism to achieve word-level alignment by calculating the similarity between the source language and target language word vectors.

8. The multilingual word alignment method based on attention mechanism according to claim 7, characterized in that: The following steps are also included: Multilingual representation: Use word alignment information to build a more informative and robust multilingual representation to improve the performance of cross-language tasks; Word alignment attention layer: A word alignment attention layer is added to the encoder and decoder of the neural machine translation model to obtain high-quality word alignment matrices between the source language and the target language, providing word alignment information for context modeling training.

9. The multilingual word alignment method based on attention mechanism according to claim 1, characterized in that: The attention mechanism can dynamically assign weights based on the semantics, context and other information of the words to capture the alignment relationship between words.

10. The multilingual word alignment method based on attention mechanism according to claim 9, characterized in that: Modeling network architectures include convolutional neural networks (CNN), recurrent neural networks (RNN), or long short-term memory networks (LSTM).