Automated methods, systems, equipment, and storage media for evaluating the quality of French translations

Abstract

This invention discloses an automated method, system, device, and storage medium for evaluating the quality of French translations. The method obtains the representation vectors of the Chinese source sentence and the French translation to be evaluated in the BGE M3-Emebdding model, calculates their distance, and uses this distance as input to a trained deep DNN network model to obtain a quantifiable translation evaluation score. Furthermore, it can combine the translation evaluation patterns of translation evaluation experts with the machine's natural language understanding capabilities, utilize the emergent capabilities of a large language model, and employ the LoRA fine-tuning method to obtain a fine-tuned large model, generating reasonable and interpretable justifications for the translation quality evaluation. This invention ensures the accuracy of French translation quality scoring and effectively improves translation efficiency.

Description

Technical Field

[0001] This invention belongs to the field of translation quality assessment, and relates to a method for assessing the quality of French translations, and more particularly to an automated method, system, device and storage medium for assessing the quality of French translations based on a large language model. Background Technology

[0002] Translation Quality Assessment (TQA) is an important research area in the field of translation. Translation scholars both domestically and internationally have proposed influential TQA models, defining evaluation standards from multiple perspectives, including different dimensions, levels, and cultural factors. However, these models have two drawbacks: First, they require manual review of translation quality by experts in the field, lacking automated methods and thus being inefficient, making them difficult for ordinary French learners and practitioners to apply in practice. Second, they lack comprehensive and quantifiable indicators for evaluating translation quality, making it difficult to intuitively measure the quality of a translation.

[0003] Existing automated translation quality assessment methods, such as BLEU, COMET, and HLEPOR, are based on the difference between the reference translation and the machine translation, and are used to evaluate the level of machine translation. They are difficult to apply in scenarios where there is no reference translation. Moreover, most of these methods can only provide translation quality scores, but cannot explain the reasons for the scores, making it difficult to provide French learners and practitioners with effective ideas for improving translation.

[0004] With the rapid development of large language models such as ChatGPT, their language understanding capabilities have significantly improved. Decoder-only models based on the Transform network structure have achieved breakthroughs in text generation. These models are pre-trained on large-scale unsupervised text data and then fine-tuned for specific tasks. Although these models are only trained to predict the next word, they exhibit strong emergent capabilities. The pre-trained models can perform various tasks such as text classification, question answering, and translation well, and can learn new tasks through context without updating model parameters. This capability offers a new possibility for building automated methods to evaluate the quality of French translations. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an automated method, system, device, and storage medium for evaluating the quality of French translations based on a large language model. This method can automatically evaluate the quality of French translations, including quantifiable comprehensive evaluation indicators and interpretable reasons for the translation evaluation.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] An automated method for evaluating the quality of French translations based on a large language model includes:

[0008] Obtain the representation vectors A and B of the Chinese source sentence and the French translation to be evaluated in the BGE M3-Emebdding model;

[0009] Calculate the distance d between the representation vector A of the Chinese source sentence and the representation vector B of the French translation, and use it as the initial translation evaluation score;

[0010] The representation vectors A and B, along with the distance d, are input into the trained deep DNN network model. Representation vectors A and B serve as inputs to the first layer of the network, while the distance d serves as a supplementary input feature for the last layer of the deep DNN network. The output of the model is the quantifiable translation evaluation score.

[0011] In the above technical solution, the deep DNN network model further includes an input layer, five intermediate layers, and an output layer. The dimensions of the five intermediate layers are 512, 256, 128, 64, and 31+1, respectively. The representation vectors of the Chinese source sentence and the corresponding French translation are used as the bottom-level cross-feature input of the DNN network model, and the initial translation evaluation score is used as the supplementary input feature of the last intermediate layer in the network model for training. After passing through the sigmoid function, the output layer outputs a quantifiable translation evaluation score.

[0012] Furthermore, the training of the deep DNN network model, including the construction of its samples, comprises:

[0013] Translation samples of Chinese source sentences and their corresponding French translations are constructed. The data comes from the United Nations Sino-French Parallel Corpus and is used as positive samples. The negative samples consist of two parts: one is random samples generated by negative sampling within a batch, and the other is hard negative samples constructed using French translations from Google Translate. For the positive and negative samples, the representation vectors of the BGE M3-embedding model of the Chinese source sentences and their corresponding French translations are obtained for training.

[0014] Furthermore, the proportion of difficult negative samples in the negative samples should not be too high. The ratio of randomly generated negative samples to difficult negative samples is 127:1. At the same time, 10% of the vector representations of the French translations from Google Translate are randomly discarded using dropout.

[0015] Furthermore, after obtaining quantifiable translation evaluation scores, the Chinese source sentences, the French translation to be evaluated, and the corresponding quantifiable translation evaluation scores are filled into a pre-designed Prompt template. The fine-tuning model for French translation quality evaluation is then accessed to obtain interpretable translation evaluation reasons.

[0016] Furthermore, the fine-tuned large model for assessing the quality of French translations is based on the LlaMA3 model and is obtained using a high-quality SFT (Supervised Fine-Tuning) dataset and the LoRA fine-tuning method.

[0017] Furthermore, the method for constructing the high-quality SFT dataset is as follows:

[0018] N data points that were not used in the training of the deep DNN network model were selected from the UN Sino-French parallel corpus. N should typically be no less than 1000 data points. At the same time, the Chinese source sentences and their corresponding French translations from Google Translate were obtained, resulting in 2N Sino-French sentence pairs. The trained deep DNN network model was used to obtain quantifiable translation evaluation scores for the Sino-French sentence pairs. Based on the Translation Quality Assessment Model, translation quality evaluation reasons were manually generated to obtain a high-quality SFT dataset.

[0019] An automated evaluation system for the quality of French translations based on a large language model, used to implement the method described in any of the preceding methods.

[0020] An electronic device, comprising:

[0021] One or more processors;

[0022] Memory, used to store one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any of the preceding methods.

[0024] A computer-readable storage medium storing computer-executable instructions, which, when executed, are used to perform...

[0025] Now, as described in any of the previous methods.

[0026] The advantages of this invention are:

[0027] (1) An automated French translation quality assessment method combining quantifiable and interpretable approaches is proposed, which can effectively improve translation efficiency; (2) By utilizing the accuracy of the BGE M3-embedding model in Chinese and French language representation and the comparability of vectors, a deep neural network structure for French translation quality assessment is constructed, and the initial score of vector distance is used as the feature input of the last layer of the neural network, ensuring the accuracy of French translation quality scoring; (3) Combining the translation assessment mode of human translation assessment experts and the natural language understanding ability of machines, the emergence capability of large language models is utilized, and the LoRA fine-tuning method is adopted. Under the premise of ensuring the model effect, the fine-tuning cost is reduced on a large scale, thereby generating reasonable and interpretable reasons for translation quality assessment. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall architecture of a specific example of the present invention.

[0029] Figure 2 This is a schematic diagram of the Prompt template design and LLM (Large Language Model) fine-tuning data and structure, representing a specific example of the present invention. Detailed Implementation

[0030] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] This invention is based on the BGE M3-embedding and LlaMA3 large language model, and proposes an automated evaluation method for the translation quality of Chinese-to-French translation. The evaluation results can include both quantifiable translation quality scores and interpretable reasons for the translation quality evaluation.

[0032] According to a specific example of the present invention, an automated method for evaluating the quality of French translations based on a large language model is provided, such as... Figure 1 As shown, it includes the following steps:

[0033] 1) Obtain the vector representations of the Chinese source sentence and the corresponding French translation using the BGE M3-Emebdding model, and calculate the distance score between the two vectors as the initial translation evaluation score;

[0034] 2) Combine the vector representations of the Chinese source sentence and the French translation as input features of a fully connected deep neural network, and use the initial translation evaluation score as supplementary input to the last layer of the deep neural network to obtain the final quantifiable translation evaluation score.

[0035] According to a specific example of the present invention, the automated evaluation method may further include:

[0036] 3) Select data from the Sino-French parallel corpus and manually construct the SFT training dataset for French translation according to the translation quality assessment model. The data includes quantifiable translation assessment scores and multiple reasons for the scores.

[0037] 4) Based on the SFT dataset, the LlaMA3 model was fine-tuned using LoRA to make it more suitable for the field of French translation evaluation, resulting in a fine-tuned large model for French translation quality evaluation.

[0038] 5) Based on the theory of translation evaluation models, a Prompt template for French translation evaluation tasks is constructed. Combining the translation evaluation scores, the CoT (Chain of Thought) thought-guided model is used to generate interpretable reasons for the translation quality evaluation.

[0039] The above steps mainly consist of two parts: quantifiable translation evaluation scores and interpretable reasons for evaluating translation quality.

[0040] For the quantifiable translation evaluation score, the BGE M3-Emebdding model is used as the basic representation of the sentence. BGE is a new embedding model released and open-sourced by the Beijing Academy of Artificial Intelligence. It is superior to other embedding models in terms of Chinese and English representation capabilities. BGE M3-Embedding is a further upgraded version of it, supporting more than 100 languages ​​including Chinese, English, and French. It has multilingual and cross-lingual representation capabilities, and different languages ​​share the same vector space. It has the characteristic of comparable vectors for Chinese and French. Vector representations of the Chinese source sentence and its corresponding French translation using the BGE M3-Emebdding model are generated respectively, and the vector distance between the two is used as the initial translation evaluation score. Although the BGE M3-Emebdding model has good language representation capabilities, it is designed as a general embedding model and cannot directly achieve optimal results in specific translation evaluation tasks. Therefore, a multi-layer deep neural network needs to be constructed to further enhance the learning of the correspondence between the Chinese source sentence and the French translation. The BGE M3-Emebdding representations of the Chinese source sentence and its corresponding French translation are used as the input cross-features of this neural network. A parallel Chinese-French corpus is used as samples to strengthen the training of the DNN model in the Chinese-French translation scenario. In addition, to prevent the loss of original embedding information in the deep neural network, the initial translation evaluation score is used as a supplementary input feature of the last layer of the deep neural network for training to improve the accuracy of translation quality evaluation.

[0041] For the section on interpretable translation quality assessment rationale, the large language model uses the open-source LlaMA3 8B model from MetaWorks as its base model. LlaMA3 is a standard decoder-only Transformer architecture, and its massive training data scale and GQA (Group Query Attention) mechanism demonstrate strong language understanding and generation capabilities. However, to better adapt to the scenario of French translation quality assessment, LlaMA3 needs to be fine-tuned to understand the translation evaluation patterns of human translation experts, thereby providing accurate and interpretable rationale for translation quality assessment. Data from a Chinese-French parallel corpus is selected, and quantifiable translation evaluation scoring methods are used to generate French translation quality scores for corresponding sentences. Then, according to the translation quality assessment pattern, human reasons are given for the scores to construct a fine-tuned SFT training dataset for French translation quality assessment. The LoRA (Low-Rank Adaptation) fine-tuning method is used to obtain a fine-tuned large model for French translation quality assessment. Based on the translation evaluation model, a Prompt template for the French translation evaluation task is constructed. Combining the translation evaluation scores, the CoT concept guides the model to produce interpretable reasons for the translation quality evaluation.

[0042] Example 1

[0043] An automated method for evaluating the quality of French translations based on a large language model includes:

[0044] 1. Construct translation samples of Chinese source sentences and their corresponding French translations. The data comes from the United Nations Parallel Corpus, which collects parallel texts of various UN documents and document types, ensuring the accuracy of the translations. The large version of the BGE M3-embedding model with better representation capabilities is selected, and the dimension of the embedding vector is 1024. The representation vectors of the BGE M3-embedding model for the Chinese source sentences and their corresponding French translations are obtained respectively, and the cosine distance of the corresponding vectors is used as the initial translation evaluation score.

[0045] 2. A deep DNN network model is constructed. The representation vectors of the Chinese source sentence and its corresponding French translation obtained in step 1 are used as the bottom-level cross-feature input of the DNN network model. The training objective is to further strengthen the learning of the correspondence between the Chinese source sentence and the French translation. To prevent the loss of original embedding information in the deep neural network, the initial translation evaluation score is used as a supplementary input feature for the last layer of the deep neural network during training. This method effectively improves the accuracy of the prediction score (AUC improved by 0.67% on the test set, from 93.10% to 93.77%). This DNN design has 5 intermediate layers with dimensions of 512, 256, 128, 64, and 31 + the initial translation evaluation score, respectively. The output layer has a dimension of 1, and the sigmoid function is used as the training prediction score for the model.

[0046] 3. Regarding the sample construction of the DNN network model in section 2, positive samples are one-to-one translated texts from parallel corpora. Negative samples consist of two parts: first, random samples generated by negative sampling within a batch; and second, difficult negative samples constructed using French translations from Google Translate. Softmax cross-entropy is used as the loss function for training. The proportion of difficult negative samples should not be too high; therefore, the batch size is set to 128, and the ratio of the two parts of negative samples is 127:1. Furthermore, to prevent positive samples from being mistakenly set as negative samples due to the accuracy of Google Translate, 10% of the vector representations from the French translations from Google Translate are randomly discarded using dropout to ensure the accuracy of the training results.

[0047] Example 2

[0048] An automated method for evaluating the quality of French translations based on a large language model, which, in addition to Example 1, further includes:

[0049] 4. For the Chinese-French parallel corpus in step 1, select 1000 sentences that did not participate in the training data in step 2, and obtain their corresponding French translations from Google Translate, thus obtaining 2000 Chinese-French sentence pairs. Use the model trained in step 2 to obtain quantifiable translation evaluation scores for the sentence pairs. Then, according to the translation quality evaluation model, combine the translation evaluation scores with language / text, register, and genre aspects to manually generate translation quality evaluation reasons and construct a high-quality SFT dataset.

[0050] The evaluation criteria for manually produced translations can be derived using the following steps:

[0051] 1) Analyze the register, subject matter, and textual function of the Chinese source sentences;

[0052] 2) Analyze the register, subject matter, and textual function of the French translation, and compare it with the source text to identify errors and mismatches in the translation;

[0053] 3) Determine whether the French translation is an explicit translation (emphasizing equivalence in language / text, register, and genre) or an implicit translation (emphasizing equivalence in textual function).

[0054] 4) Conduct a comprehensive evaluation of the translation quality.

[0055] 5. The base large language model is the LlaMA3 model released and open source by Meta, which has surpassed similar models in the industry in several key benchmark tests. In order to balance the efficiency of fine-tuning and the cost of prediction, this example selects the LlaMA3 8B (8 billion parameters) version, uses the SFT dataset obtained in step 4, and adopts the LoRA fine-tuning method (the rank r of the matrix is ​​set to 2) to obtain the fine-tuned large model for French translation quality assessment.

[0056] 6. Adopting the concept of CoT, a Prompt template is designed. For the Chinese source sentence and the French translation to be evaluated, a quantifiable translation evaluation score is first obtained based on the model obtained in step 2. Then, the sentence and score are populated into the Prompt template. The fine-tuning model for French translation quality evaluation in step 5 is accessed to obtain an interpretable rationale for the translation evaluation. The Prompt template is as follows: Figure 2 As shown, the design can be as follows: 1) Scenario and function preset: "You are an expert in the field of Sino-French translation evaluation. Based on your knowledge in the field of translation evaluation, you can provide detailed evaluation reasons according to the given Chinese source sentence, the corresponding French translation sentence, and the estimated translation evaluation score." 2) Data input: "The following are the details: Chinese source sentence {X}; French translation sentence {Y}; estimated translation evaluation score {Z}." 3) Result requirements: "Based on the above information, please provide detailed reasons and thought process for the quality evaluation of the French translation. There should be no less than 3 reasons, and each reason should be at least 15 words."

[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] The specific embodiments described herein are merely illustrative examples of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for automated evaluation of the quality of a French translation, characterized in that, The method comprises the following steps: obtaining the Chinese source sentence and the French translation to be evaluated in the representation vectors A and B of the BGE M3-Emebdding model; calculating the distance d between the representation vector A of the Chinese source sentence and the representation vector B of the French translation as the initial translation evaluation score; inputting the representation vector A, the representation vector B and the distance d into the trained deep DNN network model, wherein the representation vector A and the representation vector B are input into the first layer of the network, and the distance d is input into the last layer of the deep DNN network as a supplementary input feature; the output of the model is a quantifiable translation evaluation score; the deep DNN network model comprises five intermediate layers with dimensions of 512, 256, 128, 64 and 31+1 respectively; the training of the model comprises the following steps: constructing translation samples of Chinese source sentences and corresponding French translations, which are obtained from the United Nations Chinese-French parallel corpus as positive samples; the negative samples comprise two parts: one is a random sample generated by batch negative sampling, and the other is a difficult negative sample constructed by using the French translation of Google translation; meanwhile, the French translation of Google translation is randomly discarded by 10% in the form of dropout; after obtaining the quantifiable translation evaluation score, filling the Chinese source sentence, the French translation to be evaluated and the corresponding quantifiable translation evaluation score into a pre-designed Prompt template, accessing the fine-tuned large model for French translation quality evaluation, and obtaining an interpretable translation evaluation reason; the fine-tuned large model for French translation quality evaluation is obtained by using the LoRA fine-tuning method based on the LlaMA3 model and using a high-quality SFT data set; the construction method of the high-quality SFT data set is as follows: selecting N pieces of data from the United Nations Chinese-French parallel corpus which are not involved in the training of the deep DNN network model, obtaining the Chinese source sentences and the corresponding French translations of Google translation, obtaining 2N Chinese-French sentence pairs, obtaining the quantifiable translation evaluation score of the Chinese-French sentence pairs by using the trained deep DNN network model, and obtaining the high-quality SFT data set by manually outputting the translation quality evaluation reasons according to the translation quality evaluation mode.

2. The method for automated evaluation of the quality of French translations according to claim 1, characterized in that, The representation vectors of the Chinese source sentences and the corresponding French translations of the positive samples and the negative samples are obtained for training by using the BGE M3-embedding model, and the softmax cross-entropy is used as the loss function for training.

3. The method of automated evaluation of French translation quality according to claim 1, characterized in that, The proportion of difficult negative samples in the negative samples should not be too high, and the proportion of randomly generated negative samples to difficult negative samples is 127:

1.

4. A system for automated evaluation of the quality of French translations, characterized in that, The method is used to realize the method of any one of claims 1-3.

5. An electronic device, comprising: The method comprises the following steps: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-3.

6. A computer-readable storage medium storing computer-executable instructions for implementing the method of any one of claims 1-3 when executed.

Images