A method, device and medium for non-autoregressive large model paragraph-level machine translation based on sentence alignment
By introducing sentence alignment loss function and non-autoregressive decoding method in the training and inference stage of large models, the problems of slow inference speed and difficult sentence alignment in chapter-level machine translation are solved, and faster and more accurate translation results are achieved.
Patent Information
- Application Number
- CN202510070774.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-16
- Publication Date
- 2025-04-08
AI Technical Summary
When large models perform chapter-level machine translation, the reasoning speed is slow and the sentence alignment is difficult, resulting in the possible problems such as missed translation and mistranslation.
The non-autoregressive large model chapter-level machine translation method based on sentence alignment is adopted, and the translation effect is optimized by introducing the sentence alignment loss function in the model training stage, and the non-autoregressive method and speculative sampling decoding are used in the inference stage.
It improves the inference speed of the big model, ensures accurate alignment of the original text and the translation, reduces the possibility of missed and mistranslations, and improves the overall effect of machine translation.
Smart Images
Figure CN119783696A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of machine translation, and in particular, relates to a non-autoregressive large model paragraph-level machine translation method, device and medium based on sentence alignment. Background Art
[0002] Compared with traditional machine translation models, large models can better capture the semantic information and contextual relationships of the input text. Using large models for paragraph-level machine translation can effectively solve the problem of input windows of traditional machine translation models, and can utilize contextual information for more accurate translation.
[0003] However, there are two problems with large models for paragraph-level machine translation: First, the inference speed is slow. Since the large model has too many parameters, it is much slower than traditional machine translation during inference and cannot meet the needs of high-concurrency online users. Second, there is the sentence alignment problem. Since the input sentences of paragraph-level machine translation are too long, the large model cannot accurately achieve sentence-by-sentence correspondence between the original text and the translated text during translation, which may lead to problems such as omissions and mistranslations. Summary of the invention
[0004] The purpose of the present invention is to provide a non-autoregressive large model paragraph-level machine translation method based on sentence alignment to solve the technical problems existing in the prior art.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: A non-autoregressive large model paragraph-level machine translation method based on sentence alignment, comprising: (1) Model training phase (11) Using the paragraph-level training data, the paragraph-level training data is segmented and aligned to obtain aligned sentence pairs, and special symbols are used to splice the aligned sentence pairs to obtain the spliced original full text and translation full text. The sentence length of each sentence in the translation is input into the non-autoregressive large model together with the original text; (12) Fine-tune all parameters of the non-autoregressive large model. When updating the model parameters, introduce the sentence alignment loss function and calculate the loss function of each sentence separately. (2) Model inference stage Using a non-autoregressive approach, all predicted texts are generated at once, and sentence length prediction and translation effect optimization are performed. The specific implementation steps are as follows: (21) For each sentence in the original text, use the small model to perform autoregressive prediction, continuously generate y prediction tokens as the output results of the next model, and obtain the sentence length; (22) Using the original text and the predicted tokens in (21) as input, perform non-autoregressive large model non-autoregressive decoding and output all predicted tokens at once; (23) For each step of the predicted token, use the logits of the large model and the small model for comparison: (231) If the generation probability of the large model of the predicted token is less than the generation probability of the small model, re-execute step (22) to obtain the updated predicted token; (232) If the effect of the large model generating the predicted token is greater than or equal to that of the small model, the predicted token output by the large model is retained as the result of the current step of reasoning; then, step (21) is re-executed, the output result of the small model is updated, and the next step of reasoning is performed until the model reasoning is completed.
[0006] In a preferred embodiment, in (11), the chapter-level training data is obtained by directly acquiring the original text and translation of the existing entire article.
[0007] In a preferred embodiment, in (11), the original text and the translation are split into sentences according to pause symbols to obtain sentence pairs; after the splitting, the sentence evaluation method is used to score the sentence pairs of the original text and the translation, and according to the threshold, the sentence pairs with scores exceeding the threshold are retained to achieve alignment of the original text and the translation.
[0008] In a preferred embodiment, in (12), the method of fine-tuning all parameters is as follows: the parameter structure of the original model is kept unchanged, and the model is further trained by using a dataset for a specific task, and all model parameters are updated.
[0009] In a preferred embodiment, in (12), the loss function is calculated as follows: according to the sentence separator " <sent>"Align the length of the standard translation and the predicted translation, using the sentence concatenation symbol" <pad>"Complete the shorter sentences and then calculate the overall loss function.
[0010] In order to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the non-autoregressive large model paragraph-level machine translation method based on sentence alignment as described above.
[0011] In order to achieve the above-mentioned objectives, the present invention also provides a non-autoregressive large model chapter-level machine translation device based on sentence alignment, comprising: a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the non-autoregressive large model chapter-level machine translation device based on sentence alignment performs the non-autoregressive large model chapter-level machine translation method based on sentence alignment as described above.
[0012] Compared with the prior art, the present invention has the following beneficial effects: (1) According to the present invention, paragraph-level machine translation is achieved by fine-tuning all parameters of a large model.
[0013] (2) According to the present invention, the problem of paragraph-level machine translation alignment is solved by adding a sentence alignment loss function in the training process.
[0014] (3) According to the present invention, a non-autoregressive approach is used to improve the inference speed of large models.
[0015] (4) According to the present invention, speculative sampling decoding is used to optimize the machine translation effect of large models. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 This is a schematic diagram of the principle of Example 1 of the present invention.
[0017] Figure 2 Schematic diagram for comparing the splicing methods of the standard translation and the predicted translation in Example 1 of the present invention. DETAILED DESCRIPTION
[0018] In order to enable those skilled in the art to have a clearer understanding and knowledge of the present invention, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described below are only used to explain the present invention, and are convenient for understanding. The technical solutions provided by the present invention are not limited to the technical solutions provided by the following embodiments, and the technical solutions provided by the embodiments should not limit the protection scope of the present invention.
[0019] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0020] Example 1 like Figure 1 As shown, this embodiment provides a non-autoregressive large model chapter-level machine translation method based on sentence alignment, which aims to solve the problems of large model chapter-level machine translation inference speed and translation effect.
[0021] In this embodiment, the non-autoregressive large model paragraph-level machine translation method based on sentence alignment mainly includes two steps, one is the model training stage, and the other is the model reasoning stage. The two steps are described in detail below.
[0022] 1. Model training stage In the model training stage, chapter-level training data is used. The training data is obtained by directly acquiring the original text and translation of the existing entire article. In order to achieve sentence alignment, the training data is first segmented and aligned. A rule-based method is used for sentence segmentation, and sentences are split according to pause symbols (such as .?! in Chinese). After segmentation, sentence evaluation methods such as PPL, BLUE value, word alignment model, etc. are used to score the original text and translation (i.e. the original text and translation of the training data). According to the threshold, sentence pairs with scores exceeding the threshold are retained to achieve alignment of the original text and the translation.
[0023] After getting the aligned sentence pairs, use special symbols such as " <sent>" is concatenated to obtain the concatenated original full text and translated full text. In addition, the sentence length of each sentence in the translated text is calculated and input into the non-autoregressive large model together with the original text.
[0024] Fine-tune all parameters of the non-autoregressive large model. When updating the model parameters, introduce the sentence alignment loss function and calculate the loss function of each sentence separately. The method of fine-tuning all parameters is as follows: keep the parameter structure of the original model unchanged, and further train the model by using task-specific datasets. All model parameters are updated to better adapt to new tasks.
[0025] In order to achieve sentence alignment, the sentence alignment loss function is introduced when the model updates the parameters. The loss function of each sentence is calculated separately. Taking Cross-Entropy Loss as an example, according to the sentence separator " <sent>"Align the length of the standard translation and the predicted translation, using the sentence concatenation symbol" <pad>"Fill the shorter sentences, and then calculate the overall loss function. In this embodiment, the loss function calculates the logarithm between the standard translation and the predicted translation. The calculation formula adopts the loss function calculation method commonly used in large models. The difference is that in this embodiment, the splicing method of the standard translation and the predicted translation is changed, such as Figure 2 As shown, A represents the traditional method and B represents the method adopted in this embodiment.
[0026] 2. Model Reasoning Stage During the model inference process, a non-autoregressive method is used to generate all predicted texts at once. However, the non-autoregressive method requires setting the translation output length, and the accuracy is not as high as the autoregressive method. Therefore, the speculative sampling decoding method is used to rely on an existing traditional machine translation model to predict sentence length and optimize translation results. The specific implementation steps are as follows: (21) For each sentence in the original text, use the small model to perform autoregressive prediction, continuously generate y prediction tokens as the output results of the next model, and get the sentence length; the small model refers to a model with a relatively small number of parameters. Commonly used large models include 7B, 13B, 32B, 72B, etc., and the small model is less than 1B; (22) Using the original text and the predicted tokens in (21) as input, perform non-autoregressive large model non-autoregressive decoding and output all predicted tokens at once; (23) For the predicted token of each step, the logits of the large model and the small model are compared: (231) If the generation probability of the large model of the predicted token is less than the generation probability of the small model, re-execute step (22) to obtain the updated predicted token; (232) If the effect of the large model generating the predicted token is greater than or equal to the small model, retain the predicted token output by the large model as the result of the current step of reasoning; then, re-execute step (21), update the output result of the small model, and proceed to the next step of reasoning until the model reasoning is completed.
[0027] Example 2 This embodiment provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the sentence alignment-based non-autoregressive large model chapter-level machine translation method provided in Example 1. A person of ordinary skill in the art can understand that all or part of the steps of implementing the method provided in Example 1 can be completed by hardware related to the computer program, and the above-mentioned computer program can be stored in a computer-readable storage medium, and when the program is executed, the steps of the method provided in Example 1 are executed; and the above-mentioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.
[0028] Example 3 This embodiment provides a non-autoregressive large model chapter-level machine translation device based on sentence alignment, including: a processor and a memory; the memory is used to store computer programs; the processor is connected to the memory, and is used to execute the computer programs stored in the memory, so that the non-autoregressive large model chapter-level machine translation device based on sentence alignment executes the non-autoregressive large model chapter-level machine translation method based on sentence alignment provided in Example 1.
[0029] Specifically, the memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.
[0030] Preferably, the processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.
[0031] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.< / pad> < / sent> < / sent> < / pad> < / sent>
Claims
1. A non-autoregressive large model paragraph-level machine translation method based on sentence alignment, characterized in that: include: (1) Model training phase (11) Using the paragraph-level training data, the paragraph-level training data is segmented and aligned to obtain aligned sentence pairs, and special symbols are used to splice the aligned sentence pairs to obtain the spliced original full text and translation full text. The sentence length of each sentence in the translation is input into the non-autoregressive large model together with the original text; (12) Fine-tune all parameters of the non-autoregressive large model. When updating the model parameters, introduce the sentence alignment loss function and calculate the loss function of each sentence separately. (2) Model inference stage Using a non-autoregressive approach, all predicted texts are generated at once, and sentence length prediction and translation effect optimization are performed. The specific implementation steps are as follows: (21) For each sentence in the original text, use the small model to perform autoregressive prediction, continuously generate y prediction tokens as the output results of the next model, and obtain the sentence length; (22) Using the original text and the predicted tokens in (21) as input, perform non-autoregressive large model non-autoregressive decoding and output all predicted tokens at once; (23) For the predicted token of each step, the logits of the large model and the small model are compared: (231) If the generation probability of the large model of the predicted token is less than the generation probability of the small model, re-execute step (22) to obtain the updated predicted token; (232) If the effect of the large model generating the predicted token is greater than or equal to the small model, retain the predicted token output by the large model as the result of the current step of reasoning; then, re-execute step (21), update the output result of the small model, and proceed to the next step of reasoning until the model reasoning is completed.
2. The non-autoregressive large model paragraph-level machine translation method based on sentence alignment according to claim 1 is characterized in that: In (11), the paragraph-level training data is obtained by directly acquiring the original text and translation of the existing entire article.
3. The sentence alignment-based non-autoregressive large model paragraph-level machine translation method according to claim 2 is characterized in that: In (11), the original text and the translation are split into sentences according to pause symbols to obtain sentence pairs; after the splitting, the sentence evaluation method is used to score the sentence pairs of the original text and the translation. According to the threshold, the sentence pairs with scores exceeding the threshold are retained to achieve alignment of the original text and the translation.
4. The sentence alignment-based non-autoregressive large model paragraph-level machine translation method according to claim 3 is characterized in that: In (12), the method of fine-tuning all parameters is as follows: keep the parameter structure of the original model unchanged, further train the model by using a dataset for a specific task, and all model parameters are updated.
5. The sentence alignment-based non-autoregressive large model paragraph-level machine translation method according to claim 4 is characterized in that: In (12), the loss function is calculated as follows: According to the sentence separator " <sent>"Align the length of the standard translation and the predicted translation, using the sentence concatenation symbol" <pad> "Complete the shorter sentences and then calculate the overall loss function.< / pad> < / sent> 6. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the sentence alignment-based non-autoregressive large model paragraph-level machine translation method as described in any one of claims 1 to 5.
7. A non-autoregressive large model paragraph-level machine translation device based on sentence alignment, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory so that the sentence alignment-based non-autoregressive large model chapter-level machine translation device performs the sentence alignment-based non-autoregressive large model chapter-level machine translation method as described in any one of claims 1 to 5.