Translation method based on entity anchoring verifiable reward reinforcement learning

Abstract

The application provides a translation method based on entity anchoring verifiable reward reinforcement learning, which can be applied to the technical field of natural language processing. The method comprises the following steps: inputting a sentence to be translated into a translation model to generate an inference track with a thinking label, the thinking label comprising at least one sub-label for defining the range of the inference track in a complete output sequence of the translation model; generating a target translation in a target cultural context based on the verification indication information about the target word unit contained in the inference track, the complete output sequence comprising the inference track and the target translation; wherein the word unit generation strategy of the translation model comprises: taking the sentence to be translated and a generated historical word unit sequence as conditions, selecting a preset word unit matching the target cultural context from a preset word table as a target word unit of a current time step through the translation model, until a terminal symbol is generated, and the historical word unit sequence comprises a plurality of historical target word units arranged in time step order.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically to a translation method based on entity-anchored verifiable reward reinforcement learning. Background Technology

[0002] With the widespread application of large language models in the field of multilingual translation, efficiently and accurately translating culturally loaded entities in the source language into culturally consistent expressions in the target language has become a hot topic in current machine translation research.

[0003] Large language models are prone to literal or mistranslation in cross-cultural contexts. Furthermore, large language models directly output translations without explicit reasoning or self-checking of translation decisions, making it difficult to detect and correct errors during the generation process. Summary of the Invention

[0004] In view of the above problems, the present invention provides a translation method based on entity-anchored verifiable reward reinforcement learning.

[0005] According to a first aspect of the present invention, a translation method based on entity-anchored verifiable reward reinforcement learning is provided, comprising: inputting a statement to be translated into a translation model; generating an inference trajectory with a thought label, wherein the thought label includes at least one sub-label for defining the range of the inference trajectory in the complete output sequence of the translation model; the inference trajectory includes the inference and verification process of the translation model on the statement to be translated; generating a target translation in the target cultural context based on the verification indication information about target lexical units contained in the inference trajectory; the complete output sequence includes the inference trajectory and the target translation; wherein the lexical unit generation strategy of the translation model includes: using the statement to be translated and the generated historical lexical unit sequence as conditions, selecting preset lexical units that match the target cultural context from a preset vocabulary at each time step by the translation model as the target lexical unit of the current time step, until an end symbol is generated; the historical lexical unit sequence includes multiple historical target lexical units arranged in the order of time steps.

[0006] According to an embodiment of the present invention, the thinking label includes an inference start sub-label and an inference end sub-label. The inference start sub-label is used to define the starting position of the inference trajectory in the complete output sequence, and the inference end sub-label is used to define the ending position of the inference trajectory in the complete output sequence.

[0007] According to an embodiment of the present invention, the translation model is trained in the following manner: for the same training sentence to be translated, multiple training translations are generated by sampling through the translation model, including reasoning and thinking training texts and translation training texts; based on the text format check results, translation length check results, and entity matching check results of the multiple training translations, the advantage value of each training translation is determined; based on the field length of the multiple training translations, the current lexical generation strategy of the translation model, and the historical lexical generation strategy, the importance ratio of each training translation is determined; the parameters of the translation model are iteratively updated based on the substitution objective function, wherein the substitution objective function is constructed based on the advantage value and importance ratio of each training translation.

[0008] According to an embodiment of the present invention, the above method further includes: checking multiple training translations according to a preset format containing a complete response sequence of thinking tags, reasoning training text, and translation training text, to obtain a text format check result; and checking multiple training translations according to a tolerance coefficient for the length of the translation field, to obtain a translation length check result.

[0009] According to an embodiment of the present invention, the advantage value of each training translation is determined based on the text format check results, translation length check results, and entity matching check results of each of the multiple training translations. This includes: determining the aggregate reward value of each training translation based on the text format check results, translation length check results, and entity matching check results of each of the multiple training translations; and normalizing the reward values ​​corresponding to the multiple training translations respectively, using the multiple training translations as the same reward group, to obtain the advantage value of each training translation.

[0010] According to an embodiment of the present invention, the aggregate reward value of each training translation is determined based on the text format check result, translation length check result, and entity matching check result of each of the multiple training translations, including: if at least one of the text format check result and translation length check result of the training translation indicates that the check failed, the aggregate reward value of the training translation is determined to be a first preset reward value.

[0011] According to an embodiment of the present invention, the aggregate reward value of each training translation is determined based on the text format check result, translation length check result, and entity matching check result of each of the multiple training translations. This includes: determining the reward value of the text format check result as a second preset reward value if the text format check result and translation length check result of the training translation are passed; determining the reward value of the translation length check result based on a preset translation field length and a tolerance coefficient, and matching multiple training words in the training translation with an entity set to obtain an entity matching check result; determining the reward value of the entity matching result if the entity matching check result indicates that there are entities in the entity set that match multiple training words one-to-one; and determining the aggregate reward value of the training translation based on the reward value of the text format check result, the reward value of the translation length check result, and the reward value of the entity matching result.

[0012] According to an embodiment of the present invention, the importance ratio is determined based on the field lengths of multiple training translations, the current lexical generation strategy of the translation model, and the historical lexical generation strategy, including: calculating the probability ratio of the current lexical generation strategy and the historical lexical generation strategy of the translation model in the training translations; and dividing the probability ratio by the field length of the training translations in exponential space to obtain the importance ratio.

[0013] According to an embodiment of the present invention, the alternative objective function is constructed based on a first pruning threshold, a second pruning threshold, the advantage value and importance ratio of each training translation.

[0014] According to an embodiment of the present invention, the parameters of the translation model are iteratively updated based on the substitution objective function, including: for each training translation, calculating a first product of the importance ratio and the advantage value; pruning the probability ratio using different first or second pruning thresholds according to the positive or negative direction of the advantage value to obtain the pruned ratio, and calculating a second product of the pruned ratio and the advantage value, wherein the first pruning threshold is used to prune the probability ratio when the advantage value is positive, and the second pruning threshold is used when the advantage value is negative; selecting the minimum value between the first and second products as the individual substitution objective value of the training translation; averaging the individual substitution objective values ​​of multiple training translations corresponding to the same training sentence to obtain the group average substitution objective value; and backpropagating to update the parameters of the translation model with the group average substitution objective value as the update direction.

[0015] A second aspect of the present invention provides a translation device based on entity-anchored verifiable reward reinforcement learning, comprising: an input module for inputting a statement to be translated into a translation model and generating an inference trajectory with a thought label, the thought label including at least one sub-label for defining the range of the inference trajectory in the complete output sequence of the translation model, the inference trajectory including the inference and verification process of the statement to be translated by the translation model; and a generation module for generating a target translation in the target cultural context based on the verification indication information about target lexical units contained in the inference trajectory, the complete output sequence including the inference trajectory and the target translation; wherein, the lexical unit generation strategy of the translation model includes: using the statement to be translated and the generated historical lexical unit sequence as conditions, selecting preset lexical units that match the target cultural context from a preset vocabulary at each time step by the translation model as the target lexical unit of the current time step, until the generation of the end symbol, the historical lexical unit sequence including multiple historical target lexical units arranged in the order of time steps.

[0016] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0017] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0018] A fifth aspect of the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0019] According to an embodiment of the present invention, a reasoning trajectory with thinking labels can be generated through a translation model. The thinking labels include at least one sub-label used to define the range of the reasoning trajectory in the complete output sequence. At the same time, the reasoning trajectory includes the reasoning and verification process of the translation model on the statement to be translated, thereby realizing reasoning and self-checking of translation decisions and improving translation accuracy.

[0020] Meanwhile, the complete output sequence of the translation model includes the inference trajectory and the target translation, which allows users to discover and correct inference translation errors, further improving the translation accuracy of the translation model. Attached Figure Description

[0021] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0022] Figure 1This diagram illustrates an application scenario of a translation method based on entity-anchored verifiable reward reinforcement learning according to an embodiment of the present invention.

[0023] Figure 2 A flowchart illustrating a translation method based on entity-anchored verifiable reward reinforcement learning according to an embodiment of the present invention is shown schematically.

[0024] Figure 3 A flowchart illustrating the reasoning method of a translation model according to an embodiment of the present invention is shown schematically.

[0025] Figure 4 A flowchart illustrating a training method for a translation model according to an embodiment of the present invention is shown schematically.

[0026] Figure 5 This schematically illustrates a structural block diagram of a translation device based on entity-anchored verifiable reward reinforcement learning according to an embodiment of the present invention;

[0027] Figure 6 A block diagram of an electronic device suitable for implementing a translation method based on entity-anchored verifiable reward reinforcement learning according to an embodiment of the present invention is illustrated. Detailed Implementation

[0028] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] With the widespread application of large language models in the field of multilingual translation, how to efficiently and accurately translate culturally loaded entities (such as books, movies, place names, songs, idioms, etc.) in the source language into expressions that conform to cultural conventions in the target language without relying on external knowledge bases has become a hot topic in current machine translation research.

[0033] In one relevant embodiment of cross-cultural machine translation, the method based on external knowledge enhancement specifically includes explicitly providing the translation model with external context through online retrieval, multilingual knowledge graphs, or document databases to eliminate entity ambiguity.

[0034] In another relevant embodiment of cross-cultural machine translation, a method based on data augmentation and supervised fine-tuning is employed. This includes synthesizing training data through code switching, entity replacement, etc., or enhancing the model's robustness to entities through joint optimization of entity alignment and translation tasks. While these methods can alleviate entity translation errors to some extent, they still have significant limitations: On the one hand, methods based on external knowledge augmentation heavily rely on the coverage and retrieval quality of external databases, requiring additional training of task-specific retrieval tools and shifting the bottleneck of translation quality from contextual reasoning ability to the degree of structuring of external knowledge; on the other hand, methods based on supervised fine-tuning primarily rely on imitating reference translations for learning. When the entities in the training and test sets do not overlap, the model struggles to generalize the mapping relationships learned during training to unseen entities, easily falling into memory traps.

[0035] In summary, translation methods based on external knowledge reinforcement are heavily reliant on external databases, supervised fine-tuning methods struggle to handle unseen entities, and reinforcement learning methods based on neural quality metrics are prone to the "fluency trap." Therefore, there is an urgent need for a cross-cultural machine translation method that can overcome dependence on external knowledge bases, directly activate model parameterized knowledge through verifiable rewards, and maintain stability during training.

[0036] In view of this, embodiments of the present invention provide a translation method based on entity-anchored verifiable reward reinforcement learning, comprising: inputting the statement to be translated into a translation model, generating an inference trajectory with a thinking label, the thinking label including at least one sub-label for defining the range of the inference trajectory in the complete output sequence of the translation model, the inference trajectory including the inference and verification process of the translation model for the statement to be translated; generating a target translation in the target cultural context based on the verification indication information about target lexicals contained in the inference trajectory, the complete output sequence including the inference trajectory and the target translation; wherein, the lexical generation strategy of the translation model includes: using the statement to be translated and the generated historical lexical sequence as conditions, selecting preset lexicals that match the target cultural context from a preset vocabulary step by step by the translation model as the target lexicals of the current time step, until the end symbol is generated, the historical lexical sequence including multiple historical target lexicals arranged in the order of time steps.

[0037] Figure 1 The illustration depicts an application scenario of a translation method based on entity-anchored verifiable reward reinforcement learning according to an embodiment of the present invention.

[0038] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0039] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0041] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0042] It should be noted that the translation method based on entity-anchored verifiable reward reinforcement learning provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the translation device based on entity-anchored verifiable reward reinforcement learning provided in this embodiment of the invention can generally be located in server 105. The translation method based on entity-anchored verifiable reward reinforcement learning provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the translation device based on entity-anchored verifiable reward reinforcement learning provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0044] Figure 2 A flowchart illustrating a translation method based on entity-anchored verifiable reward reinforcement learning according to an embodiment of the present invention is shown.

[0045] like Figure 2 As shown, the translation method based on entity anchoring verifiable reward reinforcement learning in this embodiment includes operations S210 to S220.

[0046] In operation S210, the sentence to be translated is input into the translation model, generating a reasoning trajectory with thought labels.

[0047] According to an embodiment of the present invention, the statement to be translated may be a natural language sequence of the source language and may contain entities that need to be adapted to the target cultural context.

[0048] According to embodiments of the present invention, the translation model can be a large language model, which can have reasoning and text generation functions.

[0049] According to an embodiment of the present invention, the thinking label includes at least one sub-label for defining the range of the inference trajectory in the complete output sequence of the translation model, the inference trajectory including the inference and verification process of the translation model on the statement to be translated.

[0050] For example, the inference of a translation model can involve internal analysis of the sentence to be translated to identify entities (person names, place names, titles, cultural terms, etc.) that need to be adapted to the target cultural context.

[0051] For example, the verification process of a translation model can be the verification of target lexical units, such as whether they match the target cultural context or whether they are in the entity set of the target cultural context.

[0052] In operation S220, based on the verification indication information about target words contained in the inference trajectory, the target translation in the target cultural context is generated, and the complete output sequence includes the inference trajectory and the target translation.

[0053] According to embodiments of the present invention, the verification indication information can be a textual indication of whether the current inference step / candidate lexical has passed the self-check, output by the translation model itself in the inference trajectory. For example, the translation model outputs a "verification passed / failed" flag in the inference trajectory, or the inference trajectory contains a score value that can be used to determine whether to accept the translation.

[0054] According to an embodiment of the present invention, the target translation may be composed of target lexical units that have passed verification at multiple time steps, in order to adapt to the target cultural context.

[0055] According to an embodiment of the present invention, the complete output sequence may further include thought tags, that is, the complete output sequence includes a reasoning trajectory surrounded by thought tags and the target translation.

[0056] According to an embodiment of the present invention, the lexical generation strategy of the translation model includes: using the sentence to be translated and the generated historical lexical sequence as conditions, the translation model selects preset lexicals that match the target cultural context from a preset vocabulary at each time step as the target lexicals for the current time step, until the end symbol is generated. The historical lexical sequence includes multiple historical target lexicals arranged in the order of time steps.

[0057] According to an embodiment of the present invention, a time step is a discrete beat unit in the autoregressive generation process of the translation model. At each time step, the translation model selects a word from a preset vocabulary as the target word for the current time step, based on the sentence to be translated and the generated historical word sequence, and appends the target word to the end of the historical word sequence for use in the next time step. When the target word selected at a certain time step is the end symbol, the generation process terminates, and the total number of time steps experienced at this time is the field length of the output sequence.

[0058] According to an embodiment of the present invention, the preset vocabulary can be the complete set of output lexical units generated by the autoregressive translation model. The translation model does not output arbitrary characters without restriction at each step; instead, it first defines a fixed set (vocabulary) and then selects one character from this set.

[0059] According to an embodiment of the present invention, the terminator can be the termination signal of the autoregressive generation process of the translation model. At each time step, the translation model selects one lexical unit from a preset vocabulary. When the terminator is selected, generation stops, and the current complete output sequence (inference trajectory + target translation) is generated.

[0060] According to an embodiment of the present invention, the lexical generation strategy of the translation model can be to model cross-cultural translation as a sequential decision process.

[0061] Given a source language, input the statement to be translated. The goal is to generate the target translation within the target cultural context that correctly renders the cultural entities in the target rendering context. .

[0062] For example, a translation model could be an autoregressive large language model. This autoregressive large language model can be viewed as a model that acts on the output target word. Lexical generation strategy Its generation process can be represented as:

[0063] (1);

[0064] θ represents the parameters of the autoregressive large language model. For the target word generated at time step t, It is a historical lexical sequence, that is, included in All target lexical units generated by the previous autoregressive large language model. The total length of the output sequence. The generation process constitutes a round-robin decision process: in the... Time step status is The action is to generate the next word element. When a terminator (such as) is generated <eos>The round ends when ( ).

[0065] According to an embodiment of the present invention, a reasoning trajectory with thinking labels can be generated through a translation model. The thinking labels include at least one sub-label used to define the range of the reasoning trajectory in the complete output sequence. At the same time, the reasoning trajectory includes the reasoning and verification process of the translation model on the statement to be translated, thereby realizing reasoning and self-checking of translation decisions and improving translation accuracy.

[0066] Meanwhile, the complete output sequence of the translation model includes the inference trajectory and the target translation, which allows users to discover and correct inference translation errors, further improving the translation accuracy of the translation model.

[0067] According to an embodiment of the present invention, the thinking label includes an inference start sub-label and an inference end sub-label. The inference start sub-label is used to define the starting position of the inference trajectory in the complete output sequence, and the inference end sub-label is used to define the ending position of the inference trajectory in the complete output sequence.

[0068] According to an embodiment of the present invention, the inference segment and the translation segment are separated. To allow the translation model to invoke implicit parameterized knowledge, the translation model is required to generate inference-initiated sub-tags (such as...) before outputting the final translation. <think> ) and reasoning end sub-labels (such as< / think> The reasoning trajectory surrounded by ) The inference trajectory includes the translation model's reasoning and verification process for the sentence to be translated. When the response format is valid, the entire output is a complete output sequence. It can be decomposed into:

[0069] (2);

[0070] This is the target translation within the target cultural context. All content-based assessments apply only to... This allows the model to freely explore candidate entities in the thought process without being penalized.

[0071] By setting inference start and end sub-labels, the range of the inference trajectory within the complete output sequence is explicitly anchored as a pair of identifiable boundaries. The translation model can accurately locate the start and end positions of the inference trajectory during the generation process, ensuring that the target translation generated based on the verification indication information contained in the inference trajectory is not confused with the inference content, thereby improving the standardization of the output structure.

[0072] According to an embodiment of the present invention, the translation model is trained in the following manner: for the same training sentence to be translated, multiple training translations are generated by sampling through the translation model, including reasoning and thinking training texts and translation training texts; based on the text format check results, translation length check results, and entity matching check results of the multiple training translations, the advantage value of each training translation is determined; based on the field length of the multiple training translations, the current lexical generation strategy of the translation model, and the historical lexical generation strategy, the importance ratio of each training translation is determined; the parameters of the translation model are iteratively updated based on the substitution objective function, wherein the substitution objective function is constructed based on the advantage value and importance ratio of each training translation.

[0073] According to embodiments of the present invention, in related embodiments, when the external knowledge retrieval method relies excessively on external databases and lacks structural constraints, the translation model is prone to learning degenerate strategies such as enumerating synonyms and outputting excessively long sequences to cheat for rewards during reinforcement learning training, resulting in training failure.

[0074] Therefore, this invention constructs an alternative objective function for training the translation model based on the text format check results, translation length check results, and entity matching check results of multiple training translations. Text format check and translation length check can reduce the occurrence of cheating for rewards by enumerating synonyms or outputting excessively long sequences, and entity matching can be performed without relying on an external knowledge base, thus accelerating training and improving the accuracy of the translation model.

[0075] According to an embodiment of the present invention, for the same training statement x to be translated, G training translations are generated by sampling through a translation model, where G is an integer greater than 1. The training translations and the complete output sequence are set according to the same preset format, that is, the training translations include reasoning and thinking training text and translation training text, and the reasoning and thinking training text is surrounded by thinking tags.

[0076] According to an embodiment of the present invention, the advantage value of the training translation is relative to the other training translations among the G training translations.

[0077] According to an embodiment of the present invention, the field length of the training translation can be the total number of lexical units in the training translation (equal to the total number of time steps).

[0078] According to an embodiment of the present invention, the current lexical generation strategy of the translation model can be the model parameters of the translation model in the current training round, and the historical lexical generation strategy can be the model parameters of the translation model in the historical training round.

[0079] According to an embodiment of the present invention, the importance ratio of the training translation can be used to correct for strategy distribution bias.

[0080] According to an embodiment of the present invention, the above method further includes: checking multiple training translations according to a preset format containing a complete response sequence of thinking tags, reasoning training text, and translation training text, to obtain a text format check result; and checking multiple training translations according to a tolerance coefficient for the length of the translation field, to obtain a translation length check result.

[0081] According to an embodiment of the present invention, the complete response sequence and the complete output sequence of the translation model application stage can be set according to the same preset format.

[0082] To ensure that the reward value truly reflects translation quality rather than degenerate behavior, two hard structural gating constraints are introduced on top of the reward function, such as format gating and length gating.

[0083] The format gating can check multiple training translations according to a preset format that includes a complete response sequence containing thinking labels, reasoning training text, and translation training text. If the text format check result is consistent with the preset format, the reward value can be 1; if the text format check result is consistent with the preset format, the reward value can be 0.

[0084] Length gating can check multiple training translations according to the tolerance coefficient of the translation field length, and output the reward value corresponding to the translation length check result.

[0085] It should be noted that the training translation structure sampled during the training phase is consistent with the complete output sequence structure output during the inference phase, so the same label "thinking label" is used.

[0086] According to an embodiment of the present invention, the aggregate reward value of each training translation is determined based on the text format check result, translation length check result, and entity matching check result of each of the multiple training translations, including: if at least one of the text format check result and translation length check result of the training translation indicates that the check failed, the aggregate reward value of the training translation is determined to be a first preset reward value.

[0087] According to an embodiment of the present invention, the first preset reward value can be 0. For example, if at least one of the text format check results or the translation length check results indicates that the check failed, the aggregate reward value of the training translation can be directly determined to be 0.

[0088] According to an embodiment of the present invention, the aggregate reward value of each training translation is determined based on the text format check result, translation length check result, and entity matching check result of each of the multiple training translations. This includes: determining the reward value of the text format check result as a second preset reward value if the text format check result and translation length check result of the training translation are passed; determining the reward value of the translation length check result based on a preset translation field length and a tolerance coefficient, and matching multiple training words in the training translation with an entity set to obtain an entity matching check result; determining the reward value of the entity matching result if the entity matching check result indicates that there are entities in the entity set that match multiple training words one-to-one; and determining the aggregate reward value of the training translation based on the reward value of the text format check result, the reward value of the translation length check result, and the reward value of the entity matching result.

[0089] Formatted gating command ,in, This represents the reward value (or format gating function) for the text format check result. If the text format check result does not meet the preset format (such as preset structural requirements), the reward value is awarded. <think> / < / think> If the text format check result meets the preset format, the gate is set to 0 (i.e., the reward value of the text format check result is the first preset reward value 0); if the text format check result meets the preset format, the gate is set to 1 (the reward value of the text format check result is the second preset reward value, which can be 1).

[0090] Length gating is used to prevent translation models from cheating by outputting excessively long translations to enumerate all possible candidate entities and thus gain rewards. Therefore, length gating is defined as follows:

[0091] (3);

[0092] Set the preset translation field length (such as the average length of the reference translation). , This is the tolerance factor, used to control the maximum allowable length as a multiple of the reference length. This represents the reward value (or length gating function) for the translation length check result. This indicates the indicator function. Length gating is used to determine the length of the generated training translations. If the output is within the allowed range, a positive reward (e.g., 1) is given; otherwise, a negative reward (e.g., 0) is given to penalize excessively long outputs. It should be noted that both the training translation on the training side and the target translation on the inference side can be... express.

[0093] Notice, It's a reward-side hard gate that sets the reward to zero for degenerate outputs without directly modifying the policy gradient. The subsequent importance ratio provides length normalization on the gradient side. These two mechanisms complement each other, jointly suppressing length exploitation at different stages of the optimization process.

[0094] Multiple training words in the training translation are matched against an entity set to obtain entity matching results. For example, normalized entity matching is performed between multiple training words in the training translation and an entity set. Each training translation is labeled with a set of acceptable entity sets. The entity set can be an alias field from Wikidata, covering various legitimate variations of the same entity in the target cultural context (such as different spellings, transliterations, abbreviations, etc.), thereby minimizing false negatives where the translation model outputs legitimate variations that are not completely consistent with the main reference.

[0095] The deterministic matching function is defined as follows:

[0096] (4);

[0097] The reward value represents the entity matching result and is used to determine whether the training translation produces an appropriate expression in the target cultural context (such as entity set). (There are entities that match one-to-one with multiple training lexical units). For the normalized entity g, the entity set Including at least one .remember For the normalized training translation, the normalization function is... Perform lowercase conversion and remove diacritics. For indicator functions, express It is a substring of 'b'. This indicator function determines... Does it exist in If the entity matches, the reward value for the matching result is 1; if the entity does not match, the reward value for the matching result is 0.

[0098] The training translation is eligible for a reward only if both the format gate and length gate conditions are met. The aggregate reward function is defined as follows:

[0099] (5);

[0100] This is a basic format reward used to reward well-structured responses; This is the reward value for the entity matching result. It can be set in the experiment. , Intuitively, if either structural gate is not satisfied, the response receives no reward; if both gates are satisfied, the response receives the basic reward. And receive an extra reward when the entity is correct.

[0101] A reward function with three properties—verifiability, alignment, and stability—was constructed. Verifiability is achieved by directly calculating from labeled datasets without requiring a learning-based reward model; alignment is achieved by directly measuring the correctness of cross-cultural entities; and stability is ensured by robustly resisting reward hacking during policy optimization.

[0102] According to an embodiment of the present invention, the advantage value of each training translation is determined based on the text format check results, translation length check results, and entity matching check results of each of the multiple training translations. This includes: determining the aggregate reward value of each training translation based on the text format check results, translation length check results, and entity matching check results of each of the multiple training translations; and normalizing the reward values ​​corresponding to the multiple training translations respectively, using the multiple training translations as the same reward group, to obtain the advantage value of each training translation.

[0103] Given the input training statement x to be translated, a historical lexical generation strategy is used. Mid-sampling generation Training Translation G is an integer greater than 1. Instead of relying on learning-based commentators, each training translation is compared to the same reward group (the same training sentence to be translated, sampled x). The rewards for other training translations were normalized to estimate the advantage:

[0104] (6);

[0105] To train the advantage value of the translation, and They are in the same reward group The sample mean and standard deviation are given, i = 1, 2, ..., G. This normalization stabilizes the training process and makes the absolute scale of the reward irrelevant.

[0106] According to an embodiment of the present invention, the importance ratio is determined based on the field lengths of multiple training translations, the current lexical generation strategy of the translation model, and the historical lexical generation strategy, including: calculating the probability ratio of the current lexical generation strategy and the historical lexical generation strategy of the translation model in the training translations; and dividing the probability ratio by the field length of the training translations in exponential space to obtain the importance ratio.

[0107] According to an embodiment of the present invention, the field length of the training translation It can also be the length of the training translation. It should be noted that each time step generates one target word for a field; therefore, the total number of time steps is equal to the field length, and t belongs to (1, ...). ).

[0108] To mitigate the problem of excessively large or small gradients on long sequences, importance ratios are calculated at the sequence level. And perform length normalization:

[0109] (7);

[0110] This formula represents the probability ratio of the current lexical generation strategy to that of historical lexical generation strategies in the training translation. It provides length normalization on the gradient side, automatically reducing the update magnitude when the sequence is long to avoid overly aggressive updates; simultaneously, it allows meaningful policy drift as the reward continues to improve. This is related to the length gating on the reward side. Collaboration constitutes a dual protection against length-exploitation behavior.

[0111] According to an embodiment of the present invention, the alternative objective function is constructed based on a first pruning threshold, a second pruning threshold, the advantage value and importance ratio of each training translation.

[0112] and These are the first pruning threshold and the second pruning threshold, used to limit the magnitude of single-step policy updates to prevent training instability. Asymmetric pruning is used to encourage updates in the exploration direction.

[0113] According to an embodiment of the present invention, the parameters of the translation model are iteratively updated based on the substitution objective function, including: for each training translation, calculating a first product of the importance ratio and the advantage value; pruning the probability ratio using different first or second pruning thresholds according to the positive or negative direction of the advantage value to obtain the pruned ratio, and calculating a second product of the pruned ratio and the advantage value, wherein the first pruning threshold is used to prune the probability ratio when the advantage value is positive, and the second pruning threshold is used when the advantage value is negative; selecting the minimum value between the first and second products as the individual substitution objective value of the training translation; averaging the individual substitution objective values ​​of multiple training translations corresponding to the same training sentence to obtain the group average substitution objective value; and backpropagating to update the parameters of the translation model with the group average substitution objective value as the update direction.

[0114] Based on the verifiable rewards mentioned above, a no-commentator policy gradient algorithm with pruning is used to optimize the policy network. It integrates the advantages of in-group normalization, sequence-level importance ratio, and asymmetric pruning techniques.

[0115] The objective function is replaced by maximizing the group average as the update direction, and the parameters of the translation model are updated via backpropagation. as follows:

[0116] (8);

[0117] For the expectation operator, For the training translation set containing the training statement x to be translated, Represents a set of training translations y i Historical Lexical Generation Strategy ; and These are the first and second clipping thresholds, which are distinct from each other. Importance ratio With advantage value First product ;clip(.) is the clipping function, which determines the cropping ratio. Importance ratio Restricted to the interval [1−ε] low ,1+ε high [Inside]. This is the key operation for implementing "with clipping". min(.) is the function to find the minimum value.

[0118] Figure 3 A flowchart illustrating the reasoning method of a translation model according to an embodiment of the present invention is shown.

[0119] like Figure 3 As shown, the reasoning method of the translation model includes operations S311~S340.

[0120] In operation S310, the sentence to be translated is input into the translation model.

[0121] When operating S320, the translation model generates inference trajectories surrounded by thought labels.

[0122] In operation S330, the inference trajectory is verified in the context of the target cultural context, and the target translation is generated.

[0123] When operating S340, the output includes a complete output sequence of inference trajectory and target translation.

[0124] Figure 4 A flowchart illustrating a training method for a translation model according to an embodiment of the present invention is shown.

[0125] like Figure 4 As shown, the reasoning method of the translation model includes operations S401~S411.

[0126] In operation S401, multiple training translations are generated by sampling through the translation model for the same training sentence to be translated.

[0127] In operation S402, determine whether the training translation has a thinking label. If yes, execute operation S403; otherwise, execute operation S404.

[0128] In operation S403, determine whether the field length of the training translation exceeds the tolerance coefficient. If yes, execute operation S405; otherwise, execute operation S406.

[0129] In operation S404, it was determined that the format gating failed and the aggregate reward value of the training translation was determined to be 0.

[0130] In operation S405, it is determined that the length gating fails and the aggregate reward value of the training translation is determined to be 0.

[0131] In operation S406, multiple training lexical units in the training translation are matched with the entity set.

[0132] In operation S407, if the entity matching check result indicates that there are entities in the entity set that match one-to-one with multiple training lexical units, the reward value of the entity matching result is determined.

[0133] In operation S408, the aggregate reward value of the training translation is determined based on the reward value of the text format check result, the reward value of the translation length check result, and the reward value of the entity matching result.

[0134] In operation S409, the advantage value of each training translation is determined based on the aggregated reward value of each of the multiple training translations.

[0135] In operation S410, the importance ratio of each training translation is calculated.

[0136] In operation S411, the parameters of the translation model are iteratively updated based on the alternative objective function of asymmetric pruning.

[0137] After operating S411, continue iteratively updating the translation model.

[0138] The translation method based on entity-anchored verifiable reward reinforcement learning, also known as Cross-Cultural Machine Translation Reinforcement Learning with Verifiable Rewards (CCMT-RLVR), proposed in this application models the cross-cultural translation task as a sequential decision-making process. It incentivizes the translation model to actively utilize its parameterized knowledge and generate entity translations that conform to the target culture's conventions through fully rule-based, automatically verifiable reward signals, without relying on external knowledge bases. The method involves having the translation model output a thought process before generating the final translation; anchoring the supervision signal to deterministic entity matching and suppressing reward hacking through structural gating; and using a commentator-free strategy to optimize and stabilize training by leveraging sequence-level importance ratios and within-group normalization advantages.

[0139] To comprehensively verify the effectiveness and robustness of the method of this invention, this embodiment conducted extensive experimental verification on the XC-Translate cross-cultural entity translation benchmark and the WMT24++ general machine translation benchmark. Qwen3-8B and Qwen3-14B were selected as the base models, involving ten target languages ​​(ar, de, es, fr, it, ja, ko, th, tr, zh), and rigorous comparisons were made with mainstream data baseline methods. The experiments used three metrics for evaluation: Entity Translation Accuracy (ETA), character-level F-score (chrF), and XCOMET-XL. The training set contained only 7,278 samples, and the test set contained 49,606 samples, with no overlap between the entities in the training and test sets.

[0140] Table 1 shows the comparison results of CCMT-RLVR on the XC-Translate benchmark with the base model, the SFT baseline (Supervised Fine-Tuning Baseline), and various external baselines. Acc represents the accuracy, and chrF represents the character-level F value. The higher the chrF value, the closer the translation generated by the model is to the reference translation in terms of wording, and the smoother and more natural it is to read.

[0141] Table 1

[0142]

[0143] The results show that: (1) Standard SFT has almost no gain on unseen entities (Qwen3-14B only +0.22%), confirming that it is trapped in the memory trap; (2) CCMT-RLVR brings an entity accuracy improvement of +8.21% to +9.06% compared to the basic model, breaking through the imitation ceiling; (3) Qwen3-14B (31.87%) using CCMT-RLVR even surpasses Qwen3-235B-A22B (31.05%), which has a larger number of parameters, in entity translation accuracy, and significantly narrows the gap with GPT-5-mini (33.87%).

[0144] Despite being trained on only 7,000 cross-cultural entity datasets and never having been exposed to WMT data or general machine translation supervision, CCMT-RLVR's learned inference strategies can still be generalized to general machine translation tasks. Table 2 presents a comparison of XCOMET-XL scores on WMT24++.

[0145] Table 2

[0146]

[0147] The results show that CCMT-RLVR consistently outperforms the base model and the SFT baseline across all ten language directions, with an average XCOMET improvement of +1.25 to +1.35 points; after training on the full dataset, the improvement further reaches +1.59 to +1.68 points. More importantly, SFT's performance actually declines with increasing data size (suspected overfitting), while CCMT-RLVR's performance continuously improves with increasing data size, indicating that the verifiable reward-optimized landscape provided by this method has good scalability.

[0148] To verify that this method learns a general reasoning strategy rather than a language-specific mapping, ten target languages ​​were divided into group A (Asian and Semitic languages: ar, ja, ko, th, zh) and group B (European and Turkic languages: de, es, fr, it, tr). After training on one group, the other group was directly evaluated. The results showed that entity accuracy improvements of +7.27% and +6.28% were achieved in the A→B and B→A directions, respectively, indicating that this method learns reasoning abilities that can be transferred across language families, rather than language-specific entity mappings.

[0149] Ablation experiments show that removing format gating or length gating Both of these can lead to training failure: the response length grows uncontrollably and explodes, the model cheats by infinitely enumerating candidate entities to obtain rewards, and although the original reward curve is higher, the generated translation is no longer usable. This phenomenon confirms Goodhard's Law: when an unconstrained entity matching metric becomes the sole objective, the model will exploit its loopholes. CCMT-RLVR's dual structural gating effectively suppresses this degradation, ensuring that training converges to a concise and correct translation strategy.

[0150] Example 1: CCMT-RLVR Training Implementation Based on Qwen3-14B. The XC-Translate benchmark dataset was used, with its official validation set repositioned as the training set (7,278 entries) and the official test set used as the evaluation set (49,606 entries), covering 10 language directions (en→ar / de / es / fr / it / ja / ko / th / tr / zh). Strict zero entity overlap between the training and test sets was ensured. Each sample contains the source English sentence, the target language, and a set of entities $\mathcal{G}(x)$ from Wikidata.

[0151] Uses the standard chat template from the Qwen3 series. The input side uses the standard translation command: Translate the following sentence into {Target_language}.

[0152] Text: {Source_text}

[0153] No task-specific instructions are introduced during the training phase, allowing the model to autonomously learn inference strategies during the RL process.

[0154] The reward function is calculated as follows: Extracted from the model output $y$ <think> ...< / think> Translation section If parsing fails... ; Calculate the reference length and ,like ,but (set up );right and Each Perform normalization (lowercase conversion + de-diacritics), and check for existence. make ,get Aggregate reward function ,in .

[0155] Table 3 shows the training hyperparameters for CCMT-RLVR.

[0156] Table 3

[0157]

[0158] The thinking mode is enabled by default in the evaluation phase, with the maximum generation length uniformly set to 4096 tokens, the sampling temperature to 0.6, and top-$p$ to 0.95 (following the best practices of DeepSeek-R1 and Qwen3), so as to avoid repeated loops caused by greedy decoding on reasoning-oriented models. For all open-source model results, the average value of 3 independent samplings (pass@1) is taken.

[0159] The behavioral differences of CCMT-RLVR compared with the base model are demonstrated below through four specific cases:

[0160] Case 1: Standardization of historical terms (Great Ming Code → Chinese). The base Qwen3-8B adopts a combined literal translation, translating "Great Ming" as "Ming Chao" and "Code" as "Fa Dian", outputting the non-standard "Ming Chao Fa Dian". The thinking segment of CCMT-RLVR triggers hypothesis testing — "Great Ming Code most likely refers to Da Ming Lü... I should verify this standard translation", and finally correctly outputs the historical academic term "Da Ming Lü".

[0161] Case 2: Analogical reasoning of cultural conventions (Once Upon a Time in Venezuela → Chinese). The base Qwen3-8B literally translates "Once Upon a Time" as "Hen Jiu Hen Jiu Yi Qian", ignoring the film and television naming conventions. The thinking segment of CCMT-RLVR recalls the prototype entity — "Once Upon a Time in America is translated as Mei Guo Wang Shi", and applies this naming convention to output "Wei Nei Rui La Wang Shi".

[0162] Case 3: Domain-specific disambiguation (Mahavira Hall → Chinese). The base Qwen3-8B associates "Mahavira" with Jainism and resorts to transliteration as "Ma Ha Wei La Ting". CCMT-RLVR correctly identifies the Buddhist context constraint — "In Chinese Buddhist terms, Mahavira is often transliterated as Da Xiong", and finally outputs "Da Xiong Bao Dian".

[0163] Case 4: Localization of target language release conventions (Magic Emi → French). The base Qwen3-8B retains the original English name. CCMT-RLVR actively localizes the media title to "Emi magique", reflecting deep compliance with the target language release specifications.

[0164] The above cases collectively embody the "Pause-Retrieve-Verify" meta-strategy learned by CCMT-RLVR: (1) detecting potential cultural entities; (2) pausing instantaneous translation and searching for domain correspondences in parametric memory; and (3) verifying candidates in the target cultural context. This shift in thinking from "translating meaning" to "matching entities" is the core driver of performance improvement.

[0165] According to embodiments of the present invention, it can be synergized with Retrieval Enhancement Generation (RAG). Using mContriever as a multilingual intensive retrieval tool on Qwen3-8B, an index is built for entity aliases in the XC-Translate test set, and top-k relevant entity aliases are retrieved for each source sentence as contextual precedence. The synergistic effect of CCMT-RLVR with RAG compared to Qwen3-8B is shown in Table 4.

[0166] Table 4

[0167]

[0168] The results show that CCMT-RLVR can outperform the RAG baseline when used alone, indicating that the bottleneck in cross-cultural translation lies in "alignment" rather than "availability". At the same time, CCMT-RLVR+RAG can achieve further gains, and the two have a synergistic effect. The model trained by CCMT-RLVR still has the ability to flexibly utilize external evidence and will not overfit its own parameters.

[0169] According to an embodiment of the invention, a fine-grained analysis was performed by cultural category. The Qwen3-14B base version and the CCMT-RLVR version were evaluated on 14 cultural categories annotated by XC-Translate. Entity translation accuracy by cultural category is shown in Table 5.

[0170] Table 5

[0171]

[0172] All 14 categories achieved positive gains, confirming that the parameterized knowledge of CCMT-RLVR activation covers a broad cultural spectrum rather than being biased towards specific entity types.

[0173] This invention primarily addresses the technical problems in existing cross-cultural machine translation methods, such as insufficient utilization of model parameterized knowledge, excessive reliance on external knowledge bases, and unstable reinforcement learning training based on neural quality indicators.

[0174] Related technologies learn by imitating reference translations. When the entity sets in the training and test sets do not overlap, the model tends to memorize specific entity mapping relationships, making it difficult to form transferable cross-cultural reasoning strategies. Experiments show that on the XC-Translate benchmark, using 7k samples to train the Qwen3-14B model with standard SFT only brings a +0.22% improvement in entity translation accuracy, almost indistinguishable from the base model. This indicates that imitation-based learning struggles to overcome the "memory ceiling."

[0175] Secondly, regarding the problem of excessive reliance on external databases for external knowledge retrieval methods, existing translation methods based on Retrieval-Augmented Generation (RAG) shift the bottleneck of translation quality from contextual entity reasoning to the structure and coverage of external knowledge sources. The performance of the translation system is severely limited by the availability and retrieval quality of external databases, and it requires training or fine-tuning of dedicated retrieval machines for specific tasks, which increases engineering complexity and inference latency.

[0176] Furthermore, reinforcement learning methods based on neural quality metrics suffer from the "fluency trap" and reward hacking issues. While reinforcement learning methods that directly use neural quality metrics as reward signals can improve overall translation fluency (XCOMET on WMT24++ can improve from 74.46 to 78.32), entity accuracy degrades on culturally dense benchmarks (such as XC-Translate) (from 20.19% to 18.94%). This is because neural metrics lack the ability to fine-grainedly distinguish specific entities and cannot penalize fluent but culturally incorrect entity translations. In addition, without structural constraints, reinforcement learning models are prone to learning degenerate strategies during training, such as enumerating synonyms or outputting excessively long sequences to cheat for rewards, leading to training failures.

[0177] Figure 5 The diagram illustrates a structural block diagram of a translation device based on entity-anchored verifiable reward reinforcement learning according to an embodiment of the present invention.

[0178] like Figure 5 As shown, the translation device 500 based on entity anchoring verifiable reward reinforcement learning in this embodiment includes an input module 510 and a generation module 520.

[0179] The input module 510 is used to input the statement to be translated into the translation model and generate an inference trajectory with thought labels. The thought labels include at least one sub-label used to define the range of the inference trajectory in the complete output sequence of the translation model. The inference trajectory includes the inference and verification process of the translation model for the statement to be translated. In one embodiment, the input module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0180] The generation module 520 is used to generate a target translation in the target cultural context based on the verification indication information about target lexical units contained in the inference trajectory. The complete output sequence includes the inference trajectory and the target translation. The lexical unit generation strategy of the translation model includes: using the sentence to be translated and the generated historical lexical unit sequence as conditions, the translation model selects preset lexical units that match the target cultural context from a preset vocabulary at each time step as the target lexical unit for the current time step, until the end marker is generated. The historical lexical unit sequence includes multiple historical target lexical units arranged in time step order. In one embodiment, the generation module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0181] According to embodiments of the present invention, any plurality of modules in the input module 510 and the generation module 520 may be combined into one module, or any one of the modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the input module 510 and the generation module 520 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the input module 510 and the generation module 520 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0182] Figure 6 A block diagram of an electronic device suitable for implementing a translation method based on entity-anchored verifiable reward reinforcement learning according to an embodiment of the present invention is illustrated.

[0183] like Figure 6 As shown, an electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0184] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0185] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0186] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0187] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0188] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the translation method based on entity-anchored verifiable reward reinforcement learning provided in the embodiments of the present invention.

[0189] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0190] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0191] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0192] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0193] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0194] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0195] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.< / eos>

Claims

1. A translation method based on entity-anchored verifiable reward reinforcement learning, characterized in that, The method includes: The sentence to be translated is input into the translation model, and an inference trajectory with thinking labels is generated. The thinking labels include at least one sub-label for defining the range of the inference trajectory in the complete output sequence of the translation model. The inference trajectory includes the inference and verification process of the translation model on the sentence to be translated. Based on the verification indication information about target words contained in the inference trajectory, a target translation in the target cultural context is generated, and the complete output sequence includes the inference trajectory and the target translation; The lexical generation strategy of the translation model includes: using the sentence to be translated and the generated historical lexical sequence as conditions, the translation model selects preset lexicals that match the target cultural context from the preset vocabulary step by step, and uses them as the target lexicals for the current time step, until the end symbol is generated. The historical lexical sequence includes multiple historical target lexicals arranged in the order of time steps.

2. The method according to claim 1, characterized in that, The thinking tag includes a reasoning start sub-tag and a reasoning end sub-tag. The reasoning start sub-tag is used to define the starting position of the reasoning trajectory in the complete output sequence, and the reasoning end sub-tag is used to define the ending position of the reasoning trajectory in the complete output sequence.

3. The method according to claim 1, characterized in that, The translation model was trained using the following method: For the same training sentence to be translated, multiple training translations are generated by sampling through a translation model. The training translations include reasoning and thinking training texts and translation training texts. Based on the text format check results, translation length check results, and entity matching check results of multiple training translations, the advantage value of each training translation is determined. Based on the field lengths of multiple training translations, the current lexical generation strategy and the historical lexical generation strategy of the translation model, the importance ratio of each training translation is determined. The parameters of the translation model are iteratively updated based on an alternative objective function, wherein the alternative objective function is constructed based on the advantage value and the importance ratio of each training translation.

4. The method according to claim 3, characterized in that, The method further includes: The multiple training translations are checked according to a preset format that includes a complete response sequence containing thinking tags, reasoning training text, and translation training text, to obtain the text format check results; The training translations are checked according to the tolerance coefficient of the translation field length to obtain the translation length check result.

5. The method according to claim 3, characterized in that, The process of determining the advantage value of each training translation based on the text format check results, translation length check results, and entity matching check results of each of the multiple training translations includes: Based on the text format check results, translation length check results, and entity matching check results of each of the training translations, the aggregate reward value of each training translation is determined. Using multiple training translations as the same reward group, the reward values ​​corresponding to the multiple training translations are normalized to obtain the advantage value of each training translation.

6. The method according to claim 5, characterized in that, The step of determining the aggregate reward value for each of the training translations based on the text format check results, translation length check results, and entity matching check results of each of the multiple training translations includes: If at least one of the text format check results and the translation length check results of the training translation fails, the aggregate reward value of the training translation is determined to be the first preset reward value.

7. The method according to claim 5, characterized in that, The step of determining the aggregate reward value for each of the training translations based on the text format check results, translation length check results, and entity matching check results of each of the multiple training translations includes: If the text format check result and the translation length check result of the training translation are both passed, the reward value of the text format check result is determined to be the second preset reward value; Based on the preset translation field length and tolerance coefficient, the reward value of the translation length check result is determined, and multiple training words in the training translation are matched with the entity set to obtain the entity matching check result; If the entity matching check result indicates that there are entities in the entity set that match one-to-one with multiple training lexical units, then determine the reward value of the entity matching result. The aggregate reward value of the training translation is determined based on the reward value of the text format check result, the reward value of the translation length check result, and the reward value of the entity matching result.

8. The method according to claim 3, characterized in that, The determination of importance ratios based on the field lengths of multiple training translations, the current lexical generation strategy of the translation model, and the historical lexical generation strategy includes: Calculate the probability ratio of the current lexical generation strategy to the historical lexical generation strategy of the translation model in the training translation; The importance ratio is obtained by dividing the probability ratio by the field length of the training translation in exponential space.

9. The method according to claim 8, characterized in that, The alternative objective function is constructed based on a first pruning threshold, a second pruning threshold, the advantage value of each of the training translations, and the importance ratio.

10. The method according to claim 9, characterized in that, The iterative update of the translation model parameters based on the alternative objective function includes: For each of the training translations, Calculate the first product of the importance ratio and the advantage value; Based on the positive or negative direction of the dominance value, the probability ratio is clipped using a first clipping threshold or a second clipping threshold that are different from each other to obtain a clipped ratio. The second product of the clipped ratio and the dominance value is then calculated. Specifically, when the dominance value is positive, the probability ratio is clipped using the first clipping threshold, and when the dominance value is negative, the probability ratio is clipped using the second clipping threshold. The minimum value between the first product and the second product is selected as the individual replacement target value of the training translation; The average of the individual substitution target values ​​of multiple training translations corresponding to the same training statement is used to obtain the group average substitution target value. The parameters of the translation model are updated by backpropagation, with the objective value of maximizing the group average as the update direction.

Images