Data processing method and device and domain translation model training method and device

Abstract

The embodiment of the invention provides a data processing method and device and a domain translation model training method and device. The data processing method comprises the steps that a term word list set and a source language text set of a target domain are obtained; comparing the term word list set with the source language text set, and determining a long-tail term word list and an uncovered source language text; generating a long-tail source language text corresponding to the long-tail term word list based on the text generation model, inputting the long-tail term word list and the long-tail source language text into a data translation model to obtain a first translation text corresponding to the long-tail source language text, and inputting the uncovered source language text into the data translation model to obtain a second translation text corresponding to the uncovered source language text; generating a second translated text corresponding to the uncovered source language text; determining field data of the target field based on the long-tail source language text, the first translated text, the uncovered source language text and the second translated text; by performing data synthesis on the long-tail term word list and the uncovered source language text, the problem of corpus scarcity in the professional field is effectively solved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, in particular to a data processing method, one or more embodiments of the present specification also relate to a data processing apparatus, a domain translation model training method, a domain translation model training apparatus, a computing device, a computer-readable storage medium, and a computer program product. BACKGROUND

[0002] In a translation scenario, it is usually necessary to use translation software or tools to translate source language text into target language text. With the increasing application of artificial intelligence, users can use large language models, neural machine translation, etc. to realize language translation.

[0003] However, for professional field translation tasks, general translation tools, large language model translation, and neural machine translation cannot accurately translate professional field terminology and are prone to hallucinations and incorrect translations. Therefore, it is necessary to train a translation model for the corresponding field using professional field training data to improve translation accuracy. However, high-quality professional field corpus is extremely scarce and the cost of manual annotation is high. SUMMARY

[0004] Therefore, embodiments of the present specification provide a data processing method. One or more embodiments of the present specification also relate to a data processing apparatus, a domain translation model training method, a domain translation model training apparatus, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0005] According to a first aspect of an embodiment of the present specification, a data processing method is provided, comprising: obtaining a set of term glossaries of a target domain and a set of source language texts, wherein each term glossary in the set of term glossaries includes a source language term and a corresponding target language term; comparing the set of term glossaries with the set of source language texts to determine long-tail term glossaries in the set of term glossaries and uncovered source language texts in the set of source language texts; generating long-tail source language texts corresponding to the long-tail term glossaries based on a text generation model, inputting the long-tail term glossaries and the long-tail source language texts into a data translation model to obtain first translation texts corresponding to the long-tail source language texts, and inputting the uncovered source language texts into the data translation model to generate second translation texts corresponding to the uncovered source language texts; determine the domain data corresponding to the target domain based on the long-tail source language text and the first translation text, the uncovered source language text and the second translation text.

[0006] According to a second aspect of the embodiments of the present specification, a data processing apparatus is provided, comprising: a data acquisition module configured to acquire a term vocabulary set of a target domain and a source language text set, wherein each term vocabulary in the term vocabulary set comprises a source language term and a corresponding target language term; a set matching module configured to compare the term vocabulary set with the source language text set, and determine a long-tail term vocabulary in the term vocabulary set and an uncovered source language text in the source language text set; a text translation module configured to generate a long-tail source language text corresponding to the long-tail term vocabulary based on a text generation model, input the long-tail term vocabulary and the long-tail source language text into a data translation model, obtain a first translation text corresponding to the long-tail source language text, input the uncovered source language text into the data translation model, and generate a second translation text corresponding to the uncovered source language text; a data determination module configured to determine domain data corresponding to the target domain based on the long-tail source language text and the first translation text, the uncovered source language text and the second translation text.

[0007] According to a third aspect of the embodiments of the present specification, a domain translation model training method is provided, comprising: determining an initial translation model and domain data of a target domain, wherein the domain data is obtained by the above-mentioned data processing method; training the initial translation model using the domain data corresponding to the target domain to obtain a trained domain translation model.

[0008] According to a fourth aspect of the embodiments of the present specification, a domain translation model training apparatus is provided, comprising: a determination module configured to determine an initial translation model and domain data of a target domain, wherein the domain data is obtained by the above-mentioned data processing method; a training module configured to train the initial translation model using the domain data corresponding to the target domain to obtain a trained domain translation model.

[0009] According to a fifth aspect of the embodiments of the present specification, a computing device is provided, comprising: a memory and a processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which realize the steps of the data processing method and the field translation model training method.

[0010] According to a sixth aspect of an embodiment of the present specification, a computer-readable storage medium is provided, which stores computer programs / instructions, which realize the steps of the data processing method and the field translation model training method when executed by a processor.

[0011] According to a seventh aspect of an embodiment of the present specification, a computer program product is provided, which includes computer programs / instructions, which realize the steps of the data processing method and the field translation model training method when executed by a processor.

[0012] The data processing method provided by the embodiments of the present specification obtains a term vocabulary set of a target field and a source language text set, accurately locates long-tail term vocabularies lacking context instances and uncovered source language texts not covered by the term vocabulary set by comparing the term vocabulary set and the source language text set, automatically constructs long-tail source language texts containing context by using a text generation model for the long-tail term vocabulary, and obtains corresponding first translation texts by using a data translation model; for the uncovered source language texts, the data translation model can be directly used for translation, thereby efficiently expanding second translation texts of general expressions, and integrating the long-tail source language texts and the corresponding first translation texts, the uncovered source language texts and the corresponding second translation texts, to form a field data with accurate terms, rich context and more comprehensive coverage, that is, by data synthesis of the long-tail term vocabulary and the uncovered source language texts, the problem of corpus scarcity in professional field translation is effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a flowchart of a data processing method provided by an embodiment of the present specification; Figure 2 is a flowchart of a field translation model training method provided by an embodiment of the present specification; Figure 3 is a schematic diagram of a processing process of a data processing method provided by an embodiment of the present specification; Figure 4 is a structural schematic diagram of a data processing device provided by an embodiment of the present specification; Figure 5 is a structural schematic diagram of a field translation model training device provided by an embodiment of the present specification; Figure 6 is a structural block diagram of a computing device provided by an embodiment of the present specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0019] Data distillation: A teacher model is used to generate "pseudo-labels" (such as the probability distribution of the next word, complete answers, etc.) on a large amount of unlabeled text. Then, supervised distillation is performed by constructing a high-quality distillation dataset and fine-tuning the data with instructions.

[0020] Continual pre-training: refers to the process of using new unlabeled text data to continue pre-training on the basis of an existing pre-trained language model, in order to expand the model's knowledge, adapt to new fields, or improve overall language understanding capabilities. Data synthesis: refers to the process of artificially generating training samples that do not exist originally, but have realistic data characteristics and semantically reasonable characteristics through algorithms, rules, models, or other automated means. The core goal is to expand the scale of training data, improve data diversity, make up for the scarcity of labeled data or privacy restrictions, etc., especially in large model training, instruction fine-tuning, security alignment, etc. RAG: Retrieval-Augmented Generation, is a hybrid artificial intelligence architecture that combines information retrieval (Retrieval) and text generation (Generation), aiming to enable language models to dynamically reference new, authoritative or domain-specific information from external knowledge sources when generating answers, thereby improving the accuracy, timeliness and explainability of answers.

[0021] Large model translation: refers to the use of a general large language model with a large number of parameters (usually ≥7B) to make it have cross-language translation capabilities through prompt engineering or fine-tuning. Such models are not designed specifically for translation, but due to exposure to a large amount of multilingual text during pre-training, they exhibit strong zero-shot or multilingual generalization capabilities. Neural machine translation: uses deep neural networks to learn translation mappings end-to-end, which can significantly improve translation fluency and accuracy.

[0022] Users often face cross-language retrieval and reading barriers when searching and reading literature or other materials, which reduces efficiency, and for more specialized fields, general translation models are limited by professional knowledge, which can lead to inaccurate translations. For example, in the medical field, general translation tools may not accurately grasp medical terminology, and existing large model translation and neural machine translation can produce hallucinations and mistranslations, making it difficult to meet the accuracy requirements of clinical practice.

[0023] Specifically, while general large model translation has zero-shot multilingual capabilities, its core weakness is the lack of domain expertise. It has not been systematically trained on medical corpus and has weak understanding of professional terminology, abbreviations and clinical expressions, which can lead to "smooth but factually incorrect" hallucination translation. For example, it may mistranslate "negative for malignancy" as "malignant tumor negative" or even invent non-existent medical concepts. In addition, its translation behavior is highly dependent on prompt design, resulting in unstable results and inability to guarantee term consistency or alignment with authoritative medical standards.

[0024] Traditional neural machine translation (NMT) performs well on general-purpose languages, but it also faces the problem of insufficient professional adaptation. Such models mainly rely on large-scale general parallel corpus training, and the proportion of medical bilingual data is extremely low, which leads to a sharp drop in accuracy when dealing with complex medical sentence patterns (such as negation, condition, and temporal logic). At the same time, NMT tends to be "conservative and direct translation", which is difficult to capture the implied meaning in the clinical context, for example, it cannot distinguish the clinical differences between "history of diabetes" (diabetes history) and "active diabetes" (current illness). More importantly, its closed architecture cannot integrate domain knowledge bases and cannot dynamically calibrate specific terminology.

[0025] That is, none of the above methods embed medical knowledge systems, and it is difficult to meet the high requirements of medical scenarios in terms of terminology accuracy, context understanding, and clinical reliability. If used directly for medical record translation, patient communication, or drug instructions, there is a significant safety risk.

[0026] If you want to train a specialized medical translation model, high-quality medical corpus is extremely scarce and the cost of manual annotation is high. Therefore, there is an urgent need for a data processing method to obtain training data for training a professional field translation model corresponding to the field.

[0027] In the present specification, a data processing method is provided, and the present specification also relates to a data processing apparatus, a field translation model training method, a field translation model training apparatus, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0028] Referring to Figure 1 , Figure 1 A flowchart of a data processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0029] Step 102: Obtain a set of term glossaries of a target field and a set of source language texts, wherein each term glossary in the set of term glossaries includes a source language term and a corresponding target language term.

[0030] Wherein, the target field can be understood as any professional field, such as the medical field, the legal field, the education field, etc., which is not limited here. The set of term glossaries can be understood as a set of term correspondence tables corresponding to the target field. The term glossary is a term correspondence table, which is usually in the form of "source language term: target language term", for example, a term glossary is "hydrorachitis: inflammatory spinal canal hydrops". The set of source language texts can be understood as a source language text library to be translated or used for training. The lengths of the source language texts in the set of source language texts are not the same, and the source language texts can be sentences or paragraphs, and the specific division standard is not limited here.

[0031] Specifically, the source of the term glossary can be a standardized term library, expert manual annotation, or authoritative document extraction, etc., which is not limited here; the source language text set can be obtained from source language documents, technical reports, etc., which usually covers professional expressions in the target field.

[0032] For example, in the medical field, the term glossary set can include multiple Chinese-English medical term glossaries, and the source language text set can include multiple English texts of sentence segments in medical documents. After obtaining the term glossary set and the source language text set in the target field, the subsequent translation of the source language text can be based on the term glossary to accurately map the terms, providing structured knowledge support for translation.

[0033] The method provided by the embodiments of the present specification provides data support for subsequent construction of translation data by obtaining high-quality, widely-covered term resources and real text corpus.

[0034] In one or more embodiments of the present specification, in the case of obtaining an initial glossary set including at least one initial term glossary through multiple sources, because there are inconsistencies in translations of the same term by different data sources, or spelling errors, etc., it is necessary to check the initial glossary set to obtain a high-quality, unified term glossary set. The specific implementation is as follows: Obtaining a term glossary set in a target field, comprising: Obtaining an initial glossary set in the target field, performing term unification and quality filtering on at least one initial term glossary in the initial glossary set, and obtaining the term glossary set.

[0035] Wherein, the initial glossary set can be understood as a set of term contrast tables collected from multiple data sources, which have not been standardized. The initial term glossary in the term contrast table set may have problems such as inconsistent term expression and inaccurate translation, for example, the same term may have two expressions "hypertension: high blood pressure" and "high blood pressure" in different sources, or the translation is not unified, such as "cancer: cancer / tumor".

[0036] The term unification can be understood as a process of standardizing the expression of the initial term table, ensuring that the same concept has a consistent source language term and target language term correspondence. For example, in the medical field, "hypertension: high blood pressure" and "high blood pressure: high blood pressure" are unified as "hypertension: high blood pressure". Quality filtering can be understood as a process of filtering and cleaning the initial term table based on accuracy, field relevance, etc.

[0037] Specifically, a set of initial term tables for the target field is obtained from a variety of data sources, such as 100 initial term tables obtained based on expert manual annotation, 200 initial term tables obtained by term extraction and translation from authoritative literature in the target field, and 300 initial term tables in the initial term table set based on the obtained initial term tables.

[0038] The initial term table set is subjected to term unification to ensure that the term corresponding to a unified concept is a term table. The term unification process can be implemented through a language model, and the initial term table set can be subjected to quality filtering through manual review, rule matching, model scoring, etc. to eliminate incorrect translations, non-professional expressions, etc. in the initial term table, and to improve the overall quality and usability of the term table.

[0039] For example, in the manual review stage, medical experts can filter the quality of the automatically collected term table pair, eliminate general terms such as "cold: cold", and retain professional terms such as "hypertension: high blood pressure". After term unification and quality filtering, the initial term table set can be merged, repeated, unified, and deleted, and the like, to finally obtain a term table set containing 150 terms.

[0040] The data processing method provided by the embodiments of the present specification can construct a high-quality, standardized, and field-adapted term table set by performing term unification and quality filtering on at least one initial term table in the initial term table set, providing accurate, consistent, and reliable knowledge support for subsequent translation processes, thereby improving the term accuracy of professional translation.

[0041] Step 104: Comparing the term table set with the source language text set to determine the long-tail term table in the term table set and the uncovered source language text in the source language text set.

[0042] The long-tail terminology vocabulary can be understood as a terminology vocabulary with a low frequency or not appearing in the source language text. The uncovered source language text can be understood as a source language text in the source language text set that does not contain any corresponding term in the terminology vocabulary, i.e., the source language term in the terminology vocabulary does not appear in the source language text. The uncovered source language text belongs to the Out-Of-Vocabulary (OOV) case.

[0043] Specifically, the efficient substring matching method is used to compare the terminology vocabulary set and the source language text set in both directions. For example, the AC automatic machine (Aho-Corasick algorithm) is used to load the terminology vocabulary set in advance, the terminology vocabulary in the terminology vocabulary set is constructed into a multi-mode matching machine, the source language text set is scanned, and the frequency of each terminology vocabulary is counted. According to the frequency, the long-tail terminology vocabulary (with a very small number of occurrences or zero) can be determined from the terminology vocabulary set. The source language text in the source language text set that does not match any terminology vocabulary, i.e., the uncovered source language text, is identified.

[0044] In a specific implementation, when the language of the source language text is the source language, the comparison between the terminology vocabulary set and the source language text set is actually a comparison between the source language terms in the terminology vocabulary and the source language text, i.e., to determine whether the corresponding source language term exists in the source language text. When one or more source language terms in the terminology vocabulary set are located and identified in the source language text, it is determined that the source language text matches the terminology vocabulary corresponding to the one or more source language terms, and the frequency of the related terminology vocabulary is accumulated and counted.

[0045] The data processing method provided by the embodiments of the present specification accurately identifies the long-tail terminology vocabulary in the terminology vocabulary set and the uncovered source language text in the source language text set by comparing the terminology vocabulary set and the source language text set, and provides a basis for subsequent differentiated processing. In the case where the long-tail terminology vocabulary and the uncovered source language text can be processed subsequently, the coverage of the field data can be finally obtained.

[0046] In one or more embodiments of the present specification, when the terminology vocabulary set and the source language text set are compared, the target terminology vocabulary and the corresponding target source language text of the bidirectional matching can be determined, so that the target terminology vocabulary is added to the prompt text as additional knowledge when the target source language text is translated. The specific implementation is as follows: After comparing the terminology vocabulary set and the source language text set, the method further includes: determining a target terminology list and corresponding target source language text that match successfully, and inputting the target terminology list and the target source language text into the data translation model to obtain a third translation text corresponding to the target source language text.

[0047] The target source language text can be understood as a source language text containing one or more target terminology lists, and the target terminology list can be understood as a terminology list in the terminology list set that successfully matches the target source language text, i.e., a matched terminology list in the target source language text. For example, in a medical literature fragment, the terms “cancer” and “hydrorachitis” contained therein have corresponding terminology lists in the terminology list set, and these terminology lists and the source language text in which they are located are the matched target terminology list and target source language text.

[0048] The data translation model can be understood as a teacher model with powerful performance and multilingual translation function, which can generate high-quality translation text. In this embodiment, the data translation model can be a large language model. The third translation text can be understood as a translation text in the target language obtained by translating the target source language text by the data translation model.

[0049] Specifically, in the case of comparing the terminology list set with the source language text set, a comparison result is determined, which includes the occurrence frequency of each terminology list and the number of matched terms contained in each source language text, etc. According to a preset matching threshold (such as terminology list occurrence frequency ≥ 50, and the number of matched terms contained in the source language text ≥ 2), the target terminology list and the target source language text that meet the conditions are screened out.

[0050] In actual application, the matching threshold includes a terminology list occurrence frequency threshold and a matched term number threshold. The terminology list set can be divided into a target terminology list (occurrence frequency greater than or equal to the occurrence frequency threshold) and a long-tail terminology list (occurrence frequency less than the occurrence frequency threshold) by the occurrence frequency threshold. The source language text set can be divided into a target source language text (containing a number of matched terms greater than or equal to the matched term number threshold) and an uncovered source language text (containing a number of matched terms less than the matched term number threshold) by the matched term number threshold. The matching threshold can be set according to actual conditions. For example, in the case of a source language text being a sentence, the matched term number threshold can be 1, i.e., a source language text matching one terminology list can be determined as a target source language text. When the source language text is a text paragraph, the matched term number threshold can be set to 3, which is not limited herein.

[0051] In specific implementation, the idea of retrieval augmentation generation (RAG) can be used to splice the successfully matched target terminology glossary in the form of "source language terminology: target language terminology" into the prompt word context, and input the same into the data translation model together with the corresponding target source language text, so that the data translation model can refer to the provided target terminology glossary when generating the translation of the target source language text, and ensure the accurate translation of professional terminology.

[0052] The data processing method provided by the embodiments of the present specification realizes the explicit injection of terminology knowledge and the controllable guidance of the translation process by inputting the matched target terminology glossary and the corresponding target source language text into the data translation model, and effectively improves the accuracy and consistency of professional terminology in the translated text.

[0053] In one or more embodiments of the present specification, the long-tail terminology glossary is determined from the terminology glossary set and the uncovered source language text is determined from the source language text set by counting the number of successful terminology matches corresponding to each terminology glossary and the number of successful text matches corresponding to each source language text in the comparison process. The specific implementation is as follows: The comparison of the terminology glossary set and the source language text set determines the long-tail terminology glossary in the terminology glossary set and the uncovered source language text in the source language text set, comprising: Each terminology glossary in the terminology glossary set is compared with each source language text in the source language text set, and in the case that the source language text contains the source language terminology in the terminology glossary, it is determined that the source language text and the terminology glossary corresponding to the source language terminology match successfully; In the case where the comparison is completed, the number of successful terminology matches corresponding to each terminology glossary and the number of successful text matches corresponding to each source language text are determined; The long-tail terminology glossary is determined from the terminology glossary set according to the number of successful terminology matches, and the uncovered source language text is determined from the source language text set according to the number of successful text matches.

[0054] Wherein, the number of successful terminology matches can be understood as the frequency of the source language terminology in each terminology glossary being identified in the source language text set in the comparison process, i.e. the occurrence frequency of each terminology glossary in the above embodiments; the number of successful text matches can be understood as the number of successfully matched terminology glossaries in each source language text, i.e. the number of matched terminologies contained in each source language text in the above embodiments.

[0055] Specifically, in the process of comparing each term glossary in the term glossary set with each source language text in the source language text set, the occurrence frequency (i.e., the number of successful term matches) of each term glossary and the number of matched terms (i.e., the number of successful text matches) in each source language text can be counted. According to the number of successful term matches of each term glossary, the term glossaries in the term glossary set can be divided into target term glossaries and long-tail term glossaries (with a very small number of successful matches or zero). According to the number of successful text matches, the source language texts in the source language text set can be divided into target source language texts and uncovered source language texts.

[0056] For example, in the medical field, a medical English document (i.e., a source language text set) containing 3,000 sentence segments and a term glossary set containing 300 term glossaries are obtained. The medical English document and the term glossary set are compared. If the source language term "cancer" appears 500 times in the medical English document, the occurrence frequency is greater than or equal to the occurrence frequency threshold 100, and it can be determined that the term glossary corresponding to the source language term belongs to a high-frequency term glossary. The source language term "hydrorachitis" may appear 10 times (less than the occurrence frequency threshold 100) in the medical English document, and the term glossary corresponding to the source language term can be classified as a long-tail term glossary. The description of the statistical method in the medical English document may not contain any professional disease or drug terms, and the description of the statistical method can be determined as an uncovered source language text.

[0057] The data processing method provided by the embodiments of the present specification can identify long-tail term glossaries and uncovered source language texts by quantitatively counting the occurrence frequency of term glossaries and the number of matched terms in source language texts, provide data basis for subsequent differentiated translation, and ensure that the field data obtained through translation can have a wider coverage.

[0058] Step 106: generating a long-tail source language text corresponding to the long-tail term glossary based on a text generation model, inputting the long-tail term glossary and the long-tail source language text into a data translation model to obtain a first translation text corresponding to the long-tail source language text, and inputting the uncovered source language text into the data translation model to generate a second translation text corresponding to the uncovered source language text.

[0059] The text generation model can be understood as a large language model or other language model with text generation capability, which can generate logical source language texts in the target field based on given source language terms. The long-tail source language text can be understood as a context text containing source language terms in the long-tail term glossary, which is generated based on the long-tail term glossary, to provide real and usable context for subsequent translation.

[0060] Specifically, for the long-tail terminology vocabulary, the text generation model is called to generate a piece of source language text that is logically complete and related to the target field for the source language terminology in the long-tail terminology vocabulary. For example, in the case of "hydrorachitis: inflammatory spinal canal hydrops" as a long-tail terminology vocabulary, a prompt text "Please write a piece of English literature abstract containing the medical terminology 'hydrorachitis: inflammatory spinal canal hydrops' in a rigorous medical literature style for the user's reference, ensuring the logicality and scientificity of the text, and ensuring that the terminology in the text is consistent with the given terminology" can be constructed. By inputting the prompt text into the text generation model, the long-tail source language text corresponding to the long-tail terminology vocabulary is obtained.

[0061] Subsequently, the long-tail terminology vocabulary and the generated long-tail source language text are input into the data translation model. The data translation model can refer to the corresponding relationship between the source language terminology and the target language terminology in the long-tail terminology vocabulary to generate an accurate translation, and obtain the first translation text corresponding to the long-tail source language text.

[0062] For the uncovered source language text, the data translation model can be directly input, and the second translation text can be generated by relying on the general multilingual translation capability of the data translation model without the guidance of the terminology.

[0063] The data processing method provided by the embodiments of the present specification effectively solves the problem of lack of real corpus for long-tail terminology by synthesizing corresponding contexts for long-tail terminology vocabularies and subsequently translating in combination with the long-tail terminology vocabularies and the generated contexts. The general translation strategy is adopted for the uncovered text to ensure that the general language content is included in the field training data, so that the professional terminology can be learned and the translation capability of the general language can be retained when the field translation model is subsequently trained. The method improves the comprehensiveness and quality of the field data through full coverage and differentiated processing of terminology translation.

[0064] Step 108: determining the field data corresponding to the target field based on the long-tail source language text and the first translation text, the uncovered source language text and the second translation text.

[0065] The field data can be understood as training data suitable for a professional translation task in the target field, and the field data includes source language text and corresponding target language translation text.

[0066] Specifically, after generating the first translation text corresponding to the long-tail source language text and the second translation text corresponding to the uncovered source language text, the source language texts and the obtained translation texts are paired and arranged to obtain training data (i.e., field data) for training the field translation model corresponding to the target field.

[0067] In specific implementation, for long-tail source language text, a first translation pair is formed according to the long-tail source language text and a corresponding first translation text; for uncovered source language text, a second translation pair is formed according to the uncovered source language text and a corresponding second translation text; and for successfully matched target source language text, a third translation pair is formed according to the target source language text and a corresponding third translation text. The three types of translation pairs together constitute the field data of the target field. The specific implementation is as follows: Based on the long-tail source language text and the first translation text and the uncovered source language text and the second translation text, the field data corresponding to the target field is determined, including: determining a first translation pair based on the long-tail source language text and the first translation text; determining a second translation pair based on the uncovered source language text and the second translation text; determining a third translation pair based on the target source language text and the third translation text; obtaining the field data corresponding to the target field according to the first translation pair, the second translation pair and the third translation pair.

[0068] Among them, the translation pair can be understood as a pair of data composed of a source language text and its corresponding target language translation text. Specifically, the first translation pair is derived from the long-tail terminology scenario, and is composed of a long-tail source language text synthesized by a text generation model and a first translation text generated by a data translation model; the second translation pair is derived from the OOV scenario, and is composed of an uncovered source language text that does not match the terminology and a second translation text generated by a data translation model; and the third translation pair is derived from the successfully matched high-frequency terminology scenario, and is composed of a successfully matched target source language text and a third translation text generated by a data translation model.

[0069] Specifically, for long-tail terminology glossary, the first translation pair supplements the translation examples of long-tail terminology in real context, realizing enhanced translation for long-tail terminology glossary; for uncovered source language text, the second translation pair provides translation examples of general text; and for successfully matched target terminology glossary, the third translation pair constitutes a professional translation example with accurate terminology and real context. By collecting and organizing the three types of translation pairs, structured field data is finally formed.

[0070] The data processing method provided by the embodiments of the present specification can construct a wide coverage and diversified corpus through the first translation pair corresponding to the long-tail terminology vocabulary and the second translation pair, and the third translation pair provides a corpus with accurate terminology and real context. On this basis, by integrating the three types of translation pairs, rich and reliable domain data is obtained, which provides high-quality and diversified training data for subsequent training of a domain translation model, and can effectively improve the performance of the domain translation system in terms of terminology consistency, semantic accuracy and domain professionalism.

[0071] In one or more embodiments of the present specification, in the case of obtaining domain data, a general initial translation model can be trained using a domain model to obtain a domain translation model of a target domain. The specific implementation is described as follows: After obtaining the domain data corresponding to the target domain, the method further includes: Determining an initial translation model, training the initial translation model using the domain data corresponding to the target domain, and obtaining a trained domain translation model.

[0072] The initial translation model can be understood as a basic model with general translation capability. The initial translation model is not trained specifically with domain data, and may have problems of insufficient accuracy and consistency in professional terminology translation. The domain translation model can be understood as a specialized translation model with domain adaptability after being trained with domain data corresponding to the target domain, which can achieve more accurate and professional translation in the target domain.

[0073] Specifically, after obtaining the domain data corresponding to the target domain (i.e., including the first translation pair, the second translation pair, and the third translation pair), the initial translation model is trained using these translation pairs. During the training, various training strategies can be used, such as continuous pre-training and In-Context Finetuning, so that the domain translation model learns the accurate corresponding relationship of the terminology vocabulary corresponding to the target domain and the consistent expression of the context.

[0074] For example, in the medical translation scenario, the domain data containing a large number of medical translation pairs can be used to perform multi-stage fine-tuning on the general large model, so that it can accurately translate professional terms such as "huangqin" and "inflammatory spinal canal hydrops" when translating medical text, and maintain a rigorous sentence and clear logic.

[0075] The data processing method provided by the embodiments of the present specification trains the initial translation model using high-quality and domain-related domain data, thereby significantly improving the translation professionalism and accuracy of the domain translation model in the target domain, and meeting the translation needs in professional fields.

[0076] In one or more embodiments of the present specification, by constructing bidirectional translation data through domain data, a symmetric supervision signal can be formed, thereby not only improving translation quality, but also enhancing the understanding of bilingual terminology mapping of the domain translation model and reducing information loss in one-way translation. The specific implementation is described as follows. training the initial translation model using the domain data corresponding to the target domain to obtain a trained domain translation model, comprising: constructing bidirectional translation data according to the domain data corresponding to the target domain, wherein the bidirectional translation data includes forward translation data of source language text and corresponding translation text and reverse translation data of translation text and corresponding source language text; training the initial translation model using the bidirectional translation data to obtain the trained domain translation model.

[0077] Wherein, the bidirectional translation data can be understood as a corpus set containing forward translation data and reverse translation data, wherein the forward translation data refers to the corresponding relationship of source language text to target language translation text, and the reverse translation data refers to the corresponding relationship of target language translation text to source language, the forward translation data can be used to train the model to translate from source language to target language, while the reverse translation data can be used to enhance the model's understanding of the consistency of the translation result with the original text and improve the bidirectional alignment capability.

[0078] Specifically, based on the first translation pair, the second translation pair and the third translation pair in the domain data, bidirectional translation data can be constructed, that is, for each pair of source language text and target language translation text, forward translation data and reverse translation data are generated. For example, for the first translation pair, the constructed forward translation data is long-tail source language text->first translation text, and the reverse translation data is first translation text->long-tail source language text; for the second translation pair, the constructed forward translation data is uncovered source language text->second translation text, and the reverse translation data is second translation text->uncovered source language text; for the third translation pair, the constructed forward translation data is target source language text->third translation text, and the reverse translation data is third translation text->target source language text.

[0079] In the case of source language English and target language Chinese, the forward translation data is used to train the model to translate English into Chinese, and the reverse translation data is used to train the model to translate Chinese into English, so that the domain translation model obtained by training can maintain terminology accuracy in bidirectional translation tasks.

[0080] The data processing method provided by the embodiment of the present specification reduces the information loss in one-way translation by constructing and using bidirectional translation data to train an initial translation model, improves the ability of the field translation model in bidirectional alignment and term consistency in the target field, and thus forms a more robust and comprehensive field translation model.

[0081] In one or more embodiments of the present specification, training the initial translation model using the field data corresponding to the target field to obtain a trained field translation model, comprising: training the initial translation model using the field data corresponding to the target field to obtain an updated translation model; training the updated translation model using each term table in the set of term tables to obtain the trained field translation model.

[0082] The updated translation model can be understood as an intermediate model with certain field adaptability after preliminary training by field data. The updated translation model has learned the term knowledge of the target field on the basis of general translation capability, but has deficiencies in long-tail term memory.

[0083] Specifically, the training process of the initial translation model is divided into two stages. In the first stage, the initial translation model is trained using field data to preliminarily master the term knowledge of the target field, and an updated translation model is obtained. In the second stage, the set of term tables is focused on, and the updated translation model is intensively trained using the term tables, for example, using the instruction of "translating English terms into Chinese" or "translating Chinese terms into English", so that the trained field translation model can deeply remember and accurately output the translation of each term, especially long-tail terms.

[0084] The data processing method provided by the embodiment of the present specification uses a two-stage training strategy, first uses field data to improve the overall field adaptability of the model, and then intensively trains the model through the term table to ensure the accuracy and consistency of term translation, so as to obtain a field translation model that has both fluent translation capability of field text and term accuracy, effectively meeting the high-standard professional translation demand.

[0085] The data processing method provided by the embodiment of the present specification can significantly improve the term accuracy rate of medical translation, and by simulating the mechanism of "looking up dictionary for translation" of human experts, the translation behavior driven by terms is modeled as text The term bidirectional matching process replaces the traditional black box generation with explicit term alignment, constructs translation pairs containing source language text and translated text, and based on the training model based on the translation pairs, the trained domain translation model can effectively reduce the clinical or scientific research risk caused by translation errors. For OOV cases and long-tail terms, data augmentation is performed, data synthesis is performed on the uncovered source language text without matching terms and the long-tail term table without matching source language text, the training of the rare words in the medical field is strengthened, and the knowledge gap of the model is repaired.

[0086] By constructing bidirectional translation data, forming a semantic closed-loop bilingual training sample, strengthening the model's understanding of bidirectional mapping of medical terms, and improving the consistency and robustness of translation, the structured term table set is embedded into the data synthesis process as prior knowledge, effectively bridging the gap between general language models and professional field requirements, and providing high-quality and high-coverage field data for subsequent continuous pre-training and fine-tuning.

[0087] Referring to Figure 2 , Figure 2 A flowchart of a domain translation model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0088] Step 202: Determine an initial translation model and domain data of a target domain, wherein the domain data is obtained by the above-mentioned data processing method.

[0089] Step 204: Training the initial translation model using the domain data corresponding to the target domain to obtain a trained domain translation model.

[0090] Training the initial translation model using the domain data corresponding to the target domain to obtain a trained domain translation model includes: Training the initial translation model using the domain data corresponding to the target domain to obtain an updated translation model; Training the updated translation model using each term table in the set of term tables corresponding to the target domain to obtain the trained domain translation model.

[0091] For details, please refer to the above embodiments, which will not be repeated here.

[0092] The domain translation model training method provided by the embodiments of the present specification uses a two-stage training strategy to first improve the overall domain adaptability of the model using domain data, and then ensures the accuracy and consistency of term translation through term table intensive training, thereby obtaining a domain translation model that has both fluent translation capability of domain text and accuracy of terms, effectively meeting the high-standard professional translation requirements.

[0093] Referring to Figure 3 ,Figure 3 A processing process schematic diagram of a data processing method provided by an embodiment of the present specification is shown.

[0094] Taking the medical field as an example, the large model translation of the medical field scene usually helps the zero-shot multilingual knowledge of the general large language model, and combines the prompt engineering or medical translation fine-tuning to improve the medical text translation capability. The translation large model fine-tuned by the translation instruction is difficult to master the massive medical translation knowledge through the post-training supervision fine-tuning mode, and is easy to overfit and lose the instruction following ability and generalization ability.

[0095] Under this background, the core problem of building a medical translation large model is how to inject hundreds of billions of multilingual medical translation data in the pre-training stage in advance to significantly improve the medical multilingual ability of the model. In addition, the medical translation model is prone to catastrophic forgetting during the training stage, for example, learning a large number of high-frequency vocabulary translation data is prone to further forgetting the translation of long-tail vocabulary, and the "seesaw" phenomenon occurs.

[0096] The data processing method provided by the embodiment of the present specification converts the term dictionary behavior of human experts into an extensible automated process through the "knowledge-guided data synthesis" paradigm, while maintaining language fluency, fundamentally improving the accuracy, consistency and professional reliability of medical translation, and providing a practical technical path for building a safe and usable medical intelligent translation system.

[0097] Specifically, a term library containing a term vocabulary table (i.e., the term vocabulary table set in the above embodiment) is determined, and the format of the term vocabulary table is "source language term: target language term", for example, "hydrorachitis: inflammatory hydrocephalus". The term vocabulary table can be generated by medical standard translation word collection, manual annotation, etc. A training content library (i.e., a source language text set) is selected, and the training content library can contain text content such as titles, abstracts and body paragraphs of documents.

[0098] The process of "checking the dictionary for translation" by human experts is simulated through a bidirectional matching strategy of text-terminology. When performing the bidirectional matching process, the text (i.e., the target source language text) with the number of matching terms ≥ n (the number of matching terms is the number of text matching successes in the above embodiment, and n can be set to 1) in the training content library is found through a substring matching method. The substring matching method usually uses the AC automatic machine method to load the term library in advance to improve the efficiency of substring matching.

[0099] In practical applications, the matching results can be divided into three cases: Case one is bidirectional matching, that is, there are terms in the training content library in the text in the term library, that is, the target source language text and the target term vocabulary table in the above embodiment.

[0100] For example, in the medical field, the target terminology list is spliced together with a fixed separator (e.g., "baicalin: Huangqin; cancer: cancer;...") and added to the prompt word of data synthesis in the form of RAG. The teacher model (i.e., the data translation model) uses its strong compliance ability to translate the medical field terminology into the translation, improving the accuracy of medical field terminology. The teacher model has strong general multi-language ability and can maintain good term replacement effect and fluent translation in slow thinking mode.

[0101] For example, the prompt text constructed according to the target term glossary and the target source language text is: Please translate the following English medical literature segment into Chinese, ensuring accurate terminology, consistent format, and smooth semantics. Do not add additional output, do not alter the format, and directly provide the translation result. During translation, please refer to the target term glossary I provided: baicalin: Huangqin; cancer: cancer; enzyme: enzyme; glucuronosyltransferase: glucuronosyltransferase; intestine: intestine; metabolism: metabolism; oral liquid: oral liquid; pharmacodynamics: pharmacodynamics; pharmacopoeia: pharmacopoeia. The target source language text is: Radix scutellariae (rs) is the dried root of the medicinal plant. It has been extensively employed in traditional formulations such as huangqin tang, huangqin shegantang and xiao-chai-hu-tang (sho-saiko-to) or proprietary traditional Chinese medicine (ptcm) products including huangqin tablet, yinhuang tablet, shuanghuanglian capsule, shuanghuanglian oral liquid, yiqing capsule, etc. Baicalein (b), baicalin (baicalein-7-o-glucuronide, bg), wogonin (w), wogonoside (wogonin-7-o-glucuronide, wg), oroxylin a (oa) and oroxylin a-7-o-glucuronide (oag) are the main bioactive components in rs. The six avones exhibited a variety of pharmacological effects such as anti-in Inflammation, anti-cancer effects and an anti-viral effect. Our previous study demonstrated that three of these compounds, namely baicalein, wogonin and oroxylin A, underwent extensive metabolism during and after oral administration. Two types of phase II enzymes including uridine 5'-diphospho-glucuronosyltransferase (UGT) and sulfotransferases (SULT) are responsible for the conjugation. In rat intestine, these three flavones are only metabolized via glucuronidation, whereas in rat liver, sulfates are generated in limited amounts. Due to the complexity of constituents in herbal medicines, there exists a higher probability of interactions among the multiple components originating from the same herbal preparation or different herbs. In fact, there exist a number of reports about interactions in pharmacokinetics or pharmacodynamics among the multiple components of a single herb.

[0102] At this time, the thinking process of the data translation model could be: I now need to process this translation request from the user. First, the user provided an English medical literature fragment and requested a translation into Chinese, with special emphasis on the accuracy of terms, format consistency, and semantic smoothness. At the same time, the user also provided a glossary containing the corresponding Chinese translations of some professional terms, such as "baicalin" should be translated as "黄芩苷", "cancer" as "癌症", etc. The user also requested not to add extra output, not to change the format, and directly give the translation result. So I need to ensure strict compliance with these requirements.

[0103] First, I need to read through the entire English text to understand its content and structure. This text mainly introduces the source of Radix Scutellariae, its applications in traditional formulas and proprietary Chinese medicines, as well as its main active ingredients and their metabolic processes. Next is about the pharmacological effects of these ingredients, and the role of metabolic enzymes, and finally mentions the interactions between herbal ingredients.

[0104] Next, I need to check the glossary provided by the user to ensure that all specified terms are correctly replaced. For example, "baicalin" should be translated as "黄芩苷", "cancer" as "癌症", "enzyme" as "酶", etc. Also, pay attention to the format. For example, the formula name like "huangqin tang" should be translated as "黄芩汤", keeping the first letter capitalized and the format consistent. Also, note whether the punctuation and the format of numerical references are consistent with the original text.

[0105] During the translation process, attention needs to be paid to the accuracy of professional terms. For example, "glucuronosyltransferase" corresponds to "葡糖醛酸基转移酶", and "sulphotransferases" is "磺基转移酶". In addition, "oralliquid" should be translated as "口服液", and "pharmacodynamics" is "药效学". The glossary provided by the user has covered most of the key terms, but also pay attention to whether other possible terms need to be adjusted to ensure the overall consistency of terms.

[0106] Then, I need to handle the sentence structure to make the Chinese sentences smooth and natural. For example, the original sentence "radixscutellariae (rs) is the dried root of the medicinal plant." needs to be accurately translated as "黄芩(RS)是药用植物黄芩(Scutellaria baicalensis Georgi)的干燥根。" Here, pay attention to the use of Chinese punctuation, such as the correct positions of commas and periods, as well as the correct translation of proper nouns.

[0107] Next is the section on the main active ingredients, such as "baicalein (b), baicalin (baicalein-7-o-glucuronide, bg), wogonin (w), wogonoside (wogonin-7-o-glucuronide, wg), oroxylin a (oa) and oroxylin a-7-o-glucuronide (oag) are the main bioactive components in rs." It is necessary to ensure that the names and abbreviations of each ingredient are correctly corresponding, and translate according to the vocabulary provided by the user. For example, "baicalein" is translated as "baicalein", "baicalin" is "baicalin", "wogonin" is "wogonin", "wogonoside" is "wogonoside", "orxylin a" is "oroxylin A", "orxylin a-7-o-glucuronide" is "oroxylin A-7-O-glucuronide". At the same time, pay attention to whether the abbreviations in the brackets and the citation format are correct.

[0108] When dealing with the description of the metabolic process, such as "the six flavones exhibited a variety of pharmacological effects such as anti-inflammation, anti-cancer effects and an anti-virus effect." It is necessary to ensure the accurate translation of the pharmacological effects. For example, "anti-inflammation" is translated as "anti-inflammatory", "anti-cancer" is "anti-cancer", "anti-virus" is "anti-virus".

[0109] The next part involves the role of metabolic enzymes, such as "uridine 5’-diphospho-glucuronosyltransferase (ugt) and sulphotransferases (sult) are responsible for the conjugation." Here, the names of the enzymes should be correctly translated. For example, "uridine 5’-diphospho-glucuronosyltransferase" corresponds to "uridine diphosphate glucuronosyltransferase (UGT)", "sulphotransferases" is "sulfotransferase (SULT)". At the same time, pay attention to whether the case and abbreviations of the terms meet the requirements of the user.

[0110] Finally, regarding the section on the interactions of herbal components, it's crucial to ensure that terms such as "interactions inpharmacokinetics or pharmacodynamics" are correctly translated as "interactions in pharmacokinetics or pharmacodynamics." Additionally, pay attention to the overall coherence of the paragraph, ensuring the translated Chinese is fluent, natural, and free of grammatical errors.

[0111] Throughout the translation process, I needed to repeatedly check whether all terms were replaced according to the user-provided thesaurus, and whether the formatting was consistent with the original text, such as the numbering of cited references, the use of parentheses, and the capitalization of proper nouns. Furthermore, I had to ensure that no extra content was added, and that the translation was a direct translation strictly according to the user's requirements, without any formatting changes. Finally, I reviewed the entire translation to ensure that it was semantically fluent, contained accurate terminology, correctly formatted, and met all the user's requirements.

[0112] The final output translation (i.e., the third translated text in the above embodiment) is as follows: Scutellaria baicalensis (RS) is the dried root of the medicinal plant Scutellaria baicalensis Georgi. It is widely used in traditional formulas such as Scutellaria baicalensis decoction, Scutellaria baicalensis and Ge Gen decoction, and Xiao Chai Hu decoction (Xiao Chai Hu Tang) or in prepared traditional Chinese medicine (PTCM) preparations, including Scutellaria baicalensis tablets, Yin Huang tablets, Shuang Huang Lian capsules, Shuang Huang Lian oral liquid, and Yi Qing capsules. Baicalein (B), baicalin-7-O-glucuronide (BG), flavonoids (W), flavonoid glycosides (WG), crotonin A (OA), and crotonin A-7-O-glucuronide (OAG) are the main bioactive components of Scutellaria baicalensis. These six flavonoid compounds exhibit various pharmacological effects, such as anti-inflammatory, anticancer, and antiviral effects. Our previous studies have shown that three compounds—baicalin, flavonoids, and oroxylin A—undergo extensive metabolism during and after oral administration. Two phase II enzymes, including uridine diphosphate glucuronyltransferase (UGT) and sulfotransferase (SULT), are responsible for their binding reactions. In the rat intestine, these three flavonoids are metabolized only via glucuronidation, while in the rat liver, only a small amount of sulfate is produced. Due to the complexity of herbal components, there is a high probability of interactions between multiple components from the same herbal preparation or from different herbal sources. In fact, there are numerous reports on the pharmacokinetic or pharmacodynamic interactions of multiple components from a single herb.

[0113] Scenario 2 involves a source language text where no terminology is matched (OOV scenario). In this case, the source language text is not covered by the target language text. The translation is synthesized using the teacher model's multilingual capabilities in slow thinking mode, without relying on a terminology vocabulary.

[0114] In this case, the prompt text is constructed according to the uncovered source language text, and is directly input into the data translation model to obtain a second translation text.

[0115] For example, the prompt text is: Please translate the following English medical literature segment into Chinese, ensuring accurate terminology, consistent format, and smooth semantics. Do not add additional output, do not alter the format, and directly provide the translation result. Text: between boys and girls, the BMI classification did not differ between thesexes. Three different criteria were used to diagnose MetS in ad-olescents: Cook, Ferranti, and InternationalDiabetes Federation. The criteria of Cook and Ferranticonsider MetS in the presence of change in three risk factors or more (glycemia, triglycerides, HDL-c,WC,and SBP). For the IDFcriterion, thediagnosis of MetS is performed in the presenceof a change in WC and at leasttwo other components (glyce-mia, triglycerides, HDL-c, SBP, and / or DBP).Forthe characterization of the sample, as well as the descrip-tion ofpresence of MetS for all diagnostic criterion andthe presence of alterationin each component, the absoluteand relative frequency were used. Comparisonof theagreement between different diagnostic criteria of MetSwas performed bythe Kappa index, considering differ-ences for p<0.05.

[0116] In this scenario, the thought process of the data translation model can be described as follows: I now need to process a user's translation request. First, the user wants to translate a passage of English medical literature into Chinese, requiring accurate terminology, consistent formatting, and fluent meaning, while maintaining the original format and not adding any extra content. The user also specifically emphasizes that the format should not be changed and the translation result should be provided directly.

[0117] First, I need to read the original text to understand the overall content. This passage mainly discusses the diagnostic criteria and statistical analysis methods for adolescent metabolic syndrome (MetS). It involves some technical terms, such as BMI classification, Cook et al.'s criteria, Ferranti et al.'s criteria, the International Diabetes Federation (IDF) criteria, and statistical methods such as the Kappa index and the use of SPSS software.

[0118] Next, I need to ensure the accuracy of the terminology. For example, "BMI classification" should be translated as "BMI classification," while "MetS" is usually translated as "metabolic syndrome" in medical literature.

[0119] Next is the description of the diagnostic criteria. The Cook and Ferranti criteria mention changes in three risk factors: blood glucose, triglycerides, HDL-c, waist circumference (WC), and systolic blood pressure (SBP). The IDF criteria, however, require changes in waist circumference plus changes in at least two other indicators. It's important to ensure that terms such as "glycemia" are translated as "blood glucose," "triglycerides" as "triglycerides," "HDL-c" as "high-density lipoprotein cholesterol," "WC" as "waist circumference," "SBP" as "systolic blood pressure," and "DBP" as "diastolic blood pressure."

[0120] Statistical methods such as "Kappa index" should be translated as "Kappa index". Note that the p-value in "p<0.05" is usually represented by "p value" in Chinese, but users may want to keep the "p<0.05" format, so it is necessary to confirm whether to keep it directly.

[0121] Additionally, the user requested that the formatting not be changed, so the paragraph structure and numbering in the original text should remain unchanged. Sentence coherence is also important; for example, "the BMI classification did not differ between thesexes" should be translated as "BMI classification showed no difference between genders." Ensure the meaning is fluent and conforms to Chinese expression habits.

[0122] Finally, check the entire translation for all user requirements: accurate terminology, consistent formatting, fluent semantics, no additional content, maintain original format. Ensure no details are missed, such as table references, statistical method descriptions, etc. Also, pay attention to the conversion of Chinese and English punctuation marks, such as periods, commas, etc. Use Chinese punctuation.

[0123] Potential difficulties may include handling the consistency of professional terms and maintaining the format structure of the original text. For example, ensure that "Kappa index" is correctly translated while using the correct terminology in the description of statistical analysis. In addition, pay attention to the expression of numbers and units.

[0124] In summary, I need to translate sentence by sentence, ensuring that each term is accurate, the format is consistent with the original, the sentence is fluent, and no additional content is added. After completing the translation, read the translation again to check if it meets all user requirements.

[0125] The final output translation (i.e., the second translation text in the above example) is: Among boys and girls, there is no difference in BMI classification between genders. The diagnosis of metabolic syndrome (MetS) in adolescents uses three different criteria: Cook, Ferranti, and International Diabetes Federation (IDF). Cook and Ferranti criteria consider that when three or more risk factors (blood glucose, triglycerides, HDL-c, waist circumference (WC), and systolic blood pressure (SBP)) change, MetS can be diagnosed. The IDF standard requires a change in waist circumference accompanied by a change in at least two other indicators (blood glucose, triglycerides, HDL-c, SBP, and / or diastolic blood pressure (DBP)). To describe the characteristics of the sample and the presence of MetS under all diagnostic criteria and the distribution of changes in each component, relative frequency is used. The consistency between different MetS diagnostic criteria is compared using the Kappa index, with p<0.05 as the significant difference criterion.

[0126] Case three is the absence of corresponding source language text for the term glossary (i.e., long tail term scenario), which is a long tail term glossary. To address the lack of real context for some long tail term glossaries, the content creation ability of the teacher model (which is now considered a text generation model) is used to generate source language text containing the corresponding long tail term glossary. Based on obtaining the long tail source language text corresponding to the long tail term glossary, the translation synthesis is performed based on the mode of case one.

[0127] For example, hydrorachitis: inflammatory spinal canal hydrops is a long-tail term vocabulary, at this time the prompt text corresponding to the text generation model can be: You are a senior doctor with many years of medical research, often read a lot of medical literature, now provide you with an English medical term vocabulary, and its corresponding Chinese explanation, please write a paragraph of English literature abstract containing the medical term vocabulary according to the rigorous medical literature style, ensure the logicality and scientificity of the text, the length should not exceed 200 words, and it is ensured that the text of the term is consistent with the given term, without expansion, without annotation, directly give the text, without additional output. Term: hydrorachitis: inflammatory spinal canal hydrops.

[0128] The thinking process of the text generation model is: the user asks me to help generate an English medical literature abstract, the term is hydrorachitis, and the corresponding Chinese explanation is inflammatory spinal canal hydrops. First, I need to confirm the correct use of the term and the relevant background. The user emphasizes the rigorous medical literature style, so I need to make sure that the words are accurate and the structure is scientific. First, I need to verify the definition of hydrorachitis. According to the information provided by the user, it corresponds to inflammatory spinal canal hydrops, which may be related to inflammation and fluid accumulation in the spinal canal. It may be necessary to consult relevant literature to confirm the common usage of the term and the related pathological mechanism to ensure that there is no misunderstanding. Next, the user requires that the term must be used accurately in the abstract, without expansion or annotation. This means that the abstract cannot explain the term, but must use it directly to describe the research content. Need to ensure that the term appears in the appropriate position in the text and is logical. The user also mentioned that the length of the abstract should not exceed 200 words, so it needs to be concise and clear. The structure usually includes research background, methods, results and conclusions. It needs to revolve around hydrorachitis, which may involve its etiology, imaging features, treatment methods and prognosis. Considering that the user may be a medical researcher or student, it is necessary to provide a high-quality abstract for them to reference or use in a paper. The deep demand may be to quickly obtain an abstract that meets academic standards, save time, and ensure that the term is used correctly to avoid errors. Need to ensure logicality, such as first introducing the disease, then describing the diagnostic method, and then the treatment and results. May refer to MRI as a diagnostic tool, mention surgical or drug treatment, and discuss prognosis. At the same time, avoid any additional explanation or expansion, strictly according to the user's requirements. Finally, check if the term is used correctly, the whole text has no grammatical errors, and meets the standard structure of an academic abstract. Make sure all information is accurate and reliable, such as average follow-up time, number of cases, etc., to enhance credibility.

[0129] The synthesized source language text (i.e., the long-tail source language text) can be: Hydrorachitis is a rare pathological condition characterized by inflammatory accumulation of fluid within the spinal canal, frequently associated with chronic granulomatous inflammation or post-infectious sequelae. This retrospective study analyzed 23 cases of hydrorachitis confirmed by MRI and histopathological examination over a 12-year period. Radiological features demonstrated focal or diffuse spinal canal dilation with T2-weighted hyperintensity and variable contrast enhancement. Etiological factors included tuberculosis (n = 14), brucellosis (n = 5), and idiopathic causes (n = 4). All patients underwent surgical decompression combined with pathogen-specific medical therapy. Postoperative follow-up (mean 26.4 months) revealed significant neurological improvement in 82.6% of cases, with recurrence observed in 3 patients.Our findings suggest that early diagnosis through multimodal imaging and targeted anti-inflammatory treatment are critical for optimizing clinical outcomes inhydrorachitis. This study provides comprehensive insights into the pathophysiological mechanisms and management strategies for this underrecognized spinal disorder. .

[0130] Next, following the pattern of Case 1, the data translation model translates the long-tail source language text based on the long-tail terminology glossary and the long-tail source language text to obtain the first translated text.

[0131] After completing the above data synthesis, a massive amount of synthesized translation pairs are used to construct parallel corpora. Specifically, the original English text and its corresponding Chinese translation are concatenated to form an "English->Chinese" translation context, while the Chinese translation and its corresponding original English text are concatenated to form a "Chinese->English" translation context, forming bidirectional symmetrical translation data (i.e., the aforementioned bidirectional translation data). For example, the instruction template for bidirectional translation data is: "English to Chinese context format: {original English text} The Chinese translation of the above English content is: {synthesized Chinese translation}; Chinese to English context format: {synthesized Chinese translation} The English version of the given Chinese text is: {original English text}."

[0132] This method can concatenate the translation data synthesized by the teacher model into a large amount of text, forming the translation pre-training corpus for a dedicated translation model (i.e., the student model). In practical applications, the synthesized training data can be used for the first stage of pre-training (CPT) of the model. In the second stage of pre-training (context adjustment), training can be performed directly using a terminology vocabulary without an instruction template, to strengthen the model's memory of long-tail terms. The training template for this stage can be: Translate from Chinese to English. Chinese: {Chinese vocabulary} English: {English vocabulary}, translating English terms into Chinese. English: {English vocabulary} Chinese: {Chinese vocabulary}.

[0133] Through the above-mentioned bidirectional matching, data synthesis, and phased training, a domain translation model with accurate terms, real-world context, and comprehensive coverage is constructed, which can effectively support automated translation in scenarios such as literature and guidelines, and provide a reliable technical foundation for applications such as cross-language retrieval and parallel reading.

[0134] The data processing method provided in the embodiments of this specification pre-constructs a high-quality medical term library (derived from standard dictionaries and manual annotations), and enforces the alignment of source terms and target terms during the synthesis process to ensure that the translation of key medical concepts strictly complies with professional norms, significantly improving term accuracy. Inject structured medical knowledge explicitly into the training corpus. Efficiently match terms in the literature through an Aho-Corasick automaton and retain the original context, enabling the model to learn the joint representation of "terms + context" during the pre-training stage, and the synthetic data is generated based on the same term library to avoid translating the same term into multiple versions in different sentences (e.g., "myocardial infarction" is sometimes translated as "heart attack" and sometimes as "myocardial infarction"). By using a large number of English medical literature as the content source and combining automated term replacement, a large number of high-quality English-Chinese / Chinese-English translation pairs can be efficiently generated without relying on scarce manually annotated parallel corpora, solving the fundamental bottleneck of insufficient data in the medical field.

[0135] In practical applications, when data distillation is achieved through the above-mentioned data synthesis, the end-to-end translation accuracy of the distilled student model is significantly improved and exceeds that of the teacher model, and the term accuracy is significantly improved. Moreover, due to retaining the original literature sentence structure, the model can better handle complex clinical expressions such as negation, time sequence, and conditions, with high context semantic fidelity, supporting rapid domain transfer, and providing a high-quality pre-training foundation for a dedicated medical translation large model.

[0136] Corresponding to the above method embodiments, this specification also provides embodiments of a data processing device. Figure 4 The structure diagram of a data processing device provided in an embodiment of this specification is shown. As Figure 4 shown, the device includes: A data acquisition module 402, configured to acquire a set of term vocabularies and a set of source language texts in the target domain, where each term vocabulary in the set of term vocabularies includes a source language term and a corresponding target language term; A set matching module 404, configured to compare the set of term vocabularies with the set of source language texts to determine the long-tail term vocabulary in the set of term vocabularies and the uncovered source language texts in the set of source language texts; The text translation module 406 is configured to generate long-tail source language text corresponding to the long-tail terminology vocabulary based on the text generation model, input the long-tail terminology vocabulary and the long-tail source language text into the data translation model to obtain the first translated text corresponding to the long-tail source language text, and input the uncovered source language text into the data translation model to generate the second translated text corresponding to the uncovered source language text. The data determination module 408 is configured to determine the domain data corresponding to the target domain based on the long-tail source language text, the first translated text, the uncovered source language text, and the second translated text.

[0137] Optionally, the text translation module 406 is further configured to: Determine the matching target term vocabulary and the corresponding target source language text, and input the target term vocabulary and the target source language text into the data translation model to obtain the third translated text corresponding to the target source language text.

[0138] Optionally, the data determination module 408 is further configured to: Based on the long-tailed source language text and the first translated text, a first translation pair is determined; Based on the uncovered source language text and the second translated text, a second translation pair is determined; Based on the target source language text and the third translated text, a third translation pair is determined; Based on the first translation pair, the second translation pair, and the third translation pair, the domain data corresponding to the target domain is obtained.

[0139] The device further includes: The model training module is configured to determine an initial translation model, train the initial translation model using domain data corresponding to the target domain, and obtain a trained domain translation model.

[0140] Optionally, the model training module is further configured as follows: Based on the domain data corresponding to the target domain, bidirectional translation data is constructed, wherein the bidirectional translation data includes forward translation data between the source language text and the corresponding translation text, and reverse translation data between the translation text and the corresponding source language text. The initial translation model is trained using the bidirectional translation data to obtain the trained domain translation model.

[0141] Optionally, the model training module is further configured as follows: The initial translation model is trained using the domain data corresponding to the target domain, and an updated translation model is obtained. The updated translation model is trained using each terminology vocabulary in the terminology vocabulary set to obtain the trained domain translation model.

[0142] Optionally, the set matching module 404 is further configured to: Each term vocabulary in the term vocabulary set is compared with each source language text in the source language text set. If the source language text contains a preset number of source language terms, it is determined that the source language text and the term vocabulary corresponding to the source language terms are successfully matched. Once the comparison is complete, determine the number of successful term matches for each term vocabulary and the number of successful text matches for each source language text. The long-tail terminology list is determined from the terminology list set based on the number of successful term matches, and the uncovered source language text is determined from the source language text set based on the number of successful text matches.

[0143] Optionally, the data acquisition module 402 is further configured to: Obtain an initial vocabulary set for the target domain, and perform terminology unification and quality filtering on at least one initial term vocabulary in the initial vocabulary set to obtain the term vocabulary set.

[0144] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0145] Corresponding to the above method embodiments, this specification also provides embodiments of a domain translation model training device. Figure 5 A schematic diagram of a domain translation model training device according to one embodiment of this specification is shown. Figure 5 As shown, the device includes: The determination module 502 is configured to determine the initial translation model and the domain data of the target domain, wherein the domain data is obtained through the data processing method described above; Training module 504 is configured to train the initial translation model using domain data corresponding to the target domain, and obtain a trained domain translation model.

[0146] Optionally, training module 504 is further configured as follows: The initial translation model is trained using the domain data corresponding to the target domain, and an updated translation model is obtained. The updated translation model is trained using each term in the terminology set corresponding to the target domain, thereby obtaining the trained domain translation model.

[0147] The above is a schematic scheme of a domain translation model training device according to this embodiment. It should be noted that the technical solution of this domain translation model training device and the technical solution of the above-described domain translation model training method belong to the same concept. For details not described in detail in the technical solution of the domain translation model training device, please refer to the description of the technical solution of the above-described domain translation model training method.

[0148] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0149] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0150] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0151] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0152] The processor 620 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned data processing method and domain translation model training method.

[0153] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method and domain translation model training method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method and domain translation model training method embodiments.

[0154] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method and domain translation model training method.

[0155] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are relatively simple in description because they are fundamentally similar to the data processing method and domain translation model training method embodiments; relevant parts can be referred to the descriptions of the data processing method and domain translation model training method embodiments.

[0156] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method and domain translation model training method.

[0157] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the data processing method and the domain translation model training method described above. For details not described in detail in the technical solution of the computer program product, please refer to the descriptions of the technical solutions of the data processing method and the domain translation model training method described above.

[0158] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0159] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0160] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0161] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Obtain a terminology vocabulary set for the target domain and a source language text set, wherein each terminology vocabulary set includes source language terms and corresponding target language terms; The terminology vocabulary set is compared with the source language text set to determine the long-tail terminology vocabulary in the terminology vocabulary set and the uncovered source language text in the source language text set; Based on the text generation model, the long-tail source language text corresponding to the long-tail terminology vocabulary is generated. The long-tail terminology vocabulary and the long-tail source language text are input into the data translation model to obtain the first translated text corresponding to the long-tail source language text. The uncovered source language text is input into the data translation model to generate the second translated text corresponding to the uncovered source language text. Based on the long-tail source language text, the first translated text, the uncovered source language text, and the second translated text, the domain data corresponding to the target domain is determined.

2. The method of claim 1, after comparing the terminology vocabulary set with the source language text set, further includes: The target term vocabulary and the corresponding target source language text that have been successfully matched are determined, and the target term vocabulary and the target source language text are input into the data translation model to obtain the third translated text corresponding to the target source language text.

3. The method as described in claim 2, wherein determining the domain data corresponding to the target domain based on the long-tail source language text and the first translated text, the uncovered source language text and the second translated text, includes: Based on the long-tailed source language text and the first translated text, a first translation pair is determined; Based on the uncovered source language text and the second translated text, a second translation pair is determined; Based on the target source language text and the third translated text, a third translation pair is determined; Based on the first translation pair, the second translation pair, and the third translation pair, the domain data corresponding to the target domain is obtained.

4. The method as described in claim 3, after obtaining the domain data corresponding to the target domain, further includes: An initial translation model is determined, and the initial translation model is trained using the domain data corresponding to the target domain to obtain a trained domain translation model.

5. The method as described in claim 4, wherein the initial translation model is trained using domain data corresponding to the target domain to obtain a trained domain translation model, comprising: Based on the domain data corresponding to the target domain, bidirectional translation data is constructed, wherein the bidirectional translation data includes forward translation data between the source language text and the corresponding translation text, and reverse translation data between the translation text and the corresponding source language text. The initial translation model is trained using the bidirectional translation data to obtain the trained domain translation model.

6. The method as described in claim 4 or 5, wherein the initial translation model is trained using domain data corresponding to the target domain to obtain a trained domain translation model, comprising: The initial translation model is trained using the domain data corresponding to the target domain, and an updated translation model is obtained. The updated translation model is trained using each terminology vocabulary in the terminology vocabulary set to obtain the trained domain translation model.

7. The method according to any one of claims 1-5, wherein comparing the terminology vocabulary set with the source language text set to determine the long-tail terminology vocabulary in the terminology vocabulary set and the uncovered source language text in the source language text set, includes: Each terminology list in the terminology list set is compared with each source language text in the source language text set. If the source language text contains a source language term from the terminology list, it is determined that the source language text and the terminology list corresponding to the source language term are successfully matched. Once the comparison is complete, determine the number of successful term matches for each term vocabulary and the number of successful text matches for each source language text. The long-tail terminology list is determined from the terminology list set based on the number of successful term matches, and the uncovered source language text is determined from the source language text set based on the number of successful text matches.

8. A method for training a domain translation model, comprising: An initial translation model and target domain data are determined, wherein the domain data is obtained by the data processing method according to any one of claims 1-7; The initial translation model is trained using the domain data corresponding to the target domain to obtain the trained domain translation model.

9. The method of claim 8, wherein the initial translation model is trained using domain data corresponding to the target domain to obtain a trained domain translation model, comprising: The initial translation model is trained using the domain data corresponding to the target domain, and an updated translation model is obtained. The updated translation model is trained using each term in the terminology set corresponding to the target domain, thereby obtaining the trained domain translation model.

10. A data processing apparatus, comprising: The data acquisition module is configured to acquire a terminology vocabulary set for the target domain and a source language text set, wherein each terminology vocabulary set includes source language terms and corresponding target language terms; The set matching module is configured to compare the term vocabulary set with the source language text set to determine the long-tail term vocabulary in the term vocabulary set and the uncovered source language text in the source language text set. The text translation module is configured to generate long-tail source language text corresponding to the long-tail terminology vocabulary based on the text generation model, input the long-tail terminology vocabulary and the long-tail source language text into the data translation model to obtain the first translated text corresponding to the long-tail source language text, and input the uncovered source language text into the data translation model to generate the second translated text corresponding to the uncovered source language text. The data determination module is configured to determine the domain data corresponding to the target domain based on the long-tail source language text, the first translated text, the uncovered source language text, and the second translated text.

11. A domain translation model training device, comprising: The determination module is configured to determine an initial translation model and domain data of the target domain, wherein the domain data is obtained by the data processing method according to any one of claims 1-7; The training module is configured to train the initial translation model using domain data corresponding to the target domain, thereby obtaining a trained domain translation model.

12. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

14. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images