A translation method, a translation device and a computer-readable storage medium
By updating and cleaning the parallel corpus, and training the translation model with the original parallel corpus, the problem of machine translation model poor translation effect on new words is solved, and the low-cost and efficient new word translation ability is achieved.
Patent Information
- Application Number
- CN202111630203.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-28
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-12-28
AI Technical Summary
The machine translation model is difficult to translate accurately when facing new words on the Internet, and the construction of training parallel corpus containing new words is costly and inefficient, and cannot keep up with the update speed of new words.
By obtaining the trained translation model and the original parallel corpus, using the new word set to update the training parallel corpus, and cleaning it, and updating the translation model with the original parallel corpus to achieve learning and iterative optimization of new words.
It reduces the cost of building training parallel corpus containing new words, improves training efficiency and model robustness, and significantly improves the translation effect of new words.
Smart Images

Figure CN114492469B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of machine translation, and in particular to a translation method, a translation device and a computer-readable storage medium. Background Art
[0002] Nowadays, information spreads at an extremely fast speed, and some new Internet words are often formed. Since new Internet words have never appeared in the historical training corpus or their meanings have changed, and the translation effect of the machine translation model comes directly from the knowledge of the training corpus, it is impossible to correctly translate new Internet words that have not appeared. Therefore, the machine translation model is particularly weak in the face of an endless stream of new words; and it is difficult to construct new training parallel corpora for new Internet words. The number of corpora containing new words that can be obtained is limited. If the parallel corpora are annotated one by one, the cost is high and the annotation cycle is long, and it is difficult to keep up with the update speed of new Internet words. Therefore, how to update and iterate the machine translation model at low cost and high efficiency so that the machine translation model can accurately translate new words has become a major problem. Summary of the invention
[0003] The present application provides a translation method, a translation device and a computer-readable storage medium, which can improve the translation effect of new words.
[0004] In order to solve the above technical problems, the technical solution adopted in the present application is: to provide a translation method, the translation method comprising: obtaining data to be processed, inputting the data to be processed into a first translation model, and obtaining translation data; wherein the training method of the first translation model comprises: obtaining a trained second translation model, an original parallel corpus, and a plurality of new word sets, the new word set comprising a first new word and at least one second new word, the second new word being a translation of the first new word; processing the new word set to obtain a first training parallel corpus containing the new word set; cleaning the first training parallel corpus to obtain a second training parallel corpus; using the original parallel corpus and the second training parallel corpus to update and train the second translation model to obtain the first translation model.
[0005] To solve the above technical problems, another technical solution adopted in the present application is: to provide a translation device, which includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the translation method in the above technical solution.
[0006] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the translation method in the above technical solution.
[0007] Through the above scheme, the beneficial effects of the present application are: firstly obtaining the trained second translation model, the original parallel corpus and multiple new word sets, then using the new word sets to update the original parallel corpus, obtaining the first training parallel corpus containing the new word sets, reducing the cost of constructing the training parallel corpus containing the new words, and improving the efficiency of constructing the training parallel corpus; and cleaning the first training parallel corpus, screening out the second training parallel corpus with good quality, improving the quality of the training parallel corpus, and further improving the training effect of the translation model and the robustness of the translation model; then using the original parallel corpus and the second training parallel corpus, updating and training the second translation model to obtain the first translation model, realizing the incremental iterative optimization of the translation model, so that the translation model continuously learns new word knowledge, and greatly improves the translation effect of the existing translation model on new words. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:
[0009] Figure 1 It is a flowchart of an embodiment of a translation method provided by the present application;
[0010] Figure 2 It is a flowchart of an embodiment of a training method for a first translation model provided by the present application;
[0011] Figure 3 is a flowchart of another embodiment of the training method of the first translation model provided by the present application;
[0012] Figure 4 It is a flowchart of an embodiment of step 32 provided by the present application;
[0013] Figure 5 It is a schematic diagram of the alignment relationship;
[0014] Figure 6 is a schematic diagram of constructing the first training parallel corpus;
[0015] Figure 7 is a flowchart of another embodiment of step 32 provided by the present application;
[0016] Figure 8 It is a schematic diagram of replacing entity phrases;
[0017] Fig. 9 It is a schematic diagram of quality screening of the first training parallel corpus;
[0018] Fig.10 It is a structural schematic diagram of an embodiment of a translation device provided by the present application;
[0019] Fig.11 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION
[0020] The present application is further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present application, but are not intended to limit the scope of the present application. Similarly, the following examples are only some embodiments of the present application rather than all embodiments, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0021] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0022] It should be noted that the terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.
[0023] See also Figure 1 , Figure 1 : is a flowchart of an embodiment of a translation method provided by the present application, the method comprising:
[0024] Step 11: Get the data to be processed.
[0025] The data to be processed is data to be translated, which may be text or voice data; it is understandable that when the data to be processed is voice data, the voice data may be first converted into text data, and then the text data may be translated.
[0026] Step 12: Input the data to be processed into the first translation model to obtain translation data.
[0027] By inputting the data to be processed into the first translation model, translation data containing the target language can be obtained; specifically, the data to be processed may be monolingual data, that is, it only contains sentences, phrases or words in one language. The first translation model can be used to translate the data to be processed into translation data in the target language. Common languages are Chinese or English, and the first translation model can be used to achieve Chinese / English translation or English / Chinese translation; it can be understood that the language of the data to be processed and the language of the target language can be set according to actual conditions, including but not limited to the above-mentioned Chinese / English translation or English / Chinese translation, to achieve translation between various languages. The data to be processed may also include two or more languages, and the first translation model can be used to translate two or more languages into the same target language.
[0028] Furthermore, the data to be processed may include new words, and the first translation model in this embodiment can be used to accurately translate the new words, wherein new words refer to words that have emerged with the development of the times or old words that have been used in a new way, and may include new Internet buzzwords or proprietary vocabulary, such as "Versailles", "Light Disk Action" or "New Coronavirus", etc.; please refer to the following Figure 2 , Figure 2 The flowchart of an embodiment of the training method of the first translation model provided by the present application is as follows. The first translation model can be trained by the following training method, so that the first translation model learns new word knowledge, so as to use the first translation model to accurately translate the to-be-processed data containing the new word, thereby improving the translation effect of the new word. The training method of the first translation model includes:
[0029] Step 21: Obtain the trained second translation model, the original parallel corpus, and multiple new word sets.
[0030] The parallel corpus includes the original text and the translated text corresponding to the original text, wherein the language of the original text is consistent with the language of the data to be processed, and the language of the translated text is consistent with the language of the target language. The parallel corpus of the corresponding language can be selected to train the translation model according to the actual translation needs, so as to use the trained translation model to realize the translation of the data to be processed.
[0031] The second translation model is a translation model trained using the original parallel corpus, and the new word set includes a first new word and at least one second new word, and the second new word is the translation of the first new word. For example, the first new word may be "new coronavirus", and the corresponding second new word may include "COVID-19" and "new coronavirus" in other languages; it can be understood that the new word set can be collected from the Internet, and the first new word may be a noun or adjective, etc., and the part of speech and usage are not limited here. The training method in this embodiment can be used to realize the learning and training of new words with different parts of speech and different usages of the translation model.
[0032] Furthermore, the original parallel corpus is a parallel corpus that does not contain new words. The second translation model trained using the original parallel corpus cannot achieve accurate translation of the new words and may mistranslate the new words. In this case, the second translation model can be trained using the parallel corpus containing new words to improve the translation effect of the second translation model on the new words. The specific implementation scheme is described below.
[0033] Step 22: Process the new word set to obtain a first training parallel corpus containing the new word set.
[0034] The new word set can be used to update the original parallel corpus to obtain a first training parallel corpus containing the new word set. The update is performed on the basis of the original parallel corpus, which can save the cost and time of reconstructing the training parallel corpus containing the new words, improve the efficiency of constructing the new training parallel corpus, and at the same time ensure the number of new training parallel corpora, thereby ensuring the training effect of the second translation model.
[0035] Step 23: Clean the first training parallel corpus to obtain a second training parallel corpus.
[0036] The first training parallel corpus may contain semantic or grammatical problems, such as incoherent sentences or grammatical errors. In this case, the first training parallel corpus may be cleaned to filter out the erroneous corpus and select high-quality corpus as the second training parallel corpus to ensure the quality of the training parallel corpus containing new words.
[0037] Step 24: Use the original parallel corpus and the second training parallel corpus to update and train the second translation model to obtain the first translation model.
[0038] The original parallel corpus and the second training parallel corpus can be mixed, and the second translation model can be updated and trained using the mixed original parallel corpus and the second training parallel corpus to obtain the first translation model, so that the first translation model can learn the knowledge of new words, thereby improving the new word translation effect of the first translation model; while the second translation model is trained using the second training parallel corpus containing the new word set, the original parallel corpus is used together in the training of the second translation model, so that the translation model can further perform new word training on the basis of the previous model training, thereby realizing the growing iterative optimization of the translation model, and at the same time ensuring the number of training corpora, thereby improving the training effect of the first translation model and the robustness of the translation model.
[0039] In this embodiment, the original parallel corpus can be updated with the new word set to obtain the first training parallel corpus containing the new word set, thereby reducing the cost of constructing the training parallel corpus containing the new words and improving the efficiency of constructing the training parallel corpus; moreover, the first training parallel corpus can also be cleaned to screen out the second training parallel corpus with good quality, thereby improving the quality of the training parallel corpus and further improving the training effect of the translation model and the robustness of the translation model; in addition, the original parallel corpus and the second training parallel corpus are used to update and train the second translation model to obtain the first translation model, thereby realizing the incremental iterative optimization of the translation model, so that the translation model continuously learns new word knowledge, thereby ensuring the model training effect, greatly improving the translation effect of the first translation model on the new words, and not restricting the part of speech and usage of the new words, so that it is applicable to the update optimization of all new words and has a wide applicability.
[0040] See also Figure 3 , Figure 3 : is a flow chart of another embodiment of the training method of the first translation model provided by the present application, the method comprising:
[0041] Step 31: Obtain the trained second translation model, the original parallel corpus, and multiple new word sets.
[0042] The above step 31 is the same as step 21 in the above embodiment and is not limited here.
[0043] Step 32: Process the new word set to obtain a first training parallel corpus containing the new word set.
[0044] The step of processing the new word set to obtain a first training parallel corpus containing the new word set may include: 1) obtaining independent keyword groups of the original parallel corpus; 2) updating the original parallel corpus based on the independent keyword groups and the new word set to obtain the first training parallel corpus.
[0045] In a specific implementation, the original parallel corpus may include a first original monolingual corpus and a second original monolingual corpus corresponding to the first original monolingual corpus, where the second original monolingual corpus is a translation of the first original monolingual corpus. Taking Chinese / English translation as an example, the first original monolingual corpus may be “Xiaoming meets Xiaohong in Beijing” in Chinese, and the corresponding second original monolingual corpus is “Xiaoming meets Xiaohong in Beijing” in English.
[0046] See also Figure 4 , obtaining independent keyword groups of the original parallel corpus, and then updating the original parallel corpus based on the independent keyword groups and the new word set to obtain the first training parallel corpus is shown in the following steps 41 to 43:
[0047] Step 41: Obtain an independent keyword group of a first original monolingual corpus to obtain a first independent keyword group.
[0048] The first original monolingual corpus and the second original monolingual corpus can be aligned to obtain an alignment result, and then based on the alignment result, the first independent keyword group can be screened out from the first phrase; specifically, the mgiza++ alignment tool can be used to align the first original monolingual corpus and the second original monolingual corpus to obtain an alignment result. It can be understood that the alignment tool includes but is not limited to mgiza++, and the appropriate alignment tool can be selected according to the actual situation to perform the alignment operation, which is not limited here.
[0049] Furthermore, the alignment result may include a first phrase in the first original monolingual corpus, a second phrase in the second original monolingual corpus, and a correspondence between the first phrase and the second phrase. At least one first phrase in the first original monolingual corpus and at least one second phrase in the second original monolingual corpus may be first divided, and then a correspondence between the first phrase and the second phrase is established. For example, the first original monolingual corpus "Xiaoming meets Xiaohong in Beijing" corresponds to multiple first phrases including "Xiaoming", "Beijing", and "Xiaohong", and the second original monolingual corpus "Xiaoming meets Xiaohong inBeijing" corresponds to multiple second phrases including "Xiaoming", "Beijing", and "Xiaohong", where "Xiaoming" corresponds to "Xiaoming", "Beijing" corresponds to "Beijing", and "Xiaohong" corresponds to "Xiaohong".
[0050] The "independent" in the independent keyword group means that there is no intersection between the alignment relationship of the first phrase and the second phrase. In a specific implementation, it can be determined based on the alignment result whether there are other phrases between the second phrase corresponding to the first phrase that do not correspond to the first phrase, thereby screening out the first independent keyword group to avoid grammatical or semantic errors in the first monolingual corpus in the following step of obtaining the first monolingual corpus containing the first new word. The specific screening steps may include: 1) Determine whether there are other phrases between the words in the second phrase; 2) If there are no other phrases between the words in the second phrase, determine that the first phrase is the first independent keyword group; 3) If there are other phrases between the words in the second phrase, determine whether the other phrases have a corresponding relationship with the first phrase; 4) If the other phrases have a corresponding relationship with the first phrase, determine that the first phrase is the first independent keyword group.
[0051] like Figure 5 As shown in the figure, the step of screening out the first independent keyword group is explained by taking the first original monolingual corpus as "I am Xiaoyi, an intelligent assistant from iFLY@@TEK" and the second original monolingual corpus as "I am Xiaoyi, an intelligent assistant from iFLY@@TEK" as examples, wherein the first phrase and the second phrase are shown with black borders in the figure, and the corresponding relationship between the first phrase and the second phrase is marked with a connecting line.
[0052] For the first phrase "iFLY@@", its corresponding second phrase is "iFLY@@" and "TEK", and there is no other phrase between the two phrases, which means that the "iFLY@@" and "TEK" corresponding to "iFLY@@" and "TEK" are "independent", and at this time "iFLY@@" can be used as a first independent keyword group; for the first phrase "intelligent assistant", its corresponding second phrase is "intelligent assistant", and there is no other phrase in between, which means that the "intelligent assistant" corresponding to "intelligentassistant" is "independent", and at this time "intelligent assistant" can be used as a first independent keyword group; and if the first phrase is "intelligent assistant Xiaoyi", its corresponding second phrase is "Xiaoyi" and "intelligent assistant", it can be seen that there is a "," and "an" between the two, so "intelligent assistant Xiaoyi" cannot be used as an "independent" first independent keyword group.
[0053] Step 42: Based on the first independent keyword group and the new word set, the first original monolingual corpus is updated to obtain a first monolingual corpus containing the first new word.
[0054] The keyword prediction model can be used to update the first original monolingual corpus to obtain the first monolingual corpus containing the first new word. Specifically, after the first independent keyword group in the first original monolingual corpus is screened out by using the above step 41, a masking mark can be used to randomly mask a preset number of first independent keyword groups, that is, the randomly selected first independent keyword group is replaced with the masking mark, and then the masked first original monolingual corpus is input into the keyword prediction model, so that the keyword prediction model replaces the masking mark with the first new word in the new word set, wherein the number of randomly masked first independent keyword groups is generally 15% of the total number of phrases in the first original monolingual corpus. The first keyword prediction model can replace appropriate masking marks with the first new word according to the semantics and grammar of the first original monolingual corpus. The keyword prediction model can also replace other inappropriate masking marks with other phrases other than the first new word or retain the original phrases before the masking process, thereby ensuring that the obtained first monolingual corpus not only contains the first new word but also has good quality.
[0055] For example, the first new word is "new coronavirus", and the first independent keyword group that is masked in the first original monolingual corpus is "iFLYTEK". The masking mark in the sentence of the first original monolingual corpus is a noun phrase meaning company information, which is inconsistent with the semantics of the noun phrase expressing the virus of the first new word "new coronavirus". In this case, you can choose to replace the masking mark corresponding to "iFLYTEK" with a noun phrase meaning company information such as "Tencent" or keep "iFLYTEK".
[0056] It is understandable that before using the keyword prediction model to update the first original monolingual corpus, the keyword prediction model needs to be trained first so that the keyword prediction model learns to predict new words, and then the keyword prediction model is used to generate a first monolingual corpus containing the first new word based on the first original monolingual corpus; specifically, the monolingual corpus containing the first new word is obtained from the Internet, and then the monolingual corpus containing the first new word is divided into phrases and masked, wherein the masked phrases may include the first new word and a preset number of other phrases randomly selected, and the number of other phrases is generally 15% of the total number of phrases in the monolingual corpus of the first new word; it is understandable that when dividing the monolingual corpus containing the first new word into phrases, there may be a situation where one first new word is divided into two phrases. When the first new word is masked, a masking mark is used to mask the above two phrases. When similar divisions occur in other phrases, the same operation can also be performed, which will not be repeated here.
[0057] In a specific implementation, the keyword prediction model may be trained based on a multilingual translation pre-training model (such as an mbart model), and the parameters of the keyword prediction model may be fine-tuned to improve the prediction effect of the keyword training model.
[0058] In other embodiments, an autoregressive keyword prediction model may be used. There is no need to use the mbart model to fine-tune the parameters of the keyword prediction model during the training process. The monolingual corpus containing the first new word after masking processing can be directly input into the keyword prediction model to achieve training of the keyword prediction model.
[0059] Step 43: construct a first training parallel corpus based on the first monolingual corpus and the second original monolingual corpus.
[0060] The process steps for constructing the first training parallel prediction can be as follows Figure 6 As shown, after obtaining the first monolingual corpus containing the first new word, the corresponding phrase in the second original monolingual corpus can be updated according to the first monolingual corpus to obtain the second monolingual corpus, and then the first training parallel corpus is constructed according to the first monolingual corpus and the second monolingual corpus; specifically, the second independent keyword group corresponding to the first independent keyword group in the second original monolingual corpus can be obtained based on the alignment result; the second independent keyword group is replaced with the second new word to obtain the second monolingual corpus; then the first monolingual corpus and the second monolingual corpus are aligned to obtain the first training parallel corpus; wherein the language of the second new word is the same as the language of the second original monolingual corpus.
[0061] It can be understood that the first independent keyword group in the above steps refers to the first independent keyword group updated to the first new word, and then the corresponding second independent keyword group in the second original monolingual corpus is replaced by the second new word. For the first independent keyword group updated to other phrases other than the first new word (hereinafter referred to as "other updated phrases"), the corresponding second independent keyword group in the second original monolingual corpus can be replaced by the translation of the other updated phrases to keep corresponding to the first monolingual corpus.
[0062] In a specific implementation, before the step of replacing the second independent keyword group with the second new word to obtain the second monolingual corpus, the first monolingual corpus may be screened first; specifically, other phrases except the independent keyword group in the first original monolingual corpus may be extracted to obtain the first other phrases; other phrases except the first new word in the first monolingual corpus may be extracted to obtain the second other phrases; then, it is determined whether the first monolingual corpus contains the first new word, and whether the first other phrases are consistent with the second other phrases; if one of the following situations occurs, that is, the first monolingual corpus does not contain the first new word or the first other phrases are inconsistent with the second other phrases, the first monolingual corpus is deleted to prevent the other phrases except the independent keyword group in the first monolingual corpus from having inconsistent formats, and the invalid first monolingual corpus with inconsistent formats or not containing the first new word is deleted to control the quality of the first monolingual corpus used for subsequent training, thereby improving the training effect.
[0063] It can be understood that in the process of updating the independent keyword group of the first original monolingual corpus, there is a situation where the independent keyword group is updated to other phrases (i.e., other updated phrases) except the first new word. In this case, when judging the format consistency, other phrases except the first new word and other updated phrases are extracted from the first monolingual corpus as the second other phrases.
[0064] In another embodiment, Figure 7 As shown, the above step 32 may also include:
[0065] Step 71: Perform named entity recognition on the original parallel corpus to obtain entity phrases.
[0066] By performing named entity recognition (NER) on the original parallel corpus, entity phrases in the original parallel corpus can be identified, and the entity phrases may include names of people, places or companies.
[0067] Step 72: Replace the entity phrase with the first new word set to obtain a first training parallel corpus.
[0068] like Figure 8 As shown, after obtaining the entity phrases in the original parallel corpus, the entity phrases can be replaced by special tags (UNK1-3) to obtain the term parallel corpus, and then the special tags corresponding to the entity phrases are replaced with the first new word set, the special tags in the first original monolingual corpus are replaced with the first new words, and the special tags in the second original monolingual corpus are replaced with the second new words, thereby obtaining the first training parallel corpus.
[0069] In the implementation of obtaining the first training parallel corpus using the NER recognition method, the original parallel corpus is directly updated to the first training parallel corpus containing the first new word set, which has high updating efficiency and convenient operation for the first training parallel corpus.
[0070] Step 33: Score the first monolingual corpus and the second monolingual corpus to obtain a quality score.
[0071] like Fig. 9 As shown, after obtaining the first training parallel corpus containing the new word set, the language model (including the first language model and the second language model) can be used to perform semantic evaluation on the first monolingual corpus and the second monolingual corpus in the first training parallel corpus to obtain a quality score, and then based on the quality score, it is judged whether the first monolingual corpus and the second monolingual corpus are semantically coherent to screen out the first training parallel corpus with better quality.
[0072] Step 34: Based on the quality score, determine whether the preset screening conditions are currently met.
[0073] A first language model can be used to perform semantic scoring on a first monolingual corpus to obtain a first quality score; then a second language model can be used to perform semantic scoring on a second monolingual corpus to obtain a second quality score; then the first quality score is added to the second quality score to obtain a quality score; and it is determined whether the quality score exceeds a preset threshold; if the quality score exceeds the preset threshold, it is determined that the preset screening condition is met.
[0074] In other implementations, a first language model may be used to perform semantic scoring on a first monolingual corpus to obtain a first quality score; then a second language model may be used to perform semantic scoring on a second monolingual corpus to obtain a second quality score; and it is determined whether both the first quality score and the second quality score exceed a preset threshold; if both the first quality score and the second quality score exceed the preset threshold, it is determined that the preset screening condition is met, and the first training parallel corpus is determined as the second training parallel corpus.
[0075] It can be understood that the preset threshold can be set according to the actual situation and is not limited here. In other embodiments, it is also possible to determine whether the preset screening condition is currently met by scoring methods such as weighted summing of the first quality score and the second quality score or calculating the mean of the first quality score and the second quality score. Examples are not given here one by one.
[0076] Step 35: If the preset screening condition is not met, the first monolingual corpus and the second monolingual corpus are deleted, and the process returns to the step of scoring the first monolingual corpus and the second monolingual corpus to obtain a quality score.
[0077] When the quality score does not exceed the preset threshold, it means that the first training parallel corpus does not meet the preset screening condition, and the quality of the first training parallel corpus is poor. The first training parallel corpus is deleted, and the process returns to the step of scoring the first monolingual corpus and the second monolingual corpus to obtain the quality score.
[0078] Step 36: If the preset screening condition is met, the first training parallel corpus is determined as the second training parallel corpus.
[0079] When the quality score exceeds the preset threshold, it means that the quality of the first training parallel corpus is good. At this time, the first training parallel corpus can be determined as the second training parallel corpus, and the quality scores of other first training parallel corpora are continued until all the first training parallel corpora are screened to obtain the second training parallel corpora.
[0080] Step 37: Mix the second training parallel corpus with the original parallel corpus to obtain a third training parallel corpus.
[0081] The ratio of the number of corpora of the second training parallel corpus to the number of corpora of the original parallel corpus in the third training parallel corpus is a preset ratio value, and the preset ratio value can be set according to actual conditions, and it can be 1:1 or 1:2, etc., and is not limited here. By mixing the second training parallel corpus with the original parallel corpus, the number of corpora for training the second translation model can be guaranteed, thereby improving the training effect and the robustness of the model training.
[0082] Step 38: Use the third training parallel corpus to update the second translation model to obtain a first translation model.
[0083] The third training parallel corpus is input into the second translation model, so that the second translation model learns relevant knowledge of the new word, and obtains the first translation model, so as to use the first translation model to translate the data to be processed containing the new word, and obtain accurate translation results, thereby greatly improving the translation effect and accuracy of the new word, and realizing incremental iterative update of the first translation model.
[0084] In this embodiment, the keyword prediction model can be trained using a parallel corpus containing a new word set, so that the keyword prediction model learns new word knowledge and extracts independent keyword groups from the original parallel corpus; then, the original parallel corpus is updated using the keyword prediction model, and the independent keyword groups are replaced with the new word set, thereby obtaining a first training parallel corpus containing the new word set; the first training parallel corpus is then quality screened using a language model (including a first language model and a second language model) to obtain a second training parallel corpus with better quality, thereby improving the quality of constructing the training parallel corpus, thereby improving the training effect, and further improving the translation effect of the translation model on new network words; in addition, the training method in this embodiment does not require a lot of manpower and material resources, and the keyword prediction model, language model, and translation model used can all be iteratively optimized, and incremental iterative updates are performed on the basis of the previous training, so that new word knowledge can be continuously learned, greatly improving training efficiency and translation effect.
[0085] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of an embodiment of a translation device provided in the present application. The translation device 100 includes a memory 101 and a processor 102 connected to each other. The memory 101 is used to store a computer program. When the computer program is executed by the processor 102, it is used to implement the translation method in the above embodiment.
[0086] See also Fig.11 , Fig.11 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 110 is used to store a computer program 111. When the computer program 111 is executed by a processor, it is used to implement the translation method in the above embodiment.
[0087] The computer-readable storage medium 110 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media that can store program codes.
[0088] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0089] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0090] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0091] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A translation method, characterized in that: include: Acquire data to be processed, and input the data to be processed into a first translation model to obtain translation data; The training method of the first translation model includes: Acquire a trained second translation model, an original parallel corpus, and a plurality of new word sets, wherein the new word set includes a first new word and at least one second new word, wherein the second new word is a translation of the first new word; Processing the new word set to obtain a first training parallel corpus containing the new word set; Cleaning the first training parallel corpus to obtain a second training parallel corpus; Using the original parallel corpus and the second training parallel corpus, updating and training the second translation model to obtain the first translation model; Among them, the second translation model is a translation model obtained by training with the original parallel corpus; the original parallel corpus is a parallel corpus that does not contain new words; the language of the original text is consistent with the language of the data to be processed, and the language of the translated text is consistent with the language of the target language.
2. The translation method according to claim 1, characterized in that: The step of processing the new word set to obtain a first training parallel corpus containing the new word set comprises: Obtaining independent keyword groups of the original parallel corpus; Based on the independent keyword group and the new word set, the original parallel corpus is updated to obtain the first training parallel corpus.
3. The translation method according to claim 2, characterized in that: The original parallel corpus includes a first original monolingual corpus and a second original monolingual corpus corresponding to the first original monolingual corpus; The second original monolingual corpus is a translation of the first original monolingual corpus, and the method further includes: Acquire an independent keyword group of the first original monolingual corpus to obtain a first independent keyword group; Based on the first independent keyword group and the new word set, updating the first original monolingual corpus to obtain a first monolingual corpus containing the first new word; The first training parallel corpus is constructed based on the first monolingual corpus and the second original monolingual corpus.
4. The translation method according to claim 3, characterized in that: The step of obtaining the independent keyword group of the first original monolingual corpus to obtain the first independent keyword group includes: Performing alignment processing on the first original monolingual corpus and the second original monolingual corpus to obtain an alignment result; the alignment result includes a first phrase in the first original monolingual corpus, a second phrase in the second original monolingual corpus, and a correspondence between the first phrase and the second phrase; Based on the alignment result, the first independent keyword group is selected from the first word group.
5. The translation method according to claim 4, characterized in that: The step of selecting the first independent keyword group from the first phrase based on the alignment result includes: Determining whether there are other phrases between the words in the second phrase; If the other phrases do not exist between the words in the second phrase, determining that the first phrase is the first independent keyword group; If the other phrase exists between the words in the second phrase, determining whether the other phrase has a corresponding relationship with the first phrase; If there is a corresponding relationship between the other word groups and the first word group, the first word group is determined to be the first independent keyword group.
6. The translation method according to claim 4, characterized in that: The step of constructing the first training parallel corpus based on the first monolingual corpus and the second original monolingual corpus includes: Based on the alignment result, obtaining a second independent keyword group corresponding to the first independent keyword group in the second original monolingual corpus; Replacing the second independent keyword group with the second new word to obtain a second monolingual corpus, wherein the language of the second new word is the same as the language of the second original monolingual corpus; The first monolingual corpus is aligned with the second monolingual corpus to obtain the first training parallel corpus.
7. The translation method according to claim 6, characterized in that: The step of cleaning the first training parallel corpus to obtain the second training parallel corpus includes: Scoring the first monolingual corpus and the second monolingual corpus to obtain a quality score; Based on the quality score, determine whether the preset screening condition is currently met; If yes, determining the first training parallel corpus as the second training parallel corpus; If not, the first monolingual corpus and the second monolingual corpus are deleted, and the process returns to the step of scoring the first monolingual corpus and the second monolingual corpus to obtain a quality score.
8. The translation method according to claim 7, characterized in that: The step of scoring the first monolingual corpus and the second monolingual corpus to obtain a quality score includes: Using a first language model to perform semantic scoring on the first monolingual corpus to obtain a first quality score; Using the second language model to perform semantic scoring on the second monolingual corpus to obtain a second quality score; Adding the first quality score to the second quality score to obtain the quality score; Determining whether the quality score exceeds a preset threshold; If so, it is determined that the preset screening condition is met.
9. The translation method according to claim 7, characterized in that: The step of scoring the first monolingual corpus and the second monolingual corpus to obtain a quality score further includes: Using a first language model to perform semantic scoring on the first monolingual corpus to obtain a first quality score; Using the second language model to perform semantic scoring on the second monolingual corpus to obtain a second quality score; Determining whether both the first quality score and the second quality score exceed a preset threshold; If so, it is determined that the preset screening condition is met, and the first training parallel corpus is determined as the second training parallel corpus.
10. The translation method according to claim 3, characterized in that: The method further comprises: Extracting other phrases except the independent keyword phrase from the first original monolingual corpus to obtain a first other phrase; Extracting other phrases except the first new word from the first monolingual corpus to obtain second other phrases; determining whether the first monolingual corpus contains the first new word, and whether the first other word group is consistent with the second other word group; If not, the first monolingual corpus is deleted.
11. The translation method according to claim 1, characterized in that: The step of processing the new word set to obtain a first training parallel corpus containing the new word set further includes: Performing named entity recognition on the original parallel corpus to obtain entity phrases; The entity phrase is replaced with the new word set to obtain the first training parallel corpus.
12. The translation method according to claim 1, characterized in that: The step of using the original parallel corpus and the second training parallel corpus to update and train the second translation model to obtain the first translation model includes: The second training parallel corpus is mixed with the original parallel corpus to obtain a third training parallel corpus; the ratio of the number of the second training parallel corpus in the third training parallel corpus to the number of the original parallel corpus is a preset ratio value; The second translation model is updated and trained using the third training parallel corpus to obtain the first translation model.
13. A translation device, characterized in that: The invention comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the translation method according to any one of claims 1 to 12.
14. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the translation method according to any one of claims 1 to 12.
