Translation corpus multiplexing method and device and electronic equipment

Abstract

The invention provides a translation corpus multiplexing method and device and electronic equipment, and the method comprises the steps: obtaining corpus data, and storing the corpus data according to an index established in a search server; the method comprises the following steps: acquiring an input translation request, disassembling an input original text included in the translation request into a text sequence, and retrieving the text sequence, an original language included in the translation request and a target language in a search server to obtain a retrieval candidate set; the similarity between the text sequence and each candidate original text in the retrieval candidate set is determined, the candidate original texts are sorted in an inverted order, the candidate original texts sorted in the inverted order are screened according to a similarity threshold value included in the translation request, and a corpus result set is constructed to serve as a response result of the translation request. According to the method and the device, the defects that in the prior art, when a new document is translated, the historical document in the corpus is selected to be imported, the historical translation in the corpus can be directly used for translation, and the reliability and the effectiveness of translation of the new document are low are overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of text translation, and in particular to a method, device and electronic equipment for reusing translation target text corpus. Background Art

[0002] In the translation industry, there are a large number of duplicate or similar original texts, especially for translations in vertical industries. For human translators, if the historical translations are provided for reference or direct use, but in the actual translation process, there are more similar or similar original texts, in which case most of the historical translations are useless. Often, this part of the data is also very useful, such as some differences in numbers, punctuation, or tenses, which can be used after appropriate review and modification by human translators without having to translate them all.

[0003] In the existing technology, a translation system similar to Trados is used to save historically translated documents through the memory library (corpus) in the system, but here human translators are required to manually upload the original documents and the translated documents. When translating a new document, if the memory library (corpus) is selected for import, the historical translations in the memory library will be directly used for translation, and the corpus will not be fully utilized, resulting in reduced translation reliability and effectiveness of the new document. Summary of the invention

[0004] The present invention provides a translation text corpus reuse method, device and electronic device, which are used to overcome the defect in the prior art that when translating a new document, historical documents in a corpus are selected for import, and the historical translations in the corpus are directly used for translation, resulting in low reliability and effectiveness of the translation of the new document.

[0005] The present invention provides a translation target corpus reuse method, comprising the following steps: Acquire corpus data, and store the corpus data in the search server according to the index established in the search server; Obtaining an input translation request, decomposing an input original text included in the translation request into a text sequence, and searching the text sequence, the original language included in the translation request, and the target language in the search server to obtain a search candidate set; Determining the similarity between the text sequence and each candidate original text in the retrieval candidate set; sorting the candidate original texts in reverse order according to the similarity, and screening the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request; A corpus result set is constructed according to the screened candidate original texts, and the corpus result set is used as a response result of the translation request.

[0006] In some embodiments, the searching the text sequence, the original language included in the translation request, and the target language in the search server to obtain a search candidate set includes: According to the original language and the target language included in the translation request, query and obtain corresponding candidate records from the corpus data stored in the search server; Preset the recall quantity of the candidate records; The recalled number of candidate records is used as a retrieval candidate set.

[0007] In some embodiments, determining the similarity between the text sequence and each candidate original text in the search candidate set includes: For each candidate original text in the retrieval candidate set, determining the Jaccard distance and the edit distance between the text sequence and the candidate original text; The average value of the Jaccard distance and the edit distance is taken as the similarity between the text sequence and the candidate original text.

[0008] In some embodiments, screening the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request includes: For the candidate original texts sorted in reverse order, if the similarity of the candidate original texts is less than the similarity threshold included in the translation request, the candidate original texts are filtered out.

[0009] In some embodiments, constructing a corpus result set based on the screened candidate original texts includes: Determine candidate translations corresponding to the screened candidate original texts; The candidate original text, the candidate translated text, the original language, the target language and the similarity corresponding to the candidate original text are taken as a corpus result set.

[0010] In some embodiments, the method further comprises: Determining the matching degree between the input original text and the search candidate set, and constructing a corresponding matching identifier according to the matching degree; The matching identifier is added to the corpus result set as a response result of the translation request.

[0011] The present invention also provides a translation target corpus reuse device, which includes the following modules: An acquisition module, used for acquiring corpus data and storing the corpus data in the search server according to the index established in the search server; A retrieval module, configured to obtain an input translation request, decompose the input original text included in the translation request into a text sequence, and retrieve the text sequence, the original language included in the translation request, and the target language in the search server to obtain a retrieval candidate set; A determination module, used to determine the similarity between the text sequence and each candidate original text in the retrieval candidate set; A screening module, configured to sort the candidate original texts in reverse order according to the similarity, and screen the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request; The construction module is used to construct a corpus result set according to the screened candidate original texts, and use the corpus result set as a response result of the translation request.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for reusing translation target text materials as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for reusing translation target text materials as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned translation target corpus reuse methods.

[0015] The translation target text corpus reuse method, device and electronic device provided by the present invention store the corpus data in a search server for storage. When the input original text needs to be translated, the corresponding candidate original text is retrieved from the search server according to the input original text, and the candidate original text and the corresponding candidate translation that meet the similarity threshold are used as the result of the translation request. Thus, by using the corpus data as a basis for retrieval and translation, the reliability and effectiveness of the translation can be improved compared with the prior art of directly translating using historical document data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the translation target corpus reuse method provided by the present invention.

[0018] Figure 2 It is a flow chart of the translation target text reuse device provided by the present invention.

[0019] Figure 3It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Combine the following Figure 1 Describe the translation target corpus reuse method of the present invention, Figure 1 FIG. 1 is a flow chart of the method for reusing translation target text provided by the present invention. Figure 1 As shown, the method includes the following steps 101 to 105.

[0022] Step 101: Acquire corpus data, and store the corpus data in the search server according to the index established in the search server.

[0023] First, a large amount of corpus data is collected from the SQL database through SQL statements, also known as historical corpus data, specifically the project data completed in the business system in the last 30 days, including project number, original language, target language, sentence identifier (ID), original text, translation, translator, translation time, etc.

[0024] Next, you can create an index in the search server (Elasticsearch, ES) in advance. For example, the index can be a sentence identifier (id) as a unique key. After obtaining the corpus data, store the corpus data in the search server according to the index established in the search server. Here, each sentence is used as the project data of the corpus data and stored in the retrieval database ES according to the sentence identifier. When storing, the original text and the translation will be decomposed into words, decomposed into individual characters or words, as two additional search fields, respectively recorded as src_text_list and transn_text_list, to facilitate subsequent retrieval.

[0025] Step 102: Obtain an input translation request, decompose the input original text included in the translation request into a text sequence, and search the text sequence, the original language included in the translation request, and the target language in the search server to obtain a search candidate set.

[0026] When a translation request is inputted by a translation terminal or other translation server, the inputted translation request is obtained, wherein the translation request includes data items such as inputted original text, original language, target language, similarity threshold (rate), etc. The inputted original text included in the translation request is decomposed into a text sequence, recorded as . And retrieve the text sequence, the original language included in the translation request, and the target language in the search server ES to obtain a search candidate set. Among them, the original language is the language of the input original text, and the target language is the language to be translated.

[0027] The retrieval process in the search server can be: according to the original language and target language included in the translation request, query the corresponding candidate records from the corpus data stored in the search server ES. This step is to limit the original language and target language, and keep it consistent with the original language and target language of the corpus data stored in the search server ES. The query is performed on the two search fields src_text_list and transn_text_list in the search server ES. The query method can use the search function More Like This (MLT) that comes with ES.

[0028] In addition, during retrieval, it is necessary to limit the recall number of candidate records to be retrieved, considering that the corpus data of the search server ES is too redundant and difficult to identify, and the search efficiency of the search server ES is low. Therefore, the embodiment of the present invention presets the recall number of candidate records, and under general conditions, the recall number is 10. Finally, the candidate records corresponding to the recall number are used as the retrieval candidate set, that is, 10 candidate records are used as the retrieval candidate set.

[0029] In the embodiment of the present invention, when responding to a translation request, a search server provides a plurality of reusable historical corpus data for an input original text in real time as a retrieval candidate set, thereby ensuring the timeliness of the translation process.

[0030] Step 103: Determine the similarity between the text sequence and each candidate original text in the retrieval candidate set.

[0031] After obtaining the search candidate set from the search server ES, the next step is to filter the search candidate set by determining the similarity between the text sequence and each candidate original text in the search candidate set. Specifically, for each candidate original text in the search candidate set, determine the text sequence The Jaccard distance (jaccard_distance) and edit distance (levenshtein_distance) of the candidate original text, and then the average of the Jaccard distance and the edit distance is used as the text sequence The similarity distance with the candidate original text is expressed as the following formula (1): (1)

[0032] In the above formula (1), Represents the text sequence corresponding to the input original text , It represents the text sequence corresponding to the candidate original text in the retrieval candidate set, levenshtein_distance represents the function for calculating the edit distance, and jaccard_distance represents the function for calculating the Jaccard distance.

[0033] Through the above formula (1), we can calculate the text sequence corresponding to the input text for each candidate text in the retrieval candidate set: The similarity distance.

[0034] Step 104: sort the candidate original texts in reverse order according to the similarity, and screen the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request.

[0035] Through step 103, after calculating the similarity distance between each candidate original text in the retrieval candidate set and the text sequence corresponding to the input original text, the candidate original texts are sorted in reverse order according to the similarity, that is, the texts with high similarity are sorted at the head of the sequence, and the texts with low similarity are sorted at the end of the sequence.

[0036] Then, the candidate original texts in reverse order are screened according to the similarity threshold (rate) included in the translation request. This similarity threshold refers to the lower limit of the similarity between the input original text and the corpus data, and the rate is generally set to 0.6. Therefore, here, for the candidate original texts in reverse order, if the similarity distance of the candidate original text is less than the similarity threshold (0.6) included in the translation request, the candidate original text is screened out. If the similarity distance of the candidate original text is greater than or equal to the similarity threshold (0.6), it is retained for the subsequent construction of the corpus result set to return the corresponding result.

[0037] Step 105: construct a corpus result set according to the screened candidate original texts, and use the corpus result set as a response result of the translation request.

[0038] After screening the search candidate set, the corpus result set is constructed based on the screened candidate original texts. The candidate translations corresponding to the screened candidate original texts can be determined from the search server ES, and then the candidate original texts, candidate translations, original languages, target languages, and similarities corresponding to the candidate original texts are taken as the corpus result set. Here, the original language and target language are the original language and target language included in the translation request, the similarity corresponding to the candidate original text is also the candidate original text, and the text sequence corresponding to the input original text is The similarity distance.

[0039] In some embodiments, after the search candidate set is retrieved, the matching degree between the input original text and the search candidate set can also be determined, and a corresponding matching identifier is constructed according to the matching degree. Here, whether the input original text is consistent with the search candidate set is determined according to the matching degree, so as to determine whether the input original text and the search candidate set are fully matched, thereby constructing a corresponding matching identifier, and then adding the matching identifier to the corpus result set as a response result of the translation request. Thus, the validity of reusing the translation results can be guaranteed through the matching identifier.

[0040] After the corpus result set is constructed, it can be digitally desensitized according to the actual translation needs. For example, all text character information in the corpus result set is replaced with symbols (such as "*"), or the text characters are encoded. The result after digital desensitization is then encapsulated into a JSON list as a response result of the translation request and returned to the translation terminal or other translation server.

[0041] In the embodiment of the present invention, corpus data is stored in a search server for storage. When the input original text needs to be translated, multiple corresponding candidate original texts are retrieved from the search server according to the input original text, thereby ensuring the timeliness of the server translation process. In addition, the candidate original texts that meet the similarity threshold and the corresponding candidate translations are used as the results of the translation request, and the matching degree of the candidate original texts is determined during the screening process. Thus, by using corpus data as a matching basis for retrieval and translation, the reliability and effectiveness of the translation are improved compared to the prior art of directly translating using historical document data.

[0042] The following is a description of the translation corpus reuse device provided by the present invention. The translation corpus reuse device described below and the translation corpus reuse method described above can be referenced to each other.

[0043] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of the translation target text reuse device provided by the present invention. Figure 2As shown, the translation target text corpus reuse device includes an acquisition module 201, a retrieval module 202, a determination module 203, a screening module 204, and a construction module 205. Specifically, the acquisition module 201 is used to acquire corpus data and store the corpus data in the search server according to the index established in the search server; the retrieval module 202 is used to acquire the input translation request, decompose the input original text included in the translation request into a text sequence, and retrieve the text sequence, the original language included in the translation request, and the target language in the search server to obtain a search candidate set; the determination module 203 is used to determine the similarity between the text sequence and each candidate original text in the search candidate set; the screening module 204 is used to sort the candidate original texts in reverse order according to the similarity, and screen the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request; the construction module 205 is used to construct a corpus result set according to the screened candidate original texts, and use the corpus result set as the response result of the translation request.

[0044] It should be noted that the beneficial effects of the translation target corpus reuse device here and the translation target corpus reuse method mentioned above can correspond to each other, so the beneficial effects of the translation target corpus reuse device are not repeated here.

[0045] Figure 3 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310 , a communications interface 320 , a memory 330 and a communication bus 340 , wherein the processor 310 , the communications interface 320 , and the memory 330 communicate with each other via the communication bus 340 . The processor 310 can call the logic instructions in the memory 330 to execute the translation target text corpus reuse method, which includes: obtaining corpus data, and storing the corpus data in the search server according to the index established in the search server; obtaining an input translation request, decomposing the input original text included in the translation request into a text sequence, and searching the text sequence, the original language included in the translation request, and the target language in the search server to obtain a search candidate set; determining the similarity between the text sequence and each candidate original text in the search candidate set; sorting the candidate original texts in reverse order according to the similarity, and filtering the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request; constructing a corpus result set according to the filtered candidate original texts, and using the corpus result set as a response result of the translation request.

[0046] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0047] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the translation text corpus reuse method provided by the above methods, which method includes: obtaining corpus data, and storing the corpus data in the search server according to the index established in the search server; obtaining an input translation request, decomposing the input original text included in the translation request into a text sequence, and searching the text sequence, the original language included in the translation request, and the target language in the search server to obtain a retrieval candidate set; determining the similarity between the text sequence and each candidate original text in the retrieval candidate set; sorting the candidate original texts in reverse order according to the similarity, and screening the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request; constructing a corpus result set based on the screened candidate original texts, and using the corpus result set as a response result of the translation request.

[0048] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the translation text corpus reuse method provided by the above-mentioned methods, the method comprising: obtaining corpus data, and storing the corpus data in the search server according to the index established in the search server; obtaining an input translation request, decomposing the input original text included in the translation request into a text sequence, and retrieving the text sequence, the original language included in the translation request, and the target language in the search server to obtain a retrieval candidate set; determining the similarity between the text sequence and each candidate original text in the retrieval candidate set; sorting the candidate original texts in reverse order according to the similarity, and screening the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request; constructing a corpus result set based on the screened candidate original texts, and using the corpus result set as a response result of the translation request.

[0049] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0050] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A translation target corpus reuse method, It is characterized in that The method comprises: Acquire corpus data, and store the corpus data in the search server according to the index established in the search server; Obtaining an input translation request, decomposing an input original text included in the translation request into a text sequence, and searching the text sequence, the original language included in the translation request, and the target language in the search server to obtain a search candidate set; Determining the similarity between the text sequence and each candidate original text in the retrieval candidate set; sorting the candidate original texts in reverse order according to the similarity, and screening the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request; A corpus result set is constructed according to the screened candidate original texts, and the corpus result set is used as a response result of the translation request.

2. The method for reusing translated text according to claim 1, It is characterized in that The searching the text sequence, the original language included in the translation request, and the target language in the search server to obtain a search candidate set includes: According to the original language and the target language included in the translation request, query and obtain corresponding candidate records from the corpus data stored in the search server; Presetting the recall quantity of the candidate records; The recalled number of candidate records is used as a retrieval candidate set.

3. The method for reusing translated text according to claim 1, It is characterized in that Determining the similarity between the text sequence and each candidate original text in the search candidate set includes: For each candidate original text in the retrieval candidate set, determining the Jaccard distance and the edit distance between the text sequence and the candidate original text; The average value of the Jaccard distance and the edit distance is taken as the similarity between the text sequence and the candidate original text.

4. The method for reusing translated text according to claim 1, It is characterized in that Said Screening the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request includes: For the candidate original texts sorted in reverse order, if the similarity of the candidate original texts is less than the similarity threshold included in the translation request, the candidate original texts are filtered out.

5. The method for reusing translation target text according to claim 1, It is characterized in that The step of constructing a corpus result set based on the screened candidate original texts includes: Determine candidate translations corresponding to the screened candidate original texts; The candidate original text, the candidate translated text, the original language, the target language and the similarity corresponding to the candidate original text are taken as a corpus result set.

6. The method for reusing translated text materials according to claim 5, It is characterized in that The method further comprises: Determining the matching degree between the input original text and the search candidate set, and constructing a corresponding matching identifier according to the matching degree; The matching identifier is added to the corpus result set as a response result of the translation request.

7. A translation target corpus reuse device, It is characterized in that The device comprises: An acquisition module, used for acquiring corpus data and storing the corpus data in the search server according to the index established in the search server; A retrieval module, configured to obtain an input translation request, decompose the input original text included in the translation request into a text sequence, and retrieve the text sequence, the original language included in the translation request, and the target language in the search server to obtain a retrieval candidate set; A determination module, used to determine the similarity between the text sequence and each candidate original text in the retrieval candidate set; A screening module, configured to sort the candidate original texts in reverse order according to the similarity, and screen the candidate original texts sorted in reverse order according to the similarity threshold included in the translation request; The construction module is used to construct a corpus result set according to the screened candidate original texts, and use the corpus result set as a response result of the translation request.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method for reusing translation target text materials as claimed in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for reusing translation target text materials as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method for reusing translation target text materials as claimed in any one of claims 1 to 6 is implemented.

Images