Method, apparatus, computer device, and readable storage medium for generating a training set

Abstract

The present disclosure relates to a method, apparatus, computer device and readable storage medium for generating a training set. The method comprises determining a sentence length probability set of a source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range, determining a sentence character length probability set of a source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range, and determining a first ratio probability set of a first ratio frequency distribution data within the first probability range. The source corpus is screened according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus and the first ratio probability set to form a source corpus training set, thereby providing a method for collecting and cleaning patent field corpus and forming a training set that can be applied in a machine translation system, thereby improving the translation quality in the patent field.

Description

Technical Field

[0001] The present disclosure relates to the technical field of corpus collection and cleaning, and in particular to a method, apparatus, computer equipment, and readable storage medium for generating a training set. Background Art

[0002] As machine learning technology becomes more and more mature, the role of machine translation systems in patent translation is becoming increasingly important. With current technology, the corpus training sets used by machine translation systems during training are mostly not corpus in the patent field. This means that the quality of the training results of machine translation systems, that is, in the field of patent translation, needs to be improved. Summary of the Invention

[0003] Based on this, it is necessary to provide a method, device, computer equipment and readable storage medium for generating training sets that can collect and clean patent field corpus and form a training set that can be used in a machine translation system, thereby improving the translation quality in the patent field, in order to address the above technical problems.

[0004] In a first aspect, the present disclosure provides a method for generating a training set. The method comprises:

[0005] Determining first ratio data according to sentence length data of the source corpus and sentence length data of the target corpus;

[0006] Determining sentence length frequency distribution data of the source corpus according to the sentence length data of the source corpus, and determining a sentence length probability set of the source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range;

[0007] Determining sentence character length frequency distribution data of the source corpus according to the sentence character length data of the source corpus, and determining a sentence character length probability set of the source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range;

[0008] determining first ratio frequency distribution data according to the first ratio data, and determining a first ratio probability set of the first ratio frequency distribution data within a first probability range;

[0009] The source corpus is screened according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set.

[0010] In one embodiment, the method further comprises:

[0011] Determining second ratio data based on the sentence length data of the source corpus and the sentence length data of the target corpus;

[0012] Determining sentence length frequency distribution data of the target corpus according to the sentence length data of the target corpus, and determining a sentence length probability set of the target corpus whose sentence length frequency distribution data of the target corpus is within a second probability range;

[0013] Determining sentence character length frequency distribution data of the target corpus based on the sentence character length data of the target corpus, and determining a sentence character length probability set of the target corpus within a second probability range for which the sentence character length frequency distribution data of the target corpus is within a second probability range;

[0014] determining second ratio frequency distribution data according to the second ratio data, and determining a second ratio probability set for the second ratio frequency distribution data within a second probability range;

[0015] The target corpus is screened according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set to form a target corpus training set.

[0016] In one embodiment, the screening of the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form the source corpus training set includes:

[0017] Deleting the source corpus outside the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set respectively;

[0018] The source corpus that contains the target corpus in the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set is deleted respectively.

[0019] In one embodiment, the screening of the target corpus according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set to form the target corpus training set includes:

[0020] Deleting target corpora outside the sentence length probability set, the sentence character length probability set, and the second ratio probability set of the target corpora respectively;

[0021] Delete the target corpus that contains the source corpus in the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set respectively.

[0022] In one embodiment, the method further comprises:

[0023] Deleting sentences containing specific characters and specific words in the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set;

[0024] Sentences containing specific characters and specific words in the target corpus are deleted according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set.

[0025] In one embodiment, the method further comprises:

[0026] The bilingual corpus is formed into a parallel corpus according to preset rules, wherein the parallel corpus includes a source corpus and a target corpus;

[0027] Determining sentence length data of the source corpus and sentence character length data of the source corpus respectively according to the parallel corpus;

[0028] The sentence length data of the target corpus and the sentence character length data of the target corpus are determined respectively according to the parallel corpus.

[0029] In one embodiment, the method further includes: forming a machine translation system training set based on the source corpus training set and the target corpus training set.

[0030] In a second aspect, the present disclosure further provides a device for generating a training set. The device comprises:

[0031] A first ratio module, configured to determine first ratio data based on the sentence length data of the source corpus and the sentence length data of the target corpus;

[0032] a sentence length probability module, configured to determine sentence length frequency distribution data of the source corpus based on the sentence length data of the source corpus, and determine a sentence length probability set of the source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range;

[0033] a character length probability module, configured to determine, based on the sentence character length data of the source corpus, sentence character length frequency distribution data of the source corpus, and determine a set of sentence character length probabilities of the source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range;

[0034] a first ratio probability module, configured to determine first ratio frequency distribution data according to the first ratio data, and determine a first ratio probability set of the first ratio frequency distribution data within a first probability range;

[0035] The training set forming module is used to screen the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set.

[0036] In a third aspect, the present disclosure further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in any embodiment of the present disclosure when executing the computer program.

[0037] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present disclosure.

[0038] In a fifth aspect, the present disclosure further provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method described in any embodiment of the present disclosure is implemented.

[0039] The embodiment provided by the present disclosure determines a sentence length probability set of the source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range, determines a sentence character length probability set of the source corpus whose sentence character length frequency distribution data of the source corpus is within the first probability range, and determines a first ratio probability set of the first ratio frequency distribution data within the first probability range. The source corpus is screened according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set, thereby providing a method for collecting and cleaning patent field corpus and forming a training set that can be used in a machine translation system, thereby improving the quality of patent field translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the implementation methods of this specification or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the implementation methods or the description of the prior art. Obviously, the drawings described below are only some implementation methods recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A diagram illustrating an application environment of a method for generating a training set in one embodiment;

[0042] Figure 2 A flowchart of a method for generating a training set in one embodiment;

[0043] Figure 3 A flowchart of a method for generating a training set in one embodiment;

[0044] Figure 4 A flowchart of a method for generating a training set in one embodiment;

[0045] Figure 5 A flowchart of a method for generating a training set in one embodiment;

[0046] Figure 6 A flowchart of a method for generating a training set in one embodiment;

[0047] Figure 7 A flowchart of a method for generating a training set in one embodiment;

[0048] Figure 8 A flowchart of a method for generating a training set in one embodiment;

[0049] Figure 9 A structural block diagram of an apparatus for generating a training set in one embodiment;

[0050] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not intended to limit the present disclosure.

[0052] The method for forming a machine translation system training set provided by the embodiment of the present disclosure can be applied to Figure 1 In the application environment shown in FIG. 1 , the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104 or placed on the cloud or other network servers.

[0053] Determine first ratio data based on the sentence length data of the source corpus and the sentence length data of the target corpus. Determine sentence length frequency distribution data of the source corpus based on the sentence length data of the source corpus, and determine a sentence length probability set of the source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range. Determine sentence character length frequency distribution data of the source corpus based on the sentence character length data of the source corpus, and determine a sentence character length probability set of the source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range. Determine first ratio frequency distribution data based on the first ratio data, and determine a first ratio probability set of the first ratio frequency distribution data within the first probability range.

[0054] The server 104 filters the source corpus based on the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0055] In one embodiment, Figure 2 As shown, a method for generating a training set is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the terminal in the figure:

[0056] S202: Determine first ratio data according to the sentence length data of the source corpus and the sentence length data of the target corpus.

[0057] The source corpus and the target corpus may comprise a language pair to be trained, for example, the source corpus is Chinese and the target corpus is English. Sentence length may comprise the number of words contained in a sentence. The sentence length data of the source corpus may comprise a set of the number of words contained in all sentences in the source corpus. The first ratio data may comprise a set of ratios of the sentence length data in the source corpus and the sentence length data in the target corpus, i.e., the ratio of the sentence length in the source corpus to the sentence length at the corresponding position in the target corpus.

[0058] Specifically, the first ratio data can be obtained by dividing the sentence length data of the source corpus by the sentence length data of the target corpus. In some embodiments, the first ratio data can be obtained by dividing the set of the number of words contained in all sentences in the source corpus by the set of the number of words contained in all sentences in the target corpus at the corresponding position, wherein the first ratio data can include a ratio set.

[0059] S204: Determine sentence length frequency distribution data of the source corpus based on the sentence length data of the source corpus, and determine a sentence length probability set of the source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range.

[0060] The sentence length frequency distribution data of the source corpus may include a sentence length frequency distribution histogram of the source corpus. The first probability range may include a probability value set based on experience or actual rule requirements when the frequency distribution histogram conforms to or approximates a Gaussian distribution. For example, a first probability range of 95% is set for a frequency distribution histogram conforming to a Gaussian distribution. Because the frequency distribution histogram conforming to the Gaussian distribution is symmetrical about a certain value, when the probability range is set to 95%, the data contained in the range of 2.5% to the left and right of the frequency distribution histogram will be eliminated. The first probability range of 95% may be a two-sided boundary value.

[0061] Specifically, based on the sentence length data of the source corpus, a bin width of 5 to 12 or another number can be used, and the difference between the maximum value and the minimum value of the sentence length data of the source corpus is divided by the bin width as the bin width, thereby plotting the sentence length frequency distribution data of the source corpus or a sentence length frequency distribution histogram of the source corpus. A source corpus sentence length probability set of the sentence length frequency distribution data of the source corpus is determined based on a first probability range.

[0062] S206 , determining sentence character length frequency distribution data of the source corpus based on the sentence character length data of the source corpus, and determining a sentence character length probability set of the source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range.

[0063] Among them, the sentence character length frequency distribution data of the source corpus can include a frequency distribution diagram of the sentence character length of the source corpus, or the proportion of the sentence character length frequency of the source corpus, and can also include a sentence character length frequency distribution histogram of the source corpus. The sentence character length can include the number of characters contained in a sentence. The sentence character length data of the source corpus can include a set of the number of characters contained in all sentences in the source corpus. The frequency distribution histogram can include a statistical diagram in which the height of the rectangle represents the frequency of the corresponding group.

[0064] Specifically, based on the sentence character length data of the source corpus, and with a bin width of 5 to 12 or other numbers, the difference between the maximum value and the minimum value of the sentence character length data of the source corpus is divided by the bin width as the bin width, thereby plotting the sentence character length frequency distribution data of the source corpus or plotting a frequency distribution histogram of the sentence character length of the source corpus. A sentence character length probability set of the source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range is determined.

[0065] S208: Determine first ratio frequency distribution data based on the first ratio data, and determine a first ratio probability set of the first ratio frequency distribution data within a first probability range.

[0066] The first ratio frequency distribution data may include the distribution of each frequency in the first ratio or the proportion of each frequency, and may also include a first ratio frequency distribution histogram.

[0067] Specifically, based on the first ratio data, and with 5 to 12 or other numbers as the group interval, the difference between the maximum value and the minimum value in the first ratio data is used and divided by the group interval as the group interval width, so as to draw the frequency distribution data of the first ratio data or to draw the frequency distribution histogram of the first ratio data to determine the first ratio probability set of the first ratio frequency distribution data within the first probability range.

[0068] S210 , screening the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set.

[0069] The training set may include a portion of the samples that are divided into for building a model in machine learning.

[0070] Specifically, the source corpus may be screened using a source corpus sentence length probability set, a source corpus sentence character length probability set, and a first ratio probability set, and the screened source corpus is used as a source corpus training set.

[0071] In the above-mentioned method for generating a training set, a sentence length probability set of the source corpus is determined, wherein the frequency distribution data of the sentence length of the source corpus is within a first probability range; a sentence character length probability set of the source corpus is determined, wherein the frequency distribution data of the sentence character length of the source corpus is within a first probability range; and a first ratio probability set of the first ratio frequency distribution data is determined within the first probability range. The source corpus is screened based on the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set, thereby providing a method for collecting and cleaning patent field corpus and forming a training set that can be used in a machine translation system, thereby improving the quality of translation in the patent field.

[0072] In one embodiment, Figure 3 As shown, the method further includes the following steps:

[0073] S302: Determine second ratio data according to the sentence length data of the source corpus and the sentence length data of the target corpus.

[0074] The second ratio data may include a set of ratios of sentence length data in the target corpus to sentence length data in the source corpus, that is, the ratio of the sentence length in the target corpus to the sentence length at the corresponding position in the source corpus.

[0075] Specifically, the second ratio data can be obtained by dividing the sentence length data of the target corpus by the sentence length data of the source corpus. In some embodiments, the second ratio data can be obtained by dividing the set of the number of words contained in all sentences in the target corpus by the set of the number of words contained in all sentences in the source corpus at corresponding positions, wherein the second ratio data can include a ratio set.

[0076] S304: Determine sentence length frequency distribution data of the target corpus based on the sentence length data of the target corpus, and determine a sentence length probability set of the target corpus whose sentence length frequency distribution data of the target corpus is within a second probability range.

[0077] Among them, the second probability range can include a probability value set according to experience or actual rule requirements when the frequency distribution histogram conforms to or is approximately Gaussian distribution. For example, a first probability range of 85% is set for a frequency distribution histogram that conforms to Gaussian distribution. Because the frequency distribution histogram that conforms to Gaussian distribution is symmetrical about a certain value, when the probability range is set to 85%, the data contained in the left and right 7.5% range of the frequency distribution histogram will be eliminated. Among them, the second probability range of 85% can be a two-sided boundary value.

[0078] Specifically, based on the sentence length data of the target corpus, and with a group interval of 5 to 12 or other numbers, the difference between the maximum value and the minimum value of the sentence length data of the target corpus is divided by the group interval as the group interval width, so as to draw the sentence length frequency distribution data of the target corpus, or the sentence length frequency distribution histogram of the target corpus can be used to draw the sentence length frequency distribution data of the target corpus and then determine the sentence length probability set of the target corpus within the second probability range of the sentence length frequency distribution data of the target corpus.

[0079] S306 , determining the sentence character length frequency distribution data of the target corpus based on the sentence character length data of the target corpus, and determining a sentence character length probability set of the target corpus whose sentence character length frequency distribution data of the target corpus is within a second probability range.

[0080] Specifically, based on the sentence character length data of the target corpus, and with a group interval of 5 to 12 or other numbers, the difference between the maximum value and the minimum value of the sentence character length data of the target corpus can be divided by the group interval as the group interval width, so as to draw the sentence character length frequency data of the target corpus, or the sentence character length frequency distribution histogram of the target corpus can be used to draw the sentence character length frequency distribution data of the target corpus and then determine the sentence character length probability set of the target corpus within the second probability range.

[0081] S308: Determine second ratio frequency distribution data based on the second ratio data, and determine a second ratio probability set of the second ratio frequency distribution data within a second probability range.

[0082] Specifically, based on the second ratio data, and with a group interval of 5 to 12 or other numbers, the difference between the maximum value and the minimum value in the first ratio data is used and divided by the group interval as the group interval width, so as to draw the frequency distribution data of the second ratio data, or the frequency distribution histogram of the second ratio data can be used to draw the second ratio frequency distribution data and then determine the second ratio probability set within the second probability range.

[0083] S310 , screening the target corpus according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set to form a target corpus training set.

[0084] Specifically, the target corpus can be screened using a target corpus sentence length probability set, a target corpus sentence character length probability set, and a second ratio probability set, and the screened target corpus is used as a target corpus training set.

[0085] In this embodiment, a sentence length probability set of the target corpus is determined to be within a second probability range for the sentence length frequency distribution data of the target corpus, a sentence character length probability set of the target corpus is determined to be within a second probability range for the sentence character length frequency distribution data of the target corpus, and a second ratio probability set of the second ratio frequency distribution data is determined to be within a second probability range. Furthermore, the target corpus is screened based on the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set to form a target corpus training set. This provides a method for collecting and cleaning patent field corpus and forming a training set that can be used in a machine translation system, thereby improving the quality of translation in the patent field.

[0086] In one embodiment, Figure 4 As shown, step S210 of screening the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set includes the following steps:

[0087] S402 , respectively deleting source corpora outside the sentence length probability set of the source corpora, the sentence character length probability set of the source corpora, and the first ratio probability set.

[0088] Specifically, source corpus that is not within the range of the sentence length probability set of the source corpus in the sentence length frequency distribution histogram of the source corpus can be deleted; source corpus that is not within the range of the sentence character length probability set of the source corpus in the sentence character length frequency distribution histogram of the source corpus can be deleted; source corpus that is not within the range of the first ratio probability set in the first ratio frequency distribution histogram can be deleted.

[0089] S404: Delete the source corpus that contains the target corpus in the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set.

[0090] Specifically, source corpora that contain target corpora within the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set can be deleted. In some embodiments, for example, if the source corpus is English and the target corpus is Chinese, English corpora that contain Chinese corpora within the range of the sentence length probability set of the English corpus can be deleted from the sentence length frequency distribution histogram of the English corpus.

[0091] In this embodiment, by deleting source corpora outside the sentence length probability set, the sentence character length probability set and the first ratio probability set of the source corpora and deleting source corpora containing target corpora within the sentence length probability set, the sentence character length probability set and the first ratio probability set of the source corpora, a method is provided that can collect and clean patent field corpora and form a training set that can be used in a machine translation system, thereby improving the translation quality in the patent field.

[0092] In one embodiment, Figure 5 As shown, step S310 of screening the target corpus according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set to form a target corpus training set includes the following steps:

[0093] S502 , respectively deleting target corpora outside the sentence length probability set of the target corpora, the sentence character length probability set of the target corpora, and the second ratio probability set.

[0094] Specifically, the target corpus whose sentence length frequency distribution histogram is not within the sentence length probability set of the target corpus can be deleted; the target corpus whose sentence character length frequency distribution histogram is not within the sentence character length probability set of the target corpus can be deleted; the target corpus whose second ratio frequency distribution histogram is not within the second ratio probability set can be deleted.

[0095] S504 , respectively deleting the target corpus that contains the source corpus in the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set.

[0096] Specifically, the target corpus that contains the source corpus within the target corpus's sentence length probability set, the target corpus's sentence character length probability set, and the second ratio probability set can be deleted. In some embodiments, for example, if the source corpus is English and the target corpus is Chinese, the Chinese corpus that contains the English corpus within the range of the Chinese corpus's sentence length probability set can be deleted from the sentence length frequency distribution histogram of the Chinese corpus.

[0097] In this embodiment, by deleting the target corpus outside the sentence length probability set, the sentence character length probability set and the second ratio probability set of the target corpus and deleting the target corpus containing the source corpus within the sentence length probability set, the sentence character length probability set and the second ratio probability set of the target corpus, a method is provided that can collect and clean patent field corpus and form a training set that can be used in a machine translation system, thereby improving the translation quality in the patent field.

[0098] In one embodiment, Figure 6 As shown, the method further includes the following steps:

[0099] S602: Delete sentences containing specific characters and specific words in the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set.

[0100] The specific characters may not conform to the format of the source language. For example, when the source language is Chinese, the special characters may include a comma in English format.

[0101] Specifically, sentences containing specific characters and specific words can be deleted within the range of the sentence length probability set of the source corpus in the sentence length frequency distribution histogram of the source corpus; sentences containing specific characters and specific words can be deleted within the range of the sentence character length probability set of the source corpus in the sentence character length frequency distribution histogram of the source corpus; sentences containing specific characters and specific words can be deleted within the range of the first ratio probability set in the first ratio frequency distribution histogram.

[0102] S604: Delete sentences containing specific characters and specific words in the target corpus according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set.

[0103] Specifically, sentences containing specific characters and specific words can be deleted within the range of the sentence length probability set of the target corpus in the sentence length frequency distribution histogram of the target corpus; sentences containing specific characters and specific words can be deleted within the range of the sentence character length probability set of the target corpus in the sentence character length frequency distribution histogram of the target corpus; sentences containing specific characters and specific words can be deleted within the range of the second ratio probability set in the second ratio frequency distribution histogram.

[0104] In this embodiment, by deleting sentences containing specific characters and specific words in the source corpus based on the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus and the first ratio probability set, and deleting sentences containing specific characters and specific words in the target corpus based on the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus and the second ratio probability set, a method is provided that can collect and clean patent field corpora and form a training set that can be used in a machine translation system, which can improve the translation quality in the patent field.

[0105] In one embodiment, Figure 7 As shown, the method further includes the following steps:

[0106] S702: Generate a parallel corpus from the bilingual corpus according to preset rules, where the parallel corpus includes a source corpus and a target corpus.

[0107] The bilingual corpus may include bilingual corpora extracted from a storage medium of a CAT system (a system that utilizes computer-assisted translation). The preset rule may include that the quantity of the bilingual corpora is the same, including the number of sentences in corresponding paragraphs and the number of words in corresponding sentences in the corpora. The preset rule may also include performing a random sampling check on the determined parallel corpora to check whether the quantity of the parallel corpora is equal. If not, re-extracting new bilingual corpora from the CAT system.

[0108] Specifically, bilingual corpora can be extracted from the storage medium of the CAT system, and then it can be checked whether the quantities of the bilingual corpora are equal, and bilingual corpora with equal quantities can be determined as parallel corpora.

[0109] S704: Determine sentence length data of the source corpus and sentence character length data of the source corpus according to the parallel corpus.

[0110] Specifically, the sentence length data of the source corpus and the sentence character length data of the source corpus can be determined respectively according to the source corpus in the parallel corpus.

[0111] S706: Determine sentence length data of the target corpus and sentence character length data of the target corpus according to the parallel corpus.

[0112] Specifically, the sentence length data of the target corpus and the sentence character length data of the target corpus can be determined respectively according to the target corpus in the parallel corpus.

[0113] In this embodiment, by forming a parallel corpus from bilingual corpora according to preset rules, determining the sentence length data of the source corpus and the sentence character length data of the source corpus based on the source corpus in the parallel corpus, and determining the sentence length data of the target corpus and the sentence character length data of the target corpus based on the target corpus in the parallel corpus, a system is provided that can collect and clean patent field corpora and form a training set that can be used in a machine translation system, thereby improving the translation quality in the patent field.

[0114] In one embodiment, the method further comprises: forming a machine translation system training set based on the source corpus training set and the target corpus training set.

[0115] Specifically, the source corpus training set and the target corpus training set are combined into one-to-one corresponding training sets to form a machine translation system training set.

[0116] In this embodiment, by forming a machine translation system training set based on the source corpus training set and the target corpus training set, a method is provided that can collect and clean patent field corpus and form a training set that can be used in a machine translation system, thereby improving the translation quality in the patent field.

[0117] In one embodiment, Figure 8 As shown, a method for generating a training set is provided, the method comprising the following steps:

[0118] S802: Generate a parallel corpus from the bilingual corpus according to preset rules, where the parallel corpus includes a source corpus and a target corpus.

[0119] S804: Determine sentence length data of the source corpus and sentence character length data of the source corpus according to the parallel corpus.

[0120] S806: Determine sentence length data of the target corpus and sentence character length data of the target corpus according to the parallel corpus.

[0121] S808: Determine first ratio data according to the sentence length data of the source corpus and the sentence length data of the target corpus.

[0122] S810: Determine sentence length frequency distribution data of the source corpus based on the sentence length data of the source corpus, and determine a sentence length probability set of the source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range.

[0123] S812. Determine sentence character length frequency distribution data of the source corpus based on the sentence character length data of the source corpus, and determine a sentence character length probability set of the source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range.

[0124] S814. Determine first ratio frequency distribution data based on the first ratio data, and determine a first ratio probability set of the first ratio frequency distribution data within a first probability range.

[0125] S816 , respectively deleting source corpora outside the sentence length probability set of the source corpora, the sentence character length probability set of the source corpora, and the first ratio probability set.

[0126] S818. Delete the source corpus that contains the target corpus in the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set respectively.

[0127] S820: Determine second ratio data based on the sentence length data of the source corpus and the sentence length data of the target corpus.

[0128] S822. Determine sentence length frequency distribution data of the target corpus based on the sentence length data of the target corpus, and determine a sentence length probability set of the target corpus whose sentence length frequency distribution data of the target corpus is within a second probability range.

[0129] S824. Determine the sentence character length frequency distribution data of the target corpus based on the sentence character length data of the target corpus, and determine a sentence character length probability set of the target corpus whose sentence character length frequency distribution data of the target corpus is within a second probability range.

[0130] S826. Determine second ratio frequency distribution data based on the second ratio data, and determine a second ratio probability set of the second ratio frequency distribution data within a second probability range.

[0131] S828. Delete the target corpus outside the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set.

[0132] S830: Delete the target corpus that contains the source corpus in the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set.

[0133] S832: Delete sentences containing specific characters and specific words from the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set.

[0134] S834. Delete sentences containing specific characters and specific words in the target corpus according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set.

[0135] S836. Form a machine translation system training set based on the source corpus training set and the target corpus training set.

[0136] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the accompanying drawings may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The order of execution of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or at least a portion of steps or stages in other steps.

[0137] Based on the same inventive concept, embodiments of the present disclosure also provide an apparatus for forming a machine translation system training set, for implementing the aforementioned method for forming a machine translation system training set. The solution provided by this apparatus is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the apparatus for forming a machine translation system training set provided below can be found in the aforementioned method for forming a machine translation system training set, and will not be further elaborated here.

[0138] In one embodiment, Figure 9 As shown, a device 900 for generating a training set is provided, comprising: a first ratio module 902, a sentence length probability module 904, a character length probability module 906, a first ratio probability module 908, and a training set forming module 910, wherein:

[0139] The first ratio module 902 is configured to determine first ratio data based on the sentence length data of the source corpus and the sentence length data of the target corpus.

[0140] The sentence length probability module 904 is used to determine the sentence length frequency distribution data of the source corpus based on the sentence length data of the source corpus, and determine the sentence length probability set of the source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range.

[0141] The character length probability module 906 is used to determine the sentence character length frequency distribution data of the source corpus based on the sentence character length data of the source corpus, and determine the sentence character length probability set of the source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range.

[0142] The first ratio probability module 908 is configured to determine first ratio frequency distribution data according to the first ratio data, and determine a first ratio probability set of the first ratio frequency distribution data within a first probability range.

[0143] The training set forming module 910 is configured to screen the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set.

[0144] Each module in the apparatus for generating a training set may be implemented in whole or in part by software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0145] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for forming a training set for a machine translation system is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0146] Those skilled in the art will understand that Figure 10The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present disclosure, and does not constitute a limitation on the computer device to which the solution of the present disclosure is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0147] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0148] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0149] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0151] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in the present disclosure may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in each embodiment provided in this disclosure may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc.

[0152] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above-described embodiments merely represent several implementation methods of the present disclosure. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present disclosure. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present disclosure, all of which fall within the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be determined by the appended claims.

Claims

1. A method for generating a training set, characterized in that: The method comprises: Determining first ratio data based on sentence length data of a source corpus and sentence length data of a target corpus; the source corpus includes corpus in the patent field; Determining sentence length frequency distribution data of the source corpus according to the sentence length data of the source corpus, and determining a sentence length probability set of the source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range; Determining sentence character length frequency distribution data of the source corpus according to the sentence character length data of the source corpus, and determining a sentence character length probability set of the source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range; Determining first ratio frequency distribution data based on the first ratio data, and determining a first ratio probability set for the first ratio frequency distribution data within a first probability range; wherein the first probability range includes a two-sided limit of a Gaussian distribution; The source corpus is screened according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set; A machine translation system training set is formed based on the source corpus training set and the target corpus training set for application in the machine translation system.

2. The method according to claim 1, characterized in that The method further comprises: Determining second ratio data based on the sentence length data of the source corpus and the sentence length data of the target corpus; Determining sentence length frequency distribution data of the target corpus according to the sentence length data of the target corpus, and determining a sentence length probability set of the target corpus whose sentence length frequency distribution data of the target corpus is within a second probability range; Determining sentence character length frequency distribution data of the target corpus based on the sentence character length data of the target corpus, and determining a sentence character length probability set of the target corpus within a second probability range for which the sentence character length frequency distribution data of the target corpus is within a second probability range; determining second ratio frequency distribution data according to the second ratio data, and determining a second ratio probability set for the second ratio frequency distribution data within a second probability range; The target corpus is screened according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set to form a target corpus training set.

3. The method according to claim 1, characterized in that The screening of the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set includes: Deleting source corpora outside the sentence length probability set of the source corpora, the sentence character length probability set of the source corpora, and the first ratio probability set respectively; The source corpus that contains the target corpus in the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set is deleted respectively.

4. The method according to claim 2, characterized in that The screening of the target corpus according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set to form a target corpus training set includes: Deleting target corpora outside the sentence length probability set, the sentence character length probability set, and the second ratio probability set of the target corpora respectively; Delete the target corpus that contains the source corpus in the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set respectively.

5. The method according to any one of claims 3 to 4, characterized in that The method further comprises: Deleting sentences containing specific characters and specific words in the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set; Sentences containing specific characters and specific words in the target corpus are deleted according to the sentence length probability set of the target corpus, the sentence character length probability set of the target corpus, and the second ratio probability set.

6. The method according to claim 1, characterized in that The method further comprises: The bilingual corpus is formed into a parallel corpus according to preset rules, wherein the parallel corpus includes a source corpus and a target corpus; Determining sentence length data of the source corpus and sentence character length data of the source corpus respectively according to the parallel corpus; The sentence length data of the target corpus and the sentence character length data of the target corpus are determined respectively according to the parallel corpus.

7. A device for generating a training set, characterized in that The device comprises: A first ratio module is configured to determine first ratio data based on sentence length data of a source corpus and sentence length data of a target corpus; the source corpus includes corpus in the patent field; a sentence length probability module, configured to determine sentence length frequency distribution data of the source corpus based on the sentence length data of the source corpus, and determine a sentence length probability set of the source corpus whose sentence length frequency distribution data of the source corpus is within a first probability range; a character length probability module, configured to determine, based on the sentence character length data of the source corpus, sentence character length frequency distribution data of the source corpus, and determine a set of sentence character length probabilities of the source corpus whose sentence character length frequency distribution data of the source corpus is within a first probability range; a first ratio probability module, configured to determine first ratio frequency distribution data based on the first ratio data, and determine a first ratio probability set of the first ratio frequency distribution data within a first probability range; wherein the first probability range includes a two-sided limit of a Gaussian distribution; a training set forming module, configured to screen the source corpus according to the sentence length probability set of the source corpus, the sentence character length probability set of the source corpus, and the first ratio probability set to form a source corpus training set; The device is used to form a machine translation system training set based on a source corpus training set and a target corpus training set for application in a machine translation system.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images