Classification model training method, translation method, device and electronic equipment

Abstract

The present disclosure provides a classification model training method, a translation method, a device and an electronic device, and relates to the technical field of natural language processing and machine learning. The method comprises: obtaining bilingual parallel corpus, the bilingual parallel corpus comprising source language corpus and target language corpus; determining the translatable segment and the non-translatable segment in the source language corpus; the translatable segment is determined based on the unambiguous segment in the source language corpus; and training a classification model based on the translatable segment and the non-translatable segment. The technical scheme of the present disclosure trains the classification model based on the translatable segment and the non-translatable segment, and applies the trained classification model to translation, which can make the translation result more accurate and meet the demand for high quality and low latency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical field of natural language processing and machine learning. BACKGROUND

[0002] Simultaneous interpretation is a process of converting a natural language (source language) into another natural language (target language) in real time. At present, simultaneous interpretation is mainly completed by artificial interpreters, but there are only 3000 qualified simultaneous interpreters in the world, and they can only work continuously for 15-20 minutes, and the translation rate is only 60%. Therefore, it is hoped that computers can assist humans in simultaneous interpretation, that is, using machines for speech translation, also known as machine simultaneous interpretation. However, the machine simultaneous interpretation in the prior art often has inaccurate translation, and the translation quality is not high. SUMMARY

[0003] The present disclosure provides a classification model training method, a translation method, a device and an electronic equipment.

[0004] According to an aspect of the present disclosure, a classification model training method is provided, comprising:

[0005] obtaining bilingual parallel corpus, the bilingual parallel corpus comprising source language corpus and target language corpus;

[0006] determining the translatable segment and the non-translatable segment in the source language corpus; the translatable segment is determined based on the unambiguous segment in the source language corpus;

[0007] training the classification model based on the translatable segment and the non-translatable segment.

[0008] According to another aspect of the present disclosure, a translation method is provided, comprising:

[0009] obtaining the source language segment to be translated;

[0010] inputting the source language segment to be translated into the classification model, and translating the source language segment to be translated in the case that the classification result of the classification model is translatable;

[0011] The classification model is trained according to the method in any embodiment of the present disclosure.

[0012] According to another aspect of the present disclosure, a classification model training device is provided, comprising:

[0013] an obtaining module configured to obtain bilingual parallel corpus, the bilingual parallel corpus comprising source language corpus and target language corpus;

[0014] a determining module configured to determine the translatable segment and the non-translatable segment in the source language corpus; the translatable segment is determined based on the unambiguous segment in the source language corpus;

[0015] The training module is used to train a classification model based on translatable and non-translatable segments.

[0016] According to another aspect of this disclosure, a translation apparatus is provided, comprising:

[0017] The acquisition module is used to acquire source language fragments to be translated;

[0018] The translation module is used to input the source language fragment to be translated into the classification model. If the classification model determines that the source language fragment is translatable, the translation module will translate the source language fragment.

[0019] The classification model is trained according to the method in any embodiment of this disclosure.

[0020] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0021] At least one processor; and

[0022] The memory is communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods in any embodiment of this disclosure.

[0024] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods of any embodiment of this disclosure.

[0025] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods of any embodiment of this disclosure.

[0026] This disclosure provides a classification model training method, translation method, apparatus, and electronic device. It determines translatable and untranslatable segments from the source language corpus of bilingual parallel corpora. The translatable segments are determined based on unambiguous segments in the source language corpus. Based on the translatable and untranslatable segments, a classification model is trained. The trained classification model is then applied to translation, which can make the translation results more accurate and meet the requirements of high quality and low latency.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0028] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0029] Figure 1 This is a schematic diagram of a classification model training method in one embodiment of the present disclosure;

[0030] Figure 2 This is a schematic diagram illustrating the acquisition of unambiguous semantic segments in one embodiment of the present disclosure;

[0031] Figure 3 This is a schematic diagram of source language sentence segmentation in one embodiment of this disclosure;

[0032] Figure 4 This is a schematic diagram illustrating the determination of a translatable segment in one embodiment of this disclosure;

[0033] Figure 5 This is a flowchart of a classification model training method in one embodiment of the present disclosure;

[0034] Figure 6 This is a flowchart of a translation method in one embodiment of the present disclosure;

[0035] Figure 7 This is a schematic diagram of a classification model training device in one embodiment of the present disclosure;

[0036] Figure 8 This is a schematic diagram of the module determination in one embodiment of the present disclosure;

[0037] Figure 9 This is a schematic diagram of a translation device in one embodiment of the present disclosure;

[0038] Figure 10 This is a block diagram of an electronic device used to implement the classification model training method or translation method of the embodiments of this disclosure. Detailed Implementation

[0039] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0040] This disclosure provides a method for training a classification model. Figure 1This is a flowchart of a classification model training method according to an embodiment of the present disclosure. This method can be applied to a classification model training device, for example, when deployed on a terminal, server, or other processing device, it can perform classification model training. The terminal can be a user equipment (UE), mobile device, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, it includes:

[0041] Step S101: Obtain bilingual parallel corpus, which includes source language corpus and target language corpus;

[0042] Bilingual parallel corpora can be obtained from a pre-built bilingual parallel corpus. Each bilingual parallel corpus consists of multiple sentence pairs, each containing a source language sentence and a corresponding target language sentence translated from that source language sentence. The source language corpus includes source language sentences, and the target language corpus includes target language sentences.

[0043] Step S102: Determine the translatable and untranslatable segments in the source language corpus; the translatable segments are determined based on the unambiguous segments in the source language corpus.

[0044] Based on bilingual parallel corpora, translatable and untranslatable segments are identified from the source language corpus. A translatable segment can be a single word, a phrase or a group of words, or a short phrase. If a source language segment is not a translatable segment, it is considered an untranslatable segment.

[0045] Based on parallel corpora, multiple source language segments and at least one target language segment corresponding to each source language segment can be obtained. An unambiguous segment is a source language segment that corresponds to a target language segment. If a source language segment is unambiguous, it is further determined whether the source language segment is translatable.

[0046] Step S103: Train a classification model based on translatable and non-translatable segments.

[0047] Based on whether a segment is translatable, source language sentences in the source language corpus are segmented. Using translatable and non-translatable segments in the source language corpus, a training sample set is constructed to train a classification model. The trained classification model can be used for translation between the source and target languages.

[0048] In training the classification model, the initial model can be any text classification model, and this disclosure does not impose any restrictions on it.

[0049] This disclosure provides a classification model training method that determines translatable and untranslatable segments from the source language corpus of bilingual parallel corpora. The translatable segments are determined based on unambiguous segments in the source language corpus. Based on the translatable and untranslatable segments, a classification model is trained. Applying the trained classification model to translation can make the translation results more accurate and meet the requirements of high quality and low latency.

[0050] The method for determining translatable segments is illustrated in the following example:

[0051] In one possible implementation, the translatable segments in the source language corpus are identified, including:

[0052] Identify multiple unambiguous segments in the source language corpus, and the first target language segment corresponding to each of the multiple unambiguous segments;

[0053] Based on multiple unambiguous segments and their corresponding first target language segments, multiple translatable segments in the source language corpus are identified.

[0054] In practical applications, based on the source language sentences and their corresponding translated target language sentences in parallel corpora, unambiguous segments in each sentence of the source language corpus can be identified, as well as the target language segments corresponding to each unambiguous segment in the target language sentence. Each unambiguous segment corresponds to one target language segment, which is the first target language segment. For each unambiguous segment, based on the unambiguous segment and its corresponding first target language segment, it can be determined whether the unambiguous segment is translatable, thus yielding multiple translatable segments.

[0055] In this embodiment of the disclosure, translatable and untranslatable segments are further selected based on unambiguous segments and corresponding first target language segments, which can be used as the training sample set for the classification model.

[0056] In one possible implementation, based on multiple unambiguous segments and their corresponding first target language segments, multiple translatable segments in the source language corpus are determined, including:

[0057] For each unambiguous segment, obtain at least one source language sentence that begins with the unambiguous segment, translate the at least one source language sentence to obtain the target language sentence corresponding to each source language sentence;

[0058] Based on each unambiguous segment, the corresponding first target language segment for each unambiguous segment, and each target language sentence, multiple translatable segments in the source language corpus are identified.

[0059] In practical applications, for each unambiguous segment, it is input into a Generative Pre-Training (GPT) model. The GPT model outputs at least one source language sentence beginning with the unambiguous segment. These source language sentences are translated to obtain the corresponding target language sentence for each source language sentence. The first target language segment corresponding to the unambiguous segment is compared with the target language segments corresponding to the unambiguous segment in the target language sentences. Based on the comparison results, it is determined whether the unambiguous segment is translatable. Since multiple source language sentences beginning with the unambiguous segment may be generated from each unambiguous segment output by the GPT model, resulting in multiple translated target language sentences, the first target language segment can be compared sequentially with each target language sentence corresponding to the unambiguous segment to determine whether the unambiguous segment is translatable.

[0060] In this embodiment of the disclosure, a sentence is constructed starting with an unambiguous fragment and translated. The unambiguous fragment is then compared with the translated target language sentence to determine whether it is a translatable fragment. The translatable fragment obtained in this way is used as a training sample for the classification model. The completed classification model can take into account more possibilities that may bring differences to the translation results. Applying the model to the translation process can make the translation results more accurate.

[0061] In one possible implementation, based on each unambiguous segment, the corresponding first target language segment for each unambiguous segment, and each target language sentence, multiple translatable segments in the source language corpus are determined, including:

[0062] For each unambiguous segment, obtain at least one second target language segment in the target language sentence that corresponds to the unambiguous segment;

[0063] Unambiguous segments that satisfy a preset matching condition between the first target language segment and at least one second target language segment are considered as translatable segments in the source language corpus.

[0064] In practical applications, the words in the source language sentence beginning with the unambiguous fragment are aligned with the words in the target language sentence translated from that sentence to obtain the target language fragment in the target language sentence corresponding to the unambiguous fragment, i.e., the second target language fragment. Since each unambiguous fragment may result in multiple source language sentences output by the GPT model, each beginning with that unambiguous fragment, and thus multiple translated target language sentences, the first target language fragment can be compared sequentially with the second target language fragment in each target language sentence. If the first target language fragment corresponding to the unambiguous fragment and each of the second target language fragments corresponding to that unambiguous fragment satisfy a preset matching condition, then the unambiguous fragment is a translatable fragment; otherwise, the unambiguous fragment is an untranslatable fragment. The preset matching condition can be that the first target language fragment matches every second target language fragment in each target language sentence, or that a preset matching degree threshold is met, and can be configured according to specific needs.

[0065] In this embodiment of the disclosure, a translatable segment is determined based on the degree of matching between the first target language segment and at least one second target language segment. The translatable segment obtained in this way is used as a training sample for the classification model. The completed classification model can be applied to the translation process to make the translation results more accurate.

[0066] In one possible implementation, multiple unambiguous segments in the source language corpus are determined, along with a first target language segment corresponding to each of the multiple unambiguous segments, including:

[0067] Align the source language words and target language words in the bilingual parallel corpus;

[0068] Based on the aligned source language words and target language words, multiple unambiguous segments in the source language corpus are identified, as well as the first target language segment corresponding to each of the multiple unambiguous segments.

[0069] In practical applications, word alignment between source and target language sentences in bilingual parallel corpora can be achieved using word alignment tools, such as GIZA, which employs the Expectation-Maximum (EM) algorithm to align words in the sentences between the source and target languages ​​and record the alignment information. Based on this alignment information, multiple unambiguous segments in the source language corpus and multiple primary target language segments can be identified.

[0070] In this embodiment of the disclosure, unambiguous segments and corresponding first target language segments can be determined from bilingual parallel corpora as the basis for determining translatable segments.

[0071] In one possible implementation, based on the aligned source language words and target language words, multiple unambiguous segments in the source language corpus are determined, along with the first target language segment corresponding to each of the multiple unambiguous segments, including:

[0072] Based on the aligned source language words and target language words, a source language fragment containing the source language words and at least one corresponding target language fragment are obtained;

[0073] The source language segment corresponding to a target language segment is taken as the unambiguous segment, and the target language segment corresponding to the unambiguous segment is taken as the first target language segment.

[0074] In practical applications, the alignment information of the source and target languages ​​is statistically analyzed and recorded in a phrase table. The phrase table records a source language segment and at least one corresponding first target language segment. Each source language segment includes at least one source language word, and each target language segment includes at least one target language word. If a source language segment has only one corresponding first target language segment, the source language segment is considered an unambiguous segment, and the corresponding first target language segment is considered the first target language segment corresponding to the unambiguous segment. For each sentence pair and its alignment information in the bilingual parallel corpus, the source language segment corresponding to each word in the target language is determined character by character from beginning to end. If the corresponding source language segment is an unambiguous segment, it is segmented at that point to obtain multiple unambiguous segments and their corresponding first target language segments.

[0075] In this embodiment of the disclosure, unambiguous segments and corresponding first target language segments can be obtained based on the alignment information of source language words and target language words, which serve as the basis for obtaining translatable segments.

[0076] In one possible implementation, a classification model is trained based on translatable and non-translatable segments, including:

[0077] Obtain the category labels for translatable and non-translatable segments;

[0078] The translatable fragments, untranslatable fragments, and their corresponding category labels are used as the training sample set to train the classification model.

[0079] In practical applications, category labels for translatable and non-translatable segments can be configured through manual or machine annotation. For example, the category label for translatable segments can be 1, and the category label for non-translatable segments can be 0. A training sample set is then constructed using the translatable and non-translatable segments and their corresponding category labels to train the classification model.

[0080] In the embodiments of the present disclosure, a classification model is trained based on translatable segments and non - translatable segments. The trained classification model is used to classify and identify the input source - language segments. Applying this classification model to translation can make the translation results more accurate.

[0081] The technical solutions of the present disclosure will be introduced in detail through several specific embodiments below.

[0082] Figure 2 It is a schematic diagram for obtaining an unambiguous semantic segment in an embodiment of the present disclosure. As Figure 2 shown,

[0083] First, sentence alignment is performed. Source - language sentence 1: This is made possible through close collaboration; corresponding target - language sentence 1: 这是通过紧密协作实现的. Source - language sentence 2: If you are cold, close the window; corresponding target - language sentence 2: 你要是冷就把窗户关上. The source - language sentences and the corresponding target - language sentences are respectively word - aligned. As Figure 2 shown, for source - language sentence 1 and corresponding target - language sentence 1: "This" is aligned with "这"; "is" is aligned with "是"; "made" is aligned with "实现的"; "possible" is aligned with "实现的"; "through" is aligned with "通过"; "close" is aligned with "紧密"; "collaboration" is aligned with "协作". For source - language sentence 2 and corresponding target - language sentence 2: "If" is aligned with "要是"; "you" is aligned with "你"; "cold" is aligned with "冷"; "close" is aligned with "关上"; "window" is aligned with "窗户".

[0084] Then, a phrase table is obtained. According to the alignment information of source - language sentence 1 and target - language sentence 1, source - language sentence 2 and target - language sentence 2, a phrase table is obtained, such as Figure 2 shown: "close→紧密 / 关上、close collaboration→紧密协作、close the window→把窗户关上".

[0085] Finally, segment the source language sentence. In the phrase table, "collaboration" and "close the window" each correspond to a target language segment, so "collaboration" and "close the window" are unambiguous segments. However, "close" corresponds to two target language segments, "tight" and "close", so "close" is not an unambiguous segment. If the corresponding source language segment is an unambiguous segment, perform a segmentation here to obtain multiple unambiguous segments. The source language sentence 1 can be segmented into 4 segments: This, is, made possible through, close collaboration; the source language sentence 2 can be segmented into 3 segments: If you, are cold, close the window.

[0086] Figure 3 This is a schematic diagram of the segmentation of the source language sentence in an embodiment of the present disclosure. As Figure 3 shown, for the source language sentence 1: This is made possible through close collaboration; the corresponding target language sentence 1: This is achieved through close collaboration. According to the phrase table, for "This" corresponding to "这", judge whether "This" only corresponds to one target language segment. If so, perform a segmentation after "This", and then judge whether "is" corresponding to "是" only corresponds to one target language segment... until the segmentation of the source language sentence is completed to obtain multiple unambiguous segments. Here, pointer represents the position order of the word in the sentence from left to right.

[0087] Figure 4 This is a schematic diagram of determining translatable segments in an embodiment of the present disclosure. As Figure 4As shown, if "Cut the banana" is an unambiguous segment and its corresponding first target language segment is "切香蕉", input "Cut the banana" into the GPT model to obtain source language sentence A: Cut the banana and the apple. Source language sentence B: Cut the banana here. Translate source language sentence A and source language sentence B to obtain target language sentence C: 切香蕉和苹果、target language sentence D: 把香蕉切在这里。Align the words of source language sentence A, source language sentence B with their respective target language sentences C, D. The second target language segment corresponding to "Cut the banana" in target language sentence C is "切香蕉", and the second target language segment corresponding to "Cut the banana" in target language sentence D is "香蕉切". If the preset matching condition is that the first target language segment of the unambiguous segment and all the second target language segments corresponding to the unambiguous segment match, then the unambiguous segment can be determined as a translatable segment. Since one of the second target language segments of the unambiguous segment "Cut the banana", "香蕉切", does not match the first target language segment corresponding to "Cut the banana" (as shown as "不一致" in the figure), the unambiguous segment "Cut the banana" is an untranslatable segment.

[0088] Figure 5 The following is a flowchart of a classification model training method in an embodiment of the present disclosure. As Figure 5 shown, the method includes:

[0089] Step S501, obtain bilingual parallel corpus, the bilingual parallel corpus includes source language corpus and target language corpus;

[0090] Step S502, determine multiple unambiguous segments in the source language corpus, and the first target language segment corresponding to each of the multiple unambiguous segments;

[0091] Step S503, for each unambiguous segment, obtain at least one source language sentence starting with the unambiguous segment, and translate the at least one source language sentence to obtain the target language sentence corresponding to each source language sentence;

[0092] Step S504, for each unambiguous segment, obtain at least one second target language segment corresponding to the unambiguous segment in the target language sentence;

[0093] Step S505, regard the unambiguous segments whose first target language segment and at least one second target language segment meet the preset matching condition as the translatable segments in the source language corpus;

[0094] Step S506: Obtain the category labels corresponding to the translatable and non-translatable segments respectively, and use the translatable segments, non-translatable segments and their corresponding category labels as training sample sets to train the classification model.

[0095] In this embodiment of the disclosure, a sentence is constructed starting with an unambiguous segment and translated. The unambiguous segment is then compared with the translated target language sentence to determine whether it is a translatable segment. The resulting translatable and untranslatable segments are used as training samples for the classification model. The completed classification model can take into account more possibilities that may cause differences in the translation results. Applying the model to the translation process can make the translation results more accurate.

[0096] This disclosure provides a translation method. Figure 6 This is a flowchart of a translation method according to one embodiment of the present disclosure. This method can be applied to a translation device, for example, when deployed on a terminal, server, or other processing device, it can perform tasks such as classification model training. The terminal can be a user equipment (UE), mobile device, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 6 As shown, translation methods may include:

[0097] Step S601: Obtain the source language fragment to be translated;

[0098] In this embodiment of the disclosure, during the translation process, a source language sentence to be translated can be received and segmented to obtain source language fragments to be translated. Alternatively, source language fragments can be received directly. For example, in the process of machine simultaneous interpretation, streaming source language fragments to be translated can be received. The source language fragment to be translated includes at least one source language word.

[0099] Step S602: Input the source language segment to be translated into the classification model. If the classification result of the classification model is translatable, translate the source language segment to be translated.

[0100] The classification model is trained according to any of the classification model training methods in the embodiments of this disclosure. After receiving the source language segment to be translated, the classification result of the classification model can be used to determine whether to translate the source language segment. It is understood that this disclosure does not limit the specific method of translating the content of the source language segment to be translated.

[0101] In related technologies, if unambiguous segments are used to train a classification model, the unambiguous segments may produce different translation results in different scenarios. Therefore, when a classification model trained on unambiguous segments is applied to translation, inaccurate translation may occur.

[0102] Therefore, the technical solution disclosed herein, based on unambiguous segments, further identifies translatable segments and uses translatable and untranslatable segments as training samples to train a classification model. This approach can take into account more possibilities that may lead to differences in translation results. When the model is applied to the translation process, it can make the translation results more accurate and meet the requirements of high quality and low latency.

[0103] In one possible implementation, the translation method in this embodiment further includes:

[0104] If the classification result of the classification model is that it is untranslatable, a new source language segment is obtained, and the source language segment to be translated and the new source language segment are concatenated to obtain the current source language segment to be translated and input into the classification model.

[0105] If the current classification result indicates that the language segment to be translated is translatable, then the current source language segment to be translated will be translated.

[0106] If the current classification result is untranslatable, new source language fragments are continuously acquired and concatenated with the previous source language fragment to be translated before being input into the classification model, until the current classification result is translatable, at which point the current source language fragment to be translated is translated.

[0107] In practical applications, when the classification model determines that the source language segment to be translated is untranslatable (meaning it is not translatable), new source language segments are continuously acquired during the ongoing translation process. These new segments are then concatenated with the source language segment to be translated, resulting in the current source language segment to be translated. This concatenated segment is then input into the classification model, which determines whether the concatenated segment is translatable. If the current classification result indicates that the segment is translatable, it is translated. If the current classification result indicates that the segment is untranslatable, new source language segments are continuously acquired and concatenated with the previous segment to be translated, then input into the classification model. This process continues until the current classification result indicates that the segment is translatable, at which point it is translated.

[0108] In this embodiment of the disclosure, source language segments to be translated are continuously acquired, and the classification results of the classification model are used to determine whether to translate the source language segments to be translated or to continue acquiring new source language segments, splicing them together and making another judgment. This allows for continuous translation of streaming source language, meeting the high-quality and low-latency requirements of machine simultaneous interpretation.

[0109] Figure 7 This is a schematic diagram of a classification model training device according to an embodiment of this disclosure. Figure 7 As shown, the classification model training device may include:

[0110] The acquisition module 701 is used to acquire bilingual parallel corpora, which include source language corpora and target language corpora;

[0111] The determination module 702 is used to determine translatable and non-translatable segments in the source language corpus; translatable segments are determined based on unambiguous segments in the source language corpus.

[0112] Training module 703 is used to train a classification model based on translatable and non-translatable segments.

[0113] This disclosure provides a classification model training device that determines translatable and untranslatable segments from the source language corpus of a bilingual parallel corpus. The translatable segments are determined based on unambiguous segments in the source language corpus. Based on the translatable and untranslatable segments, a classification model is trained. Applying the trained classification model to translation can make the translation results more accurate and meet the requirements of high quality and low latency.

[0114] Figure 8 This is a schematic diagram illustrating the determination of a module in one embodiment of this disclosure. For example... Figure 8 As shown, in one possible implementation, the determining module includes a first determining unit 801 and a second determining unit 802;

[0115] The first determining unit 801 is used to determine multiple unambiguous segments in the source language corpus, and the first target language segment corresponding to each of the multiple unambiguous segments.

[0116] The second determining unit 802 is used to determine multiple translatable segments in the source language corpus based on multiple unambiguous segments and their corresponding first target language segments.

[0117] In one possible implementation, the second determining unit 802 is specifically used for:

[0118] For each unambiguous segment, obtain at least one source language sentence that begins with the unambiguous segment, translate the at least one source language sentence to obtain the target language sentence corresponding to each source language sentence;

[0119] Based on each unambiguous segment, the corresponding first target language segment for each unambiguous segment, and each target language sentence, multiple translatable segments in the source language corpus are identified.

[0120] In one possible implementation, the second determining unit 802, when determining multiple translatable segments in the source language corpus based on each unambiguous segment, the corresponding first target language segment, and each target language sentence, is used to:

[0121] For each unambiguous segment, obtain at least one second target language segment in the target language sentence that corresponds to the unambiguous segment;

[0122] Unambiguous segments that satisfy a preset matching condition between the first target language segment and at least one second target language segment are considered as translatable segments in the source language corpus.

[0123] In one possible implementation, the first determining unit 801 is specifically used for:

[0124] Align the source language words and target language words in the bilingual parallel corpus;

[0125] Based on the aligned source language words and target language words, multiple unambiguous segments in the source language corpus are identified, as well as the first target language segment corresponding to each of the multiple unambiguous segments.

[0126] In one possible implementation, when the first determining unit 801 determines multiple unambiguous segments in the source language corpus, and the first target language segment corresponding to each of the multiple unambiguous segments, based on the aligned source language words and target language words, it is used to:

[0127] Based on the aligned source language words and target language words, a source language fragment containing the source language words and at least one corresponding target language fragment are obtained;

[0128] The source language segment corresponding to a target language segment is taken as the unambiguous segment, and the target language segment corresponding to the unambiguous segment is taken as the first target language segment.

[0129] In one possible implementation, training module 703 is used for:

[0130] Obtain the category labels for translatable and non-translatable segments;

[0131] The translatable fragments, untranslatable fragments, and their corresponding category labels are used as the training sample set to train the classification model.

[0132] Figure 9 This is a schematic diagram of a translation device according to an embodiment of the present disclosure. Figure 9 As shown, the translation device includes:

[0133] Module 901 is used to acquire source language fragments to be translated;

[0134] The translation module 902 is used to input the source language fragment to be translated into the classification model, and translate the source language fragment to be translated if the classification result of the classification model is translatable.

[0135] The classification model is trained according to any one of the methods in the embodiments of this disclosure.

[0136] In one possible implementation, the translation module 902 is also used for:

[0137] If the classification result of the classification model is that it is untranslatable, a new source language segment is obtained, and the source language segment to be translated and the new source language segment are concatenated to obtain the current source language segment to be translated and input into the classification model.

[0138] If the current classification result indicates that the language segment to be translated is translatable, then the current source language segment to be translated will be translated.

[0139] If the current classification result is untranslatable, new source language fragments are continuously acquired and concatenated with the previous source language fragment to be translated before being input into the classification model, until the current classification result is translatable, at which point the current source language fragment to be translated is translated.

[0140] The functions of each unit, module, or sub-module in the various devices of this disclosure embodiment can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0141] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0142] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0143] At least one processor; and

[0144] The memory is communicatively connected to the at least one processor; wherein,

[0145] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods in any embodiment of this disclosure.

[0146] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods of any embodiment of this disclosure.

[0147] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods of any embodiment of this disclosure.

[0148] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0149] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0150] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0151] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as classification model training methods and translation methods. For example, in some embodiments, the classification model training methods and translation methods can be implemented as computer software programs, which are tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the classification model training methods and translation methods described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform classification model training methods or translation methods by any other suitable means (e.g., by means of firmware).

[0152] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0156] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0157] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0158] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0159] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A classification model training method, the method comprising: Obtain bilingual parallel corpora, which include source language corpora and target language corpora; The translatable and untranslatable segments in the source language corpus are determined; the translatable segments are determined based on unambiguous segments in the source language corpus. Based on the translatable and untranslatable segments, a classification model is trained; The step of determining the translatable segments in the source language corpus includes: Identify multiple unambiguous segments in the source language corpus, and the first target language segment corresponding to each of the multiple unambiguous segments; For each unambiguous segment, at least one source language sentence starting with the unambiguous segment is obtained, and the at least one source language sentence is translated to obtain the target language sentence corresponding to each source language sentence; wherein, the at least one source language sentence is a new sentence starting with the unambiguous segment obtained by inputting the unambiguous segment into a generative pre-trained model; For each unambiguous segment, obtain at least one second target language segment in the target language sentence that corresponds to the unambiguous segment; An unambiguous segment that satisfies a preset matching condition with the first target language segment and at least one second target language segment is taken as a translatable segment in the source language corpus; wherein, the preset matching condition is that the first target language segment matches the second target language segment in each target language sentence, or satisfies a preset matching degree threshold.

2. The method according to claim 1, wherein, The step of determining multiple unambiguous segments in the source language corpus, and the first target language segment corresponding to each of the multiple unambiguous segments, includes: Align the source language words and target language words in the bilingual parallel corpus; Based on the aligned source language words and target language words, multiple unambiguous segments in the source language corpus are determined, as well as the first target language segment corresponding to each of the multiple unambiguous segments.

3. The method according to claim 2, wherein, The step of determining multiple unambiguous segments in the source language corpus, and the first target language segment corresponding to each of the multiple unambiguous segments, based on the aligned source language words and target language words, includes: Based on the aligned source language words and target language words, a source language fragment containing the source language words and at least one corresponding target language fragment are obtained; The source language segment corresponding to a target language segment is taken as an unambiguous segment, and the target language segment corresponding to the unambiguous segment is taken as the first target language segment.

4. The method according to claim 1, wherein, The process of training a classification model based on the translatable and untranslatable segments includes: Obtain the category labels corresponding to the translatable and untranslatable segments respectively; The translatable fragments, untranslatable fragments, and their corresponding category labels are used as the training sample set to train the classification model.

5. A translation method, the method comprising: Obtain the source language fragment to be translated; The source language segment to be translated is input into a classification model. If the classification result of the classification model is translatable, the source language segment to be translated is translated. The classification model is trained using the method described in any one of claims 1-4.

6. The method according to claim 5, further comprising: If the classification result of the classification model is that it is untranslatable, a new source language segment is obtained, and the source language segment to be translated and the new source language segment are concatenated to obtain the current source language segment to be translated and input into the classification model. If the current classification result indicates that the language segment to be translated is translatable, then the current source language segment to be translated is translated. If the current classification result is untranslatable, new source language fragments are continuously acquired and concatenated with the previous source language fragment to be translated, and then input into the classification model until the current classification result is translatable, at which point the current source language fragment to be translated is translated.

7. A classification model training device, the device comprising: The acquisition module is used to acquire bilingual parallel corpora, which include source language corpora and target language corpora; A determination module is used to determine translatable and untranslatable segments in the source language corpus; the translatable segments are determined based on unambiguous segments in the source language corpus. The training module is used to train a classification model based on the translatable and untranslatable segments; The determining module includes a first determining unit and a second determining unit; The first determining unit is used to determine multiple unambiguous segments in the source language corpus, and a first target language segment corresponding to each of the multiple unambiguous segments; The second determining unit is used to determine multiple translatable segments in the source language corpus based on the multiple unambiguous segments and their respective corresponding first target language segments; The second determining unit is specifically used for: For each unambiguous segment, at least one source language sentence starting with the unambiguous segment is obtained, and the at least one source language sentence is translated to obtain the target language sentence corresponding to each source language sentence; wherein, the at least one source language sentence is a new sentence starting with the unambiguous segment obtained by inputting the unambiguous segment into a generative pre-trained model; Based on each unambiguous segment, the first target language segment corresponding to each unambiguous segment, and each target language sentence, multiple translatable segments in the source language corpus are determined; Wherein, when the second determining unit determines multiple translatable segments in the source language corpus based on each unambiguous segment, each unambiguous segment's corresponding first target language segment, and each target language sentence, it is used to: For each unambiguous segment, obtain at least one second target language segment in the target language sentence that corresponds to the unambiguous segment; An unambiguous segment that satisfies a preset matching condition with the first target language segment and at least one second target language segment is taken as a translatable segment in the source language corpus; wherein, the preset matching condition is that the first target language segment matches the second target language segment in each target language sentence, or satisfies a preset matching degree threshold.

8. The apparatus according to claim 7, wherein, The first determining unit is specifically used for: Align the source language words and target language words in the bilingual parallel corpus; Based on the aligned source language words and target language words, multiple unambiguous segments in the source language corpus are determined, as well as the first target language segment corresponding to each of the multiple unambiguous segments.

9. The apparatus according to claim 8, wherein, When the first determining unit determines multiple unambiguous segments in the source language corpus, and the first target language segment corresponding to each of the multiple unambiguous segments, based on the aligned source language words and target language words, it is used to: Based on the aligned source language words and target language words, a source language fragment containing the source language words and at least one corresponding target language fragment are obtained; The source language segment corresponding to a target language segment is taken as an unambiguous segment, and the target language segment corresponding to the unambiguous segment is taken as the first target language segment.

10. The apparatus according to claim 7, wherein, The training module is used for: Obtain the category labels corresponding to the translatable and untranslatable segments respectively; The translatable fragments, untranslatable fragments, and their corresponding category labels are used as the training sample set to train the classification model.

11. A translation apparatus, the apparatus comprising: The acquisition module is used to acquire source language fragments to be translated; The translation module is used to input the source language fragment to be translated into a classification model, and translate the source language fragment to be translated if the classification result of the classification model is translatable. The classification model is trained using the method described in any one of claims 1-4.

12. The apparatus according to claim 11, wherein, The translation module is also used for: If the classification result of the classification model is that it is untranslatable, a new source language segment is obtained, and the source language segment to be translated and the new source language segment are concatenated to obtain the current source language segment to be translated and input into the classification model. If the current classification result indicates that the language segment to be translated is translatable, then the current source language segment to be translated is translated. If the current classification result is untranslatable, new source language fragments are continuously acquired and concatenated with the previous source language fragment to be translated, and then input into the classification model until the current classification result is translatable, at which point the current source language fragment to be translated is translated.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images