Data processing method, device and electronic equipment

By performing more fine-grained encoding of source language text, the problem of semantic imbalance between Chinese and European languages ​​in machine translation is solved, the translation quality of sparse text is improved, and more accurate translation effect is achieved.

CN112926343BActive Publication Date: 2024-11-05BEIJING SOGOU TECHNOLOGY DEVELOPMENT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201911244080.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-12-06
Publication Date
2024-11-05
Estimated Expiration
2039-12-06

AI Technical Summary

Technical Problem

In machine translation, semantic imbalance between Chinese and European languages ​​leads to low translation quality of training sparse texts.

Method used

By performing finer granular encoding of source language text, including splitting Chinese characters with radicals as the smallest unit, encoding glyphs and spatial information respectively, and inputting the coded information into the machine translation model for translation.

Benefits of technology

The translation quality of sparse text by machine translation model is improved, and the capture and processing capabilities of Chinese character structure information are enhanced through finer granular coding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112926343B_ABST
    Figure CN112926343B_ABST
Patent Text Reader

Abstract

The embodiments of the present invention provide a data processing method, device and electronic device, wherein the method comprises: obtaining a source language text; encoding each character in the source language text according to text element information to obtain encoding information corresponding to the source language text; using a machine translation model to translate the source language text into a target language text according to the encoding information corresponding to the source language text; compared with the prior art that directly encodes the characters in the source language text, the embodiments of the present invention can encode the source language text in a more fine-grained manner, thereby improving the quality of translation by the machine translation model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device and electronic equipment. Background Art

[0002] Artificial intelligence includes a very broad range of sciences, consisting of different fields, such as machine learning, computer vision, etc. In general, one of the main goals of artificial intelligence research is to enable machines to perform some complex tasks that usually require human intelligence; since the birth of artificial intelligence, the theory and technology have become increasingly mature, and the application fields have continued to expand. For example, in the field of machine translation, for example, translating Chinese into English, translating English into Chinese, etc.

[0003] In the process of machine translation, Chinese characters are usually directly encoded and then translated based on the encoding results. However, compared with European languages ​​such as English, Chinese has more words and richer semantics, which makes Chinese and other languages ​​semantically unbalanced. For some sparsely trained characters, it is impossible to accurately translate them. Summary of the invention

[0004] An embodiment of the present invention provides a data processing method to improve the quality of machine translation.

[0005] Correspondingly, an embodiment of the present invention further provides a data processing device and an electronic device to ensure the implementation and application of the above method.

[0006] In order to solve the above problems, an embodiment of the present invention discloses a data processing method, which specifically includes: obtaining a source language text; encoding each character in the source language text according to text element information to obtain encoding information corresponding to the source language text; using a machine translation model to translate the source language text into a target language text according to the encoding information corresponding to the source language text.

[0007] Optionally, encoding each character in the source language text according to the text element information to obtain the encoding information corresponding to the source language text includes: encoding each character in the source language text according to the glyph of the character to obtain the encoding information corresponding to each character; splicing the encoding information corresponding to each character to generate the encoding information corresponding to the source language text.

[0008] Optionally, encoding each character in the source language text according to the glyph of the character to obtain coding information corresponding to each character includes: performing the following operations for each character in the source language text: splitting the character with radicals as the smallest unit; encoding each radical contained in the character separately to obtain glyph coding information corresponding to each radical; generating the coding information of the character based on the glyph coding information of each radical contained in the character.

[0009] Optionally, encoding each character in the source language text according to the glyph of the character to obtain coding information corresponding to each character also includes: after splitting the character based on radicals as the smallest unit, determining the spatial information of each radical contained in the character according to the glyph structure of the character; encoding the spatial information of each radical contained in the character to obtain spatial coding information corresponding to each radical; generating the coding information of the character based on the glyph coding information of each radical contained in the character, includes: using the glyph coding information of each radical contained in the character and the corresponding spatial coding information to form the coding information of the character.

[0010] Optionally, the glyph encoding information and corresponding spatial encoding information of each radical contained in the text are used to form the encoding information of the text, including: using the glyph encoding information and corresponding spatial encoding information of each radical contained in the text to form the encoding information corresponding to each radical; according to the order of each radical in the text, splicing the encoding information of each radical to obtain the encoding information of the text.

[0011] Optionally, the method further includes the step of training the machine translation model: obtaining training corpus, the training corpus including: a source language training text and a corresponding target language training text; encoding each character in the source language training text according to text element information to obtain encoding information corresponding to the source language training text; and training the machine translation model based on the encoding information corresponding to the source language training text and the target language training text.

[0012] The embodiment of the present invention also discloses a data processing device, which specifically includes: an acquisition module, used to acquire a source language text; an encoding module, used to encode each character in the source language text according to the text element information to obtain the encoding information corresponding to the source language text; and a translation module, used to use a machine translation model to translate the source language text into a target language text according to the encoding information corresponding to the source language text.

[0013] Optionally, the encoding module includes: a text encoding submodule, used to encode each character in the source language text according to the character shape to obtain the encoding information corresponding to each character; and a splicing submodule, used to splice the encoding information corresponding to each character to generate the encoding information corresponding to the source language text.

[0014] Optionally, the text encoding submodule includes: a splitting unit, used to split the text in the source language text based on radicals as the smallest unit; a glyph encoding unit, used to encode each radical contained in the text respectively to obtain glyph encoding information corresponding to each radical; and an information generation unit, used to generate encoding information of the text based on the glyph encoding information of each radical contained in the text.

[0015] Optionally, the text encoding submodule also includes: a spatial information determination unit, which is used to determine the spatial information of each radical contained in the text according to the glyph structure of the text after the text is split into the radical as the smallest unit; a spatial encoding unit, which is used to encode the spatial information of each radical contained in the text to obtain the spatial encoding information corresponding to each radical; and the information generation unit, which is used to use the glyph encoding information of each radical contained in the text and the corresponding spatial encoding information to form the encoding information of the text.

[0016] Optionally, the information generating unit is used to adopt the glyph encoding information and corresponding spatial encoding information of each radical contained in the text to form the encoding information corresponding to each radical; and to splice the encoding information of each radical according to the order of the radicals in the text to obtain the encoding information of the text.

[0017] Optionally, the device further includes: a training module, used to obtain training corpus, the training corpus including: a source language training text and a corresponding target language training text; encoding each character in the source language training text according to text element information to obtain encoding information corresponding to the source language training text; and training the machine translation model based on the encoding information corresponding to the source language training text and the target language training text.

[0018] An embodiment of the present invention further discloses a readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute a data processing method as described in any one of the embodiments of the present invention.

[0019] An embodiment of the present invention also discloses an electronic device, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations: obtaining a source language text; encoding each character in the source language text according to text element information to obtain encoding information corresponding to the source language text; and translating the source language text into a target language text using a machine translation model according to the encoding information corresponding to the source language text.

[0020] Optionally, encoding each character in the source language text according to the text element information to obtain the encoding information corresponding to the source language text includes: encoding each character in the source language text according to the glyph of the character to obtain the encoding information corresponding to each character; splicing the encoding information corresponding to each character to generate the encoding information corresponding to the source language text.

[0021] Optionally, encoding each character in the source language text according to the glyph of the character to obtain coding information corresponding to each character includes: performing the following operations for each character in the source language text: splitting the character with radicals as the smallest unit; encoding each radical contained in the character separately to obtain glyph coding information corresponding to each radical; generating the coding information of the character based on the glyph coding information of each radical contained in the character.

[0022] Optionally, encoding each character in the source language text according to the glyph of the character to obtain coding information corresponding to each character also includes: after splitting the character based on radicals as the smallest unit, determining the spatial information of each radical contained in the character according to the glyph structure of the character; encoding the spatial information of each radical contained in the character to obtain spatial coding information corresponding to each radical; generating the coding information of the character based on the glyph coding information of each radical contained in the character, includes: using the glyph coding information of each radical contained in the character and the corresponding spatial coding information to form the coding information of the character.

[0023] Optionally, the glyph encoding information and corresponding spatial encoding information of each radical contained in the text are used to form the encoding information of the text, including: using the glyph encoding information and corresponding spatial encoding information of each radical contained in the text to form the encoding information corresponding to each radical; according to the order of each radical in the text, splicing the encoding information of each radical to obtain the encoding information of the text.

[0024] Optionally, the method further comprises instructions for training the machine translation model as follows: obtaining training corpus, the training corpus comprising: a source language training text and a corresponding target language training text; encoding each character in the source language training text according to text element information to obtain encoding information corresponding to the source language training text; and training the machine translation model based on the encoding information corresponding to the source language training text and the target language training text.

[0025] The embodiments of the present invention include the following advantages:

[0026] In an embodiment of the present invention, a source language text can be obtained, and then each character in the source language text can be encoded according to the text element information to obtain the encoding information corresponding to the source language text, and then a machine translation model is used to translate the source language text into a target language text according to the encoding information corresponding to the source language text; compared with the prior art that directly encodes the characters in the source language text, the embodiment of the present invention can encode the source language text in a more fine-grained manner, thereby improving the quality of translation by the machine translation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of steps of an embodiment of a data processing method of the present invention;

[0028] Figure 2 is a flowchart of steps of an optional embodiment of a data processing method of the present invention;

[0029] Figure 3 is a flowchart of a method for training a machine translation model according to an embodiment of the present invention;

[0030] Figure 4 is a structural block diagram of an embodiment of a data processing device of the present invention;

[0031] Figure 5 is a structural block diagram of an optional embodiment of a data processing device of the present invention;

[0032] Figure 6 A structural block diagram of an electronic device for data processing according to an exemplary embodiment;

[0033] Figure 7 It is a structural schematic diagram of an electronic device for data processing according to another exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] One of the core concepts of the embodiments of the present invention is to encode text according to text element information and then input the encoded result into a machine translation model for translation, so as to encode the text in a more fine-grained manner and thus improve the quality of machine translation.

[0036] Reference Figure 1 , shows a flow chart of steps of a data processing method embodiment of the present invention, which may specifically include the following steps:

[0037] Step 102: Obtain source language text.

[0038] In the embodiment of the present invention, when a certain text needs to be translated, the text to be translated can be obtained. The language corresponding to the text to be translated can be called the source language, and the text to be translated can be called the source language text. Then, steps 104 to 106 can be executed to translate the source language text.

[0039] Step 104: Encode each character in the source language text according to the character element information to obtain encoding information corresponding to the source language text.

[0040] Step 106: Use a machine translation model to translate the source language text into a target language text according to the encoding information corresponding to the source language text.

[0041] In an embodiment of the present invention, a method of translating a source language text may be to encode the text in the source language text; then input the encoding result into a machine translation model, and the machine translation model translates based on the encoding result, thereby translating the source language text into a target language text. The target language and the source language are different languages.

[0042] Among them, in order to improve the translation quality of the machine translation model, in the process of encoding the characters in the source language text, each character in the source language text can be encoded according to the text element information of each character in the source language text; and then the encoding information corresponding to the source language text can be obtained. Then the encoding information corresponding to the source language text can be input into the trained machine translation model (the training process will be described later), and the trained machine translation model will translate and output the target language text. In the embodiment of the present invention, the text element information can include multiple elements of the text, such as glyphs, pronunciations, etc., and the embodiment of the present invention does not limit this; therefore, compared with the prior art that directly encodes the characters in the source language text, the embodiment of the present invention can encode the source language text in a more fine-grained manner, thereby improving the translation quality of the machine translation model.

[0043] In summary, in the embodiments of the present invention, a source language text can be obtained, and then each character in the source language text can be encoded according to the text element information to obtain the encoding information corresponding to the source language text, and then a machine translation model is used to translate the source language text into a target language text according to the encoding information corresponding to the source language text; compared with the prior art that directly encodes the characters in the source language text, the embodiments of the present invention can encode the source language text in a more fine-grained manner, thereby improving the quality of translation by the machine translation model.

[0044] In an optional embodiment of the present invention, the text element information may include a glyph; each character in the source language text may be encoded according to the glyph, as follows:

[0045] Reference Figure 2 , shows a flow chart of steps of an optional embodiment of a data processing method of the present invention, which may specifically include the following steps:

[0046] Step 202: Obtain source language text.

[0047] In the embodiment of the present invention, the source language may refer to a language whose corresponding text is an ideographic language, such as Chinese, Japanese, etc., and the embodiment of the present invention does not limit this; wherein the ideographic language may refer to a writing system that uses symbolic writing symbols to record information, and does not directly or simply represent speech. The source language text may be a character, a word, or one or more sentences, and the embodiment of the present invention does not limit this.

[0048] Step 204: Encode each character in the source language text according to the character's glyph to obtain encoding information corresponding to each character.

[0049] The following description is made by taking the source language as Chinese and the text in the source language text as Chinese characters as an example. Among them, step 204 may include the following steps (204-2)-(204-6):

[0050] Among them, steps (204-2)-(204-6) are operations performed on each character in the source language text.

[0051] Step 204-2: split the characters into parts based on radicals as the smallest unit.

[0052] Step 204-4: Encode each radical contained in the text to obtain the glyph encoding information corresponding to each radical.

[0053] Step 204-6: Generate encoding information of the text according to the glyph encoding information of each radical contained in the text.

[0054] In the embodiments of the present invention, from the perspective of the glyphs of Chinese characters, Chinese characters can be divided into three types of radicals: semantic radicals (rational radicals constructed according to meaning), phonetic radicals (rational radicals constructed according to sound), and accessory radicals (irrational radicals without basis). Among them, a Chinese character with only one radical is called a single-structured character, and a Chinese character with more than one radical is called a compound-structured character. Therefore, the Chinese character can be split with the radical as the smallest unit; the Chinese character is split into one or more radicals. Then, each radical included in the Chinese character can be encoded to obtain the glyph encoding information corresponding to each radical included in the Chinese character. Optionally, each radical can be encoded with the five-stroke encoding or the root encoding of the Zhengma encoding to obtain the glyph encoding information of each radical. Among them, when the five-stroke encoding or the root encoding of the Zhengma encoding corresponding to different radicals is the same, different identifiers can be added to different radicals, and the identifier can be used to uniquely identify a radical to distinguish different radicals with the same glyph encoding information. Then, according to the glyph encoding information of each radical included in the Chinese character, the encoding information of the Chinese character is generated. Among them, in one way, according to the order of each radical in the Chinese character, the glyph encoding information of each radical included in the Chinese character is spliced to obtain the encoding information corresponding to the Chinese character. Among them, the order of each radical in the Chinese character can refer to the writing order of each radical of the Chinese character when writing the Chinese character.

[0055] As an example of the present invention, for example, for the Chinese character "她" (she), splitting the Chinese character with the radical as the smallest unit, the radicals "女" (female) and "也" (also) can be obtained. Then, "女" and "也" are encoded respectively to obtain the glyph encoding information "gdi_0" of "女" and the glyph encoding information "rnb_0" of "也". Then, the glyph encoding information "gdi_0" of "女" and the glyph encoding information "rnb_0" of "也" are spliced to obtain the encoding information corresponding to the Chinese character "她" as: {gdi_0, rnb_0}. Another example is the Chinese character "他" (he). Splitting the Chinese character with the radical as the smallest unit, the radicals "亻" (person) and "也" (also) can be obtained. Then, "亻" and "也" are encoded respectively to obtain the glyph encoding information "gdi_1" of "亻" and the glyph encoding information "rnb_0" of "也". Then, the glyph encoding information "gdi_1" of "亻" and the glyph encoding information "rnb_0" of "也" are spliced to obtain the encoding information corresponding to the Chinese character "他" as: {gdi_1, rnb_0}.

[0056] In one embodiment of the present invention, the structures of Chinese characters can be divided into two categories: the structures corresponding to single-component characters (which can be called single-component structures) and the structures corresponding to compound characters. Among them, the structures corresponding to compound characters can include various types, such as the upper-lower structure, the left-right structure, and so on. For compound characters, encoding the radicals cannot reflect the spatial information of the radicals in compound characters of different structures. Therefore, spatial encoding of the radicals can also be performed. Correspondingly, when encoding each character in the source language text according to the glyph of the character to obtain the encoding information corresponding to each character, it further includes: after splitting the character with the radical as the smallest unit in step 204-2, determining the spatial information of each radical included in the character according to the glyph structure of the character; encoding the spatial information of each radical included in the character to obtain the spatial encoding information corresponding to each radical; when generating the encoding information of the character according to the glyph encoding information of each radical included in the character, it includes: using the glyph encoding information and the corresponding spatial encoding information of each radical included in the character to form the encoding information of the character.

[0057] In the embodiment of the present invention, after splitting the character with the radical as the smallest unit, the spatial information of each radical included in the character can be determined according to the glyph structure of the character. Among them, the spatial information can include: empty, left, middle, right, upper, lower, upper in the character structure of three characters arranged one above the other, lower left in the character structure of three characters arranged one above the other, lower right in the character structure of three characters arranged one above the other, fully enclosed, semi-enclosed, and internal. Among them, the empty can refer to the spatial information of single-component characters. The left, middle, right, upper, and lower can refer to the spatial information of the radicals in the characters corresponding to the upper-lower structure, the upper-middle-lower structure, the left-right structure, and the left-middle-right structure. The upper in the character structure of three characters arranged one above the other, the lower left in the character structure of three characters arranged one above the other, and the lower right in the character structure of three characters arranged one above the other can refer to the spatial information of the radicals in the characters corresponding to the character structure of three characters arranged one above the other (such as pin and sen). The fully enclosed and internal can refer to the spatial information of the radicals in the characters corresponding to the fully enclosed structure (such as yuan). The semi-enclosed and internal can refer to the spatial information of the radicals in the characters corresponding to the semi-enclosed structure (such as qing).

[0058] Then, the spatial information of each radical included in the character can be encoded to obtain the spatial encoding information corresponding to each radical. For example, for the Chinese character "ta", the spatial encoding information "kdi_1" of the radical "female" and the spatial encoding information "kdi_3" of the radical "ye" are corresponding. Among them, the spatial encoding information corresponding to different spatial information is different. Of course, the spatial information can also be encoded in advance, and then a word list of the spatial information and the corresponding spatial encoding information is established. Furthermore, when encoding the spatial information of each radical included in the character to obtain the spatial encoding information corresponding to each radical, the word list can be searched according to the spatial information of each radical to determine the spatial encoding information corresponding to each radical; the embodiment of the present invention does not limit this.

[0059] Among them, the encoding information of the character is formed by using the glyph encoding information and the corresponding spatial encoding information of each radical contained in the character; it may include the following sub-steps:

[0060] Sub-step 22: Use the glyph encoding information and the corresponding spatial encoding information of each radical contained in the character to form the encoding information corresponding to each radical.

[0061] Sub-step 24: According to the order of the radicals in the character, splice the encoding information of each radical to obtain the encoding information of the character.

[0062] In the embodiment of the present invention, for each radical contained in the character, the encoding information corresponding to the radical can be formed by using the glyph encoding information and the corresponding spatial encoding information of the radical. Among them, there are various ways to form the encoding information corresponding to the radical by using the glyph encoding information and the corresponding spatial encoding information of the radical. For example, the spatial encoding information of the radical is spliced after the glyph encoding information of the radical; for another example, the spatial encoding information of the radical is used as the superscript or subscript of the glyph encoding information of the radical, etc. The embodiment of the present invention does not limit this.

[0063] Then, according to the order of the radicals in the character, the encoding information of each radical can be spliced to obtain the encoding information of the character. For example, for the Chinese character "她" (she), the corresponding radicals include "女" (female) and "也" (also). Among them, the glyph encoding information corresponding to "女" is "gdi_0" and the spatial encoding information is "kdi_1"; the encoding information corresponding to "女" can be obtained as {gdi_0, kdi_1}. The glyph encoding information corresponding to "也" is "rnb_0" and the spatial encoding information is "kdi_3"; the encoding information corresponding to "也" can be obtained as {rnb_0, kdi_3}. Then, the encoding information of "女" and "也" is spliced to obtain the encoding information of "她" as {gdi_0, kdi_1, rnb_0, kdi_3}. Correspondingly, for the Chinese character "他" (he), the corresponding encoding information can be {gdi_1, kdi_1, rnb_0, kdi_3}.

[0064] In the prior art, direct encoding using text is adopted. For example, for the Chinese character "她 (tā)", direct encoding is performed to obtain encoding information X, and for the Chinese character "他 (tā)", direct encoding is performed to obtain encoding information Y. Whether the encoding information of a character is generated using the glyph encoding information of radicals or using the glyph encoding information of radicals and the corresponding spatial encoding information, the encoding information obtained in the embodiments of the present invention has a finer granularity than the encoding information obtained in the prior art. In addition, among the encoding information {gdi_0, kdi_1, rnb_0, kdi_3} of "她 (tā)" and the corresponding encoding information {gdi_1, kdi_1, rnb_0, kdi_3} of "他 (tā)", the last three encodings "kdi_1, rnb_0, kdi_3" are the same; thus, for sparsely trained characters, the machine translation model can find the correlation between the sparsely trained characters and the trained characters based on the encoding information, improving the translation quality of the sparse characters.

[0065] Certainly, in the embodiments of the present invention, the source language corpus may also be obtained in advance, and then for each character in the source language corpus, according to the method described in step 204, the encoding information corresponding to each character in the source language corpus is generated; then a vocabulary table of each character and the corresponding encoding information is established. Further, when step 204 is executed, the vocabulary table can be directly searched based on each character in the source language text to determine the encoding information corresponding to each character in the source language text; the embodiments of the present invention do not limit this.

[0066] Step 206: Concatenate the encoding information corresponding to each character to generate the encoding information corresponding to the source language text.

[0067] One implementation manner of the embodiments of the present invention may be to determine the order of each character in the source language file; then concatenate the encoding information corresponding to each character in the order of each character to generate the encoding information corresponding to the source language text.

[0068] Step 208: Use a machine translation model to translate the source language text into a target language text based on the encoding information corresponding to the source language text.

[0069] Then the encoding information of the source language text can be input into the machine translation model, and the machine translation model translates the source language text into a target language text based on the encoding information of the source language text.

[0070] In summary, in an embodiment of the present invention, after obtaining a source language text, each character in the source language text can be encoded according to the glyph of the character to obtain encoding information corresponding to each character, and then the encoding information corresponding to each character can be spliced ​​to generate the encoding information corresponding to the source language text; wherein, the character can be split with radicals as the smallest unit, and each radical contained in the character can be encoded respectively to obtain glyph encoding information corresponding to each radical, and the encoding information of the character can be generated based on the glyph encoding information of each radical contained in the character; since different characters may contain the same radical, for sparsely trained characters, the machine translation model can find the correlation between sparsely trained characters and trained characters based on the encoding information, thereby improving the translation quality of sparsely trained characters.

[0071] Secondly, in an embodiment of the present invention, after the text is split into radicals as the smallest unit, the spatial information of each radical contained in the text can be determined according to the glyph structure of the text, and the spatial information of each radical contained in the text can be encoded to obtain the spatial coding information corresponding to each radical; then, the glyph coding information and the corresponding spatial coding information of each radical contained in the text are used to form the coding information of the text; and then the text coding is further refined, thereby further improving the translation quality of sparse training data.

[0072] In an optional embodiment of the present invention, the text element information may also include pronunciation information. Correspondingly, another method of encoding each character in the source language text according to the text element information to obtain the encoding information corresponding to the source language text may be: encoding each character in the source language text according to the pronunciation of the character to obtain the encoding information corresponding to the source language text. Among them, each character in the source language text may be annotated with pronunciation to obtain the pronunciation sequence corresponding to each character; then the pronunciation sequence is encoded to obtain the encoding information corresponding to each character. Then the encoding information corresponding to each character is spliced ​​to generate the encoding information corresponding to the source language text. Of course, each character in the source language corpus may also be annotated with pronunciation in advance, and then the pronunciation sequence of each character is encoded to obtain the corresponding encoding information; then a vocabulary of each character and the corresponding encoding information is established. Then, when encoding each character in the source language text according to the pronunciation of the character to obtain the encoding information corresponding to the source language text, the vocabulary may be searched according to each character in the source language text to determine the encoding information corresponding to each character. Compared with the above-mentioned vocabulary generated by encoding based on the glyphs of the characters, the vocabulary generated by encoding based on the pronunciation of the characters has less data volume, thereby improving the efficiency of determining the encoding information corresponding to the source language text.

[0073] In another embodiment of the present invention, the machine translation model may be trained in advance using training corpus, so that the machine translation model learns how to accurately translate the encoded information obtained by encoding the text element information. The training process of the machine translation model is described below.

[0074] Reference Figure 3 , shows a flowchart of a method for training a machine translation model according to an embodiment of the present invention, which may specifically include the following steps:

[0075] Step 302: Obtain training corpus, wherein the training corpus includes: a source language training text and a corresponding target language training text;

[0076] Step 304: Encode each character in the source language training text according to the text element information to obtain encoding information corresponding to the source language training text.

[0077] Step 306: Train the machine translation model according to the encoding information corresponding to the source language training text and the target language training text.

[0078] In the embodiment of the present invention, training corpus can be collected; the training corpus includes: source language training text and corresponding target language training text, wherein the source language training text may include multiple pieces, and the corresponding target training text may also include multiple pieces; a target language training text corresponding to a source language training text may be a target language text obtained by translating the source language training text. Then, each character in the source language training text is encoded according to the text element information to obtain the encoding information corresponding to the source language training text; the specific details are similar to steps 204 to 206, and will not be repeated here.

[0079] Then, the machine translation model can be trained based on the encoding information corresponding to the source language training text and the target language training text, so that the machine translation model can learn how to accurately translate the encoding information obtained by encoding according to the text element information.

[0080] Among them, taking the encoding information corresponding to a source language training text and the corresponding target language training text as an example, the training process of the machine translation model is explained. Forward training: the encoding information corresponding to the source language training text can be input into the machine translation model to obtain multiple target language texts and corresponding probabilities output by the machine translation model. Reverse training: Determine whether the target language text with the highest probability output by the machine translation model matches the target language training text corresponding to the source language training text. If not, adjust the weight of the machine translation model until the encoding information corresponding to the source language training text is input, and the target language text with the highest probability output by the machine translation model matches the target language training text corresponding to the source language training text.

[0081] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0082] Reference Figure 4 , shows a structural block diagram of a data processing device embodiment of the present invention, which may specifically include the following modules:

[0083] An acquisition module 402 is used to acquire a source language text;

[0084] The encoding module 404 is used to encode each character in the source language text according to the character element information to obtain encoding information corresponding to the source language text;

[0085] The translation module 406 is used to translate the source language text into the target language text by using a machine translation model according to the encoding information corresponding to the source language text.

[0086] Reference Figure 5 , showing a structural block diagram of an optional embodiment of a data processing device of the present invention.

[0087] In an optional embodiment of the present invention, the encoding module 404 includes:

[0088] The character encoding submodule 4042 is used to encode each character in the source language text according to the character shape to obtain the encoding information corresponding to each character;

[0089] The concatenation submodule 4044 is used to concatenate the encoding information corresponding to each character to generate the encoding information corresponding to the source language text.

[0090] In an optional embodiment of the present invention, the text encoding submodule 4042 includes:

[0091] A splitting unit 40422 is used to split each character in the source language text using radicals as the smallest unit.

[0092] The glyph encoding unit 40424 is used to encode each radical contained in the character to obtain glyph encoding information corresponding to each radical;

[0093] The information generating unit 40426 is used to generate the encoding information of the text according to the glyph encoding information of each radical contained in the text.

[0094] In an optional embodiment of the present invention, the text encoding submodule 4042 further includes:

[0095] A spatial information determining unit 40428, configured to determine the spatial information of each radical contained in the character according to the glyph structure of the character after the character is split into the radical as the smallest unit;

[0096] A spatial encoding unit 404210, configured to encode the spatial information of each radical contained in the character to obtain spatial encoding information corresponding to each radical;

[0097] The information generating unit 40426 is used to form the coding information of the text by using the glyph coding information of each radical contained in the text and the corresponding spatial coding information.

[0098] In an optional embodiment of the present invention, the information generating unit 40426 is used to adopt the glyph encoding information and the corresponding spatial encoding information of each radical contained in the text to form the encoding information corresponding to each radical; and to splice the encoding information of each radical according to the order of the radicals in the text to obtain the encoding information of the text.

[0099] In an optional embodiment of the present invention, the device further comprises:

[0100] The training module 408 is used to obtain training corpus, which includes: a source language training text and a corresponding target language training text; encode each character in the source language training text according to text element information to obtain encoding information corresponding to the source language training text; and train the machine translation model based on the encoding information corresponding to the source language training text and the target language training text.

[0101] In summary, in the embodiments of the present invention, a source language text can be obtained, and then each character in the source language text can be encoded according to the text element information to obtain the encoding information corresponding to the source language text, and then a machine translation model is used to translate the source language text into a target language text according to the encoding information corresponding to the source language text; compared with the prior art that directly encodes the characters in the source language text, the embodiments of the present invention can encode the source language text in a more fine-grained manner, thereby improving the quality of translation by the machine translation model.

[0102] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0103] Figure 6 1 is a block diagram of an electronic device 600 for data processing according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0104] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0105] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0106] The memory 604 is configured to store various types of data to support operations on the device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0107] The power component 606 provides power to the various components of the electronic device 600. The power component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0108] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0109] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0110] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0111] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the device 600, the relative positioning of components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0112] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 614 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 614 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0113] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0114] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0115] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a data processing method, the method comprising: obtaining a source language text; encoding each character in the source language text according to character element information to obtain encoding information corresponding to the source language text; using a machine translation model to translate the source language text into a target language text according to the encoding information corresponding to the source language text.

[0116] Optionally, encoding each character in the source language text according to the text element information to obtain the encoding information corresponding to the source language text includes: encoding each character in the source language text according to the glyph of the character to obtain the encoding information corresponding to each character; splicing the encoding information corresponding to each character to generate the encoding information corresponding to the source language text.

[0117] Optionally, encoding each character in the source language text according to the glyph of the character to obtain coding information corresponding to each character includes: performing the following operations for each character in the source language text: splitting the character with radicals as the smallest unit; encoding each radical contained in the character separately to obtain glyph coding information corresponding to each radical; generating the coding information of the character based on the glyph coding information of each radical contained in the character.

[0118] Optionally, encoding each character in the source language text according to the glyph of the character to obtain coding information corresponding to each character also includes: after splitting the character based on radicals as the smallest unit, determining the spatial information of each radical contained in the character according to the glyph structure of the character; encoding the spatial information of each radical contained in the character to obtain spatial coding information corresponding to each radical; generating the coding information of the character based on the glyph coding information of each radical contained in the character, includes: using the glyph coding information of each radical contained in the character and the corresponding spatial coding information to form the coding information of the character.

[0119] Optionally, the glyph encoding information and corresponding spatial encoding information of each radical contained in the text are used to form the encoding information of the text, including: using the glyph encoding information and corresponding spatial encoding information of each radical contained in the text to form the encoding information corresponding to each radical; according to the order of each radical in the text, splicing the encoding information of each radical to obtain the encoding information of the text.

[0120] Optionally, the method further includes the step of training the machine translation model: obtaining training corpus, the training corpus including: a source language training text and a corresponding target language training text; encoding each character in the source language training text according to text element information to obtain encoding information corresponding to the source language training text; and training the machine translation model based on the encoding information corresponding to the source language training text and the target language training text.

[0121] Figure 7 : is a structural diagram of an electronic device 700 for data processing according to another exemplary embodiment of the present invention. The electronic device 700 may be a server, which may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 722 (for example, one or more processors) and a memory 732, and one or more storage media 730 (for example, one or more mass storage devices) storing application programs 742 or data 744. Among them, the memory 732 and the storage medium 730 may be temporary storage or permanent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processor 722 may be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the server.

[0122] The server may also include one or more power supplies 726, one or more wired or wireless network interfaces 750, one or more input and output interfaces 758, one or more keyboards 756, and / or one or more operating systems 741, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0123] An electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: obtaining a source language text; encoding each character in the source language text according to character element information to obtain encoding information corresponding to the source language text; and translating the source language text into a target language text using a machine translation model according to the encoding information corresponding to the source language text.

[0124] Optionally, encoding each character in the source language text according to the text element information to obtain the encoding information corresponding to the source language text includes: encoding each character in the source language text according to the glyph of the character to obtain the encoding information corresponding to each character; splicing the encoding information corresponding to each character to generate the encoding information corresponding to the source language text.

[0125] Optionally, encoding each character in the source language text according to the glyph of the character to obtain coding information corresponding to each character includes: performing the following operations for each character in the source language text: splitting the character with radicals as the smallest unit; encoding each radical contained in the character separately to obtain glyph coding information corresponding to each radical; generating the coding information of the character based on the glyph coding information of each radical contained in the character.

[0126] Optionally, encoding each character in the source language text according to the glyph of the character to obtain coding information corresponding to each character also includes: after splitting the character based on radicals as the smallest unit, determining the spatial information of each radical contained in the character according to the glyph structure of the character; encoding the spatial information of each radical contained in the character to obtain spatial coding information corresponding to each radical; generating the coding information of the character based on the glyph coding information of each radical contained in the character, includes: using the glyph coding information of each radical contained in the character and the corresponding spatial coding information to form the coding information of the character.

[0127] Optionally, the glyph encoding information and corresponding spatial encoding information of each radical contained in the text are used to form the encoding information of the text, including: using the glyph encoding information and corresponding spatial encoding information of each radical contained in the text to form the encoding information corresponding to each radical; according to the order of each radical in the text, splicing the encoding information of each radical to obtain the encoding information of the text.

[0128] Optionally, the method further comprises instructions for training the machine translation model as follows: obtaining training corpus, the training corpus comprising: a source language training text and a corresponding target language training text; encoding each character in the source language training text according to text element information to obtain encoding information corresponding to the source language training text; and training the machine translation model based on the encoding information corresponding to the source language training text and the target language training text.

[0129] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0130] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0134] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0135] The data processing method, the data processing device and the electronic device provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and the scope of application. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data processing method, characterized in that: including: Obtain the source language text; Encode each character in the source language text according to the character element information to obtain the encoded information corresponding to the source language text, including: encode each character in the source language text according to the glyph of the character to obtain the encoded information corresponding to each character; splice the encoded information corresponding to each character to generate the encoded information corresponding to the source language text; the encoded information corresponding to each character includes: the glyph encoded information corresponding to each character and the spatial encoded information of the glyph corresponding to each character; Use a machine translation model to translate the source language text into a target language text according to the encoded information corresponding to the source language text; The encoding each character in the source language text according to the glyph of the character to obtain the encoded information corresponding to each character includes: Perform the following operations on each character in the source language text: Split the character with the radical as the smallest unit; Encode each radical included in the character respectively to obtain the glyph encoded information corresponding to each radical; Determine the spatial information of each radical included in the character according to the glyph structure of the character, where the spatial information includes empty, left, middle, right, upper, lower, upper in the character pattern, lower left in the character pattern, lower right in the character pattern, full package, half package, and inside; Encode the spatial information of each radical included in the character to obtain the spatial encoded information corresponding to each radical; Use the glyph encoded information and the corresponding spatial encoded information of each radical included in the character to form the encoded information of the character.

2. The method according to claim 1, characterized in that The using the glyph encoded information and the corresponding spatial encoded information of each radical included in the character to form the encoded information of the character includes: Use the glyph encoded information and the corresponding spatial encoded information of each radical included in the character to form the encoded information corresponding to each radical; Splice the encoded information of each radical in the order of the radicals in the character to obtain the encoded information of the character.

3. The method according to claim 1, characterized in that The method further includes the step of training the machine translation model: Obtain training corpus, the training corpus includes: source language training text and corresponding target language training text; Encode each character in the source language training text according to the text element information to obtain the encoded information corresponding to the source language training text; Train the machine translation model according to the encoded information corresponding to the source language training text and the target language training text.

4. A data processing device, characterized in that: including: An obtaining module, configured to obtain the source language text; An encoding module, configured to encode each character in the source language text according to the character element information to obtain the encoded information corresponding to the source language text; The encoding module includes: a character encoding sub-module, configured to encode each character in the source language text according to the glyph of the character to obtain the encoded information corresponding to each character; a splicing sub-module, configured to splice the encoded information corresponding to each character to generate the encoded information corresponding to the source language text; the glyph encoded information corresponding to each character and the spatial encoded information of the glyph corresponding to each character; A translation module for translating the source language text into a target language text by using a machine translation model based on the encoding information corresponding to the source language text; The character encoding sub-module includes: A splitting unit for splitting each character in the source language text with the radical as the smallest unit; A glyph encoding unit for encoding each radical included in the character to obtain the glyph encoding information corresponding to each radical; A spatial information determination unit for determining the spatial information of each radical included in the character according to the glyph structure of the character, wherein the spatial information includes empty, left, middle, right, upper, lower, upper in a character pattern, lower left in a character pattern, lower right in a character pattern, fully enclosed, semi-enclosed, and internal; A spatial encoding unit for encoding the spatial information of each radical included in the character to obtain the spatial encoding information corresponding to each radical; An information generation unit for using the glyph encoding information and the corresponding spatial encoding information of each radical included in the character to form the encoding information of the character.

5. The apparatus according to claim 4, wherein: The information generation unit is configured to use the glyph encoding information and the corresponding spatial encoding information of each radical included in the character to form the encoding information corresponding to each radical; and splice the encoding information of each radical in the order of the radicals in the character to obtain the encoding information of the character.

6. The device according to claim 4, characterized in that The apparatus further includes: A training module for obtaining training corpora, the training corpora including: source language training texts and corresponding target language training texts; encoding each character in the source language training texts according to text element information to obtain the encoding information corresponding to the source language training texts; and training the machine translation model according to the encoding information corresponding to the source language training texts and the target language training texts.

7. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data processing method according to any one of method claims 1-3.

8. An electronic device, characterized in that: It includes a memory, and one or more programs, wherein one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the data processing method according to any one of method claims 1-3.