Multi-language mixed text translation method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a multi-language mixed text translation method, which comprises the following steps: acquiring an input text, and converting the input text into a UTF-8 code; splicing the prompt text of the target language and the input text converted into the UTF-8 code to obtain a spliced text, and converting the spliced text into the UTF-8 code; training a ByT5 model on the basis of the input text which is converted into the UTF-8 code and the spliced text to obtain a pre-trained ByT5 model; and inputting a to-be-processed multi-language mixed text into the pre-trained ByT5 model to generate translation output of a target language. The invention further provides a multi-language mixed text translation device, computer equipment and a storage medium. According to the method, language texts used by most countries in the world can be supported, the problem that a traditional machine translation model is limited by token coding is solved, and a single model can flexibly process a translation task of inputting many-to-many bilingual texts into single-language texts and a translation task of inputting multi-language mixed texts.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multi-language mixed text translation method, device, computer equipment and storage medium. Background Art

[0002] In recent years, machine translation aims to solve the translation tasks of single-to-single or single-to-many single-language input. As people interact more in areas such as social media, news reports, and business exchanges, the mixed use of multiple languages ​​in the same text has become common. The concept of multilingual mixed text has emerged. For example, on social media, people often use multiple languages ​​to express their opinions in posts or comments. Machine translation of multilingual mixed text can promote cross-language communication and understanding, enhance the user experience of multilingual social media, broaden the scope of information acquisition, and promote cross-border business cooperation. Therefore, the development and improvement of multilingual mixed text machine translation technology is of great significance to meet the language needs of modern society and promote cross-cultural communication.

[0003] In the prior art, examples of many-to-many single language input are as follows:

[0004] Input 1 (English): "The latest technology trends are shaping the future."

[0005] Input 2 (French): "Les dernières tendances technologiquesfa onnent l'avenir."

[0006] Input 3 (Chinese): "The latest technology trends are shaping the future."

[0007] Enter 4 (Spanish): "Lasúltimas tendencias tecnológicas están dando formaal futuro."

[0008] In the many-to-many single-language input above, the input text is in a single different language and the translated output will be in the specified target language.

[0009] The following are examples of multi-language mixed text input:

[0010] Input (mixed language): "The future of technology is bright. Latechnologie dedemain sera incroyable. The future of technology will be bright"

[0011] In the above example of multilingual mixed text input, the input text combines English, French and Chinese.

[0012] Traditional manual translation has disadvantages such as high cost and poor real-time performance, and is only applicable to a small number of scenarios. Modern machine translation models convert input sentences into sentences in another language through neural networks. The current mainstream neural machine translation uses an "encoder-decoder" structure, where the encoder encodes the source language sentence into a vector in a continuous space, and the decoder generates a sentence in the target language based on this vector. However, for the translation task of multilingual mixed text input, there are currently few systems that support the translation of multilingual mixed text input because it involves the interweaving of different language structures, grammar, vocabulary, and culture. Dealing with this situation requires taking into account the context of language switching to ensure that the translation result is both accurate and natural, which is a very complex task. Summary of the invention

[0013] The purpose of the embodiments of the present application is to propose a multi-language mixed text translation method, apparatus, computer equipment and storage medium, the main purpose of which is to improve the efficiency and accuracy of multi-language mixed text translation.

[0014] In order to solve the above technical problems, the present application embodiment provides a multi-language mixed text translation method, which adopts the following technical solution:

[0015] A multi-language mixed text translation method comprises the following steps:

[0016] Get the input text and convert it into UTF-8 encoding;

[0017] Concatenate the prompt text in the target language with the input text converted into UTF-8 encoding to obtain a concatenated text, and convert the concatenated text into UTF-8 encoding;

[0018] Training the ByT5 model based on the input text converted into UTF-8 encoding and the concatenated text to obtain a pre-trained ByT5 model;

[0019] The multilingual mixed text to be processed is input into the pre-trained ByT5 model to generate a translation output in the target language.

[0020] Furthermore, the step of splicing the prompt text in the target language with the input text converted into UTF-8 encoding to obtain the spliced ​​text specifically includes:

[0021] Convert the prompt text in the target language into UTF-8 encoding. The prompt text is used to prompt the pre-trained ByT5 model to translate the input text into the target language.

[0022] The prompt text converted into UTF-8 encoding is used as a prefix and concatenated with the input text converted into UTF-8 encoding.

[0023] Further, the step of training the ByT5 model based on the input text converted into UTF-8 encoding and the concatenated text to obtain a pre-trained ByT5 model specifically includes:

[0024] The input text converted to UTF-8 encoding is used as the input of the ByT5 model;

[0025] Using an encoder of the ByT5 model to extract positional relationship features of the input text;

[0026] Using the prompt text in the target language in the concatenated text converted into UTF-8 encoding as the target language of the decoder, and inputting the positional relationship features of the input text into the decoder;

[0027] The positional relationship features of the input text are trained according to the target language of the decoder to obtain a pre-trained ByT5 model.

[0028] Furthermore, training the ByT5 model using the target language corresponding to the generated prompt text as output also includes:

[0029] The parameters of the ByT5 model were optimized by gradient descent.

[0030] Furthermore, after the step of inputting the multilingual mixed text to be processed into the pre-trained ByT5 model to generate the translation output in the target language, the method further includes:

[0031] Furthermore, after the step of inputting the multilingual mixed text to be processed into the pre-trained ByT5 model to generate the translation output in the target language, the method further includes:

[0032] Learning the feature relationship between the multilingual mixed text to be processed and the language of the translated output, and mapping the feature relationship to a shared feature space;

[0033] The ByT5 model is optimized using the characteristic relationship evaluation.

[0034] Furthermore, the learning of the feature relationship between the multilingual mixed text to be processed and the language of the translated output and mapping the feature relationship to a shared feature space specifically includes:

[0035] Learning the lexical features and syntactic features of the multilingual mixed text to be processed and the language of the translated output;

[0036] Vectorizing the lexical features and syntactic features;

[0037] The vectorized lexical features and syntactic features are mapped to a shared feature space for storage, and then the feature relationship is used to evaluate and optimize the ByT5 model.

[0038] Furthermore, the use of the characteristic relationship to evaluate and optimize the ByT5 model specifically includes:

[0039] Compare the similarity of lexical features and syntactic features of different multilingual mixed texts to be processed and the language of the translated output;

[0040] If the similarity reaches a preset value, the lexical features and syntactic features of the multilingual mixed text to be processed are associated with the lexical features and syntactic features of the language of the translated output;

[0041] When translating multi-language mixed text, it is preferred to search for lexical features and syntactic features whose similarity reaches a preset value in the shared feature space;

[0042] The translation is performed in the language of the translation output corresponding to the lexical features and syntactic features whose similarity reaches a preset value.

[0043] In order to solve the above technical problems, the embodiment of the present application also provides a multi-language mixed text translation device, which adopts the following technical solution:

[0044] A multi-language mixed text translation device, comprising:

[0045] The acquisition module is used to obtain the input text and convert the input text into UTF-8 encoding;

[0046] A splicing module, used for splicing the prompt text in the target language with the input text converted into UTF-8 encoding to obtain a spliced ​​text, and converting the spliced ​​text into UTF-8 encoding;

[0047] A training module, used for training the ByT5 model based on the input text converted into UTF-8 encoding and the concatenated text to obtain a pre-trained ByT5 model;

[0048] The processing module is used to input the multilingual mixed text to be processed into the pre-trained ByT5 model to generate a translation output in the target language.

[0049] Furthermore, the device also includes:

[0050] The feature relationship learning module is used to learn the feature relationship between the multilingual mixed text to be processed and the language of the translated output, and map the feature relationship to a shared feature space, and use the feature relationship as an evaluation criterion when evaluating the ByT5 model.

[0051] Furthermore, the splicing module includes:

[0052] The conversion submodule is used to convert the prompt text in the target language into UTF-8 encoding. The prompt text is used to prompt the pre-trained ByT5 model to translate the input text into the target language;

[0053] The concatenation submodule is used to concatenate the prompt text converted into UTF-8 encoding with the input text converted into UTF-8 encoding as a prefix.

[0054] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0055] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the multi-language mixed text translation method described above when executing the computer-readable instructions.

[0056] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0057] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the multi-language mixed text translation method as described above.

[0058] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application obtains input text and converts it into UTF-8 encoding. With the characteristics of UTF-8, it can support language texts used in most countries in the world, and solves the problem that traditional machine translation models only support single-language translation due to token encoding issues. The prompt text in the target language is spliced ​​with the input text converted into UTF-8 encoding, and then through the pre-trained ByT5 model, it can make full use of the translation data of each language to learn the same high-dimensional semantic distribution, so that a single model can flexibly handle many-to-many bilingual to single-language text input translation tasks and multi-language mixed text input translation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0061] Figure 2A flowchart of an embodiment of a multi-language mixed text translation method according to the present application;

[0062] Figure 3 This Figure 2 A flowchart of an embodiment of step S202;

[0063] Figure 4 This Figure 2 A flowchart of an embodiment of step S204;

[0064] Figure 5 is a schematic structural diagram of an embodiment of a multi-language mixed text translation device according to the present application;

[0065] Figure 6 yes Figure 5 A structural schematic diagram of a specific implementation of the splicing module shown;

[0066] Figure 7 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0068] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0069] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0070] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0071] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0072] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0073] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0074] It should be noted that the multi-language mixed text translation method provided in the embodiment of the present application is generally executed by a server, and correspondingly, the multi-language mixed text translation device is generally set in a terminal device.

[0075] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0076] Continue to refer Figure 2 , shows a flow chart of an embodiment of a multi-language mixed text translation method according to the present application. The multi-language mixed text translation method comprises the following steps:

[0077] Step S201, obtaining input text, and converting the input text into UTF-8 encoding;

[0078] In this embodiment, UTF-8 (Unicode Transformation Format-8-bit) is a variable-length character encoding, which is a part of the Unicode standard. UTF-8 can represent almost all characters, including characters used by all languages ​​in the world.

[0079] Specifically, you can save UTF-8 encoded files in a text editor, that is, select UTF-8 encoding when saving the file. Generally, most text editors, such as Notepad++, Sublime Text, etc., provide a UTF-8 encoding save option.

[0080] In this embodiment, for example, in financial technology or digital medical services, when a user publishes a multi-language mixed text on a platform, the method for obtaining the input text can be to capture the multi-language mixed text by scanning.

[0081] In this embodiment, the multi-language mixed text translation method is executed on the electronic device (eg Figure 1 The server shown / Terminal equipment ) can obtain input text through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0082] Specifically, the input text can be obtained from the page input by the user, and can be obtained by scanning, taking a photo, etc.; UTF-8 encoding is used in this embodiment to ensure that the method can accurately process various languages ​​​​and special characters.

[0083] Step S202, concatenating the prompt text in the target language with the input text converted into UTF-8 encoding to obtain a concatenated text;

[0084] Specifically, the target language is the language into which the input text needs to be converted. For example, if the input text is "Chinese" and the target language into which it needs to be translated is "English", the prompt text in the target language is "English".

[0085] See also Figure 3 In this embodiment, the step of splicing the prompt text in the target language with the input text converted into UTF-8 encoding to obtain the spliced ​​text specifically includes:

[0086] Step S2021, converting the prompt text in the target language into UTF-8 encoding, where the prompt text is used to prompt the pre-trained ByT5 model to translate the input text into the target language;

[0087] For details about the ByT5 model, please refer to the paper "ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models" published on March 8, 2022.

[0088] Step S2022, using the prompt text converted into UTF-8 encoding as a prefix to concatenate with the input text converted into UTF-8 encoding to obtain the concatenated text.

[0089] For example, the following Table 1 can represent this step.

[0090] Table 1

[0091] Hint text: Enter text Output English: Hello. Hello. French: Hello. Bonjour. Chinese: Bonjour. Today is Friday Hello. Today is Friday.

[0092] The "Hello" in the first row and first column of Table 1 is the input text, and the "English" in front is the prompt text, that is to say: "Hello" needs to be converted into English, so the "output" in the first row and second column is "Hello".

[0093] Step S203, training the ByT5 model based on the input text converted into UTF-8 encoding and the concatenated text to obtain a pre-trained ByT5 model;

[0094] See also Figure 4 In this embodiment, the ByT5 model is trained based on the input text converted into UTF-8 encoding and the concatenated text to obtain the pre-trained ByT5 model, which specifically includes:

[0095] Step S2031, using the input text converted into UTF-8 encoding as the input of the ByT5 model.

[0096] Step S2032: Use the encoder of the ByT5 model to extract the positional relationship features of the input text.

[0097] Step S2033, using the prompt text in the target language in the concatenated text converted into UTF-8 encoding as the target language of the decoder, and inputting the positional relationship features of the input text into the decoder.

[0098] Step S2034, training the positional relationship features of the input text according to the target language of the decoder to obtain a pre-trained ByT5 model.

[0099] In this embodiment, the target language corresponding to the generated prompt text is used as output to train the ByT5 model, which further includes:

[0100] The parameters of the ByT5 model were optimized by gradient descent.

[0101] Step S204: input the multilingual mixed text to be processed into the pre-trained ByT5 model to generate a translation output in the target language.

[0102] In this embodiment, the ByT5 model adopts the paper "ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models" published by LintingXue et al. on May 8, 2022.

[0103] The data set in this embodiment (including the input text converted into UTF-8 encoding and the corresponding concatenated text) can be divided into a training set and a test set. The ByT5 model is trained with the training set and tested with the test set.

[0104] Step S205 , learning the feature relationship between the multi-language mixed text to be processed and the language of the translated output, and mapping the feature relationship to a shared feature space.

[0105] Step S206: Optimizing the ByT5 model by using the feature relationship evaluation.

[0106] In this embodiment, the learning of the feature relationship between the multilingual mixed text to be processed and the language of the translated output and mapping the feature relationship to a shared feature space specifically includes:

[0107] The lexical features and syntactic features of the multilingual mixed text to be processed and the language of the translated output are learned.

[0108] The lexical features and syntactic features are vectorized.

[0109] The vectorized lexical features and syntactic features are mapped to a shared feature space for storage, and then the feature relationship is used to evaluate and optimize the ByT5 model.

[0110] In this embodiment, the optimization of the ByT5 model by evaluating the characteristic relationship specifically includes:

[0111] The similarity of the lexical features and syntactic features of the different multilingual mixed texts to be processed and the language of the translated output is compared.

[0112] If the similarity reaches a preset value, the multi-language mixed text to be processed is associated with the lexical features and syntactic features of the language of the translated output.

[0113] When translating multi-language mixed texts, it is preferred to search for lexical features and syntactic features whose similarity reaches a preset value in the shared feature space.

[0114] The translation is performed in the language of the translation output corresponding to the lexical features and syntactic features whose similarity reaches a preset value.

[0115] Through this step, this embodiment can incorporate the similarity between different languages ​​into the evaluation criteria, so that the ByT5 model can adapt to many-to-many translation needs more comprehensively and accurately.

[0116] With the characteristics of UTF-8, this application can support language texts used in most countries in the world, solving the problem that traditional machine translation models only support single-language translation due to token encoding issues. Through the ByT5 model, it can make full use of the translation data of each language to learn the same high-dimensional semantic distribution, so that a single model can flexibly handle many-to-many bilingual to single-language text input translation tasks and multi-language mixed text input translation tasks.

[0117] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0118] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0120] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0121] Further references Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a multi-language mixed text translation device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0122] like Figure 5 As shown, the multi-language mixed text translation device 500 described in this embodiment includes: an acquisition module 501, a splicing module 502, a training module 503, a processing module 504 and a feature relationship learning module 505. Among them:

[0123] The acquisition module 501 is used to acquire input text and convert the input text into UTF-8 encoding.

[0124] The splicing module 502 is used to splice the prompt text in the target language with the input text converted into UTF-8 encoding to obtain a spliced ​​text, and convert the spliced ​​text into UTF-8 encoding.

[0125] The training module 503 is used to train the ByT5 model based on the input text converted into UTF-8 encoding and the concatenated text to obtain a pre-trained ByT5 model.

[0126] The processing module 504 is used to input the multi-language mixed text to be processed into the pre-trained ByT5 model to generate a translation output in the target language.

[0127] This embodiment uses the acquisition module 501 to acquire the input text and convert it into UTF-8 encoding. With the characteristics of UTF-8, it can support language texts used in most countries in the world, solving the problem that traditional machine translation models only support single-language translation due to token encoding issues. The splicing module 502 splices the prompt text in the target language with the input text converted into UTF-8 encoding, and then through the pre-trained ByT5 model, it can make full use of the translation data of each language to learn the same high-dimensional semantic distribution, so that a single model can flexibly handle the translation tasks of many-to-many bilingual to single-language text input and the translation tasks of multi-language mixed text input.

[0128] In some optional implementations of this embodiment, the multilingual mixed text translation device also includes: a feature relationship learning module 505 is used to learn the feature relationship between the multilingual mixed text to be processed and the language of the translation output, and map the feature relationship to a shared feature space, and use the feature relationship as an evaluation criterion when evaluating the ByT5 model.

[0129] See also Figure 6 , is a structural diagram of a specific implementation of the splicing module 502, and the splicing module 502 includes a conversion submodule 5021 and a splicing submodule 5022. The conversion submodule 5021 is used to convert the prompt text in the target language into UTF-8 encoding, and the prompt text is used to prompt the pre-trained ByT5 model to translate the input text into the target language. The splicing submodule 5022 is used to splice the prompt text converted into UTF-8 encoding as a prefix with the input text converted into UTF-8 encoding.

[0130] To solve the above technical problems, the present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0131] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 7 with components 71-73, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0132] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0133] The memory 71 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 7. Of course, the memory 71 can also include both the internal storage unit of the computer device 7 and its external storage device. In this embodiment, the memory 71 is generally used to store the operating system and various application software installed on the computer device 7, such as computer-readable instructions of a multi-language mixed text translation method, etc. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or are to be output.

[0134] The processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 72 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 72 is used to run the computer-readable instructions stored in the memory 71 or process data, such as running the computer-readable instructions of the multi-language mixed text translation method.

[0135] The network interface 73 may include a wireless network interface or a wired network interface. The network interface 73 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0136] This embodiment obtains input text and converts it into UTF-8 encoding. With the characteristics of UTF-8, it can support language texts used in most countries in the world, solving the problem that traditional machine translation models only support single-language translation due to token encoding issues. The prompt text in the target language is concatenated with the input text converted into UTF-8 encoding, and then through the pre-trained ByT5 model, it can make full use of the translation data of each language to learn the same high-dimensional semantic distribution, so that a single model can flexibly handle many-to-many bilingual to single-language text input translation tasks and multi-language mixed text input translation tasks.

[0137] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the multi-language mixed text translation method as described above.

[0138] This embodiment obtains input text and converts it into UTF-8 encoding. With the characteristics of UTF-8, it can support language texts used in most countries in the world, solving the problem that traditional machine translation models only support single-language translation due to token encoding issues. The prompt text in the target language is concatenated with the input text converted into UTF-8 encoding, and then through the pre-trained ByT5 model, it can make full use of the translation data of each language to learn the same high-dimensional semantic distribution, so that a single model can flexibly handle many-to-many bilingual to single-language text input translation tasks and multi-language mixed text input translation tasks.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0140] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A multi-language mixed text translation method, characterized in that: The steps include: Get the input text and convert it into UTF-8 encoding; Concatenate the prompt text in the target language with the input text converted into UTF-8 encoding to obtain a concatenated text, and convert the concatenated text into UTF-8 encoding; Training the ByT5 model based on the input text converted into UTF-8 encoding and the concatenated text to obtain a pre-trained ByT5 model; The multilingual mixed text to be processed is input into the pre-trained ByT5 model to generate a translation output in the target language.

2. The multilingual mixed text translation method according to claim 1, characterized in that: The step of splicing the prompt text in the target language with the input text converted into UTF-8 encoding to obtain the spliced ​​text specifically includes: Convert the prompt text in the target language into UTF-8 encoding. The prompt text is used to prompt the pre-trained ByT5 model to translate the input text into the target language. The prompt text converted into UTF-8 encoding is used as a prefix to be concatenated with the input text converted into UTF-8 encoding to obtain the concatenated text.

3. The multilingual mixed text translation method according to claim 1, characterized in that: The step of training the ByT5 model based on the input text converted into UTF-8 encoding and the concatenated text to obtain a pre-trained ByT5 model specifically includes: The input text converted to UTF-8 encoding is used as the input of the ByT5 model; Using an encoder of the ByT5 model to extract positional relationship features of the input text; Using the prompt text in the target language in the concatenated text converted into UTF-8 encoding as the target language of the decoder, and inputting the positional relationship features of the input text into the decoder; The positional relationship features of the input text are trained according to the target language of the decoder to obtain a pre-trained ByT5 model.

4. The multilingual mixed text translation method according to claim 3, characterized in that: Training the ByT5 model with the target language corresponding to the generated prompt text as output also includes: The parameters of the ByT5 model were optimized by gradient descent.

5. The multi-language mixed text translation method according to claim 1, characterized in that: After the step of inputting the multilingual mixed text to be processed into the pre-trained ByT5 model to generate the translation output in the target language, the method further includes: Learning the feature relationship between the multilingual mixed text to be processed and the language of the translated output, and mapping the feature relationship to a shared feature space; The ByT5 model is optimized using the characteristic relationship evaluation.

6. The multi-language mixed text translation method according to claim 5, characterized in that: The learning of the feature relationship between the multilingual mixed text to be processed and the language of the translated output and mapping the feature relationship to a shared feature space specifically includes: Learning the lexical features and syntactic features of the multilingual mixed text to be processed and the language of the translated output; Vectorizing the lexical features and syntactic features; The vectorized lexical features and syntactic features are mapped to a shared feature space for storage, and then the feature relationship is used to evaluate and optimize the ByT5 model.

7. The multi-language mixed text translation method according to claim 6, characterized in that: The utilizing the characteristic relationship to evaluate and optimize the ByT5 model specifically includes: Compare the similarity of lexical features and syntactic features of different multilingual mixed texts to be processed and the language of the translated output; If the similarity reaches a preset value, the lexical features and syntactic features of the multilingual mixed text to be processed are associated with the lexical features and syntactic features of the language of the translated output; When translating multi-language mixed text, it is preferred to search for lexical features and syntactic features whose similarity reaches a preset value in the shared feature space; The translation is performed in the language of the translation output corresponding to the lexical features and syntactic features whose similarity reaches a preset value.

8. A multi-language mixed text translation device, characterized in that: include: The acquisition module is used to obtain the input text and convert the input text into UTF-8 encoding; A splicing module, used for splicing the prompt text in the target language with the input text converted into UTF-8 encoding to obtain a spliced ​​text, and converting the spliced ​​text into UTF-8 encoding; A training module, used for training the ByT5 model based on the input text converted into UTF-8 encoding and the concatenated text to obtain a pre-trained ByT5 model; The processing module is used to input the multilingual mixed text to be processed into the pre-trained ByT5 model to generate a translation output in the target language.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the multi-language mixed text translation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the multi-language mixed text translation method according to any one of claims 1 to 7 are implemented.

Images