Text data processing method, device, equipment and medium

Abstract

The embodiments of the present application provide a text data processing method, apparatus, device, and medium, which relates to the field of artificial intelligence, and includes: obtaining a first text pair and a second text pair, obtaining a first subtext from the first text pair, and obtaining a second subtext from the second text pair; determining the edit distance between the first subtext and the second subtext, and if the edit distance satisfies a similarity condition, generating a first target subtext associated with the semantic information of the first subtext and belonging to a third language type, and generating a second target subtext associated with the semantic information of the second subtext and belonging to a second language type; generating a text sample pair based on the first text pair, the second text pair, the first target subtext, and the second target subtext. By using the present application, text sample pairs composed of different language types can be generated, thereby increasing the quantity of the corpus while ensuring the quality of the corpus.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text data processing method, apparatus, device, and medium. Background Art

[0002] Existing corpora typically contain English-Centric (i.e., English is the source language or the target language) text pairs (i.e., sentence pairs). To increase the amount of data in existing corpora, it is necessary to generate non-English (non-English direction, i.e., both the source language and the target language are non-English) sentence pairs based on the English-Centric sentence pairs in the corpus.

[0003] Currently, corpora are generated using an extraction-based approach. When extracting parallel corpora from the existing English-Centric corpus, this extraction-based approach typically aligns identical English-sides to create a multi-way parallel corpus. For example, when the English sides of a "French-English" sentence pair and an "English-German" sentence pair are identical, a "French-English-German" three-way aligned sentence pair can be obtained, and then a "French-German" two-way aligned sentence pair can be obtained. However, this method requires strict alignment of the English-sides, resulting in the scale of the extracted corpus being much smaller than that of the English-Centric corpus. This means that the existing method of generating text pairs is limited in the amount of generated corpus, which limits the quality of machine translation. Summary of the Invention

[0004] The embodiments of the present application provide a text data processing method, apparatus, device, and medium, which can increase the quantity of corpus while ensuring the quality of the corpus.

[0005] An embodiment of the present application provides a text data processing method, including:

[0006] Obtain a first text pair and a second text pair, obtain a first subtext from the first text pair, and obtain a second subtext from the second text pair; the first subtext and the second subtext both belong to a first language type; the first text pair further includes a third subtext having the same semantic information as the first subtext and belonging to the second language type; the second text pair further includes a fourth subtext having the same semantic information as the second subtext and belonging to the third language type;

[0007] determining an edit distance between the first subtext and the second subtext, and if the edit distance satisfies a similarity condition, generating a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and generating a second target subtext associated with the semantic information of the second subtext and belonging to the second language type;

[0008] A text sample pair is generated according to the first text pair, the second text pair, the first target subtext, and the second target subtext.

[0009] An embodiment of the present application provides a text data processing device, including:

[0010] a text acquisition module configured to acquire a first text pair and a second text pair, acquire a first subtext from the first text pair, and acquire a second subtext from the second text pair; the first subtext and the second subtext both belong to a first language type; the first text pair further includes a third subtext having the same semantic information as the first subtext and belonging to the second language type; and the second text pair further includes a fourth subtext having the same semantic information as the second subtext and belonging to the third language type;

[0011] a text generation module, configured to determine an edit distance between a first subtext and a second subtext, and if the edit distance satisfies a similarity condition, generate a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and generate a second target subtext associated with the semantic information of the second subtext and belonging to the second language type;

[0012] The sample pair generation module is used to generate a text sample pair according to the first text pair, the second text pair, the first target subtext and the second target subtext.

[0013] The device further comprises:

[0014] a length determination module, configured to determine the text length of the first subtext as a first text length, determine the text length of the second subtext as a second text length, and determine a target length corresponding to a similarity condition based on the first text length and the second text length;

[0015] A distance determination module is used to obtain similarity parameters associated with the similarity condition and determine the similarity distance corresponding to the similarity condition based on the target length and the similarity parameters;

[0016] a first comparison module, configured to determine that the edit distance satisfies a similarity condition if the edit distance is less than or equal to the similarity distance;

[0017] The second comparison module is configured to determine that the edit distance does not satisfy the similarity condition if the edit distance is greater than the similarity distance.

[0018] Among them, the text generation module includes:

[0019] a text extraction unit, configured to extract a first unit text from the first subtext and a second unit text from the second subtext;

[0020] The distance determining unit is configured to determine an edit distance between the first subtext and the second subtext based on the first unit text and the second unit text.

[0021] a target acquisition unit, configured to acquire a set of network models associated with the first subtext and the second subtext if the edit distance satisfies a similarity condition;

[0022] The text generation unit is configured to generate, based on the network model set, a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and a second target subtext associated with the semantic information of the second subtext and belonging to the second language type.

[0023] The network model set includes a first target network model and a second target network model; the first target network model is associated with the first language type and the third language type, and the second target network model is associated with the first language type and the second language type;

[0024] The text generation unit includes:

[0025] A first splicing subunit is used to splice the first subtext and the fourth subtext to obtain a first spliced ​​text;

[0026] A first generating sub-unit is configured to input the first concatenated text into a first target network model, and generate a first target sub-text associated with the semantic information of the first sub-text and belonging to a third language type through the first target network model;

[0027] A second splicing subunit is used to splice the second subtext and the third subtext to obtain a second spliced ​​text;

[0028] The second generating subunit is configured to input the second concatenated text into a second target network model, and generate a second target subtext associated with the semantic information of the second subtext and belonging to a second language type through the second target network model.

[0029] The first target network model includes an encoder for encoding and a decoder for decoding;

[0030] The first generation subunit includes:

[0031] An encoding processing subunit, configured to input the first concatenated text into an encoder in the first target network model, and perform encoding processing on the first concatenated text through the encoder to obtain a first feature vector corresponding to the first concatenated text;

[0032] A decoding processing subunit, configured to input the first feature vector into a decoder in the first target network model, and decode the first feature vector through the decoder to obtain a first text vector corresponding to the first feature vector;

[0033] The text generation subunit is configured to generate, based on the first text vector, a first target subtext associated with the semantic information of the first subtext and belonging to a third language type.

[0034] The sample pair generation module includes:

[0035] A first combining unit is configured to combine the first target subtext and the third subtext into a first sample pair; the first target subtext is associated with semantic information of the third subtext;

[0036] A second combining unit is configured to combine the second target subtext and the fourth subtext into a second sample pair; and the second target subtext is associated with semantic information of the fourth subtext.

[0037] The sample pair determining unit is configured to determine the first text pair, the second text pair, the first sample pair, and the second sample pair as a text sample pair.

[0038] The text generation unit further includes:

[0039] an initial acquisition subunit, configured to acquire a first initial network model associated with the first language type and the third language type;

[0040] a transformation subunit, configured to perform editing transformation on the fourth subtext to obtain a first transformed text associated with the fourth subtext;

[0041] a splicing processing subunit, configured to splice the second subtext and the first transformed text to obtain a first spliced ​​sample;

[0042] The model training subunit is used to obtain the first sample vector of the first spliced ​​sample through the first initial network model, and perform model training on the first initial network model based on the first sample vector and the fourth subtext to obtain a first target network model.

[0043] The fourth subtext contains N unit texts; N is a positive integer;

[0044] The transformation subunit includes:

[0045] A probability generation subunit is used to generate a random transformation probability corresponding to each unit text in the N unit texts, and determine the unit text whose random transformation probability is within the transformable probability interval as the unit text to be edited;

[0046] The editing transformation subunit is used to obtain the editing transformation method associated with the unit text to be edited, perform editing transformation on the unit text to be edited according to the editing transformation method, obtain the editing transformation result, and generate the first transformation text associated with the fourth subtext according to the editing transformation result.

[0047] The editing transformation subunit is specifically used to obtain the editing transformation mode associated with the unit text to be edited;

[0048] The edit transformation subunit is further specifically configured to, if the edit transformation mode is a replace operation, obtain a dictionary table corresponding to the third language type, obtain the first edit text from the dictionary table, replace the unit text to be edited with the first edit text, and obtain an edit transformation result;

[0049] The edit transformation subunit is further specifically configured to obtain a second edit text from the dictionary table if the edit transformation mode is an insert operation, insert the second edit text into an adjacent position of the unit text to be edited, and obtain an edit transformation result;

[0050] The editing transformation subunit is further specifically configured to delete the unit text to be edited in the fourth subtext if the editing transformation mode is a deletion operation, to obtain an editing transformation result.

[0051] Among them, the model training subunit includes:

[0052] a text prediction subunit, configured to obtain a first sample vector of the first concatenated sample through a first initial network model, and generate a predicted sample subtext belonging to a third language type based on the first sample vector;

[0053] a loss generating subunit, configured to obtain a sample semantic similarity between the predicted sample subtext and the fourth subtext, and generate a model loss function of the first initial network model according to the sample semantic similarity;

[0054] The parameter adjustment subunit is used to adjust the parameters of the first initial network model based on the model loss function to obtain a first target network model.

[0055] The device further comprises:

[0056] The model training module is used to obtain an initial translation model associated with a text sample pair, iteratively train the initial translation model based on the text sample pair, and determine the iteratively trained initial translation model as a target translation model; the target translation model is used to translate text between any two language types among the first language type, the second language type, and the third language type.

[0057] In one aspect, an embodiment of the present application provides a computer device, including: a processor and a memory;

[0058] The processor is connected to a memory, wherein the memory is used to store a computer program. When the computer program is executed by the processor, the computer device executes the method provided in the embodiment of the present application.

[0059] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method provided by the embodiment of the present application.

[0060] In one aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the embodiments of the present application.

[0061] In an embodiment of the present application, when a computer device obtains a first text pair and a second text pair, it can obtain a first subtext from the first text pair and a second subtext from the second text pair. Wherein, the first subtext and the second subtext both belong to the first language type, and the first text pair here also includes a third subtext having the same semantic information as the first subtext and belonging to the second language type, and the second text pair here also includes a fourth subtext having the same semantic information as the second subtext and belonging to the third language type. Further, the computer device can determine the edit distance between the first subtext and the second subtext, and if the edit distance meets the similarity condition, a first target subtext associated with the semantic information of the first subtext and belonging to the third language type is generated, and a second target subtext associated with the semantic information of the second subtext and belonging to the second language type is generated. Further, the computer device can generate a text sample pair based on the first text pair, the second text pair, the first target subtext and the second target subtext. Thus, it can be seen that the embodiment of the present application can generate a multi-path aligned corpus based on the extraction and generation steps. The extraction step can determine the edit distance between the first subtext in the first text pair and the second subtext in the second text pair, and identify the first text pair and the second text pair corresponding to the edit distance that meets the similarity condition as candidate sentence pairs; the generation step can eliminate the semantic differences between the first subtext and the second subtext, converting the "partially aligned" candidate sentence pairs into "completely aligned" text sample pairs. Based on this, through the above-mentioned extraction and generation steps, a large amount of high-quality semantically aligned corpus (i.e., text sample pairs) can be quickly and accurately generated, thereby increasing the quantity of the corpus while ensuring the quality of the corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 This is a schematic diagram of a network architecture provided by an embodiment of the present application;

[0064] Figure 2 This is a schematic diagram of a data interaction scenario provided by an embodiment of the present application;

[0065] Figure 3 This is a flowchart of a text data processing method provided in an embodiment of the present application;

[0066] Figure 4 This is a schematic diagram of a scenario for generating candidate text pairs provided by an embodiment of the present application;

[0067] Figure 5 This is a schematic diagram of a process for generating a target subtext provided by an embodiment of the present application;

[0068] Figure 6 This is a schematic diagram of a scenario for generating text sample pairs provided by an embodiment of the present application;

[0069] Figure 7 This is a flowchart of a text data processing method provided in an embodiment of the present application;

[0070] Figure 8 This is a schematic diagram of a scenario for generating a target subtext provided by an embodiment of the present application;

[0071] Figure 9 This is a flowchart of a text data processing method provided in an embodiment of the present application;

[0072] Figure 10 This is a schematic diagram of a performance comparison scenario provided by an embodiment of the present application;

[0073] Figure 11 This is a structural diagram of a text data processing device provided in an embodiment of the present application;

[0074] Figure 12 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0076] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0077] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0078] Among them, the solutions provided in the embodiments of the present application mainly involve natural language processing (NLP) technology and machine learning (ML) technology of artificial intelligence.

[0079] Machine learning is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0080] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0081] For details, see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a service server 2000 and a user terminal cluster. The user terminal cluster may specifically include one or more user terminals, and the number of user terminals in the user terminal cluster is not limited here. Figure 1 As shown, the multiple user terminals may specifically include user terminal 3000a, user terminal 3000b, user terminal 3000c, ..., user terminal 3000n; user terminal 3000a, user terminal 3000b, user terminal 3000c, ..., user terminal 3000n may respectively be directly or indirectly connected to the business server 2000 via wired or wireless communication, so that each user terminal may exchange data with the business server 2000 via the network connection.

[0082] Among them, the business server 2000 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0083] Each user terminal in the user terminal cluster may include: smart phones, tablet computers, laptop computers, desktop computers, smart homes, wearable devices, vehicle terminals and other smart terminals with text data processing functions. Figure 1 Each user terminal in the user terminal cluster shown can be integrated with an application client. When the application client runs in each user terminal, it can be respectively connected to the above-mentioned Figure 1Data is exchanged between the business servers 2000 shown. The application clients may include: in-vehicle clients, smart home clients, entertainment clients (e.g., game clients), multimedia clients (e.g., video clients), social clients, and information clients (e.g., news clients).

[0084] For ease of understanding, the embodiments of the present application can be Figure 1 Select one user terminal from the multiple user terminals shown as the target user terminal. For example, the embodiment of the present application can Figure 1 The user terminal 3000a shown is used as a target user terminal, and the target user terminal may be integrated with an application client having a text data processing function. At this time, the target user terminal can implement data interaction with the service server 2000 through the application client.

[0085] It should be understood that the computer device in the embodiment of the present application can obtain the first text pair and the second text pair from the corpus. Among them, the corpus may include text pairs composed of sub-texts of multiple language types, the first text pair may include a first sub-text belonging to the first language type, and a third sub-text having the same semantic information as the first sub-text and belonging to the second language type, and the second text pair may include a second sub-text belonging to the first language type, and a fourth sub-text having the same semantic information as the second sub-text and belonging to the third language type. Furthermore, the computer device can generate text pairs (i.e., text sample pairs) composed of sub-texts of other language types based on the first text pair and the second text pair, thereby enriching the existing corpus.

[0086] It can be understood that the text data processing method provided in the embodiment of the present application can be executed by the above-mentioned business server 2000 (that is, the above-mentioned computer device can be the business server 2000), or can be executed by the above-mentioned target user terminal (that is, the above-mentioned computer device can be the target user terminal), or can be executed jointly by the business server 2000 and the target user terminal.

[0087] In which, when the text data processing method is jointly executed by the business server 2000 and the target user terminal, the business server 2000 can perform model training on the initial translation model used for text translation based on the text sample pairs obtained above to obtain a target translation model. In this way, the user corresponding to the target user terminal (for example, user Y) can send a text translation request to the business server 2000 through the application client in the target user terminal. In which, the text translation request here can include the initial translation text requested by user Y for text translation. Furthermore, after receiving the text translation request, the business server 2000 can obtain the initial translation text from the text translation request, perform text translation on the initial translation text through the above-mentioned target translation model to obtain the target translation text, and then return the target translation text to the target user terminal.

[0088] Optionally, when the text data processing method is executed by the service server 2000, the service server 2000 may directly translate the initial translation text using the target translation model trained above to obtain the target translation text. Optionally, when the text data processing method is executed by the target user terminal, the target user terminal may train the initial translation model based on the text sample pairs obtained above to obtain the target translation model, and then directly translate the initial translation text using the target translation model trained on the target user terminal to obtain the target translation text.

[0089] For further understanding, please refer to Figure 2 , Figure 2 This is a schematic diagram of a data interaction scenario provided by an embodiment of the present application. Figure 2 The server 20a shown may be the above Figure 1 The service server 2000 in the corresponding embodiment is as follows: Figure 2 The user terminal 20b shown can be the above Figure 1 For ease of understanding, any user terminal in the user terminal cluster of the corresponding embodiment is described in the embodiment of the present application. Figure 1 The user terminal 3000a shown is used as an example of the user terminal 20b to illustrate Figure 2 The specific process of data interaction between the server 20a and the user terminal 20b is shown. The user terminal 20b is installed with an application client, which can be used to display the initial translation text and the target translation text. The user corresponding to the user terminal 20b can be user 20c.

[0090] It is understandable that, Figure 2 The text database 20d shown may include multiple databases, which may include Figure 2As shown, databases 30a, 30b, ..., and 30n are shown. This means that text database 20d can be used to store text pairs consisting of subtexts of different language types. For example, database 30a can be used to store text pairs consisting of subtexts of a first language type and a second language type, database 30b can be used to store text pairs consisting of subtexts of a first language type and a third language type, ..., and database 30n can be used to store text pairs consisting of subtexts of a first language type and a fourth language type.

[0091] like Figure 2 As shown, server 20a can obtain a first text pair and a second text pair from a text database (e.g., text database 20d), for example, obtaining first text pair 21a from database 30a and second text pair 21b from database 30b. First text pair 21a may include a first subtext in a first language type and a third subtext in a second language type, and the first and third subtexts are semantically aligned (i.e., have the same semantic information); second text pair 21b may include a second subtext in the first language type and a fourth subtext in the third language type, and the second and fourth subtexts are semantically aligned (i.e., have the same semantic information).

[0092] Furthermore, server 20a may obtain a first subtext and a second subtext of the same language type (i.e., the first language type) from first text pair 21a and second text pair 21b, and then determine the text similarity between the first subtext and the second subtext. For example, the text similarity between the first subtext and the second subtext may be determined using the edit distance. It is understood that if the text similarity between the first subtext and the second subtext satisfies the similarity condition (i.e., the edit distance satisfies the similarity condition), server 20a may generate a target subtext 21c based on first text pair 21a and second text pair 21b. The target subtext 21c may include the first target subtext and the second target subtext.

[0093] It can be understood that the server 20a can generate a first target subtext that is associated with the semantic information of the first subtext and belongs to the third language type based on the first subtext and the fourth subtext; and the server 20a can generate a second target subtext that is associated with the semantic information of the second subtext and belongs to the second language type based on the second subtext and the third subtext.

[0094] like Figure 2As shown, server 20a can generate text sample pairs based on first text pair 21a, second text pair 21b, and target subtext 21c, and then obtain an initial translation model for text translation. Based on the text sample pairs, the initial translation model is trained to obtain a target translation model. The target translation model can be used to translate text between any two of the first, second, and third language types.

[0095] like Figure 2 As shown, when user 20c needs to carry out text translation to a certain paragraph (i.e. initial translation text), a text translation request carrying this initial translation text can be sent to server 20a by user terminal 20b.For example, the initial translation text here can belong to a second language type, and user 20c expects that the language type obtained by text translation of the initial translation text of the second language type can be a third language type.Like this, server 20a, after receiving the initial translation text, can obtain the target translation model for carrying out text translation, and the initial translation text belonging to the second language type is carried out text translation by the target translation model, and obtains the target translation text belonging to the third language type.Optionally, the target translation model can also carry out text translation to the initial translation text belonging to the second language type, and obtains the target translation text belonging to the first language type. It should be understood that the language type of the target translation text is not limited in the embodiment of the present application.

[0096] Furthermore, the server 20a may return the target translation text obtained by translation to the user terminal 20b. In this way, the user terminal 20b may display the target translation text at the same time as the initial translation text, so that the user 20c may obtain the target translation text corresponding to the initial translation text.

[0097] It can be seen from this that the embodiment of the present application can extract "highly similar" text pairs (i.e., the first text pair and the second text pair) of the first language type (for example, the English side) through the edit distance, and then eliminate the semantic differences of the text pairs through the model generation method. Among them, the "highly similar" text pairs can ensure the scale of the generated corpus; the "highly similar" text pairs serve as prior knowledge, and this prior knowledge can effectively avoid the diversity of the generated corpus being lower than the diversity of the original corpus. Therefore, the embodiment of the present application can generate a corpus with large scale and diversity, and when the initial translation model is trained with the corpus, the translation performance of the target translation model can be improved.

[0098] Further, see Figure 3 , Figure 3This is a flowchart of a text data processing method provided by an embodiment of the present application. The method can be executed by a server, or by a user terminal, or by both a server and a user terminal. The server can be the above-mentioned Figure 2 The corresponding implementation of the server 20a, the user terminal can be the above Figure 2 For ease of understanding, the present invention uses the method executed by the server as an example to illustrate the method. The text data processing method may include the following steps S101 to S103:

[0099] Step S101: obtaining a first text pair and a second text pair, obtaining a first subtext from the first text pair, and obtaining a second subtext from the second text pair;

[0100] The first subtext and the second subtext both belong to the first language type; the first text pair further includes a third subtext having the same semantic information as the first subtext and belonging to the second language type; the second text pair further includes a fourth subtext having the same semantic information as the second subtext and belonging to the third language type.

[0101] It should be understood that the first text pair and the second text pair may be obtained from a corpus, which may include text pairs consisting of subtexts in multiple language types. When the corpus is a bilingual corpus, the corpus may include text pairs consisting of texts in two language types, for example, a first text pair consisting of a first subtext in a first language type and a third subtext in a second language type, and a second text pair consisting of a second subtext in the first language type and a fourth subtext in a third language type.

[0102] It is understood that the corpus used in the embodiments of the present application may be a bilingual corpus, which may be a WMT-5 dataset, which is an English-centric dataset (i.e., English-centric, meaning that the source or target language is English). It should be understood that the embodiments of the present application do not limit the specific type of bilingual corpus.

[0103] Among them, it can be understood that the WMT-5 dataset is composed of 5 groups of datasets, which can specifically include: WMT13EnEs dataset, WMT14EnDe dataset, WMT15EnFr dataset, WMT18EnCs dataset and WMT18EnRu dataset. Among them, the WMT13EnEs dataset is a dataset composed of English (abbreviated as En / en) and Spanish (i.e. Spanish, abbreviated as Es / es), and each pair of text pairs in the WMT13EnEs dataset uses English and Spanish as the source language end or the target language end; the WMT14EnDe dataset is a dataset composed of English and German (i.e. German, abbreviated as De / de), and each pair of text pairs in the WMT14EnDe dataset uses English and German as the source language end or the target language end; the WMT15EnFr dataset is a dataset composed of English and French (i.e. French, abbreviated as Fr / fr ), each text pair in the WMT15EnFr dataset uses English and French as the source language or target language; the WMT18EnCs dataset is composed of English and Czech (i.e., Czech, abbreviated as Cs / cs), each text pair in the WMT18EnCs dataset uses English and Czech as the source language or target language; the WMT18EnRu dataset is composed of English and Russian (i.e., Russian, abbreviated as Ru / ru), each text pair in the WMT18EnRu dataset uses English and Russian as the source language or target language.

[0104] It should be understood that a bilingual corpus may also include sample data sets consisting of text pairs composed of subtexts in other language types. This embodiment of the application uses the bilingual corpus including the WMT-5 dataset as an example for explanation. The process of processing text data for text pairs in the sample dataset can be found in the description of processing text data for the first and second text pairs. The WMT-5 dataset can include subtexts in six language types: a first language type, a second language type, a third language type, a fourth language type, a fifth language type, and a sixth language type.

[0105] It should be understood that the embodiments of this application do not limit the specific language types of the first language type, the second language type, and the third language type. For example, when the first text pair belongs to the WMT15EnFr dataset, the first language type can be English and the second language type can be French; when the second text pair belongs to the WMT14EnDe dataset, the first language type can be English and the third language type can be German. The second language type can be used as either a source language or a target language; the third language type can be used as either a source language or a target language.

[0106] Similarly, the embodiments of the present application do not limit the specific language types of the fourth, fifth, and sixth language types. For example, when the first language type is English, the second language type is French, and the third language type is German, the fourth language type may be Czech, the fifth language type may be Spanish, and the sixth language type may be Russian.

[0107] Step S102: determining an edit distance between a first subtext and a second subtext; if the edit distance satisfies a similarity condition, generating a first target subtext associated with the semantic information of the first subtext and belonging to the third language type; and generating a second target subtext associated with the semantic information of the second subtext and belonging to the second language type;

[0108] Specifically, the server may extract a first unit text from the first subtext and a second unit text from the second subtext. Furthermore, the server may determine an edit distance between the first subtext and the second subtext based on the first unit text and the second unit text. Furthermore, if the edit distance satisfies a similarity condition, the server may generate a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and generate a second target subtext associated with the semantic information of the second subtext and belonging to the second language type.

[0109] It is understood that the present application can apply the edit distance to the word level. For example, when the first subtext is "How do you do?", the first unit text extracted from the first subtext can be: "How", "do", "you", "do", "?". For another example, when the second subtext is "How are you?", the second unit text extracted from the second subtext can be: "How", "are", "you", "?".

[0110] Among them, the embodiments of the present application do not distinguish between text (for example, "How", "do") and symbols (for example, "?"), and the text and symbols here can be collectively referred to as unit text.

[0111] It is understandable that the edit distance measures the similarity between sentences by the minimum number of operations required to convert one sentence into another, so the similar sentence pairs obtained in this way often have more similar words and more similar sentence structures. In addition, the calculation of the edit distance only includes three operations: replacement, insertion, and deletion, which is relatively simple. Among them, the distance calculation method can be a dynamic programming method. It should be understood that the embodiments of the present application do not limit the specific method type of the distance calculation method.

[0112] It should be understood that the relationship between the edit distance and the similarity condition can be described as follows: the server can determine the text length of the first subtext as the first text length, and the text length of the second subtext as the second text length, and determine the target length corresponding to the similarity condition based on the first text length and the second text length. Further, the server can obtain a similarity parameter associated with the similarity condition, and determine the similarity distance corresponding to the similarity condition based on the target length and the similarity parameter. Further, if the edit distance is less than or equal to the similarity distance, the server can determine that the edit distance satisfies the similarity condition. Optionally, if the edit distance is greater than the similarity distance, the server can determine that the edit distance does not satisfy the similarity condition.

[0113] It is understood that the server can give (X 1 ,Y 1 ) and (X 2 ,Y 2 ) is a language pair (i.e., a text pair), X 1 and X 2 is the first language type (for example, English), Y 1 and Y 2 Language L a (e.g., second language type) and language L b (e.g., third language type). Therefore, Can be the first text pair, Represents the first subtext, Indicates the third subtext; Can be the second text pair, Represents the second subtext, Indicates the fourth subtext.

[0114] The relationship between the edit distance and the similarity distance can be seen in the following formula (1):

[0115]

[0116] in, It can represent the text length of the first subtext (i.e. the first text length), It can represent the text length of the second subtext (ie, the second text length). It can represent the smaller value of the first text length and the second text length. Therefore, in the embodiment of the present application, the smaller value of the first text length and the second text length can be used as the target length. When the first text length is less than the second text length, the target length can be equal to the first text length; when the first text length is greater than the second text length, the target length can be equal to the second text length. γ can represent a similarity parameter, It can represent the similarity distance corresponding to the similarity condition, It can represent the edit distance between the first subtext and the second subtext.

[0117] If the relationship between the edit distance and the similarity distance satisfies formula (1), it is determined that the edit distance is less than or equal to the similarity distance. At this time, the server can As a candidate text pair (ie, a candidate sentence pair), the candidate text pair is then added to the candidate set. Optionally, if the relationship between the edit distance and the similarity distance does not satisfy formula (1), it is determined that the edit distance is greater than the similarity distance.

[0118] It can be understood that γ is an adjustable hyperparameter that controls the similarity of language pairs (i.e., candidate text pairs) in the candidate set, that is, the degree of similarity between the first subtext and the second subtext, γ∈[0,1]. The larger γ is, the less similar the first subtext and the second subtext are, and the greater the semantic difference; the smaller γ is, the more similar the first subtext and the second subtext are, and the smaller the semantic difference. When γ = 0, the similarity distance is equal to 0. When the first subtext and the second subtext are exactly the same, the edit distance between the first subtext and the second subtext (i.e., the edit distance is equal to 0) satisfies the similarity condition. In this case, the first subtext and the second subtext belong to a fully aligned multi-parallel corpus. Otherwise, more similar sentence pairs with semantic differences will be obtained.

[0119] For easier understanding, see Figure 4 , Figure 4 This is a schematic diagram of a scenario for generating candidate text pairs provided by an embodiment of the present application. Figure 4 As shown, the first text pair 41a may include a first subtext in a first language type and a third subtext in a second language type, and the second text pair 41b may include a second subtext in the first language type and a fourth subtext in a third language type. The first subtext and the third subtext have the same semantic information (i.e., first semantic information), and the second subtext and the fourth subtext have the same semantic information (i.e., second semantic information).

[0120] like Figure 4As shown, the server can obtain subtexts of the same language type in the first text pair 41a and the second text pair 41b, that is, obtain the first subtext of the first language type in the first text pair 41a, and obtain the second subtext of the first language type in the second text pair 41b, and then determine the edit distance between the first subtext and the second subtext. Furthermore, when the server determines that the edit distance meets the similarity condition, it can generate a candidate text pair based on the first text pair 41a and the second text pair 41b. The candidate text pair here can be candidate text pair 41c, and candidate text pair 41c can be expressed as (first subtext, third subtext, second subtext, fourth subtext). Furthermore, after obtaining the candidate text pair 41c, the server can store the candidate text pair 41c in the candidate set.

[0121] For easier understanding, see Figure 5 , Figure 5 This is a flow chart of generating a target subtext provided by an embodiment of the present application. Figure 5 As shown, (E1, A1) may be the first text pair 50a, and (E2, Z1) may be the second text pair 50b; Figure 5 As shown, (E3, A2) may be the first text pair 50c, and (E4, Z2) may be the second text pair 50d. Optionally, the first text pair 50c may also be referred to as the third text pair 50c, and the second text pair 50d may also be referred to as the fourth text pair 50d. Figure 5 As shown, (E5, A3) may be the first text pair 50e, and (E6, Z3) may be the second text pair 50f. Optionally, the first text pair 50e may also be referred to as the fifth text pair 50e, and the second text pair 50f may also be referred to as the sixth text pair 50f. Figure 5 The shown (E7, A4) may be the first text pair 50g, and (E8, Z4) may be the second text pair 50h. Optionally, the first text pair 50g may also be referred to as the seventh text pair 50g, and the second text pair 50h may also be referred to as the eighth text pair 50h.

[0122] It can be understood that, here, (E1, A1), (E3, A2), (E5, A3) and (E7, A4) are taken as different first text pairs for illustration, and (E2, Z1), (E4, Z2), (E6, Z3) and (E8, Z4) are taken as different second text pairs for illustration. In other words, here, (E1, A1), (E3, A2), (E5, A3) and (E7, A4) are collectively referred to as the first text pair for illustration, and (E2, Z1), (E4, Z2), (E6, Z3) and (E8, Z4) are collectively referred to as the second text pair for illustration. Here, the first text pair and the second text pair may belong to the OPUS dataset.

[0123] Among them, E can represent the first language type, A can represent the second language type, and Z can represent the third language type. For example, the first language type can be English, the second language type can be Arabic, and the third language type can be Chinese. Therefore, Figure 5 Examples of semantic alignment from Arabic to Chinese and from Chinese to Arabic generated by the method provided in the embodiments of the present application are shown.

[0124] like Figure 5 As shown, the server can generate a first target subtext 61a based on the first text pair 50a and the second text pair 50b. The first target subtext 61a can be Z5. The Z5 and the fourth subtext Z1 both belong to the third language type, and the Z5 is associated with the semantic information of the first subtext E1; the server can generate a second target subtext 61b based on the first text pair 50a and the second text pair 50b. The second target subtext 61b can be A5. The A5 and the third subtext A1 both belong to the second language type, and the A5 is associated with the semantic information of the second subtext E2.

[0125] Similarly, if Figure 5 As shown, the server can generate a first target subtext 62a and a second target subtext 62b based on the first text pair 50c and the second text pair 50d. The first target subtext 62a can be Z6, and the second target subtext 62b can be A6. The server can generate a first target subtext 63a and a second target subtext 63b based on the first text pair 50e and the second text pair 50f. The first target subtext 63a can be Z7, and the second target subtext 63b can be A7. The server can generate a first target subtext 64a and a second target subtext 64b based on the first text pair 50g and the second text pair 50h. The first target subtext 64a can be Z8, and the second target subtext 64b can be A8.

[0126] For example, the first subtext E1 can be “Did you have anything to do with him.”, the second subtext E2 can be “Did you have anything to do with it?”, and the fourth subtext Z1 can be “Are you related to this matter?”. Therefore, the first target subtext Z5 can be “Are you related to him?”. 1 and x 2 Highly similar, but still with semantic differences: x 1 There is "him" in 2There is none in it. By adding "he" to the generated Chinese sentence, which corresponds to "him" in English, the semantic difference is eliminated, so the generated and x 1 is semantically aligned, that is, with y 1 is semantically aligned, thus obtaining high-quality semantically aligned bilingual corpora (or multi-way parallel corpora).

[0127] For another example, the first sub-text E3 can be "You want justice, right?", the second sub-text E4 can be "You want out, right?", the fourth sub-text Z2 can be "你想出去,嗯?", so the first target sub-text Z6 can be "你想伸张正义对吧,嗯?". For another example, the first sub-text E5 can be "Item 56 of the provisional agenda*", the second sub-text E6 can be "Item 100 of the provisional agenda*", the fourth sub-text Z3 can be "临时议程项目100", so the first target sub-text Z7 can be "临时议程项目56". For another example, the first sub-text E7 can be "What does he want?", the second sub-text E8 can be "What does he know?", the fourth sub-text Z4 can be "他刚说什么,他知道什么?", so the first target sub-text Z8 can be "他想知道什么?".

[0128] Step S103, generate text sample pairs according to the first text pair, the second text pair, the first target sub-text and the second target sub-text.

[0129] Specifically, the server can combine the first target sub-text and the third sub-text into the first sample pair. Among them, the semantic information of the first target sub-text is associated with that of the third sub-text. Further, the server can combine the second target sub-text and the fourth sub-text into the second sample pair. Among them, the semantic information of the second target sub-text is associated with that of the fourth sub-text. Further, the server can determine the first text pair, the second text pair, the first sample pair and the second sample pair as text sample pairs.

[0130] Optionally, the server can also combine the first target sub-text and the first sub-text into the third sample pair. Further, the server can also combine the second target sub-text and the second sub-text into the fourth sample pair. Further, the server can determine the first text pair, the second text pair, the first sample pair, the second sample pair, the third sample pair and the fourth sample pair as text sample pairs.

[0131] Optionally, the server may combine the first subtext, the third subtext, and the first target subtext into a first aligned text pair. Furthermore, the server may combine the second subtext, the fourth subtext, and the second target subtext into a second aligned text pair. The first aligned text pair and the second aligned text pair may be a three-way aligned text pair.

[0132] For easier understanding, see Figure 6 , Figure 6 This is a schematic diagram of a scenario for generating text sample pairs provided by an embodiment of the present application. Figure 6 The first text pair 60a shown may be the above Figure 5 The first text pair 50a in the corresponding embodiment is as follows: Figure 6 The second text pair 60b shown may be the above Figure 5 The corresponding embodiment is the second text pair 50b. Here, E1 and E2 can be subtexts of the first language type, A1 can be a subtext of the second language type, and Z1 can be a subtext of the third language type. Here, E1 can be the first subtext, E2 can be the second subtext, A1 can be the third subtext, and Z1 can be the fourth subtext.

[0133] like Figure 6 As shown, Z5 can be the first target subtext generated based on the first text pair 60a and the second text pair 60b, and A5 can be the second target subtext generated based on the first text pair 60a and the second text pair 60b. The first target subtext Z5 is associated with the semantic information of the first subtext E1 and the third subtext A1, and the second target subtext A5 is associated with the semantic information of the second subtext E2 and the fourth subtext Z1.

[0134] like Figure 6 As shown, the server can combine the first target subtext Z5 and the third subtext A1 into a first sample pair 60c, and combine the second target subtext A5 and the fourth subtext Z1 into a second sample pair 60d. Optionally, the server can also combine the first target subtext Z5 and the first subtext E1 into a third sample pair (not shown in the figure), and combine the second target subtext A5 and the second subtext E2 into a fourth sample pair (not shown in the figure). Furthermore, the server can determine the first text pair 60a, the second text pair 60b, the first sample pair 60c, the second sample pair 60d, the third sample pair, and the fourth sample pair as text sample pairs.

[0135] It can be seen that the embodiment of the present application can generate multi-way aligned corpora based on the steps of extraction and generation. The extraction step can determine the edit distance between the first subtext in the first text pair and the second subtext in the second text pair, and determine the first text pair and the second text pair corresponding to the edit distance that meets the similarity condition as candidate sentence pairs; the generation step can eliminate the semantic difference between the first subtext and the second subtext, and convert the "partially aligned" candidate sentence pairs into "completely aligned" text sample pairs. Based on this, through the above-mentioned steps based on extraction and generation, a large number of high-quality semantically aligned corpora (i.e., text sample pairs) can be generated quickly and accurately, thereby ensuring the quality of the corpus while increasing the quantity of the corpus.

[0136] Further, see Figure 7 , Figure 7 This is a flowchart of a text data processing method provided by an embodiment of the present application. The method can be executed by a server, or by a user terminal, or by both a server and a user terminal. The server can be the above-mentioned Figure 2 The corresponding implementation of the server 20a, the user terminal can be the above Figure 2 For ease of understanding, the present invention uses the method executed by the server as an example to illustrate the method. The text data processing method may include the following steps S201-S202:

[0137] Step S201, determining the edit distance between the first subtext and the second subtext, and if the edit distance satisfies a similarity condition, obtaining a set of network models associated with the first subtext and the second subtext;

[0138] Specifically, the server may extract a first unit text from the first subtext and a second unit text from the second subtext. Furthermore, the server may determine an edit distance between the first subtext and the second subtext based on the first unit text and the second unit text. Furthermore, if the edit distance satisfies a similarity condition, the server may obtain a set of network models associated with the first subtext and the second subtext.

[0139] The network model set may include a first target network model and a second target network model. The first target network model is obtained by training the first initial network model, and the second target network model is obtained by training the second initial network model. It is understood that the first target network model and the first initial network model are associated with the first language type and the third language type, while the second target network model and the second initial network model are associated with the first language type and the second language type.

[0140] Optionally, the network model set may further include a third target network model, a fourth target network model, and a fifth target network model. The third target network model is obtained after model training of the third initial network model, the fourth target network model is obtained after model training of the fourth initial network model, and the fifth target network model is obtained after model training of the fifth initial network model. It will be understood that the third target network model and the third initial network model are associated with the first language type and the fourth language type, the fourth target network model and the fourth initial network model are associated with the first language type and the fifth language type, and the fifth target network model and the fifth initial network model are associated with the first language type and the sixth language type.

[0141] It should be understood that the target network model in the network model set (for example, the first target network model, the second target network model) can be an NMT (Neural Machine Translation) model, and the embodiment of the present application does not limit the model type of the target network model. It is understandable that the specific process of training the initial network model (for example, the first initial network model, the second initial network model) to obtain the target network model (for example, the first target network model corresponding to the first initial network model, the second target network model corresponding to the second initial network model) can be found in the following Figure 9 The description of steps S302 to S305 in the corresponding embodiment.

[0142] Step S202 : Based on the network model set, a first target subtext associated with the semantic information of the first subtext and belonging to the third language type is generated, and a second target subtext associated with the semantic information of the second subtext and belonging to the second language type is generated.

[0143] Specifically, the server may concatenate the first subtext and the fourth subtext to obtain a first concatenated text. Furthermore, the server may input the first concatenated text into a first target network model, and generate a first target subtext associated with the semantic information of the first subtext and belonging to a third language type through the first target network model. Furthermore, the server may concatenate the second subtext and the third subtext to obtain a second concatenated text. Furthermore, the server may input the second concatenated text into a second target network model, and generate a second target subtext associated with the semantic information of the second subtext and belonging to a second language type through the second target network model.

[0144] The first target network model is based on the first concatenated text, and the output of the first target subtext can refer to the following formula (2):

[0145]

[0146] in, can represent the first concatenated text, m can represent the first target network model, It can represent the first target subtext.

[0147] The second target network model is based on the second concatenated text, and the output of the second target subtext can refer to the following formula (3):

[0148]

[0149] in, can represent the second concatenated text, m can represent the second target network model, It can represent the second target subtext.

[0150] It can be understood that in formula (2) and formula (3), m(x) means that x is taken as input and the decoding process of m is run to obtain the prediction result of the model. and The splicing is used as the model input to get the output result Generated and Semantic alignment (i.e. and The semantic information of Semantic alignment (i.e. and By associating the semantic information of and The splicing is used as the model input to get the output result Generated and Semantic alignment (i.e. and The semantic information of Semantic alignment (i.e. and semantic information).

[0151] It can be understood that when the accuracy of the model's prediction results is very high, "associated" and "same" have the same meaning, that is, the first target subtext has the same semantic information as the first subtext and the third subtext, and the second target subtext has the same semantic information as the second subtext and the fourth subtext.

[0152] Among them, the first target network model includes an encoder for encoding processing and a decoder for decoding processing. It should be understood that the specific process of the server generating the first target sub-text through the first target network model can be described as: the server can input the first concatenated text into the encoder in the first target network model, and the encoder (i.e., the encoder in the first target network model) encodes the first concatenated text to obtain the first feature vector corresponding to the first concatenated text. Further, the server can input the first feature vector into the decoder in the first target network model, and the decoder (i.e., the decoder in the first target network model) decodes the first feature vector to obtain the first text vector corresponding to the first feature vector. Further, the server can generate a first target sub-text that is associated with the semantic information of the first sub-text and belongs to the third language type based on the first text vector.

[0153] Similarly, the second target network model includes an encoder for encoding and a decoder for decoding. It should be understood that the specific process of the server generating the second target sub-text through the second target network model can be described as follows: the server can input the second concatenated text into the encoder in the second target network model, and the encoder (i.e., the encoder in the second target network model) encodes the second concatenated text to obtain a second feature vector corresponding to the second concatenated text. Furthermore, the server can input the second feature vector into the decoder in the second target network model, and the decoder (i.e., the decoder in the second target network model) decodes the second feature vector to obtain a second text vector corresponding to the second feature vector. Furthermore, the server can generate a second target sub-text that is associated with the semantic information of the second sub-text and belongs to the second language type based on the second text vector.

[0154] It is understood that the server can perform vector processing on the first concatenated text to obtain a first concatenated vector corresponding to the first concatenated text, and then input the second concatenated vector into the first target network model, and perform encoding processing on the first concatenated vector using the first target network model. Similarly, the server can perform vector processing on the second concatenated text to obtain a second concatenated vector corresponding to the second concatenated text, and then input the second concatenated vector into the second target network model, and perform encoding processing on the second concatenated vector using the second target network model.

[0155] For easier understanding, see Figure 8 , Figure 8 This is a schematic diagram of a scenario for generating a target subtext provided by an embodiment of the present application. Figure 8 As shown, the server can obtain a candidate text pair from the candidate set, where the candidate text pair can be a candidate text pair 80a. The candidate text pair 80a can be Figure 4The candidate text pair 41c in the corresponding embodiment. The target network model 80d may be a first target network model 80d, which may include a first encoder and a first decoder, and the target network model 80e may be a second target network model 80e, which may include a second encoder and a second decoder.

[0156] like Figure 8 As shown, the server can generate a concatenated text based on the candidate text pair 80a. The concatenated text here can include a first concatenated text and a second concatenated text. The first concatenated text here can be a first concatenated text 80b, and the second concatenated text here can be a second concatenated text 80c. The first concatenated text 80b can be expressed as "first subtext; fourth subtext", and the second concatenated text 80c can be expressed as "second subtext; third subtext".

[0157] like Figure 8 As shown, the server can input the first concatenated text 80b into the first target network model 80d and the second concatenated text 80c into the second target network model 80e. It can be understood that the first encoder in the first target network model 80d can output the first feature vector corresponding to the first concatenated text 80b, and the first decoder in the first target network model 80d can output the first text vector corresponding to the first feature vector, and then the first target subtext can be determined based on the first text vector. Similarly, the second encoder in the second target network model 80e can output the second feature vector corresponding to the second concatenated text 80c, and the second decoder in the second target network model 80e can output the second text vector corresponding to the second feature vector, and then the second target subtext can be determined based on the second text vector.

[0158] Optionally, if the edit distance does not meet the similarity condition, the server may determine that the first text pair and the second text pair do not meet the similarity condition, obtain a new first text pair (e.g., a third text pair) and a new second text pair (e.g., a fourth text pair) from the candidate set, and then perform the same operation on the third text pair and the fourth text pair as on the first text pair and the second text pair. The third text pair may include a fifth subtext belonging to the first language type and a seventh subtext belonging to the fourth language type, and the fourth text pair may include a sixth subtext belonging to the first language type and an eighth subtext belonging to the fifth language type. It should be understood that the embodiment of the present application does not limit the language type of the seventh subtext and the eighth subtext here.

[0159] It can be seen that the embodiment of the present application can eliminate semantic differences through the network model set, generate the first target subtext and the second target subtext based on the "partially aligned" first text pair and the second text pair, and then generate a completely aligned (Y 1 ,Y 2 ) Parallel sentence pairs in language and (Y 2 ,Y 1 ) Parallel sentence pairs in language This ensures that the quality of the corpus is guaranteed while increasing the quantity of the corpus.

[0160] Further, see Figure 9 , Figure 9 This is a flowchart of a text data processing method provided by an embodiment of the present application. The method can be executed by a server, or by a user terminal, or by both a server and a user terminal. The server can be the above-mentioned Figure 2 The corresponding implementation of the server 20a, the user terminal can be the above Figure 2 For ease of understanding, the present application embodiment takes the method executed by the server as an example for explanation. The text data processing method may include the following steps S301-S309:

[0161] Step S301: obtaining a first text pair and a second text pair, obtaining a first subtext from the first text pair, and obtaining a second subtext from the second text pair;

[0162] The first subtext and the second subtext both belong to the first language type; the first text pair also includes a third subtext that has the same semantic information as the first subtext and belongs to the second language type; the second text pair also includes a fourth subtext that has the same semantic information as the second subtext and belongs to the third language type.

[0163] Step S302: determining an edit distance between the first subtext and the second subtext, and if the edit distance satisfies a similarity condition, obtaining a first initial network model associated with the first language type and the third language type;

[0164] Optionally, the edit distance can be used to determine the text similarity between the first subtext and the second subtext. In the embodiment of the present application, other similarity measurement methods can also be used to determine the text similarity between the first subtext and the second subtext, such as sentence embedding, TF-IDF (term frequency–inverse document frequency, a commonly used weighting technique for information retrieval and data mining).

[0165] It is understood that when the edit distance satisfies the similarity condition, the server may use the first text pair and the second text pair that meet the similarity condition as candidate text pairs, and then add the candidate text pairs to the candidate set. Furthermore, the server may obtain a first initial network model associated with the first language type and the third language type, and then perform model training on the first initial network model in steps S303-S305 below based on the candidate text pairs in the candidate set. Similarly, the server may obtain a second initial network model associated with the first language type and the second language type, and then perform model training on the second initial network model based on the candidate text pairs in the candidate set.

[0166] The specific process of the server determining the edit distance between the first subtext and the second subtext can be found in the above Figure 3 The description of step S102 in the corresponding embodiment will not be repeated here.

[0167] Step S303: performing editing transformation on the fourth subtext to obtain a first transformed text associated with the fourth subtext;

[0168] Specifically, the server may generate a random transformation probability corresponding to each of the N unit texts, and determine the unit text whose random transformation probability falls within the transformable probability interval as the unit text to be edited. The fourth subtext includes N unit texts, where N can be a positive integer. Furthermore, the server may obtain an editing transformation mode associated with the unit text to be edited, perform an editing transformation on the unit text to be edited according to the editing transformation mode, obtain an editing transformation result, and generate a first transformed text associated with the fourth subtext based on the editing transformation result.

[0169] It is understandable that the server may perform word segmentation on the fourth subtext to obtain unit texts of the fourth subtext, and further determine that the number of unit texts is N. For example, when the fourth subtext is "How do you do?", the number of unit texts may be 5, and the 5 unit texts may be: "How", "do", "you", "do", "?".

[0170] It is understood that the server may generate a random transformation probability corresponding to each unit text, and the random transformation probability may be a randomly generated decimal between 0 and 1. For example, the random transformation probability corresponding to the unit text "How" may be G1 (e.g., G1 is equal to 0.1), the random transformation probability corresponding to the unit text "do" may be G2 (e.g., G2 is equal to 0.2), ..., and the random transformation probability corresponding to the unit text "?" may be G5 (e.g., G5 is equal to 0.7).

[0171] It is understandable that the transformable probability interval can belong to the interval [0,1]. For example, the transformable probability interval can be [0,0.25], [0.25,0.35]. For example, when the transformable probability interval is [0,0.25], if the five unit texts are: "How", "do", "you", "do", "?", then the random transformation probability G1 corresponding to the unit text "How" (the random transformation probability G1 is equal to 0.1) and the random transformation probability G2 corresponding to the unit text "do" (the random transformation probability G2 is equal to 0.2) are in the transformable probability interval. Therefore, the server can determine the unit text "How" and the unit text "do" among the five unit texts as the unit texts to be edited.

[0172] Alternatively, in other words, the server may obtain a transformation threshold associated with N unit texts, and then generate a random transformation probability corresponding to each unit text in the N unit texts, and determine the unit text with a random transformation probability less than the transformation threshold as the unit text to be edited. In this case, the transformable probability interval corresponding to the transformation threshold may be [0, transformation threshold).

[0173] It should be understood that the specific process of the server performing an edit transformation on the unit text to be edited according to the edit transformation mode and obtaining the edit transformation result can be described as follows: the server can obtain the edit transformation mode associated with the unit text to be edited. Furthermore, if the edit transformation mode is a replacement operation, the server can obtain a dictionary table corresponding to the third language type, obtain the first edit text from the dictionary table, replace the unit text to be edited with the first edit text, and obtain the edit transformation result. Furthermore, if the edit transformation mode is an insert operation, the server can obtain a second edit text from the dictionary table, insert the second edit text at an adjacent position of the unit text to be edited, and obtain the edit transformation result. Furthermore, if the edit transformation mode is a delete operation, the server can delete the unit text to be edited in the fourth subtext and obtain the edit transformation result.

[0174] The dictionary table, or word list, can also be called a lexicon. This dictionary is used to select the edited text to replace or insert during replace and insert operations. It is understood that different language types may correspond to different dictionary tables. Thus, the dictionary table corresponding to the third language type may be the first dictionary table, the dictionary table corresponding to the second language type may be the second dictionary table, and so on. Dictionary tables corresponding to other language types can be obtained accordingly.

[0175] Optionally, different language types may correspond to the same dictionary table, which may store edit texts of different language types, for example, a first edit text and a second edit text corresponding to a third language type, and a third edit text corresponding to a second language type. Therefore, when performing an edit transformation on the fourth subtext, the first edit text or the second edit text corresponding to the third language type may be obtained from the dictionary table; when performing an edit transformation on other subtexts, edit texts corresponding to other language types may be obtained from the dictionary table. For example, when performing an edit transformation on the third subtext, the third edit text corresponding to the second language type may be obtained from the dictionary table.

[0176] It is understood that the number of unit texts to be edited can be zero (i.e., 0), one (i.e., 1), or more. When the number of unit texts to be edited is zero, the server does not need to perform an edit transformation on the fourth subtext; when the number of unit texts to be edited is one, the server can perform an edit change on the one unit text to be edited to achieve an edit transformation on the fourth subtext; when the number of unit texts to be edited is multiple, the server can obtain the edit transformation methods corresponding to the multiple unit texts to be edited, and perform an edit transformation on each unit text to be edited based on the edit transformation method corresponding to each unit text to be edited, to achieve an edit transformation on the fourth subtext.

[0177] Among them, the editing transformation method corresponding to each unit text to be edited in the multiple unit texts to be edited can include replacement operation, insertion operation and deletion operation. The editing transformation method corresponding to each unit text to be edited in the multiple unit texts to be edited is independent and can be randomly the above-mentioned replacement operation, insertion operation or deletion operation.

[0178] It can be understood that the adjacent position of the unit text to be edited can be the previous position of the unit to be edited, or the next position of the unit text to be edited. The embodiment of the present application can be explained by taking the adjacent position as the previous position of the edited unit text as an example.

[0179] Step S304: splicing the second subtext and the first transformed text to obtain a first spliced ​​sample;

[0180] Step S305: obtaining a first sample vector of the first concatenated sample through the first initial network model, and performing model training on the first initial network model based on the first sample vector and the fourth subtext to obtain a first target network model;

[0181] Specifically, the server may obtain a first sample vector of the first concatenated sample through the first initial network model and, based on the first sample vector, generate a predicted sample subtext belonging to the third language type. Furthermore, the server may obtain a sample semantic similarity between the predicted sample subtext and the fourth subtext and, based on the sample semantic similarity, generate a model loss function for the first initial network model. Furthermore, the server may adjust parameters of the first initial network model based on the model loss function to obtain a first target network model.

[0182] Among them, the first initial network model includes an encoder for encoding processing and a decoder for decoding processing. It should be understood that the specific process of the server obtaining the first sample vector of the first spliced ​​sample through the first initial network model can be described as: the server can input the first spliced ​​sample into the encoder in the first initial network model, and encode the first spliced ​​sample through the encoder in the first initial network model to obtain the first encoding vector corresponding to the first spliced ​​sample. Furthermore, the server can input the first encoding vector into the decoder in the first initial network model, and decode the first encoding vector through the decoder in the first initial network model to obtain the first sample vector corresponding to the first encoding vector (i.e., the first sample vector of the first spliced ​​sample).

[0183] It should be understood that the specific process of the server adjusting the parameters of the first initial network model based on the model loss function (i.e., iteratively training the first initial network model) can be described as follows: when the model loss function of the first initial network model does not meet the model convergence condition, the server can adjust the model parameters of the first initial network model based on the model loss function that does not meet the model convergence condition. Furthermore, the server can determine the first initial network model after adjusting the model parameters as a transition network model, iteratively train the transition network model until the model loss function of the transition network model after iterative training meets the model convergence condition, and then use the transition network model that meets the model convergence condition as the first target network model.

[0184] It should be understood that for the aligned examples in the candidate set (i.e. candidate text pairs) Can generate training samples in, is the input of the first initial network model (i.e. the first splicing sample), “;” represents the splicing operation, is the prediction target of the first initial network model, It is through The sentence generated by the replacement operation, insertion operation and deletion operation (ie, the first transformed text) is Each word in the position has a probability of β (i.e., the transformation threshold) to perform replacement, insertion, or deletion operations. The first initial network model is trained by the constructed training corpus, and the prediction results of the model have the following two characteristics: the prediction results are consistent with the left part of the input (i.e., ) semantic alignment; the prediction result can be obtained by the right part of the input (i.e. ) is obtained through replacement, insertion and deletion operations.

[0185] Optionally, the server may train the second initial network model to obtain the second target network model as follows: the server may perform an edit transformation on the third subtext to obtain a second transformed text associated with the third subtext. Furthermore, the server may concatenate the first subtext and the second transformed text to obtain a second concatenated sample. Furthermore, the server may obtain a second sample vector of the second concatenated sample through the second initial network model, and perform model training on the second initial network model based on the second sample vector and the third subtext to obtain a second target network model.

[0186] It should be understood that the specific process of the server editing and transforming the third sub-text can be referred to the above description of editing and transforming the fourth sub-text, and will not be repeated here. It should be understood that the specific process of the server obtaining the second sample vector of the second spliced ​​sample can be referred to the above description of obtaining the first sample vector of the first spliced ​​sample, and will not be repeated here. It should be understood that the specific process of the server training the second initial network model based on the second sample vector and the third sub-text can be referred to the above description of training the first initial network model based on the first sample vector and the fourth sub-text, and will not be repeated here.

[0187] Step S306, obtaining a network model set associated with the first subtext and the second subtext;

[0188] It is understandable that the server may determine the first target network model and the second target network model obtained through training in the above steps S302 to S305 as a network model set.

[0189] The first initial network model and the first target network model can be collectively referred to as the first generalized model. The first initial network model and the first target network model are names for the first generalized network at different times. During the training phase, the first generalized network can be referred to as the first initial network model, and during the prediction phase, the first generalized network can be referred to as the first target network model. Similarly, the second initial network model and the second target network model can be collectively referred to as the second generalized model. The second initial network model and the second target network model are names for the second generalized network at different times. During the training phase, the second generalized network can be referred to as the second initial network model, and during the prediction phase, the second generalized network can be referred to as the second target network model.

[0190] Step S307: Based on the network model set, generating a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and generating a second target subtext associated with the semantic information of the second subtext and belonging to the second language type;

[0191] It can be understood that the first target subtext refers to the subtext that can be obtained by editing and transforming the first spliced ​​text, and the first target subtext has a similar structure and word composition to the fourth subtext; the second target subtext refers to the subtext that can be obtained by editing and transforming the second spliced ​​text, and the second target subtext has a similar structure and word composition to the third subtext. The first spliced ​​text is obtained by splicing the first subtext and the fourth subtext, and the second spliced ​​text is obtained by splicing the second subtext and the third subtext.

[0192] The specific process of the server generating the first target subtext and the second target subtext based on the network model set can be found in the above Figure 7 The description of step S202 in the corresponding embodiment will not be repeated here.

[0193] Step S308, generating a text sample pair according to the first text pair, the second text pair, the first target subtext, and the second target subtext;

[0194] Among them, step S302 to step S308 may specifically include three steps: training data generation step, model training step and model generation step. In these three steps, given a candidate sentence pair (x 1 ,y 1 ,x 2 ,y 2 ), generalized network model m, noise probability β (i.e., transformation threshold) and word list W b , which returns a semantically aligned corpus

[0195] In the model training step, the model m can be initialized. If the model m does not converge, the training data generation step is used to generate training data. When generating training data, Generate a random number α (i.e., random transformation probability) for each position of , if the random number α is less than the noise probability β, then based on the word list W b right The corresponding position of the random insertion operation, deletion operation or replacement operation is performed to obtain the first transformed text (i.e. ). Further, in the model training step, based on the first transformed text and Can generate Based on Model m is trained. It is understood that if model m converges, the model m is returned; if model m does not converge, training data can be generated through the training data generation step. For ease of understanding, the generated training data is described here using a single example (i.e., the first transformed text) as an example.

[0196] In the model generation step, Input into the trained model m, and the final Paired alignment example Then return It is understandable that the candidate sentence pair (x 1 ,y 1 ,x 2 ,y 2 ) can be multiple, and here, alignment examples corresponding to each candidate sentence pair are generated for multiple candidate sentence pairs. It should be understood that selecting candidate sentence pairs based on edit distance makes it easier to imitate the three operations corresponding to edit distance to eliminate semantic differences during the model generation step, which is highly operational.

[0197] It is understood that the text sample pairs can be used to train the initial translation model. That is, the text sample pairs can serve as the training corpus for training the initial translation model. The size of the training corpus is twice the size of the corpus in the candidate set. It should be understood that the specific process of training the initial translation model based on the text sample pairs can be seen in step S309 below.

[0198] Step S309 : obtaining an initial translation model associated with the text sample pair, iteratively training the initial translation model based on the text sample pair, and determining the iteratively trained initial translation model as a target translation model.

[0199] The target translation model is used to translate text between any two language types among the first language type, the second language type, and the third language type.

[0200] Optionally, the target translation model can also be used to translate text between any two of the first, fourth, fifth, and sixth language types. In other words, the target translation model can be used to translate text between any two of the first, second, third, fourth, fifth, and sixth language types.

[0201] Among them, the initial translation model and the target translation model here can be C-MNMT (Complete Multilingual Neural Machine Translation) models. The embodiment of the present application does not limit the model types of the initial translation model and the target translation model.

[0202] It should be understood that the initial translation model and the target translation model can be collectively referred to as the generalized translation model. The initial translation model and the target translation model are names for the generalized translation model at different times. During the training phase, the generalized translation model can be referred to as the initial translation model, and during the prediction phase, the generalized translation model can be referred to as the target translation model.

[0203] It is understood that the method proposed in this application can be experimented and verified on the public dataset WMT-5. The experimental results show that the EAG (Extract and Generate) algorithm proposed in this application generates much more corpus on the WMT-5 dataset than the corpus generated by the existing technology (i.e., the extraction-based method). Among them, on average, about 10 times more high-quality semantically aligned bilingual pairs are generated for each non-English language pair. Among them, the extraction-based method is a method in the prior art that generates aligned corpus by completely aligning, which is the method used by the C-MNMT model.

[0204] For easier understanding, see Figure 10 , Figure 10 This is a schematic diagram of a performance comparison scenario provided by an embodiment of the present application. Figure 10 Shown are the experimental results corresponding to the training corpus provided by the prior art (i.e., the first training corpus) and the training corpus provided by the embodiment of the present application (i.e., the second training corpus). For ease of understanding, the model obtained by training the initial translation model with the first training corpus is referred to as the first translation model, and the model obtained by training the initial translation model with the second training corpus is referred to as the second translation model (i.e., the target translation model).

[0205] Among them, this application selects C-MNMT as the baseline system, and the performance comparison between the EAG method and the C-MNMT method is as follows: Figure 10 As shown, experimental result 90a is the experimental result corresponding to the first translation model, and experimental result 90b is the experimental result corresponding to the second translation model. Figure 10 As can be seen, the EAG method significantly outperforms the C-MNMT method in most language pairs. In the non-English language, the EAG method achieves a BLEU (Bilingual Evaluation Understudy) score of +1.1 compared to the baseline system. The BLEU score is a machine translation evaluation method that can be used to automatically assess machine translation performance.

[0206] like Figure 10 As shown, both the first and second translation models can be used for translation between Czech, German, English, Spanish, French, and Russian. The experimental results fully demonstrate the effectiveness of this method, which can translate between any two language types, horizontally for different source languages ​​and vertically for different target languages. Furthermore, experimental results 90a show that the translation quality of the C-MNMT method for non-English translations lags behind that for English translations.

[0207] Among them, experimental results 90b show that when the source language is Czech (i.e. cs) and the target language is German (i.e. de), the BLEU value increases by +1.8; when the source language is Czech and the target language is English (i.e. en), the BLEU value decreases by -0.1; when the source language is Czech and the target language is Spanish (i.e. es), the BLEU value increases by +1.5; when the source language is Czech and the target language is French (i.e. fr), the BLEU value increases by +2.4; when the source language is Czech and the target language is Russian (i.e. ru), the BLEU value increases by +1.5.

[0208] Among them, Figure 10 The experimental results shown show that, with the exception of Czech and English as the source and target languages, the BLEU values ​​of all languages ​​improved to varying degrees compared to the existing techniques. The BLEU values ​​improved significantly when Czech was the source language and French was the target language, and when Russian was the source language and German was the target language.

[0209] It can be seen that the embodiment of the present application can extract and generate a multi-way aligned corpus (i.e., training corpus) for a generalized translation model (a fully multilingual neural network machine translation model, i.e., a C-MNMT model) through the EAG method. The EAG method is mainly divided into two steps: extraction and generation. The first step is to extract candidate sentence pairs using the edit distance. When the edit distance of the first language type (i.e., the English end) is less than the threshold (i.e., the similarity distance), the two sentence pairs (i.e., the first text pair and the second text pair) are added to the candidate set; the second step is to use the model generation method to eliminate semantic differences and convert the sentence pairs in the "partially aligned" candidate set into "fully aligned" training corpus (i.e., text sample pairs). Based on this, the method proposed in this application can be used in any task that requires alignment of training corpora of different language pairs. After the above process, a large amount of high-quality semantically aligned training corpus can be generated from the existing bilingual corpus, which can significantly improve the translation effect (i.e., performance) of the generalized translation model.

[0210] Further, see Figure 11 , Figure 11 1 is a structural diagram of a text data processing device provided in an embodiment of the present application. The text data processing device 1 may include: a text acquisition module 10, a text generation module 20, and a sample pair generation module 30; further, the text data processing device 1 may also include: a length determination module 40, a distance determination module 50, a first comparison module 60, a second comparison module 70, and a model training module 80;

[0211] A text acquisition module 10 is configured to acquire a first text pair and a second text pair, acquire a first subtext from the first text pair, and acquire a second subtext from the second text pair; the first subtext and the second subtext both belong to a first language type; the first text pair further includes a third subtext having the same semantic information as the first subtext and belonging to a second language type; the second text pair further includes a fourth subtext having the same semantic information as the second subtext and belonging to a third language type;

[0212] a text generation module 20 for determining an edit distance between a first subtext and a second subtext, and if the edit distance satisfies a similarity condition, generating a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and generating a second target subtext associated with the semantic information of the second subtext and belonging to the second language type;

[0213] The text generation module 20 includes: a text extraction unit 201, a distance determination unit 202, a target acquisition unit 203, and a text generation unit 204;

[0214] A text extraction unit 201 is configured to extract a first unit text from a first subtext and a second unit text from a second subtext;

[0215] The distance determining unit 202 is configured to determine an edit distance between the first subtext and the second subtext based on the first unit text and the second unit text.

[0216] A target acquisition unit 203 is configured to acquire a network model set associated with the first subtext and the second subtext if the edit distance satisfies a similarity condition;

[0217] The text generation unit 204 is configured to generate, based on the network model set, a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and a second target subtext associated with the semantic information of the second subtext and belonging to the second language type.

[0218] The network model set includes a first target network model and a second target network model; the first target network model is associated with the first language type and the third language type, and the second target network model is associated with the first language type and the second language type;

[0219] The text generation unit 204 includes: a first splicing subunit 2041, a first generating subunit 2042, a second splicing subunit 2043, and a second generating subunit 2044; optionally, the text generation unit 204 may further include: an initial acquisition subunit 2045, a transformation subunit 2046, a splicing processing subunit 2047, and a model training subunit 2048;

[0220] A first splicing subunit 2041 is configured to splice the first subtext and the fourth subtext to obtain a first spliced ​​text;

[0221] A first generating sub-unit 2042 is configured to input the first concatenated text into a first target network model, and generate a first target sub-text associated with the semantic information of the first sub-text and belonging to a third language type through the first target network model;

[0222] The first target network model includes an encoder for encoding and a decoder for decoding;

[0223] The first generating subunit 2042 includes: an encoding processing subunit 20421, a decoding processing subunit 20422, and a text generating subunit 20423;

[0224] The encoding processing subunit 20421 is configured to input the first concatenated text into an encoder in the first target network model, and perform encoding processing on the first concatenated text through the encoder to obtain a first feature vector corresponding to the first concatenated text;

[0225] The decoding processing subunit 20422 is configured to input the first feature vector into a decoder in the first target network model, and decode the first feature vector through the decoder to obtain a first text vector corresponding to the first feature vector;

[0226] The text generation sub-unit 20423 is configured to generate, based on the first text vector, a first target sub-text associated with the semantic information of the first sub-text and belonging to a third language type.

[0227] The specific implementation of the encoding processing subunit 20421, the decoding processing subunit 20422 and the text generation subunit 20423 can be found in the above Figure 7 The description of step S202 in the corresponding embodiment will not be repeated here.

[0228] The second splicing subunit 2043 is used to splice the second subtext and the third subtext to obtain a second spliced ​​text;

[0229] The second generating sub-unit 2044 is configured to input the second concatenated text into the second target network model, and generate a second target sub-text associated with the semantic information of the second sub-text and belonging to the second language type through the second target network model.

[0230] Optionally, the initial acquisition subunit 2045 is configured to acquire a first initial network model associated with the first language type and the third language type;

[0231] The transformation subunit 2046 is configured to perform editing transformation on the fourth subtext to obtain a first transformed text associated with the fourth subtext;

[0232] The fourth subtext contains N unit texts; N is a positive integer;

[0233] The transformation subunit 2046 includes: a probability generation subunit 20461 and an editing transformation subunit 20462;

[0234] The probability generating subunit 20461 is used to generate a random transformation probability corresponding to each unit text in the N unit texts, and determine the unit text whose random transformation probability is within the transformable probability interval as the unit text to be edited;

[0235] The editing transformation subunit 20462 is used to obtain the editing transformation method associated with the unit text to be edited, perform editing transformation on the unit text to be edited according to the editing transformation method, obtain the editing transformation result, and generate the first transformation text associated with the fourth subtext according to the editing transformation result.

[0236] The edit transformation subunit 20462 is specifically used to obtain the edit transformation mode associated with the unit text to be edited;

[0237] The edit transformation subunit 20462 is further specifically configured to, if the edit transformation mode is a replace operation, obtain a dictionary table corresponding to the third language type, obtain the first edit text from the dictionary table, replace the unit text to be edited with the first edit text, and obtain an edit transformation result;

[0238] The edit transformation subunit 20462 is further specifically configured to obtain a second edit text from the dictionary table if the edit transformation mode is an insert operation, insert the second edit text into a position adjacent to the unit text to be edited, and obtain an edit transformation result;

[0239] The editing transformation subunit 20462 is further specifically configured to delete the unit text to be edited in the fourth subtext to obtain an editing transformation result if the editing transformation mode is a deletion operation.

[0240] The specific implementation of the probability generation subunit 20461 and the editing transformation subunit 20462 can be found in the above Figure 9 The description of step S303 in the corresponding embodiment will not be repeated here.

[0241] The splicing processing subunit 2047 is used to splice the second subtext and the first transformed text to obtain a first spliced ​​sample;

[0242] The model training subunit 2048 is used to obtain the first sample vector of the first spliced ​​sample through the first initial network model, and perform model training on the first initial network model based on the first sample vector and the fourth subtext to obtain a first target network model.

[0243] The model training subunit 2048 includes: a text prediction subunit 20481, a loss generation subunit 20482, and a parameter adjustment subunit 20483;

[0244] The text prediction subunit 20481 is configured to obtain a first sample vector of the first concatenated sample through the first initial network model, and generate a predicted sample subtext belonging to the third language type based on the first sample vector;

[0245] a loss generating subunit 20482 for obtaining a sample semantic similarity between the predicted sample subtext and the fourth subtext, and generating a model loss function of the first initial network model based on the sample semantic similarity;

[0246] The parameter adjustment subunit 20483 is used to adjust the parameters of the first initial network model based on the model loss function to obtain a first target network model.

[0247] The specific implementation of the text prediction subunit 20481, the loss generation subunit 20482 and the parameter adjustment subunit 20483 can be found in the above Figure 9 The description of step S305 in the corresponding embodiment will not be repeated here.

[0248] The specific implementation of the first splicing subunit 2041, the first generating subunit 2042, the second splicing subunit 2043 and the second generating subunit 2044 can be found in the above Figure 7 The description of step S202 in the corresponding embodiment will not be repeated here. Figure 9 The description of steps S302 to S305 in the corresponding embodiment will not be repeated here.

[0249] The specific implementation of the text extraction unit 201 and the distance determination unit 202 can be found in the above Figure 3 The description of step S102 in the corresponding embodiment will not be repeated here. Figure 7 The description of step S201-step S202 in the corresponding embodiment will not be repeated here.

[0250] The sample pair generating module 30 is configured to generate a text sample pair according to the first text pair, the second text pair, the first target subtext, and the second target subtext.

[0251] The sample pair generation module 30 includes: a first combination unit 301, a second combination unit 302; a sample pair determination unit 303;

[0252] The first combining unit 301 is configured to combine the first target subtext and the third subtext into a first sample pair; the semantic information of the first target subtext is associated with the semantic information of the third subtext;

[0253] The second combining unit 302 is configured to combine the second target subtext and the fourth subtext into a second sample pair; the second target subtext is associated with the semantic information of the fourth subtext;

[0254] The sample pair determining unit 303 is configured to determine the first text pair, the second text pair, the first sample pair, and the second sample pair as a text sample pair.

[0255] The specific implementation of the first combination unit 301, the second combination unit 302 and the sample pair determination unit 303 can be found in the above Figure 3The description of step S103 in the corresponding embodiment will not be repeated here.

[0256] Optionally, a length determination module 40 is configured to determine the text length of the first subtext as a first text length, determine the text length of the second subtext as a second text length, and determine a target length corresponding to the similarity condition based on the first text length and the second text length;

[0257] a distance determination module 50 for obtaining a similarity parameter associated with a similarity condition and determining a similarity distance corresponding to the similarity condition based on the target length and the similarity parameter;

[0258] a first comparison module 60 for determining that the edit distance satisfies a similarity condition if the edit distance is less than or equal to the similarity distance;

[0259] The second comparison module 70 is configured to determine that the edit distance does not satisfy the similarity condition if the edit distance is greater than the similarity distance.

[0260] Optionally, the model training module 80 is used to obtain an initial translation model associated with the text sample pair, iteratively train the initial translation model based on the text sample pair, and determine the iteratively trained initial translation model as a target translation model; the target translation model is used to translate text between any two language types among the first language type, the second language type, and the third language type.

[0261] The specific implementation of the text acquisition module 10, the text generation module 20, the sample pair generation module 30, the length determination module 40, the distance determination module 50, the first comparison module 60 and the second comparison module 70 can be found in the above Figure 3 In the corresponding embodiment, steps S101 to S103, Figure 7 In the corresponding embodiment, steps S201-S202 and Figure 9 The description of steps S301 to S308 in the corresponding embodiment will not be repeated here. Figure 9 The description of step S309 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0262] Further, see Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 12As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. Optionally, the memory 1005 may also be at least one storage device located away from the aforementioned processor 1001. As Figure 12 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0263] In such Figure 12 In the computer device 1000 shown, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an interface for user input; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0264] Obtain a first text pair and a second text pair, obtain a first subtext from the first text pair, and obtain a second subtext from the second text pair; the first subtext and the second subtext both belong to a first language type; the first text pair further includes a third subtext having the same semantic information as the first subtext and belonging to the second language type; the second text pair further includes a fourth subtext having the same semantic information as the second subtext and belonging to the third language type;

[0265] determining an edit distance between the first subtext and the second subtext, and if the edit distance satisfies a similarity condition, generating a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and generating a second target subtext associated with the semantic information of the second subtext and belonging to the second language type;

[0266] A text sample pair is generated according to the first text pair, the second text pair, the first target subtext, and the second target subtext.

[0267] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 、 Figure 7 or Figure 9The description of the text data processing method in the corresponding embodiment can also be performed Figure 11 The description of the text data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0268] In addition, it should be noted that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the text data processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned text data processing device 1. Figure 3 、 Figure 7 or Figure 9 The description of the text data processing method in the corresponding embodiment will therefore not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0269] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or computer program, which may include computer instructions, which may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, so that the computer device performs the above Figure 3 、 Figure 7 or Figure 9 The description of the text data processing method in the corresponding embodiment will therefore not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated here. For technical details not disclosed in the computer program product or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0270] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0271] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A text data processing method, characterized in that: include: Obtaining a first text pair and a second text pair, obtaining a first subtext from the first text pair, and obtaining a second subtext from the second text pair; the first subtext and the second subtext both belong to a first language type; the first text pair further includes a third subtext having the same semantic information as the first subtext and belonging to a second language type; the second text pair further includes a fourth subtext having the same semantic information as the second subtext and belonging to a third language type; determining an edit distance between the first subtext and the second subtext, and if the edit distance satisfies a similarity condition, obtaining a set of network models associated with the first subtext and the second subtext; Based on the network model set, generating a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and generating a second target subtext associated with the semantic information of the second subtext and belonging to the second language type; generating a text sample pair according to the first text pair, the second text pair, the first target subtext, and the second target subtext; The first target subtext is obtained by training the first initial network model included in the network model set. The training of the first initial network model includes: obtaining the first initial network model associated with the first language type and the third language type; performing editing transformation on the fourth subtext to obtain a first transformed text associated with the fourth subtext; performing splicing processing on the second subtext and the first transformed text to obtain a first spliced ​​sample; obtaining a first sample vector of the first spliced ​​sample through the first initial network model, and performing model training on the first initial network model based on the first sample vector and the fourth subtext to obtain a first target network model. The prediction result obtained by the model training of the first initial network model is semantically aligned with the second subtext and allows it to be obtained from the first transformed text through replacement operations, insertion operations, and deletion operations.

2. The method according to claim 1, characterized in that The method further comprises: Determining the text length of the first subtext as a first text length, determining the text length of the second subtext as a second text length, and determining a target length corresponding to a similarity condition based on the first text length and the second text length; Acquire a similarity parameter associated with the similarity condition, and determine a similarity distance corresponding to the similarity condition according to the target length and the similarity parameter; If the edit distance is less than or equal to the similarity distance, determining that the edit distance satisfies the similarity condition; If the edit distance is greater than the similarity distance, it is determined that the edit distance does not satisfy the similarity condition.

3. The method according to claim 1, characterized in that The determining the edit distance between the first subtext and the second subtext includes: extracting a first unit text from the first subtext, and extracting a second unit text from the second subtext; An edit distance between the first subtext and the second subtext is determined based on the first unit text and the second unit text.

4. The method according to claim 1, wherein The network model set includes a first target network model and a second target network model; the first target network model is associated with the first language type and the third language type, and the second target network model is associated with the first language type and the second language type; The step of generating, based on the network model set, a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and generating a second target subtext associated with the semantic information of the second subtext and belonging to the second language type comprises: performing splicing processing on the first subtext and the fourth subtext to obtain a first spliced ​​text; Inputting the first concatenated text into the first target network model, and generating a first target subtext associated with the semantic information of the first subtext and belonging to the third language type through the first target network model; performing splicing processing on the second subtext and the third subtext to obtain a second spliced ​​text; The second concatenated text is input into the second target network model, and a second target subtext associated with the semantic information of the second subtext and belonging to the second language type is generated through the second target network model.

5. The method according to claim 4, characterized in that The first target network model includes an encoder for performing encoding processing and a decoder for performing decoding processing; Inputting the first concatenated text into the first target network model, and generating a first target subtext associated with the semantic information of the first subtext and belonging to the third language type through the first target network model, includes: Inputting the first concatenated text into an encoder in the first target network model, encoding the first concatenated text by the encoder to obtain a first feature vector corresponding to the first concatenated text; Inputting the first feature vector into a decoder in the first target network model, and decoding the first feature vector by the decoder to obtain a first text vector corresponding to the first feature vector; Based on the first text vector, a first target sub-text is generated that is associated with the semantic information of the first sub-text and belongs to the third language type.

6. The method according to claim 1, characterized in that Generating a text sample pair according to the first text pair, the second text pair, the first target subtext, and the second target subtext includes: Combining the first target subtext and the third subtext into a first sample pair; associating the semantic information of the first target subtext with the semantic information of the third subtext; combining the second target subtext and the fourth subtext into a second sample pair; associating the second target subtext with the semantic information of the fourth subtext; The first text pair, the second text pair, the first sample pair, and the second sample pair are determined as a text sample pair.

7. The method according to claim 1, characterized in that The fourth subtext includes N unit texts; N is a positive integer; The editing and transforming the fourth subtext to obtain a first transformed text associated with the fourth subtext includes: Generating a random transformation probability corresponding to each unit text in the N unit texts, and determining a unit text whose random transformation probability is within a transformable probability interval as a unit text to be edited; Obtain an editing transformation mode associated with the unit text to be edited, perform editing transformation on the unit text to be edited according to the editing transformation mode to obtain an editing transformation result, and generate a first transformation text associated with the fourth subtext according to the editing transformation result.

8. The method according to claim 7, characterized in that The acquiring of the editing transformation mode associated with the unit text to be edited, performing editing transformation on the unit text to be edited according to the editing transformation mode, and obtaining an editing transformation result includes: Obtaining an editing transformation mode associated with the unit text to be edited; If the editing transformation mode is a replacement operation, obtaining a dictionary table corresponding to the third language type, obtaining a first editing text from the dictionary table, replacing the unit text to be edited with the first editing text, and obtaining an editing transformation result; If the editing transformation mode is an insert operation, obtaining a second editing text from the dictionary table, inserting the second editing text into a position adjacent to the unit text to be edited, and obtaining an editing transformation result; If the editing transformation mode is a deletion operation, the unit text to be edited is deleted from the fourth subtext to obtain an editing transformation result.

9. The method according to claim 1, characterized in that The step of obtaining a first sample vector of the first spliced ​​sample by using the first initial network model, and performing model training on the first initial network model based on the first sample vector and the fourth subtext to obtain the first target network model includes: obtaining a first sample vector of the first concatenated sample through the first initial network model, and generating a predicted sample subtext belonging to the third language type based on the first sample vector; Obtaining a sample semantic similarity between the predicted sample subtext and the fourth subtext, and generating a model loss function of the first initial network model according to the sample semantic similarity; Parameters of the first initial network model are adjusted based on the model loss function to obtain the first target network model.

10. The method according to claim 1, characterized in that The method further comprises: An initial translation model associated with the text sample pair is obtained, the initial translation model is iteratively trained based on the text sample pair, and the iteratively trained initial translation model is determined as a target translation model; the target translation model is used to translate text between any two of the first language type, the second language type, and the third language type.

11. A text data processing device, characterized in that: include: a text acquisition module configured to acquire a first text pair and a second text pair, acquire a first subtext from the first text pair, and acquire a second subtext from the second text pair; the first subtext and the second subtext both belong to a first language type; the first text pair further includes a third subtext having the same semantic information as the first subtext and belonging to a second language type; and the second text pair further includes a fourth subtext having the same semantic information as the second subtext and belonging to a third language type; a text generation module configured to determine an edit distance between the first subtext and the second subtext, and if the edit distance satisfies a similarity condition, obtain a set of network models associated with the first subtext and the second subtext; and based on the set of network models, generate a first target subtext associated with the semantic information of the first subtext and belonging to the third language type, and generate a second target subtext associated with the semantic information of the second subtext and belonging to the second language type; a sample pair generating module, configured to generate a text sample pair based on the first text pair, the second text pair, the first target subtext, and the second target subtext; The first target subtext is obtained by training the first initial network model included in the network model set using the first target network model, and the text generation module includes a text generation unit, which includes: an initial acquisition subunit, configured to acquire a first initial network model associated with the first language type and the third language type; a transformation subunit, configured to perform editing transformation on the fourth subtext to obtain a first transformed text associated with the fourth subtext; a splicing processing subunit, configured to splice the second subtext and the first transformed text to obtain a first spliced ​​sample; a model training subunit, configured to obtain a first sample vector of the first spliced ​​sample through the first initial network model, and perform model training on the first initial network model based on the first sample vector and the fourth subtext to obtain a first target network model; The prediction result obtained by training the first initial network model is semantically aligned with the second subtext and can be obtained from the first transformed text through replacement, insertion and deletion operations.

12. A computer device, characterized in that: include: processor and memory; The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 10.

14. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 10 is performed.

Images