Text translation method and device, equipment and storage medium
By utilizing matching pairs to find matching reference text features during the text translation process, the problem of decreased translation quality caused by inference errors in the translation model is solved, thus improving the accuracy of the translation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2026-05-22
AI Technical Summary
In the process of translating long texts, inference errors in the translation model lead to a decline in the quality of subsequent translations, affecting the accuracy of the final translation result.
By extracting text features from the text to be translated, using a set of matching pairs to find matching reference text features, and determining the final text translation result based on the reference text features, the error in predictive vocabulary inference is reduced.
It improves the accuracy of text translation results and reduces the problem of poor translation quality caused by errors in predictive vocabulary inference.
Smart Images
Figure CN122072823A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of computer and artificial intelligence technology, and in particular to a text translation method, apparatus, device, and storage medium. Background Technology
[0002] With the continuous development of technology, research on Natural Language Processing (NLP) has received much attention. Machine Translation (MT) aims to achieve automatic conversion between different languages, enabling machines to understand and generate text content in different languages through NLP.
[0003] In related technologies, a translation model extracts features from the text to be translated, obtaining its textual features. Each time, the translation model can infer a word from the target sequence based on these textual features. The process involves generating a probability distribution for each word based on its textual features. Finally, the word with the highest probability from this probability distribution is selected as the predicted word for this inference. The resulting target sequence is the translated text.
[0004] In related technologies, when the text to be translated is a long text, if the translation model makes a reasoning error for a single word during the reasoning process, it will cause subsequent reasoning errors to occur one after another, affecting the final translation quality. Summary of the Invention
[0005] This application provides a text translation method, apparatus, device, and storage medium. The technical solutions provided by this application are as follows:
[0006] According to one aspect of the embodiments of this application, a text translation method is provided, the method comprising:
[0007] Get the text to be translated;
[0008] Extract text features from the text to be translated, the text features being used to indicate the semantic information of the text to be translated;
[0009] Based on the text features, a reference text feature that matches the text features is found in the matching pair set as the first reference text feature. The matching pair set includes at least one matching pair, and each matching pair includes a reference text feature and a reference word.
[0010] Based on the reference vocabulary corresponding to the first reference text features, the text translation result of the text to be translated is determined.
[0011] According to one aspect of the embodiments of this application, a text translation apparatus is provided, the apparatus comprising:
[0012] The text acquisition module is used to acquire the text to be translated.
[0013] The feature extraction module is used to extract text features of the text to be translated, and the text features are used to indicate the semantic information of the text to be translated;
[0014] The feature matching module is used to find a reference text feature that matches the text feature in a set of matching pairs, as the first reference text feature. The set of matching pairs includes at least one matching pair, and each matching pair includes a reference text feature and a reference word.
[0015] The result determination module is used to determine the text translation result of the text to be translated based on the reference vocabulary corresponding to the features of the first reference text.
[0016] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described text translation method.
[0017] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the above-described text translation method.
[0018] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program, the computer program being loaded and executed by a processor to implement the above-described text translation method.
[0019] The technical solutions provided in this application have at least the following beneficial effects:
[0020] After extracting the text features of the text to be translated, the matching reference text features are searched in the matching pair set based on the text features. Based on the reference words corresponding to the reference text features, the final text translation result is determined. This achieves more accurate selection of the predicted words corresponding to the text features, reduces the problem of poor translation quality due to inference errors of predicted words, and improves the accuracy of the text translation result. Attached Figure Description
[0021] Figure 1 This is a schematic diagram of the implementation environment of a solution provided in one embodiment of this application;
[0022] Figure 2This is a flowchart of a text translation method provided in one embodiment of this application;
[0023] Figure 3 This is a flowchart illustrating the reasoning and prediction of vocabulary provided in one embodiment of this application;
[0024] Figure 4 This is a flowchart illustrating the construction of a set of matching pairs according to one embodiment of this application;
[0025] Figure 5 This is a flowchart of an application text translation method provided in one embodiment of this application;
[0026] Figure 6 This is a block diagram of a text translation apparatus provided in one embodiment of this application;
[0027] Figure 7 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0029] Please refer to Figure 1 This diagram illustrates an implementation environment provided by an embodiment of this application. This implementation environment can be implemented as a computer system. The implementation environment may include: a terminal device 10 and a server 20.
[0030] Terminal device 10 includes, but is not limited to, mobile phones, tablets, smart voice interaction devices, game consoles, wearable devices, multimedia playback devices, PCs (Personal Computers), in-vehicle terminals, smart home appliances, AR (Augmented Reality) devices, VR (Virtual Reality) devices, and other electronic devices. In some embodiments, terminal device 10 may run a client of a target application. Optionally, the target application may be an application that needs to be downloaded and installed, or it may be in the form of a webpage or a mini-program; this application embodiment does not limit this.
[0031] In some embodiments, the target application described above is an application that requires machine translation. Exemplarily, the target application may include, but is not limited to: game applications, video applications, game engine applications, office tool applications, social applications, instant messaging applications, academic applications, educational applications, etc., and may also include other applications that require machine translation. This application embodiment does not limit this.
[0032] In some embodiments, the execution entity of the above-described text translation method can be either the terminal device 10 or the server 20. When the execution entity is the server 20, the server 20 translates the text to be translated, obtains the text translation result, and then sends the text translation result to the terminal device 10.
[0033] Server 20 is used to provide backend services for the client of the target application in terminal device 10. For example, server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but it is not limited to these.
[0034] Terminal device 10 and server 20 can communicate with each other via a network. This network can be a wired network or a wireless network.
[0035] Please refer to Figure 2 The diagram illustrates a flowchart of a text translation method provided in one embodiment of this application. The execution entity for each step of the method can be a computer device; for example, the computer device can be... Figure 1 The terminal device 10 in the illustrated implementation environment can also be a server 20. The method may include at least one of the following steps (210-240):
[0036] Step 210: Obtain the text to be translated.
[0037] The text to be translated refers to the input text that is converted from the source language to the target language in machine translation. The source language refers to the original language of the text to be translated. In some embodiments, the text to be translated may be in at least one of the following forms: words, phrases, sentences, paragraphs, articles, etc., and may also include other forms, which are not limited in this application embodiment. The target language refers to the final language after the text to be translated is translated. Optionally, the source language and the target language may be any of the following: Chinese, English, Vietnamese, Thai, Cantonese, French, German, Spanish, etc., and may also be other languages, which are not limited in this application embodiment.
[0038] In some embodiments, in the context of game localization, the text to be translated refers to at least one piece of text data included in the game. Game localization refers to the process of adapting a game from one language and cultural background to another. In the context of game localization, it is necessary to translate at least one piece of text data included in the game from a first language to a second language. It should be noted that the first language and the second language are two different languages. The text data in the game refers to information presented in text form used in the game, which may include, but is not limited to: game dialogue text, task description text, task guidance text, plot text, user interface (UI) text, help text, prompt text, etc., and may also include other text data, which are not limited in this embodiment.
[0039] Step 220: Extract text features from the text to be translated. These text features are used to indicate the semantic information of the text to be translated.
[0040] Text features refer to feature vectors used to represent the semantic, contextual, and positional information of the text to be translated. Extracting text features from the text to be translated refers to the process of converting the text into feature vectors. Semantic information refers to the meaning of words, phrases, or sentences in the text to be translated, and may include, but is not limited to, concepts, emotions, intentions, etc., and may also include other information; this embodiment of the application does not limit this. Contextual information refers to the context in which words, phrases, or sentences in the text to be translated exist. Positional information refers to the relative position of words, phrases, or sentences in the text to be translated.
[0041] In some embodiments, the methods for extracting text features from the text to be translated may include one of the following: word embeddings, deep learning models, N-gram models, bag-of-words models, neural network models, etc. Other methods may also be used to extract text features from the text to be translated; this application embodiment does not limit this approach. Optionally, the aforementioned deep learning model may be a pre-trained language model used to capture the contextual and semantic information of the text to be translated, or it may be other deep learning models used to extract text features; this application embodiment does not limit this approach. Optionally, the neural network model may be a model used to implement the word embedding method, or it may be other neural network models; this application embodiment does not limit this approach.
[0042] In some embodiments, the text features of the text to be translated are extracted by a translation model.
[0043] Translation models are used to convert text to be translated from a first language to a second language. In some embodiments, the translation model can be a Neural Machine Translation (NMT) model, a Statistical Machine Translation (SMT) model, or other types of translation models; this application embodiment does not limit the specific type of translation model. A Neural Machine Translation (NMT) model is a translation model that generates text translation results from the text to be translated using a deep neural network. In some embodiments, the NMT model is implemented based on any of the following neural network models: Sequence to Sequence (Seq2Seq) models, Attention Mechanisms, Transformer models, etc., and may also include other neural network architectures; this application embodiment does not limit the specific type of translation model. A Statistical Machine Translation (SMT) model is a translation model constructed through statistical analysis of a large amount of parallel corpora. In this application embodiment, unless otherwise specified, the translation model refers to a Neural Machine Translation (NMT) model.
[0044] In some embodiments, the translation model includes at least one hidden layer; the text to be translated is input into the translation model for processing, and the features input by the last hidden layer in the at least one hidden layer are determined as the text features of the text to be translated; or, the text to be translated is input into the translation model for processing, and the features output by the last hidden layer in the at least one hidden layer are determined as the text features of the text to be translated.
[0045] The translation model progressively extracts text features from the text to be translated through at least one hidden layer. In some embodiments, the hidden layer may include at least one of the following: a feedforward neural network layer, a self-attention layer, a convolutional layer, a recursive layer, etc., and may also include other types of hidden layers, which are not limited in this application embodiment.
[0046] In translation models, when there are multiple hidden layers, these layers are constructed hierarchically. Each layer consists of one hidden layer, and the input to each layer is the output of the layer above it. The hidden layer closest to the output layer is the last hidden layer.
[0047] An FFN (Feed-Forward Network) layer is a fully connected neural network layer used to process hidden layer vectors (i.e., features). In translation models implemented based on attention mechanisms or Transformer models, the translation model includes at least one feed-forward neural network layer. In translation models with multiple feed-forward neural network layers, these layers are constructed hierarchically. Each layer includes one feed-forward neural network layer, and the input of each layer is the output of the previous layer. It is important to note that the input of the first feed-forward neural network layer can be the text to be translated or the text to be translated after processing by other layers, depending on the structure of the translation model; this application does not limit this. That is, the input data of the feed-forward neural network layer can be at least one of the following: features output from the previous layer, the text to be translated, and the text to be translated after processing by other layers, etc., and the output of the feed-forward neural network layer is the feature. The hidden layer vectors input to the FFN layer typically represent abstract representations of words or phrases in the text to be translated, containing word meaning, grammar, and contextual information. After processing by the FFN layer, these hidden layer vectors can extract higher-level features and semantic information.
[0048] In some embodiments, the translation model is a transformer model based on an encoder-decoder structure, where both the encoder and decoder of the translation model include feedforward neural network layers. For the input text to be translated, the encoder in the translation model encodes the text into a context vector. Then, at least one feedforward neural network layer in the decoder of the translation model extracts deeper features layer by layer. Understandably, the features extracted by deeper feedforward neural network layers contain more complex, abstract, and high-level information.
[0049] Therefore, in this application, the input or output of the last hidden layer of at least one hidden layer included in the translation model is determined as the text features of the text to be translated, and the text features of the text to be translated contain more comprehensive information.
[0050] In some embodiments, before extracting text features from the text to be translated, the text to be translated is preprocessed to obtain preprocessed text to be translated. Preprocessing includes, but is not limited to, at least one of the following: text cleaning, word segmentation, stop word removal, handling missing values, spell correction, etc., and may also include other preprocessing operations, which are not limited in this embodiment.
[0051] Step 230: Based on the text features, find a reference text feature that matches the text features in the matching pair set as the first reference text feature. The matching pair set includes at least one matching pair, and each matching pair includes a reference text feature and a reference word.
[0052] A matching pair set is a collection that stores reference text features and reference words with corresponding mapping relationships. A matching pair includes a set of reference text features and reference words with corresponding mapping relationships. Reference text features are used to match text features to help determine the predicted words corresponding to the text features. Reference words refer to words or phrases predicted based on reference text features. Predicted words refer to the most suitable words inferred based on text features and the matching pair set. The matching pair set is used to find the reference text feature that best matches the text feature, thereby determining the predicted words corresponding to that text feature.
[0053] In some embodiments, the matching pair can be implemented as a key-value pair. A key-value pair consists of a pair of elements: a key and a value. The key is used to uniquely identify the value; each key is unique within a set and can be used to index and look up the corresponding value. The value is the data associated with the key. In this application, the key corresponds to a reference text feature in the matching pair, and the value pair should be a reference word in the matching pair.
[0054] In some embodiments, a set of matching pairs corresponding to a third language is determined based on the third language, where the third language refers to the language corresponding to the reference vocabulary of the matching pairs included in the set. In some embodiments, different languages may correspond to different sets of matching pairs, or different languages may correspond to the same set of matching pairs.
[0055] In some embodiments, the similarity between the text feature and the reference text features included in all or part of the matching pairs in the matching pair set is calculated respectively to obtain the similarity of at least one reference text feature; the reference text feature with the highest similarity among the at least one reference text feature is determined as the first reference text feature.
[0056] Similarity is used to measure the degree of similarity between two elements. In this application, similarity is used to measure the degree of similarity between text features and reference text features. It is understood that the higher the similarity, the more similar the text features and reference text features are; conversely, the lower the similarity, the less similar the text features and reference text features are. In some embodiments, the similarity between text features and reference text features can be calculated using any of the following methods: Euclidean distance, cosine similarity, Manhattan distance, etc. Other methods can also be used to calculate the similarity between text features and reference text features, and this application does not limit this approach.
[0057] Euclidean distance is used to measure the straight-line distance between two points in a multidimensional space. In this application, multidimensional space refers to the vector space in which the text features and the reference text features reside.
[0058] In some embodiments, the text features include the feature values of each of the n feature elements with respect to the text to be translated, and the reference text features include the feature values of the aforementioned n feature elements with respect to the reference text, where n is an integer greater than or equal to 1. The text features are subtracted from the reference text features to obtain n feature value differences. The squares of each of the n feature value differences are calculated, and the n squared values are summed to obtain the sum of the squared differences between the text features and the reference text features. Based on the sum of the squared differences between the text features and the reference text features, the Euclidean distance between the text features and the reference text features is determined. It should be noted that a larger Euclidean distance between the text features and the reference text features indicates a smaller similarity between them, and a smaller Euclidean distance indicates a greater similarity.
[0059] For example, assuming that both the text features and the reference text features are n-dimensional vectors, the text features can be represented as X = [x1, x2, ..., x...]. n The reference text features can be represented as Y = [y1, y2, ..., y]. n The Euclidean distance d(X,Y) between the text features and the reference text features is expressed as follows:
[0060]
[0061] Cosine similarity is used to measure the cosine value of the angle between two vectors. In this application, the two vectors refer to the text feature and the reference text feature.
[0062] In some embodiments, a dot product is performed on the text feature and the reference text feature to obtain the dot product of the text feature and the reference text feature; the modulus of the text feature and the reference text feature are calculated respectively to obtain a first modulus and a second modulus; based on the dot product, the first modulus and the second modulus, the cosine similarity between the text feature and the reference text feature is determined. It should be noted that the larger the cosine similarity between the text feature and the reference text feature, the greater the similarity between the text feature and the reference text feature; the smaller the cosine similarity between the text feature and the reference text feature, the smaller the similarity between the text feature and the reference text feature.
[0063] Based on the previous example, the cosine similarity c(X,Y) between the text features and the reference text features is expressed as follows:
[0064]
[0065] Manhattan distance is used to measure the difference between two points across various dimensions. In this application, two points represent the difference between text features and reference text features in a multidimensional space.
[0066] In some embodiments, the absolute values of the differences between the text features and the reference text features are subtracted and then summed to obtain the absolute values of the differences between the n feature values. The Manhattan distance between the text features and the reference text features is then calculated. It is important to note that a larger Manhattan distance indicates a lower similarity between the text features and the reference text features, and a smaller Manhattan distance indicates a higher similarity between the text features and the reference text features.
[0067] Based on the previous example, the Manhattan distance m(X,Y) between the text features and the reference text features is expressed as follows:
[0068] m(X,Y)=(x1-y1)+(x2-y2)+…+(x n -y n )
[0069] In some embodiments, when calculating the similarity between text features and a subset of matching pairs in the matching pair set, each including reference text features, these matching pairs can be determined by random selection or set by a person skilled in the art; this application embodiment does not limit this.
[0070] The above method determines the reference text feature most similar to the text feature by calculating the similarity between the text feature and the reference text feature of at least one matching pair in the matching pair set, thereby enabling more accurate selection of the predicted words corresponding to the text feature.
[0071] In some embodiments, the matching pair set includes K matching pair subsets, each matching pair subset including at least one matching pair, where K is an integer greater than or equal to 1; the similarity between the text feature and the central feature of each of the K matching pair subsets is calculated to obtain the similarity of the K matching pair subsets, and the central feature of the matching pair subset is used to indicate the matching pair subset; the matching pair subset with the highest similarity among the K matching pair subsets is determined as the target matching pair subset; the similarity between the text feature and the reference text feature of each of the at least one matching pair included in the target matching pair subset is calculated to obtain the similarity of at least one reference text feature.
[0072] A subset of matching pairs is a subset of a set of matching pairs. The central feature of a subset of matching pairs is the feature that best represents the reference text features included in that subset, and is used to distinguish different subsets of matching pairs. In some embodiments, at least one matching pair included in the set of matching pairs is uniformly divided into K subsets. In some embodiments, K subsets are obtained by clustering at least one matching pair included in the set of matching pairs. In some embodiments, the average value of the reference text features included in each subset of matching pairs is determined as its central feature.
[0073] Since the central features best represent the reference text features included in the subset of matching pairs, calculating the similarity between the text features and the K central features determines the subset of reference text features most likely to match the text features. In other words, the target subset of matching pairs refers to the reference text features most likely to match the text features. It's important to note that the method for calculating the similarity between the text features and the central features here can refer to the method for calculating the similarity between the text features and reference text features described above.
[0074] In some embodiments, the number of first reference text features may be one or at least two. When the number of first reference text features is one, the reference text feature with the highest similarity among at least one reference text feature is determined as the first reference text feature.
[0075] In some embodiments, the number of first reference text features is 'a'; the 'a' reference text features with the highest similarity among at least one reference text feature are determined as the first reference text features, where 'a' is an integer greater than or equal to 2. Optionally, 'a' is preset by those skilled in the art, and this embodiment does not limit this.
[0076] The above method divides at least one matching pair in the matching pair set into at least one subset of matching pairs. When searching for reference text features that match the text features, it first determines the target subset of matching pairs based on the central matching pairs of the at least one subset of matching pairs. Finally, it determines the matching reference text features from the target subset of matching pairs. When the number of matching pairs in the matching pair set is large, this method effectively reduces the number of similarity calculations and searches, avoids unnecessary global searches, and helps improve the accuracy and speed of finding matching reference text features.
[0077] Step 240: Determine the text translation result of the text to be translated based on the reference vocabulary corresponding to the features of the first reference text.
[0078] In some embodiments, the text translation result includes at least one predicted word, and different predicted words are determined at different times based on the reference words corresponding to the features of the first reference text.
[0079] When the translation model is a neural machine translation model, at least one predicted word included in the text translation result is obtained through reasoning, and each reasoning depends on the previously determined predicted word.
[0080] In some embodiments, the text to be translated includes M words to be translated and N predicted words, where M is an integer greater than or equal to 1 and N is an integer less than or equal to M. The reference word corresponding to the first reference text feature is determined as the (N+1)th predicted word of the text to be translated, resulting in an updated text to be translated. If the reference word corresponding to the first reference text feature is a terminator, the text translation result of the text to be translated is determined based on the (N+1)th predicted word. The terminator is used to indicate the end point of the text translation result. The (N+1)th predicted word includes N predicted words and the (N+1)th predicted word. If the reference word corresponding to the first reference text feature is not a terminator, the step of extracting the text features of the text to be translated is executed again.
[0081] The vocabulary to be translated is obtained after segmenting the text to be translated. It should be noted that the vocabulary to be translated and the predicted vocabulary can be at least one of the following: characters, words, or phrases. The vocabulary to be translated and the predicted vocabulary correspond one-to-one, but the number of characters, words, or phrases contained in the vocabulary to be translated and the predicted vocabulary can be different.
[0082] When the text to be translated includes M words to be translated and N predicted words, it indicates that the current step is the (N+1)th inference and prediction of words based on the text to be translated, and the previous N inferences have already yielded the aforementioned N predicted words. Each inference and prediction of words based on the text to be translated requires feature extraction and analysis by combining the original text to be translated and the already inferred predicted words.
[0083] In machine translation, terminators are used to indicate the end of a text sequence, and start symbols are used to indicate the beginning of a text sequence. A terminator is... <eos>It is an abbreviation for "End of Sequence," representing the end of a sequence. The start symbol is... <bos>"Beginning of Sequence" is an abbreviation for "Beginning of Sequence". It is understood that in this application, the text sequence refers to the result of text translation.
[0084] After the reasoning for the Mth predicted word is completed, the predicted word obtained through reasoning is processed to generate the final text translation result. In some embodiments, this processing may include, but is not limited to, adding punctuation marks, case conversion, formatting adjustments, etc., and may also include other processing, which is not limited in this application embodiment.
[0085] For example, please refer to Figure 3 This document illustrates a flowchart of the inference prediction vocabulary provided in one embodiment of this application. The matching pair set is constructed based on the three sentences "I love you", "I like you", and "I like playing football". The original text to be translated is "I like playing football". The translation model decodes its corresponding English translation. The output at the first time step is decoded as "I", and the input at the second time step is "I like playing football, I". At the second time step, after feature extraction of "I like playing football, I", the resulting F("I like playing football, I") is matched against at least one reference text feature included in the matching pair set. First, the Euclidean distance between F("I like playing football, I") and at least one reference text feature in the matching pair set is calculated. The two reference text features with the closest Euclidean distance correspond to the reference words "love" and "like", with Euclidean distances of 0.2 and 0.8 respectively. Considering the above distance selection, we adopt the proximity principle and select "like" as the optimal prediction vocabulary. Therefore, the prediction vocabulary for "I like playing football, I" at the second time step is selected as "like".
[0086] The decoding process of the translation model is an incremental, continuous decoding process. Therefore, the above process evolves step by step until the translation inference of the text to be translated is completed. The result is: 'I like football'.
[0087] It is important to note that the logic corresponding to steps 210 to 240 is implemented in all of the translation models described above. In other words, the translation model can complete the translation of the text to be translated and obtain the translation result by executing steps 210 to 240.
[0088] In summary, the technical solution provided in this application, after extracting the text features of the text to be translated, searches for matching reference text features in the matching pair set based on the text features, and determines the final text translation result based on the reference words corresponding to the reference text features. This achieves more accurate selection of predicted words corresponding to text features, reduces the problem of poor translation quality of the final text translation result due to errors in prediction word inference, and improves the accuracy of the text translation result.
[0089] The following describes the process of constructing a set of matching pairs.
[0090] Please refer to Figure 4 This document illustrates a flowchart of constructing a matching pair set according to an embodiment of this application. In some embodiments, the process of constructing the matching pair set includes at least one of the following steps (410-430):
[0091] Step 410: Obtain at least one corpus of data, each corpus of data including a reference text and a reference translation text.
[0092] Corpus data refers to clean data used to generate matching pairs; clean data means data that has been cleaned, filtered, and optimized. In some embodiments, the source of at least one corpus data may include at least one of the following: a corpus of a specified domain, a centralized public dataset, a cultural background dataset, etc., and may also be other sources, which are not limited in this application embodiment. Reference translation text refers to a standard translation of a reference text converted from a first language to a second language. Optionally, the reference translation text is obtained by human translation of the reference text by relevant technical personnel.
[0093] A designated domain refers to a specific field, such as, but not limited to, the gaming field, the academic field, the sports field, and the e-commerce field. The designated domain can be any of these fields. For example, if the designated domain is the gaming field, a series of matching pairs related to the gaming field can be generated based on a corpus of gaming data. Similarly, if the designated domain is the academic field, a series of matching pairs related to the academic field can be generated based on a corpus of academic data.
[0094] A domain-specific corpus refers to a corpus of language data stored within that specific domain. This domain-specific corpus data records various information related to that domain. Optionally, a domain-specific corpus can be constructed based on domain-specific corpus data; for example, all or part of the corpus data in a domain can be used to create the corpus. For instance, corpus data in the gaming domain could include game guides, game news, information about relevant game characters and skills, etc. The methods for collecting corpus data are numerous; for example, it can be obtained from news articles, blogs, academic papers, video comments and bullet screens on video platforms, or automatically extracted from video and audio using speech recognition. This application does not limit the specific types and sources of data included.
[0095] Furthermore, the aforementioned designated domain can also be a sub-domain within a broader domain. For example, within the gaming domain, it can be divided into different sub-domains based on game type, such as shooting games, MOBA (Multiplayer Online Battle Arena) games, and auto-chess games. This way, different matchmaking pairs can be generated for different game types.
[0096] Step 420: Generate at least one matching pair based on at least one corpus data.
[0097] In some embodiments, at least one matching pair can be generated based on the reference text of any one of the above-mentioned corpus data.
[0098] In some embodiments, a matching pair may further include reference text and reference vocabulary. The process of generating at least one matching pair based on a corpus of data involves: performing word segmentation on the reference text and the reference translation text in the corpus data, respectively, such that the reference text includes at least one word to be translated, and the reference translation text includes at least one reference vocabulary. Optionally, the correspondence between the at least one word to be translated included in the reference text and the at least one reference vocabulary included in the reference translation text is annotated by those skilled in the art, and this embodiment does not limit this. During the process of successively extracting reference text from reference text, the reference text or reference text features are determined as the key of a matching pair, and the reference vocabulary corresponding to the current reference text features is determined as the value of the matching pair.
[0099] The process of generating at least one matching pair based on a corpus of data is also a process of progressively determining reference text features and reference vocabulary. In some embodiments, the reference vocabulary corresponding to the extracted reference text features is determined based on the correspondence between at least one word to be translated included in the reference text and at least one reference word included in the reference translation text. Therefore, a matching pair can be generated each time reference text features are extracted. It should be noted that after each matching pair is generated, the determined reference vocabulary is added to the reference text to obtain an updated reference text, which is used to determine the reference vocabulary for the next iteration. That is, a matching pair can be represented as {reference text features, reference vocabulary} or {reference text, reference vocabulary}.
[0100] For example, when the matching pair is represented as {reference text, reference vocabulary}, taking "I love you" as the reference text and "I love you" as the reference translation text, the four matching pairs generated based on this corpus data are as follows:
[0101] {"I love you, <bos>","I"}
[0102] {"I love you, <bos>I”,"love”}
[0103] {"I love you, <bos>I love,"U"}
[0104] {"I love you, <bos>I love U”," <eos>}
[0105] When a matching pair is represented as {reference text, reference vocabulary}, at least one set of mapping relationships between reference text and reference vocabulary is generated based on the correspondence between at least one word to be translated included in the reference text and at least one reference word included in the reference translation text. The corresponding reference text can be matched in the matching pair set based on the text to be translated to obtain the reference vocabulary corresponding to the matched reference text.
[0106] In some embodiments, the reference translation text includes at least one reference word; for a first reference text in a first corpus of at least one corpus, the reference text features of the first reference text at a first time point are extracted to obtain second reference text features; among the at least one reference word included in the reference translation text of the first corpus, a reference word that matches the second reference text features is found and used as the second reference word; a matching pair is generated based on the second reference text features and the second reference word.
[0107] The first reference text is any reference text from any of the at least one corpus data. It is understood that in the process of generating at least one matching pair based on the first reference text, at least one process of determining reference words is required. Each determination of reference words corresponds to a moment in the process of progressively extracting the reference text features of the first reference text. That is, the process of progressively extracting the reference text features of the first reference text includes at least one moment. The first moment is any one of the aforementioned at least one moment. Optionally, the moment can be a time step.
[0108] In some embodiments, a second reference word is added to a first reference text to obtain an updated first reference text, which is used to determine the reference word at a second time point; wherein the first time point is earlier than the second time point.
[0109] At each time step, features are extracted from the first reference text, and the corresponding reference words are determined. These reference words are then added to the first reference text to obtain the updated first reference text. When extracting reference text features from the first reference text at the next time step, feature extraction is performed on the updated first reference text.
[0110] In some embodiments, the second reference text feature is obtained by a translation model, which includes at least one hidden layer; the first reference text is input into the translation model for processing, and the feature input by the last hidden layer in the at least one hidden layer is determined as the second reference text feature; or, the first reference text is input into the translation model for processing, and the feature output by the last hidden layer in the at least one hidden layer is determined as the second reference text feature.
[0111] The process of extracting the reference text features of the first reference text is similar to the process of extracting the reference text features of the text to be translated, and can be referred to the corresponding part above.
[0112] For example, when the matching pair is represented as {reference text features, reference vocabulary}, taking "I love you" as the reference text and "I love you" as the reference translation text, the four matching pairs generated based on this corpus data are as follows:
[0113] {F(I love you, <bos>),"I"}
[0114] {F(I love you, <bos>I),"love"}
[0115] {F(I love you, <bos>I love), "U"}
[0116] {F(I love you, <bos>I love U)," <eos>”}
[0117] Among them, F(*) represents the reference text features corresponding to the reference text *.
[0118] In the case where the matching pair is represented as {reference text, reference vocabulary}, although the process of determining the text translation result based on the set of matching pairs can also be achieved, {reference text, reference vocabulary} only represents the most superficial mapping relationship between vocabulary and vocabulary, and cannot learn the deep representation of vocabulary. For example, when the input is "我好爱你” (I love you very much), due to the above-mentioned mapping relationship of the matching pair, it is very likely to be translated as "I good love U”, resulting in a rather rigid text translation result and grammar errors.
[0119] However, in the case where the matching pair is represented as {reference text features, reference vocabulary}, when determining the text translation result of the text to be translated, by comparing the text features of the text to be translated and the reference text features of at least one matching pair in the set of matching pairs, at least one predicted vocabulary in the text translation result can be more accurately determined. That is to say, by comparing the text features of the text to be translated and the reference text features of at least one matching pair in the set of matching pairs, the true context meaning of "我爱你” (I love you) can be obtained. Naturally, when the input is "我好爱你” (I love you very much), it can be understood as a further expression of "我爱你” (I love you). Then, the worst result is also to translate it as "I love U”, not "I good love U”. If the set of matching pairs is powerful enough, the text to be translated can be perfectly translated as "I really love U”.
[0120] Step 430, generate a set of matching pairs based on at least one matching pair.
[0121] In some embodiments, index at least one matching pair to obtain a set of matching pairs. In some embodiments, the following any one of the ways can be used to index at least one matching pair included in the set of matching pairs: hash index, tree index, inverted index, etc. Other index methods can also be used to establish an index, and the embodiments of the present application do not limit this.
[0122] In some embodiments, select K matching pairs from at least one matching pair as K central matching pairs, where K is an integer greater than or equal to 1; cluster at least one matching pair based on the K central matching pairs to obtain K subsets of matching pairs, and each subset of matching pairs includes at least one matching pair; determine the central features of each of the K subsets of matching pairs based on the K subsets of matching pairs; generate a set of matching pairs based on the K subsets of matching pairs and the central features of each of the K subsets of matching pairs.
[0123] Clustering is used to divide at least one matching pair into several subsets of matching pairs based on similarity. In some embodiments, the clustering method may include, but is not limited to, k-means clustering, hierarchical clustering, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), and other clustering methods, which are not limited in this application embodiment.
[0124] In some embodiments, the average feature of each of the K matching pair subsets is calculated; the average feature of each of the K matching pair subsets is determined as the central feature of each of the K matching pair subsets. The average feature of a matching pair subset refers to the feature value obtained by averaging the reference text features of at least one matching pair included in the matching pair subset.
[0125] In some embodiments, an index is created on K subsets of matching pairs and at least one matching pair included in the K subsets of matching pairs to obtain a set of matching pairs.
[0126] The above method, by dividing at least one matching pair into K subsets of matching pairs, only requires searching a small number of the most relevant subsets of matching pairs when looking for reference text features most similar to the text features, which greatly reduces the number of matching attempts and significantly improves the search speed.
[0127] The following describes the complete process of applying the above text translation method.
[0128] Please refer to Figure 5 The diagram illustrates a flowchart of an application text translation method provided in one embodiment of this application.
[0129] The process of applying text translation methods consists of two main parts: the first part is the process of constructing a set of matching pairs, and the second part is the process of translating the text to be translated based on the set of matching pairs.
[0130] In the first part, at least one corpus of data is input into the translation model, and after passing through at least one hidden layer, at least one reference text feature is extracted. Then, based on the at least one reference text feature, at least one matching pair is established. By indexing the at least one matching pair, a set of matching pairs can be obtained.
[0131] In the second part, the text to be translated is input into the translation model, which extracts the text features of the text to be translated. Based on the text features of the text to be translated, the translation model searches for the reference text features most similar to the text features of the text to be translated in the matching pair set. Combining the reference words corresponding to the text to be translated and the determined reference text features, at least one predicted word included in the text translation result is gradually determined, and finally the text translation result is obtained.
[0132] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.
[0133] Please refer to Figure 6 This diagram illustrates a block diagram of a text translation apparatus according to an embodiment of this application. The apparatus has the functions described above, which can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the computer device described above, or it can be installed within a computer device. Figure 6 As shown, the device 600 may include a text acquisition module 610, a feature extraction module 620, a feature matching module 630, and a result determination module 640.
[0134] The text acquisition module 610 is used to acquire the text to be translated.
[0135] The feature extraction module 620 is used to extract text features of the text to be translated, and the text features are used to indicate the semantic information of the text to be translated.
[0136] The feature matching module 630 is used to find a reference text feature that matches the text feature in a matching pair set as a first reference text feature, the matching pair set including at least one matching pair, each matching pair including a reference text feature and a reference word.
[0137] The result determination module 640 is used to determine the text translation result of the text to be translated based on the reference vocabulary corresponding to the features of the first reference text.
[0138] In some embodiments, the feature matching module 630 includes a similarity calculation submodule and a feature matching submodule (in... Figure 6 (Not shown in the image).
[0139] The similarity calculation submodule is used to calculate the similarity between the text feature and the reference text features included in all or part of the matching pairs in the matching pair set, so as to obtain the similarity of at least one reference text feature.
[0140] The feature matching submodule is used to determine the reference text feature with the highest similarity among the at least one reference text feature as the first reference text feature.
[0141] In some embodiments, the matching pair set includes K matching pair subsets, each matching pair subset including at least one matching pair, where K is an integer greater than or equal to 1; the similarity calculation submodule is used to calculate the similarity between the text feature and the central feature of each of the K matching pair subsets, to obtain the similarity of the K matching pair subsets, where the central feature of the matching pair subset is used to indicate the matching pair subset; the matching pair subset with the highest similarity among the K matching pair subsets is determined as the target matching pair subset; and the similarity between the text feature and the reference text feature of each of the at least one matching pair included in the target matching pair subset is calculated, to obtain the similarity of the at least one reference text feature.
[0142] In some embodiments, the text features of the text to be translated are extracted by a translation model, which includes at least one hidden layer; the feature extraction module 620 is used to input the text to be translated into the translation model for processing, and determine the features input by the last hidden layer of the at least one hidden layer as the text features of the text to be translated; or, input the text to be translated into the translation model for processing, and determine the features output by the last hidden layer of the at least one hidden layer as the text features of the text to be translated.
[0143] In some embodiments, the text to be translated includes M words to be translated and N predicted words, where M is an integer greater than or equal to 1 and N is an integer less than or equal to M; the result determination module 640 is used to determine the reference word corresponding to the first reference text feature as the (N+1)th predicted word of the text to be translated, to obtain an updated text to be translated; if the reference word corresponding to the first reference text feature is a terminator, the text translation result of the text to be translated is determined based on the (N+1)th predicted word, where the terminator is used to indicate the end point of the text translation result, and the (N+1)th predicted word includes the N predicted words and the (N+1)th predicted word; if the reference word corresponding to the first reference text feature is not a terminator, the step of extracting the text features of the text to be translated is executed again.
[0144] In some embodiments, the apparatus 600 further includes a data acquisition module, a matching pair generation module, and a matching pair set generation module (in... Figure 6 (Not shown in the image).
[0145] The data acquisition module is used to acquire at least one corpus of data, each corpus of data including a reference text and a reference translation text.
[0146] A matching pair generation module is used to generate the at least one matching pair based on the at least one corpus data.
[0147] A matching pair set generation module is used to generate the matching pair set based on the at least one matching pair.
[0148] In some embodiments, the reference translation text includes at least one reference word; the matching pair generation module is configured to extract reference text features of the first reference text at a first time moment for the first reference text in the first corpus data of the at least one corpus data to obtain second reference text features; find reference words that match the second reference text features among the at least one reference word included in the reference translation text of the first corpus data, and use them as second reference words; and generate a matching pair based on the second reference text features and the second reference words.
[0149] In some embodiments, the matching pair generation module is further configured to add the second reference vocabulary to the first reference text to obtain an updated first reference text, wherein the updated first reference text is used to determine the reference vocabulary at the second time point; wherein the first time point is earlier than the second time point.
[0150] In some embodiments, the second reference text feature is obtained by a translation model, which includes at least one hidden layer; the matching pair generation module is used to input the first reference text into the translation model for processing, and determine the feature input by the last hidden layer in the at least one hidden layer as the second reference text feature; or, input the first reference text into the translation model for processing, and determine the feature output by the last hidden layer in the at least one hidden layer as the second reference text feature.
[0151] In some embodiments, the matching pair set generation module is configured to select K matching pairs from the at least one matching pair as K center matching pairs, where K is an integer greater than or equal to 1; cluster the at least one matching pair based on the K center matching pairs to obtain K matching pair subsets, each matching pair subset including at least one matching pair; determine the central features of each of the K matching pair subsets based on the K matching pair subsets; and generate the matching pair set based on the K matching pair subsets and their respective central features.
[0152] In summary, the technical solution provided in this application, after extracting the text features of the text to be translated, searches for matching reference text features in the matching pair set based on the text features, and determines the final text translation result based on the reference words corresponding to the reference text features. This achieves more accurate selection of predicted words corresponding to text features, reduces the problem of poor translation quality of the final text translation result due to errors in prediction word inference, and improves the accuracy of the text translation result.
[0153] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0154] Please refer to Figure 7 This diagram illustrates a structural block diagram of a computer device 700 provided in one embodiment of this application. The computer device 700 may be... Figure 1 The terminal device 10 in the implementation environment shown can also be Figure 1 The server 20 in the illustrated implementation environment is used to implement the text translation method provided in the above embodiments. Specifically:
[0155] Typically, computer device 700 includes a processor 710 and a memory 720.
[0156] Processor 710 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 710 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 710 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 710 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 710 may also include an AI processor for handling computational operations related to machine learning.
[0157] The memory 720 may include one or more computer-readable storage media, which may be non-transitory. The memory 720 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 720 are used to store a computer program configured to be executed by one or more processors to implement a text translation method.
[0158] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.
[0159] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program, when executed by a processor, implements the above-described text translation method. Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).
[0160] In an exemplary embodiment, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the text translation method described above.
[0161] It should be noted that the collection and processing of relevant data (such as reference texts) in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.
[0162] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in the order shown in the figures, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the figures. This application does not limit this.
[0163] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.< / eos> < / bos> < / bos> < / bos> < / bos> < / eos> < / bos> < / bos> < / bos> < / bos> < / bos> < / eos>
Claims
1. A text translation method, characterized in that, The method includes: Get the text to be translated; Extract text features from the text to be translated, the text features being used to indicate the semantic information of the text to be translated; Based on the text features, a reference text feature that matches the text features is found in the matching pair set as the first reference text feature. The matching pair set includes at least one matching pair, and each matching pair includes a reference text feature and a reference word. Based on the reference vocabulary corresponding to the first reference text features, the text translation result of the text to be translated is determined.
2. The method according to claim 1, characterized in that, The step of searching for a reference text feature that matches the text feature in the matching pair set, as the first reference text feature, includes: Calculate the similarity between the text feature and the reference text features included in all or part of the matching pairs in the matching pair set, respectively, to obtain the similarity of at least one reference text feature; The reference text feature with the highest similarity among the at least one reference text feature is determined as the first reference text feature.
3. The method according to claim 2, characterized in that, The set of matching pairs includes K subsets of matching pairs, each subset of matching pairs includes at least one matching pair, where K is an integer greater than or equal to 1; The step of calculating the similarity between the text feature and the reference text features of all or part of the matching pairs in the matching pair set to obtain the similarity of at least one reference text feature includes: The similarity between the text feature and the central feature of each of the K matching pair subsets is calculated to obtain the similarity of the K matching pair subsets. The central feature of the matching pair subset is used to indicate the matching pair subset. The subset of matching pairs with the highest similarity among the K subsets of matching pairs is determined as the target subset of matching pairs. The similarity between the text feature and the reference text feature of each of the at least one matching pair included in the target matching pair subset is calculated to obtain the similarity of the at least one reference text feature.
4. The method according to any one of claims 1 to 3, characterized in that, The text features of the text to be translated are extracted by a translation model, which includes at least one hidden layer. The extraction of text features from the text to be translated includes: The text to be translated is input into the translation model for processing, and the features input from the last hidden layer in the at least one hidden layer are determined as the text features of the text to be translated. or, The text to be translated is input into the translation model for processing, and the features output by the last hidden layer in the at least one hidden layer are determined as the text features of the text to be translated.
5. The method according to any one of claims 1 to 4, characterized in that, The text to be translated includes M words to be translated and N predicted words, where M is an integer greater than or equal to 1 and N is an integer less than or equal to M. The step of determining the text translation result of the text to be translated based on the reference vocabulary corresponding to the first reference text features includes: The reference words corresponding to the first reference text features are determined as the N+1th predicted words of the text to be translated, and the updated text to be translated is obtained. When the reference word corresponding to the first reference text feature is a terminator, the text translation result of the text to be translated is determined based on N+1 predicted words. The terminator is used to indicate the end point of the text translation result. The N+1 predicted words include the N predicted words and the N+1 predicted words. If the reference word corresponding to the first reference text feature is not a terminator, the step of extracting the text features of the text to be translated is executed again.
6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Acquire at least one corpus of data, each corpus of data including a reference text and a reference translation text; Based on the at least one corpus data, generate the at least one matching pair; Based on the at least one matching pair, the set of matching pairs is generated.
7. The method according to claim 6, characterized in that, The reference translation text includes at least one reference term; The step of generating the at least one matching pair based on the at least one corpus data includes: For a first reference text in the first corpus data of the at least one corpus data, extract the reference text features of the first reference text at a first time point to obtain the second reference text features; In the reference translation text of the first corpus data, a reference word that matches the features of the second reference text is found and used as the second reference word. A matching pair is generated based on the second reference text features and the second reference vocabulary.
8. The method according to claim 7, characterized in that, The method further includes: The second reference vocabulary is added to the first reference text to obtain the updated first reference text, which is used to determine the reference vocabulary at the second moment. The first moment is earlier than the second moment.
9. The method according to claim 7 or 8, characterized in that, The second reference text features are extracted through a translation model, which includes at least one hidden layer; The step of extracting the reference text features of the first reference text at the first moment to obtain the second reference text features includes: The first reference text is input into the translation model for processing, and the feature input from the last hidden layer in the at least one hidden layer is determined as the feature of the second reference text. or, The first reference text is input into the translation model for processing, and the feature output by the last hidden layer in the at least one hidden layer is determined as the feature of the second reference text.
10. The method according to any one of claims 6 to 9, characterized in that, The step of generating the set of matching pairs based on the at least one matching pair includes: Select K matching pairs from the at least one matching pair as K center matching pairs, where K is an integer greater than or equal to 1; Based on the K center matching pairs, the at least one matching pair is clustered to obtain K matching pair subsets, each matching pair subset including at least one matching pair; Based on the K matching pairs subsets, determine the central features of each of the K matching pairs subsets; The matching pair set is generated based on the K matching pair subsets and the central features of each of the K matching pair subsets.
11. A text translation device, characterized in that, The device includes: The text acquisition module is used to acquire the text to be translated. The feature extraction module is used to extract text features of the text to be translated, and the text features are used to indicate the semantic information of the text to be translated; The feature matching module is used to find a reference text feature that matches the text feature in a set of matching pairs, as the first reference text feature. The set of matching pairs includes at least one matching pair, and each matching pair includes a reference text feature and a reference word. The result determination module is used to determine the text translation result of the text to be translated based on the reference vocabulary corresponding to the features of the first reference text.
12. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as described in any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 10.
14. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the method as claimed in any one of claims 1 to 10.