A language translation method, electronic device and system based on big data

Abstract

The present invention relates to the field of natural language processing technology, and in particular to a language translation method, electronic device and system based on big data, including: constructing an initial NMT model and translating a sentence segment to be translated to obtain an initial translation; constructing a network corpus, combining the network corpus to perform semantic analysis on the relationship between the sentence segment to be translated and the word unit in the initial translation in the sentence structure, obtaining the sentence structure parameters of the word unit, and replacing the word unit in the corresponding sentence through the synonyms and near synonyms of the word unit in the network corpus, analyzing the semantic change characteristics before and after the replacement, obtaining the semantic feature value of the word unit, combining the sentence structure parameter of the word unit with the semantic feature value, obtaining the translation degree of the word unit, thereby providing feedback to the NMT algorithm, and iteratively translating the sentence segment to be translated to obtain the final translation. The present invention effectively optimizes the semantic expression of the translation, and improves the accuracy and context adaptability of the translation.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a language translation method, electronic equipment and system based on big data. Background Art

[0002] The advancement of globalization and the increasing number of cross-border exchanges have made language translation an indispensable part of people's daily lives and business activities. In addition, the amount of data around the world has shown explosive growth, and the emergence of big data technology has provided new opportunities for the improvement of language translation systems. By utilizing big data, translation systems can more accurately capture the relationship between languages, analyze context, identify vocabulary collocations and common expressions, thereby improving the accuracy and naturalness of translation.

[0003] At present, language translation technology based on NMT (Neural Machine Translation) algorithm has gradually become mainstream. Compared with traditional statistical translation methods, neural network translation can not only better capture grammatical information at the sentence level, but also provide more natural and fluent translation through contextual information. However, although the translation method based on neural network has made significant progress in accuracy and fluency, the translation results still have problems with word order and context, which makes the NMT algorithm still have certain limitations. Summary of the invention

[0004] The present invention provides a language translation method, electronic device and system based on big data to solve the existing problems.

[0005] A language translation method, electronic device and system based on big data of the present invention adopts the following technical solutions: An embodiment of the present invention provides a language translation method based on big data, the method comprising the following steps: Obtain corpus data and preliminarily train the NMT model to obtain the initial NMT model; Obtain the sentence segment to be translated, and translate it through the initial NMT model to obtain the initial translation; Use crawler technology to crawl network corpus data to build a network corpus; Combined with the network corpus, semantic analysis is performed on the relationship between each word unit in the initial translation and the sentence structure parameters of each word unit in the initial translation are obtained. The synonyms and near synonyms of the word unit in the network corpus are used to replace the word unit in the corresponding sentence, and the semantic change characteristics before and after the replacement are analyzed to obtain the semantic feature value of each word unit in the initial translation. The sentence structure parameters of any word unit in the initial translation are combined with the semantic feature value to obtain the translation degree of the word unit. The translation degree of word units is combined to provide feedback to the NMT model, a feedback function is constructed, and the sentences to be translated are translated iteratively to obtain the final translation.

[0006] Furthermore, the semantic analysis of the relationship between each word unit in the initial translation and the sentence structure in the network corpus is performed to obtain the sentence structure parameters of each word unit in the initial translation, including the specific method of: The initial translation is segmented using Jieba word segmentation, and each phrase in the segmentation result is taken as a word unit; for any word unit, several sentence segments containing the arbitrary word unit in the network corpus are obtained and recorded as the corpus sentence segments of the arbitrary word unit; Perform text structure analysis on the initial translation of any word unit and the corpus sentence segment to obtain the cosine structure sequence of the initial translation and each corpus sentence segment; According to the similarity of the cosine structure sequences corresponding to the initial translation and all the corpus sentence segments corresponding to each word unit in the initial translation, and combined with the structural features of the corpus sentence segment corresponding to each word unit, the sentence structure parameters of any word unit are obtained.

[0007] Furthermore, the text structure analysis of the initial translation and the corpus sentence segment of any word unit is performed to obtain the cosine structure sequence of the initial translation and each corpus sentence segment, including the specific method of: First, the corpus segments are segmented to obtain several phrases in any corpus segment, called corpus word units. The word2vec algorithm is used to convert the word units in the initial translation and the corpus word units in all corpus segments into word vectors to obtain word units and word vectors of corpus word units. Then, the cosine value of the word vector corresponding to any adjacent word unit in the initial translation is obtained, and a sequence formed by the cosine values ​​between the word vectors of all adjacent word units in the initial translation is obtained, which is recorded as the cosine structure sequence of the initial translation; and so on, the cosine structure sequence of any corpus sentence segment is obtained.

[0008] Furthermore, the specific method for obtaining the sentence structure parameters of the arbitrary word unit is: Obtain the structural factors between the initial translation and the corpus segment; The specific calculation method of the sentence structure parameters of the word unit is: in, The first Sentence structure parameters of word units; The initial translation and the first word unit The DTW distance between the cosine structure sequences corresponding to the corpus sentences; The first The number of corpus segments with word units; The initial translation and the first word unit Structural factors between corpus segments; The first word unit The number of all the participles in the corpus segment that are identical to the word units in the initial translation; The first word unit The number of all participles in a corpus segment; is the standard deviation function.

[0009] Furthermore, the specific method for obtaining the structural factor between the initial translation and the corpus sentence segment is: Get the same participles in the initial translation and the corpus segment; take a participle as a node, connect them according to the order of the participles in the corresponding text, and take the backward order as the corresponding connection direction; The intervals between adjacent segmented words in the corresponding text are used as the edge weights of the corresponding connecting edges, forming chain segmented graph structures corresponding to the initial translation and the corpus sentence segments; The graph similarity between the word segmentation graph structures corresponding to the initial translation and the corpus segment is obtained through the graph edit distance as a structural factor between the initial translation and the corpus segment.

[0010] Furthermore, the specific method for obtaining the semantic feature value of the word unit is: Obtain synonyms and near-synonyms of any word unit, collectively referred to as approximate words of the corresponding word unit, and obtain corpus segments containing the approximate words of the word unit in the network corpus, referred to as approximate corpus segments of the word unit; The semantic feature analysis of each word unit is performed based on the initial translation and the approximate corpus sentence of each word unit to obtain the semantic feature value of the word unit.

[0011] Furthermore, the specific method for obtaining the semantic feature value of the word unit is: For any word unit, obtain all approximate corpus segments corresponding to the word unit, obtain the approximate words corresponding to the word unit in the approximate corpus segments, and the words adjacent to the approximate words on the left and right. The word segmentation is used as the neighboring word segmentation of the approximate word, and the word group formed by the cosine value of the word vector of the approximate word and each corresponding neighboring word segmentation is obtained, which is recorded as the semantic feature group of the approximate word, where is the preset quantity parameter; For any word unit, replace the word unit with all the similar words of the word unit to obtain a new initial translation, and obtain the semantic feature group of all the similar words of the word unit in the new initial translation; and so on, obtain the semantic feature group corresponding to the word unit; The semantic feature value of the word unit is calculated based on the cosine value between the semantic feature groups of the word unit and the corresponding similar words obtained in different texts.

[0012] Furthermore, the specific calculation method of the semantic feature value of the word unit is: in, Indicates the first The semantic feature value of each word unit; The first The semantic feature group of each word unit; The first The word unit is corresponding to After the similar words are replaced, the semantic feature group of the similar words; The first The word unit corresponds to the Similar words in the The semantic feature groups in the similar corpus segments; To contain The number of similar corpus segments with similar words; The first The number of similar words to the word unit; is the standard deviation function; is the cosine function; is an exponential function with a natural constant as its base.

[0013] An embodiment of the present invention provides a language translation electronic device based on big data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a language translation method based on big data when executing the computer program.

[0014] An embodiment of the present invention provides a language translation system based on big data, which includes the following modules: The initial model module is used to obtain corpus data and preliminarily train the NMT model to obtain the initial NMT model; The preliminary translation module is used to obtain the sentence segments to be translated, and translate the sentence segments to be translated through the initial NMT model to obtain the initial translation; Corpus collection module, used to crawl network corpus data using crawler technology, so as to build a network corpus; The translation analysis module is used to perform semantic analysis on the relationship between each word unit in the initial translation and the sentence structure in combination with the network corpus, obtain the sentence structure parameters of each word unit in the initial translation, and replace the word unit in the corresponding sentence with the synonyms and near synonyms in the network corpus, analyze the semantic change characteristics before and after the replacement, obtain the semantic feature value of each word unit in the initial translation, and combine the sentence structure parameters and semantic feature values ​​of any word unit in the initial translation to obtain the translation degree of the word unit; The feedback translation module is used to provide feedback to the NMT model based on the translation degree of word units, build a feedback function and iteratively translate the sentences to be translated to obtain the final translation.

[0015] The beneficial effect of the technical solution of the present invention is that after constructing the initial NMT model and the network corpus, after performing semantic comparative analysis on the word units in the initial translation and the synonyms and near-synonyms in the network corpus in terms of syntactic structure and synonym replacement, a translation degree index is further quantified and generated, and translation ambiguity points in the initial translation are effectively identified. The translation degree parameter is integrated into the NMT model weight adjustment through an iterative optimization mechanism to achieve a gradual improvement in translation quality. This method can significantly improve the translation quality by combining big data technology, thereby optimizing the semantic expression of the translation, and improving the accuracy and contextual adaptability of the translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of the steps of a language translation method based on big data of the present invention; Figure 2 A schematic diagram of a process for obtaining a structure factor according to an embodiment of the present invention; Figure 3 A schematic diagram of the positional relationship between a word unit and similar words and corresponding adjacent participles provided in one embodiment of the present invention; Figure 4 A schematic diagram of a semantic feature space provided for an embodiment of the present invention; Figure 5 It is a structural block diagram of a language translation system based on big data of the present invention. DETAILED DESCRIPTION

[0018] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the language translation method, electronic device and system based on big data proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0019] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0020] The following is a detailed description of a language translation method, electronic device and system based on big data provided by the present invention in conjunction with the accompanying drawings.

[0021] See also Figure 1 , which shows a flow chart of a language translation method based on big data provided by an embodiment of the present invention, the method comprising the following steps: Step S001: Acquire corpus data and preliminarily train the NMT model to obtain an initial NMT model.

[0022] It should be noted that when performing language translation, NMT models usually need to be trained with a large amount of bilingual parallel corpora, which include sentences in the source language and the target language, such as parallel corpora in English and Chinese.

[0023] Specifically, in order to implement a language translation method based on big data proposed in this embodiment, it is first necessary to collect bilingual parallel corpus data and train the NMT model. The specific process is as follows: Step S101, obtaining corpus data and preprocessing it.

[0024] In one embodiment of the present invention, the specific steps of acquiring and preprocessing corpus data include: First, bilingual parallel corpora are collected to obtain corpus data.

[0025] Then, the bilingual parallel corpus is preprocessed, including data cleaning, word segmentation, standardization and data segmentation. Among them, special characters, HTML tags, blank lines, etc. in the bilingual parallel corpus are removed through data cleaning. The bilingual parallel corpus is segmented, and the Chinese corpus is processed by Jieba word segmentation, and the English corpus is processed by Nltk word segmentation. The standardization process is to uniformly encode the vocabulary formed by the words after the bilingual parallel corpus word segmentation based on the Subword Unit technology, and divide the data set formed by the corpus data into training set, validation set and test set.

[0026] It should be noted that in the embodiment of the present invention, the data set is divided into a training set, a validation set, and a test set in proportions of 80%, 10%, and 10%, respectively. The specific division ratios can be adjusted according to actual conditions, and the embodiment of the present invention does not make specific limitations.

[0027] Step S102: construct an NMT model and train the NMT model in combination with the corpus data to obtain an initial NMT model.

[0028] First, we build an NMT neural network model based on the encoder-decoder architecture and combined with the self-attention mechanism and Transformer structure as the NMT model, and then further implement the NMT model through a deep learning framework.

[0029] As an embodiment of the present invention, the deep learning framework is PyTorch, thereby realizing the NMT model.

[0030] Then, the source language sentences in the training set are input into the NMT model, and the target language sentences are used as labels. The model parameters are adjusted during the training process by optimizing the cross entropy loss function.

[0031] At this point, the initial NMT model that has been preliminarily trained is obtained through the above method.

[0032] Step S002: Obtain a sentence segment to be translated, and translate the sentence segment to be translated using an initial NMT model to obtain an initial translation.

[0033] In one embodiment of the present invention, the specific steps of obtaining the initial translation are: First, the sentence segment to be translated is obtained and input into the initial NMT model, and a context vector is generated by the encoder in the initial NMT model.

[0034] Then, the decoder generates a sentence in the target language based on the context vector. During the generation process, a greedy search algorithm is used to generate the optimal translation result.

[0035] It should be noted that since the greedy algorithm is an existing optimal solution-finding algorithm, it is not described in detail in the embodiment of the present invention.

[0036] Finally, the initial translation of the segment to be translated is output.

[0037] At this point, the initial translation of the sentence to be translated is obtained through the above method.

[0038] Step S003: crawling web corpus data using crawler technology to build a web corpus.

[0039] It should be noted that the NMT model trained by the existing fixed corpus will be affected by the number of words in the training data when performing language translation, resulting in underfitting or overfitting problems when translating some words, causing some words in the translation result to have inappropriate semantics in the translation, resulting in an incoherent translation result. Therefore, the embodiment of the present invention improves the translation effect of the translation by subsequently performing semantic analysis on other corpus data and combining it with the words in the initial translation.

[0040] In order to achieve the purpose of subsequent semantic analysis, the embodiment of the present invention needs to obtain network corpus data and build a network corpus, including: First, based on the Scrapy crawler library, the sentences, paragraphs, phrases, and words on the Internet platform are obtained through crawler technology.

[0041] The Internet platforms include news websites, forums and blogs.

[0042] Then, the crawled web corpus data is cleaned to remove irrelevant content such as advertisements and navigation bars on the web pages; and duplicate content is removed.

[0043] Finally, all processed phrases, sentences, paragraphs, etc. are stored as text files and stored in the database as a corpus, recorded as the network corpus.

[0044] At this point, the network corpus is obtained through the above method.

[0045] Step S004: semantically analyze the relationship between each word unit in the initial translation and the sentence structure in the network corpus to obtain the sentence structure parameters of each word unit in the initial translation, and replace the word unit in the corresponding sentence with the synonyms and near-synonyms in the network corpus, analyze the semantic change characteristics before and after the replacement, and obtain the semantic feature value of each word unit in the initial translation. Combine the sentence structure parameters and semantic feature values ​​of any word unit in the initial translation to obtain the translation degree of the word unit.

[0046] It should be noted that, in the process of translating sentences through the NMT model, NMT may misunderstand the translated sentences, or the attributives in the translated results may be too long and do not conform to the Chinese expression habits, resulting in unsatisfactory translation results. Therefore, in order to improve the translation effect of NMT, we choose to use the network corpus to analyze the translation effect of the initial translation of NMT and obtain the translation degree of the word units in the initial translation.

[0047] As an embodiment, a specific method for obtaining the translation degree includes: Step S401, obtaining a number of word units of the initial translation, combining the sentence structure of the word units in the network corpus, and calculating the sentence structure parameters of each word unit in the initial translation.

[0048] It should be noted that when translating a text, it is not only necessary to convert the vocabulary in one language into the corresponding vocabulary in another language, but also necessary to capture the structural and semantic properties of the language, that is, language is essentially composed of a hierarchical structure and semantics containing implicit meanings. Therefore, when translating vocabulary and generating sentences, the core task is to capture these structural and semantic information. Therefore, in an embodiment of the present invention, it is chosen to perform semantic analysis on the relationship between each word unit in the initial translation in the sentence structure by combining the network corpus to obtain the sentence structure parameters of each word unit in the initial translation.

[0049] As an embodiment, the method for obtaining the sentence structure parameters of the word unit includes: Step S401.1, use Jieba word segmentation to segment the initial translation, and each phrase in the segmentation result is taken as a word unit; for any word unit, obtain several sentence segments containing the arbitrary word unit in the network corpus, and record them as the corpus sentence segments of the arbitrary word unit.

[0050] Step S401.2: Perform text structure analysis on the initial translation of any word unit and the corpus sentence segment to obtain the cosine structure sequence of the initial translation and each corpus sentence segment.

[0051] As an embodiment, the specific process of obtaining the cosine structure sequence includes: First, the corpus sentences are segmented to obtain several phrases in any corpus sentence, called corpus word units. The word2vec algorithm is used to convert the word units in the initial translation and the corpus word units in all corpus sentences into word vectors to obtain word units and word vectors of corpus word units.

[0052] Then, the cosine value of the word vector corresponding to any adjacent word unit in the initial translation is obtained, and a sequence formed by the cosine values ​​between the word vectors of all adjacent word units in the initial translation is obtained, which is recorded as the cosine structure sequence of the initial translation.

[0053] And so on, the cosine structure sequence of any corpus sentence segment can be obtained.

[0054] Finally, the DTW distance between the initial translation and the cosine structure sequence corresponding to any corpus segment is obtained.

[0055] It should be noted that the word units in the initial translation and the corpus word vectors in the corpus sentence segments essentially extract the semantic information in the initial translation or corpus sentence segments through word vector conversion, that is, through word vector conversion, each word can be converted into a dense vector representation, and these vectors can retain the semantic similarity between words. For example, the word vectors of "apple" and "orange" are closer in the semantic space, while the word vectors of "apple" and "car" are farther away; in addition, by calculating the cosine similarity of all adjacent word units, the similarity distribution between words in the entire sentence is reflected, which can further help identify the structural features of the translation, thereby facilitating the subsequent analysis of the degree of conformity between the text structure of the initial translation and common expressions in terms of semantic structure.

[0056] Step S401.3, based on the similarity of the cosine structure sequences corresponding to the initial translation and all the corpus sentence segments corresponding to each word unit in the initial translation, and in combination with the sentence structure features between the initial translation and the corpus sentence segments corresponding to each word unit in the initial translation, the sentence structure parameters of any word unit are obtained.

[0057] As an embodiment, the specific calculation method of the sentence structure parameter of the word unit is: in, The first Sentence structure parameters of word units; The initial translation and the first word unit The DTW distance between the cosine structure sequences corresponding to the corpus sentences; The first The number of corpus segments with word units; The initial translation and the first word unit Structural factors between corpus segments; The first word unit The number of all the participles in the corpus segment that are identical to the word units in the initial translation; The first word unit The number of all participles in a corpus segment; is the standard deviation function.

[0058] As an embodiment, the specific method for obtaining the structure factor includes: First, obtain the same participles in the initial translation and the corpus sentence segment; take a participle as a node, connect them according to the order of the participles in the corresponding text, and use the backward order as the corresponding connection direction.

[0059] Then, the intervals between adjacent word segments in the corresponding text are used as the edge weights of the corresponding connecting edges to form chained word segmentation graph structures corresponding to the initial translation and the corpus segments.

[0060] Finally, the graph similarity between the word segmentation graph structures corresponding to the initial translation and the corpus segment is obtained through the graph edit distance as the structural factor between the initial translation and the corpus segment.

[0061] like Figure 2 Shown is a schematic diagram of the process for obtaining structure factors.

[0062] It should be noted that the structural factor reflects the similarity between the sentence structures formed in the word segmentation order of the initial translation and the corpus sentence segment, and the sentence structure parameter of the word unit reflects the characteristics of the sentence structure of the word unit in the initial translation. The more similar text structures there are in the online corpus for the text structure formed between the word segments in the initial translation, and the more similar the structures are, the more the text structure of the corresponding word unit in the initial translation conforms to the language habits, and the larger the corresponding sentence structure parameter is.

[0063] Step S402: Acquire a corpus segment containing synonyms and near-synonyms of a word unit in a network corpus, and then perform semantic feature analysis on the synonyms and near-synonyms and the corresponding word units contained in the corpus segment by replacement to obtain a semantic feature value of the word unit.

[0064] It should be noted that, usually, although the corresponding synonyms and near-synonyms of a word are similar to its own semantic information in a sentence during use, the semantic information may change in some cases. Therefore, in order to more accurately describe the semantic features reflected by the word, the present invention chooses to combine the synonyms and near-synonyms of the word unit to further obtain the semantic features of the word unit.

[0065] As an embodiment, a specific method for obtaining the semantic feature value of a word unit includes: Step S402.1, obtaining synonyms and near-synonyms of any word unit, collectively referred to as approximate words of the corresponding word unit, and obtaining corpus segments in the network corpus that contain the approximate words of the word unit, referred to as the approximate corpus segments of the word unit.

[0066] Step S402.2, performing semantic feature analysis on the word unit in combination with the initial translation and the approximate corpus segment of each word unit to obtain the semantic feature value of the word unit.

[0067] As an embodiment, the specific steps of obtaining the semantic features of a word unit include: First, for any word unit, obtain all the approximate corpus segments corresponding to the word unit, obtain the approximate words corresponding to the word unit in the approximate corpus segments, and the words adjacent to the approximate words on the left and right. The word group formed by the cosine value of the word vector of the approximate word and each corresponding neighboring word is called the semantic feature group of the approximate word, where The preset quantity parameter.

[0068] It should be noted that the quantity parameters are preset based on experience. The value is 3, which can be adjusted according to actual conditions and is not specifically limited in the embodiment of the present invention.

[0069] Then, for any word unit, all the similar words of the word unit are used to replace the word unit to obtain a new initial translation, and the semantic feature group of all the similar words of the word unit in the new initial translation is obtained. Similarly, the semantic feature group corresponding to the word unit is obtained.

[0070] like Figure 3 The figure shows the positional relationship between word units and similar words and corresponding adjacent participles, including word units The initial translation, as well as the word unit Similar words of and The similar corpus segment, word unit and similar words and Adjacent to the left and right Each cell is a neighboring word segment of a word unit or a similar word in the corresponding sentence segment.

[0071] It should be noted that there is a high correlation between the meaning expressed by a word in a text and the combination relationship formed between its neighboring words, but the neighboring words cannot be too many, otherwise the amount of information to be conveyed by the text will be too large to effectively express this correlation feature. Therefore, the present invention chooses to obtain a semantic feature group constructed by segmentation within a preset range to express the semantic feature space, taking the three-dimensional space as an example for demonstration, such as Figure 4 The following is a schematic diagram of the semantic feature space, where is the word vector of the approximate word, , as well as is the word vector corresponding to the neighboring word of the approximate word, then the corresponding , as well as is the angle between the word vectors, then the cosine value obtained by the corresponding angle forms the semantic feature group of the approximate word. The smaller the angle between the word vectors, the closer the semantic information between the corresponding words, that is, the closer the meaning expressed. Then, the semantic feature group formed by the cosine values ​​of the word vectors between multiple adjacent words in a sentence segment expresses the semantic features of the words in the meaning conveyed by the corresponding sentence segment. Therefore, the embodiment of the present invention further uses the semantic feature group to quantify the semantic features of the word vectors in the initial translation.

[0072] Finally, for any word unit in the initial translation, the semantic feature value of the word unit is calculated based on the cosine value between the semantic feature groups of the word unit and the corresponding similar words obtained in different texts.

[0073] As an embodiment, the specific calculation method of the semantic feature value of the word unit is: in, Indicates the first The semantic feature value of each word unit; The first The semantic feature group of each word unit; The first The word unit is corresponding to After the similar words are replaced, the semantic feature group of the similar words; The first The word unit corresponds to the Similar words in the The semantic feature groups in the similar corpus segments; To contain The number of similar corpus segments with similar words; The first The number of similar words to the word unit; is the standard deviation function; is the cosine function; is an exponential function with a natural constant as its base.

[0074] It should be noted that The cosine value between the semantic feature group of the word unit and the semantic feature group in the initial translation after the word unit is replaced by a similar word, The cosine value between the semantic feature group of the word unit and the corresponding semantic feature group of the similar words in the similar corpus segment to which it belongs reflects the degree of semantic change between the word unit and its similar words in different contexts. It reflects the discreteness of the semantic changes of all approximate words corresponding to the word unit of the initial translation, that is, taking into account whether the context changes in the translation process, and the semantics of the word unit itself and the approximate words are easily affected by the context changes, resulting in too high a discrete degree of semantic changes and inappropriate translation, and then obtaining semantic feature values ​​to reflect the semantic stability characteristics of the word unit. When the initial translation before and after the word unit is replaced by the corresponding approximate words (that is, synonyms and near-synonyms), and the semantic information expressed by the corresponding approximate words in the sentence to which they belong do not change significantly, that is, the semantics are relatively stable, then the translation of the word unit in the translation result is sufficiently accurate, and the corresponding semantic feature value is larger.

[0075] Step S403: For any word unit in the initial translation, the sentence structure parameters and semantic feature values ​​of the word unit are comprehensively analyzed to calculate the translation degree of the word unit.

[0076] As an embodiment, the specific calculation method of the translation degree of any word unit in the initial translation is: in, is the translation degree of word unit; is the sentence structure parameter of the word unit; is the semantic feature value of the word unit; is the sigmoid normalization function.

[0077] It should be noted that the translation degree is the degree to which the word unit causes the initial translation to affect the semantic information during the translation process. The greater the translation degree, the more the word unit's expression of semantic structure and semantic information in the initial translation conforms to the corresponding language habits, and the better the translation effect.

[0078] So far, the translation degree of each word unit in the initial translation is obtained through the above method.

[0079] Step S005: Feedback is given to the NMT model in combination with the translation degree of the word unit, a feedback function is constructed, and the sentence segment to be translated is translated iteratively to obtain a final translation.

[0080] It should be noted that the translation degree describes the degree to which each word unit in the initial translation conforms to the expression habits of the corresponding language for the corresponding semantic information in the translation sentence after the translation of the translation sentence. By quantifying the degree of conformity, the translation effect of the initial translation can be effectively clarified, which can be fed back to the initial NMT model to further continuously adjust the initial translation, improve the translation effect of the initial NMT model, and make the words in the translation more in line with the expression habits of the target language.

[0081] As an embodiment, the specific method for obtaining the final translation includes: First, construct the feedback function.

[0082] It should be noted that the feedback function is constructed based on the translation degree of the word unit in the initial translation. The purpose of this feedback function is to adjust the current translation result based on the translation degree so that important word units can be better processed in the translation.

[0083] As an embodiment, the basic structure of the feedback function can be expressed as: in, It is The degree of translation of word units; is the translation result of the word unit; Represents the total feedback after weighting the translation degree of all word units.

[0084] It should be noted that the feedback function adjusts the translation result by weighting the translation degree of each word unit, thereby optimizing the target translation.

[0085] Then, after obtaining the feedback function, the translation model is adjusted through the optimization algorithm. The specific steps are as follows: Calculate the loss function: Based on the feedback function, calculate the difference between the translation result and the expected translation (for example, using the cross entropy loss function).

[0086] Gradient update: The gradient is calculated through the back-propagation algorithm and the model parameters are updated to reduce the loss.

[0087] Retranslation process: Use the optimized model to retranslate the source sentence to generate a new translation.

[0088] Finally, iterative optimization is performed to obtain the final translation.

[0089] It should be noted that each iteration optimizes the model based on the previous round of translation results and feedback functions to make the translation results more accurate and smooth. After multiple rounds of iterative adjustments, the final translation should fully consider the translation degree of each word unit, that is, the translation quality of important word units is high, while less important words can be reasonably processed in the translation, thus generating high-quality target language translation.

[0090] Through the above steps, the translation of the sentence to be translated is completed and the final translation is obtained.

[0091] In other embodiments of the present invention, a language translation electronic device based on big data is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the contents of steps S001 to S005 in the language translation method based on big data are implemented.

[0092] Further, in an optional embodiment, the above-mentioned memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories, wherein the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0093] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0094] See also Figure 5, which shows a structural block diagram of a language translation system based on big data provided by an embodiment of the present invention, the system includes the following modules: The initial model module is used to obtain corpus data and preliminarily train the NMT model to obtain the initial NMT model; The preliminary translation module is used to obtain the sentence segments to be translated, and translate the sentence segments to be translated through the initial NMT model to obtain the initial translation; Corpus collection module, used to crawl network corpus data using crawler technology, so as to build a network corpus; The translation analysis module is used to perform semantic analysis on the relationship between each word unit in the initial translation and the sentence structure in combination with the network corpus, obtain the sentence structure parameters of each word unit in the initial translation, and replace the word unit in the corresponding sentence with the synonyms and near synonyms in the network corpus, analyze the semantic change characteristics before and after the replacement, obtain the semantic feature value of each word unit in the initial translation, and combine the sentence structure parameters and semantic feature values ​​of any word unit in the initial translation to obtain the translation degree of the word unit; The feedback translation module is used to provide feedback to the NMT model based on the translation degree of word units, build a feedback function and iteratively translate the sentences to be translated to obtain the final translation.

[0095] After constructing the initial NMT model and the online corpus, this embodiment further quantifies and generates a translation degree index by performing a semantic comparative analysis of the word units in the initial translation and the synonyms and near-synonyms in the online corpus in terms of syntactic structure and synonym replacement, effectively identifying translation ambiguities in the initial translation, and integrating the translation degree parameters into the NMT model weight adjustment through an iterative optimization mechanism to achieve a gradual improvement in translation quality. This method can significantly improve translation quality by combining big data technology, thereby optimizing the semantic expression of the translation and improving translation accuracy and contextual adaptability.

[0096] It should be noted that the The model is only used to represent negative correlation and constrain the output of the model to be in In the specific implementation, it can be replaced by other models with the same purpose. This embodiment is only based on The model is described as an example without any specific limitation. is the input to the model.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A language translation method based on big data, characterized in that: The method comprises the following steps: Obtain corpus data and preliminarily train the NMT model to obtain the initial NMT model; Obtain the sentence segment to be translated, and translate it through the initial NMT model to obtain the initial translation; Use crawler technology to crawl network corpus data to build a network corpus; Combined with the network corpus, semantic analysis is performed on the relationship between each word unit in the initial translation and the sentence structure parameters of each word unit in the initial translation are obtained. The synonyms and near synonyms of the word unit in the network corpus are used to replace the word unit in the corresponding sentence, and the semantic change characteristics before and after the replacement are analyzed to obtain the semantic feature value of each word unit in the initial translation. The sentence structure parameters of any word unit in the initial translation are combined with the semantic feature value to obtain the translation degree of the word unit. The translation degree of word units is combined to provide feedback to the NMT model, a feedback function is constructed, and the sentences to be translated are translated iteratively to obtain the final translation.

2. According to the language translation method based on big data as claimed in claim 1, it is characterized in that: The method of combining the network corpus to semantically analyze the relationship between each word unit in the initial translation and the sentence structure to obtain the sentence structure parameters of each word unit in the initial translation includes the following specific methods: The initial translation is segmented using Jieba word segmentation, and each phrase in the segmentation result is taken as a word unit; for any word unit, several sentence segments containing the arbitrary word unit in the network corpus are obtained and recorded as the corpus sentence segments of the arbitrary word unit; Perform text structure analysis on the initial translation of any word unit and the corpus sentence segment to obtain the cosine structure sequence of the initial translation and each corpus sentence segment; According to the similarity of the cosine structure sequences corresponding to the initial translation and all the corpus sentence segments corresponding to each word unit in the initial translation, and combined with the structural features of the corpus sentence segment corresponding to each word unit, the sentence structure parameters of any word unit are obtained.

3. According to the language translation method based on big data as claimed in claim 2, it is characterized in that: The text structure analysis of the initial translation and the corpus sentence segment of any word unit is performed to obtain the cosine structure sequence of the initial translation and each corpus sentence segment, including the specific method of: First, the corpus segments are segmented to obtain several phrases in any corpus segment, called corpus word units. The word2vec algorithm is used to convert the word units in the initial translation and the corpus word units in all corpus segments into word vectors to obtain word units and word vectors of corpus word units. Then, the cosine value of the word vector corresponding to any adjacent word unit in the initial translation is obtained, and a sequence formed by the cosine values ​​between the word vectors of all adjacent word units in the initial translation is obtained, which is recorded as the cosine structure sequence of the initial translation; and so on, the cosine structure sequence of any corpus sentence segment is obtained.

4. According to the language translation method based on big data as claimed in claim 2, it is characterized in that: The specific method for obtaining the sentence structure parameters of the arbitrary word unit is: Obtain the structural factors between the initial translation and the corpus segment; The specific calculation method of the sentence structure parameters of the word unit is: in, The first Sentence structure parameters of word units; The initial translation and the first word unit The DTW distance between the cosine structure sequences corresponding to the corpus sentences; The first The number of corpus segments with word units; The initial translation and the first word unit Structural factors between corpus segments; The first word unit The number of all the participles in the corpus segment that are identical to the word units in the initial translation; The first word unit The number of all participles in a corpus segment; is the standard deviation function.

5. According to the language translation method based on big data as claimed in claim 4, it is characterized in that: The specific method for obtaining the structural factor between the initial translation and the corpus sentence segment is: Get the same participles in the initial translation and the corpus segment; take a participle as a node, connect them according to the order of the participles in the corresponding text, and take the backward order as the corresponding connection direction; The intervals between adjacent segmented words in the corresponding text are used as the edge weights of the corresponding connecting edges, forming chain segmented graph structures corresponding to the initial translation and the corpus sentence segments; The graph similarity between the word segmentation graph structures corresponding to the initial translation and the corpus segment is obtained through the graph edit distance as a structural factor between the initial translation and the corpus segment.

6. The language translation method based on big data according to claim 1, characterized in that: The specific method for obtaining the semantic feature value of the word unit is: Obtain synonyms and near-synonyms of any word unit, collectively referred to as approximate words of the corresponding word unit, and obtain corpus segments containing the approximate words of the word unit in the network corpus, referred to as approximate corpus segments of the word unit; The semantic feature analysis of each word unit is performed based on the initial translation and the approximate corpus sentence of each word unit to obtain the semantic feature value of the word unit.

7. The language translation method based on big data according to claim 6, characterized in that: The specific method for obtaining the semantic feature value of the word unit is: For any word unit, obtain all approximate corpus segments corresponding to the word unit, obtain the approximate words corresponding to the word unit in the approximate corpus segments, and the words adjacent to the approximate words on the left and right. The word segmentation is used as the neighboring word segmentation of the approximate word, and the word group formed by the cosine value of the word vector of the approximate word and each corresponding neighboring word segmentation is obtained, which is recorded as the semantic feature group of the approximate word, where is the preset quantity parameter; For any word unit, replace the word unit with all the similar words of the word unit to obtain a new initial translation, and obtain the semantic feature group of all the similar words of the word unit in the new initial translation; and so on, obtain the semantic feature group corresponding to the word unit; The semantic feature value of the word unit is calculated based on the cosine value between the semantic feature groups of the word unit and the corresponding similar words obtained in different texts.

8. The language translation method based on big data according to claim 7, characterized in that: The specific calculation method of the semantic feature value of the word unit is: in, Indicates the first The semantic feature value of each word unit; The first The semantic feature group of each word unit; The first The word unit is corresponding to After the similar words are replaced, the semantic feature group of the similar words; The first The word unit corresponds to the Similar words in the The semantic feature groups in the similar corpus segments; To contain The number of similar corpus segments with similar words; The first The number of similar words to the word unit; is the standard deviation function; is the cosine function; is an exponential function with a natural constant as its base.

9. A language translation electronic device based on big data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the language translation method based on big data as described in any one of claims 1 to 8 are implemented.

10. A language translation system based on big data, using a language translation method based on big data as claimed in any one of claims 1 to 8, characterized in that: The system includes the following modules: The initial model module is used to obtain corpus data and preliminarily train the NMT model to obtain the initial NMT model; The preliminary translation module is used to obtain the sentence segments to be translated, and translate the sentence segments to be translated through the initial NMT model to obtain the initial translation; Corpus collection module, used to crawl network corpus data using crawler technology, so as to build a network corpus; The translation analysis module is used to perform semantic analysis on the relationship between each word unit in the initial translation and the sentence structure in combination with the network corpus, obtain the sentence structure parameters of each word unit in the initial translation, and replace the word unit in the corresponding sentence with the synonyms and near synonyms in the network corpus, analyze the semantic change characteristics before and after the replacement, obtain the semantic feature value of each word unit in the initial translation, and combine the sentence structure parameters and semantic feature values ​​of any word unit in the initial translation to obtain the translation degree of the word unit; The feedback translation module is used to provide feedback to the NMT model based on the translation degree of word units, build a feedback function and iteratively translate the sentences to be translated to obtain the final translation.

Images