Intelligent translation method and system based on big data

Abstract

The invention relates to the technical field of natural language processing, and discloses an intelligent translation method and system based on big data, and the method comprises the steps: obtaining initial English text data, and carrying out the sentence division to obtain English sentences and English words; calculating to obtain a first vocabulary weight, a second vocabulary weight and a third vocabulary weight; calculating a part-of-speech confidence coefficient of the word through a preset part-of-speech probability, and calculating a final vocabulary weight according to the part-of-speech confidence coefficient and the third vocabulary weight; according to the final vocabulary weight, generating a text vector of the English sentence; and inputting the text vector into a preset translation decoder to generate a translation result of the target language. According to the method, the word class of the vocabulary in the English text can be effectively associated with the context, so that the accuracy of subsequent translation of the English text is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an intelligent translation method and system based on big data. Background Art

[0002] At present, the rapid development of globalization has made cross-language communication an indispensable core demand in the fields of economy, culture, science and technology, and has promoted the rapid development of intelligent translation technology. With the widespread application of big data technology and deep learning models, translation technology has entered a new stage based on semantic understanding and context association. However, despite the significant progress made in existing technologies, there are still technical bottlenecks that cannot fully meet the needs of high-precision translation in multiple scenarios, multiple languages, and multiple fields.

[0003] In one existing technology, a word2vec model is used to process English text translation. The word2vec model is used to generate word vectors, which represent the semantic and grammatical information between words. This information can be used to cluster words with similar semantics. By combining deep learning, text features are extracted layer by layer in a bottom-up manner, thereby capturing information in the text.

[0004] However, when using the word2vec model for feature extraction, it is impossible to filter words from the context dimension, and the effect of semantic filtering of words is poor, which affects the accuracy of translation. Summary of the invention

[0005] The present invention provides an intelligent translation method and system based on big data, so as to effectively link the word parts of speech in English text with the context, thereby improving the accuracy of subsequent translation of the English text.

[0006] In a first aspect, in order to solve the above technical problems, the present invention provides an intelligent translation method based on big data, which is executed by a computer and includes:

[0007] Obtain initial English text data and perform sentence segmentation to obtain English sentences and English words;

[0008] Counting the occurrence frequency of the English words and calculating a first vocabulary weight;

[0009] Calculating the semantic relevance of the English word, and calculating a second vocabulary weight according to the first vocabulary weight and the semantic relevance;

[0010] Input the English word into the machine learning model to calculate an offset value, calculate an offset weight according to the offset value and a preset knowledge graph, and calculate a third vocabulary weight according to the offset weight and the second vocabulary weight;

[0011] Calculating the part-of-speech confidence of the word by using the preset part-of-speech probability, and calculating the final word weight according to the part-of-speech confidence and the third word weight;

[0012] Generating a text vector of the English sentence according to the final vocabulary weight;

[0013] The text vector is input into a preset translation decoder to generate a translation result in the target language.

[0014] As an optional implementation manner, counting the occurrence frequency of the English words and calculating the first vocabulary weight includes:

[0015] Count the occurrence frequency of the English words;

[0016] The word activity is calculated based on the frequency of occurrence, and the calculation formula is:

[0017]

[0018] Among them, a j is the word activity, f j is the frequency of occurrence;

[0019] The first vocabulary weight is calculated according to the word activity, and its calculation formula is:

[0020] w first =a j α

[0021] Among them, w first is the first word weight, α is the activity adjustment parameter, and its value range is [0.1,10].

[0022] As an optional implementation, the calculating the semantic association of the English word and calculating the second vocabulary weight according to the first vocabulary weight and the semantic association includes:

[0023] The semantic relevance of the English words is calculated using the following formula:

[0024]

[0025] Among them, r j is the semantic association between the jth English word and the kth English word, sim(v j ,v k ) indicates v j With v k The cosine similarity between j is the jth English word, v k is the kth English word, and m is the total number of English words;

[0026] The second vocabulary weight of the English word is calculated according to the semantic association, and the calculation formula is:

[0027] w second =w first ·(1+r j )

[0028] Among them, w second is the second word weight, w first is the first word weight.

[0029] As an optional implementation, the English word is input into a machine learning model to calculate an offset value, an offset weight is calculated according to the offset value and a preset knowledge graph, and a third vocabulary weight is calculated according to the offset weight and the second vocabulary weight, including:

[0030] Input the English word into the machine learning model to calculate the offset value of the word;

[0031] The offset weight is calculated based on the offset value and a preset knowledge graph, wherein the preset knowledge graph includes the feature importance weight and feature quantity of English words, and the offset weight is calculated as follows:

[0032]

[0033] Among them, θ is the bias weight, δ i is the offset value, K j is the feature importance weight, the value range is [0,1];

[0034] The third word weight is obtained by combining the preset knowledge graph. The calculation method of the third word weight is:

[0035] w third =w second (θ β + (1-θ) γ)

[0036] Among them, w third is the third vocabulary weight, β is the first offset adjustment parameter, and its value range is [0.5, 1.5]; γ is the second offset adjustment parameter, and its value range is [0.5, 2.0].

[0037] As an optional implementation, the part-of-speech confidence of the word is calculated by a preset part-of-speech probability, and the final word weight is calculated according to the part-of-speech confidence and the third word weight, including:

[0038] The word's part-of-speech confidence is calculated using the preset part-of-speech probability, which includes the part-of-speech occurrence probability and the total part-of-speech probability. The calculation formula for the part-of-speech confidence is:

[0039]

[0040] Among them, conf pos is the part-of-speech confidence, f pos is the probability of part of speech occurrence, F total is the total probability of part of speech, which means the sum of the probability of occurrence of the part of speech of all words in the English sentence;

[0041] The final vocabulary weight is calculated according to the part-of-speech confidence, and its calculation formula is:

[0042] w final =w third (1+conf pos )

[0043] Among them, w final is the final vocabulary weight, w third is the third word weight.

[0044] As an optional implementation manner, generating the text vector of the English sentence according to the final vocabulary weight includes:

[0045] The sentence weight of the English sentence is calculated by the final vocabulary weight, and the calculation formula is:

[0046]

[0047] Among them, S i is the sentence weight of the i-th English sentence, w final,j is the final vocabulary weight of the jth word in the i-th English sentence; n i is the total number of words in the i-th English sentence, used to indicate the length of the sentence;

[0048] By normalizing the sentence length and text length, a text vector is generated, which is calculated as follows:

[0049]

[0050] Among them, V text is a text vector, p is the total number of sentences in the text, and is used to indicate the length of the text.

[0051] As an optional implementation, the inputting the text vector into a preset translation decoder to generate a translation result in a target language includes:

[0052] Based on the received text vector, the initial context representation is calculated through the attention mechanism, the decoder generates the probability distribution of the first word, and selects the word with the highest probability to add to the blank target language text sequence;

[0053] The attention mechanism is used to calculate the correlation between the current input text vector and the generated target language text sequence to obtain a weighted context representation to update the current context representation.

[0054] Based on the current context representation, the decoder generates the next word and a probability distribution, which represents the probability of selecting the next word;

[0055] Select the word with the highest probability from the probability distribution and add it to the target language text sequence;

[0056] When the generated word is a terminal symbol, the decoder stops generating and outputs the final target language text.

[0057] In a second aspect, the present invention further provides an intelligent translation system based on big data, comprising:

[0058] A data acquisition module is used to acquire initial English text data and perform sentence segmentation to obtain English sentences and English words;

[0059] A frequency calculation module, used to count the occurrence frequency of the English words and calculate the first vocabulary weight;

[0060] A semantic relevance module, used to calculate the semantic relevance of the English word, and calculate a second vocabulary weight according to the first vocabulary weight and the semantic relevance;

[0061] A knowledge graph fusion module, configured to input the English word into a machine learning model to calculate an offset value, calculate an offset weight according to the offset value and a preset knowledge graph, and calculate a third vocabulary weight according to the offset weight and the second vocabulary weight;

[0062] A part-of-speech confidence optimization module, used to calculate the part-of-speech confidence of a word by using a preset part-of-speech probability, and to calculate a final word weight based on the part-of-speech confidence and the third word weight;

[0063] A text vector generation module, used to generate a text vector of the English sentence according to the final vocabulary weight;

[0064] The translation generation module is used to generate a translation result in a target language based on the text vector.

[0065] In a third aspect, the present invention also provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the processor implements any one of the above-mentioned big data-based intelligent translation methods.

[0066] In a fourth aspect, the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned big data-based intelligent translation methods.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] The present invention converts English text into target language text with contextual and semantic accuracy through a series of computational steps, aiming to enhance the translation system's understanding of the original text and the quality of translation. The accuracy of the translation is improved by considering the contextual relevance of the vocabulary. Part-of-speech changes and polysemy are better handled by calculating part-of-speech confidence. The use of knowledge graphs provides additional semantic information, enhancing the semantic understanding of the translation. The entire process is automated, improving translation efficiency.

[0069] The present invention proposes an intelligent English translation method based on big data, which pre-processes the English text from the vocabulary dimension and sentence dimension before translation, effectively connects the vocabulary parts of speech in the English text with the context, thereby improving the accuracy of subsequent translation of the English text. The present invention can obtain the text vector of the English text through the steps of encoder processing and vocabulary set construction, and translate the English text according to the text vector, thereby improving the efficiency and accuracy of translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flowchart of an intelligent translation method based on big data provided by the first embodiment of the present invention;

[0071] Figure 2 It is a schematic diagram of the structure of an intelligent translation system based on big data provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] In order to solve the above technical problems, refer to Figure 1 The first embodiment of the present invention provides an intelligent translation method based on big data, comprising the following steps:

[0074] S11, obtaining initial English text data and dividing it into sentences to obtain English sentences and English words;

[0075] S12, counting the occurrence frequency of the English words and calculating a first vocabulary weight;

[0076] S13, calculating the semantic relevance of the English word, and calculating a second vocabulary weight according to the first vocabulary weight and the semantic relevance;

[0077] S14, inputting the English word into the machine learning model to calculate an offset value, calculating an offset weight according to the offset value and a preset knowledge graph, and calculating a third vocabulary weight according to the offset weight and the second vocabulary weight;

[0078] S15, calculating the part-of-speech confidence of the word by using the preset part-of-speech probability, and calculating the final word weight according to the part-of-speech confidence and the third word weight;

[0079] S16, generating a text vector of the English sentence according to the final vocabulary weight;

[0080] S17, inputting the text vector into a preset translation decoder to generate a translation result in a target language.

[0081] The present invention proposes an intelligent English translation method based on big data, which pre-processes the English text from the vocabulary dimension and sentence dimension before translation, effectively connects the vocabulary parts of speech in the English text with the context, thereby improving the accuracy of subsequent translation of the English text. The present invention can obtain the text vector of the English text through the steps of encoder processing and vocabulary set construction, and translate the English text according to the text vector, thereby improving the efficiency and accuracy of translation.

[0082] In step S11, it is necessary to obtain initial English text data and perform sentence segmentation to obtain English sentences and English words.

[0083] First, a large amount of English text data can be obtained from English corpora. These corpora cover multiple fields such as literature, news, and technology, and contain hundreds of millions of words of English text, which can well ensure the diversity and comprehensiveness of the corpus. After obtaining the corpus data, it needs to be preprocessed. Through methods such as regular expressions, non-English characters, numbers, punctuation marks, etc. in the text are removed, and all letters are converted to lowercase. After preprocessing, text data consisting of pure English words is obtained.

[0084] Next, the text data is divided into sentences using a rule-based method. By identifying punctuation marks such as periods, question marks, exclamation marks, and using a maximum matching algorithm, the text is divided into independent sentences. On this basis, the sentence data obtained by the division is processed by word segmentation. Commonly used English word segmentation algorithms have methods based on dictionaries, such as maximum matching algorithms, minimum word count algorithms, etc., and methods based on statistical machine learning are also available, such as conditional random fields (CRF), hidden Markov models (HMM), etc. In the present embodiment, a maximum matching algorithm based on a dictionary is used, and a pre-built English dictionary is used to perform forward and reverse maximum matching segmentation on sentences to obtain the English word sequence contained in each sentence. On the basis of word segmentation, it is also necessary to normalize the word data, convert the word into a stem or root form, remove stop words such as "the", "a", and "an", and obtain standardized English word data.

[0085] Finally, the sentence data and the corresponding word data are mapped to form sentence-word correspondence data. After verification and statistical analysis of the data, it is found that among the 1 million sentences processed, the average sentence length is 20 words, the longest sentence contains 150 words, and the shortest sentence contains only 1 word, and the sentence length presents a normal distribution. At the same time, the data contains a total of 500,000 different English words, among which the most frequently appearing word is "the", which appears 1 million times, and there are 200,000 words that appear only once. Through the above series of processing, high-quality English sentence and English word data are finally obtained.

[0086] In step S12, the occurrence frequency of the English words is counted to calculate the first vocabulary weight, including:

[0087] Count the occurrence frequency of the English words;

[0088] The word activity is calculated based on the frequency of occurrence, and the calculation formula is:

[0089]

[0090] Among them, a j is the word activity, f j is the frequency of occurrence;

[0091] The first vocabulary weight is calculated according to the word activity, and its calculation formula is:

[0092] w first =a j α

[0093] Among them, w first is the first word weight, α is the activity adjustment parameter, and its value range is [0.1,10].

[0094] It should be noted that the activity of a word in this example is defined as the inverse of the frequency of occurrence. This means that the less frequent a word is, the more active it is. Although this is counterintuitive, as it is believed that words with a high frequency of occurrence are more "active". However, in the context of translation, this definition is intended to emphasize words that are uncommon in a particular text but have a specific meaning. At the same time, this formula adjusts the activity of a word through an activity adjustment parameter to obtain the first vocabulary weight. This weight can reflect the importance of a word in a particular text. The adjustment parameter allows the system to adjust the word weight according to the specific needs of the translation task, increasing the flexibility of the method. The word weight can be flexibly adjusted by adjusting the parameter to adapt to different translation scenarios and needs. This method emphasizes the importance of a word in a specific context, rather than just based on its general frequency in a large corpus. By considering the activity and weight of a word, the translation system can more accurately capture the intent and sentiment of the original text, thereby improving the naturalness and accuracy of the translation.

[0095] In step S13, the semantic relevance of the English word is calculated, and a second vocabulary weight is calculated according to the first vocabulary weight and the semantic relevance, including:

[0096] The semantic relevance of the English words is calculated using the following formula:

[0097]

[0098] Among them, r j is the semantic association between the jth English word and the kth English word, sim(v j ,v k ) indicates v j With v k The cosine similarity between j is the jth English word, v k is the kth English word, and m is the total number of English words;

[0099] The second vocabulary weight of the English word is calculated according to the semantic association, and the calculation formula is:

[0100] w second =w first ·(1+r j )

[0101] Among them, w second is the second word weight, w first is the first word weight.

[0102] It is worth noting that cosine similarity is a method for measuring the similarity between text vectors. It can reflect the similarity of two vectors in direction, regardless of their size. This method can capture the semantic relationship between words, not just based on word frequency statistics. In addition, this formula adds the semantic relevance r to the first word weight. j The relevant weights are used to calculate the second vocabulary weight, taking into account not only the statistical characteristics of the word (through w first ), and also considers the semantic properties of words (through r j ), thus providing richer information for the translation system.

[0103] In this embodiment, by considering the semantic relationship between words, the translation system's ability to understand the text semantically is enhanced. By combining statistical and semantic information, the intent of the original text can be captured more accurately, thereby improving the accuracy of the translation. For words with multiple meanings, this method can determine the most appropriate translation based on the semantic relevance of the context, thereby better handling polysemous words.

[0104] In step S14, the English word is input into the machine learning model to calculate an offset value, an offset weight is calculated according to the offset value and a preset knowledge graph, and a third vocabulary weight is calculated according to the offset weight and the second vocabulary weight, including:

[0105] Input the English word into the machine learning model to calculate the offset value of the word;

[0106] The offset weight is calculated based on the offset value and a preset knowledge graph, wherein the preset knowledge graph includes the feature importance weight and feature quantity of English words, and the offset weight is calculated as follows:

[0107]

[0108] Among them, θ is the bias weight, δ i is the offset value, K j is the feature importance weight, the value range is [0,1];

[0109] The third word weight is obtained by combining the preset knowledge graph. The calculation method of the third word weight is:

[0110] w third =w second (θ β + (1-θ) γ)

[0111] Among them, w third is the third vocabulary weight, β is the first offset adjustment parameter, and its value range is [0.5, 1.5]; γ is the second offset adjustment parameter, and its value range is [0.5, 2.0].

[0112] Among them, in this example, presetting the knowledge graph is a key step, which involves collecting and organizing entity and relationship information related to translation in order to provide additional semantic information and contextual understanding during the translation process. First, natural language text data is collected, and preprocessed, cleaned and tokenized to prepare data for building a knowledge graph. Entities and relationships are extracted from text data through natural language processing techniques (such as named entity recognition, relationship extraction, etc.). Next, entity-relationship-entity (ERG) triples are constructed, and the extracted entities and relationships are organized into ERG triples to form a knowledge graph. These triples contain the type of entity, the relationship between entities, and related attributes or features. Then, the feature importance weight is determined, and a feature importance weight is assigned to each entity and relationship in the knowledge graph, which can be determined based on statistical analysis or machine learning models. For example, statistical methods such as correlation analysis, chi-square test, etc. are used to evaluate the relationship between different features and target variables (such as translation quality, information retrieval accuracy, etc.). Through these analyses, it can be determined which features have a strong correlation with the target variable, so that they are assigned higher weights. Alternatively, build machine learning models, such as support vector machines, random forests, etc., to predict the importance of entities. During model training, feature importance weights can be determined by feature importance scores. For example, a random forest model can provide feature importance scores that can be used to adjust the weights of entities in the knowledge graph.

[0113] It should be noted that by setting the offset value δ of each word i and its feature importance weight K j Multiply and sum, then divide by the sum of all feature importance weights to calculate the offset weight θ. This method can adjust the impact of the offset value according to the feature importance of the word, so as to more accurately reflect the meaning of the word in a specific context. At the same time, the second vocabulary weight and the offset weight are combined to calculate the third vocabulary weight. The first offset adjustment parameter and the second offset adjustment parameter allow the system to adjust the word weight according to the specific needs of the translation task, which increases the flexibility of the method. This embodiment enhances the translation system's ability to understand the text context by considering the offset value of the word in the machine learning model and the feature importance in the knowledge graph. At the same time, combining semantic relevance, offset value and feature importance, the intention of the original text can be captured more accurately, thereby improving the naturalness and accuracy of the translation. In particular, for words with multiple meanings or terms in a specific field, the most appropriate translation can be determined based on their feature importance, thereby better handling polysemous words and professional terms.

[0114] In step S15, the part-of-speech confidence of the word is calculated by the preset part-of-speech probability, and the final word weight is calculated according to the part-of-speech confidence and the third word weight, including:

[0115] The part-of-speech confidence of a word is calculated by the preset part-of-speech probability, which includes the part-of-speech occurrence probability and the total part-of-speech probability. The calculation formula of the part-of-speech confidence is:

[0116]

[0117] Among them, conf pos is the part-of-speech confidence, f pos is the probability of part of speech occurrence, F total is the total probability of part of speech, which means the sum of the probability of occurrence of the part of speech of all words in the English sentence;

[0118] The final vocabulary weight is calculated according to the part-of-speech confidence, and its calculation formula is:

[0119] w final =w third (1+conf pos )

[0120] Among them, w final is the final vocabulary weight, w third is the third word weight.

[0121] It is worth noting that by calculating the part-of-speech confidence by dividing the probability of occurrence of a specific part of speech by the total probability of all parts of speech, it is possible to quantify whether a word belongs to a specific part of speech, thereby providing the translation system with information about the word's part of speech. At the same time, by adding the weight associated with the part-of-speech confidence to the third lexical weight to calculate the final lexical weight, the semantic, statistical and part-of-speech information of the word is combined to provide the translation system with a more comprehensive lexical weight. For complex linguistic phenomena such as part-of-speech changes and polysemy, this method can provide more detailed processing to better adapt to these phenomena. The calculation of the final lexical weight takes into account the part-of-speech information, which helps to optimize the translation results and make them more in line with the grammar and expression habits of the target language.

[0122] In step S16, a text vector of the English sentence is generated according to the final vocabulary weight, including:

[0123] The sentence weight of the English sentence is calculated by the final vocabulary weight, and the calculation formula is:

[0124]

[0125] Among them, S i is the sentence weight of the i-th English sentence, w final,j is the final vocabulary weight of the jth word in the ith English sentence, n i is the total number of words in the i-th English sentence, used to indicate the length of the sentence;

[0126] By normalizing the sentence length and text length, a text vector is generated, which is calculated as follows:

[0127]

[0128] Among them, V text is a text vector, p is the total number of sentences in the text, and is used to indicate the length of the text.

[0129] It should be noted that by adding up the final vocabulary weights of all words in a sentence and then dividing it by the total number of words to calculate the sentence weight, the importance of each word in the sentence can be comprehensively considered to obtain the overall weight of the sentence. Next, the text vector is generated by adding up the weights of all sentences and then dividing it by the total number of sentences. Through normalization, the text vector can reflect the central trend of the entire text while reducing the impact of the length of a single sentence on the text vector.

[0130] In the above formula, the vector representation of the entire text is obtained by averaging the sentence weights, thereby taking into account the importance of each sentence in the text. At the same time, the weights of all sentences are treated equally when calculating the average value, regardless of their length. This embodiment comprehensively considers the final vocabulary weight of each word in the sentence, which helps to capture the semantic information of the sentence. By normalizing the sentence length and text length, the text vector can more fairly reflect the semantic features of the entire text, rather than being determined solely by the number or length of sentences.

[0131] In step S17, the text vector is input into a preset translation decoder to generate a translation result in the target language, including:

[0132] Based on the received text vector, the initial context representation is calculated through the attention mechanism, the decoder generates the probability distribution of the first word, and selects the word with the highest probability to add to the blank target language text sequence;

[0133] The attention mechanism is used to calculate the correlation between the current input text vector and the generated target language text sequence to obtain a weighted context representation to update the current context representation.

[0134] Based on the current context representation, the decoder generates the next word and a probability distribution, which represents the probability of selecting the next word;

[0135] Select the word with the highest probability from the probability distribution and add it to the target language text sequence;

[0136] When the generated word is a terminal symbol, the decoder stops generating and outputs the final target language text.

[0137] In one implementation, the context representation is first calculated. Based on the received text vector, the initial context representation is calculated through the attention mechanism, the decoder generates the probability distribution of the first word, and the word with the highest probability is selected and added to the blank target language text sequence; the correlation between the current input text vector and the generated target language text sequence is calculated through the attention mechanism to obtain the weighted context representation to update the current context representation.

[0138] Exemplarily, the decoder receives a text vector (or context vector) converted from the source language text. This vector is calculated by the word weights of the source language sentence and contains the overall semantic information of the source language sentence. The generated target language word sequence will be encoded into a vector to represent the progress and context information of the current translation. The encoding method may include a Transformer architecture, which uses a self-attention mechanism to encode the generated word sequence to capture the global dependencies in the sequence. For example, assuming that the currently generated target language word sequence is [w1, w2, ..., wn], it is encoded using the self-attention mechanism in the Transformer architecture to obtain the encoded vector sequence [E1, E2, ..., En]. The attention mechanism is used to calculate the correlation between the input text vector and the encoded vector [E1, E2, ..., En] of the generated word sequence to obtain a weighted context representation. Then, the context representation is input to the output layer of the decoder (such as the Softmax layer) to obtain the probability distribution of the next word.

[0139] This representation is the key for the decoder to understand the text and original text vector generated so far. Then, the word and probability distribution are generated, and the decoder then generates the next word and the corresponding probability distribution based on the current context representation. This probability distribution reflects the probability of occurrence of each word given the context. Then, the highest probability word is selected, and the word with the highest probability is selected from the probability distribution and added to the target language text sequence. This selection process is the core of the decoder to generate translation, which determines the fluency and accuracy of the translation. Finally, the terminator is judged. When the word generated by the decoder is a terminator (such as a period or a question mark), the generation process is stopped and the final target language text is output. In this embodiment, the decoder takes contextual information into account when generating each word, which helps to generate a more contextual translation. By selecting words through probability distribution, the decoder can better handle ambiguity and polysemous words and improve the accuracy of translation. The decoder can adapt to different translation needs and generate translation texts with different styles and tones. In addition, the entire translation process is automated, without manual intervention, which improves translation efficiency.

[0140] For example, suppose there is an English sentence "The quick brown fox jumps over the lazydog" and you want to translate it into Chinese. First, the text vector of the sentence is input into the translation decoder. Context representation calculation: The decoder calculates the initial context representation. Word generation: The decoder starts generating the first word, such as "quick" (corresponding to "quick"), and gives the probability distribution of subsequent words. Select and add words: The decoder selects the word with the highest probability "brown" (corresponding to "brown") and adds it to the Chinese translation sequence. Repeat process: The decoder repeats this process until the translation of the entire sentence is generated, such as "The quick brown fox jumps over the lazy dog". End translation: When the end symbol is encountered, the decoder stops generating and outputs the complete Chinese translation "The quick brown fox jumps over the lazy dog".

[0141] To facilitate understanding of the present invention, the following is a specific translation process example:

[0142] Hypothetical original English text:

[0143] "The quick brown fox jumps over the lazy dog."

[0144] The translation process is as follows:

[0145] Step 1, data acquisition and sentence segmentation: The system first acquires the above English text and divides it into sentences and words.

[0146] Step 2, count the word frequency and calculate the first vocabulary weight: count the frequency of occurrence of each word; calculate the activity of each word (activity is defined as the inverse of the frequency of occurrence); calculate the first vocabulary weight according to the activity and activity adjustment parameter α.

[0147] Step 3, calculating the semantic association and the second vocabulary weight: using cosine similarity to calculate the semantic association between words; combining the first vocabulary weight, calculating the second vocabulary weight.

[0148] Step 4: The machine learning model calculates the offset value and the third vocabulary weight: input the word into the machine learning model and calculate the offset value; calculate the offset weight in combination with the feature importance weight in the knowledge graph; calculate the third vocabulary weight in combination with the offset weight and the offset adjustment parameter.

[0149] Step 5, calculate the part-of-speech confidence and final vocabulary weight: calculate the part-of-speech confidence according to the part-of-speech occurrence probability and the total probability; combine with the third vocabulary weight to optimize and obtain the final vocabulary weight.

[0150] Step 6, generate text vector: calculate the sentence weight of each sentence according to the final vocabulary weight; generate text vector by normalizing the sentence length and text length.

[0151] Step 7: The translation decoder generates the translation result: the text vector is input into the translation decoder; the decoder generates the next word and probability distribution based on the current context representation; the word with the highest probability is selected and added to the target language text sequence. This process is repeated until the end symbol is generated and the final translation result is output.

[0152] Hypothetical translation: "The quick brown fox jumps over the lazy dog."

[0153] This embodiment converts English text into target language text with contextual and semantic accuracy through a series of calculation steps, aiming to enhance the translation system's understanding of the original text and the quality of translation. The accuracy of translation is improved by considering the contextual relevance of vocabulary. Part-of-speech changes and polysemy are better handled by calculating part-of-speech confidence. The use of knowledge graphs provides additional semantic information, enhancing the semantic understanding of translation. The entire process is automated, improving translation efficiency.

[0154] In summary, the present invention proposes an intelligent English translation method based on big data, which pre-processes the English text from the vocabulary dimension and sentence dimension before translation, effectively connects the vocabulary parts of speech in the English text with the context, thereby improving the accuracy of subsequent translation of the English text. The present invention can obtain the text vector of the English text through the steps of encoder processing and vocabulary set construction, and translate the English text according to the text vector, thereby improving the efficiency and accuracy of translation.

[0155] Reference Figure 2 The second embodiment of the present invention provides an intelligent translation system based on big data, including:

[0156] A data acquisition module is used to acquire initial English text data and divide it into sentences to obtain English sentences and English words;

[0157] A frequency calculation module, used to count the occurrence frequency of the English words and calculate the first vocabulary weight;

[0158] A semantic relevance module, used to calculate the semantic relevance of the English word, and calculate a second vocabulary weight according to the first vocabulary weight and the semantic relevance;

[0159] A knowledge graph fusion module, configured to input the English word into a machine learning model to calculate an offset value, calculate an offset weight according to the offset value and a preset knowledge graph, and calculate a third vocabulary weight according to the offset weight and the second vocabulary weight;

[0160] A part-of-speech confidence optimization module, used to calculate the part-of-speech confidence of a word by using a preset part-of-speech probability, and to calculate a final word weight based on the part-of-speech confidence and the third word weight;

[0161] A text vector generation module, used to generate a text vector of the English sentence according to the final vocabulary weight;

[0162] The translation generation module is used to generate a translation result in a target language based on the text vector.

[0163] Preferably, the frequency calculation module is used to:

[0164] Count the occurrence frequency of the English words;

[0165] The word activity is calculated based on the frequency of occurrence, and the calculation formula is:

[0166]

[0167] Among them, a j is the word activity, f j is the frequency of occurrence;

[0168] The first vocabulary weight is calculated according to the word activity, and its calculation formula is:

[0169] w first =a j α

[0170] Among them, w first is the first word weight, and α is the activity adjustment parameter.

[0171] Preferably, the semantic relevance module is used to:

[0172] The semantic relevance of the English words is calculated using the following formula:

[0173]

[0174] Among them, r j is the semantic relevance of the jth English word, sim(v j ,v k ) indicates v j With v k The cosine similarity between j is the jth English word, v k is the kth English word, and m is the total number of English words;

[0175] The second vocabulary weight of the English word is calculated according to the semantic association, and the calculation formula is:

[0176] w second =w first ·(1+r j )

[0177] Among them, w second is the second word weight, w first is the first word weight.

[0178] Preferably, the knowledge graph fusion module is used to:

[0179] Input the English word into the machine learning model to calculate the offset value of the word;

[0180] The offset weight is calculated based on the offset value and a preset knowledge graph, wherein the preset knowledge graph includes the feature importance weight and feature quantity of English words, and the offset weight is calculated as follows:

[0181]

[0182] Among them, θ is the bias weight, δ i is the offset value, K j is the feature importance weight;

[0183] The third word weight is obtained by combining the preset knowledge graph. The calculation method of the third word weight is:

[0184] w third =w second (θ β + (1-θ) γ)

[0185] Among them, w third is the third vocabulary weight, β is the first offset adjustment parameter, and γ is the second offset adjustment parameter.

[0186] Preferably, the part-of-speech confidence optimization module is used to:

[0187] The word's part-of-speech confidence is calculated using the preset part-of-speech probability, which includes the part-of-speech occurrence probability and the total part-of-speech probability. The calculation formula for the part-of-speech confidence is:

[0188]

[0189] Among them, conf pos is the part-of-speech confidence, f pos is the probability of part of speech occurrence, F total is the total probability of part of speech;

[0190] The final vocabulary weight is calculated according to the part-of-speech confidence, and its calculation formula is:

[0191] w final =w third (1+confpos )

[0192] Among them, w final is the final vocabulary weight, w third is the third word weight.

[0193] Preferably, the text vector generating module is used to:

[0194] The sentence weight of the English sentence is calculated by the final vocabulary weight, and the calculation formula is:

[0195]

[0196] Among them, S i is the sentence weight of the i-th English sentence, w final,j is the final vocabulary weight of the jth word in the ith English sentence, n i is the total number of words in the i-th English sentence, used to indicate the length of the sentence;

[0197] By normalizing the sentence length and text length, a text vector is generated, which is calculated as follows:

[0198]

[0199] Among them, V text is a text vector, p is the total number of sentences in the text, and is used to indicate the length of the text.

[0200] Preferably, the translation generation module is used to:

[0201] Based on the received text vector, the initial context representation is calculated through the attention mechanism, the decoder generates the probability distribution of the first word, and selects the word with the highest probability to add to the blank target language text sequence;

[0202] The attention mechanism is used to calculate the correlation between the current input text vector and the generated target language text sequence to obtain a weighted context representation to update the current context representation.

[0203] Based on the current context representation, the decoder generates the next word and a probability distribution, which represents the probability of selecting the next word;

[0204] Select the word with the highest probability from the probability distribution and add it to the target language text sequence;

[0205] When the generated word is a terminal symbol, the decoder stops generating and outputs the final target language text.

[0206] In summary, the present invention proposes an intelligent English translation method based on big data, which pre-processes the English text from the vocabulary dimension and sentence dimension before translation, effectively connects the vocabulary parts of speech in the English text with the context, thereby improving the accuracy of subsequent translation of the English text. The present invention can obtain the text vector of the English text through the steps of encoder processing and vocabulary set construction, and translate the English text according to the text vector, thereby improving the efficiency and accuracy of translation.

[0207] It should be noted that the intelligent translation system based on big data provided in an embodiment of the present invention is used to execute all the process steps of the intelligent translation method based on big data in the above embodiment. The working principles and beneficial effects of the two correspond one to one, so they will not be repeated here.

[0208] The embodiment of the present invention further provides an electronic device. The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as an intelligent translation program based on big data. When the processor executes the computer program, the steps in the above-mentioned intelligent translation method embodiments based on big data are implemented, such as Figure 1 Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned system embodiments are realized, such as the translation generation module.

[0209] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the electronic device.

[0210] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, and a smart tablet. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than the above components, or may combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0211] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0212] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the electronic device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0213] Wherein, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or system capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0214] It should be noted that the system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiment provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0215] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent translation method based on big data, characterized in that: Executed by a computer, including: Obtain initial English text data and perform sentence segmentation to obtain English sentences and English words; Counting the occurrence frequency of the English words and calculating a first vocabulary weight; Calculating the semantic relevance of the English word, and calculating a second vocabulary weight according to the first vocabulary weight and the semantic relevance; Input the English word into the machine learning model to calculate an offset value, calculate an offset weight according to the offset value and a preset knowledge graph, and calculate a third vocabulary weight according to the offset weight and the second vocabulary weight; Calculating the part-of-speech confidence of the word by using the preset part-of-speech probability, and calculating the final word weight according to the part-of-speech confidence and the third word weight; Generating a text vector of the English sentence according to the final vocabulary weight; The text vector is input into a preset translation decoder to generate a translation result in the target language.

2. The intelligent translation method based on big data according to claim 1, characterized in that: The counting of the occurrence frequencies of the English words to calculate the first vocabulary weight includes: Count the occurrence frequency of the English words; The word activity is calculated based on the frequency of occurrence, and the calculation formula is: Among them, a j is the word activity, f j is the frequency of occurrence; The first vocabulary weight is calculated according to the word activity, and its calculation formula is: w first =a j ·a Among them, w first is the first word weight, α is the activity adjustment parameter, and its value range is [0.1,10].

3. The intelligent translation method based on big data according to claim 1, characterized in that: The calculating of the semantic relevance of the English words and calculating the second vocabulary weight according to the first vocabulary weight and the semantic relevance includes: The semantic relevance of the English words is calculated using the following formula: Among them, r j is the semantic association between the jth English word and the kth English word, sim(v j ,v k ) indicates v j With v k The cosine similarity between j is the jth English word, v k is the kth English word, and m is the total number of English words; The second vocabulary weight of the English word is calculated according to the semantic association, and the calculation formula is: w second =w first ·(1+r j ) Among them, w second is the second word weight, w first is the first word weight.

4. The intelligent translation method based on big data according to claim 1, characterized in that: Input the English word into the machine learning model to calculate an offset value, calculate an offset weight according to the offset value and a preset knowledge graph, and calculate a third vocabulary weight according to the offset weight and the second vocabulary weight, including: Input the English word into the machine learning model to calculate the offset value of the word; The offset weight is calculated based on the offset value and a preset knowledge graph, wherein the preset knowledge graph includes the feature importance weight and feature quantity of English words, and the offset weight is calculated as follows: Among them, θ is the bias weight, δ i is the offset value, K j is the feature importance weight, the value range is [0,1]; The third word weight is obtained by combining the preset knowledge graph. The calculation method of the third word weight is: w third =w second ·(θ·β+(1-θ)·γ) Among them, w third is the third vocabulary weight; β is the first offset adjustment parameter, and its value range is [0.5,1.5]; γ is the second offset adjustment parameter, and its value range is [0.5,2.0].

5. The intelligent translation method based on big data according to claim 1, characterized in that: The part-of-speech confidence of the word is calculated by using the preset part-of-speech probability, and the final word weight is calculated according to the part-of-speech confidence and the third word weight, including: The word's part-of-speech confidence is calculated using the preset part-of-speech probability, which includes the part-of-speech occurrence probability and the total part-of-speech probability. The calculation formula for the part-of-speech confidence is: Among them, conf pos is the part-of-speech confidence, f pos is the probability of part of speech occurrence, F total is the total probability of part of speech, which means the sum of the probability of occurrence of the part of speech of all words in the English sentence; The final vocabulary weight is calculated according to the part-of-speech confidence, and its calculation formula is: w final =w third ·(1+conf pos ) Among them, w final is the final vocabulary weight, w third is the third word weight.

6. The intelligent translation method based on big data according to claim 1, characterized in that: Generating the text vector of the English sentence according to the final vocabulary weight includes: The sentence weight of the English sentence is calculated by the final vocabulary weight, and the calculation formula is: Among them, S i is the sentence weight of the i-th English sentence, w final,j is the final vocabulary weight of the jth word in the i-th English sentence; n i is the total number of words in the i-th English sentence, used to indicate the length of the sentence; By normalizing the sentence length and text length, a text vector is generated, which is calculated as follows: Among them, V text is a text vector; p is the total number of sentences in the text, which is used to indicate the length of the text.

7. The intelligent translation method based on big data according to claim 1, characterized in that: The step of inputting the text vector into a preset translation decoder to generate a translation result in a target language includes: Based on the received text vector, the initial context representation is calculated through the attention mechanism, the decoder generates the probability distribution of the first word, and selects the word with the highest probability to add to the blank target language text sequence; The attention mechanism is used to calculate the correlation between the current input text vector and the generated target language text sequence to obtain a weighted context representation to update the current context representation. Based on the current context representation, the decoder generates the next word and a probability distribution, which represents the probability of selecting the next word; Select the word with the highest probability from the probability distribution and add it to the target language text sequence; When the generated word is a terminal symbol, the decoder stops generating and outputs the final target language text.

8. An intelligent translation system based on big data, characterized in that: include: A data acquisition module is used to acquire initial English text data and divide it into sentences to obtain English sentences and English words; A frequency calculation module, used to count the occurrence frequency of the English words and calculate the first vocabulary weight; A semantic relevance module, used to calculate the semantic relevance of the English word, and calculate a second vocabulary weight according to the first vocabulary weight and the semantic relevance; A knowledge graph fusion module, configured to input the English word into a machine learning model to calculate an offset value, calculate an offset weight according to the offset value and a preset knowledge graph, and calculate a third vocabulary weight according to the offset weight and the second vocabulary weight; A part-of-speech confidence optimization module, used to calculate the part-of-speech confidence of a word by using a preset part-of-speech probability, and to calculate a final word weight based on the part-of-speech confidence and the third word weight; A text vector generation module, used to generate a text vector of the English sentence according to the final vocabulary weight; The translation generation module is used to generate a translation result in a target language based on the text vector.

9. An electronic device, characterized in that: The invention comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the intelligent translation method based on big data as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the big data-based intelligent translation method according to any one of claims 1 to 7.

Images