A deep learning-based multilingual translation method and system

Abstract

The application belongs to the technical field of real-time translation, and discloses a multilingual translation method and system based on deep learning, which comprises the following steps: collecting a voice signal, performing feature extraction and fusion on the voice signal to obtain voice features; identifying the voice features based on a pre-trained voice recognition model to obtain a voice recognition result; generating a language text to be translated based on the voice recognition result, performing feature extraction on the language text to be translated to obtain language word embedding representation; expanding the language word embedding representation based on a pre-constructed sample vocabulary representation to obtain expanded language word embedding representation; and inputting the expanded language word embedding representation into a pre-trained translation model, wherein the translation model outputs a multilingual translation text. The application can reduce the influence of background noise, improve the accuracy of voice recognition, and the high-accuracy voice recognition result is conducive to improving the accuracy of translation, and the accuracy of small language translation is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of real-time translation, and particularly relates to a multilingual translation method and system based on deep learning. BACKGROUND

[0002] With the development of China's international trade services, China is promoting the high-quality development of service trade at a high level of opening-up and contributing more to global economic development. The service trade fair has attracted a large number of domestic and foreign enterprises to participate in the exhibition and display the latest achievements and technologies in the fields of digitization, intelligentization and greenization.

[0003] Therefore, with the deepening of exchanges between China and the international community, multilingual translation technology can solve the problem of language incompatibility between domestic and foreign languages. The traditional multilingual translation method is to find the corresponding English of the required language by manual searching, then find the translation of the required language through the StringID translation index of English, and then update the correct translation to the target language library, and then realize the packaging of the language library supported by the product into the system software. This multilingual translation method has great defects, and there are problems such as gray areas of manual searching and the need for a large amount of manpower, that is, this multilingual translation method will cause low development efficiency of multilingual translation due to the need for manual searching to realize translation.

[0004] With the development of deep learning, translation software trained by deep learning, i.e. neural machine translation, can automatically convert one natural language (source language) into another natural language (target language). However, such translation software still has the following problems in the application process of real-time translation:

[0005] 1. In the process of real-time communication between people, real-time speech needs to be collected, and there are often a large amount of background noise in the process of collecting speech, which leads to low speech recognition accuracy and in turn leads to low translation accuracy, which is prone to translation errors;

[0006] 2. Neural machine translation has achieved remarkable translation quality at the level of millions of sentences. For small languages, due to the lack of massive small language corpus for training, the accuracy of small language translation is low. SUMMARY

[0007] The purpose of the present application is to provide a multilingual translation method and system based on deep learning to solve the problem of low translation accuracy in the process of real-time communication of existing translation methods.

[0008] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0009] In the first aspect, the present application provides a multilingual translation method based on deep learning, which comprises:

[0010] collecting a voice signal, performing feature extraction and fusion on the voice signal to obtain voice features;

[0011] performing voice recognition on the voice features based on a pre-trained voice recognition model to obtain a voice recognition result;

[0012] generating a language text to be translated based on the voice recognition result, performing feature extraction on the language text to be translated to obtain a language word embedding representation;

[0013] extending the language word embedding representation based on a pre-constructed sample vocabulary representation to obtain an extended language word embedding representation;

[0014] inputting the extended language word embedding representation into a pre-trained translation model, wherein the translation model outputs a multilingual translation text.

[0015] Preferably, the method further comprises:

[0016] obtaining a target language required by a user, and extracting a translation text in the target language from the multilingual translation text according to the target language required by the user;

[0017] sending the translation text in the target language to the user.

[0018] Preferably, the voice features are obtained by performing feature extraction and fusion on the voice signal, comprising:

[0019] performing feature extraction on the voice signal based on an MFCC algorithm to obtain a first feature set;

[0020] obtaining a power spectrum in the process of performing feature extraction on the voice signal by the MFCC algorithm;

[0021] determining a linear prediction signal of the voice signal based on the power spectrum, and calculating a prediction deviation signal according to the voice signal and the linear prediction signal;

[0022] performing feature extraction on the prediction deviation signal to obtain a second feature set;

[0023] performing equal-dimension fusion on the first feature set and the second feature set to obtain fused features, and taking the fused features as the voice features.

[0024] Preferably, the equal-dimension fusion on the first feature set and the second feature set to obtain the fused features comprises:

[0025] calculating the importance of each element in the first feature set to obtain a plurality of first importance values;

[0026] sorting the plurality of first importance values in descending order to obtain a first importance sequence;

[0027] Take elements in the first feature set corresponding to the first k first importance values in the first importance sequence as first to-be-fused features, where k is a positive integer;

[0028] Calculate the importance of each element in the second feature set to obtain a plurality of second importance values;

[0029] Sort the plurality of second importance values in descending order to obtain a second importance sequence;

[0030] Take elements in the second feature set corresponding to the first k second importance values in the second importance sequence as second to-be-fused features;

[0031] Multiply and fuse the k first to-be-fused features and the k second to-be-fused features to obtain fused features.

[0032] Preferably, the speech recognition model comprises a down-sampling layer, an encoder layer and a decoder layer connected in sequence; wherein the encoder layer is constructed based on an attention mechanism.

[0033] Preferably, the method further comprises constructing a sample vocabulary representation, comprising:

[0034] Obtaining general monolingual data, pre-processing the general monolingual data to obtain general monolingual corpus;

[0035] Translating the general monolingual corpus based on a preset translation tool to obtain a multilingual corpus;

[0036] Extracting features from the general monolingual corpus and the multilingual corpus based on a bag-of-words model to obtain general monolingual embedding representation and multilingual embedding representation;

[0037] Aligning the general monolingual embedding representation and the multilingual embedding representation, and after alignment, mapping the multilingual embedding representation into the general monolingual embedding representation to obtain a multilingual language shared representation, taking the multilingual language shared representation as the sample vocabulary representation.

[0038] Preferably, based on the pre-constructed sample vocabulary representation, the language word embedding representation is expanded to obtain an expanded language word embedding representation, comprising:

[0039] Obtaining the initial language of the language text to be translated;

[0040] According to the initial language of the language text to be translated and the language word embedding representation, matching the general monolingual embedding representation corresponding to the language from the sample vocabulary representation;

[0041] Splicing the language word embedding representation and the general monolingual embedding representation to obtain an expanded language word embedding representation.

[0042] Preferably, the method further comprises: taking the sample vocabulary representation as a training set, training a neural network based on an attention mechanism with the training set to obtain a trained neural network, and taking the trained neural network as a pre-trained translation model.

[0043] Preferably, the method further comprises:

[0044] characteristics of the multilingual translation text are extracted to obtain language embedding representation of the text;

[0045] The language embedding representation of the text is input into the translation model to obtain the reverse text in the initial language;

[0046] The loss value of the reverse text in the initial language and the language text to be translated is calculated;

[0047] The parameters of the translation model are gradient-optimized based on the loss value of the reverse text in the initial language and the language text to be translated.

[0048] In a second aspect, the present application provides a multilingual translation system based on deep learning, which is used to implement the multilingual translation method based on deep learning described above, and comprises:

[0049] The first extraction module is used to collect the speech signal, perform feature extraction and fusion on the speech signal, and obtain the speech feature.

[0050] The feature recognition module is used to recognize the speech feature based on the pre-trained speech recognition model to obtain the speech recognition result.

[0051] The second extraction module is used to generate the language text to be translated based on the speech recognition result, perform feature extraction on the language text to be translated, and obtain the language word embedding representation.

[0052] The feature expansion module is used to expand the language word embedding representation based on the pre-constructed sample vocabulary representation to obtain the expanded language word embedding representation.

[0053] The text translation module is used to input the expanded language word embedding representation into the pre-trained translation model, and the translation model outputs the multilingual translation text.

[0054] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multilingual translation method based on deep learning described above when executing the computer program.

[0055] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the multilingual translation method based on deep learning described above.

[0056] Beneficial effects:

[0057] 1、The application identifies the speech of different speakers in real-time translation and automatically converts it into corresponding language text, which is convenient for translation of language text; meanwhile, the speech signal is subjected to feature extraction and fusion, and the speech feature is identified by using a speech recognition model, which can reduce the influence of background noise and improve the accuracy of speech recognition, and the speech recognition result with high accuracy is conducive to improving the accuracy of translation;

[0058] 2、The language word embedding representation is expanded by the pre-constructed sample vocabulary representation, which can enrich the semantics of the language text to be translated, is conducive to improving the BLEU value of small language translation, and improves the accuracy of small language translation. BRIEF DESCRIPTION OF DRAWINGS

[0059] The accompanying drawings are used to provide a further understanding of the embodiments of the application, and constitute a part of the specification, and are used to explain the embodiments of the application together with the following detailed description, but do not constitute a limitation on the embodiments of the application. In the drawings:

[0060] Figure 1 is a flowchart of a multilingual translation method based on deep learning provided by an embodiment of the application;

[0061] Figure 2 is a block diagram of a multilingual translation system based on deep learning provided by an embodiment of the application. DETAILED DESCRIPTION

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the application will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the drawings structure is only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings. It should be noted that the description of these embodiment modes is used to help understand the application, but does not constitute a limitation on the application.

[0063] Embodiment one

[0064] Figure 1 is a flowchart of a multilingual translation method based on deep learning provided by an embodiment of the application. As shown in Figure 1 , the embodiment provides a multilingual translation method based on deep learning, which comprises:

[0065] Step S10: Collecting a speech signal, performing feature extraction and fusion on the speech signal, and obtaining a speech feature.

[0066] In the present embodiment, the voice signal of the speaker is collected by a microphone, and then feature extraction and fusion are performed on the voice signal. The specific steps of feature extraction and fusion are as follows:

[0067] Step S101: Feature extraction is performed on the voice signal based on the MFCC algorithm to obtain a first feature set. The MFCC (Mel-Frequency Cepstral Coefficients) algorithm is an important feature in audio signal processing, which is used to capture the spectral characteristics of audio signals and is widely used in speech recognition, music information retrieval, voiceprint recognition, etc. The steps of feature extraction based on the MFCC algorithm include:

[0068] Pre-emphasis: A high-pass filter is used to increase the energy of the high-frequency part to balance the frequency spectrum, making the high-frequency component more prominent and suitable for subsequent feature extraction.

[0069] Frame division: The audio signal is divided into multiple small segments (frames), each frame usually containing 20 to 40 milliseconds of data. There is usually a certain overlap (usually 50%) between adjacent frames.

[0070] Windowing: A window function (usually Hamming window) is applied to each frame of signal to reduce spectral leakage and make the connection of data within the frame smoother.

[0071] Fourier transform and power spectrum: Fast Fourier transform (FFT) is performed on the windowed signal to obtain the frequency spectrum of each frame. Then the power spectrum of each frame is calculated, i.e. the square of each frequency component.

[0072] Mel filter bank: A set of triangular filters is designed to map the frequency spectrum to the Mel frequency scale and perform weighted averaging on the frequency spectrum through these filters. The relationship between Mel frequency and normal frequency f is logarithmic, which simulates the nonlinear perception characteristics of the human ear to frequency.

[0073] Logarithmic transformation: Take the logarithm of each value output by the filter bank to simulate the human ear's perception of sound intensity.

[0074] Discrete cosine transform (DCT): Perform DCT on the log Mel spectrum to obtain Mel frequency cepstral coefficients (MFCC). The purpose of DCT is to compress the frequency spectrum into fewer coefficients and remove correlation; usually the first 12 to 13 coefficients are retained because the low-order coefficients contain the main spectral information and the high-order coefficients are often related to noise.

[0075] Feature normalization: when using MFCC feature vectors, in order to average the contribution of each dimension of the feature value to the prediction, the feature vector needs to be normalized; common normalization processing is cepstrum mean normalization (CMN) and global feature standardization; normalization can ensure that the dimensions of the feature vector are in a similar dynamic range, improving the stability and performance of the model.

[0076] Through the above steps, the first feature matrix of the speech signal can be extracted, and the first feature matrix is taken as the first feature set.

[0077] Step S102: obtaining the power spectrum of the speech signal in the feature extraction process of the MFCC algorithm.

[0078] Step S103: based on the power spectrum, determining the linear prediction signal of the speech signal, and calculating the prediction deviation signal according to the speech signal and the linear prediction signal.

[0079] In this embodiment, since the MFCC algorithm is in a noisy environment, these features are easily disturbed, resulting in poor speech recognition effect; therefore, the power spectrum of each frame of the speech signal is analyzed, for example, the Levinson-Durbin algorithm is used to analyze each frame of the power spectrum to obtain the LPC (Linear Predictive Coding) coefficient of each frame, the LPC coefficient represents the coefficient of each time delay component in the prediction signal, and the Levinson-Durbin algorithm is an effective linear prediction coefficient (LPC) calculation method, which is used in signal processing to solve the linear prediction model, and then each frame of the speech signal is linearly predicted by using the LPC coefficient to obtain the LPC prediction signal of each frame, and the LPC prediction signal is taken as the linear prediction signal of the speech signal; then each frame of the speech signal is subtracted from each frame of the LPC prediction signal to obtain the prediction deviation signal of each frame.

[0080] Step S104: performing feature extraction on the prediction deviation signal to obtain a second feature set; the specific steps of performing feature extraction on the prediction deviation signal by using the MFCC algorithm are as follows: the Mel filter bank converts each frame of the prediction deviation signal into a Mel frequency spectrum to simulate the sensitivity of the human auditory system to different frequencies; then the Mel frequency spectrum is logarithmically operated to convert the frequency information into a logarithmic scale, which is more in line with the human auditory perception characteristics; finally, the logarithmic Mel frequency spectrum is subjected to discrete cosine transform (DCT) to obtain a second feature matrix, and the second feature matrix is taken as the second feature set.

[0081] In this embodiment, by calculating the prediction deviation signal of each frame and then performing feature extraction on the prediction deviation signal, residual information in the speech signal that cannot be described by the MFCC can be extracted, which can effectively improve the accuracy of speech recognition.

[0082] Step S105: isodimensional fusion of the first feature set and the second feature set is performed to obtain fused features, and the fused features are taken as speech features.

[0083] In the embodiment, the dimensions of the first feature matrix and the second feature matrix are MxN, where M is the number of frames of the speech signal, and N is the number of mel filters, where the number of mel filters is 20-40; therefore, the first feature set and the second feature set can be spliced and fused, but the directly spliced features have a high dimension, which increases the complexity of model training and calculation, and there may be redundant information, and in a noisy environment, as the signal-to-noise ratio decreases, the acoustic features of the speech are destroyed.

[0084] Therefore, the first feature set and the second feature set are isodimensionally fused, unnecessary features can be removed in the isodimensional fusion process, and the feature dimension is reduced to solve the above problems. The first feature set and the second feature set are isodimensionally fused to obtain fused features, including:

[0085] Step a10: the importance of each element in the first feature set is calculated to obtain a plurality of first importance values; in the embodiment, the first feature set has MxN features, the first feature set has M elements in total, each element has N features, that is, one element corresponds to one frame of speech signal, and the influence degree of each element on the whole speech signal recognition needs to be judged.

[0086] For each element, any other element is selected from all elements to combine with the element to obtain a combination of each element, and the recognition accuracy of the combination of each element is calculated through experiments, and the recognition accuracy can be calculated through error rate and character error rate and the like;

[0087] The calculation expression of the first importance value of the i-th element is:

[0088] (1);

[0089] In formula (1), is the first importance value of the i-th element, is the recognition accuracy of the combination of the i-th element, is the recognition accuracy of the combination of the i-1-th element, is the recognition accuracy of the combination of the i+1-th element, i=1, 2, …, M, wherein, and a random number between 0 and 1 is taken.

[0090] Step a20: sort the first importance values in descending order to obtain a first importance sequence.

[0091] Step a30: take elements in the first feature set corresponding to the first importance values in the first importance sequence as first to-be-fused features.

[0092] Step a40: calculate the importance of each element in the second feature set to obtain a plurality of second importance values; in this embodiment, the calculation manner of the second importance values is the same as that of step a10.

[0093] Step a50: sort the second importance values in descending order to obtain a second importance sequence.

[0094] Step a60: take elements in the second feature set corresponding to the second importance values in the second importance sequence as second to-be-fused features.

[0095] Step a70: multiply fuse the first to-be-fused features and the second to-be-fused features to obtain fused features.

[0096] In this embodiment, the first feature set and the second feature set are extracted, and then the first feature set and the second feature set are fused to obtain the final speech features, which can exhibit the highest speech recognition accuracy and robustness in a complex noise environment; and the features are sorted and selected according to the size of the influence degree, which can better obtain meaningful features, thereby reducing the complexity of calculation and improving the speech recognition accuracy.

[0097] Step S20: identifying the speech features based on a pre-trained speech recognition model to obtain a speech recognition result; the speech recognition model in this embodiment includes a down-sampling layer, an encoder layer and a decoder layer connected in sequence; wherein the encoder layer is constructed based on an attention mechanism, which can enable the encoder layer to focus on more critical information for the current task among a plurality of input information, reduce the attention degree to other information, and even filter out irrelevant information, thereby improving the efficiency and accuracy of task processing.

[0098] Step S30: generating a language text to be translated based on the speech recognition result, and performing feature extraction on the language text to be translated to obtain a language word embedding representation; in this embodiment, the speech recognition result includes text, language type and other information corresponding to the speech signal; the language text to be translated can be directly extracted from the speech recognition result; this embodiment can use a bag-of-words model such as Bag-of-Words (BOW) and a Word2vec model to map the words of the language text to be translated to a vector representation in a high-dimensional space.

[0099] Step S40: based on the pre-constructed sample vocabulary representation, the language word embedding representation is expanded to obtain an expanded language word embedding representation.

[0100] In this embodiment, there are not enough training examples for some small language, which causes the model to fail to learn the low-resource vocabulary representation; therefore, it is necessary to expand the small language, and a sample vocabulary representation for expansion needs to be constructed before expansion; therefore, the method further comprises: constructing a sample vocabulary representation, comprising:

[0101] Step b10: obtaining general monolingual data, and preprocessing the general monolingual data to obtain general monolingual corpus; in this embodiment, the general monolingual data can be Chinese language or English language, which is a commonly used language at home and abroad; the general monolingual data can be directly obtained from existing corpus, or can be collected from the Internet through a crawler, and after the collected data on the Internet is preprocessed, such as data cleaning and standardization, a general monolingual corpus is obtained.

[0102] Step b20: translating the general monolingual corpus based on a preset translation tool to obtain a multilingual corpus; wherein the translation tool can be a Google translation tool, a Baidu translation tool, and a Yodao translation tool, etc., and the general monolingual corpus is translated into a multilingual corpus.

[0103] Step b30: based on a bag-of-words model, features are extracted from the general monolingual corpus and the multilingual corpus respectively to obtain general monolingual embedding representation and multilingual embedding representation; wherein the bag-of-words model can be a BOW model and a Word2vec model, etc.

[0104] Step b40: word alignment is performed on the general monolingual embedding representation and the multilingual embedding representation, and after alignment, the multilingual embedding representation is mapped into the general monolingual embedding representation to obtain a multilingual shared representation, and the multilingual shared representation is used as a sample vocabulary representation.

[0105] Therefore, through steps b10-b40, the multilingual shared representation can map general language knowledge to small language, and each word from any language is represented as a probability of general language word embedding, so that words with similar semantics in different languages will naturally have similar representations.

[0106] After obtaining the sample vocabulary representation, the language word embedding representation is expanded based on the pre-constructed sample vocabulary representation to obtain an expanded language word embedding representation, comprising:

[0107] Step S401: obtaining an initial language of the language text to be translated; the initial language can be directly extracted from the speech recognition result;

[0108] Step S402: According to the initial language of the language text to be translated and the language word embedding representation, the general monolingual embedding representation corresponding to the language is matched from the sample vocabulary representation, and the matched general monolingual embedding representation has similarity with the language word embedding representation.

[0109] Step S403: The language word embedding representation is spliced with the general monolingual embedding representation to obtain an expanded language word embedding representation; the expanded language word embedding representation can enrich the semantics of the language text to be translated, which is beneficial to improve the BLEU value (Bilingual Evaluation Understudy, referred to as BLEU, an index for evaluating the quality of machine translation, whose score value range is 0 to 1, and the closer the score is to 1, the higher the translation quality is) of small language translation, and improves the accuracy of small language translation.

[0110] Step S50: The expanded language word embedding representation is input into the pre-trained translation model, and the translation model outputs a multilingual translation text, wherein the translation model is a neural machine translation model based on Transformer_base; in this embodiment, the sample vocabulary representation is used as a training set, the neural network based on the attention mechanism is trained by using the training set, and the trained neural network is used as the pre-trained translation model.

[0111] As a further optimization of this embodiment, the method further comprises: obtaining a target language required by a user, and extracting a translation text in the target language from the multilingual translation text according to the target language required by the user; and sending the translation text in the target language to the user.

[0112] The multilingual translation text of this example contains translation texts in multiple languages, and the corresponding translation text in the target language can be output according to the selection of the user and returned to the user.

[0113] As a further optimization of this embodiment, the method further comprises:

[0114] Step c10: extracting features of the multilingual translation text to obtain a language embedding representation of the translation text;

[0115] Step c20: inputting the language embedding representation of the translation text into the translation model to obtain a reverse translation text in the initial language;

[0116] Step c30: calculating a loss value of the reverse translation text in the initial language and the language text to be translated;

[0117] Step c40: performing gradient optimization on the parameters of the translation model based on the loss value of the reverse translation text in the initial language and the language text to be translated.

[0118] In the embodiment, the translated multilingual translation text is subjected to one-time back translation, the back translation of the initial language is back translated, and then the loss value of the back translation of the initial language and the language text to be translated is calculated. According to the loss value, the translation model is optimized again, the translation performance of the translation model can be optimized, and the translation accuracy of the translation model is improved.

[0119] Embodiment two

[0120] Figure 2 is a block diagram of a multilingual translation system based on deep learning provided by an embodiment of the application, as Figure 2 shown, the embodiment provides a multilingual translation system based on deep learning, the system is used to realize the multilingual translation method based on deep learning in the embodiment one, the system comprises:

[0121] The first extraction module is used to collect the speech signal, extract and fuse the features of the speech signal, and obtain the speech features.

[0122] The feature recognition module is used to recognize the speech features based on the pre-trained speech recognition model, and obtain the speech recognition result.

[0123] The second extraction module is used to generate the language text to be translated based on the speech recognition result, extract the features of the language text to be translated, and obtain the language word embedding representation.

[0124] The feature expansion module is used to expand the language word embedding representation based on the pre-constructed sample vocabulary representation, and obtain the expanded language word embedding representation.

[0125] The text translation module is used to input the expanded language word embedding representation into the pre-trained translation model, and the translation model outputs the multilingual translation text.

[0126] The embodiment also provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor realizes the multilingual translation method based on deep learning in the embodiment one when executing the computer program.

[0127] The embodiment also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the multilingual translation method based on deep learning in the embodiment one.

[0128] In the real-time translation process, the speech of different speakers is recognized and automatically converted into corresponding language text, facilitating translation of the language text; meanwhile, feature extraction and fusion are performed on the speech signal, and the speech feature is recognized by using a speech recognition model, which can reduce the influence of background noise and improve the accuracy of speech recognition, and the speech recognition result with high accuracy is conducive to improving the accuracy of translation; and the language word embedding representation is expanded by using the pre-constructed sample vocabulary representation, which can enrich the semantics of the language text to be translated, is conducive to improving the BLEU value of small language translation, and improves the accuracy of small language translation.

[0129] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0130] The present application is described with reference to flowcharts and / or block diagrams according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0131] The above is only an embodiment of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A multilingual translation method based on deep learning, characterized in that, The method includes: Acquire speech signals, extract and fuse speech features to obtain speech features, including: The first feature set is obtained by extracting features from the speech signal based on the MFCC algorithm. Obtain the power spectrum of the MFCC algorithm during the feature extraction process of speech signals; Based on the power spectrum, the linear prediction signal of the speech signal is determined, and the prediction deviation signal is calculated based on the speech signal and the linear prediction signal. Feature extraction is performed on the prediction deviation signal to obtain the second feature set; The first feature set and the second feature set are fused in the same dimension to obtain the fused features, which are then used as speech features. The speech features are identified based on the pre-trained speech recognition model to obtain the speech recognition result; The language text to be translated is generated based on the speech recognition results. Feature extraction is performed on the language text to be translated to obtain the language word embedding representation. Based on the pre-constructed sample vocabulary representation, the language word embedding representation is extended to obtain the extended language word embedding representation, including: Obtain the initial language of the text to be translated; Based on the initial language and language word embedding representation of the text to be translated, the general monolingual embedding representation corresponding to that language is matched from the sample lexical representation; The extended language word embedding representation is obtained by concatenating the language word embedding representation with the general monolingual embedding representation. The expanded language word embeddings are input into a pre-trained translation model, which outputs multilingual translations.

2. The deep learning-based multilingual translation method according to claim 1, characterized in that, The method further includes: Obtain the target language required by the user, and extract the target language translation from the multilingual translation text based on the user's required target language; Send the translation in the target language to the user.

3. The deep learning-based multilingual translation method according to claim 1, characterized in that, The first feature set and the second feature set are fused in equal dimensions to obtain the fused features, including: Calculate the importance of each element in the first feature set to obtain several first importance values; Sort the first importance values ​​in descending order to obtain the first importance sequence; The elements in the first feature set corresponding to the first k first importance values ​​in the first importance sequence are taken as the first features to be fused, where k is a positive integer; Calculate the importance of each element in the second feature set to obtain several second importance values; The second importance values ​​are sorted in descending order to obtain the second importance sequence; The elements in the second feature set corresponding to the first k second importance values ​​in the second importance sequence are taken as the second features to be fused. Multiply and fuse the k first features to be fused with the k second features to be fused to obtain the fused features.

4. The deep learning-based multilingual translation method according to any one of claims 2-3, characterized in that, The speech recognition model comprises a downsampling layer, an encoder layer, and a decoder layer connected in sequence; wherein the encoder layer is constructed based on an attention mechanism.

5. The deep learning-based multilingual translation method according to claim 1, characterized in that, The method further includes: constructing sample vocabulary representations, including: Obtain general monolingual data, preprocess the general monolingual data, and obtain general monolingual corpus; Translate general monolingual corpora using pre-set translation tools to obtain multilingual corpora; Based on the bag-of-words model, features are extracted from general monolingual corpora and multilingual corpora to obtain general monolingual embedding representations and multilingual embedding representations; Word alignment is performed between the general monolingual embedding representation and the multilingual embedding representation. After alignment, the multilingual embedding representation is mapped back to the general monolingual embedding representation to obtain a multilingual shared representation. This multilingual shared representation is then used as the sample vocabulary representation.

6. The deep learning-based multilingual translation method according to claim 1, characterized in that, The method further includes: using the sample vocabulary representation as a training set, training the neural network constructed based on the attention mechanism using the training set to obtain a trained neural network, and using the trained neural network as a pre-trained translation model.

7. The deep learning-based multilingual translation method according to claim 6, characterized in that, The method further includes: Feature extraction of multilingual translation texts yields a linguistic embedding representation of the translation texts. The language embedding representation of the translation is input into the translation model to obtain the reverse translation in the initial language; Calculate the loss value between the reverse translation of the initial language and the language text to be translated; The parameters of the translation model are optimized using gradients based on the loss values ​​of the reverse translation in the initial language and the language text to be translated.

8. A deep learning-based multilingual translation system, said system being used to implement the deep learning-based multilingual translation method according to any one of claims 1-7, characterized in that, The system includes: The first extraction module is used to acquire speech signals, extract and fuse features from the speech signals to obtain speech features, including: extracting features from the speech signals based on the MFCC algorithm to obtain a first feature set; Obtain the power spectrum of the MFCC algorithm during the feature extraction process of speech signals; Based on the power spectrum, the linear prediction signal of the speech signal is determined, and the prediction deviation signal is calculated based on the speech signal and the linear prediction signal. Feature extraction is performed on the prediction deviation signal to obtain the second feature set; The first feature set and the second feature set are fused in the same dimension to obtain the fused features, which are then used as speech features. The feature recognition module is used to recognize speech features based on a pre-trained speech recognition model to obtain speech recognition results. The second extraction module is used to generate the language text to be translated based on the speech recognition results, extract features from the language text to be translated, and obtain language word embedding representations. The feature extension module is used to extend the language word embedding representation based on the pre-built sample vocabulary representation to obtain the extended language word embedding representation, including: Obtain the initial language of the text to be translated; Based on the initial language and language word embedding representation of the text to be translated, the general monolingual embedding representation corresponding to that language is matched from the sample lexical representation; The extended language word embedding representation is obtained by concatenating the language word embedding representation with the general monolingual embedding representation. The text translation module is used to input the expanded language word embedding representation into a pre-trained translation model, which outputs multilingual translations.

Images