Translation method and system based on multimodal interaction

Abstract

The present invention relates to the technical field of machine translation, and specifically to a translation method and system based on multimodal interaction. On the one hand, when a text modality exists, different operations are performed respectively according to whether an audio modality of the text modality exists; wherein, when an audio modality exists, the audio modality is translated into a text vector and then fused with the corresponding vector of the text modality; when the audio modality does not exist, the text modality is preliminarily translated and then spliced with the corresponding vector of the text modality, and temporal semantic analysis is performed; on the other hand, when the text modality does not exist, the audio modality and / or the image modality are translated into a text vector; finally, the audio-text fusion feature vector, or the text temporal feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fusion imitation text vector obtained by the above-mentioned feature extraction is processed by machine translation to obtain a more accurate translation result, thereby achieving the comprehensiveness, flexibility, adaptability and accuracy of machine translation.

Description

Technical Field

[0001] The present invention relates to the technical field of machine translation, and in particular to a translation method and system based on multimodal interaction. Background Art

[0002] Machine translation is the process of automatically converting one natural language into another using computer technology. It aims to break down language barriers and facilitate information exchange and communication across different languages. Machine translation has a wide range of applications across various fields. In international business, it can be used to translate documents such as contracts and business emails; in tourism, it helps tourists understand local signs, menus, and other information; and in academic research, it can assist researchers in quickly obtaining a general understanding of foreign literature. Furthermore, it plays a vital role in social media, online education, and other fields, facilitating communication and learning across language barriers.

[0003] Currently, most machine translation systems perform translation analysis solely based on text modality, including lexical, syntactic, and semantic analysis, to understand the structure and meaning of the text modality. Based on specific translation models and strategies, these systems then convert the analysis results into a target language representation. Finally, the generated target language text is adjusted and optimized to conform to the target language's grammar and expression conventions. However, when text modality is missing, existing machine translation systems, lacking a semantic foundation for the text, are unable to perform syntactic structure analysis and struggle to capture contextual associations. This makes it difficult to generate target language text and fails to meet actual translation needs. Summary of the Invention

[0004] The purpose of the present invention is to provide a translation method and system based on multimodal interaction, which can stably perform modal translation in the presence or absence of text modality and improve the accuracy of translation results.

[0005] The technical solutions of the present invention are as follows:

[0006] A translation method based on multimodal interaction includes the following operations:

[0007] S1. Determine whether there is a text mode in the data to be translated; if so, execute S2; if not, execute S3;

[0008] S2. Determine whether there is an audio modality; if so, perform noise reduction and normalization on the audio modality to obtain an audio normalized modality; translate the audio normalized modality into a text vector to obtain a first audio imitation text vector; fuse the first audio imitation text vector with the corresponding vector of the text modality to obtain an audio-text fusion vector, and execute S4; if not, divide the text modality into several text sub-modalities, translate them through a neural network and transform them into vectors in sequence, and then splice them to obtain an initial translation vector; splice the initial translation vector and the corresponding vector of the text modality to obtain a text time series vector; perform time series semantic analysis on the text time series vector to obtain a text time series feature vector, and execute S4;

[0009] S3. If only audio modality exists, the audio modality is processed by speech feature extraction to obtain speech signal features; acoustic features are obtained based on the speech signal features, and decoding is performed to obtain an acoustic hidden state sequence; the acoustic hidden state sequence is converted into a phoneme sequence, and the phoneme sequence is converted into a text vector according to the pronunciation rules and vocabulary of the language to obtain a second audio imitation text vector, and S4 is executed; if only image modality exists, the image modality is subjected to image enhancement processing to obtain an image enhancement modality; the image enhancement modality is subjected to text feature extraction to obtain an image imitation text vector, and S4 is executed; if both audio modality and image modality exist, the context features of the audio modality and image modality are extracted respectively to obtain audio features and image features; the feature corresponding to the maximum information entropy in the audio features and image features is used as the imitated feature, and after translating another feature into the imitated feature, it is fused with the imitated feature to obtain an audio-image fusion feature; the audio-image fusion feature is translated into a text vector to obtain a fused imitation text vector, and S4 is executed;

[0010] S4. The audio-text fusion feature vector, or the text time sequence feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fusion imitation text vector is processed by machine translation to obtain a translation result.

[0011] The specific operation of obtaining the first audio imitation text mode in S2 is as follows: the audio normalized mode is frame-processed, and linear prediction is performed on each frame of the audio normalized mode to obtain several linear prediction sub-vectors; the Euclidean distance of each linear prediction sub-vector with each subspace in the vector space formed by multiple non-intersecting subspaces is obtained, and each linear prediction sub-vector is mapped to the text representative vector in the subspace corresponding to the minimum Euclidean distance value, to obtain the respective audio imitation text sub-vectors, thereby forming the first audio imitation text vector.

[0012] The specific operation of obtaining the audio-text fusion feature vector in S2 is as follows: the first audio imitation text vector and the text modality corresponding vector are processed by a multi-layer perceptron respectively to obtain an audio imitation text vector sequence and a text vector sequence; the attention weight matrix between the audio imitation text vector sequence and the text vector sequence is obtained, and based on the attention weight matrix, the audio imitation text vector sequence and the text vector sequence are weightedly summed to obtain the audio-text fusion feature vector.

[0013] The specific operations of speech feature extraction and processing in S3 are as follows: performing high-frequency enhancement processing on the audio mode to obtain an audio high-frequency enhanced mode; dividing the audio high-frequency enhanced mode into several short frames to form a sequence to obtain a discrete audio frame sequence; performing frame feature enhancement processing on the discrete audio frame sequence to obtain an audio frame enhanced sequence; obtaining frequency domain information of the audio frame enhanced sequence and analyzing it on the Mel frequency scale to obtain a Mel frequency feature vector; compressing the energy dynamic range of the Mel frequency feature vector to the standard energy dynamic range and then converting it to the cepstrum domain to obtain the speech signal features.

[0014] The operation of character feature processing in S3 is achieved by splicing the visual feature vector obtained based on the image enhancement mode, or / and the text word vector feature vector, or / and the text semantic feature vector, and then performing a nonlinear transformation.

[0015] In S2, if there is an image modality, the visual feature vector, or / and the text word vector feature vector, or / and the text semantic feature vector of the image modality are obtained; the audio-text fusion feature vector or the text timing feature vector is respectively spliced with the visual feature vector, or / and the text word vector feature vector, or / and the text semantic feature vector to obtain an optimized audio-text fusion feature vector or an optimized text timing feature vector for executing the operation in S4.

[0016] In S3, when image features are the features to be imitated, the operations for translating image features into audio features are as follows: multiplying the image features with the key matrix and the value matrix respectively to obtain the image key vector and the image value vector; multiplying the audio features with the query matrix to obtain the audio query vector; after the audio query vector and the image key vector are subjected to dot product operation, scaling processing and linear processing, the dot product operation, residual connection processing, normalization and nonlinear processing are performed with the image value vector to obtain the image-to-audio feature for fusion with the audio feature.

[0017] A translation system based on multimodal interaction, used to implement the above-mentioned translation method based on multimodal interaction, comprising:

[0018] The text modality existence judgment module is used to determine whether the text modality exists in the translation material; if so, the audio modality existence judgment and modality feature extraction module is executed; if not, the audio and image modality feature extraction modules are executed;

[0019] The audio modality existence judgment and modality feature extraction module is used to judge whether the audio modality exists; if it exists, the audio modality is subjected to noise reduction and normalization processing to obtain an audio normalized modality; the audio normalized modality is translated into a text vector to obtain a first audio imitation text vector; the first audio imitation text vector is fused with the corresponding vector of the text modality to obtain an audio-text fusion vector, and the translation result generation module is executed; if it does not exist, the text modality is divided into several text sub-modalities, which are sequentially translated by a neural network and converted into vectors, and then spliced to obtain an initial translation vector; the initial translation vector and the corresponding vector of the text modality are spliced to obtain a text time series vector; the text time series vector is subjected to time series semantic analysis to obtain a text time series feature vector, and the translation result generation module is executed;

[0020] Audio and image modality feature extraction module: if only audio modality exists, the audio modality is processed by speech feature extraction to obtain speech signal features; acoustic features are obtained based on the speech signal features, and decoding is performed to obtain an acoustic hidden state sequence; the acoustic hidden state sequence is converted into a phoneme sequence, and the phoneme sequence is converted into a text vector according to the pronunciation rules and vocabulary of the language to obtain a second audio imitation text vector, and the translation result generation module is executed; if only image modality exists, the image modality is subjected to image enhancement processing to obtain an image enhancement modality; the image enhancement modality is subjected to text feature extraction to obtain an image imitation text vector, and the translation result generation module is executed; if both audio modality and image modality exist, the context features of the audio modality and image modality are extracted respectively to obtain audio features and image features; the feature corresponding to the maximum information entropy in the audio features and image features is used as the imitated feature, and after translating the other feature into the imitated feature, it is fused with the imitated feature to obtain an audio-image fusion feature; the audio-image fusion feature is translated into a text vector to obtain a fused imitation text vector, and the translation result generation module is executed;

[0021] The translation result generation module obtains a translation result by machine translation processing of the audio-text fusion feature vector, or the text time sequence feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fusion imitation text vector.

[0022] A translation device based on multimodal interaction includes a processor and a memory, wherein the processor implements the above-mentioned translation method based on multimodal interaction when executing a computer program stored in the memory.

[0023] A computer-readable storage medium is used to store a computer program, wherein the computer program implements the above-mentioned translation method based on multimodal interaction when executed by a processor.

[0024] The beneficial effects of the present invention are:

[0025] The present invention provides a translation method based on multimodal interaction, which respectively executes different methods for extracting features to be translated according to whether there is a text modality in the material to be translated; on the one hand, when there is a text modality, different operations are respectively performed according to whether there is an audio modality whose impact on translation accuracy is second only to the text modality; wherein, when there is an audio modality, the audio modality is translated into a text vector and then fused with the corresponding vector of the text modality to realize the interaction between the audio modality information and the text modality information, thereby improving the feature expression ability of the features to be translated and obtaining an audio-text fusion vector; when there is no audio modality, the initial translation vector obtained after the preliminary translation of the text modality, which contains a preliminary understanding and conversion of the source language text, is fused with the corresponding vector of the text modality containing the feature information of the source language text After splicing, the temporal semantic analysis is performed to further improve the feature expression ability of the text modality; on the other hand, when there is no text modality, the audio modality and / or image modality is translated into a text vector with stronger feature expression ability and higher translation efficiency, so as to improve the expression ability of the audio modality and / or image modality in vocabulary, grammatical structure and semantics, and obtain a second audio imitation text vector, or a second audio imitation text vector, or a fused imitation text vector; finally, the audio-text fusion feature vector, or the text temporal feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fused imitation text vector with stronger vocabulary, grammatical structure and semantic expression ability is processed by machine translation to obtain a more accurate translation result, thereby realizing the comprehensiveness, flexibility, adaptability and accuracy of machine translation. DETAILED DESCRIPTION

[0026] This embodiment provides a translation method based on multimodal interaction, including the following operations:

[0027] S1. Determine whether there is a text mode in the data to be translated; if so, execute S2; if not, execute S3;

[0028] S2. Determine whether there is an audio modality; if so, perform noise reduction and normalization on the audio modality to obtain an audio normalized modality; translate the audio normalized modality into a text vector to obtain a first audio imitation text vector; fuse the first audio imitation text vector with the corresponding vector of the text modality to obtain an audio-text fusion vector, and execute S4; if not, divide the text modality into several text sub-modalities, translate them through a neural network and transform them into vectors in sequence, and then splice them to obtain an initial translation vector; splice the initial translation vector and the corresponding vector of the text modality to obtain a text time series vector; perform time series semantic analysis on the text time series vector to obtain a text time series feature vector, and execute S4;

[0029] S3. If only audio modality exists, the audio modality is processed by speech feature extraction to obtain speech signal features; acoustic features are obtained based on the speech signal features, and decoding is performed to obtain an acoustic hidden state sequence; the acoustic hidden state sequence is converted into a phoneme sequence, and the phoneme sequence is converted into a text vector according to the pronunciation rules and vocabulary of the language to obtain a second audio imitation text vector, and S4 is executed; if only image modality exists, the image modality is subjected to image enhancement processing to obtain an image enhancement modality; the image enhancement modality is subjected to text feature extraction to obtain an image imitation text vector, and S4 is executed; if both audio modality and image modality exist, the context features of the audio modality and image modality are extracted respectively to obtain audio features and image features; the feature corresponding to the maximum information entropy in the audio features and image features is used as the imitated feature, and after translating another feature into the imitated feature, it is fused with the imitated feature to obtain an audio-image fusion feature; the audio-image fusion feature is translated into a text vector to obtain a fused imitation text vector, and S4 is executed;

[0030] S4. The audio-text fusion feature vector, or the text time sequence feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fusion imitation text vector is processed by machine translation to obtain a translation result.

[0031] S1. Determine whether there is a text mode in the data to be translated; if so, execute S2; if not, execute S3.

[0032] Compared with audio and image modalities, text modalities have clearer vocabulary and grammatical structures, clearer semantics, and a greater impact on the accuracy and efficiency of translation results. Therefore, this embodiment executes different translation methods based on whether text modalities exist in the material to be translated, thereby improving the accuracy, flexibility, and comprehensiveness of the translation method.

[0033] S2. Determine whether there is an audio modality; if so, perform noise reduction and normalization on the audio modality to obtain an audio normalized modality; translate the audio normalized modality into a text vector to obtain a first audio imitation text vector; fuse the first audio imitation text vector with the corresponding vector of the text modality to obtain an audio-text fusion vector, and execute S4; if not, divide the text modality into several text sub-modalities, translate them through a neural network and transform them into vectors in turn, and then splice them to obtain an initial translation vector; splice the initial translation vector and the corresponding vector of the text modality to obtain a text time series vector; perform time series semantic analysis on the text time series vector to obtain a text time series feature vector, and execute S4.

[0034] When there is a text modality, different operations are performed according to whether there is an audio modality whose impact on translation accuracy is second only to the text modality; when there is an audio modality, the audio modality is translated into a text vector and then fused with the corresponding vector of the text modality, thereby improving the feature expression ability of the features to be translated and obtaining an audio-text fusion vector; when there is no audio modality, the text modality is divided into several sub-modalities and then translated in sequence through a neural network, so that the latter sub-modality will be translated according to the translation result of the previous sub-modality during translation, thereby improving the accuracy of the initial translation vector, and then the initial translation vector containing a preliminary understanding and conversion of the source language text is spliced with the corresponding vector of the text modality containing the feature information of the source language text and then subjected to temporal semantic analysis to obtain a text temporal feature vector, thereby further improving the feature expression ability of the text modality and improving the accuracy of subsequent translation.

[0035] Compared with the image modality, the audio modality acquires language information more directly and is richer in semantic and emotional information, and has a greater impact on the accuracy and efficiency of translation results. Therefore, when the text modality exists in the material to be translated, in order to further improve the accuracy of the translation results, it is necessary to further determine whether the audio modality exists.

[0036] If an audio modality exists, in order to improve the vocabulary, grammatical structure, and semantic expression capabilities of the audio modality while improving translation efficiency, the audio modality is translated into a text vector and fused with the corresponding vector of the text modality (the vector obtained after the text modality is embedded) to obtain an audio-text fusion vector with stronger feature expression capabilities. The specific steps are as follows.

[0037] Step 1: De-noise the audio mode (which can be achieved through the Wiener filter method) and normalize it to make the energy of different audio signals comparable, facilitate subsequent feature analysis, and obtain the audio normalized mode.

[0038] The normalization operation can be achieved through the following formula:

[0039]

[0040] y(n) is the normalized audio mode, x(n) is the noise-reduced audio mode of length n (obtained after the audio mode is denoised), and N is the total length of the noise-reduced audio mode.

[0041] Step 2: Translate the normalized audio modality into a text vector to obtain a first audio pseudo-text vector. Specifically, the normalized audio modality is framed and linearly predicted for each frame of the normalized audio modality to obtain a number of linear prediction sub-vectors; the Euclidean distance between each linear prediction sub-vector and each sub-space in a vector space formed by multiple non-intersecting sub-spaces is obtained (each sub-space is represented by a text representative vector), and each linear prediction sub-vector is mapped to the text representative vector in the sub-space corresponding to the minimum Euclidean distance. In this way, each linear prediction sub-vector is encoded as a corresponding text codeword index, obtaining its own audio pseudo-text sub-vector. The vector composed of these text codeword indices can be regarded as a text-like representation, forming the first audio pseudo-text vector.

[0042] The linear prediction operation can be achieved through the following formula:

[0043]

[0044] S i is the i-th linear predictor vector, α k is the kth order linear prediction coefficient, K is the total order of linear prediction, s i The normalized mode of the audio of the i-th frame.

[0045] Step 3: The first audio-imitation text vector and the text modality corresponding vector are fused to obtain an audio-text fusion vector, and then S4 is executed.

[0046] The specific process is as follows: the first audio-simulated text vector and the corresponding text modality vector are processed by a multi-layer perceptron to obtain a first audio-simulated text vector sequence and a text vector sequence; an attention weight matrix is obtained between the first audio-simulated text vector sequence and the text vector sequence, and based on the attention weight matrix, the first audio-simulated text vector sequence and the text vector sequence are weighted and summed to obtain an audio-text fusion feature vector. In the above attention weight matrix, the attention weight is obtained based on the cosine similarity between the u-th subsequence of the first audio-simulated text vector sequence and the v-th first subsequence in the text vector sequence, that is, the attention weight is obtained based on the cosine similarity of the two subsequences.

[0047] Furthermore, if there is both text modality, audio modality, and image modality, in order to further improve the information richness of the features to be translated, the visual feature vector of the image modality, or / and the text word vector feature vector, or / and the text semantic feature vector are obtained; the audio-text fusion feature vector is spliced with the visual feature vector, or / and the text word vector feature vector, or / and the text semantic feature vector to achieve interactive fusion of text modality information, audio modality information, and image modality information, and obtain an optimized audio-text fusion feature vector for executing the operation in S4.

[0048] The visual feature vector of the above image modality is achieved by processing the image modality with a convolutional neural network.

[0049] The specific operations for obtaining the text word vector feature vector of the above-mentioned image modality are as follows: grayscale processing is performed on the image modality to simplify the color information of the image and highlight the brightness characteristics of the image to obtain a grayscale image; the grayscale image is subjected to median filtering to remove the salt and pepper noise of the image and Gaussian filtering to remove the Gaussian noise of the image, so that the image can maintain good quality after noise removal, providing more reliable image data for subsequent text recognition, and obtaining a denoised image; the denoised image is subjected to binarization processing so that the text part in the image is represented by black pixels and the background part can be represented by white pixels, thereby clearly separating the text from the background to obtain a binary image; the text information in the binary image is obtained, and after embedding processing, a text word vector feature vector is obtained.

[0050] The operation of obtaining the text semantic feature vector of the above-mentioned image modality can be achieved by performing a neural network-based mapping process on the image modality. The specific operation of the neural network-based mapping process is as follows: the image modality is embedded to obtain an image vector; the image vector is multiplied by the weight matrix in the hidden layer, and the bias vector of the hidden layer is added, and then a nonlinear transformation is performed through an activation function to obtain the image hidden layer output features; the image hidden layer output features are multiplied by the weight matrix of the output layer and the bias vector of the output layer is added, thereby mapping the image modality features into a simulated text vector with text semantic features to obtain a text semantic feature vector.

[0051] If there is no audio modality and only text modality exists, the following operations are performed to further enhance the feature expression capability of the text modality.

[0052] Step 1: Split the text modality into several sub-modalities. These sub-modalities are sequentially translated by a neural network (so that each subsequent sub-modality is translated based on the previous sub-modality, improving the accuracy of the initial translation) and converted to vectors, and then concatenated to obtain an initial translation vector. This neural network translation includes, but is not limited to, implementation via a recurrent neural network language model (RNN-LM).

[0053] Step 2: Because some words and expressions in the source text modality often have multiple meanings, their specific meanings may not be accurately determined based on the initial translation results alone. Therefore, in this embodiment, the initial translation vector that contains a preliminary understanding and conversion of the source language text is spliced with the text modality corresponding vector (i.e., the text vector) containing the feature information of the source language text to obtain a text time series vector that contains both historical translation information and richer semantic information, which can improve the accuracy of subsequent translation.

[0054] Step 3: The text time series vector is subjected to time series semantic analysis to further extract text features to obtain a text time series feature vector, and then execute S4.

[0055] The operations of temporal semantic analysis can be: the text temporal vector is multiplied with the query item parameter matrix, key item parameter matrix and value item parameter matrix respectively to obtain the text temporal query vector, text temporal key vector and text temporal value vector; the text temporal query vector, text temporal key vector and text temporal value vector are multiplied with different parameter weights respectively to obtain several text temporal query sub-vectors, several text temporal key sub-vectors and several text temporal value sub-vectors; the attention scores of each text temporal query sub-vector and each text temporal key sub-vector are obtained, and the scores are added to the corresponding mask matrix and processed by probability mapping to obtain several mask scores; the several mask scores are weighted summed with each text temporal value sub-vector, and then the text temporal feature vector is obtained through splicing, linear processing and nonlinear processing. The above attention score is obtained based on the product of the text temporal query sub-vector and the text temporal key sub-vector.

[0056] The operation of temporal semantic analysis can also be implemented through the Long Short-Term Memory (LSTM) network.

[0057] Furthermore, if there is text modality and image modality, the visual feature vector of the image modality, or / and the text word vector feature vector, or / and the text semantic feature vector are obtained; the text timing feature vector is spliced with the visual feature vector, or / and the text word vector feature vector, or / and the text semantic feature vector to realize the interaction between text modality information and image modality information, and obtain the optimized text timing feature vector for executing the operation in S4.

[0058] The methods for obtaining the visual feature vector of the image modality, the text word vector feature vector, and the text semantic feature vector have been described above and will not be repeated here to save space.

[0059] S3. If only audio modality exists, the audio modality is processed by speech feature extraction to obtain speech signal features; acoustic features are obtained based on the speech signal features, and decoding is performed to obtain an acoustic hidden state sequence; the acoustic hidden state sequence is converted into a phoneme sequence, and the phoneme sequence is converted into a text vector according to the pronunciation rules and vocabulary of the language to obtain a second audio imitation text vector, and execute S4; if only image modality exists, the image modality is subjected to image enhancement processing to obtain an image enhancement modality; the image enhancement modality is subjected to text feature extraction to obtain an image imitation text vector, and execute S4; if both audio modality and image modality exist, the context features of the audio modality and image modality are extracted respectively to obtain audio features and image features; the feature corresponding to the maximum information entropy in the audio features and image features is used as the imitated feature, and after translating the other feature into the imitated feature, it is fused with the imitated feature to obtain an audio-image fusion feature; the audio-image fusion feature is translated into a text vector to obtain a fused imitation text vector, and execute S4.

[0060] When there is no text modality, in order to improve the vocabulary, grammatical structure and semantic expression ability of the existing modality (audio modality and / or image modality), the audio modality and / or image modality are translated into a text vector with stronger feature expression ability and higher translation efficiency to obtain a second audio imitation text vector, or a second audio imitation text vector, or a fused imitation text vector, which is used to improve the accuracy of subsequent translation results.

[0061] When only the audio modality exists, rather than the text modality, to increase the information richness of the features to be translated, we extract detailed speech signal features from the audio modality, decode them into an acoustic hidden state sequence, and convert them into a phoneme sequence. Based on the pronunciation rules and vocabulary of the language, we convert the phoneme sequence into a text vector that is rich in details and easy to translate. The specific steps are as follows.

[0062] Step 1: The audio modality is processed by speech feature extraction to obtain speech signal features.

[0063] The specific operation of speech feature extraction processing is as follows: during the transmission process of the speech signal in the audio mode, the high-frequency part will be attenuated to a certain extent. Therefore, the audio mode is first subjected to high-frequency enhancement processing (including but not limited to being implemented by a first-order FIR filter) to enhance the high-frequency components of the speech signal and obtain an audio high-frequency enhanced mode, which is conducive to subsequent feature extraction and analysis; then, in order to facilitate digital signal processing, the audio high-frequency enhanced mode is divided into several short frames to form a sequence to obtain a discrete audio frame sequence; subsequently, in order to prevent and reduce the spectrum leakage of the discrete audio frame sequence, the discrete audio frame sequence is subjected to frame feature enhancement processing (including but not limited to being implemented by applying a preset window function to each frame sequence) so that each frame sequence signal is gradually enhanced at both ends. Gradually attenuate, effectively reduce spectral leakage, and obtain an audio frame enhancement sequence; then, obtain frequency domain information of the audio frame enhancement sequence (including but not limited to through fast Fourier transform), and analyze it on the Mel frequency scale to obtain speech signal characteristics of different frequencies, thereby improving the robustness and distinguishability of acoustic feature analysis, and obtaining a Mel frequency feature vector; compress the energy dynamic range of the Mel frequency feature vector to the standard energy dynamic range (which can be achieved by performing a logarithmic operation on the Mel frequency feature vector) and then convert it to the cepstrum domain (including but not limited to through a discrete cosine transform method) to achieve the extraction of speech signal details on the basis of highlighting the energy changes of the low-frequency part and reducing the noise influence of the high-frequency part, and obtain speech signal characteristics.

[0064] Step 2: Acoustic features are obtained based on the speech signal features, and decoding is performed to obtain an acoustic hidden state sequence. The above-mentioned operation of obtaining acoustic features based on the speech signal features can be achieved by training a deep neural network to process the speech signal features, and the decoding operation includes but is not limited to being achieved through a hidden Markov model.

[0065] Step 3: Convert the acoustic hidden state sequence into a phoneme sequence. Based on the pronunciation rules and vocabulary of the language, convert the phoneme sequence into a text vector to obtain a second audio-imitation text vector, and execute S4. The above-mentioned operation of converting the acoustic hidden state sequence into a phoneme sequence includes, but is not limited to, implementing it through an N-Gram model, a recurrent neural network language model, or a long short-term memory network language model.

[0066] If only the image modality exists, the image modality is subjected to image enhancement processing (including but not limited to being achieved through an adaptive equalization method) to obtain an image enhancement modality; the image enhancement modality is subjected to text feature extraction to obtain an image imitation text vector, and S4 is executed. The text feature processing operation is achieved by splicing the visual feature vector, or / and the text word vector feature vector, or / and the text semantic feature vector obtained based on the image enhancement modality and performing a nonlinear transformation. The method for obtaining the visual feature vector, the text word vector feature vector, and the text semantic feature vector has been described above and will not be repeated here to save space.

[0067] If both audio and image modalities exist, the interaction between the audio and image modal information is translated into a text vector to improve the expressiveness of the features to be translated and the translation efficiency. The operation is as follows.

[0068] Step 1: Extract context features of the audio modality and image modality respectively to obtain audio features and image features.

[0069] The specific operations for obtaining audio features are as follows: the audio modality is processed by multi-head attention to obtain audio attention features; the audio attention features are processed by residual connection to obtain audio residual features; the audio residual features and the corresponding audio attention weights are processed by layer normalization and nonlinearity to obtain audio features.

[0070] The operation of obtaining image features can be the same as the operation of obtaining audio features described above. Alternatively, the image modality can be subjected to spatial pyramid pooling to obtain contextual information at different levels of the image to obtain image features.

[0071] Step 2: The feature with the maximum information entropy among the audio features and image features is used as the simulated feature. After translating the other feature into the simulated feature, it is fused with the simulated feature to obtain the audio-image fusion feature.

[0072] Information entropy can be obtained by the following formula:

[0073]

[0074] H is information entropy. The larger the information entropy, the richer the corresponding modal information. i ) is the probability of occurrence of the i-th information feature in the audio feature or image feature, and I is the total number of information feature types in the audio feature or image feature.

[0075] In the process of translating another modality into the imitated modality, when the audio feature is the imitated feature,

[0076] The specific operations for translating image features into audio features are as follows: multiplying the image features with the key matrix and value matrix, respectively, to obtain the image key vector and image value vector; multiplying the audio features with the query matrix to obtain the audio query vector; and performing dot product operations on the audio query vector and the image key vector, scaling, and linear processing. The dot product operation, residual connection processing, normalization, and nonlinear processing are then performed with the image value vector to obtain image-to-audio features for fusion with the audio features. The operations for translating audio features into image features are the same as those for translating image features into audio features.

[0077] Step 3: Translate the audio image fusion features into a text vector (which can be achieved by training a Transformer pre-trained model or other training neural network), obtain a fused imitation text vector, and execute S4.

[0078] S4. The audio-text fusion feature vector, or the text time sequence feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fusion imitation text vector is processed by machine translation to obtain a translation result.

[0079] The audio-text fusion feature vector, or the text time series feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fusion imitation text vector with stronger vocabulary, grammatical structure and semantic expression capabilities will be processed by machine translation (which can be achieved through the N-Gram model, or the recurrent neural network language model, or the long short-term memory network language model, or the training of the Transformer pre-training model) to obtain more accurate translation results.

[0080] This embodiment further provides a translation system based on multimodal interaction, which is used to implement the above-mentioned translation method based on multimodal interaction, including:

[0081] The text modality existence judgment module is used to determine whether the text modality exists in the translation material; if so, the audio modality existence judgment and modality feature extraction module is executed; if not, the audio and image modality feature extraction modules are executed;

[0082] The audio modality existence judgment and modality feature extraction module is used to judge whether the audio modality exists; if it exists, the audio modality is subjected to noise reduction and normalization processing to obtain an audio normalized modality; the audio normalized modality is translated into a text vector to obtain a first audio imitation text vector; the first audio imitation text vector is fused with the corresponding vector of the text modality to obtain an audio-text fusion vector, and the translation result generation module is executed; if it does not exist, the text modality is divided into several text sub-modalities, which are sequentially translated by a neural network and converted into vectors, and then spliced to obtain an initial translation vector; the initial translation vector and the corresponding vector of the text modality are spliced to obtain a text time series vector; the text time series vector is subjected to time series semantic analysis to obtain a text time series feature vector, and the translation result generation module is executed;

[0083] Audio and image modality feature extraction module: if only audio modality exists, the audio modality is processed by speech feature extraction to obtain speech signal features; acoustic features are obtained based on the speech signal features, and decoding is performed to obtain an acoustic hidden state sequence; the acoustic hidden state sequence is converted into a phoneme sequence, and the phoneme sequence is converted into a text vector according to the pronunciation rules and vocabulary of the language to obtain a second audio imitation text vector, and the translation result generation module is executed; if only image modality exists, the image modality is subjected to image enhancement processing to obtain an image enhancement modality; the image enhancement modality is subjected to text feature extraction to obtain an image imitation text vector, and the translation result generation module is executed; if both audio modality and image modality exist, the context features of the audio modality and image modality are extracted respectively to obtain audio features and image features; the feature corresponding to the maximum information entropy in the audio features and image features is used as the imitated feature, and after translating the other feature into the imitated feature, it is fused with the imitated feature to obtain an audio-image fusion feature; the audio-image fusion feature is translated into a text vector to obtain a fused imitation text vector, and the translation result generation module is executed;

[0084] The translation result generation module obtains a translation result by machine translation processing of the audio-text fusion feature vector, or the text time sequence feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fusion imitation text vector.

[0085] This embodiment further provides a translation device based on multimodal interaction, including a processor and a memory, wherein the processor implements the above-mentioned translation method based on multimodal interaction when executing a computer program stored in the memory.

[0086] This embodiment further provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the above-mentioned translation method based on multimodal interaction when executed by a processor.

[0087] The present embodiment provides a translation method based on multimodal interaction, which executes different methods for extracting features to be translated according to whether there is a text modality in the material to be translated. On the one hand, when there is a text modality, different operations are performed according to whether there is an audio modality whose impact on translation accuracy is second only to the text modality. Among them, when there is an audio modality, the audio modality is translated into a text vector and then fused with the corresponding vector of the text modality to realize the interaction between the audio modality information and the text modality information, thereby improving the feature expression ability of the features to be translated and obtaining an audio-text fusion vector. When there is no audio modality, the initial translation vector obtained after the preliminary translation of the text modality, which contains a preliminary understanding and conversion of the source language text, is fused with the corresponding vector of the text modality containing the feature information of the source language text. After splicing, the temporal semantic analysis is performed to further improve the feature expression ability of the text modality; on the other hand, when there is no text modality, the audio modality and / or image modality is translated into a text vector with stronger feature expression ability and higher translation efficiency. In order to improve the expression ability of the audio modality and / or image modality in vocabulary, grammatical structure and semantics, a second audio imitation text vector, or a second audio imitation text vector, or a fused imitation text vector is obtained; finally, the audio-text fusion feature vector, or the text temporal feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fused imitation text vector with stronger vocabulary, grammatical structure and semantic expression ability is processed by machine translation to obtain a more accurate translation result, thereby realizing the comprehensiveness, flexibility, adaptability and accuracy of machine translation.

Claims

1. A translation method based on multimodal interaction, characterized in that: The following operations are included: S1. Determine whether there is a text mode in the data to be translated; if so, execute S2; if not, execute S3; S2, determining whether there is an audio mode; If it exists, the audio mode is subjected to noise reduction and normalization processing to obtain an audio normalized mode; Translate the audio normalized modality into a text vector to obtain a first audio imitation text vector; The first audio imitation text vector and the text modality corresponding vector are fused to obtain an audio text fusion vector, and S4 is executed; If it does not exist, the text modality is divided into several text sub-modalities, which are sequentially translated by neural network and transformed into vectors, and then spliced to obtain an initial translation vector; the initial translation vector and the corresponding vector of the text modality are spliced to obtain a text time sequence vector; the text time sequence vector is subjected to time sequence semantic analysis to obtain a text time sequence feature vector, and S4 is executed; S3. If only the audio mode exists, the audio mode is subjected to speech feature extraction processing to obtain speech signal features; acoustic features are obtained based on the speech signal features, and decoding processing is performed to obtain an acoustic hidden state sequence; the acoustic hidden state sequence is converted into a phoneme sequence, and the phoneme sequence is converted into a text vector based on the pronunciation rules and vocabulary of the language to obtain a second audio imitation text vector, and then S4 is executed; If only the image modality exists, the image modality is subjected to image enhancement processing to obtain an image enhancement modality; the image enhancement modality is subjected to text feature extraction to obtain an image imitation text vector, and S4 is executed; If both audio and image modalities exist, context features of the audio and image modalities are extracted to obtain audio features and image features respectively; the feature corresponding to the maximum information entropy among the audio and image features is used as the imitated feature, and the other feature is translated into the imitated feature and then fused with the imitated feature to obtain an audio-image fusion feature; the audio-image fusion feature is translated into a text vector to obtain a fused imitation text vector, and S4 is executed; S4. The audio-text fusion feature vector, or the text time sequence feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fusion imitation text vector is processed by machine translation to obtain a translation result.

2. The translation method based on multimodal interaction according to claim 1, characterized in that: In S2, the operation of obtaining the first audio imitation text mode is specifically as follows: The audio normalized mode is processed by dividing the frames, and linear prediction is performed on each frame of the audio normalized mode to obtain several linear prediction sub-vectors; Obtain the Euclidean distance of each linear prediction sub-vector with each subspace in the vector space formed by multiple non-intersecting subspaces, map each linear prediction sub-vector to the text representative vector in the subspace corresponding to the minimum Euclidean distance, obtain the respective audio imitation text sub-vector, and form a first audio imitation text vector.

3. The translation method based on multimodal interaction according to claim 1, characterized in that: In S2, the operation of obtaining the audio-text fusion feature vector is specifically as follows: The first audio-imitation-text vector and the text modality corresponding vector are processed by a multi-layer perceptron respectively to obtain an audio-imitation-text vector sequence and a text vector sequence; an attention weight matrix between the audio-imitation-text vector sequence and the text vector sequence is obtained, and based on the attention weight matrix, the audio-imitation-text vector sequence and the text vector sequence are weightedly summed to obtain an audio-text fusion feature vector.

4. The translation method based on multimodal interaction according to claim 1, characterized in that: In S3, the speech feature extraction process is specifically performed as follows: Performing high-frequency enhancement processing on the audio mode to obtain an audio high-frequency enhancement mode; dividing the audio high-frequency enhancement mode into a plurality of short frames to form a sequence to obtain a discrete audio frame sequence; The discrete audio frame sequence is processed by frame feature enhancement to obtain an audio frame enhancement sequence; the frequency domain information of the audio frame enhancement sequence is obtained and analyzed on the Mel-frequency scale to obtain a Mel-frequency feature vector; the energy dynamic range of the Mel-frequency feature vector is compressed to the standard energy dynamic range and then converted to the cepstrum domain to obtain the speech signal characteristics.

5. The translation method based on multimodal interaction according to claim 1, characterized in that: In S3, the text feature processing operation is achieved by splicing the visual feature vector obtained based on the image enhancement mode, or / and the text word vector feature vector, or / and the text semantic feature vector, and then performing nonlinear transformation.

6. The translation method based on multimodal interaction according to claim 1, characterized in that: In S2, if there is an image modality, a visual feature vector of the image modality, or / and a text word vector feature vector, or / and a text semantic feature vector are obtained; The audio-text fusion feature vector or the text timing feature vector is concatenated with the visual feature vector, or / and the text word vector feature vector, or / and the text semantic feature vector to obtain an optimized audio-text fusion feature vector or an optimized text timing feature vector for executing the operation in S4.

7. The translation method based on multimodal interaction according to claim 1, characterized in that: In S3, when the image feature is the feature to be imitated, the operation of translating the image feature into the audio feature is as follows: The image features are multiplied with the key matrix and the value matrix respectively to obtain the image key vector and the image value vector; the audio features are multiplied with the query matrix to obtain the audio query vector; the audio query vector and the image key vector are dot-product processed, scaled, and linearly processed, and then dot-product processed, residual connected, normalized, and nonlinearly processed with the image value vector to obtain the image-to-audio features for fusion with the audio features.

8. A translation system based on multimodal interaction, used to implement the translation method based on multimodal interaction according to claim 1, characterized in that: include: A text modality existence judgment module is used to judge whether there is a text modality in the material to be translated; If it exists, execute the audio mode existence judgment and modal feature extraction module; If it does not exist, execute the audio and image modality feature extraction module; Audio mode existence judgment and modal feature extraction module, used to determine whether the audio mode exists; If it exists, the audio modality is subjected to noise reduction and normalization processing to obtain an audio normalized modality; the audio normalized modality is translated into a text vector to obtain a first audio imitation text vector; the first audio imitation text vector is fused with the corresponding vector of the text modality to obtain an audio-text fusion vector, and the translation result generation module is executed; if it does not exist, the text modality is divided into several text sub-modalities, which are sequentially translated by a neural network and converted into vectors, and then spliced to obtain an initial translation vector; the initial translation vector and the corresponding vector of the text modality are spliced to obtain a text time series vector; the text time series vector is subjected to time series semantic analysis to obtain a text time series feature vector, and the translation result generation module is executed; The audio and image modality feature extraction module, if only the audio modality exists, extracts speech features from the audio modality to obtain speech signal features; obtains acoustic features based on the speech signal features, performs decoding processing to obtain an acoustic hidden state sequence; converts the acoustic hidden state sequence into a phoneme sequence, and converts the phoneme sequence into a text vector based on the pronunciation rules and vocabulary of the language to obtain a second audio-to-text vector, and executes the translation result generation module; If only the image modality exists, the image modality is subjected to image enhancement processing to obtain the image enhancement modality; the image enhancement modality is subjected to text feature extraction to obtain the image imitation text vector, and the translation result generation module is executed; If both audio and image modalities exist, context features of the audio and image modalities are extracted to obtain audio and image features respectively. The feature with the maximum information entropy among the audio and image features is used as the imitated feature. Another feature is translated into the imitated feature and then fused with the imitated feature to obtain an audio-image fusion feature. The audio-image fusion feature is translated into a text vector to obtain a fused imitation text vector, and the translation result generation module is executed. The translation result generation module obtains a translation result by machine translation processing of the audio-text fusion feature vector, or the text time sequence feature vector, or the second audio imitation text vector, or the image imitation text vector, or the fusion imitation text vector.

9. A translation device based on multimodal interaction, characterized in that: The system comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, the translation method based on multimodal interaction according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the translation method based on multimodal interaction according to any one of claims 1 to 7 is implemented.

Images