Multimodal fusion speech translation method, system and device
Through the multimodal fusion method, combining audio and text features, and using adaptive weight allocation and multi-level encoding technology, the accuracy of existing speech translation methods under environmental noise and context changes is solved, and higher quality Voice translation results.
Patent Information
- Application Number
- CN202411192430.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-28
AI Technical Summary
Existing speech translation methods rely on single audio information and are susceptible to environmental noise, accent differences and context changes, resulting in low accuracy of translation results, especially in professional fields and complex semantic scenarios.
Using a multimodal fusion method, the feature data is processed by combining audio feature extraction and text feature extraction, using adaptive weight allocation fusion strategy and multi-level coding units, combining real-time voice transcription, historical chat records and professional knowledge base information , perform multi-level encoding and translation of speech signals.
It improves the accuracy and fluency of pronunciation translation, can better understand the context and meaning of pronunciation signals, adapt to translation needs in different scenarios, and improves the quality and nature of translation results.
Smart Images

Figure CN118692446B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speech recognition, and in particular to a multimodal fusion speech translation method, system and device. Background Art
[0002] Traditional speech translation methods mainly rely on single audio information for translation processing. However, translation methods based solely on audio have many limitations.
[0003] On the one hand, audio signals may be disturbed by environmental noise, accent differences, speech speed changes, etc. during transmission, which makes it difficult to accurately extract semantic information from audio alone. For example, in noisy public places or in voice scenarios with dialect accents, the accuracy of traditional voice translation systems will be greatly reduced.
[0004] On the other hand, for some speech contents that contain a lot of professional terms, complex semantic structures, or rely on specific contexts, it is difficult to fully and accurately understand their meanings by relying solely on audio features. For example, in professional communication such as academic seminars and business negotiations, the same words may have completely different meanings in different contexts, and a single audio model cannot fully combine background knowledge and relevant text information to accurately interpret them.
[0005] In addition, existing speech translation technology also has deficiencies in the encoding and translation processing stages. Traditional encoding methods usually use fixed encoding algorithms and parameters, which cannot be adaptively adjusted according to the characteristics and changes of speech signals, resulting in the encoded features not being able to well reflect the essential characteristics of speech, affecting the accuracy of subsequent translation.
[0006] Therefore, the above defects make the current translation method have low translation accuracy. Summary of the invention
[0007] The main purpose of the present invention is to provide a multimodal fusion speech translation method, system and device, aiming to overcome the defect of low translation accuracy in current translation methods.
[0008] To achieve the above object, the present invention provides a multimodal fusion speech translation method, comprising the following steps:
[0009] Performing audio feature extraction processing on the input speech signal to obtain audio feature data;
[0010] Acquiring text information associated with the voice signal, performing text feature extraction processing on the text information to obtain text feature data;
[0011] Adopting an adaptive weight allocation fusion strategy to fuse the audio feature data and the text feature data to obtain fused feature data;
[0012] Based on the multi-level coding unit, the fused feature data is encoded to obtain the encoding feature; wherein, in the multi-level coding unit, each level of the serially connected coding unit adopts a different encoding algorithm, and the encoding parameters are adaptively adjusted according to the output result of the previous level;
[0013] The encoded features are input into a pre-trained translation model for translation processing to obtain a speech translation result.
[0014] Further, the acquiring text information associated with the voice signal includes:
[0015] Extracting the real-time speech transcription text corresponding to the speech signal as the first text information;
[0016] In the historical chat records, text content related to the scene to which the voice signal belongs is retrieved, and text related to the semantics of the voice signal is screened out as the second text information;
[0017] Searching for text information related to the keywords involved in the speech signal from a preset domain expertise database as the third text information;
[0018] The first text information, the second text information and the third text information are combined as text information associated with the voice signal.
[0019] Furthermore, the adaptive weight distribution fusion strategy is adopted to fuse the audio feature data and the text feature data to obtain fused feature data, including:
[0020] Analyzing the complexity and change rate of the audio feature data in real time, and determining the initial weight value of the audio feature data in the fusion process according to the complexity and change rate;
[0021] Evaluate the semantic richness of text feature data, and determine the initial weight value of the text feature data in the fusion process based on the evaluation results;
[0022] Combined with the preset voice and text importance bias rules, the initial weight values of the audio feature data and text feature data are dynamically adjusted;
[0023] Based on the adjusted weight values, the audio feature data and the text feature data are fused by weighted summation to obtain fused feature data.
[0024] Furthermore, the multi-stage encoding unit includes two-stage encoding units connected in series.
[0025] Furthermore, the encoding process of the fused feature data based on the multi-level encoding unit to obtain the encoding feature includes:
[0026] Based on the first-level coding unit, the fused feature data is preliminarily encoded based on the discrete cosine transform coding algorithm, the fused feature data is converted from the time domain to the frequency domain, the frequency component features are extracted, and the feature information of the key frequency interval is screened from the frequency component features to generate the coding intermediate result;
[0027] According to the complexity and information amount of the intermediate result of the encoding, the network structure and parameters of the second-level encoding unit are automatically adjusted; based on the adjusted second-level encoding unit, a coding algorithm based on an autoencoder is used to perform feature compression and abstract representation on the intermediate result of the encoding to generate the encoding feature.
[0028] Further, the multi-level encoding unit includes three-level encoding units connected in series;
[0029] Among them, the first-level coding unit uses a coding algorithm based on discrete cosine transform to preliminarily encode the fused feature data, convert the fused feature data from the time domain to the frequency domain, extract the main frequency components and energy distribution information, and obtain preliminary coding features;
[0030] The second-level encoding unit adopts the encoding algorithm of the autoencoder, and adaptively adjusts the number of hidden layer neurons and the type of activation function of the autoencoder according to the distribution and complexity of the preliminary encoding features; the adjusted second-level encoding unit compresses and extracts features from the preliminary encoding features to obtain intermediate encoding features;
[0031] The third-level encoding unit adopts a recurrent neural network-based encoding method, and adaptively adjusts the time step of the recurrent neural network according to the feature performance of the first two levels of encoding results and the requirements of the translation task; the adjusted third-level encoding unit processes the intermediate encoding features according to the time series, captures the time dependency and dynamic change information therein, and obtains the final encoding features.
[0032] Furthermore, after the encoding feature is input into a pre-trained translation model for translation processing and a speech translation result is obtained, the method further includes:
[0033] Determine a user terminal and a management terminal for receiving the speech translation result;
[0034] Generate an authorization code based on the characteristic information of the user terminal and the management terminal; separate the authorization code into a first authorization code and a second authorization code; wherein the first authorization code is sent to the user terminal, and the second authorization code is sent to the management terminal;
[0035] The permission code is configured as the reading permission of the speech translation result and sent to the user terminal; when the user terminal reads the speech translation result, it is necessary to request the second permission code from the management terminal.
[0036] The present invention also provides a multi-modal fusion speech translation system, comprising:
[0037] The first extraction unit is used to perform audio feature extraction processing on the input speech signal to obtain audio feature data;
[0038] A second extraction unit is used to obtain text information associated with the voice signal, and perform text feature extraction processing on the text information to obtain text feature data;
[0039] A fusion unit, used for adopting an adaptive weight distribution fusion strategy to fuse the audio feature data and the text feature data to obtain fused feature data;
[0040] A coding unit, used for encoding the fused feature data based on a multi-level coding unit to obtain a coding feature; wherein, in the multi-level coding unit, each level of the serially connected coding unit adopts a different coding algorithm, and performs adaptive adjustment of the coding parameters according to the output result of the previous level;
[0041] The translation unit is used to input the encoding features into a pre-trained translation model for translation processing to obtain a speech translation result.
[0042] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.
[0043] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.
[0044] The multimodal fusion speech translation method, system and device provided by the present invention include: performing audio feature extraction processing on the input speech signal to obtain audio feature data; obtaining text information associated with the speech signal, performing text feature extraction processing on the text information to obtain text feature data; adopting an adaptive weight distribution fusion strategy to fuse the audio feature data and text feature data to obtain fusion feature data; encoding the fusion feature data based on a multi-level coding unit to obtain a coding feature; wherein, in the multi-level coding unit, each level of the serially connected coding unit adopts a different coding algorithm, and adaptively adjusts the coding parameters according to the output result of the previous level; the coding feature is input into a pre-trained translation model for translation processing to obtain a speech translation result. In the present invention, the accuracy of the translation result is improved by fusing the speech information and its associated text information; at the same time, the adaptive adjustment of the coding parameters of the multi-level coding unit is adopted to further improve the accuracy of the translation result. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 Schematic diagram of a multimodal fusion speech translation method in one embodiment of the present invention
[0046] Figure 2 It is a schematic diagram of the steps of a multimodal fusion speech translation method in one embodiment of the present invention;
[0047] Figure 3 Schematic diagram of a multimodal fusion speech translation method in another embodiment of the present invention
[0048] Figure 4 It is a structural block diagram of a multimodal fusion speech translation system in one embodiment of the present invention;
[0049] Figure 5 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.
[0050] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0052] Reference Figure 1 as well as Figure 2 In one embodiment of the present invention, a multimodal fusion speech translation method is provided, comprising the following steps:
[0053] Step S1, performing audio feature extraction processing on the input voice signal to obtain audio feature data;
[0054] Step S2, obtaining text information associated with the voice signal, performing text feature extraction processing on the text information to obtain text feature data;
[0055] Step S3, adopting an adaptive weight allocation fusion strategy to fuse the audio feature data and the text feature data to obtain fused feature data;
[0056] Step S4, encoding the fused feature data based on a multi-level encoding unit to obtain encoding features; wherein, in the multi-level encoding unit, each level of the serially connected encoding unit adopts a different encoding algorithm, and adaptively adjusts the encoding parameters according to the output result of the previous level;
[0057] Step S5: input the encoding features into a pre-trained translation model for translation processing to obtain a speech translation result.
[0058] In this embodiment, as described in step S1 above, a signal processing algorithm is used, such as a combination of multiple audio feature extraction technologies such as short-time Fourier transform and Mel frequency cepstral coefficient (MFCC). First, the input speech signal is divided into short-time frames, and each frame signal is subjected to spectrum analysis to extract basic audio features such as frequency, energy, and zero-crossing rate. In other embodiments, a convolutional neural network (CNN) in deep learning can also be introduced to perform feature learning on the audio signal, and a more advanced and abstract audio feature representation is automatically extracted through a trained CNN model. These features can better capture the semantic information in the speech signal and the characteristics of the speaker. After the audio feature extraction processing combined with multiple technologies, the obtained audio feature data has rich information content, which not only contains the basic acoustic features of the speech, but also contains the advanced semantic features extracted by the deep learning model, providing a more comprehensive and accurate audio information basis for subsequent translation processing.
[0059] As described in step S2 above, a multi-channel approach is adopted to obtain text information associated with the voice signal. On the one hand, the real-time voice transcription technology is used to convert the input voice signal into text in real time to ensure the synchronization and relevance of the text information and the voice. On the other hand, by retrieving text resources such as historical chat records and documents related to the scene or topic to which the current voice belongs, potential related text information is mined. In terms of text feature extraction, the word vector technology in natural language processing is used to convert each word in the text into a high-dimensional vector representation, and then these word vectors are sequence modeled by building a deep learning model, such as a recurrent neural network (RNN) or a long short-term memory network (LSTM), to extract the semantic features, grammatical structure features, etc. of the text. It can fully obtain text information closely related to the voice signal and convert it into text feature data with rich semantic information, which provides important text information support for multimodal fusion and helps to understand the meaning and background of the voice more accurately.
[0060] As described in step S3 above, an adaptive weight allocation fusion strategy is adopted to achieve effective fusion of audio features and text features, giving full play to the advantages of the two modal information, and being able to automatically adjust the fusion weight according to different situations, so that the fused feature data can more accurately reflect the true meaning and background information of the speech.
[0061] In one embodiment, the above-mentioned adaptive weight allocation fusion strategy is based on a fuzzy logic system and a machine learning algorithm. First, by learning and analyzing a large amount of multimodal data, an importance assessment model for audio features and text features is established. The model takes into account multiple factors, such as the context of the current voice signal (for example, whether it is in a quiet office environment or a noisy outdoor environment), the language type (characteristics and expressions of different languages), and the statistical analysis results of historical fusion data. According to the output of the importance assessment model, the fusion weights assigned to the audio feature data and the text feature data are dynamically adjusted. For example, when the context is a noisy environment, the fusion weight proportion of the audio feature data is increased to ensure the accuracy of the audio information; when the text information contains rich professional terms or key semantic information, the fusion weight of the text feature data is increased accordingly. Finally, using the weighted fusion algorithm, the audio feature data and the text feature data are weighted and summed according to the adjusted fusion weights to obtain fused feature data.
[0062] As described in step S4 above, a multi-level coding unit structure is constructed, and each level of coding unit adopts a different coding algorithm. For example, the first-level coding unit adopts a coding algorithm based on discrete cosine transform, converts the fused feature data from the time domain to the frequency domain, extracts the main frequency components and energy distribution information, obtains preliminary coding features, and adaptively adjusts the parameters of the discrete cosine transform according to the statistical characteristics of the fused feature data, such as the selection of the transformation basis function and the size of the transformation block. The second-level coding unit uses an autoencoder structure based on deep learning, and adaptively adjusts the network structure and parameters of the autoencoder according to the distribution and complexity of the first-level coding features, such as the number of hidden layer neurons and the type of activation function, to obtain intermediate coding features. The preliminary coding features obtained at the first level are used as input, and the encoder part in the autoencoder is further compressed and feature extracted by training. The third-level coding unit uses a coding method based on a recurrent neural network, and adaptively adjusts the time step of the recurrent neural network and the state update rule of the memory unit according to the feature performance of the first two levels of coding results and the requirements of the translation task, processes the intermediate coding features according to the time series, captures the time dependency and dynamic change information therein, and obtains the final coding features.
[0063] In this embodiment, through the step-by-step processing and adaptive parameter adjustment of multi-level encoding units, the encoded features can better meet the needs of speech translation tasks, effectively retain the key information of the speech, and remove redundant and noise information, providing high-quality encoding feature input for subsequent translation models.
[0064] As described in step S5 above, a pre-trained translation model is used, which is based on a neural network architecture in deep learning, such as a Transformer model. The Transformer model has a powerful sequence-to-sequence learning capability and can effectively process the encoded feature data after multimodal fusion. When training the translation model, a large-scale multimodal parallel corpus is used, including audio and corresponding text translation data. By optimizing the parameters of the model, the model can learn the complex mapping relationship between audio features, text features and target language translation results. In order to further improve the translation quality, an attention mechanism is introduced in the translation process so that the model can dynamically focus on the key information in the input encoding features, better understand the meaning of the source language, and generate accurate and fluent target language translation results.
[0065] In this embodiment, high-quality speech translation results are accurately generated based on the encoded feature data after multimodal fusion. The translation results have high accuracy, natural fluency and semantic coherence, and can meet the speech translation needs in different scenarios.
[0066] In one embodiment, the acquiring text information associated with the voice signal includes:
[0067] Extracting the real-time speech transcription text corresponding to the speech signal as the first text information;
[0068] In the historical chat records, text content related to the scene to which the voice signal belongs is retrieved, and text related to the semantics of the voice signal is screened out as the second text information;
[0069] Searching for text information related to the keywords involved in the speech signal from a preset domain expertise database as the third text information;
[0070] The first text information, the second text information and the third text information are combined as text information associated with the voice signal.
[0071] In this embodiment, the input voice signal is analyzed and processed by the acoustic model and the language model. The acoustic model is responsible for converting the voice signal into an acoustic feature vector, and then identifying the basic speech units such as phonemes or syllables in the voice by pattern matching and classification of these vectors. The language model is trained based on a large amount of text corpus to learn the knowledge of language grammar, semantic rules and vocabulary collocation. In the process of speech recognition, the language model predicts the most likely vocabulary sequence based on the phoneme sequence output by the acoustic model and the learned language knowledge, thereby realizing the real-time speech transcription text that converts the voice signal into text form. The above technical solution can quickly and accurately convert the input voice signal into text information in real time, and provide a text data foundation synchronized with the voice for subsequent text feature extraction and multimodal fusion. This real-time performance enables the system to use the text information corresponding to the voice in a timely manner for translation processing, especially for voices with a fast speaking speed or containing complex vocabulary, the meaning can be understood more accurately with the help of text.
[0072] In this embodiment, a historical chat record database is first established to classify and index all historical chat records. Classification can be performed according to different scene themes, such as business negotiations, academic discussions, daily life communication, etc. When a new voice signal is received, key topic vocabulary or semantic concepts are extracted from the voice signal through keyword extraction technology. Then, the historical chat record database is searched for text content related to these keywords or semantic concepts using an information retrieval algorithm. Semantic analysis technology is further used to evaluate the semantic relevance of the retrieved text content. By calculating the semantic similarity between the text and the voice signal, a text segment semantically related to the current voice signal is screened out as the second text information.
[0073] In this embodiment, the information in the historical chat records is fully utilized to provide context and supplementary information for the current speech translation. By retrieving the historical chat records, the meaning of the current speech can be better understood, and the accuracy and coherence of the translation can be improved.
[0074] In this embodiment, a preset domain expertise base is also constructed, which covers a large amount of text information in multiple professional fields, including professional terms, industry standards, technical documents, academic papers, etc. The knowledge in each professional field is organized and stored according to a certain structure and classification for quick retrieval. When processing speech signals, the professional fields and key topics involved in the speech are determined by semantic analysis and keyword extraction of the speech signals. Then, accurate text information search is performed in the professional knowledge base based on this information. Using knowledge graph technology and text matching algorithm, the extracted keywords are associated and matched with the knowledge nodes in the knowledge base, and text information related to the keywords in the speech signal is found and extracted as the third text information.
[0075] Finally, the extracted first text information (real-time speech transcription text), the screened second text information (relevant text in historical chat records), and the searched third text information (relevant text in the professional knowledge base) are integrated. These three types of text information are merged and deduplicated. When merging, the texts are arranged and combined according to the chronological order and semantic relevance to ensure the coherence and logic of the text information. Deduplication is to remove repeated words and information to improve the quality and efficiency of text information. The text information in real-time, historical and professional fields is integrated to provide comprehensive and rich text background support for speech translation. By combining these three types of text information, the meaning and context of speech can be understood more comprehensively, thereby improving the accuracy and quality of speech translation and meeting translation needs in different scenarios.
[0076] In one embodiment, the adaptive weight distribution fusion strategy is used to fuse the audio feature data and the text feature data to obtain fused feature data, including:
[0077] Analyzing the complexity and change rate of the audio feature data in real time, and determining the initial weight value of the audio feature data in the fusion process according to the complexity and change rate;
[0078] Evaluate the semantic richness of text feature data, and determine the initial weight value of the text feature data in the fusion process based on the evaluation results;
[0079] Combined with the preset voice and text importance bias rules, the initial weight values of the audio feature data and text feature data are dynamically adjusted;
[0080] Based on the adjusted weight values, the audio feature data and the text feature data are fused by weighted summation to obtain fused feature data.
[0081] In this embodiment, first, for the complexity analysis of audio feature data, various indicators and algorithms in signal processing can be used. For example, the spectral complexity of the audio signal is calculated, and the energy distribution in different frequency bands is analyzed by performing a fast Fourier transform on the audio signal. If the frequency distribution is relatively dispersed and contains multiple prominent frequency peaks, it means that the audio complexity is high. The calculation of the rate of change is measured by the difference between consecutive audio frames. The characteristic parameters of adjacent audio frames are compared, such as the difference in Mel-frequency cepstral coefficients, and the rate of change of the audio feature data is determined based on the size and change trend of these differences. Then, according to the pre-established mapping relationship model between complexity and rate of change and weight value, the calculated complexity and rate of change values are substituted into the model to determine the initial weight value of the audio feature data in the fusion process.
[0082] In this embodiment, when the audio complexity is high or the rate of change is large, it means that it may contain more key information or be in a complex voice environment. At this time, it is given a correspondingly higher initial weight to ensure that the important features of the audio can be fully reflected during the fusion process, thereby improving the adaptability to complex audio situations.
[0083] In terms of semantic richness assessment, the technology of natural language processing is used. First, the text feature data is subjected to lexical analysis to count the number of different words in the text, the diversity of words, and the proportion of professional terms and complex words. Then, by constructing a semantic network, the degree of semantic association between words in the text and the structural complexity of sentences are analyzed. For example, the semantic similarity and semantic relevance between words in a sentence, as well as the grammatical structure complexity of the sentence, are calculated. The semantic richness of the text feature data is comprehensively evaluated based on the above indicators, and the initial weight value of the text feature data in the fusion process is determined based on the preset correspondence between semantic richness and weight value.
[0084] In this embodiment, the amount of information and semantic value contained in the text feature data can be accurately measured, and an appropriate initial weight can be assigned to it. When the text semantic richness is high, it is given a correspondingly higher weight, so that the important role of text information in understanding the meaning of speech can be fully utilized during the fusion process, especially in cases involving professional fields or complex semantics, highlighting the value of the text.
[0085] The preset voice and text importance bias rules are obtained based on a large amount of multimodal data training and analysis in advance. The above rules take into account a variety of factors, such as the relative importance of voice and text in different scenarios, the characteristics of different language types, and user preferences. For example, in some scenarios dominated by oral communication, such as real-time conversations or speeches, voice may be relatively more important. At this time, the weight adjustment range of audio feature data is appropriately increased according to the rules; while in some scenarios where written text is an important reference, such as reading and explaining voice content, the weight adjustment of text may be more valued. According to the above rules, combined with the current specific scenario and the initial weight values of audio and text obtained by analysis, a dynamic adjustment algorithm is used to adjust the initial weight values of the two. For example, an adjustment algorithm based on fuzzy logic can be used to automatically adjust the increase or decrease of the weight value through a fuzzy reasoning system according to the input scenario information and the initial weight value.
[0086] In this embodiment, the weights of audio and text feature data are flexibly adjusted according to actual conditions to adapt them to different application scenarios and needs. By combining preset rules for dynamic adjustment, it is possible to ensure the fusion effect under general circumstances, highlight the role of key information sources in special scenarios, and improve the accuracy and adaptability of fusion.
[0087] After determining the weight values of the adjusted audio and text feature data, a weighted sum fusion method is used. Multiply the audio feature data and text feature data by their corresponding weight values, and then add the products to obtain the fused feature data. For example, if the audio feature data is A, its weight is w1; the text feature data is B, its weight is w2, then the fused feature data C = w1 * A + w2 * B.
[0088] In this embodiment, the feature information of audio and text is organically fused according to their importance by weighted summation to obtain fused feature data that combines the advantages of both. This fused feature data contains both the real-time dynamic information of the audio and the rich semantic information of the text, providing a more comprehensive and accurate information basis for subsequent encoding and translation processing, and helping to improve the performance and effect of the entire speech translation system.
[0089] In one embodiment, the multi-stage encoding unit includes two-stage encoding units connected in series.
[0090] In this embodiment, the encoding process of the fused feature data based on the multi-level encoding unit to obtain the encoding feature includes:
[0091] Based on the first-level coding unit, the fused feature data is preliminarily encoded based on the discrete cosine transform coding algorithm, the fused feature data is converted from the time domain to the frequency domain, the frequency component features are extracted, and the feature information of the key frequency interval is screened from the frequency component features to generate the coding intermediate result;
[0092] According to the complexity and information amount of the intermediate result of the encoding, the network structure and parameters of the second-level encoding unit are automatically adjusted; based on the adjusted second-level encoding unit, a coding algorithm based on an autoencoder is used to perform feature compression and abstract representation on the intermediate result of the encoding to generate the encoding feature.
[0093] In this embodiment, the first-level encoding unit uses a discrete cosine transform (DCT) encoding algorithm as a basic tool. Discrete cosine transform is a mathematical method for converting signals from the time domain to the frequency domain, and has good energy concentration characteristics and decorrelation. First, the fused feature data is divided into appropriate time segments or data blocks, and a discrete cosine transform operation is performed on each data block. Through DCT calculation, each data point in the fused feature data is converted into a corresponding frequency domain coefficient, and these coefficients represent the distribution of the data at different frequencies. Then, the obtained frequency domain coefficients are analyzed, and the feature information of the key frequency interval with important information is screened out from a large number of frequency component features according to the preset frequency screening rules or the key frequency interval determined based on data analysis. For example, the feature information corresponding to the frequency interval with higher energy can be retained according to the energy distribution of the frequency. Finally, the filtered feature information is sorted and combined to generate an intermediate encoding result. In this embodiment, by converting the fused feature data to the frequency domain and screening the feature information of the key frequency interval, the preliminary compression of the data and the extraction of key information are achieved. Converting data from the time domain to the frequency domain can better reveal the inherent structure and frequency characteristics of the data, and help remove some redundant information and noise interference. At the same time, screening the characteristic information of key frequency intervals can highlight the important parts of the data, provide a more targeted information basis for subsequent further processing, and improve the efficiency and accuracy of encoding.
[0094] The second-level encoding unit first analyzes the complexity and information content of the intermediate encoding result. The complexity can be measured by calculating the data change rate of the intermediate encoding result, the distribution characteristics of the data (such as standard deviation, entropy, etc.). The amount of information can be determined by evaluating the number of key features, the diversity of features, and the relevance to the target task. According to the complexity and information content results obtained by the analysis, an adaptive adjustment algorithm is used to automatically adjust the network structure and parameters of the second-level encoding unit. For example, if the complexity of the intermediate encoding result is high, it is necessary to increase the number of hidden layer nodes in the autoencoder network (the number of nodes can be increased one by one) or adjust the type of activation function (you can choose from a variety of preset types) to enhance the network's ability to process complex data; if the amount of information is large, it is necessary to optimize the parameters of the network to better retain and process this information. The second-level encoding unit uses an encoding algorithm based on an autoencoder to process the intermediate encoding result. The autoencoder is an unsupervised learning neural network model consisting of an encoder and a decoder. The encoder compresses the input data into a low-dimensional encoding feature representation, and the decoder attempts to reconstruct the original data from the encoding feature. In this process, the encoder learns the abstract feature representation of the data and realizes feature compression and abstraction of the data. The adjusted autoencoder network is applied to the encoding intermediate result. By training the autoencoder network, it is able to learn the potential patterns and features in the encoding intermediate result and transform it into more compact and representative encoding features.
[0095] In this embodiment, the structure and parameters of the first-level coding result can be automatically adjusted according to the characteristics of the first-level coding result, realizing adaptive processing of different types of data. Through the coding algorithm based on the autoencoder, the intermediate coding result is further compressed and abstractly represented, making the coding features more concise and efficient, while retaining the key information and semantic features of the data. This multi-level coding method improves the flexibility and adaptability of the coding, provides high-quality coding feature input for subsequent translation tasks, and helps to improve the performance and effect of the entire system.
[0096] Combined with reference Figure 3 , in one embodiment, the multi-stage encoding unit includes three-stage encoding units connected in series;
[0097] Among them, the first-level coding unit uses a coding algorithm based on discrete cosine transform to preliminarily encode the fused feature data, convert the fused feature data from the time domain to the frequency domain, extract the main frequency components and energy distribution information, and obtain preliminary coding features;
[0098] The second-level encoding unit adopts the encoding algorithm of the autoencoder, and adaptively adjusts the number of hidden layer neurons and the type of activation function of the autoencoder according to the distribution and complexity of the preliminary encoding features; the adjusted second-level encoding unit compresses and extracts features from the preliminary encoding features to obtain intermediate encoding features;
[0099] The third-level encoding unit adopts a recurrent neural network-based encoding method, and adaptively adjusts the time step of the recurrent neural network according to the feature performance of the first two levels of encoding results and the requirements of the translation task; the adjusted third-level encoding unit processes the intermediate encoding features according to the time series, captures the time dependency and dynamic change information therein, and obtains the final encoding features.
[0100] In this embodiment, the first-level encoding unit uses discrete cosine transform (DCT) to achieve the conversion from time domain to frequency domain. The fused feature data is regarded as a time series signal and is divided into multiple data segments according to a certain time window. Discrete cosine transform calculation is performed on each data segment, and the representation of the data is converted from a function in the time domain to a set of coefficients in the frequency domain through a series of mathematical operations. These coefficients reflect the energy distribution of the data at different frequencies. By analyzing these frequency coefficients, the main frequency components are extracted, that is, those frequency parts with higher energy and greater contribution to the data characteristics. At the same time, the distribution of energy in different frequency intervals is also analyzed to fully understand the energy distribution information of the data. The preliminary processing and feature extraction of the fused feature data are realized, and the data is converted from the original time representation to a form that better reflects its inherent frequency characteristics. By extracting the main frequency components and energy distribution information, basic frequency feature data is provided for subsequent encoding processing, the complexity of the data is reduced, the key frequency information in the data is highlighted, and it helps to better understand and process the data.
[0101] The second-level encoding unit adopts the deep learning model of autoencoder for encoding processing. The autoencoder has the ability to automatically learn data feature representation, can compress input data into low-dimensional encoding representation, and retain the key information of the data as much as possible. First, analyze the distribution characteristics of the preliminary encoding features, such as calculating the mean, variance, kurtosis and other statistics to understand the distribution form of the data. At the same time, evaluate the complexity of the preliminary encoding features, such as by calculating the entropy and complexity index of the data. According to the analysis results, the structural parameters of the autoencoder are adaptively adjusted. For example, the number of hidden layer neurons is adjusted according to the complexity of the data and the richness of the features. If the data is more complex, increase the number of hidden layer neurons to improve the expression ability of the model; if the data is relatively simple, the number of neurons can be appropriately reduced to avoid overfitting. For the type of activation function, select a suitable activation function according to the distribution of the data and the requirements of the task, such as ReLU, Sigmoid, etc. The adjusted autoencoder is used to further compress and extract the preliminary encoding features. Through the training process of the autoencoder, the model automatically learns a higher-level and more abstract feature representation of the data, removes redundant information, and obtains intermediate encoding features. It can perform adaptive optimization and processing based on the characteristics of the first-level coding results, improving the efficiency and accuracy of coding. By compressing and extracting the initial coding features, more representative and abstract intermediate coding features are obtained, providing better data representation for subsequent processing, while also reducing the dimension of the data and the computational complexity.
[0102] The third-level encoding unit introduces an encoding method based on a recurrent neural network (RNN). RNN is good at processing time series data and can capture the time dependency and dynamic change information in the data.
[0103] First, we analyze the characteristics of the first two levels of encoding results. For example, we evaluate the time series characteristics of the intermediate encoding features, the trend of data changes, and possible periodicity. At the same time, we consider the degree of dependence and requirements on time information in the translation process in combination with the specific needs of the translation task.
[0104] Based on the above analysis, the time step of the recurrent neural network is adaptively adjusted. The time step determines the time span that the RNN considers when processing data. For data that changes rapidly or tasks that need to capture time dynamics more finely, the time step can be appropriately reduced; for data that changes relatively slowly or requires a more macroscopic grasp of time trends, the time step can be increased.
[0105] The intermediate coding features are input into the adjusted recurrent neural network in time series for processing. Through its internal memory unit and loop structure, RNN can process the time series data step by time, update its internal state at each time step, so as to capture the time dependency and dynamic change information in the data, and finally obtain the final coding features. In this embodiment, the time characteristics of the data and the requirements of the translation task are fully considered, and further optimization and processing of the intermediate coding features are achieved. By capturing the time dependency and dynamic change information, a richer and more accurate time series feature representation is provided for the translation model, so that the translation result can better reflect the time change and semantic coherence of the speech data, and improve the quality and accuracy of the translation.
[0106] In one embodiment, the coding feature is input into a pre-trained translation model for translation processing, and after obtaining a speech translation result, the method further includes:
[0107] Determine a user terminal and a management terminal for receiving the speech translation result;
[0108] Generate an authorization code based on the characteristic information of the user terminal and the management terminal; separate the authorization code into a first authorization code and a second authorization code; wherein the first authorization code is sent to the user terminal, and the second authorization code is sent to the management terminal;
[0109] The permission code is configured as the reading permission of the speech translation result and sent to the user terminal; when the user terminal reads the speech translation result, it is necessary to request the second permission code from the management terminal.
[0110] In this embodiment, the user terminal and the management terminal that receive the speech translation result can be identified and associated in a variety of ways. For example, when the user initiates a speech translation request, the device information used by the user (such as the unique identifier of the device, IP address, etc.) is recorded as the identification of the user terminal. At the same time, for the management terminal, a specific management device is pre-set or determined based on the administrator's login information and device connection status.
[0111] When generating the permission code, a combination of encryption algorithm and random number generator can be used. For example, an initial random string is generated based on multiple factors such as the current timestamp, the unique number of the translation task, and the relevant characteristic information of the user and management terminal (such as device identification information). Then, this random string is encrypted by an encryption algorithm (such as the symmetric encryption algorithm AES or the asymmetric encryption algorithm RSA) to obtain the final permission code.
[0112] When the generated permission code is separated, a specific separation algorithm is used or the separation is performed according to a preset rule. For example, the permission code can be separated according to the length or specific character position, and divided into a first permission code and a second permission code. In this embodiment, the terminal that needs to receive the translation result can be accurately identified, and a permission code with certain security and uniqueness is generated for it. By means of encryption and random generation, it is ensured that the permission code is difficult to be deciphered and forged, thereby ensuring the security of the translation result and the accuracy of access control.
[0113] When the permission code is configured as the reading permission of the speech translation result, the corresponding permission information is associated with the translation result in the database or file system storing the translation result, and the permission code is stored and managed as the key verification information for accessing the translation result.
[0114] When the user terminal needs to read the speech translation result, it is first detected whether the user terminal has a complete permission code (including the first permission code and the second permission code obtained from the management terminal). The process of the user terminal requesting the second permission code from the management terminal can be implemented through a network communication protocol, such as sending a request message in a specific format to the management terminal. After receiving the request, the management terminal can decide whether to provide the second permission code to the user terminal according to the preset approval process and security policy. The management terminal can review the user terminal's request, such as checking the user's identity, the legitimacy of the request, and whether it complies with relevant security policies and business rules. If the review is passed, the management terminal sends the second permission code to the user terminal; if the review is not passed, the second permission code is refused to be provided.
[0115] In this embodiment, fine access control of speech translation results is achieved. By separating the permission codes and sending them to the user terminal and the management terminal respectively, the management terminal can effectively manage and supervise the access of the user terminal. This method ensures the security of the translation results and prevents unauthorized access. At the same time, it also provides the management terminal with a means to control sensitive translation results, meeting the flexible management requirements for access rights to translation results in different application scenarios.
[0116] In one embodiment, the generating of the permission code based on the characteristic information of the user terminal and the management terminal includes:
[0117] Obtain a first identifier of the user terminal and a second identifier of the management terminal;
[0118] Concatenate the first identifier and the second identifier to obtain a concatenated identifier;
[0119] Get the total number of characters in the concatenated identifier;
[0120] Creating a plurality of evenly distributed character grids on a circle, and adding the concatenated identifiers to the character grids one by one in sequence; wherein the number of the character grids is the same as the total number of characters;
[0121] Randomly generate three straight lines passing through the center of the circle on the circle; wherein the three straight lines do not overlap, and the three straight lines and the circle generate six intersection points;
[0122] The character grids closest to the six intersection points are obtained on the circle as target character grids, and the characters in the target character grids are combined in sequence to obtain the permission code.
[0123] In this embodiment, each user terminal and management terminal has a unique identifier, which can be in the form of a device serial number, MAC address, unique user account, etc. Obtaining these identifiers is a basic step in generating permission codes, because they contain key information related to a specific device or user, and by processing this information, a targeted and unique permission code can be generated. For example, the first identifier of the user terminal can be a device serial number composed of numbers and letters, which can uniquely identify the device used by the user; the second identifier of the management terminal may be the administrator's specific account code, which is used to distinguish different levels of management authority. For the first identifier of the user terminal, it can be obtained through the interface provided by the device's operating system or a specific hardware information reading program. For example, on a mobile device, the unique identifier of the device can be obtained by calling the system's device information API; on a computer, the MAC address and other identifiers can be obtained by reading the relevant register information of the hardware device.
[0124] The second identification of the management terminal is usually assigned during the login or registration process of the management system, or generated based on the specific identity information of the administrator. For example, when the administrator logs in to the management system, the system will generate a corresponding identification based on his account information for subsequent permission management and operation tracking.
[0125] The purpose of concatenating the first identification of the user terminal and the second identification of the management terminal is to merge the two independent identifications into a unified identification string for subsequent overall processing and encoding. Through concatenation, the information of the user and the management terminal can be integrated to form a unique identification sequence related to a specific user-management terminal combination. For example, if the first identification of the user terminal is "ABC123" and the second identification of the management terminal is "XYZ789", then the concatenated identification is "ABC123XYZ789". This concatenated identification contains the dual information of the user and the management terminal, providing richer raw data for generating permission codes. Simple string concatenation functions or algorithms can be used to implement concatenation operations. In programming, common programming languages such as Python and Java provide string concatenation methods. For example, in Python, the "+" operator can be used to concatenate two strings together; in Java, the append method of the StringBuilder or StringBuffer class can be used to implement string concatenation.
[0126] Furthermore, the total number of characters in the concatenated identifier is calculated in order to determine the number of character grids that need to be created. The number of character grids is the same as the total number of characters, which ensures that each character has a corresponding character grid for placement and processing. The total number of characters can be obtained by calculating the length of the concatenated identifier string. For example, using the string length function in a programming language, such as the len() function in Python, the character length of the concatenated identifier can be quickly obtained. If the concatenated identifier is "ABC123XYZ789", then the total number of characters is 10.
[0127] Create multiple evenly distributed character grids on a virtual circle. This is a way to spatially layout the characters, providing a geometric basis for the subsequent random selection and encoding process. Create the same number of character grids as the total number of characters, ensuring that each character grid has a unique position and number. Then, add the characters in the concatenated logo to the character grid one by one in sequence. For example, the first character is placed in the first character grid, the second character is placed in the second character grid, and so on. In this way, each character is assigned to a specific position on the circle.
[0128] Three straight lines are randomly generated on the circle that pass through the center of the circle. This is to introduce a random factor, making the generated permission code unpredictable and secure. Through the intersection of these three straight lines with the circle, specific characters can be selected from the character grid on the circle as the key element for generating the permission code. The process of randomly generating straight lines can be implemented using a random number generator. For example, in a two-dimensional coordinate system, with the center of the circle as the origin, three straight lines at different angles are randomly generated to ensure that they do not overlap and can generate six intersections with the circle.
[0129] Then, find the character grids closest to the six intersections on the circle, and mark these character grids as target character grids. The closest judgment can be determined based on the calculation method of geometric distance, for example, the Euclidean distance from the intersection to the center of each character grid is calculated, and the character grid with the closest distance is the target character grid. The characters in the target character grid are combined in sequence to obtain the final permission code. This permission code is generated based on the feature information of the user terminal and the management terminal through a series of processing and randomization operations. It is unique and random and is used to control the user's access rights and operation scope in a specific management system. For example, if the characters in the six target character grids are "1", "A", "5", "B", "9", and "C", respectively, then the combined permission code is "1A5B9C".
[0130] Reference Figure 4 In another embodiment of the present invention, a multimodal fusion speech translation system is provided, comprising:
[0131] The first extraction unit is used to perform audio feature extraction processing on the input speech signal to obtain audio feature data;
[0132] A second extraction unit is used to obtain text information associated with the voice signal, and perform text feature extraction processing on the text information to obtain text feature data;
[0133] A fusion unit, used for adopting an adaptive weight distribution fusion strategy to fuse the audio feature data and the text feature data to obtain fused feature data;
[0134] A coding unit, used for encoding the fused feature data based on a multi-level coding unit to obtain a coding feature; wherein, in the multi-level coding unit, each level of the serially connected coding unit adopts a different coding algorithm, and performs adaptive adjustment of the coding parameters according to the output result of the previous level;
[0135] The translation unit is used to input the encoding features into a pre-trained translation model for translation processing to obtain a speech translation result.
[0136] In this embodiment, for the specific implementation of each unit in the above system embodiment, please refer to the description in the above method embodiment, which will not be repeated here.
[0137] Reference Figure 5 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.
[0138] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.
[0139] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.
[0140] In summary, the multimodal fusion speech translation method, system and device provided in the embodiment of the present invention include: performing audio feature extraction processing on the input speech signal to obtain audio feature data; obtaining text information associated with the speech signal, performing text feature extraction processing on the text information to obtain text feature data; adopting an adaptive weight distribution fusion strategy to fuse the audio feature data and text feature data to obtain fusion feature data; encoding the fusion feature data based on a multi-level encoding unit to obtain encoding features; wherein, in the multi-level encoding unit, each level of the serially connected encoding unit adopts a different encoding algorithm, and adaptively adjusts the encoding parameters according to the output result of the previous level; the encoding features are input into a pre-trained translation model for translation processing to obtain a speech translation result. In the present invention, the accuracy of the translation result is improved by fusing the speech information and its associated text information; at the same time, the adaptive adjustment of the encoding parameters of the multi-level encoding unit is adopted to further improve the accuracy of the translation result.
[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.
[0142] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.
[0143] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A multimodal fusion speech translation method, characterized in that: The following steps are involved: Performing audio feature extraction processing on the input speech signal to obtain audio feature data; Acquiring text information associated with the voice signal, performing text feature extraction processing on the text information to obtain text feature data; Adopting an adaptive weight allocation fusion strategy to fuse the audio feature data and the text feature data to obtain fused feature data; Based on the multi-level coding unit, the fused feature data is encoded to obtain the encoding feature; wherein, in the multi-level coding unit, each level of the serially connected coding unit adopts a different encoding algorithm, and the encoding parameters are adaptively adjusted according to the output result of the previous level; Inputting the encoded features into a pre-trained translation model for translation processing to obtain a speech translation result; The acquiring of text information associated with the voice signal comprises: Extracting the real-time speech transcription text corresponding to the speech signal as the first text information; In the historical chat records, text content related to the scene to which the voice signal belongs is retrieved, and text related to the semantics of the voice signal is screened out as the second text information; Searching for text information related to the keywords involved in the speech signal from a preset domain expertise database as the third text information; The first text information, the second text information and the third text information are combined as text information associated with the voice signal.
2. The multimodal fusion speech translation method according to claim 1, characterized in that: The method adopts an adaptive weight allocation fusion strategy to fuse the audio feature data and the text feature data to obtain fused feature data, including: Analyzing the complexity and change rate of the audio feature data in real time, and determining the initial weight value of the audio feature data in the fusion process according to the complexity and change rate; Evaluate the semantic richness of text feature data, and determine the initial weight value of the text feature data in the fusion process based on the evaluation results; Combined with the preset voice and text importance bias rules, the initial weight values of the audio feature data and text feature data are dynamically adjusted; Based on the adjusted weight values, the audio feature data and the text feature data are fused by weighted summation to obtain fused feature data.
3. The multimodal fusion speech translation method according to claim 1, characterized in that: The multi-stage encoding unit includes two-stage encoding units connected in series.
4. The multimodal fusion speech translation method according to claim 3, characterized in that: The encoding process of the fused feature data based on the multi-level encoding unit to obtain the encoding feature includes: Based on the first-level coding unit, the fused feature data is preliminarily encoded based on the discrete cosine transform coding algorithm, the fused feature data is converted from the time domain to the frequency domain, the frequency component features are extracted, and the feature information of the key frequency interval is screened from the frequency component features to generate the coding intermediate result; According to the complexity and information amount of the intermediate result of the encoding, the network structure and parameters of the second-level encoding unit are automatically adjusted; based on the adjusted second-level encoding unit, a coding algorithm based on an autoencoder is used to perform feature compression and abstract representation on the intermediate result of the encoding to generate the encoding feature.
5. The multimodal fusion speech translation method according to claim 1, characterized in that: The multi-level encoding unit includes three-level encoding units connected in series; Among them, the first-level coding unit uses a coding algorithm based on discrete cosine transform to preliminarily encode the fused feature data, convert the fused feature data from the time domain to the frequency domain, extract the main frequency components and energy distribution information, and obtain preliminary coding features; The second-level encoding unit adopts the encoding algorithm of the autoencoder, and adaptively adjusts the number of hidden layer neurons and the type of activation function of the autoencoder according to the distribution and complexity of the preliminary encoding features; the adjusted second-level encoding unit compresses and extracts features from the preliminary encoding features to obtain intermediate encoding features; The third-level encoding unit adopts a recurrent neural network-based encoding method, and adaptively adjusts the time step of the recurrent neural network according to the feature performance of the first two levels of encoding results and the requirements of the translation task; the adjusted third-level encoding unit processes the intermediate encoding features according to the time series, captures the time dependency and dynamic change information therein, and obtains the final encoding features.
6. The multimodal fusion speech translation method according to claim 1, characterized in that: The encoding feature is input into a pre-trained translation model for translation processing to obtain a speech translation result, and the method further includes: Determine a user terminal and a management terminal for receiving the speech translation result; Generate an authorization code based on the characteristic information of the user terminal and the management terminal; separate the authorization code into a first authorization code and a second authorization code; wherein the first authorization code is sent to the user terminal, and the second authorization code is sent to the management terminal; The permission code is configured as the reading permission of the speech translation result and sent to the user terminal; when the user terminal reads the speech translation result, it is necessary to request the second permission code from the management terminal.
7. A multimodal fusion speech translation system, characterized in that: include: The first extraction unit is used to perform audio feature extraction processing on the input speech signal to obtain audio feature data; A second extraction unit is used to obtain text information associated with the voice signal, and perform text feature extraction processing on the text information to obtain text feature data; A fusion unit, used for adopting an adaptive weight distribution fusion strategy to fuse the audio feature data and the text feature data to obtain fused feature data; A coding unit, used for encoding the fused feature data based on a multi-level coding unit to obtain a coding feature; wherein, in the multi-level coding unit, each level of the serially connected coding unit adopts a different coding algorithm, and performs adaptive adjustment of the coding parameters according to the output result of the previous level; A translation unit, used for inputting the encoding features into a pre-trained translation model for translation processing to obtain a speech translation result; The acquiring of text information associated with the voice signal comprises: Extracting the real-time speech transcription text corresponding to the speech signal as the first text information; In the historical chat records, text content related to the scene to which the voice signal belongs is retrieved, and text related to the semantics of the voice signal is screened out as the second text information; Searching for text information related to the keywords involved in the speech signal from a preset domain expertise database as the third text information; The first text information, the second text information and the third text information are combined as text information associated with the voice signal.
8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
