Speech recognition, synthesis and translation method and device and storage medium

Abstract

The invention relates to the technical field of natural language processing, in particular to a speech recognition, synthesis and translation method and device and a storage medium. According to the method, the speech recognition model and the machine translation model are jointly trained, so that the task correlation between speech recognition and translation can be optimized in the training process, and the accuracy of a translation result is remarkably improved; according to the method, a deep learning technology and optimization of a joint loss function are utilized, so that a speech recognition module and a translation module can work cooperatively, error accumulation during processing of a single module is reduced, and the overall translation quality is improved; denoising and volume standardization processing are carried out in the voice data preprocessing stage, so that the input data quality in the subsequent voice recognition and translation process is optimized; by adopting the speech recognition model and the machine translation model based on the feedforward neural network, speech input can be efficiently processed, and a corresponding translation result can be generated.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a speech recognition synthesis translation method, device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) and natural language processing (NLP) technologies, speech recognition and machine translation technologies have gradually become core technologies for applications such as intelligent voice interaction and real-time translation. The speech recognition system can convert speech signals into text, while machine translation can translate text in the source language into text in the target language. The combination of these two technologies makes multilingual communication more convenient. However, in practical applications, existing technologies still face some problems, especially in the integration and optimization of speech recognition and translation systems.

[0003] First, existing speech recognition systems are often affected by environmental noise, differences in voice quality, and pronunciation, resulting in low accuracy in audio signal recognition after processing. Although there are denoising and enhancement methods, these methods are often effective in specific noise environments and are difficult to cope with complex real-world situations.

[0004] Secondly, current speech recognition systems mainly rely on Mel-frequency cepstral coefficients in the feature extraction stage, but due to the diversity of speech signals themselves, existing MFCC feature extraction methods perform poorly for complex speech signals.

[0005] Finally, in the prior art, speech recognition and machine translation are usually two independent processing steps. Although machine translation can be performed after the speech recognition module outputs text, this processing flow may lead to the accumulation of errors. Especially when dealing with complex conversations and real-time speech translation, how to effectively combine speech recognition with machine translation models.

[0006] In view of the above problems, it is necessary to propose a speech recognition synthesis translation method, device and storage medium. Summary of the invention

[0007] The purpose of the present invention is to solve the problems existing in the background technology and to provide a speech recognition synthesis translation method, device and storage medium.

[0008] The purpose of the present invention can be achieved through the following technical solutions:

[0009] In a first aspect, the present invention provides a speech recognition synthesis translation method, comprising the following steps:

[0010] Step 1: Preparation and preprocessing of training data set;

[0011] Collect and prepare speech data pairs containing source and target languages. The speech data is used in various scenarios in daily conversations to ensure the practicality and generalization ability of the system.

[0012] S101 , pre-processing the speech data, including noise removal and volume normalization.

[0013] The denoising process is specifically as follows: obtaining the time-series audio signal x(t) corresponding to each source language speech data. The time-series audio signal x(t) is converted from the time domain to the frequency domain by fast Fourier transform FFT to obtain the frequency domain signal X(f). The fast Fourier transform formula is: Wherein f is the frequency, T is the number of sampling points of the time-series audio signal; where j is the imaginary unit and j×j=-1.

[0014] By formula Calculate the frequency spectral density P of the original signal part in the frequency domain signal X(f) s (f) and the frequency spectrum density P of the morning signal part n (f).

[0015] Compute the spectrum of a frequency-domain signal A filtering operation S^(f)=H(f)×X(f) is performed on the signal spectrum to filter the noise signal part n(t) to obtain a filtered frequency domain signal S^(f).

[0016] Through the inverse Fourier transform formula The frequency domain signal S^(f) is restored from the frequency domain to the time domain to obtain the denoised time series audio signal s^(t).

[0017] The volume normalization process is specifically as follows: obtaining the time series audio signal x(t) after the noise reduction process, and adjusting the volume of the time series audio signal through normalization operation so that it fluctuates within a certain range.

[0018] The normalization formula is: Where A is the target volume, max(|x(t)|) is the maximum value in the time series audio signal after noise reduction, and y(t) is the time series audio signal obtained after normalization.

[0019] S102, audio feature extraction;

[0020] The time-series audio signal y(t) obtained by normalization operation is divided into n short-time frames y(i)=y(1), y(2), ..., y(n) at preset time intervals; wherein i=1, 2, ..., n.

[0021] Perform a short-time Fourier transform on each short-time frame y(1) to obtain the spectrum information X(i, f) in the frequency domain. The formula for short-time Fourier transform is: Where h(i-τ) is the window function and τ is the backtracking index.

[0022] As a preferred embodiment of the present invention, the spectrum is converted from a linear frequency scale to a Mel scale that is more consistent with human auditory perception through Mel frequency transformation, and the spectrum information X(i, f) of each short-time frame is weighted by a preset m Mel filter to obtain the spectrum information E on M Mel frequency scales. m ; where m = 1, 2, ..., M.

[0023] The Mel frequency conversion formula is:

[0024] The weighted calculation formula is: where |X(i,f mel )| 2 is the square of the amplitude of the frequency information X(i, f); where H m (f mel ) is the response value of the mth Mel filter.

[0025] As a preferred embodiment of the present invention, for each Mel frequency f output mel Logarithmic processing L m =log(E m ) to simulate the nonlinear response of the human ear.

[0026] Finally, the logarithmic Mel spectrum is subjected to discrete cosine transform to reduce the correlation between features and generate the final MFCC feature coefficients. The discrete cosine transform formula is: Where m is the number of the Mel filter, M is the total number of Mel filters, and c n is the nth MFCC feature coefficient.

[0027] S103, data annotation and preparation;

[0028] The speech data containing the source language and the target language are annotated manually, including the speech and the corresponding phonemes and words, to help train the acoustic model to learn the mapping relationship between phonemes.

[0029] Step 2: Build a speech recognition model;

[0030] Build a speech recognition model based on ASR automatic speech recognition technology based on deep learning.

[0031] The speech recognition model is responsible for converting the input source language speech signal into a corresponding source language text sequence.

[0032] The speech recognition model includes:

[0033] Feedforward neural network for acoustic analysis, including input layer, hidden layer and output layer.

[0034] The input parameter of the input layer is the MFCC feature coefficient c n ;

[0035] The hidden layer maps the input features to the phoneme probability space through convolution operations.

[0036] The output parameter of the output layer is to output the probability distribution of the phoneme category corresponding to each time frame i, indicating the probability of occurrence of each phoneme;

[0037] As a preferred embodiment of the present invention, the output layer takes the maximum phoneme category probability corresponding to each time frame i, matches it to a preset phoneme annotation, obtains a phoneme annotation sequence, and then generates a corresponding source language text sequence.

[0038] Step 3: Build a machine translation model;

[0039] Build a machine translation model based on natural language processing technology.

[0040] The machine translation model is responsible for translating the source language text sequence output by the speech recognition module into the target language text sequence.

[0041] The machine translation model includes: an encoder and a decoder.

[0042] The encoder processes the text sequence of the source language and converts it into a context vector of fixed length;

[0043] The encoder of the encoder contains a feed-forward neural network for encoder processing, including input layer, hidden layer and output layer.

[0044] The input parameter of the input layer is the source language text sequence;

[0045] The hidden layer maps the input source language text sequence to the phoneme probability space through convolution operations.

[0046] The output parameter of the output layer is the probability distribution of the translation results corresponding to each character in the source language text sequence, which represents the probability of occurrence of each target language character;

[0047] As a preferred embodiment of the present invention, the output layer takes the maximum translation result probability corresponding to each character in the source language text sequence, matches it to the preset target language character, obtains the target language character, and then generates the corresponding target language sequence.

[0048] The decoder generates a text sequence in the target language based on the output of the encoder.

[0049] Step 4: Joint model training;

[0050] Combine speech recognition models and machine translation models for joint optimization.

[0051] For the same source language speech data, obtain the speech recognition result output by the speech recognition model, that is, the source language text sequence Y = {yp} = {y1, y2, ..., yP}, where P is the number of words contained in the source language text sequence, and p = 1, 2, ..., P; then, input the source language text sequence into the machine translation model to obtain the target text sequence Z = {zq} = {z1, z2, ..., zQ} output by the machine translation model, where q is the number of words contained in the target text sequence. And q = 1, 2, ..., Q;

[0052] Setting speech recognition loss function where c n ∈yp represents all MFCC feature coefficients c belonging to the source language character yp n . Where θ1 is the speech model parameter vector, that is, the vector composed of weight factors and bias factors in the feedforward neural network in the speech recognition model.

[0053] Setting the machine translation model loss function Where P(z targer |zq, θ2) is the target language character zq predicted as the target character z targer The probability of . Among them, θ2 is the translation model parameter vector, that is, the vector composed of weight factors and bias factors in the feedforward neural network in the machine translation model.

[0054] As a preferred embodiment of the present invention, the joint loss function L=λ1×L ASR +λ2×L MT . λ1 and λ2 are preset hyperparameters used to adjust the accuracy training balance between speech recognition and translation tasks. Set the training goal: minimize the joint loss function, that is, minL = min(λ1×L ASR +λ2×L MT ).

[0055] During the joint training process, an optimization algorithm is used to optimize the hyperparameters λ1, λ2, the speech model parameter vector θ1, and the translation model parameter vector θ2, and the gradient of the joint loss is calculated. The model gradually adjusts the parameters and updates the parameters, so that the accuracy of both speech recognition and translation tasks can be improved.

[0056] The parameter update formula is: Among them, η1, η2, η3 and η4 are preset learning rate factors. and is the gradient of the joint loss function L with respect to λ1, λ2, θ1 and θ2.

[0057] Step 5: Speech recognition and translation application;

[0058] The preprocessed speech signal is input into the trained speech recognition module and machine translation module to carry out speech recognition and translation.

[0059] Specifically, the collected audio signal is first input into step 1 for preprocessing to obtain a number of corresponding MFCC feature coefficients.

[0060] The MFCC feature coefficients are input into the speech recognition module to obtain the corresponding speech recognition result, that is, the source language text sequence.

[0061] Subsequently, the source language text sequence is input into the machine translation module to obtain the corresponding translation result, i.e., the target language text sequence.

[0062] In a second aspect, the present invention provides a speech recognition synthesis translation device, comprising a speech recognition module and a machine translation module.

[0063] The speech recognition module is used to store the speech recognition model, and includes several computers and servers for carrying feedforward neural networks. The input source language speech signal is converted into a text sequence of the source language. This process is completed by the acoustic model and language model in deep learning technology.

[0064] The machine translation module is used to store machine translation models. It includes several computers and servers that are used to carry feedforward neural networks, encoders, and decoders. It is responsible for translating the source language text sequence output by the speech recognition module into the target language text sequence. This process is based on the encoder-decoder framework in the neural network.

[0065] In a third aspect, the present invention provides a speech recognition synthesis translation storage medium, comprising a plurality of computers and servers, for carrying a feedforward neural network, an encoder and a decoder, on which a computer program is stored.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] 1. The present invention can optimize the task relevance between speech recognition and translation during the training process by jointly training the speech recognition model and the machine translation model, and significantly improve the accuracy of the translation results. The method uses deep learning technology and optimization of the joint loss function to enable the speech recognition and translation modules to work together, thereby reducing the error accumulation when a single module is processed and improving the overall translation quality;

[0068] 2. The present invention optimizes the input data quality of subsequent speech recognition and translation processes by performing denoising and volume standardization in the speech data preprocessing stage. Denoising effectively reduces the impact of environmental noise on speech recognition, while volume standardization ensures the uniformity of speech signals, helps the system maintain a high recognition accuracy rate under different volume conditions, improves the system's generalization ability, and adapts to more practical application scenarios;

[0069] 3. The present invention can efficiently process voice input and generate corresponding translation results by adopting a speech recognition model and a machine translation model based on a feedforward neural network. Combined with modern computing hardware such as high-performance servers and computer clusters, it can effectively support the processing of large-scale voice data and ensure the efficiency of the system in real-time translation applications. In addition, the system architecture has good scalability and can be continuously optimized and upgraded according to demand to adapt to the ever-changing technological development and application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings:

[0071] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0072] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] See also Figure 1 As shown, a speech recognition synthesis translation method comprises the following steps:

[0074] Step 1: Preparation and preprocessing of training data set;

[0075] Collect and prepare speech data pairs in both the source and target languages. The speech data should be used in a variety of scenarios in daily conversations to ensure the system's practicality and generalizability.

[0076] S101, pre-processing the speech data, including noise removal and volume normalization, to improve the accuracy of subsequent speech recognition and translation.

[0077] The denoising process is as follows: obtain the time-series audio signal x(t) corresponding to each source language speech data. Convert the time-series audio signal x(t) from the time domain to the frequency domain through the fast Fourier transform FFT to obtain the frequency domain signal X(f). The fast Fourier transform formula is: Wherein f is the frequency, T is the number of sampling points of the time-series audio signal; where j is the imaginary unit and j×j=-1.

[0078] By formula Calculate the frequency spectral density P of the original signal part in the frequency domain signal X(f) s (f) and the frequency spectrum density P of the morning signal part n (f).

[0079] Compute the spectrum of a frequency-domain signal A filtering operation S^(f)=H(f)×X(f) is performed on the signal spectrum to filter the noise signal part n(t) to obtain a filtered frequency domain signal S^(f).

[0080] Through the inverse Fourier transform formula The frequency domain signal S^(f) is restored from the frequency domain to the time domain to obtain the denoised time series audio signal s^(t).

[0081] The volume normalization process is specifically as follows: obtaining the time series audio signal x(t) after noise reduction processing, and adjusting the volume of the time series audio signal through normalization operation so that it fluctuates within a certain range.

[0082] The normalization formula is: Where A is the target volume, max(|x(t)|) is the maximum value in the time series audio signal after noise reduction, and y(t) is the time series audio signal obtained after normalization.

[0083] S102, audio feature extraction;

[0084] The time-series audio signal y(t) obtained by normalization operation is divided into n short-time frames y(i)=y(1), y(2), ..., y(n) at preset time intervals; wherein i=1, 2, ..., n.

[0085] Perform a short-time Fourier transform on each short-time frame y(1) to obtain the spectrum information X(i, f) in the frequency domain. The formula for short-time Fourier transform is: Where h(i-τ) is the window function and τ is the backtracking index.

[0086] Furthermore, the spectrum is converted from a linear frequency scale to a Mel scale that is more consistent with human auditory perception through a Mel frequency transform, and the spectrum information X(i, f) of each short-time frame is weighted by a preset m Mel filter to obtain the spectrum information E on M Mel frequency scales. m ; where m = 1, 2, ..., M.

[0087] The Mel frequency transform formula is:

[0088] The weighted calculation formula is: where |X(i,f mel )| 2 is the square of the amplitude of the frequency information X(i, f); where H m (f mel ) is the response value of the mth Mel filter.

[0089] Furthermore, for each Mel frequency f output mel Logarithmic processing L m =log(E m ) to simulate the nonlinear response of the human ear.

[0090] Finally, the logarithmic Mel spectrum is subjected to discrete cosine transform to reduce the correlation between features and generate the final MFCC feature coefficients. The discrete cosine transform formula is: Where m is the number of the Mel filter, M is the total number of Mel filters, and c n is the nth MFCC feature coefficient.

[0091] S103, data annotation and preparation;

[0092] The speech data containing the source language and the target language are annotated manually, including the speech and the corresponding phonemes and words, to help train the acoustic model to learn the mapping relationship between phonemes.

[0093] Step 2: Build a speech recognition model;

[0094] Build a speech recognition model based on ASR automatic speech recognition technology based on deep learning.

[0095] The speech recognition model is responsible for converting the input source language speech signal into the corresponding source language text sequence.

[0096] The speech recognition models include:

[0097] Feedforward neural network for acoustic analysis, including input layer, hidden layer and output layer.

[0098] The input parameter of the input layer is the MFCC feature coefficient cn

[0099] The hidden layer maps the input features to the phoneme probability space through convolution operations.

[0100] The output parameter of the output layer is to output the probability distribution of the phoneme category corresponding to each time frame i, indicating the probability of occurrence of each phoneme;

[0101] Furthermore, the output layer takes the maximum phoneme category probability corresponding to each time frame i, matches it to the preset phoneme annotation, obtains the phoneme annotation sequence, and then generates the corresponding source language text sequence.

[0102] Step 3: Build a machine translation model;

[0103] Build a machine translation model based on natural language processing technology.

[0104] The machine translation model is responsible for translating the source language text sequence output by the speech recognition module into the target language text sequence.

[0105] The machine translation model includes: encoder and decoder.

[0106] The encoder processes the text sequence of the source language and converts it into a fixed-length context vector;

[0107] The encoder of the encoder contains a feed-forward neural network for encoder processing, including input layer, hidden layer and output layer.

[0108] The input parameter of the input layer is the source language text sequence;

[0109] The hidden layer maps the input source language text sequence to the phoneme probability space through convolution operations.

[0110] The output parameter of the output layer is the probability distribution of the translation results corresponding to each character in the source language text sequence, which represents the probability of occurrence of each target language character;

[0111] Furthermore, the output layer takes the maximum translation result probability corresponding to each character in the source language text sequence, matches it to the preset target language character, obtains the target language character, and then generates the corresponding target language sequence.

[0112] The decoder generates a text sequence in the target language based on the output of the encoder.

[0113] Step 4: Joint model training;

[0114] Combine speech recognition models and machine translation models for joint optimization.

[0115] For the same source language speech data, obtain the speech recognition result output by the speech recognition model, that is, the source language text sequence Y = {yp} = {y1, y2, ..., yP}, where P is the number of words contained in the source language text sequence, and p = 1, 2, ..., P; then, input the source language text sequence into the machine translation model to obtain the target text sequence Z = {zq} = {z1, z2, ..., zQ} output by the machine translation model, where q is the number of words contained in the target text sequence. And q = 1, 2, ..., Q;

[0116] Setting speech recognition loss function where c n ∈yp represents all MFCC feature coefficients c belonging to the source language character yp n . Where θ1 is the speech model parameter vector, that is, the vector composed of weight factors and bias factors in the feedforward neural network in the speech recognition model.

[0117] Setting the machine translation model loss function Where P(z targer |zq, θ2) is the target language character zq predicted as the target character z targer The probability of . Among them, θ2 is the translation model parameter vector, that is, the vector composed of weight factors and bias factors in the feedforward neural network in the machine translation model.

[0118] Furthermore, we set the joint loss function L = λ1×L ASR +λ2×L MT . λ1 and λ2 are preset hyperparameters used to adjust the accuracy training balance between speech recognition and translation tasks. Set the training goal: minimize the joint loss function, that is, minL = min(λ1×L ASR +λ2×L MT ).

[0119] During the joint training process, an optimization algorithm is used to optimize the hyperparameters λ1, λ2, the speech model parameter vector θ1, and the translation model parameter vector θ2, and the gradient of the joint loss is calculated. The model gradually adjusts the parameters and updates the parameters, so that the accuracy of both speech recognition and translation tasks can be improved.

[0120] The parameter update formula is: Among them, η1, η2, η3 and η4 are preset learning rate factors. and is the gradient of the joint loss function L with respect to λ1, λ2, θ1 and θ2.

[0121] Step 5: Speech recognition and translation application;

[0122] The preprocessed speech signal is input into the trained speech recognition module and machine translation module to carry out speech recognition and translation.

[0123] Specifically, the collected audio signal is first input into step 1 for preprocessing to obtain a number of corresponding MFCC feature coefficients.

[0124] The MFCC feature coefficients are input into the speech recognition module to obtain the corresponding speech recognition results, that is, the source language text sequence.

[0125] Subsequently, the source language text sequence is input into the machine translation module to obtain the corresponding translation result, i.e., the target language text sequence.

[0126] A speech recognition synthesis translation device comprises a speech recognition module and a machine translation module.

[0127] The speech recognition module is used to store the speech recognition model, and includes several computers and servers for carrying feedforward neural networks. The input source language speech signal is converted into a text sequence of the source language. This process is completed by the acoustic model and language model in deep learning technology.

[0128] The machine translation module is used to store machine translation models. It includes several computers and servers that are used to carry feedforward neural networks, encoders, and decoders. It is responsible for translating the source language text sequence output by the speech recognition module into the target language text sequence. This process is based on the encoder-decoder framework in the neural network.

[0129] A speech recognition synthesis translation storage medium includes several computers and servers, which are used to carry a feedforward neural network, an encoder and a decoder. A computer program is stored thereon, and the computer program is used to implement the above-mentioned speech recognition synthesis translation method.

[0130] It should be understood that the terms “include” and “comprising” used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0131] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and the claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and the claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations;

[0132] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A speech recognition synthesis translation method, characterized in that: The following steps are involved: Step 1: Preparation and preprocessing of training data set; Collect and prepare speech data pairs containing source and target languages; S101, pre-processing the voice data, including noise removal and volume normalization; S102, audio feature extraction: converting the source language speech data into MFCC feature coefficients through Mel frequency transformation; S103, data annotation and preparation; Manually annotate the speech data of the source language and the target language, including speech and corresponding phoneme and word annotations; help train the acoustic model to learn the mapping relationship between phonemes; Step 2: Build a speech recognition model; Build a speech recognition model based on ASR automatic speech recognition technology based on deep learning; The speech recognition model is responsible for converting the input source language speech signal into the corresponding source language text sequence; Step 3: Build a machine translation model; Build machine translation models based on natural language processing technology; The machine translation model is responsible for translating the source language text sequence output by the speech recognition module into the target language text sequence; Step 4: Joint model training; Combine speech recognition models and machine translation models for joint optimization; Step 5: Speech recognition and translation application; The preprocessed speech signal is input into the trained speech recognition module and machine translation module to carry out speech recognition and translation.

2. A speech recognition synthesis translation method according to claim 1, characterized in that: The specific process of preprocessing speech data is as follows: This includes noise reduction and volume normalization to improve the accuracy of subsequent speech recognition and translation; The denoising process is specifically as follows: obtaining a time-series audio signal x(t) corresponding to each source language speech data; converting the time-series audio signal x(t) from the time domain to the frequency domain by fast Fourier transform FFT to obtain a frequency domain signal X(f); the fast Fourier transform formula is: Where f is the frequency, T is the number of sampling points of the time-series audio signal; where j is the imaginary unit and j×j=-1; By formula Calculate the frequency spectral density Ps(f) of the original signal part and the frequency spectral density Pn of the morning signal part in the frequency domain signal X(f) ( f ) ; Compute the spectrum of a frequency-domain signal Perform filtering operation S^(f)=H(f)×X(f) through the signal spectrum, filter the noise signal part n(t), and obtain the filtered frequency domain signal S^(f); Through the inverse Fourier transform formula The frequency domain signal S^(f) is restored from the frequency domain to the time domain to obtain the denoised time series audio signal s^(t); The volume normalization process is specifically as follows: obtaining the time series audio signal x(t) after the noise reduction process, and adjusting the volume of the time series audio signal through normalization operation so that it fluctuates within a certain range; The normalization operation formula is: Where A is the target volume, where max(x(t)) is the maximum value in the time series audio signal after noise reduction, and where y(t) is the time series audio signal obtained after normalization operation.

3. A speech recognition synthesis translation method according to claim 1, characterized in that: The specific process of audio feature extraction is: The time-series audio signal y(t) obtained by the normalization operation is divided into n short-time frames y(i)=y(1), y(2), ..., y(n) at preset time intervals; wherein i=1, 2, ..., n; Perform a short-time Fourier transform on each short-time frame y(1) to obtain the spectrum information X(i, f) in the frequency domain; the short-time Fourier transform formula is: Where h(i-τ) is the window function; where τ is the backtracking index; The spectrum is converted from a linear frequency scale to a Mel scale that is more consistent with human auditory perception through the Mel frequency transform, and the spectrum information X(i, f) of each short-time frame is weighted by the preset m Mel filters to obtain the spectrum information E on the M Mel frequency scale. m ; Wherein m = 1, 2, ..., M; The Mel frequency conversion formula is: The weighted calculation formula is: Where X(i, fmel) 2 is the square of the amplitude of the frequency information X(i, f); where Hm ( fmel ) is the response value of the mth Mel filter; Perform logarithmic processing on each Mel frequency fmel output m =log(E m ), to simulate the nonlinear response of the human ear; Finally, the logarithmic Mel spectrum is subjected to discrete cosine transform to reduce the correlation between features and generate the final MFCC feature coefficients; the discrete cosine transform formula is: Where m is the number of the Mel filter, M is the total number of Mel filters, and cn is the nth MFCC feature coefficient.

4. The speech recognition synthesis translation method according to claim 1, characterized in that: The speech recognition model includes: A feed-forward neural network for acoustic analysis, including an input layer, a hidden layer, and an output layer; The input parameter of the input layer is the MFCC feature coefficient cn; The hidden layer maps the input features to the phoneme probability space through convolution operations; The output parameter of the output layer is to output the probability distribution of the phoneme category corresponding to each time frame i, indicating the probability of occurrence of each phoneme; The output layer takes the maximum phoneme category probability corresponding to each time frame i, matches it to the preset phoneme annotation, obtains the phoneme annotation sequence, and then generates the corresponding source language text sequence.

5. The speech recognition synthesis translation method according to claim 1, characterized in that: The machine translation model includes: Encoders and decoders; The encoder processes the text sequence of the source language and converts it into a context vector of fixed length; The encoder of the encoder contains a feed-forward neural network for encoder processing, including input layer, hidden layer and output layer; The input parameter of the input layer is the source language text sequence; The hidden layer maps the input source language text sequence to the phoneme probability space through convolution operation; The output parameter of the output layer is the probability distribution of the translation results corresponding to each character in the source language text sequence, which represents the probability of occurrence of each target language character; The output layer takes the maximum translation result probability corresponding to each character in the source language text sequence, matches it to the preset target language character, obtains the target language character, and then generates the corresponding target language sequence; The decoder generates a text sequence in the target language based on the output of the encoder.

6. A speech recognition synthesis translation method according to claim 1, characterized in that: The specific process of combining the speech recognition model and the machine translation model for joint optimization is as follows: For the same source language speech data, obtain the speech recognition result output by the speech recognition model, that is, the source language text sequence Y={yp}={y1, y2, ..., yP}, where P is the number of words contained in the source language text sequence, and p=1, 2, ..., P; Then, the source language text sequence is input into the machine translation model to obtain the target text sequence Z={zq}={z1, z2, ..., zQ} outputted by the model, where q is the number of words contained in the target text sequence; and q=1, 2, ..., Q; The speech recognition loss function, machine translation model loss function and joint loss function are set to carry out joint training. During the joint training process, the optimization algorithm is used to optimize the model hyperparameters, the gradient of the joint loss is calculated, and the model gradually adjusts the parameters to update the parameters.

7. A speech recognition synthesis translation method according to claim 6, characterized in that: The speech recognition loss function, machine translation model loss function and joint loss function are specifically: Setting speech recognition loss function Where cn∈yp represents all MFCC feature coefficients cn belonging to the source language character yp; where θ1 is the speech model parameter vector, that is, the vector composed of weight factors and bias factors in the feedforward neural network in the speech recognition model; Setting the machine translation model loss function Where P(z targer zq, θ2) is the target language character zq predicted as the target character z targer θ2 is the translation model parameter vector, that is, the vector composed of weight factors and bias factors in the feedforward neural network in the machine translation model; Set the joint loss function L = λ1×L ASR +λ2×L MT ; λ1 and λ2 are preset hyperparameters used to adjust the accuracy training balance between speech recognition and translation tasks; Set the training goal: minimize the joint loss function, that is, minL = min(λ1×L ASR +λ2×L MT ).

8. A speech recognition synthesis translation method according to claim 7, characterized in that: The specific process of using the optimization algorithm to optimize the model hyperparameters is: Calculate the gradient of the joint loss, and the model gradually adjusts the parameters to update the parameters; The parameter update formula is: Among them, η1, η2, η3 and η4 are preset learning rate factors; and is the gradient of the joint loss function L with respect to λ1, λ2, θ1 and θ2.

9. A speech recognition synthesis translation device, used to implement a speech recognition synthesis translation method according to any one of claims 1 to 8, comprising a speech recognition module and a machine translation module, characterized in that: The speech recognition module is used to store the speech recognition model, and includes several computers and servers for carrying feedforward neural networks; it converts the input source language speech signal into a text sequence of the source language; this process is completed by the acoustic model and language model in deep learning technology; The machine translation module is used to store machine translation models and includes several computers and servers equipped with feedforward neural networks, encoders and decoders. It is responsible for translating the source language text sequence output by the speech recognition module into the target language text sequence. This process is based on the encoder-decoder framework in the neural network.

10. A storage medium having a computer program stored thereon, characterized in that: The computer program is used to implement a speech recognition synthesis translation method according to any one of claims 1 to 8.

Images