Multi-channel voice conversion and synchronous transmission system applied to simultaneous interpretation

Abstract

The invention relates to the technical field of voice recognition, in particular to a multi-channel voice conversion and synchronous transmission system applied to simultaneous interpretation. Multi-language voice signals are collected through a multi-microphone array, background noise is dynamically eliminated by adopting a noise suppression technology combining spectral subtraction and deep learning, and a dynamic information source verification symbol is generated to ensure data synchronization integrity; a blind source separation technology is combined with time-frequency analysis and time delay estimation to realize multi-language signal separation and synchronization, and a time sequence is dynamically adjusted through voice activity detection; an end-to-end ASR-NMT-TTS model is constructed to realize voice real-time translation and synthesis, and low-delay transmission is carried out based on a 5G network; the real-time monitoring module is adopted to dynamically adjust the output delay and the signal-to-noise ratio, and the translation delay and the synchronization precision are optimized in combination with user feedback. According to the method, dynamic noise reduction, multi-source synchronization and a self-adaptive feedback mechanism are integrated, the problems of distortion and delay of multi-language simultaneous transmission in a complex noise environment are solved, and the obvious technical synergistic effect and practicability are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a multi-channel speech conversion and synchronous transmission system applied to simultaneous interpretation. Background Art

[0002] With the acceleration of globalization, cross-language communication has become more frequent in various international conferences, business cooperation, cultural exchanges and other occasions. Especially in the field of simultaneous interpretation, the demand for fast and accurate language conversion is increasing. However, traditional simultaneous interpretation systems face multiple challenges in practical applications, including the processing of multilingual signals, interference from background noise, voice synchronization problems and delays in real-time translation.

[0003] A Chinese invention patent application with publication number CN118762699A discloses a multi-channel speech conversion and synchronous transmission system for simultaneous interpretation, which includes a speech emotion analysis module, a collaborative pronunciation acoustic model and a designated audio channel marking module. The speech emotion analysis module is combined with the acoustic model to analyze the range state of each channel speech in the content overlap set, and the designated audio channel marking module is combined with the speech output requirements of the target receiving position to match the speech and text signals of the corresponding range state, locate the channel corresponding to the speech and text signal, perform marking processing, formulate a channel designated output signal, and send it to the speech output module. The speech output module outputs speech and text signals for the marked channel, so that in the same type of speech transmitted back, emotionally stable speech and text information are selected for live interpretation, emotionally unstable sound and text information are avoided, and the probability of live broadcast accidents is reduced.

[0004] In a complex language environment, accurate separation and synchronous transmission of multi-language signals are particularly important. Especially in the process of capturing and translating multi-channel voice signals, how to efficiently and accurately process and transform multiple signal sources is one of the key difficulties of simultaneous interpretation technology. In addition, due to the influence of background noise, echo and other factors, improving voice quality has also become a problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to solve the problems existing in the background technology and to propose a multi-channel voice conversion and synchronous transmission system for simultaneous interpretation.

[0006] The technical solution of the present invention is a multi-channel voice conversion and synchronous transmission system for simultaneous interpretation, comprising:

[0007] A microphone array is used to collect voice signals and transmit the voice signals to an environmental noise recognition unit;

[0008] The environmental noise identification unit dynamically estimates the noise power spectrum through short-time Fourier transform and minimum variance distortion-free response algorithm, combines spectral subtraction for preliminary noise reduction, and uses convolutional neural network to optimize the residual noise to obtain denoised data, construct a dynamic source verifier, and transmit it to the signal source separation and synchronization module;

[0009] The signal source separation and synchronization module verifies the data synchronization integrity through hash operation, separates the mixed signal by independent component analysis, calculates the delay difference synchronization signal in combination with the cross-correlation function, determines the language source based on the spectrum characteristics and adjusts the timestamp through voice activity detection, and outputs the synchronized voice to the multi-channel voice conversion module;

[0010] The multi-channel speech conversion module builds an end-to-end speech translation model, extracts the input speech signal through Mel-spectrogram features, uses a recurrent neural network for speech recognition to generate source language text, performs neural machine translation on the text, uses the encoding and decoding structure to extract syntactic and semantic information and outputs the target language text, and finally converts the target language text into the target speech signal;

[0011] Synchronous transmission module, based on the dedicated data transmission link of 5G network, transmits the target voice signal;

[0012] Real-time monitoring module, which dynamically schedules multi-channel voice stream timing based on real-time delay and network status, triggers channel switching through signal-to-noise ratio threshold, monitors delay data and automatically compensates for over-limit channels;

[0013] The user feedback and calibration module uses real-time feedback data to drive deep learning to optimize translation delay, noise suppression and multi-channel synchronization, and dynamically adjust audio enhancement strategies and synchronization algorithms.

[0014] Preferably, the generation process of denoised data is as follows:

[0015] S21, based on the short-time Fourier transform (STFT) noise estimation algorithm, the power spectrum of the noise is estimated

[0016] S22, subtract the spectrum of the original signal from the estimated noise spectrum to remove the noise component and obtain a clearer speech signal:

[0017]

[0018] In the formula, represents the spectrum after noise suppression; |S i (f,t)| 2 Represents the spectral power of the original signal; S i (f, t) represents the signal spectrum at frequency f and time t; represents the spectral power of the noise; δ represents the set threshold;

[0019] S23. Based on the noise suppression model, the noise characteristics are dynamically identified, the spectral power of the noise is predicted, and the speech signal spectrum is processed to obtain the spectrum|S i (f,t)| 2 ;

[0020] S24, restore the time domain signal through inverse short time Fourier transform ISTFT, the recovery process is as follows:

[0021]

[0022] In the formula, represents the denoised audio signal; ISTFT() represents the inverse short-time Fourier transform model.

[0023] Preferably, the power spectrum of the noise The estimation process is as follows:

[0024] S31, the input original signal s i (t) After short-time Fourier transform (SIFT) processing, the spectrum representation S of the signal is obtained. i (f,t):

[0025]

[0026] In the formula, S i (f, t) represents the signal spectrum at frequency f and time t; w(τ-t) represents the windowing function; T represents the window width; s i (τ) represents the original speech signal s i (t) is the amplitude at time τ; τ is the variable of time; e -j2πfτ represents the basis function of Fourier transform; e represents a natural constant; j represents an imaginary unit;

[0027] S32. Estimate the power spectrum of the noise by the minimum variance distortionless response (MVDR) method:

[0028]

[0029] In the formula, represents the estimated power spectrum of the noise; λ represents the noise smoothing parameter, which is used to adjust the smoothness of the noise estimation; E() represents the expected operation, that is, the average estimation of the signal in the past time frame.

[0030] Preferably, the noise suppression model construction process is as follows:

[0031] S41. Train a noise suppression model based on a convolutional neural network. The model input is the spectrum graph at each moment |S i (f,t)| 2, the output is the noise power spectrum estimate processed by the deep neural network

[0032] S42. Data preparation: Collect audio datasets of various environmental noises;

[0033] S43, Feature extraction: Convert the audio signal into a spectrum through STFT | S i (f,t)| 2 ;

[0034] S44, Model training: Based on the convolutional neural network (CNN) model, it is trained to extract effective noise features from the spectrum graph and predict the spectral power of the noise.

[0035] During the training process, the loss function uses the mean square error (MSE) to measure the difference between the predicted noise spectrum and the actual noise spectrum:

[0036] Where L represents the loss function; represents the noise power spectrum predicted by the model; represents the true noise power spectrum.

[0037] Preferably, the generation process of the dynamic source verification code is as follows:

[0038] S51, randomly generate a 16-bit string r, and calculate the encapsulated data data = H(S'(t))||H(S'(t)||r);

[0039] Wherein, H is a precompiled hash function; || represents a concatenation operation; S'(t) represents the binary string form of the denoised data S(t);

[0040] S52, generate dynamic source verification symbol Sign = (data) α mod n;

[0041] Where α is the preset information encoding parameter; α = I -1 mod[(p-1)(q-1)]; p and q are preset prime numbers, satisfying: p≡3(mod 4); q≡3(mod 4), p≠q; n represents a preset first-order information sharing parameter; I represents a preset second-order information sharing parameter, I={H(p||q)mod[(p-1)(q-1)]}+1;

[0042] S53. Output the dynamic source verification symbol {r, Sign}.

[0043] Preferably, the verification process of verifying data synchronization integrity through hash operation is as follows:

[0044] S61, calculate the first-order verification code C 1 =H(BS'(t))||H(BS'(t)||r);

[0045] Wherein, H is a precompiled hash function; || represents a concatenation operation; BS'(t) represents the binary string form of the received denoised data S(t);

[0046] S62, calculate the second-order verification code C 2 =Sign I mod n;

[0047] Wherein, n represents the preset first-order information sharing parameter; I represents the preset second-order information sharing parameter;

[0048] S63, if C 1 =C 2 , it means that the received denoised data S(t) has synchronous integrity; otherwise, an automatic alarm will be issued immediately.

[0049] Preferably, the end-to-end speech translation model is a deep learning model based on a sequence-to-sequence Seq2Seq architecture, including a speech recognition model ASR and a neural machine translation model NMT:

[0050] Speech recognition model: The input speech signal is X = [x 1 ,x 2 ,…,x t ,…,x T ]; Through the recurrent neural network, the input speech signal is converted into a text sequence:

[0051]

[0052] Where X represents the input speech feature sequence; Y = [y 1 ,y 2 ,…,y t ,…,y T ] represents the output text sequence; P(y t |x t ,h t-1 ) represents the predicted character y at time step t t The probability of h t-1 represents the hidden state of the previous moment; T is the length of the speech signal; x t represents the feature vector at time step t;

[0053] Neural Machine Translation Model NMT: The converted text sequence is input into the neural machine translation model for translation. The neural machine translation model NMT adopts the Transformer-based NMT architecture. The input source language text passes through the self-attention and feedforward neural network layers, and finally the translation result in the target language is obtained:

[0054]

[0055] Where, X source Represents the text in the source language; Y target Represents the text in the target language; Represents the tth word translated into the target language.

[0056] Preferably, the translation process of the end-to-end speech translation model is as follows:

[0057] S81, audio input and feature extraction: convert the synchronized speech signal into Mel spectrum features;

[0058] Among them, each speech signal X i Corresponding to a feature sequence x t ;

[0059] S82, Real-time speech recognition: feature sequence x t Input into the ASR model, after acoustic modeling and decoding, the source language text sequence Y is obtained source ;

[0060] S83, Speech to text conversion: The ASR model converts the source language speech signal into the corresponding text output through back propagation and sequence generation technology;

[0061] S84, machine translation processing: source language text Y source Input the NMT model for translation and generate text Y in the target language target ;

[0062] S85, speech synthesis output: translated text Y target The preset speech synthesis model TTS generates speech signals in the target language.

[0063] Preferably, the synchronization signal generation process is as follows:

[0064] S91. Calculate the time delay difference TDOA: Estimate the time delay difference between signals through the cross-correlation function. The calculation method of the cross-correlation function C(t) is:

[0065]

[0066] Where C(t) represents the cross-correlation value of the two signals; X1 (t) and X 2 (t) are the signals of two channels selected arbitrarily; t represents the time delay;

[0067] S92. Find the maximum correlation point: find the time point t corresponding to the maximum correlation value max , that is, t max = arg max(C(t)), and the time delay Δt between the two signal sources is obtained;

[0068] S93, synchronization signal: the delay difference between the two signals is Δt, and the voice signal is time-aligned: X' 2 (t) = X 2 (t-Δt);

[0069] Among them, X' 2 (t) represents the signal after time synchronization;

[0070] S94. Repeat steps S91 to S93 to calculate the delay difference between each pair of signals, and gradually adjust the time axis of each signal until all signals are synchronized.

[0071] Preferably, the speech synthesis model TTS is one of WaveNet and Tacotron.

[0072] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0073] The present invention designs a multi-channel voice conversion and synchronous transmission system for simultaneous interpretation. By introducing a number of technologies such as multi-channel signal processing, deep learning, noise suppression, voice recognition, and machine translation, the real-time performance, accuracy, voice quality, and stability of the simultaneous interpretation system are significantly improved, providing an efficient and reliable solution for simultaneous interpretation applications:

[0074] (1) Efficient noise suppression and clarity improvement: The adaptive noise suppression method based on spectral subtraction and deep learning can effectively remove background noise, and further optimize the noise suppression effect through deep neural networks, ensuring that the clarity of speech signals is significantly improved even in complex noise environments, thereby improving the overall quality of simultaneous interpretation.

[0075] (2) High-precision separation and synchronization of multi-channel speech signals: The introduction of blind source separation (BSS) technology and multi-channel time-frequency analysis methods can accurately separate and synchronize speech signals from different languages. The independent component analysis (ICA) and time difference estimation (TDOA) technologies are used to ensure the timing consistency between multi-language speech signals, avoiding synchronization problems caused by signal delay or channel interference, greatly improving the translation accuracy and real-time performance during simultaneous interpretation.

[0076] (3) Guarantee of real-time performance and translation fluency: A deep learning model based on a sequence-to-sequence (Seq2Seq) architecture is used, combined with speech recognition (ASR) and neural machine translation (NMT) technologies, so that speech signals can be quickly and accurately converted into target language text, and fluent target language speech is generated through text-to-speech (TTS) technology, ensuring the real-time performance and translation fluency of the simultaneous interpretation system and avoiding the time lag problem in the traditional translation process.

[0077] (4) Precise scheduling and quality monitoring of multi-channel audio streams: The 5G network dedicated data transmission link is used, combined with the precise scheduling and signal-to-noise ratio (SNR) calculation of the real-time monitoring module to ensure the real-time transmission and output quality of the target voice signal. By dynamically adjusting the delay, the timing consistency of all voice streams is guaranteed to avoid the occurrence of asynchrony problems. At the same time, real-time monitoring of the signal-to-noise ratio can ensure that the audio output achieves the best quality, improving the stability and reliability of the system.

[0078] (5) Dynamic Source Verifier: Ensures the authenticity and reliability of multi-channel voice signals. By generating and verifying source identifiers in real time, the system can identify and filter out potential forged or unreliable voice sources, avoiding translation errors or synchronization failures caused by misidentification or signal interference. The dynamic source verifier not only enhances the security and stability of the system, but also improves the accuracy of the signal source during simultaneous interpretation, making voice processing and synchronous transmission more accurate and reliable, thereby further improving translation quality and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a system architecture diagram of a multi-channel voice conversion and synchronous transmission system for simultaneous interpretation proposed by the present invention. DETAILED DESCRIPTION

[0080] Embodiment 1, as Figure 1 As shown, the present invention proposes a multi-channel voice conversion and synchronous transmission system for simultaneous interpretation, including: a voice acquisition module, a signal source separation and synchronization module, a multi-channel voice conversion module, a real-time monitoring module, a synchronous transmission module and a user feedback and calibration module.

[0081] The voice acquisition module is used to collect and process audio data. The voice acquisition module includes several microphone arrays and an environmental noise recognition unit;

[0082] The microphone array is used to collect speech signals of different languages ​​in multiple channels;

[0083] The environmental noise recognition unit automatically adjusts and pre-processes the collected sound source;

[0084] The signal source separation and synchronization module extracts speech signals of different languages ​​from multiple channels and ensures that the signals are synchronized in time for subsequent translation and transmission;

[0085] The multi-channel speech conversion module uses deep learning speech synthesis technology to convert the collected source language into the target language, supports multi-language translation, and is optimized according to the real-time requirements of translation;

[0086] The synchronous transmission module ensures real-time transmission of voice between multiple channels while reducing delay and packet loss;

[0087] The real-time monitoring module is responsible for monitoring the quality and synchronization of voice signals in all channels to ensure that the output voice signals can be accurately conveyed to the audience;

[0088] The User Feedback and Calibration module collects feedback from listeners and interpreters and automatically adjusts system parameters such as volume and speech delay to optimize speech quality and transmission synchronization.

[0089] Embodiment 2, a multi-channel voice conversion and synchronous transmission method for simultaneous interpretation proposed by the present invention is applied to a multi-channel voice conversion and synchronous transmission system for simultaneous interpretation proposed in Embodiment 1, and its specific implementation steps are as follows:

[0090] S1. Speech signals of different languages ​​are collected through multiple high-sensitivity microphone arrays. Each microphone array is deployed in the interpreter and audience areas to capture audio information of different source languages. The speech collection module obtains the speech signals collected by each microphone array and processes each audio signal independently. The environmental noise recognition algorithm is used to monitor and suppress background noise in real time. Specifically:

[0091] S11. Assume that the system contains N microphone arrays, and the speech signal collected by each microphone array i is represented as s i (t), and the output signal of each microphone array is: s i (t) = x i (t)+n i (t);

[0092] where i∈{1,2,…,N}; x i (t) represents the target speech signal captured by microphone array i; n i (t) represents background noise or interference signal;

[0093] S12, Environmental noise identification unit designs an adaptive noise suppression method based on spectral subtraction and deep learning. It combines the traditional spectral subtraction and deep neural network noise suppression methods to improve the noise suppression effect. It accurately identifies the noise components in different environments through adaptive estimation of the noise spectrum and dynamically adjusts the noise suppression strategy according to environmental changes. Specifically:

[0094] S1201, estimating the noise component in the input signal. In a simultaneous interpretation environment, the noise signal changes dynamically. A noise estimation algorithm based on short-time Fourier transform (STFT) is used to convert the time domain signal into a frequency domain signal in order to better separate the speech signal from the noise signal. Specifically:

[0095] (1) Input original signal s i (t) After being processed by short-time Fourier transform (SIFT), the spectrum representation S of the signal is obtained i (f,t):

[0096]

[0097] In the formula, S i (f, t) represents the signal spectrum at frequency f and time t; w(τ-t) represents the windowing function. In this embodiment, a Hamming window or a Gaussian window is used to ensure that the time domain truncation of the signal minimizes the impact on the spectrum; T represents the window width, which is set to 10ms to 40ms in this embodiment; s i (τ) represents the original speech signal s i (t) is the amplitude at time τ; τ is the variable of time; e -j2πfτ represents the basis function of Fourier transform, which is used to map the signal to the frequency domain. The frequency f is the coordinate mapped to the frequency domain. e represents a natural constant. j represents an imaginary unit.

[0098] (2) Assume that the noise component N i (f, t) is a stationary process, and the power spectrum of the noise is estimated by the minimum variance distortionless response (MVDR) method:

[0099]

[0100] In the formula, represents the estimated power spectrum of the noise; λ represents the noise smoothing parameter, which is used to adjust the smoothness of the noise estimation; E() represents the expected operation, that is, the average estimation of the signal in the past period of time;

[0101] S1202. Once the power spectrum of the noise The noise component is removed by subtracting the spectrum of the original signal from the estimated noise spectrum to obtain a clearer speech signal. The spectrum of the output signal of the spectral subtraction method is expressed as:

[0102]

[0103] In the formula, represents the spectrum after noise suppression; |S i (f,t)| 2 Represents the spectral power of the original signal; Indicates the spectral power of the noise; δ indicates the threshold in the spectral subtraction method, which prevents excessive suppression of noise from causing distortion of the speech signal. In this embodiment, the value is 0.1 to 0.5;

[0104] S1203. In order to further improve the noise suppression effect, a deep neural network (DNN) is combined to enhance the performance of noise suppression. In a complex noise environment, noise features are dynamically identified and fine-tuned in the frequency domain. Specifically:

[0105] (1) Train a noise suppression model based on a convolutional neural network (CNN), whose input is the spectrum graph |S at each moment i (f,t)| 2 , the output is the noise power spectrum estimate processed by the deep neural network

[0106] (2) Data preparation: Collect audio datasets of various environmental noises, including but not limited to office noise and fan noise;

[0107] (3) Feature extraction: The audio signal is converted into a spectrum graph through STFT. i (f,t)| 2 ;

[0108] (4) Model training: Training based on the convolutional neural network (CNN) model enables it to extract effective noise features from the spectrum graph and predict the spectral power of the noise.

[0109] During the training process, the loss function uses the mean square error (MSE) to measure the difference between the predicted noise spectrum and the actual noise spectrum:

[0110] Where L represents the loss function; represents the noise power spectrum predicted by the model; represents the true noise power spectrum;

[0111] After the training is completed, according to the input spectrum graph |S i (f,t)|2 To predict a more accurate noise power spectrum, so as to achieve more efficient denoising in the noise suppression process;

[0112] S1204, through the spectrum diagram processed by spectral subtraction and deep learning model, we can get a clear speech signal spectrum | S i (f,t)| 2 , and restore the time domain signal through the inverse short-time Fourier transform (ISTFT). The restoration process is as follows:

[0113]

[0114] In the formula, Represents the denoised audio signal; ISTFT() represents the inverse short-time Fourier transform model;

[0115] It should be noted that spectral subtraction, as a preliminary processing step of noise suppression, first performs simple noise removal on the signal to effectively reduce most of the background noise. At this time, the noise spectrum estimation is relatively rough and the signal quality has improved, but there may still be slight distortion or some incompletely removed noise components. Deep learning-enhanced noise suppression is a further optimization process based on spectral subtraction. The deep learning model can further eliminate residual noise and enhance the clarity of speech signals based on more complex features. The combination of the two in the simultaneous interpretation system can maintain the efficiency of denoising and ensure the improvement of speech quality in complex environments.

[0116] S14. Summarize the denoised data Generate a dynamic source verifier for the denoised data S(t). The generation process of the dynamic source verifier is as follows:

[0117] S1401, randomly generate a 16-bit string r, and calculate the encapsulated data data = H(S'(t))||H(S'(t)||r);

[0118] Wherein, H is a precompiled hash function; || represents a concatenation operation; S'(t) represents the binary string form of the denoised data S(t);

[0119] S1402, generate dynamic source verification code Sign = (data) α mod n;

[0120] Among them, α is the preset information encoding parameter; n represents the first-order information sharing parameter;

[0121] S1403, output dynamic source verification symbol {r, Sign};

[0122] It should be noted that the generation process of the preset information coding parameter α is as follows:

[0123] (1) Select two prime numbers p and q: p≡3(mod 4); q≡3(mod 4), p≠q;

[0124] (2) Calculate the first-order information sharing parameter n = p × q;

[0125] (3) Calculate the second-order information sharing parameter I = {H(p||q)mod[(p-1)(q-1)]}+1;

[0126] (4) Based on this, the preset information coding parameter α=I is calculated -1 mod[(p-1)(q-1)];

[0127] S15, transmitting the {dynamic source verification symbol {r, Sign}, the denoised data S(t)} to the signal source separation and synchronization module.

[0128] S2, signal source separation and synchronization module proposes a method that combines blind source separation (BSS) technology and multi-channel time-frequency analysis to effectively separate and synchronize multilingual speech signals in complex environments. Specifically:

[0129] S21, receiving {dynamic source verification symbol {r, Sign}, denoised data S(t)}, extracting the dynamic source verification symbol {r, Sign} and the denoised data S(t), and verifying the synchronization integrity of the received denoised data S(t). The verification process is as follows:

[0130] S2101, calculate the first-order verification code C 1 =H(BS'(t))||H(BS'(t)||r);

[0131] Wherein, H is a precompiled hash function; || represents a concatenation operation; BS'(t) represents the binary string form of the received denoised data S(t);

[0132] S2102, calculate the second-order verification code C 2 =Sign I mod n;

[0133] S2103, if C 1 =C 2 , it means that the received denoised data S(t) has synchronous integrity; otherwise, an automatic alarm is immediately issued;

[0134] S22. Assume that after denoising, a clear signal X of M channels is obtained 1 (t), X 2 (t),…,X M(t), the signal received by each channel is a linear combination of multiple speech sources, that is: X(t) = A·S(t);

[0135] Wherein, S(t) represents an independent source signal vector (including but not limited to speech signals of different languages); A represents a mixing matrix, that is, a mixing relationship between channels; X(t) represents the signal vector of the received M channels;

[0136] In order to separate the independent source signal S(t) from the received mixed signal X(t), the independent component analysis (ICA) method is used to solve the separation matrix W by maximizing the non-Gaussianity between the signals, thereby obtaining the source signal: S(t) = W·X(t);

[0137] Where W represents the source separation matrix;

[0138] S23, using the time difference (TDOA) estimation technology to synchronize the signals by calculating the time delay between different speech sources, specifically:

[0139] S2301, calculate the delay difference TDOA: Assume there are two receiving channels X 1 (t) and X 2 (t), corresponding to the received signals of the two speech signal sources, due to the different propagation times of the sound waves, there will be a time delay in the signals received by the two channels. The delay difference between the signals is estimated by the cross-correlation function. The calculation method of the cross-correlation function C(t) is:

[0140]

[0141] Where C(t) represents the cross-correlation value of the two signals; X 1 (t) and X 2 (t) is the signal of two channels; t represents the time delay;

[0142] S2302, find the maximum correlation point: after calculating the cross-correlation function C(t), find the time point t corresponding to the maximum correlation value max , that is, t max = arg max(C(t));

[0143] Based on this, the time delay Δt between the two signal sources is obtained;

[0144] S2303, synchronization signal: The delay difference between the two signals is Δt, and one of the signals is time-aligned in the following manner: X' 2 (t) = X 2 (t-Δt);

[0145] Among them, X' 2(t) represents the signal after time synchronization;

[0146] S2304, calculating the delay difference between each pair of signals accordingly, and gradually adjusting the time axis of each signal until all signals reach a synchronized state;

[0147] S24. After the multi-language signal source separation and synchronization analysis are completed, multiple synchronized voice source outputs will be generated. In order to ensure the smoothness of simultaneous interpretation, the final signal optimization is performed before signal transmission, specifically:

[0148] S2401, separating language sources: after signal separation, by analyzing the spectrum of the signal, based on the spectrum characteristics of the speech, determining the language corresponding to each signal, and extracting speech data of each language by processing and analyzing the signals of different languages;

[0149] S2402, dynamic synchronization adjustment based on voice activity detection (VAD): Taking into account the differences in duration and speech speed between languages ​​during simultaneous interpretation, voice activity detection (VAD) determines the start and end time of the voice signal, specifically:

[0150] (1) Voice Activity Detection (VAD): Determines whether there is voice activity by detecting the energy changes of the voice signal:

[0151]

[0152] Where E(t) represents the signal energy at time t; |S(f,t)| 2 Represents the spectral intensity of the signal at frequency f and time t;

[0153] If the signal energy E(t) exceeds the set threshold, it is considered that there is voice activity at this time, and a timestamp is assigned to the signal;

[0154] (2) Dynamic synchronization adjustment: Combined with the speech activity timestamp detected by VAD, dynamic time synchronization is performed according to the actual speech activity time to ensure that speech signals of different languages ​​remain synchronized during simultaneous interpretation;

[0155] S25. After signal separation and time synchronization, a plurality of synchronized voice signals are obtained, and the plurality of synchronized voice signals are transmitted to a multi-channel voice conversion module.

[0156] S3, the multi-channel speech conversion module uses a deep learning method that combines end-to-end neural machine translation (NMT) and speech recognition (ASR) to extract speech information from multiple languages ​​and convert it into the target language in real time. The specific implementation process is as follows:

[0157] S31. Build a deep learning model based on the sequence-to-sequence (Seq2Seq) architecture. The architecture includes a speech recognition model (ASR) and a neural machine translation model (NMT). The model is optimized through joint training. The input of the speech signal is firstly subjected to feature extraction and acoustic modeling, then converted into text through speech recognition, and then language conversion is performed through the machine translation model. The specific structure is as follows:

[0158] S3101, speech recognition model (ASR): Assume that the input speech signal is X = [x 1 ,x 2 ,…,x t ,…,x T ]; where T is the length of the speech signal; x t Represents the feature vector at time step t; through the recurrent neural network, the input speech signal is converted into a text sequence:

[0159]

[0160] Where X represents the input speech feature sequence; Y = [y 1 ,y 2 ,…,y t ,…,y T ] represents the output text sequence; P(y t |x t ,h t-1 ) represents the predicted character y at time step t t The probability of h t-1 Represents the hidden state of the previous moment;

[0161] S3102, Neural Machine Translation Model (NMT): Once the speech signal is converted into text in the target language, it is input into the Neural Machine Translation Model (NMT) for translation. The Neural Machine Translation Model NMT uses the Transformer-based NMT architecture. The input source language text passes through the self-attention and feed-forward neural network layers, and finally the translation result in the target language is obtained:

[0162]

[0163] Where, X source Represents the text in the source language; Y target Represents the text in the target language; represents the tth word translated into the target language;

[0164] S32. In order to ensure the fluency and real-time performance of the entire translation process, a speech translation model is constructed, combining ASR and NMT. ​​The construction process is as follows:

[0165] S3201, audio input and feature extraction: Get the synchronized speech signal and convert it into Mel spectrum feature through the feature extraction module. i Corresponding to a feature sequence x t , and use it as the input of the ASR model;

[0166] S3202, Real-time speech recognition: feature sequence x t Input into the ASR model, after acoustic modeling and decoding, the source language text sequence Y is obtained source ;

[0167] S3203, speech to text conversion: The ASR model converts the source language speech signal into the corresponding text output through back propagation and sequence generation technology;

[0168] S3204, Machine Translation Processing: Source Language Text Y source It is sent to the NMT model for translation to generate text Y in the target language target , the NMT model uses the self-attention mechanism and encoding-decoding process to extract information from the syntax and semantics of the source language and generate fluent translation text that conforms to the grammar of the target language;

[0169] S3205, speech synthesis output: translated text Y target The speech synthesis model TTS generates speech signals in the target language, and the text is converted into natural speech output;

[0170] The speech synthesis model TTS in this embodiment is one of WaveNet and Tacotron;

[0171] S33, outputting the target speech;

[0172] It should be noted that WaveNet is a generative waveform model proposed by DeepMind. It directly generates high-fidelity audio samples through stacked dilated causal convolutions and can capture long-term dependencies. Tacotron is an end-to-end sequence-to-sequence model developed by Google. It directly maps text into Mel-spectrograms through an encoder-attention-decoder framework, and then generates speech in combination with a vocoder.

[0173] S4. The translated voice signal is transmitted in real time through the synchronous transmission module, using a dedicated data transmission link based on the 5G network to ensure the real-time transmission of the voice stream.

[0174] S5, the real-time monitoring module outputs these target speech signals in an effective multi-channel manner, ensures the output quality through real-time monitoring, adjusts the translation and output results in a timely manner, ensures that the translated target speech is output synchronously through a multi-channel manner, and performs real-time audio quality monitoring at the same time, specifically:

[0175] S51. Precisely schedule each voice stream according to the latency of each audio stream, the network condition, and the capabilities of the output device. During the scheduling process, ensure the temporal consistency of the audio streams, that is, ensure that the output signals of each channel are as synchronized as possible to avoid out-of-sync problems caused by latency. The modeling process is as follows:

[0176]

[0177] In the formula, represents the target language voice signal of the final output; represents the target language voice signal after translation; Δt i represents the latency adjustment amount, that is, the temporal difference between the voice stream of each channel and the voice streams of other channels;

[0178] Accordingly: Adjust the latency of each signal based on real-time network and device information to ensure that all voice streams are output at the same moment;

[0179] S52. For each voice signal Calculate the signal-to-noise ratio:

[0180]

[0181] In the formula, represents the target voice signal at time t; represents the estimation of the noise part in the target voice signal;

[0182] Accordingly: The larger the SNR value, the better the quality of the output voice signal. By calculating the SNR value in real time, determine whether the voice signal has reached the expected quality. If the SNR value is lower than the set threshold, automatically adjust the output channel;

[0183] S53. Monitor the latency of each channel in real time and make adjustments when excessive latency is detected. The latency monitoring can be modeled by the following formula:

[0184] Δt max = max(Δt 1 , Δt 2 ,..., Δt i ,..., Δt N );

[0185] Among them, Δt i represents the latency of each channel; Δt max represents the maximum latency, which is used to determine whether the output order needs to be adjusted;

[0186] Accordingly: If Δt maxWhen the preset threshold is exceeded, the delay adjustment mechanism is automatically started to ensure that the voice signals of each channel are output at the same time by adjusting the output time point or compensating for the channel with larger delay.

[0187] S6, User Feedback and Calibration Module collects real-time feedback from listeners and interpreters and automatically adjusts various system parameters, including but not limited to voice quality, translation delay, and channel allocation;

[0188] Through deep learning and user behavior analysis, we can intelligently learn and optimize the following aspects:

[0189] (1) Automatically adjust translation delay: Dynamically adjust the delay of the translation process based on user feedback to achieve optimal real-time performance;

[0190] (2) Voice quality improvement: Analyze noise in different environments and automatically adjust audio enhancement strategies;

[0191] (3) Synchronization optimization: By using the feedback of time error data, the multi-channel synchronization processing algorithm is adjusted to ensure more accurate language alignment.

[0192] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A multi-channel voice conversion and synchronous transmission system for simultaneous interpretation, characterized in that: include: A microphone array is used to collect voice signals and transmit the voice signals to an environmental noise recognition unit; The environmental noise identification unit dynamically estimates the noise power spectrum through short-time Fourier transform and minimum variance distortion-free response algorithm, combines spectral subtraction for preliminary noise reduction, and uses convolutional neural network to optimize the residual noise to obtain denoised data, construct a dynamic source verifier, and transmit it to the signal source separation and synchronization module; The signal source separation and synchronization module verifies the data synchronization integrity through hash operation, separates the mixed signal by independent component analysis, calculates the delay difference synchronization signal in combination with the cross-correlation function, determines the language source based on the spectrum characteristics and adjusts the timestamp through voice activity detection, and outputs the synchronized voice to the multi-channel voice conversion module; The multi-channel speech conversion module builds an end-to-end speech translation model, extracts the input speech signal through Mel-spectrogram features, uses a recurrent neural network for speech recognition to generate source language text, performs neural machine translation on the text, uses the encoding and decoding structure to extract syntactic and semantic information and outputs the target language text, and finally converts the target language text into the target speech signal; Synchronous transmission module, based on the dedicated data transmission link of 5G network, transmits the target voice signal; Real-time monitoring module, which dynamically schedules multi-channel voice stream timing based on real-time delay and network status, triggers channel switching through signal-to-noise ratio threshold, monitors delay data and automatically compensates for over-limit channels; The user feedback and calibration module uses real-time feedback data to drive deep learning to optimize translation delay, noise suppression and multi-channel synchronization, and dynamically adjust audio enhancement strategies and synchronization algorithms.

2. The multi-channel voice conversion and synchronous transmission system for simultaneous interpretation according to claim 1, characterized in that: The generation process of denoised data is as follows: S21, based on the short-time Fourier transform (STFT) noise estimation algorithm, the power spectrum of the noise is estimated S22, subtract the spectrum of the original signal from the estimated noise spectrum to remove the noise component and obtain a clearer speech signal: In the formula, represents the spectrum after noise suppression; |S i (f,t)| 2 Represents the spectral power of the original signal; S i (f, t) represents the signal spectrum at frequency f and time t; represents the spectral power of the noise; δ represents the set threshold; S23. Based on the noise suppression model, the noise characteristics are dynamically identified, the spectral power of the noise is predicted, and the speech signal spectrum is processed to obtain the spectrum|S i (f,t)| 2 ; S24, restore the time domain signal through inverse short time Fourier transform ISTFT, the recovery process is as follows: In the formula, represents the denoised audio signal; ISTFT() represents the inverse short-time Fourier transform model.

3. The multi-channel voice conversion and synchronous transmission system for simultaneous interpretation according to claim 2, characterized in that: Power spectrum of noise The estimation process is as follows: S31, the input original signal s i (t) After short-time Fourier transform (SIFT) processing, the spectrum representation S of the signal is obtained. i (f,t): In the formula, S i (f, t) represents the signal spectrum at frequency f and time t; w(τ-t) represents the windowing function; T represents the window width; s i (τ) represents the original speech signal s i (t) is the amplitude at time τ; τ is the variable of time; e -j2πfτ represents the basis function of Fourier transform; e represents a natural constant; j represents an imaginary unit; S32. Estimate the power spectrum of the noise by the minimum variance distortionless response (MVDR) method: In the formula, represents the estimated power spectrum of the noise; λ represents the noise smoothing parameter, which is used to adjust the smoothness of the noise estimation; E() represents the expected operation, that is, the average estimation of the signal in the past time frame.

4. The multi-channel voice conversion and synchronous transmission system for simultaneous interpretation according to claim 2, characterized in that: The noise suppression model construction process is as follows: S41. Train a noise suppression model based on a convolutional neural network. The model input is the spectrum graph at each moment |S i (f,t)| 2 , the output is the noise power spectrum estimate processed by the deep neural network S42. Data preparation: Collect audio datasets of various environmental noises; S43, Feature extraction: Convert the audio signal into a spectrum through STFT | S i (f,t)| 2 ; S44, Model training: Based on the convolutional neural network (CNN) model, it is trained to extract effective noise features from the spectrum graph and predict the spectral power of the noise. During the training process, the loss function uses the mean square error (MSE) to measure the difference between the predicted noise spectrum and the actual noise spectrum: Where L represents the loss function; represents the noise power spectrum predicted by the model; represents the true noise power spectrum.

5. The multi-channel voice conversion and synchronous transmission system for simultaneous interpretation according to claim 1, characterized in that: The generation process of the dynamic source authenticator is as follows: S51, randomly generate a 16-bit string r, and calculate the encapsulated data data = H(S'(t))||H(S'(t)||r); Wherein, H is a precompiled hash function; || represents a concatenation operation; S'(t) represents the binary string form of the denoised data S(t); S52, generate dynamic source verification symbol Sign = (data) α mod n; Where α is the preset information encoding parameter; α = I -1 mod[(p-1)(q-1)]; p and q are preset prime numbers, satisfying: p≡3(mod 4); q≡3(mod 4), p≠q; n represents a preset first-order information sharing parameter; I represents a preset second-order information sharing parameter, I={H(p||q)mod[(p-1)(q-1)]}+1; S53. Output the dynamic source verification symbol {r, Sign}.

6. The multi-channel voice conversion and synchronous transmission system for simultaneous interpretation according to claim 5, characterized in that: The verification process of verifying data synchronization integrity through hashing is as follows: S61, calculate the first-order verification code C1=H(BS'(t))||H(BS'(t)||r); Wherein, H is a precompiled hash function; || represents a concatenation operation; BS'(t) represents the binary string form of the received denoised data S(t); S62, calculate the second-level verification code C2 = Sign I mod n; Wherein, n represents the preset first-order information sharing parameter; I represents the preset second-order information sharing parameter; S63. If C1=C2, it means that the received denoised data S(t) has synchronous integrity; otherwise, an automatic alarm is immediately issued.

7. The multi-channel voice conversion and synchronous transmission system for simultaneous interpretation according to claim 1, characterized in that: The end-to-end speech translation model is a deep learning model based on the sequence-to-sequence Seq2Seq architecture, which includes a speech recognition model ASR and a neural machine translation model NMT: Speech recognition model: The input speech signal is X = [x1, x2, ..., x t ,…,x T ]; Through the recurrent neural network, the input speech signal is converted into a text sequence: Where X represents the input speech feature sequence; Y = [y1, y2, …, y t ,…,y T ] represents the output text sequence; P(y t |x t ,h t-1 ) represents the predicted character y at time step t t The probability of h t-1 represents the hidden state of the previous moment; T is the length of the speech signal; x t represents the feature vector at time step t; Neural Machine Translation Model NMT: The converted text sequence is input into the neural machine translation model for translation. The neural machine translation model NMT adopts the Transformer-based NMT architecture. The input source language text passes through the self-attention and feedforward neural network layers, and finally the translation result in the target language is obtained: Where, X source Represents the text in the source language; Y target Represents the text in the target language; Represents the tth word translated into the target language.

8. The multi-channel voice conversion and synchronous transmission system for simultaneous interpretation according to claim 7, characterized in that: The translation process of the end-to-end speech translation model is as follows: S81, audio input and feature extraction: convert the synchronized speech signal into Mel spectrum features; Among them, each speech signal X i Corresponding to a feature sequence x t ; S82, Real-time speech recognition: feature sequence x t Input into the ASR model, after acoustic modeling and decoding, the source language text sequence Y is obtained source ; S83, speech to text conversion: The ASR model converts the source language speech signal into the corresponding text output through back propagation and sequence generation technology; S84, machine translation processing: source language text Y source Input the NMT model for translation and generate text Y in the target language target ; S85, speech synthesis output: translated text Y target The preset speech synthesis model TTS generates speech signals in the target language.

9. The multi-channel voice conversion and synchronous transmission system for simultaneous interpretation according to claim 1, characterized in that: The synchronization signal generation process is as follows: S91. Calculate the time delay difference TDOA: Estimate the time delay difference between signals through the cross-correlation function. The calculation method of the cross-correlation function C(t) is: In the formula, C(t) represents the cross-correlation value of two signals; X1(t) and X2(t) are the signals of two channels selected arbitrarily; t represents the time delay; S92. Find the maximum correlation point: find the time point t corresponding to the maximum correlation value max , that is, t max = arg max(C(t)), and the time delay Δt between the two signal sources is obtained; S93, synchronization signal: the delay difference between the two signals is Δt, and the voice signal is time-aligned: X'2(t)=X2(t-Δt); Among them, X'2(t) represents the signal after time synchronization; S94. Repeat steps S91 to S93 to calculate the delay difference between each pair of signals, and gradually adjust the time axis of each signal until all signals are synchronized.

10. The multi-channel voice conversion and synchronous transmission system for simultaneous interpretation according to claim 8, characterized in that: The speech synthesis model TTS is one of WaveNet and Tacotron.

Images