Tablet computer real-time voice recognition and translation system based on side cloud collaboration

Abstract

The invention discloses a tablet computer real-time speech recognition and translation system based on side cloud collaboration, which relates to the technical field of speech processing, and comprises a speech noise reduction module for performing multi-stage enhancement by adopting beam forming and a deep residual network, performing multi-channel feature fusion in combination with adaptive noise estimation and an attention mechanism, and performing speech recognition and translation. Obtaining clean voice data after signal-to-noise ratio optimization; the language recognition module inputs the clean voice data into a language recognition network, extracts a voice feature vector, performs language recognition through a language clustering model and generates a language tag; and the translation module is used for carrying out semantic optimization through semantic understanding and context modeling according to the preliminary recognition text, and carrying out translation processing by utilizing a neural network translation model to generate a translated text. According to the invention, the technical effect of effectively improving the signal-to-noise ratio of the voice signal in a complex acoustic environment is realized.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a tablet computer real-time speech recognition and translation system based on edge-cloud collaboration. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, speech recognition and machine translation have been widely used in scenarios such as cross-language communication, intelligent assistants, and human-computer interaction, becoming important research areas in natural language processing. In recent years, deep learning-based end-to-end speech recognition networks and neural network translation models have made significant progress, significantly improving the efficiency and accuracy of converting speech signals to text representations and cross-language text conversion. With the increasing popularity of mobile devices such as smartphones and tablets, users are increasingly demanding real-time multilingual speech recognition and translation. Traditional cloud-based speech processing systems typically rely on uploading raw speech data to remote servers for computation and translation. While this centralized architecture offers powerful computing capabilities, it is susceptible to network bandwidth, latency, and privacy issues, making it difficult to meet real-time and robustness requirements. To improve response speed and task processing efficiency, edge computing technology has been gradually applied to speech processing in recent years, enabling some computational tasks to be performed locally, thus realizing an edge-cloud collaborative processing architecture.

[0003] While edge-cloud collaborative architectures show promising development prospects in speech recognition and translation scenarios, existing technologies still face numerous limitations. Chief among these is the difficulty of achieving high-accuracy speech recognition and translation results in multilingual mobile terminals due to hardware resources, algorithm adaptability, and inter-module collaboration efficiency. Furthermore, they cannot guarantee real-time response performance in complex acoustic environments. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a tablet computer real-time speech recognition and translation system based on edge-cloud collaboration to solve the speech recognition problem in multilingual scenarios in the prior art.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] The present invention provides a real-time speech recognition and translation system for a tablet computer based on edge-cloud collaboration, which includes a speech acquisition module, which acquires speech signals through multiple microphone arrays of a tablet computer and obtains speech signal direction information using a TDOA algorithm; a speech noise reduction module, which uses beamforming and a deep residual network for multi-level enhancement, combines adaptive noise estimation with an attention mechanism for multi-channel feature fusion, and obtains clean speech data with optimized signal-to-noise ratio; a language recognition module, which inputs the clean speech data into a language recognition network, extracts speech feature vectors, and performs language recognition and generates a language label through a language clustering model; a text conversion module, which matches a speech recognition network corresponding to the current language label from a language category library according to the language label, and performs speech-to-text processing on the clean speech data through the speech recognition network to generate preliminary recognition text; and a translation module, which performs semantic optimization based on the preliminary recognition text through semantic understanding and context modeling, and performs translation processing using a neural network translation model to generate translated text.

[0008] As a preferred solution of the edge-cloud collaboration-based tablet computer real-time speech recognition and translation system of the present invention, the specific steps of obtaining the direction information of the speech signal are as follows:

[0009] The tablet computer's multiple microphone arrays simultaneously collect voice signals in the environment, remove noise from the collected voice signals using spectral subtraction, and adjust the amplitude of the voice signals using maximum normalization.

[0010] Using the GCC-PHAT algorithm, the time difference of the preprocessed speech signal is calculated to obtain the time difference between the speech signal reaching different microphones;

[0011] Based on the spatial arrangement of multiple microphone arrays and the calculated time difference, the TDOA algorithm is used to determine the propagation direction of the signal and obtain the direction information of the voice signal.

[0012] As a preferred solution of the edge-cloud collaboration-based tablet computer real-time speech recognition and translation system of the present invention, the steps of obtaining clean speech data with optimized signal-to-noise ratio are as follows:

[0013] Using beamforming, the speech signal is weighted and phase-adjusted, and noise interference from other directions is suppressed to enhance the speech signal.

[0014] The enhanced speech signal is input into the deep residual denoising network deployed on the edge computing platform. The nonlinear feature extraction and noise separation mechanism are used to obtain the initial denoised speech signal.

[0015] The preliminarily denoised speech signal is combined with the current ambient sound characteristics to perform an adaptive noise estimation process. By dynamically modeling and suppressing the residual noise components, a speech signal with significantly reduced noise interference is obtained.

[0016] The speech signal with significantly reduced noise interference is jointly processed with the speech and environmental features extracted by multiple microphone channels. The weighted fusion of multimodal features is completed through the fusion attention mechanism to generate clean speech data with optimized signal-to-noise ratio.

[0017] As a preferred solution of the edge-cloud collaboration-based tablet computer real-time speech recognition and translation system of the present invention, the specific steps of generating language tags are as follows:

[0018] Perform pre-emphasis, frame division, and windowing on clean speech data to segment the continuous speech stream into temporally localized speech signal frames.

[0019] Perform a fast Fourier transform on each frame of speech signal to obtain spectral information, map the spectrum to a Mel-scale filter bank, calculate the linear energy vector distribution, and extract the Mel-frequency cepstral coefficients and logarithmic filter bank energy to form speech features;

[0020] The speech features are input into the language recognition neural network, embedded into the low-dimensional space and then input into the language clustering model to generate language labels.

[0021] As a preferred solution of the edge-cloud collaboration-based tablet computer real-time speech recognition and translation system described in the present invention, the steps of extracting Mel-frequency cepstral coefficients and logarithmic filter bank energy are as follows:

[0022] Perform natural logarithm operation on the linear energy calculated by Mel filter bank in each frame to form the logarithmic Mel frequency energy feature after nonlinear compression, enhance the resolution of low energy, and obtain the logarithmic filter bank energy;

[0023] The energy of the logarithmic filter bank of each frame is subjected to discrete cosine transform, the frequency domain energy information is mapped to the cepstral domain, and the first N coefficients are intercepted according to the energy concentration to obtain the Mel-frequency cepstral coefficients.

[0024] As a preferred solution of the edge-cloud collaboration-based tablet computer real-time speech recognition and translation system of the present invention, wherein: according to the language label, the speech recognition network corresponding to the current language label is matched from the language category library, and the specific steps are as follows:

[0025] Inputting the language label into the language category library and retrieving the speech recognition network identification information corresponding to the language label;

[0026] According to the speech recognition network identification information, the matching speech recognition network parameters and structure configuration are called and loaded from the preset speech recognition network.

[0027] As a preferred solution of the edge-cloud collaboration-based tablet computer real-time speech recognition and translation system of the present invention, wherein: the speech-to-text processing of clean speech data is performed through the speech recognition network to generate preliminary recognition text. The specific steps are as follows:

[0028] The clean speech data is passed into the speech recognition network to extract acoustic features and recognize the audio pattern of the speech based on the acoustic features;

[0029] The speech recognition network decodes the speech data based on the extracted acoustic features and preset language rules, converts the acoustic features into text format, and performs spelling correction and grammatical analysis to generate preliminary recognized text.

[0030] As a preferred solution of the edge-cloud collaboration-based tablet computer real-time speech recognition and translation system of the present invention, wherein: according to the preliminary recognition text, semantic optimization is performed through semantic understanding and context modeling. The specific steps are as follows:

[0031] Input the preliminary recognition text into the context modeling structure, model the contextual relationship in the text, and obtain the global context vector;

[0032] Using context analysis methods, semantic ambiguity and incoherent expressions in the initial recognition text are identified and annotated to generate semantic correction candidates;

[0033] Based on language structure information and common expression paradigms, semantic correction candidates are semantically reconstructed and language optimized, and the optimized recognition text is output.

[0034] As a preferred solution of the edge-cloud collaboration-based tablet computer real-time speech recognition and translation system described in the present invention, the contextual relationship in the text is modeled, and the specific steps are as follows:

[0035] Inputting the preliminary recognition text into the context modeling structure, extracting context information in the preliminary recognition text and identifying the relationship between words;

[0036] Perform semantic analysis on the initially recognized text based on contextual information, identify the grammatical structure of keywords, phrases, and sentences, and annotate their role in the context;

[0037] Use preset language rules to optimize the text structure, adjust incoherent parts, and output a global context vector.

[0038] As a preferred solution of the edge-cloud collaboration-based tablet computer real-time speech recognition and translation system of the present invention, the neural network translation model is used for translation processing to generate translation text. The specific steps are as follows:

[0039] The optimized recognized text is input into the encoder part of the neural network translation model to convert the recognized text into an intermediate semantic representation;

[0040] The decoder of the neural network translation model generates a word sequence in the target language based on the semantic representation generated by the encoder, combined with the vocabulary of the target language and preset language rules;

[0041] The target language word sequence is subjected to post-grammatical reconstruction methods to perform reverse segmentation, punctuation completion and word order adjustment to generate a translated text.

[0042] The beneficial effects of the present invention are as follows: By weighting and phase-adjusting speech signals, noise interference from other directions is effectively suppressed, thereby enhancing speech signal quality. The deep residual noise reduction network is used to process speech signals, significantly reducing noise components through nonlinear feature extraction and noise separation. Adaptive noise estimation and dynamic modeling further suppress residual noise, significantly reducing the final speech signal interference. This effectively improves the clarity and intelligibility of speech data, providing high-quality input for subsequent speech recognition and translation, and enhancing accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is a schematic diagram of the tablet computer real-time speech recognition and translation system based on edge-cloud collaboration in the present invention.

[0045] Figure 2 This is a flow chart of the voice signal direction in the present invention.

[0046] Figure 3 This is a flow chart of language recognition in the present invention.

[0047] Figure 4 Flowchart of the speech translation process in the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0050] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0051] Reference Figures 1 to 4 , this embodiment provides a tablet computer real-time speech recognition and translation system based on edge-cloud collaboration, including the following steps:

[0052] The voice acquisition module collects voice signals through the tablet computer's multiple microphone arrays and uses the TDOA algorithm to obtain the direction information of the voice signals.

[0053] The tablet computer's multiple microphone arrays are used to simultaneously collect voice signals in the environment. Spectral subtraction is used to remove noise from the collected voice signals, and maximum value normalization is used to adjust the amplitude of the voice signals.

[0054] It should be explained that when the multiple microphone arrays of a tablet computer simultaneously collect voice signals in the environment, the multiple microphone array channels first sample the original voice signals in the same time period and form multiple audio data streams; then, Fourier transform is performed on each audio signal to convert the time domain signal into a frequency domain signal; in the frequency domain, spectral subtraction is applied to each audio signal to estimate the static or dynamic noise spectrum, and the noise spectrum is subtracted from the original power spectrum frequency point by frequency point to obtain a preliminary purified voice spectrum; then, the denoised frequency domain signal is inverse Fourier transformed to restore it to the time domain signal; finally, a maximum normalization operation is performed on the restored voice signal, that is, the absolute maximum amplitude of the voice signal is counted, and all amplitude samples are linearly scaled according to the maximum value to normalize them to the [-1,1] interval. For example: if the maximum amplitude is 0.8, all sample values ​​are divided by 0.8 to complete the normalization.

[0055] The GCC-PHAT algorithm is used to calculate the time difference of the preprocessed speech signal to obtain the time difference between the speech signal reaching different microphones.

[0056] Specifically, the preprocessed speech signals x1 and x2 collected by each pair of microphone channels are Fourier transformed to obtain frequency domain representations X1(f) and X2(f), respectively. Then, the cross power spectrum density is calculated, which is expressed as:

[0057]

[0058] Among them, G 12 (f) represents the cross power spectrum density of the speech signals collected by microphone 1 and microphone 2 in the frequency domain, * represents the conjugate operation; then the cross power spectrum density is PHAT weighted to obtain

[0059]

[0060] Among them, G ' 12 (f) represents the cross-power spectral density after PHAT normalization. The weighted spectrum is then inverse Fourier transformed to obtain the normalized cross-correlation function delay time. Finally, the time delay corresponding to the maximum value in the delay time is found and recorded as the maximum delay time, which is the propagation time difference of the speech signal from one microphone to the other. For example, when the sampling frequency is 16 kHz and the maximum cross-correlation occurs at a delay time of 12 samples, the propagation time difference is 12 / 16000 = 0.00075 seconds.

[0061] Based on the spatial arrangement of multiple microphone arrays and the calculated time difference, the TDOA algorithm is used to determine the propagation direction of the signal and obtain the direction information of the voice signal.

[0062] Specifically, the spatial position coordinate information between each group of microphones is recorded, and the time delay between each pair of microphones is used as the TDOA input data; then, based on the known spatial position difference and the corresponding time delay between each pair of microphones, the least squares method based on geometric constraints in the TDOA algorithm is used to establish the cosine relationship expression between the target voice signal propagation path and the connection line of each pair of microphones; then all possible propagation directions are searched, the theoretical time delay corresponding to each direction is calculated, and the error is compared with the actual measured time delay, and the propagation direction with the smallest error is selected as the direction information output of the voice signal; for example, if the microphone spacing is 0.2 meters and the propagation time difference of the voice signal between the two microphones is 0.0006 seconds, the TDOA algorithm will search for the direction angle that best matches the propagation time of 0.0006 seconds, and output this angle as the direction information of the voice signal.

[0063] The speech noise reduction module uses beamforming and deep residual networks for multi-level enhancement, and combines adaptive noise estimation and attention mechanism for multi-channel feature fusion to obtain clean speech data with optimized signal-to-noise ratio.

[0064] The beamforming method is used to weight and phase-adjust the speech signal, suppress noise interference from other directions, and enhance the speech signal.

[0065] It should be noted that the voice signals collected by multiple microphone arrays are short-time Fourier transformed to obtain frequency domain voice signals on each microphone channel; then, based on the propagation direction information of the voice signal, the relative delay of each microphone channel signal is calculated, and the frequency domain signals of each channel are phase compensated so that the voice signal in the target direction remains coherent in the frequency domain; then, a weighting factor matching the target direction is assigned to each channel, and all compensated frequency domain signals are weightedly superimposed to obtain an enhanced frequency domain voice signal; finally, the enhanced frequency domain voice signal is inversely short-time Fourier transformed to obtain an enhanced voice signal in the time domain; for example, when using a fixed beamforming algorithm, the weight can be set according to the delay calculated from the sound source direction. If the target direction is 30 degrees, the time delays of each channel are 0ms, 0.12ms, and 0.25ms, respectively, and the channels are subjected to corresponding phase correction and weighted superposition before voice enhancement processing.

[0066] The enhanced speech signal is input into the deep residual denoising network deployed on the edge computing platform, and the speech signal after preliminary denoising is obtained by using nonlinear feature extraction and noise separation mechanism.

[0067] Specifically, the input enhanced speech signal undergoes multi-layer convolution operations through a deep residual neural network. Each convolution layer filters the speech signal, extracts high-order features in the time and frequency domains, and gradually decomposes the details and structural information of the speech signal into different levels. The convolution layer helps capture the different frequency components and complex temporal features in the speech signal; after each layer of convolution operation, the input speech signal is fused with the speech signal output by the current layer using residual connections to avoid information loss and enhance the network's deep transmission capabilities. The details of the speech signal are more effectively preserved and restored, and the important features of the speech signal are retained even in the multi-layer processing process;

[0068] The Deep Residual Denoising Network uses nonlinear activation functions (such as ReLU) to perform a nonlinear transformation on the output of each layer, enhancing key signal components in the speech signal. Key signals refer to those parts of speech that help convey speech content, tone, emotion, and speech details, such as clear speech components and high-frequency features. This nonlinear transformation amplifies the high-frequency components of the speech signal and suppresses low-frequency noise, thereby enabling a clearer separation between speech and background noise. After multiple layers of convolution and nonlinear transformation, the output speech signal is the initially denoised speech signal.

[0069] The initial denoised speech signal is combined with the current ambient sound characteristics to perform an adaptive noise estimation process. By dynamically modeling and suppressing residual noise components, a speech signal with significantly reduced noise interference is obtained.

[0070] Specifically, the amplitude and spectral characteristics of the ambient noise are estimated based on the acoustic characteristics of the current environment, such as the spectrum and time domain waveform. The ambient noise characteristics are dynamically updated at each moment based on changes in the input speech signal. The initially denoised speech signal and the estimated ambient noise signal are each converted into frequency domain representations, and the amplitude and phase spectra are extracted. The amplitude difference between the speech signal and the ambient noise signal is calculated, and the difference between the two is obtained by comparing their amplitude spectra. Combined with the estimated amplitude spectrum of the ambient noise, the phase difference between the speech signal and the ambient noise signal is calculated to ensure that the naturalness of the speech signal is not affected during processing. Based on the amplitude difference, the amplitude spectrum of the speech signal is adjusted using weighting coefficients to suppress the noise component. Combining the updated amplitude spectrum with the original phase spectrum, the speech signal is reconstructed through an inverse Fourier transform, outputting a speech signal with significantly reduced noise interference.

[0071] It should also be noted that when converting the pre-noised speech signal and the estimated ambient noise signal into a frequency domain representation, a Fast Fourier Transform (FFT) is first applied to each of the pre-noised speech signal and the estimated ambient noise signal, converting them from the time domain to the frequency domain. For each moment in time, the FFT generates a set of complex values ​​representing the amplitude and phase information of the pre-noised speech signal and the estimated ambient noise signal at different frequencies. Next, the magnitude spectrum is extracted by calculating the modulus (i.e., the absolute value) of the complex value. This represents the intensity or energy distribution of the pre-noised speech signal and the estimated ambient noise signal at each frequency component. Simultaneously, the phase spectrum is extracted by calculating the phase angle (i.e., the argument) of the complex value. This represents the relative phase information of the pre-noised speech signal and the estimated ambient noise signal at each frequency component. The magnitude spectrum reflects the frequency composition of the pre-noised speech signal and the estimated ambient noise signal, while the phase spectrum preserves the temporal characteristics of the pre-noised speech signal and the estimated ambient noise signal. Next, these two frequency domain features are used to further process and optimize the speech signal and noise signal.

[0072] The speech signal with significantly reduced noise interference is jointly processed with the speech and environmental features extracted by multiple microphone channels. The weighted fusion of multimodal features is completed through the fusion attention mechanism to generate clean speech data with optimized signal-to-noise ratio.

[0073] Specifically, the speech signal with significantly reduced noise interference is combined with speech features and environmental features extracted through multiple microphone channels, and these multimodal features are weighted using a fusion attention mechanism. During this process, the speech features and environmental features extracted from multiple microphone channels are fused with the speech signal with significantly reduced noise interference to generate a joint feature set. The joint feature set is weighted using a fusion attention mechanism, and the weights are dynamically adjusted according to changes in the speech signal and environmental noise to ensure that the speech signal is strengthened while effectively suppressing noise components. The weighted feature set is further processed to optimize the signal-to-noise ratio, and clean speech data with an optimized signal-to-noise ratio is output, making the speech signal clearer and significantly reducing background noise.

[0074] The language recognition module inputs clean speech data into the language recognition network, extracts speech feature vectors, and uses the language clustering model to perform language recognition and generate language labels.

[0075] The clean speech data is pre-emphasized, frame-divided and windowed to segment the continuous speech stream into temporally localized speech signal frames.

[0076] It should be noted that pre-emphasis processing is to perform high-pass filtering on the voice signal. Usually a first-order differential filter is used to weight the voice signal to reduce the impact of low-frequency components in the spectrum on subsequent processing. The pre-emphasis processing formula is:

[0077] y(t)=x(t)-α·x(t-1);

[0078] Where y(t) represents the value of the pre-emphasized signal at time t, x(t) represents the value of the original input signal at time t, α represents the pre-emphasis coefficient, which is usually between 0 and 1 and is usually 0.95, and x(t-1) represents the value of the original input signal at time t-1, which is a sample point before the current time.

[0079] Next, the pre-emphasized speech signal is divided into frames. Each frame typically contains 20ms to 40ms of speech data, with some overlap between frames, typically 50% (for example, a 20ms frame length with a 10ms overlap). This frame division process divides the continuous speech signal into multiple temporally localized speech signal blocks.

[0080] Then, for each frame of speech signal, a window function is applied to perform weighted processing. Common window functions include Hamming window and Hanning window. Window functions can effectively reduce the distortion of signal endpoints. Using window functions for weighted processing, the expression is:

[0081]

[0082] Where w(n) is the window function value, N is the number of sampling points per frame (N>1), and n is the index of each sampling point within the window. Windowing effectively reduces edge effects during spectrum analysis of the signal.

[0083] Perform fast Fourier transform on each frame of speech signal to obtain spectral information, map the spectrum to the Mel-scale filter bank, calculate the linear energy vector distribution, and extract the Mel-frequency cepstral coefficients and logarithmic filter bank energy to form speech features.

[0084] Specifically, a fast Fourier transform is performed on each frame of the speech signal to obtain the frame's spectral information. This spectral information is then mapped to a Mel-scale filter bank, and the spectrum is weighted to simulate the human auditory perception characteristics. The linear energy vector distribution of each filter output is then calculated, representing the signal's energy distribution on the Mel scale. Next, the Mel-frequency cepstral coefficients and logarithmic filter bank energy are extracted. The logarithmic filter energy is obtained by taking the logarithm of each filter energy. Finally, the Mel-frequency cepstral coefficients and logarithmic filter bank energy are combined to form the speech recognition features.

[0085] Furthermore, a natural logarithm operation is performed on the linear energy calculated by the Mel filter bank in each frame to form a logarithmic Mel frequency energy feature after nonlinear compression, thereby enhancing the resolution of low energy and obtaining the logarithmic filter bank energy.

[0086] It should be noted that the linear energy obtained by passing the Mel filter bank through each frame of speech signal is processed. For the output energy value of each filter, a natural logarithm operation is performed to calculate the logarithmic energy value of each filter output. That is, the energy value output by each filter is logarithmically transformed, and the expression is:

[0087] L m (t) = ln(E m (t)+∈);

[0088] Among them, L m (t) represents the logarithmic energy feature value of the mth Mel filter at time t, E m (t) represents the linear energy of the mth Mel filter at time t, and ∈ is a small constant used to avoid zero values ​​in the logarithmic calculation. This performs nonlinear compression on low-energy signals, enhancing the ability to resolve low-energy components. Ultimately, the logarithmic energies of all filters are combined to form the log filter bank energy.

[0089] The energy of the logarithmic filter bank of each frame is subjected to discrete cosine transform, the frequency domain energy information is mapped to the cepstral domain, and the first N coefficients are intercepted according to the energy concentration to obtain the Mel-frequency cepstral coefficients.

[0090] Specifically, the energy of the logarithmic filter bank of each frame is discrete cosine transformed to calculate the frequency domain features. The expression of discrete cosine transform is:

[0091]

[0092] Among them, C k Represents the kth Mel frequency cepstral coefficient, M ' Indicates the number of Mel filter banks, k indicates the sequence number of the cepstral coefficient, and m indicates the number of Mel filters ranging from 1 to M. ' Next, from the obtained cepstral coefficients, according to the principle of energy concentration, the first N coefficients are intercepted to obtain the Mel-frequency cepstral coefficients.

[0093] The speech features are input into the language recognition neural network, embedded into the low-dimensional space and then input into the language clustering model to generate language labels.

[0094] Specifically, the speech features are sequentially input into the convolutional layer, pooling layer and fully connected layer of the language recognition neural network, the input features are processed by a nonlinear activation function, and the dimensionally compressed embedding vector is extracted in the output layer; after all the embedding vectors are formed into a sample set, the k-means method is used to construct a language clustering model, which specifically includes: first setting the number of categories A, initializing the cluster center, calculating the Euclidean distance between each embedding vector and all cluster centers and assigning them to the nearest center, updating the center of each category to the mean of all the embedding vectors within it, and repeating the iteration until the cluster center no longer changes or the set maximum number of iterations is reached; finally, generating a language label based on the cluster label to which the embedding vector belongs.

[0095] The text conversion module matches the speech recognition network corresponding to the current language label from the language category library based on the language label, and performs speech-to-text processing on the clean speech data through the speech recognition network to generate preliminary recognition text.

[0096] The language label is input into the language category library, and the speech recognition network identification information corresponding to the language label is retrieved.

[0097] It should be noted that the language label is used as the key value, and the hash search algorithm is called to search in the language category library. First, the language label is converted into the corresponding hash index address through a hash function. For example, the hash value H(i) = imodN is calculated using the division and remainder method, where i represents the numerical code corresponding to the language label, and N represents the capacity of the language category library; then the corresponding storage entry is searched at the hash index address. If the language label stored at the location is exactly the same as the input language label, the speech recognition network identification information in its corresponding storage unit is read as the query result; if a hash conflict occurs at the location, a conflict resolution strategy such as a linear probing method or a chain address method is used to continue matching language labels at subsequent locations until the match is successful or the traversal is completed; and finally, the speech recognition network identification information that completely corresponds to the language label is determined.

[0098] According to the speech recognition network identification information, the matching speech recognition network parameters and structure configuration are called and loaded from the preset speech recognition network.

[0099] Specifically, first retrieve the speech recognition network configuration path pointed to by the corresponding identifier in the parameter index table on the local or edge side; then read the speech recognition network structure configuration data from the parameter configuration file according to the speech recognition network configuration path, such as reading the input dimension, number of hidden layers, number of units per layer, activation function type, convolution kernel size, step size, pooling strategy and other structural information; then load the trained weight parameters and bias items from the specified parameter file, such as the ".pt" or ".pth" file in PyTorch format or the ".ckpt" file in TensorFlow format, and pass the standard API interfaces such as torch.load() or tf.train.load_checkpoint() complete the loading; then, the computational graph structure is constructed based on the structural configuration, and the loaded weight parameters are bound one-to-one to the weight matrix in the structure graph to complete the instantiation of the speech recognition network; the preset speech recognition network includes a set of recognition network models for multiple languages. Each recognition network is obtained through transfer learning or independent training based on a unified structural template. During the deployment phase, the structure files and weight parameter files are stored separately according to the language dimension, and a structure-parameter-identifier mapping table is established as a retrieval basis.

[0100] The clean speech data is passed into the speech recognition network, acoustic features are extracted, and the audio pattern of the speech is recognized based on the acoustic features.

[0101] It should be noted that after the clean speech data is input into the input layer of the speech recognition network frame by frame, the acoustic feature vector containing time-frequency distribution features is first extracted frame by frame through the acoustic front-end structure set in the speech recognition network, such as the convolutional neural network layer, the long short-term memory network layer or the Transformer encoding layer; in the feature extraction process, the speech recognition network performs time series modeling based on the short-time spectrum or Mel-frequency cepstral coefficients of the clean speech data, and extracts the intermediate expression representing the pronunciation pattern between speech frames and the movement law of the vocal organs; then the acoustic feature vector is passed to the subsequent structure of the speech recognition network, and the predicted probability distribution corresponding to the audio pattern candidate set is output through the fully connected layer, and the category corresponding to the maximum probability is used as the audio pattern recognition result of the current frame, for example, it can be identified as a specific phoneme, syllable or character fragment, and finally an audio pattern output sequence in the form of a time series is formed.

[0102] The speech recognition network decodes the speech data based on the extracted acoustic features and preset language rules, converts the acoustic features into text format, and performs spelling correction and grammatical analysis to generate preliminary recognized text.

[0103] Specifically, after the acoustic features are input into the speech recognition network, the speech recognition network first aligns the acoustic feature sequence with the audio pattern output by the speech recognition network through a connection-time classification method or an attention mechanism to generate a corresponding audio pattern sequence. The audio pattern sequence is then decoded based on the language rules preset in the speech recognition network.

[0104] The process of presetting language rules includes: first, obtaining large-scale text corpus data from public corpus resources or customized corpus resources; then, cleaning the text corpus data, including removing special symbols, unifying the encoding format, and excluding non-language character lines; then, segmenting the cleaned text corpus, dividing morpheme boundaries using a rule dictionary matching method or a two-way maximum matching algorithm, and uniformly converting it into a standard word sequence format; then, based on morphemes or words as the smallest language units, statistically analyzing the probability of sequence combinations between words, saving the parameters of the statistical language model in the form of a probability table, and saving the parameters of the neural language model in the form of neural network weights. After the construction is completed, all parameters are written into the language rule configuration file, annotated with version information and language type tags, for on-demand loading and calling during speech recognition network inference;

[0105] The decoding stage uses the Viterbi algorithm or beam search algorithm combined with language rules to select the optimal word sequence path from the candidate paths of the audio pattern and output a preliminary text; then, spelling correction is performed on the non-standard words contained in the preliminary text, and a method based on edit distance matching or corpus dictionary comparison is used to replace words that may be misspelled; the spelling-corrected text is then input into the syntactic analysis module, and a syntactic structure analysis based on context-free grammar or a grammatical relationship extraction algorithm based on Transformer structure is used to construct a grammatical dependency tree of the text, and output a preliminary recognition text with a standardized grammatical structure.

[0106] The translation module performs semantic optimization based on the preliminary recognized text through semantic understanding and context modeling, and uses a neural network translation model to perform translation processing and generate translated text.

[0107] The preliminary recognition text is input into the context modeling structure, the contextual relationship in the text is modeled, and the global context vector is obtained.

[0108] Furthermore, the preliminary recognition text is input into a context modeling structure to extract context information in the preliminary recognition text and identify the relationship between words.

[0109] Specifically, word segmentation is performed on the preliminary recognition text, and the text is divided into word sequences using a maximum forward matching method based on dictionary matching; the word sequence is then input into a word vector generation algorithm, such as the Word2Vec algorithm or the GloVe algorithm, to obtain the word vector representation of each word and form a word vector sequence; the word vector sequence is then input into a context modeling structure. If the context modeling structure adopts a bidirectional long short-term memory network, the word vector sequence is encoded in forward and reverse order respectively to obtain the forward hidden state and backward hidden state of each word, and the forward and backward hidden states are concatenated at each word position to obtain a context representation sequence containing forward and backward semantic information; if the context modeling structure adopts a Transformer structure, position encoding is added to the word vector sequence and then input into a multi-head self-attention mechanism. The inter-word dependency is extracted by calculating the attention weights between words, and the context features are nonlinearly transformed through residual connections and feedforward neural networks to generate updated word vectors that integrate context semantics; subsequently, the semantic association between words in the preliminary recognition text is identified by analyzing the attention distribution or dependency paths between hidden states in the word representation sequence output by the context modeling structure.

[0110] Perform semantic analysis on the initially recognized text based on contextual information, identify the grammatical structure of keywords, phrases, and sentences, and annotate their role in the context.

[0111] It should be noted that the input preliminary recognition text is processed by word segmentation to obtain a word sequence; the word sequence is then input into the context modeling structure to extract the context information of each word in the sentence, and the context representation extracted by the context modeling structure is used to identify the dependency relationship and grammatical role between words; then the dependency syntactic analysis method is used (for example, a graph-based dependency analysis algorithm or a neural network-based dependency syntactic analysis model) to analyze the grammatical relationship between words, construct a dependency tree of the sentence, and mark out syntactic structures such as subject-predicate relationship, object relationship and modifiers; then, combined with the context information, the keywords and phrases in the sentence are identified, and the keywords and phrases with the most semantic information in the sentence are extracted based on the word frequency and the importance of the sentence in the context; then the role of the keywords and phrases in the context is marked, such as identifying grammatical roles such as subject, predicate, and object, and the phrases are extracted based on semantic associations to identify their role and meaning in the entire context.

[0112] Use preset language rules to optimize the text structure, adjust incoherent parts, and output a global context vector.

[0113] It should be noted that after word segmentation and grammatical analysis of the input text, word and sentence structures are obtained. Then, the text structure is optimized using preset language rules, and incoherent parts are adjusted. For example, the conjunctions, pronouns, and syntactic components in the sentence are analyzed to ensure the logical consistency of the context. Then, according to grammatical rules and contextual relationships, the order of words, phrases, and sentences is rearranged to correct grammatical errors and enhance the fluency and coherence of the sentence. Then, a context-based grammatical rule model is used to calculate the global context vector, which is generated by integrating the grammatical structure, semantic relationships, and contextual information in the sentence. Finally, the optimized text and global context vector are output as the basis for further analysis and processing.

[0114] A context analysis method is used to identify and annotate semantic ambiguities and incoherent expressions in the preliminary recognized text, and to generate semantic correction candidates.

[0115] It should be noted that the preliminary recognized text is input into the context analysis structure to extract the individual sentences in the text and their grammatical relationships; then, by analyzing the context information, possible semantic ambiguity and incoherent expressions are identified, such as through the polysemy of vocabulary, errors in grammatical structure, or unclear logical relationships between sentences, and the specific locations of these problems are marked; then, combined with the results of the context analysis, multiple semantic correction candidates are generated, such as by replacing ambiguous words, reorganizing incoherent sentences, or adjusting the expression method to resolve semantic confusion or inconsistency; finally, the semantic correction candidates are output for subsequent error correction and optimization processing.

[0116] Based on language structure information and common expression paradigms, semantic correction candidates are semantically reconstructed and language optimized, and the optimized recognition text is output.

[0117] It should be noted that language structure information and common expression paradigms refer to the extraction of common syntactic structures and expressions in a language by analyzing large text corpora. Language structure information typically includes the grammatical structure of sentences, syntactic dependencies, word collocation patterns, etc., while common expression paradigms refer to the common semantic and grammatical collocations and expressions of words, phrases, and sentences in a specific language environment. Through the analysis of large-scale corpora and the generation of machine learning methods (such as deep learning models, recurrent neural networks, Transformers, etc.), more natural and accurate language optimization is achieved in text processing.

[0118] Obtain candidate texts for semantic correction and analyze their language structure information to identify possible inconsistencies or inappropriate expressions in the text. Optimize these candidate texts based on common expression paradigms, such as reconstructing sentence structures and adjusting word order according to grammatical rules to ensure that the text is more logically coherent. Then, combine the language structure information to optimize vocabulary selection, avoid polysemous words or uncommon usage, and improve the semantic consistency and fluency of the text. Finally, output the optimized recognized text to make it more consistent with conventional expression patterns, ensuring that the semantics are clearer and easier to understand.

[0119] The optimized recognized text is input into the encoder part of the neural network translation model to convert the recognized text into an intermediate semantic representation.

[0120] Specifically, the construction of the neural network translation model first prepares a large number of parallel corpus pairs containing source language text and target language text, and performs preprocessing operations such as word segmentation, removal of low-frequency words, and unified encoding on the source and target languages ​​respectively; then initializes the word vector embedding layer to map each language unit to a vector representation of fixed dimension; then constructs the encoder and decoder structures of the neural network translation model. The encoder uses a bidirectional long short-term memory network structure or a Transformer structure to encode the source language word vector sequence, and the decoder uses a long short-term memory network structure or a Transformer structure with an attention mechanism to generate the target language sequence; during the training process, the objective function is to maximize the conditional probability of the target language sequence, and the model parameters are optimized through backpropagation and gradient descent algorithms. After the training is completed, the encoder and decoder parameters are saved for use in the translation inference stage;

[0121] The optimized recognized text is segmented, and the recognized text is divided into processable language units using a dictionary matching-based method or a subword unit-based Byte Pair Encoding method. Each language unit is then converted into a corresponding word vector representation through a randomly initialized and trained embedding layer. The obtained word vector sequence is then input into the encoder part of the neural network translation model, which adopts a bidirectional long short-term memory network structure or a Transformer structure, wherein the bidirectional long short-term memory network structure forward and backward encodes the word vector sequence and splices it to obtain a contextual representation of each position. The Transformer structure uses a multi-head self-attention mechanism combined with position encoding to model the global context and generate a semantic representation of each position. The encoder part of the neural network translation model then outputs a semantic vector sequence equal to the number of input language units as an intermediate semantic representation result for the subsequent translation generation step.

[0122] The decoder of the neural network translation model generates a word sequence in the target language based on the semantic representation generated by the encoder, combined with the vocabulary of the target language and preset language rules.

[0123] It should be noted that according to the input of the decoder of the initialized neural network translation model, the input content is a special start symbol and the semantic representation of the encoder output. If the Transformer structure is used, the word vector and all semantic representations of the generated target language word sequence are input at each moment; then based on the attention mechanism, the attention weight between the current decoding position and the entire semantic representation is calculated to obtain the encoder output representation that is most relevant to the current context; then the attention context vector and the current word vector are concatenated or linearly transformed and input into the multi-layer feedforward network or long short-term memory unit of the Transformer structure, and the probability distribution of the target language word at the current moment is output; then, based on the probability distribution, the word with the highest probability is selected as the current word, or the beam search method is used to select the word path with the best score from multiple possible word paths as the word generation sequence of the target language; then the word vector of the currently generated word and the previously generated word vector are used together as the decoding input of the next moment, and the above operation is repeated until a special termination symbol is generated, and finally the word sequence of the target language is output.

[0124] The target language word sequence is subjected to post-grammatical reconstruction methods to perform reverse segmentation, punctuation completion and word order adjustment to generate a translated text.

[0125] It should be noted that, according to the language structure information of the target language, the words in the target language word sequence that have been split due to word segmentation are subjected to reverse word segmentation processing, for example, "sub" and "way" are merged into "subway", and a method based on dictionary comparison or word vector similarity is used to determine whether they belong to the same compound word; then, according to the common expression paradigms and part-of-speech tagging results of the target language, punctuation marks are inserted at appropriate positions, for example, a comma or period is inserted at the position indicating a pause or the end of a sentence. The methods used include punctuation prediction based on conditional random fields or sequence tagging models; then, the language structure information is used to adjust the word order of word sequences that do not conform to the grammatical habits of the target language, for example, the fragments with incorrect subject, predicate and object order in the target language word sequence are reordered to meet the grammatical rules. The methods used include grammatical structure restoration based on dependency syntax analysis; finally, the processed target language word sequences are combined and output as continuous text to generate a semantically coherent and grammatically correct translation text.

[0126] In summary, the present invention enhances speech signal quality by weighting and phase-adjusting speech signals to effectively suppress noise interference from other directions. It also processes speech signals using a deep residual noise reduction network, significantly reducing noise components through nonlinear feature extraction and noise separation. Adaptive noise estimation and dynamic modeling further suppress residual noise, significantly reducing the final speech signal interference. This effectively improves the clarity and intelligibility of speech data, providing high-quality input for subsequent speech recognition and translation, and enhancing accuracy and robustness.

[0127] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A tablet computer real-time speech recognition and translation system based on edge-cloud collaboration, featuring: include, The voice acquisition module collects voice signals through the tablet computer's multiple microphone arrays and uses the TDOA algorithm to obtain the voice signal direction information; The speech noise reduction module uses beamforming and deep residual networks for multi-level enhancement, combined with adaptive noise estimation and attention mechanism for multi-channel feature fusion, to obtain clean speech data with optimized signal-to-noise ratio; The language recognition module inputs clean speech data into the language recognition network, extracts speech feature vectors, and uses the language clustering model to perform language recognition and generate language labels; The text conversion module matches the speech recognition network corresponding to the current language label from the language category library based on the language label, and performs speech-to-text processing on the clean speech data through the speech recognition network to generate preliminary recognition text; The translation module performs semantic optimization based on the preliminary recognized text through semantic understanding and context modeling, and uses a neural network translation model to perform translation processing and generate translated text.

2. The tablet computer real-time speech recognition and translation system based on edge-cloud collaboration as claimed in claim 1, characterized in that: The specific steps of obtaining the voice signal direction information are as follows: The tablet computer's multiple microphone arrays simultaneously collect voice signals in the environment, remove noise from the collected voice signals using spectral subtraction, and adjust the amplitude of the voice signals using maximum normalization. Using the GCC-PHAT algorithm, the time difference of the preprocessed speech signal is calculated to obtain the time difference between the speech signal reaching different microphones; Based on the spatial arrangement of multiple microphone arrays and the calculated time difference, the TDOA algorithm is used to determine the propagation direction of the signal and obtain the direction information of the voice signal.

3. The tablet computer real-time speech recognition and translation system based on edge-cloud collaboration as claimed in claim 1, characterized in that: The steps for obtaining clean speech data after optimizing the signal-to-noise ratio are as follows: Using beamforming, the speech signal is weighted and phase-adjusted, and noise interference from other directions is suppressed to enhance the speech signal. The enhanced speech signal is input into the deep residual denoising network deployed on the edge computing platform. The nonlinear feature extraction and noise separation mechanism are used to obtain the initial denoised speech signal. The preliminarily denoised speech signal is combined with the current ambient sound characteristics to perform an adaptive noise estimation process. By dynamically modeling and suppressing the residual noise components, a speech signal with significantly reduced noise interference is obtained. The speech signal with significantly reduced noise interference is jointly processed with the speech and environmental features extracted by multiple microphone channels. The weighted fusion of multimodal features is completed through the fusion attention mechanism to generate clean speech data with optimized signal-to-noise ratio.

4. The tablet computer real-time speech recognition and translation system based on edge-cloud collaboration as claimed in claim 1, characterized in that: The specific steps for generating language labels are as follows: Perform pre-emphasis, frame division, and windowing on clean speech data to segment the continuous speech stream into temporally localized speech signal frames. Perform a fast Fourier transform on each frame of speech signal to obtain spectral information, map the spectrum to a Mel-scale filter bank, calculate the linear energy vector distribution, and extract the Mel-frequency cepstral coefficients and logarithmic filter bank energy to form speech features; The speech features are input into the language recognition neural network, embedded into the low-dimensional space and then input into the language clustering model to generate language labels.

5. The tablet computer real-time speech recognition and translation system based on edge-cloud collaboration as claimed in claim 4, characterized in that: Extract the Mel frequency cepstral coefficients and logarithmic filter bank energy. The specific steps are as follows: Perform natural logarithm operation on the linear energy calculated by Mel filter bank in each frame to form the logarithmic Mel frequency energy feature after nonlinear compression, enhance the resolution of low energy, and obtain the logarithmic filter bank energy; The energy of the logarithmic filter bank of each frame is subjected to discrete cosine transform to map the frequency domain energy information to the cepstral domain. The first N coefficients are intercepted according to the energy concentration to obtain the Mel-frequency cepstral coefficients.

6. The tablet computer real-time speech recognition and translation system based on edge-cloud collaboration as claimed in claim 1, characterized in that: The specific steps of matching the speech recognition network corresponding to the current language label from the language category library according to the language label are as follows: Inputting the language label into the language category library and retrieving the speech recognition network identification information corresponding to the language label; According to the speech recognition network identification information, the matching speech recognition network parameters and structure configuration are called and loaded from the preset speech recognition network.

7. The tablet computer real-time speech recognition and translation system based on edge-cloud collaboration as claimed in claim 6, characterized in that: The speech recognition network is used to perform speech-to-text processing on the clean speech data to generate preliminary recognition text. The specific steps are as follows: The clean speech data is passed into the speech recognition network to extract acoustic features and recognize the audio pattern of the speech based on the acoustic features; The speech recognition network decodes the speech data based on the extracted acoustic features and preset language rules, converts the acoustic features into text format, and performs spelling correction and grammatical analysis to generate preliminary recognized text.

8. The tablet computer real-time speech recognition and translation system based on edge-cloud collaboration as claimed in claim 1, characterized in that: According to the preliminary recognition text, semantic optimization is performed through semantic understanding and context modeling. The specific steps are as follows: Input the preliminary recognition text into the context modeling structure, model the contextual relationship in the text, and obtain the global context vector; Using context analysis methods, semantic ambiguity and incoherent expressions in the initial recognition text are identified and annotated to generate semantic correction candidates; Based on language structure information and common expression paradigms, semantic correction candidates are semantically reconstructed and language optimized, and the optimized recognition text is output.

9. The tablet computer real-time speech recognition and translation system based on edge-cloud collaboration as claimed in claim 8, characterized in that: Modeling the contextual relationship in the text, the specific steps are as follows: Inputting the preliminary recognition text into the context modeling structure, extracting context information in the preliminary recognition text and identifying the relationship between words; Perform semantic analysis on the initially recognized text based on contextual information, identify the grammatical structure of keywords, phrases, and sentences, and annotate their role in the context; Use preset language rules to optimize the text structure, adjust incoherent parts, and output a global context vector.

10. The tablet computer real-time speech recognition and translation system based on edge-cloud collaboration as claimed in claim 1, characterized in that: The neural network translation model is used to perform translation processing and generate translation text. The specific steps are as follows: The optimized recognized text is input into the encoder part of the neural network translation model to convert the recognized text into an intermediate semantic representation; The decoder of the neural network translation model generates a word sequence in the target language based on the semantic representation generated by the encoder, combined with the vocabulary of the target language and preset language rules; The target language word sequence is subjected to post-grammatical reconstruction methods to perform reverse segmentation, punctuation completion and word order adjustment to generate a translated text.

Images