Speech translation method and device, equipment and medium

Abstract

The invention relates to a speech translation method and device, equipment and a medium in the technical field of the Internet, and the method comprises the steps: responding to a speech translation event, continuously receiving an original audio signal flow corresponding to the event, and determining a plurality of voice-containing frame sequences corresponding to the original audio signal flow; for each human voice-containing frame sequence, performing non-human voice noise reduction processing on the human voice-containing frame sequence to obtain a pure human voice frame sequence; reasoning a to-be-translated text sequence corresponding to the pure human voice frame sequence by adopting a voice recognition model; and translating the to-be-translated text sequence into a target text sequence according to a target language corresponding to the speech translation event. The method can efficiently and accurately translate the voice in a streaming manner, and greatly improves the translation robustness in the acoustic environment of noise interference and mixed sound sources.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a voice translation method and corresponding device, electronic device, and computer-readable storage medium. Background Technology

[0002] Real-time speech translation technology aims to break down language barriers and achieve seamless cross-language communication. In recent years, with the development of deep learning technology, the accuracy of speech recognition and machine translation has significantly improved in quiet, ideal acoustic environments, and has been successfully applied in some online conferencing and instant messaging scenarios. However, when these technologies are deployed in real, complex acoustic environments, their performance suffers a catastrophic decline. Most existing translation services are based on the ideal assumption that the input audio is a single, clear, target human voice close to the microphone. This assumption is extremely fragile in reality, leading to severe challenges in environmental robustness.

[0003] In real-world streaming speech translation scenarios, such as noisy airports, bustling restaurants, or conference rooms with multiple speakers taking turns, background noise (such as conversations, traffic noise, or background music) severely contaminates the acoustic features of the target speech. This causes existing speech recognition engines to generate numerous substitution, insertion, and deletion errors. For example, "I need a quiet place." might be incorrectly recognized as "I need a quite place." or "I need Mike Blake.", rendering subsequent translations meaningless—a "garbage in, garbage out" problem. Clearly, traditional technologies struggle to adapt to complex and changing acoustic environments; their fixed audio recognition methods lack scene adaptability, and performance deteriorates drastically with environmental changes. Therefore, given the shortcomings of traditional technologies, an alternative approach is being explored to address this industry challenge. Summary of the Invention

[0004] The primary objective of this application is to solve at least one of the aforementioned problems by providing a speech translation method and corresponding apparatus, electronic device, and computer program product.

[0005] To achieve the various objectives of this application, the following technical solution is adopted: A speech translation method provided for one of the purposes of this application includes the following steps: In response to a speech translation event, continuously receive the raw audio signal stream corresponding to the event, and determine the multiple frames containing human voices corresponding to the raw audio signal stream; For each of the aforementioned human voice frame sequences, non-human voice noise reduction processing is performed on the human voice frame sequences to obtain pure human voice frame sequences. The speech recognition model is used to infer the text sequence to be translated corresponding to the pure human voice frame sequence. The text sequence to be translated is translated into a target text sequence according to the target language corresponding to the speech translation event.

[0006] On the other hand, a speech translation device provided to meet one of the purposes of this application includes an event response module, a non-human voice noise reduction module, a model inference module, and a text translation module. The event response module is used to respond to a speech translation event, continuously receive the original audio signal stream corresponding to the event, and determine multiple sequences of frames containing human voices corresponding to the original audio signal stream. The non-human voice noise reduction module is used to perform non-human voice noise reduction processing on each of the sequences of frames containing human voices to obtain a pure human voice frame sequence. The model inference module is used to infer the text sequence to be translated corresponding to the pure human voice frame sequence using a speech recognition model. The text translation module is used to translate the text sequence to be translated into a target text sequence according to the target language corresponding to the speech translation event.

[0007] In another aspect, an electronic device provided to suit one of the purposes of this application includes a central processing unit and a memory, the central processing unit being used to invoke and run a computer program stored in the memory to perform the steps of the speech translation method described in this application.

[0008] In another aspect, a computer program product provided for another purpose of this application includes a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0009] The technical solution of this application has many advantages, including but not limited to the following aspects: This application, firstly, responds to speech translation events and continuously receives the original audio signal stream. By identifying multiple sequences of frames containing human voices from the original audio signal stream, and performing non-human voice noise reduction processing on each sequence to obtain a pure human voice frame sequence, it can significantly reduce the interference of irrelevant non-human voice noise on the target speech, enhancing adaptability in noisy environments. Therefore, it effectively avoids speech recognition errors caused by acoustic feature distortion, fundamentally improving the reliability of speech translation in real-world scenarios. Simultaneously, by reasonably discarding non-human voice content, it improves processing efficiency, achieving a balance between accuracy and timeliness.

[0010] Secondly, speech recognition and translation based on pure human voice frame sequences can significantly improve the accuracy of text generation. Because noise interference is effectively suppressed, the speech recognition model can more accurately capture speech features, thereby reducing substitution, insertion, and deletion errors during the recognition process. This provides high-quality input text for the subsequent translation stage, ensuring that the entire translation process maintains high semantic consistency even in noisy environments, significantly improving the practicality and user experience of cross-language communication.

[0011] Furthermore, the streaming processing mechanism enables continuous analysis and real-time response of audio signals, maintaining stable and accurate translation performance in complex real-world acoustic environments. This not only avoids redundant processing of unnecessary audio data and optimizes the utilization and processing efficiency of computing resources, but also ensures smooth and accurate translation results even in complex scenarios such as multi-person conversations and fluctuating background noise, thereby expanding the application scope and reliability of speech translation technology in real-world settings. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A typical network deployment architecture diagram related to the implementation of the technical solution of this application; Figure 2 This is a flowchart illustrating a typical embodiment of the speech translation method of this application; Figure 3 This is a schematic block diagram of the speech translation device of this application; Figure 4 This is a schematic diagram of the structure of an electronic device used in this application. Detailed Implementation

[0013] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0014] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0015] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0016] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDAs (Personal Digital Assistants) that may include radio frequency receivers, pagers, internet / intranet access, web browsers, notebooks, calendars, and / or GPS (Global Positioning System) receivers; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0017] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0018] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method described in this application.

[0019] Please see Figure 1 The hardware infrastructure required for implementing the technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 mentioned in this application is deployed in the cloud and acts as a business server. It can further connect to relevant data servers and other servers providing related support, thereby forming a logically related service cluster to provide services to relevant terminal devices such as the smartphone 81 and personal computer 82 shown in the figure, or third-party servers (not shown). Both the smartphone and personal computer can access the Internet through known network access methods and establish a data communication link with the cloud server 80 to run terminal applications related to the services provided by the server.

[0020] For servers, the application is usually built as a service process, with corresponding program interfaces exposed for remote calls by applications running on various terminal devices. The relevant technical solutions in this application that are suitable for running on servers can be implemented in servers in this way.

[0021] The application mentioned refers to an application running on a server or terminal device. This application implements the relevant technical solutions of this application in a programmed manner. Its program code can be stored in a non-volatile storage medium that can be recognized by a computer in the form of computer-executable instructions, and is loaded into memory by the central processing unit for execution. The relevant device of this application is constructed by the operation of the application on the computer.

[0022] For servers, the application is usually built as a service process, with corresponding program interfaces exposed for remote calls by applications running on various terminal devices. The relevant technical solutions in this application that are suitable for running on servers can be implemented in servers in this way.

[0023] For various popular terminal devices, especially mobile devices such as tablets and mobile phones, they are usually equipped with built-in camera devices, or personal computers can also be connected to the aforementioned camera devices. Theoretically, the application of the terminal device in this application can call the camera devices in these situations.

[0024] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0025] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0026] Please see Figure 2 A speech translation method according to this application, in a typical embodiment, includes the following steps: Step S1100: Respond to the speech translation event, continuously receive the original audio signal stream corresponding to the event, and determine the multiple frames containing human voices corresponding to the original audio signal stream; Within the client application, voice translation events can be automatically triggered by various specific user interactions or client states. A typical scenario is that the user actively initiates the voice translation service provided by the client (such as instant messaging software, email clients, or web browsers). After the service begins, it enters a waiting state, at which point the audio acquisition module is activated to prepare for receiving analog signal input, thus triggering the voice translation event. Another common scenario is that the user initiates the translation function within the client by clicking a built-in "Translate" button or voice icon, thereby triggering the voice translation event.

[0027] Furthermore, voice translation events can also be triggered in a controlled manner by the client's context-aware functionality. When the client uses a GPS or network with user authorization and detects that the user is in a pre-defined foreign language environment (such as an airport, international hotel, or a specific country / region), a prompt can automatically appear asking the user whether to enable real-time voice translation. If the user confirms, the event is triggered. Another automatic triggering scenario is based on audio analysis. For example, when the client detects during a call that the other party's language is inconsistent with the user's preset native language, a voice translation event can be automatically initiated to provide real-time translation assistance.

[0028] In some integration scenarios, voice translation events are invoked as a sub-service provided by the client. For example, in the client of a video conferencing application, it can be configured to automatically provide translations for each participant's speech. In this case, whenever a new speaker is detected to begin speaking, a separate voice translation event is triggered for that speaker's audio stream.

[0029] In response to a speech translation event, the system continuously receives analog signals from a microphone or other audio input device, which are then digitized into a digital signal as the raw audio signal stream. This digital signal is typically digitized at a fixed sampling rate and quantization precision, such as 16kHz sampling rate and 16-bit quantization precision. Those skilled in the art can configure this fixed sampling rate and quantization precision as needed. In real-world scenarios, the raw audio signal stream is prone to mixing ambient sound and noise with the human voice, thus requiring the extraction of the human voice portion for subsequent processing. The process of determining multiple frames containing human voices corresponding to the raw audio signal stream involves segmenting the continuous audio stream into short-time frame sequences and identifying which frame sequences contain valid human voices.

[0030] It is understood that the event can be responded to locally by the client or by the server. In this process, in one embodiment, the original audio signal stream is first preprocessed by framing to obtain multiple short frames. The duration of each frame is usually 10 to 30 milliseconds. There may be overlap between frames, and each frame represents the frequency domain characteristics of the continuous digital signal within the corresponding duration. That is, by preprocessing the continuous digital signal within each duration, its original temporal characteristics are transformed into corresponding frequency domain characteristics.

[0031] A sequence of frames containing human voices refers to a sequence of multiple consecutive frames that contains human voice features. Human voice features can be identified in various ways, such as based on spectral features, energy thresholds, or machine learning models.

[0032] In one embodiment, a human voice detection model is directly applied to classify each audio frame. This model is pre-trained to learn the reasoning ability to distinguish between human voices and non-human voices. Convolutional neural networks and / or recurrent neural networks can be used to analyze the Mel frequency cepstral coefficient features or logarithmic filter bank energy of the frames, and output the probability that each frame belongs to human voice. When the probability exceeds a threshold, it is marked as a frame containing human voice. Frames that are continuously marked as containing human voice are grouped to form a sequence of frames containing human voice.

[0033] In another embodiment, coarse-grained sound activity detection is first performed to identify all frame sequences containing continuous sound, and then fine-grained human voice verification is performed on these sequences. For example, the fundamental frequency or formant frequency of the frame can be calculated to verify whether it falls within the typical range of human speech.

[0034] Step S1200: For each of the frames containing human voices, perform non-human voice noise reduction processing on the frames containing human voices to obtain pure human voice frames. For each known frame sequence containing human voice, non-human voice noise reduction processing is performed on the sequence. The purpose is to suppress or eliminate noise components related to non-human voice in the frame sequence, thereby obtaining a higher quality pure human voice frame sequence.

[0035] In one implementation, end-to-end noise reduction is performed using a pre-trained deep learning model. A sequence of frames containing human voices is input into a speech separation neural network model, which learns to directly map clean human voice signals from mixed audio. For example, the model could be a U-Net model, a CNN (Convolutional Neural Network) + LSTM (Long Short-Term Memory) model, a temporal fully convolutional model, a dual-path recurrent neural network model, or a multi-scale encoding / decoding model combining time-frequency convolution and time-frequency fully connected layers. The input is a sequence of frames containing human voices, composed of multiple frames representing Mel-frequency cepstral coefficient features or logarithmic filter bank energy. The model outputs an ideal ratio mask, which, when multiplied point-by-point by the noisy spectrum, filters out non-human voice components, generating an enhanced spectrum, which is then reconstructed into a sequence of pure human voice frames. Such models need to be trained on a mixed audio dataset containing clean human voices and various types of noise to learn the distinguishing features between human voices and common noise. Those skilled in the art can flexibly adapt and train the model based on the information disclosed herein.

[0036] Step S1300: Use a speech recognition model to infer the text sequence to be translated corresponding to the pure human voice frame sequence; A speech recognition model is a trained machine learning model that can identify speech content from audio signals and generate corresponding text representations. After being deployed online, this model uses pre-trained parameters to perform forward inference on the input pure human voice frame sequence, outputs a probability distribution sequence, and generates the final text sequence through a decoding algorithm.

[0037] A pure voice frame sequence is usually represented as a series of short-time audio frames, and each frame contains corresponding voice frequency domain features, such as Mel-frequency cepstral coefficients or log filter bank energies. These features capture the acoustic characteristics of the voice and serve as the input to the speech recognition model. The speech recognition model is designed based on the sequence-to-sequence mapping principle, where the input is a variable-length pure voice frame sequence and the output is a variable-length text token sequence. Common model architectures include CNN (Convolutional Neural Network) + CTC (Connectionist Temporal Classification) model, BiLSTM (Bidirectional Long Short-Term Memory Network) + CTC (Connectionist Temporal Classification) model, and attention-based encoder-decoder + CTC (Connectionist Temporal Classification) model.

[0038] In one embodiment, the convolutional neural network works in cooperation with the connectionist temporal classification model. The pure voice frame sequence is organized into an input feature map and used as two-dimensional features to input into the convolutional neural network. This network contains multiple convolutional layers and pooling layers. For example, a 3x3 convolutional kernel is used to extract the local correlation of acoustic features, and the time dimension resolution is reduced through the max pooling layer. The high-level feature sequence extracted by the convolutional neural network is flattened and then fed into the fully connected layer, and the probability distribution of all characters in the character set corresponding to each time step is output. The connectionist temporal classification layer receives this probability distribution, calculates the probabilities of all possible character alignment paths through the forward-backward algorithm, and uses beam search decoding to find the most likely character sequence during inference. For example, when the input is the pure voice frame sequence of the English speech "good morning", after the convolutional neural network represents the input data with features, the connectionist temporal classification decodes and outputs a character sequence such as "g o o d m o r n i n g", and the final text "good morning" is obtained after deduplication and blank character deletion.

[0039] In another embodiment, the bidirectional long short-term memory network is used in conjunction with the connectionist temporal classification model. The pure voice frame sequence is input into the bidirectional long short-term memory network in chronological order. This network contains forward and backward propagation layers and can utilize the past and future context information of each audio frame simultaneously. The hidden state at each time step is linearly transformed into the character probability space to form a probability distribution sequence. The connectionist temporal classification calculates the loss function on this sequence and efficiently calculates the conditional probability between the output sequence and the target sequence through dynamic programming. During inference, the prefix beam search algorithm is adopted to balance the search efficiency and accuracy while maintaining multiple candidate sequences. For example, when processing the Chinese speech "你好", the bidirectional long short-term memory network analyzes the temporal dependence relationship of the frame sequence, and the connectionist temporal classification decodes and generates a Chinese character sequence such as "你-好", and the final text is directly formed into a readable text after deduplication and blank character deletion.

[0040] In another embodiment, an attention-based encoding / decoding model is combined with a connection-time classifier. The encoder consists of a stack of BERT (a bidirectional encoder from a transformer) and bidirectional long short-term memory networks, encoding the input frame sequence into a high-level representation. The decoder employs an attention mechanism, dynamically calculating the weight relationship with the encoder at each time step when generating each output character. The connection-time classifier serves as an auxiliary training objective, working together with the decoder output of the attention mechanism to optimize the model, improving convergence speed and recognition robustness. During inference, the confidence of the attention-generated sequence and the candidate sequences from the connection-time classifier are combined to select the optimal result. For example, when processing the French speech "au revoir," the attention mechanism guides the decoder to generate characters one by one, while the connection-time classifier provides alternative alignment paths, ultimately outputting the text "au revoir" as correctly inferred by the model.

[0041] In practice, the speech recognition model can be optimized into a lightweight version. Optimization can be achieved through model knowledge distillation, quantization, or pruning to adapt to the computing resources of physical devices. The inference engine uses frameworks such as TensorFlow Lite or ONNX Runtime to ensure efficient execution. The entire inference process is automated, requiring no manual intervention. Pure human voice frame sequences serve as input, and the text sequence to be translated serves as output, directly used in subsequent translation steps.

[0042] Step S1400: Translate the text sequence to be translated into a target text sequence according to the target language corresponding to the speech translation event.

[0043] The target language is the language predetermined or set in real-time during a voice translation event. It can be achieved through one or more methods, such as explicit user configuration, automatic client inference, or context awareness. Explicit user configuration involves the user specifying their preferred target language in the client settings, for example, selecting Japanese before the voice translation function is activated. Automatic client inference is based on device status or environmental information; for example, when a user is detected as being in France via GPS, the target language is automatically set to French. Context awareness involves the client analyzing the current application scenario; for example, in an instant messaging chat window, if the other user's historical message language is German, the target language is set to German. Another common method is to pass the target language through the trigger parameters of the voice translation event; for example, when the user clicks the translate button, a dropdown menu selects English as the target, and this selection is bound to the translation process as an event attribute.

[0044] The translation process uses machine translation technology to convert the sequence of text to be translated from the source language into the target text sequence in the target language. The machine translation model is a trained sequence-to-sequence mapping model that learns the probability distribution between the source language text and the target language text. The model architecture includes encoder and decoder components. The encoder encodes the input text sequence into a sequence of context vectors with a fixed dimension, and the decoder generates the target text sequence autoregressively based on this vector sequence. Specifically, the encoder consists of a word embedding layer and a feature encoding network layer. The word embedding layer maps each source language word to a dense vector, and the feature encoding network layer, such as a bidirectional long short-term memory network or a Transformer Encoder, extracts sequence features. The decoder generates a target word at each step, depending on the encoder output and the history of the generated words.

[0045] When implementing machine translation, a pre-trained model is deployed or a remote translation service is called. Before local deployment, the model is lightweighted to adapt to the computing resources of physical devices. For example, the weights are compressed from 32-bit floating point to 8-bit integer through model quantization, or redundant connections are removed through pruning. Model selection includes architectures based on Transformer, etc., which use self-attention mechanisms to process the input sequence in parallel, improving the translation speed. For example, when the text sequence to be translated is "你好" in Chinese and the target language is English, the model encodes "你好" into a vector sequence, and the decoder generates the word sequence "hello". When calling the remote service, the text sequence to be translated is sent to the cloud server through a network interface, and the target text sequence returned by the cloud server is obtained. The receiving end only needs to handle network communication.

[0046] The translation process includes text preprocessing, model inference, and postprocessing steps. Text preprocessing performs word segmentation and normalization on the text sequence to be translated. For example, English text is segmented into a word sequence, and Chinese text is divided into word units using a word segmentation algorithm. When performing model decoding inference, the beam search algorithm can be used to balance the generation quality and efficiency. The beam width is set to 3 to 10, and multiple candidate sequences are retained and finally the optimal sequence is selected as the output result sequence to avoid local optimality. Postprocessing includes any one or more of the following processes on the generated sequence: capitalization correction, punctuation insertion, and grammar rule adjustment, etc., to ensure that the target text sequence conforms to language habits.

[0047] The training data of the machine translation model contains a parallel corpus, that is, sentence pairs in the source language and the target language. The model learns the mapping relationship through maximum likelihood estimation. Those skilled in the art can train the model flexibly according to the disclosure here.

[0048] According to the typical embodiments of this application, it can be known that the technical solutions of this application have many advantages, including but not limited to the following aspects: This application, firstly, responds to speech translation events and continuously receives the original audio signal stream. By identifying multiple sequences of frames containing human voices from the original audio signal stream, and performing non-human voice noise reduction processing on each sequence to obtain a pure human voice frame sequence, it can significantly reduce the interference of irrelevant non-human voice noise on the target speech, enhancing adaptability in noisy environments. Therefore, it effectively avoids speech recognition errors caused by acoustic feature distortion, fundamentally improving the reliability of speech translation in real-world scenarios. Simultaneously, by reasonably discarding non-human voice content, it improves processing efficiency, achieving a balance between accuracy and timeliness.

[0049] Secondly, speech recognition and translation based on pure human voice frame sequences can significantly improve the accuracy of text generation. Because noise interference is effectively suppressed, the speech recognition model can more accurately capture speech features, thereby reducing substitution, insertion, and deletion errors during the recognition process. This provides high-quality input text for the subsequent translation stage, ensuring that the entire translation process maintains high semantic consistency even in noisy environments, significantly improving the practicality and user experience of cross-language communication.

[0050] Furthermore, the streaming processing mechanism enables continuous analysis and real-time response of audio signals, maintaining stable and accurate translation performance in complex real-world acoustic environments. This not only avoids redundant processing of unnecessary audio data and optimizes the utilization and processing efficiency of computing resources, but also ensures smooth and accurate translation results even in complex scenarios such as multi-person conversations and fluctuating background noise, thereby expanding the application scope and reliability of speech translation technology in real-world settings.

[0051] In a further embodiment, step S1100, determining multiple frames containing human voices corresponding to the original audio signal stream, includes the following steps: Step S1110: Based on sound activity detection of multiple target audio frame sequences corresponding to the original audio signal stream, obtain each sound frame sequence in the original signal stream; In speech translation, identifying multiple sequences of human voice frames within the original audio signal stream requires initial preprocessing by framing the original audio signal stream. This results in multiple short frames, each typically ranging from 10 to 30 milliseconds in duration. Frames may overlap, and each frame represents the frequency domain characteristics of a continuous digital signal within its corresponding duration. Subsequently, sound activity detection is performed on these frames to identify segments containing any sound activity from the continuous audio signal stream, rather than just silence or background noise. Sound activity detection distinguishes between areas with and without sound by analyzing the time or frequency domain characteristics of the audio signal.

[0052] In one embodiment, the calculation is based on short-time energy and zero-crossing rate, where short-time energy represents the sum of squares of the signal amplitude within a frame, and zero-crossing rate represents the frequency at which the signal crosses the zero axis; when the energy exceeds a preset threshold and the zero-crossing rate is within a certain range, the frame is marked as an audio frame, which contains audio activity.

[0053] In another embodiment, a statistical model, such as a Gaussian mixture model, is used. This model learns the probability distribution of background noise and sound activity during the training phase and calculates the posterior probability that the current frame belongs to sound activity during the inference phase. When the probability exceeds a threshold, it is determined to be a sound frame.

[0054] In one embodiment, multiple consecutive audio frames obtained from preprocessing the original signal stream can be combined into different individual audio frame sequences. Furthermore, if the number of frames in an audio frame sequence is too large, to ensure translation efficiency, the sequence can be truncated according to a preset number of frames, thereby obtaining shorter new sequences. Those skilled in the art can further configure the preset number of frames as needed based on the disclosure herein.

[0055] Step S1120: For each sound frame sequence, check whether the sound frame sequence contains human voice features. If it does, the sound frame sequence is regarded as a human voice-containing frame sequence.

[0056] For each sound frame sequence obtained through sound activity detection, it is necessary to further verify whether the sound frame sequence contains human voice features. Human voice features refer to the acoustic characteristics that can distinguish human speech from other types of sounds (such as environmental noise, music, and animal calls). The purpose of verification is to confirm that the sound activity originates from human speech, rather than other sound sources, thereby ensuring the effectiveness of subsequent speech recognition and translation processing.

[0057] The verification process is based on the unique patterns exhibited by the human vocal mechanism in acoustic signals. One implementation involves analyzing the formant characteristics of a sequence of sound frames. Formants are energy-concentrated regions in the speech spectrum, determined by the shape of resonating cavities such as the oral cavity and nasal cavity. Different vowels correspond to different formant frequency patterns. For example, the vowel / i / (as in "ee") has a lower first formant frequency and a higher second formant frequency; while the vowel / a / (as in "ah") has a higher first formant frequency and a lower second formant frequency. During verification, a linear predictive coding algorithm is used to estimate the frequencies of the first three formants of representative frames in the sequence, and the combination of these frequency values ​​is checked to see if it falls within the typical vowel formant distribution area defined by the International Phonetic Alphabet. If a match is found, it is determined to contain human vocal features.

[0058] Another approach is to use a pre-trained classifier model for discrimination. This model takes a sequence of audio frames as input and outputs a score representing the probability that the sequence contains human voices. The classifier model can be a support vector machine (SVM), which learns a classification hyperplane during the training phase using labeled human and non-human voice samples; during the inference phase, it calculates the position of the current sequence features relative to this hyperplane to obtain the classification result. Alternatively, a deep learning model, such as a multidimensional convolutional neural network (CNN), can be used as the classifier model. This network can automatically learn discriminative patterns from the original frame sequence or its features. When the probability score output by the model exceeds a preset threshold (e.g., 0.8), the audio frame sequence is classified as containing human voices.

[0059] If the verification passes, the audio frame sequence is confirmed as containing human voices and proceeds to the subsequent non-human voice noise reduction processing stage. If the verification fails, the sequence is discarded or marked as invalid to avoid unnecessary consumption of computing resources.

[0060] In this embodiment, by first performing coarse-grained sound activity detection to obtain a sound frame sequence, and then finely verifying human voice features, efficient layered processing of the audio stream is achieved. This significantly reduces the false detection rate and avoids misclassifying non-human noise (such as ambient noise or mechanical sounds) as valid input, thereby reducing unnecessary subsequent noise reduction and recognition calculations and improving processing efficiency. Simultaneously, through multiple verification mechanisms such as fundamental frequency, formant, or classifier models, it is ensured that only genuine human voice segments enter the subsequent process, guaranteeing robustness in complex acoustic environments. Therefore, resource allocation is fundamentally optimized, enabling the speech translation service to more accurately focus on valid speech data in real-time scenarios, balancing accuracy and energy consumption control.

[0061] In a further embodiment, step S1110, obtaining each sound frame sequence in the original signal stream by performing sound activity detection on multiple target audio frame sequences corresponding to the original audio signal stream, includes the following steps: Step S1111: Using a preset sound activity detection model, continuously detect whether each target audio frame sequence corresponding to the original audio signal stream contains sound activity features and its start and end activity endpoints; Sound activity detection models are pre-trained machine learning models used to analyze audio frame sequences and determine whether they contain sound activity features, while also identifying the start and end points of sound activity. Sound activity features refer to acoustic patterns in an audio signal that distinguish it from silence or background noise, such as signal energy significantly higher than the silence baseline, changes in spectral entropy, or energy concentration in a specific frequency band.

[0062] The start active endpoint represents the moment when the sound event begins, i.e., the boundary point from the silent state to the sound state; the end active endpoint represents the moment when the sound event ends, i.e., the boundary point from the sound state back to the silent state.

[0063] The sound activity detection model continuously detects each target audio frame sequence one by one. These sequences consist of preprocessed short audio frames, each containing frequency domain features such as Mel-frequency cepstral coefficients or logarithmic filter bank energy. During detection, the model calculates the sound activity probability for each frame and locates endpoints based on probability change points. In one embodiment, the sound activity model is specifically implemented as a buffered FSMN-Monophone VAD model. It takes the feature vector of each frame in the sequence as input and outputs the probability value of each frame belonging to sound activity. When the probability jumps from below a threshold to above a threshold, it is marked as the starting activity endpoint; when the probability drops from above a threshold to below a threshold, it is marked as the ending activity endpoint. Furthermore, those skilled in the art will understand that for the model's buffer, the feature representations corresponding to all historical input target frame sequences stored therein are concatenated with the feature representation of the current input sequence with each input, so that it can be used to determine whether sound activity features exist and their starting and ending activity endpoints in the current inference. That is, after each output, the feature representation of the current input sequence is stored in the buffer for use in the next inference.

[0064] Step S1112: When the target audio frame sequence is detected to contain sound activity features, the target audio frame sequence is stored in the cache; When a target audio frame sequence is detected to contain sound activity features, the sequence is stored in a buffer. The buffer is a temporary storage area in memory used to accumulate consecutive frame sequences in chronological order. The storage operation ensures that the frame sequence is not processed immediately, but is retrieved all at once after subsequent confirmation. For example, the buffer can be implemented as a first-in, first-out queue, and newly detected frame sequences containing sound activity features are appended to the end of the queue.

[0065] Step S1113: When it is detected that the target audio frame sequence also contains the end of the sound activity feature, or when the cumulative number of target audio frame sequences containing sound activity features exceeds a preset threshold, all target audio frame sequences in the cache are retrieved according to the storage order and concatenated with the currently detected target audio frame sequence to extract all corresponding sound frames and obtain the sound frame sequence.

[0066] The generation of sound frame sequences is triggered under two conditions: the first condition is that the current target audio frame sequence contains the end of the sound activity feature, indicating that a sound is complete; the second condition is that the cumulative number of target audio frame sequences containing sound activity features exceeds a preset threshold, for example, 8 consecutive frame sequences are marked as having sound activity. In this case, to prevent buffer overflow or to process long-term sound, the sequence is forcibly truncated and the sound activity model's buffer is cleared, so that a new starting activity endpoint can be determined.

[0067] Upon triggering, all stored target audio frame sequences are retrieved from the cache according to their storage order, and concatenated with the latest detected sequence in chronological order to form a longer continuous frame sequence. Subsequently, corresponding to the first condition, the frame portion containing only sound activity is extracted from the concatenated sequence, that is, starting from the frame corresponding to the detected start activity endpoint and ending at the frame corresponding to the end activity endpoint or threshold trigger point, to obtain the final sound frame sequence, thus achieving continuous frames containing sound activity as output; corresponding to the second condition, the frame portion containing only sound activity is extracted from the concatenated sequence, that is, starting from the frame corresponding to the detected start activity endpoint and ending at the frame corresponding to the threshold trigger point, to obtain the final sound frame sequence, thus achieving continuous frames containing sound activity as output.

[0068] In this embodiment, a pre-trained sound activity detection model is employed, combined with a caching mechanism to dynamically locate the start and end points of sound activities. Sequence truncation is triggered based on these endpoints or cumulative frame counts, implementing a caching and splicing mechanism. This ensures the integrity of long speech segments, avoiding speech fragmentation issues caused by fixed window segmentation. Furthermore, threshold truncation prevents excessively long speech segments from affecting subsequent processing wait times, guaranteeing the stability and timeliness of streaming processing. Therefore, this improves the accuracy and real-time performance of sound activity detection, providing coherent and high-quality input for subsequent noise reduction and recognition, thereby enhancing the overall coherence and reliability of the translation process.

[0069] In a further embodiment, before step S1111, which uses a preset sound activity detection model to continuously detect whether each target audio frame sequence corresponding to the original audio signal stream contains sound activity features and its start and end activity endpoints, the following steps are included: Step S2000: Segment the original signal stream according to a preset duration to obtain multiple original signal segments in the original signal stream; In speech translation, the raw audio signal stream is first divided into multiple segments for subsequent processing. This segmentation is based on a preset duration, such as one second or two seconds per segment, cutting the continuous data stream into discrete units. Those skilled in the art can configure the preset duration as needed. The choice of preset duration balances processing real-time performance and computational efficiency; shorter segments reduce processing latency but increase overhead, while longer segments have the opposite effect. One implementation divides the signal stream into non-overlapping segments with a preset duration of one thousand milliseconds.

[0070] Step S2010: For each original signal segment, perform sliding windowing framing processing on the original signal segment according to the number of frames covered by the framing window and the movement step size to obtain the audio signal frame sequence; Each original signal segment then undergoes sliding windowing framing, breaking the segment down into shorter audio signal frames. The number of frames covered by the framing window defines the time span of each frame, typically between twenty and thirty milliseconds, to capture the short-term stationary characteristics of speech. Those skilled in the art can configure the number of covered frames as needed. The movement step size specifies the distance the window slides each time, usually set to ten milliseconds, generating overlapping frames to maintain temporal continuity. Those skilled in the art can configure the movement step size as needed. Window functions such as Hamming or Hanning windows are applied to each frame to mitigate spectral leakage. In one embodiment, a one-second signal segment sliding with a twenty-five-millisecond window and a ten-millisecond step size can generate 98 audio signal frames. In the specific windowing implementation, the signal segments within the corresponding framing window are multiplied point-by-point by the window function to generate the windowed frame sequence.

[0071] Step S2020: Perform frequency domain feature extraction processing on each audio signal frame in the audio signal frame sequence to obtain the audio frame sequence to be optimized; Frequency domain feature extraction transforms each audio signal frame from the time domain to the frequency domain to reveal its acoustic properties. Acoustic features include Mel-frequency cepstral coefficients, which simulate the human ear's perception of frequency. The extraction process first performs a Fast Fourier Transform on the frame to obtain the amplitude spectrum, then applies a Mel-scale filter bank to compress the frequency band, calculates the logarithmic energy of each filter, and finally obtains the cepstral coefficients through a Discrete Cosine Transform. In another embodiment, the acoustic features are the logarithmic filter bank energy; the Discrete Cosine Transform step is omitted, and the logarithmic filter energy values ​​are directly output. This transformation can be performed in batches through mathematical operations, ultimately generating the audio frame sequence to be optimized.

[0072] Step S2030: Normalize each audio frame in the audio frame sequence to be optimized according to the preset mean and variance to obtain a target audio frame sequence composed of all processed target audio frames.

[0073] The purpose of normalization is to eliminate feature distribution biases introduced by differences in recording equipment, channels, or speaker volume among different audio segments, ensuring that the feature data conforms to the distribution expected during model training, thereby improving the robustness and accuracy of model recognition. Normalization is performed based on preset means and variances, which are typically global statistics pre-calculated on large-scale speech datasets. The specific calculation of normalization usually employs the standard fractional normalization method, which involves subtracting the preset mean from each feature value of each audio frame in the audio frame sequence to be optimized, and then dividing by the preset variance. The result after normalization is the target audio frame, and the sequence formed by arranging these target audio frames in their original temporal order is the target audio frame sequence.

[0074] In this embodiment, a preprocessing procedure, including signal segmentation, sliding windowing framing, frequency domain feature extraction, and normalization, transforms the original audio stream into a standardized target audio frame sequence. Segmentation and framing preserve the short-term stationary characteristics of speech, while overlapping window design avoids information loss, providing a temporally complete input for subsequent model analysis. Frequency domain feature extraction (such as Mel-frequency cepstral coefficients) effectively captures the acoustic essence, while normalization based on preset mean and variance eliminates feature biases introduced by device, environment, or speaker differences, improving the model's generalization ability in different scenarios. Therefore, standardized preprocessing ensures the consistency of input data, reduces the uncertainty of model inference, and thus significantly improves the robustness and accuracy of the speech recognition and translation modules.

[0075] In a further embodiment, step S1300, inferring the text sequence to be translated corresponding to the pure human voice frame sequence using a speech recognition model, includes: Step S1310: Based on the preset low frame rate modeling strategy, the speech recognition model is invoked to infer the pure human voice frame sequence to obtain the corresponding text sequence to be translated.

[0076] Low frame rate modeling is a method to optimize the input data of speech recognition models, aiming to reduce the number of frames that need to be processed while maintaining recognition accuracy, thereby improving inference efficiency. This strategy reduces the temporal resolution of the model's input sequence by merging multiple adjacent short audio frames in a pure human voice frame sequence into a superframe unit. Each superframe corresponds to an average output target, allowing the model to process acoustic features with coarser temporal granularity during inference, reducing computational load.

[0077] In one embodiment, the speech recognition model employs a connected temporal classifier decoder, whose input sequence is a low-frame-rate superframe sequence. During model inference, the connected temporal classifier calculates the alignment probability between the superframe sequence and the character sequence, and generates the most probable text sequence through beam search. Due to the reduced frame rate, the number of beam search paths is decreased, thus improving decoding speed. The model's inference speed is increased exponentially, while recognition accuracy remains stable through context compensation.

[0078] Low frame rate modeling strategies can also be combined with other optimization techniques, such as feature dimensionality reduction or model quantization, to further reduce computational load. During inference deployment, the speech recognition model preloads the processing parameters of the bound superframes, applies frame bindings to the input pure human voice frame sequence in real time, generates a low frame rate sequence, performs forward inference, and finally outputs the text sequence to be translated.

[0079] In this embodiment, a low frame rate modeling strategy is used to merge pure human voice frame sequences into superframe units, reducing the number of input frames and significantly lowering the computational complexity of the speech recognition model, thereby improving inference speed. This is particularly suitable for real-time or resource-constrained devices. Simultaneously, the low frame rate processing, combined with a context compensation mechanism (such as connection-based temporal classifier decoding), maintains recognition accuracy without significant impact, achieving a balance between efficiency and accuracy. Therefore, it can process longer input data more efficiently, reduce latency, and enhance user experience, especially in mobile or embedded devices, expanding the practical application scope of speech translation technology.

[0080] In a further embodiment, after step S1400, which translates the text sequence to be translated into a target text sequence according to the target language corresponding to the speech translation event, the following steps are included: Step S1510: Based on all previously inferred target text sequences and the target text sequence inferred this time, verify whether there are any duplicate expression errors in the target text sequence inferred this time; In streaming speech translation, after outputting the target text sequence obtained from the inference, a check for repetition errors is performed. Repetition errors refer to non-semantic repetitions in the translation result, such as invalid repetitions of adjacent identical words, phrases, or characters (e.g., "hello hello" instead of "hello"). These errors may stem from audio overlap processing during speech recognition, redundant output due to probability distribution fluctuations in the translation model during decoding, or repetitive pronunciations in the source language itself. The purpose of the check is to eliminate these non-semantically necessary repetitions, improving the conciseness and naturalness of the output text.

[0081] During validation, contextual analysis is performed by combining all previously inferred target text sequences with the target text sequence inferred this time. All previously inferred target text sequences refer to the complete translation results history that has been output and cached before this translation. Typically, a dynamically updated text buffer is maintained to store these sequences to ensure contextual coherence. One specific implementation of validation is to use a pre-trained large language model (such as a Transformer-based generative model) to determine whether there are any non-semantically coherent duplicate text contents in the target text sequence inferred this time, based on all previously inferred target text sequences and the target text sequence inferred this time. If duplicate expression errors are found, the corresponding erroneous text and its correction text in the target text sequence inferred this time are output.

[0082] Step S1520: When the repeated expression error exists, correct the corresponding erroneous text in the target text sequence inferred this time.

[0083] At this point, the erroneous text in the target text sequence of this inference is replaced with the corrected text output by the above model to complete the correction of the result.

[0084] In this embodiment, by combining historical translation results to dynamically verify and correct repetitive errors in the current translation, non-semantic repetitions caused by model fluctuations or audio processing can be effectively identified and eliminated, significantly improving the conciseness and fluency of the output text. The implemented post-processing mechanism, as a quality assurance link in the translation pipeline, compensates for the shortcomings of end-to-end models in terms of generation stability, reduces the burden of manual correction for users, and thus enhances the practicality and user experience of real-time speech translation.

[0085] In a further embodiment, after step S1400, which translates the text sequence to be translated into a target text sequence according to the target language corresponding to the speech translation event, the following steps are included: Step S1501: Based on all previously inferred target text sequences and the target text sequence inferred this time, verify whether there are semantic expression errors in the target text sequence inferred this time; During the process of streaming speech translation, after outputting the target text sequence obtained from the current inference, semantic expression errors are further verified. Semantic expression errors refer to problems such as inaccurate semantics, incorrect grammatical structures, logical contradictions in context, or non - compliance with cultural habits in the translation result. For example, deviation in word - meaning selection (such as mis - translating "bank" as "river bank" instead of "bank"), disordered word order (such as mis - translating the Chinese sentence "我吃饭" as "eat I food"), or conflict with historical translation result information (such as inconsistent anaphora). Such errors may stem from insufficient understanding of the source - language context by the translation model, preference biases in the target - language generation model, or incomplete coverage of the training data.

[0086] During verification, context analysis is performed by combining all previously inferred target text sequences and the target text sequence inferred in the current round. All previously inferred target text sequences refer to the complete historical translation results that have been output and cached before the current translation. Usually, a dynamically updated text buffer is maintained to store these sequences to ensure context coherence. A specific implementation of verification is to use a pre - trained large - language model (such as a Transformer - based generative model) to determine whether there is duplicate text content that is not semantically coherent in the target text sequence inferred in the current round, and correspondingly output whether there is a semantic expression error. When there is a semantic expression error, the corresponding incorrect text and its corrected text in the target text sequence inferred in the current round are output.

[0087] Step S1502: When there is such a semantic expression error, correct the corresponding incorrect text in the target text sequence inferred in the current round.

[0088] At this time, replace the corresponding incorrect text in the target text sequence of the current inference with the corrected text output by the above - mentioned model to complete the correction of the result.

[0089] In this embodiment, by introducing a verification and correction mechanism for semantic expression errors, the quality of the translation result in terms of semantic accuracy, grammatical correctness, and context consistency can be effectively improved. The implemented deep post - processing not only compensates for the limitations of the translation model in complex contexts but also significantly reduces communication misunderstandings caused by semantic deviations, making real - time speech translation more reliable in rigorous scenarios such as cross - language conversations and business meetings, thus expanding the application boundaries of the technology and user trust.

[0090] Please refer to Figure 3This application provides a speech translation device to fulfill one of its objectives, which is a functional embodiment of the speech translation method of this application. On another level, this speech translation device, also fulfilling one of its objectives, includes an event response module 1100, a non-human voice noise reduction module 1200, a model inference module 1300, and a text translation module 1400. The event response module 1100 is used to respond to a speech translation event, continuously receive the original audio signal stream corresponding to the event, and determine multiple sequences of frames containing human voices corresponding to the original audio signal stream. The non-human voice noise reduction module 1200 is used to perform non-human voice noise reduction processing on each sequence of frames containing human voices to obtain a pure human voice frame sequence. The model inference module 1300 is used to infer the text sequence to be translated corresponding to the pure human voice frame sequence using a speech recognition model. The text translation module 1400 is used to translate the text sequence to be translated into a target text sequence according to the target language corresponding to the speech translation event.

[0091] In a further embodiment, the event response module 1100 includes: a sound detection submodule, used to perform sound activity detection based on multiple target audio frame sequences corresponding to the original audio signal stream to obtain each sound frame sequence in the original signal stream; and a human voice detection submodule, used to verify whether each sound frame sequence contains human voice features, and if so, to regard the sound frame sequence as a human voice-containing frame sequence.

[0092] In a further embodiment, the sound detection submodule includes: a feature endpoint detection submodule, used to continuously detect, one by one, whether each target audio frame sequence corresponding to the original audio signal stream contains sound activity features and its start and end activity endpoints using a preset sound activity detection model; a sequence caching submodule, used to store the target audio frame sequence into a cache when it is detected that the target audio frame sequence contains sound activity features; and a first sequence construction submodule, used to retrieve all target audio frame sequences from the cache according to the storage order and concatenate them with the currently detected target audio frame sequence to extract all corresponding sound frames and obtain a sound frame sequence when it is detected that the target audio frame sequence also contains the end activity endpoint of the sound activity feature, or when the cumulative number of target audio frame sequences containing sound activity features exceeds a preset threshold.

[0093] In a further embodiment, before the feature endpoint detection submodule, the system includes: a signal stream segmentation submodule, used to segment the original signal stream according to a preset duration to obtain multiple original signal segments in the original signal stream; a second sequence construction submodule, used to perform sliding windowing framing processing on each original signal segment according to the number of covered frames and the movement step size of the framing window to obtain an audio signal frame sequence; a third sequence construction submodule, used to perform frequency domain feature extraction processing on each audio signal frame in the audio signal frame sequence to obtain an audio frame sequence to be optimized; and a fourth sequence construction submodule, used to normalize each audio frame to be optimized in the audio frame sequence to be optimized according to a preset mean and variance to obtain a target audio frame sequence composed of all processed target audio frames.

[0094] In a further embodiment, the model inference module 1300 includes: a strategy inference submodule, used to call a speech recognition model to infer the pure human voice frame sequence based on a preset low frame rate modeling strategy, and obtain the corresponding text sequence to be translated.

[0095] In a further embodiment, after the text translation module 1400, there are: a first error verification submodule, used to verify whether there is a duplicate expression error in the target text sequence inferred this time based on all previously inferred target text sequences and the target text sequence inferred this time; and a first error correction submodule, used to correct the corresponding erroneous text in the target text sequence inferred this time when the duplicate expression error exists.

[0096] In a further embodiment, after the text translation module 1400, there is a second error verification submodule, used to verify whether there are semantic expression errors in the target text sequence inferred this time based on all previously inferred target text sequences and the target text sequence inferred this time; and a second error correction submodule, used to correct the corresponding erroneous text in the target text sequence inferred this time when the semantic expression error exists.

[0097] To address the aforementioned technical problems, embodiments of this application also provide an electronic device. For example... Figure 4The diagram shows the internal structure of an electronic device. The electronic device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store a sequence of control information. When the computer-readable instructions are executed by the processor, the processor can implement the speech translation method described in this application. The processor of the electronic device provides computing and control capabilities to support the operation of the entire electronic device. The memory of the electronic device may store computer-readable instructions, which, when executed by the processor, can cause the processor to execute the speech translation method described in this application. The network interface of the electronic device is used for communication with a terminal. Those skilled in the art will understand that… Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0098] In this execution method, the processor is used for execution. Figure 3 The system contains the specific functions of each module and its submodules. The memory stores the program code and various data required to execute these modules or submodules. The network interface is used for data transmission between the user terminal and the server. In this execution method, the memory stores the program code and data required to execute all modules / submodules in the speech translation device of this application. The server can call the server's program code and data to execute the functions of all submodules.

[0099] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the method described in any embodiment of this application.

[0100] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0101] In summary, this application enables efficient and accurate streaming speech translation, significantly improving translation robustness in acoustic environments with noise interference and mixed sound sources.

[0102] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those in the open-source operations, methods, and processes of this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0103] The above description is only a partial implementation method of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A speech translation method, characterized in that, Includes the following steps: In response to a speech translation event, continuously receive the raw audio signal stream corresponding to the event, and determine the multiple frames containing human voices corresponding to the raw audio signal stream; For each of the aforementioned human voice frame sequences, non-human voice noise reduction processing is performed on the human voice frame sequences to obtain pure human voice frame sequences. The speech recognition model is used to infer the text sequence to be translated corresponding to the pure human voice frame sequence. The text sequence to be translated is translated into a target text sequence according to the target language corresponding to the speech translation event.

2. The speech translation method according to claim 1, characterized in that, Determining the sequence of multiple frames containing human voices corresponding to the original audio signal stream includes the following steps: Based on sound activity detection of multiple target audio frame sequences corresponding to the original audio signal stream, each sound frame sequence in the original signal stream is obtained. For each audio frame sequence, check whether the audio frame sequence contains human voice features. If it does, the audio frame sequence is regarded as a human voice-containing frame sequence.

3. The speech translation method according to claim 2, characterized in that, Based on sound activity detection of multiple target audio frame sequences corresponding to the original audio signal stream, each sound frame sequence in the original signal stream is obtained, including the following steps: Using a pre-defined sound activity detection model, the system continuously detects whether each target audio frame sequence corresponding to the original audio signal stream contains sound activity features and its start and end activity endpoints. When a target audio frame sequence is detected to contain sound activity features, the target audio frame sequence is stored in the cache. When the target audio frame sequence is detected to contain the end of the sound activity feature, or when the cumulative number of target audio frame sequences containing sound activity features is continuously detected to exceed a preset threshold, all target audio frame sequences in the cache are retrieved according to the storage order and concatenated with the currently detected target audio frame sequence to extract all corresponding sound frames and obtain the sound frame sequence.

4. The speech translation method according to claim 3, characterized in that, Before continuously detecting whether each target audio frame sequence corresponding to the original audio signal stream contains sound activity features and its start and end activity endpoints using a preset sound activity detection model, the following steps are included: The original signal stream is segmented according to a preset duration to obtain multiple original signal segments in the original signal stream. For each original signal segment, sliding windowing framing is performed on the original signal segment according to the number of frames covered by the framing window and the movement step size to obtain the audio signal frame sequence; Frequency domain feature extraction is performed on each audio signal frame in the audio signal frame sequence to obtain the audio frame sequence to be optimized. The audio frames in the audio frame sequence to be optimized are normalized according to the preset mean and variance to obtain the target audio frame sequence composed of all the processed target audio frames.

5. The speech translation method according to claim 1, characterized in that, The text sequence to be translated is derived from the pure human voice frame sequence using a speech recognition model, including: Based on a preset low frame rate modeling strategy, the speech recognition model is invoked to infer the pure human voice frame sequence to obtain the corresponding text sequence to be translated.

6. The speech translation method according to claim 1, characterized in that, After translating the text sequence to be translated into the target text sequence according to the target language corresponding to the speech translation event, the following steps are included: Based on all previously inferred target text sequences and the target text sequence inferred this time, check whether there are duplicate expression errors in the target text sequence inferred this time; When the aforementioned repetitive expression error exists, the corresponding erroneous text in the target text sequence inferred this time is corrected.

7. The speech translation method according to claim 1, characterized in that, After translating the text sequence to be translated into the target text sequence according to the target language corresponding to the speech translation event, the following steps are included: Based on all previously inferred target text sequences and the target text sequence inferred this time, verify whether there are semantic expression errors in the target text sequence inferred this time; When a semantic expression error exists, the corresponding erroneous text in the target text sequence inferred this time is corrected.

8. A voice translation device, characterized in that, It includes: The event response module is used to respond to speech translation events, continuously receive the original audio signal stream corresponding to the event, and determine the multiple frames containing human voices corresponding to the original audio signal stream. The non-human voice noise reduction module is used to perform non-human voice noise reduction processing on each of the human voice frame sequences to obtain a pure human voice frame sequence. The model inference module is used to infer the text sequence to be translated corresponding to the pure human voice frame sequence using a speech recognition model; The text translation module is used to translate the text sequence to be translated into a target text sequence according to the target language corresponding to the speech translation event.

9. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images