Speech recognition method and system based on multi-model dynamic switching, and electronic device

Abstract

The present disclosure provides a speech recognition method and system based on multi-model dynamic switching, and an electronic device, relating to the technical field of speech recognition, comprising: acquiring audio data stream and corresponding sensor data stream in real time to form multi-modal data stream; extracting audio features in the audio data stream and scene features in the sensor data stream, and encoding into a unified context state vector; predicting scene information at a future time step and generating a model weight vector based on the unified context state vector; injecting the unified context state vector into a plurality of expert models in an expert model library for parallel processing, and each outputting a token probability distribution; dynamically weighting and summing the token probability distributions according to the model weight vector, and decoding the final probability distribution after fusion to generate a speech recognition text. Seamless and smooth transition between models can be achieved, ensuring high coherence of the final recognition text.

Description

Technical Field

[0001] This disclosure relates to the field of speech recognition technology, and in particular to a speech recognition method, system, and electronic device based on dynamic switching of multiple models. Background Technology

[0002] Automatic speech recognition technology, as a core component of modern human-computer interaction, is being widely applied in smart terminal devices, in-vehicle infotainment systems, smart homes, and various professional service fields.

[0003] In existing technical solutions, the system typically first uses an acoustic scene classifier to determine the user's current environment, and then loads and runs an expert model corresponding to the determination result to perform subsequent speech recognition tasks. Although existing technologies, especially those employing multiple expert models, have improved the performance of speech recognition systems in specific scenarios to some extent, they still have some shortcomings, mainly including: Model switching decisions are lagging and prone to errors: Existing scene switching decisions are typically a passive response. The system can only identify the change by analyzing audio features after the actual acoustic environment has changed, which naturally introduces a decision delay. More importantly, this decision relies entirely on a single acoustic feature, which is itself ambiguous. For example, a well-insulated, quiet car interior may be highly similar to a quiet indoor office. This decision-making mechanism, which depends on a single, ambiguous information source, is highly susceptible to misjudging the scene and selecting a suboptimal expert model from the outset, thereby reducing the accuracy of speech recognition. The model switching process is abrupt, resulting in the loss of contextual information: When an existing system decides to replace an expert model, a "hard switch" strategy is commonly used, which completely terminates the current model and starts a new one. This interruptive operation causes all established linguistic contextual information (such as the context, speech rate, and prosody of the dialogue) to be completely discarded at the moment of switching. The new model must re-analyze and rebuild the context from scratch, which can easily lead to semantic breaks, incoherence, or even recognition errors in the final recognition results near the switching point, severely impacting the smoothness of the user experience. Unable to proactively handle uncertain scenarios: When existing systems, based on acquired acoustic features, still cannot make a high-confidence judgment about the current scenario (i.e., facing uncertain or ambiguous situations), they lack effective coping mechanisms. The system cannot proactively acquire additional, more discriminative information to eliminate this ambiguity; it can only guess among multiple possible options. This passive approach makes the system vulnerable in critical, ambiguous scenarios, prone to making incorrect decisions, and further compromises its reliability in real-world, ever-changing environments.

[0004] Therefore, there is an urgent need for a speech recognition technology solution that can solve the above problems. Summary of the Invention

[0005] This disclosure provides a speech recognition method, system, and electronic device based on dynamic switching of multiple models. Its main purpose is to solve the problems of inaccurate model selection and discontinuous recognition results caused by the single switching decision basis and rigid switching mechanism in existing automatic speech recognition technology in dynamically changing scenarios.

[0006] According to a first aspect of this disclosure, a speech recognition method based on dynamic switching of multiple models is provided, comprising: Real-time acquisition of audio data streams and corresponding sensor data streams to form a multimodal data stream; Audio features from the audio data stream and context features from the sensor data stream are extracted and encoded into a unified context state vector; Based on the unified context state vector, the scene information of future time steps is predicted and the model weight vector is generated; The unified context state vector is injected into several expert models in the expert model library for parallel processing, and each model outputs a word probability distribution. The word probability distributions are dynamically weighted and summed according to the model weight vector, and the final fused probability distribution is decoded to generate speech recognition text.

[0007] According to a second aspect of this disclosure, a speech recognition system based on multi-model dynamic switching is provided, comprising: The data acquisition module is used to acquire audio data streams and corresponding sensor data streams in real time, forming a multimodal data stream; A predictive switching controller is used to extract audio features from an audio data stream and scene features from a sensor data stream, encode them into a unified context state vector; and, based on the unified context state vector, predict scene information for future time steps and generate a model weight vector. An expert model library, comprising several expert models, is used to process the unified context state vector in parallel and each model outputs a word probability distribution. The weighted model fusion decoder is used to dynamically weight and sum the word probability distributions according to the model weight vector, and the final fused probability distribution is decoded to generate speech recognition text.

[0008] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0009] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect above.

[0010] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0011] The embodiments of the present invention have the following advantages: This application integrates real-time audio streams with contextual data streams from sensors such as accelerometers, gyroscopes, and GPS. Through a predictive switching controller, it not only judges the current scene but, more importantly, predicts possible future scene changes, providing a forward-looking decision-making basis for model switching. This fundamentally solves the problem of decision lag and error susceptibility caused by the reliance on a single audio signal in traditional methods. Abandoning the traditional "hard switching" mechanism, this method runs multiple expert models in parallel and uses dynamic weight vectors generated by the predictive controller to perform real-time weighted fusion of the outputs (such as word probability distributions) of each model. This "soft switching" approach achieves a seamless and smooth transition between models, effectively avoiding recognition interruptions and inconsistent results caused by loss of context during switching. The unified context state vector generated by fusing multimodal data is synchronously injected into all parallel expert model decoders. This ensures that all models share a continuous and unified context starting point when processing audio, fundamentally solving the problem of context information loss caused by hard switching and guaranteeing the high coherence of the final recognized text. When passively acquired data (audio and sensor data) alone is insufficient to make a high-confidence judgment, the system can actively emit specific acoustic signals and analyze their echo characteristics. This ability to actively acquire environmental physical information provides the system with a final means of eliminating ambiguity, greatly enhancing its decision-making robustness and recognition accuracy in ambiguous and boundary-based scenarios.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating the steps of an embodiment of the speech recognition method based on dynamic switching of multiple models according to the present invention. Figure 2 This is a schematic diagram of the structure of a speech recognition system based on dynamic switching of multiple models according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the unified context state vector generation process in a specific example of the present invention; Figure 4 This is a schematic diagram of the uncertainty handling process for active acoustic detection in a specific example of the present invention; Figure 5 This is a schematic diagram of a shared context vector injection expert model decoder in a specific example of the present invention; Figure 6 This is a schematic diagram of the weighted model fusion decoding process in a specific example of the present invention. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] The following description, with reference to the accompanying drawings, outlines a speech recognition method, apparatus, electronic device, and storage medium based on multi-model dynamic switching according to embodiments of the present disclosure.

[0016] like Figure 1 As shown in one embodiment of this application, the speech recognition method based on dynamic switching of multiple models does not passively respond only after detecting recognition errors or environmental changes. Instead, it utilizes a predictive switching controller to predict scene information for future time steps based on the comprehensive context reflected by the two current data streams, and proactively generates a model weight vector. Each component of this vector corresponds one-to-one with multiple expert models in a heterogeneous expert automatic speech recognition model library.

[0017] Subsequently, the method invokes multiple expert models from the model library to process the audio data stream in parallel, obtaining their respective recognition results. Finally, based on the previously generated model weight vector, the method performs weighted fusion of the recognition results output by the multiple expert models to generate the final speech-recognized text. This predictive dynamic weight harmonization mechanism replaces the traditional hard switching mechanism, achieving a smooth transition during scene changes.

[0018] The method specifically includes the following steps: Step 1: Acquire audio data streams and corresponding sensor data streams in real time to form a multimodal data stream; In this step, the audio data stream is acquired in real time via one or more microphones deployed on the smart terminal device. The sensor data stream is acquired synchronously via non-acoustic sensors built into the smart terminal device, which may include a GPS system, accelerometer, and gyroscope. To ensure the effectiveness of subsequent processing, the acquired audio data stream and sensor data stream are strictly aligned using their respective timestamps to ensure that audio data at any given time corresponds to sensor data at the same time.

[0019] Step 2: Extract audio features from the audio data stream and context features from the sensor data stream, and encode them into a unified context state vector; In this step, the system performs feature extraction and joint encoding on the two received data streams. Specifically, a set of acoustic features, such as Mel-frequency cepstral coefficients (MFCC) or filter bank energy (Fbank) features, are extracted from the audio data stream.

[0020] Simultaneously, a set of contextual features is extracted from the sensor data stream. For example, geographic location coordinates are extracted from GPS data and mapped to predefined region types; the user's motion state is extracted from accelerometer and gyroscope data.

[0021] Subsequently, for each time step, the audio features of that time step are concatenated with the corresponding scene features to form a combined feature vector. The sequence of combined feature vectors from several time steps is input into a joint encoder network (e.g., a gated recurrent unit (GRU) network or a Transformer encoder structure). The joint encoder network processes the input combined feature vector sequence in time sequence and updates its internal hidden state at each time step. After processing the pre-defined sequence of combined feature vectors, the hidden state vector of the last time step output by the co-encoder network is used as the unified context state vector. The unified context state vector is a fixed-dimensional vector that can uniformly represent multimodal information; this vector is a high-dimensional abstract representation of the comprehensive context of the current time step.

[0022] Step 3: Based on the unified context state vector, predict the scene information for future time steps and generate the model weight vector; In this step, after generating or updating the unified context state vector, the model weight vector is calculated by passing it through a fully connected layer and applying the softmax function. Each component of this vector corresponds to the applicable weight of an expert model in a heterogeneous expert model library. The scene information output in this step can be used in other modules, such as the front-end noise reduction module.

[0023] The weight vector of this model is represented by the following formula: ; In the formula, This is the model weight vector; To unify the context state vector; and These are a learnable weight matrix and a bias vector, respectively. It is a normalized exponential function.

[0024] Step 4: Inject the unified context state vector into several expert models in the expert model library for parallel processing, and output the word probability distribution for each model. In this step, the expert model library is a heterogeneous expert automatic speech recognition model library. During use, multiple expert models from this heterogeneous library are invoked in parallel. The weight values ​​in the model weight vector correspond one-to-one with the expert models.

[0025] The model library includes at least: a general model with high recognition rates in standard, quiet environments; a noise-resistant model trained on noisy data and highly robust to background noise; and a dialect model optimized for the vocabulary and pronunciation patterns of specific regional accents. To ensure the continuous transmission of dialogue context information when model weights change, the generated unified context state vector is used as a shared additional context condition and synchronously input into the decoder of each expert model as the initial state or additional input for its decoding process.

[0026] Step 5: The word probability distribution is dynamically weighted and summed according to the model weight vector, and the final probability distribution after fusion is decoded to generate speech recognition text.

[0027] In this step, the system performs weighted fusion of the recognition results output by multiple expert models. Specifically, at each time step of decoding, each expert model outputs a probability distribution covering all words. The system uses the model weight vector calculated by the predictive switching controller to perform a weighted sum of these probability distribution vectors to obtain a final word probability distribution. Finally, based on this final word probability distribution sequence, a decoding algorithm (e.g., BeamSearch) is used to search for the optimal word sequence and combine them into the final speech recognition text.

[0028] The word probability distribution is dynamically weighted and summed according to the model weight vector, and represented by the following formula: ; In the formula, The total number of expert models; For the model weight vector, the one that is related to the first... The weights corresponding to each expert model; For the first The probability distribution of word terms output by an expert model; It serves as a unique index for a specific expert model within the expert automatic speech recognition model library; This represents the final probability distribution obtained after weighted fusion.

[0029] This application also includes a predictive front-end denoising step, as follows: Using the scene information output in the previous time step, noise reduction parameters are adaptively selected and configured. For example, when the scene information is vehicle-mounted, targeted wind noise and engine noise suppression parameters are configured.

[0030] This step performs noise reduction on the raw audio data stream at the current time step and uses the purified audio signal for subsequent acoustic feature extraction, thereby improving signal quality before entering the core recognition process.

[0031] like Figure 2 As shown, this embodiment provides a speech recognition system 100 employing the above-described speech recognition method based on multi-model dynamic switching. The system includes a data acquisition module 110, a predictive switching controller 120, a heterogeneous expert automatic speech recognition model library 130, and a weighted model fusion decoder 140, wherein: Data acquisition module 110 is used to acquire real-time audio data streams and sensor data streams synchronized with the audio data streams in time. In one specific implementation, data acquisition module 110 includes one or more microphones for acquiring the audio data streams; it also includes at least one non-acoustic sensor, such as a GPS module, accelerometer, or gyroscope, for acquiring the sensor data streams. The output of this module is two timestamp-aligned data streams. A predictive switching controller 120 has its input connected to the output of the data acquisition module 110. The controller 120 is configured to receive audio data streams and sensor data streams, and generate a unified context state vector based on these. Subsequently, the controller 120 generates a model weight vector and scene information based on the unified context state vector. The generation process of the model weight vector is characterized by the following formula: ; In the formula, This is the model weight vector; To unify the context state vector; and These are a learnable weight matrix and a bias vector, respectively. It is a normalized exponential function.

[0032] A heterogeneous expert automatic speech recognition model library 130 is configured to receive an audio data stream from a data acquisition module 110 and a unified context state vector from a predictive switching controller 120. The model library 130 internally contains multiple expert models, such as a general model for handling standard environments, a noise-resistant model for handling high-noise environments, and a dialect model for handling specific regional accents.

[0033] Each expert model independently processes the input audio data stream and combines it with the received unified context state vector as additional context conditions to output its own recognition result, which is a probability distribution for the next word.

[0034] A weighted model fusion decoder 140 has its inputs connected to the outputs of a predictive switching controller 120 and a heterogeneous expert automatic speech recognition model library 130, respectively. The decoder 140 is configured to receive a model weight vector generated by the controller 120 and multiple word probability distributions output by the model library 130. The decoder 140 generates a final word probability distribution by weighted summation of the received multiple word probability distributions, a process characterized by the following formula: ; In the formula, The total number of expert models; For the model weight vector and the first The weights corresponding to each expert model; For the first The probability distribution of word terms output by an expert model; It serves as a unique index for a specific expert model within the expert automatic speech recognition model library; This represents the final probability distribution obtained after weighted fusion.

[0035] Finally, the decoder 140 generates the final speech recognition text based on this final probability distribution.

[0036] In some embodiments, system 100 may further include a front-end noise reduction module 150. This module 150 is located between the data acquisition module 110 and the predictive switching controller 120 and model library 130. It is configured to receive the raw audio data stream from the data acquisition module 110 and simultaneously receive scene information output from the predictive switching controller 120. The module 150 configures its internal noise reduction parameters according to the scene information and performs noise reduction processing on the audio data stream. The purified audio signal output is then transmitted to the controller 120 and the model library 130.

[0037] In other embodiments, system 100 may further include an active acoustic detection module 160. This module 160 is connected to the predictive switching controller 120. The active acoustic detection module 160 is triggered to operate when the prediction confidence level generated by controller 120 is lower than a preset threshold. Its operation involves emitting a detection signal, acquiring environmental echoes, and extracting an acoustic fingerprint feature. This acoustic fingerprint feature is sent back to the predictive switching controller 120 to update its internal unified context state vector.

[0038] In one embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, it implements the above-described speech recognition method based on dynamic switching of multiple models.

[0039] In a specific example, the speech recognition process using this speech recognition system is as follows: Audio data streams are continuously acquired through one or more microphones deployed on smart terminal devices. For example, audio signals are digitized at a sampling rate of 16 kHz and a bit depth of 16 bits to generate raw Pulse Code Modulation (PCM) data streams.

[0040] Meanwhile, sensor data streams are synchronously acquired through non-acoustic sensors built into smart terminal devices. For example, the Global Positioning System (GPS) module provides latitude and longitude coordinates, altitude, and speed data at a preset frequency (e.g., 1 Hz); while the triaxial accelerometer and triaxial gyroscope provide real-time triaxial acceleration vector and angular velocity vector data at higher frequencies (e.g., 50 Hz or 100 Hz).

[0041] To ensure the accuracy of subsequent multimodal feature fusion, the audio data stream and sensor data stream are processed synchronously. The data acquisition module 110 adds a high-precision timestamp to each acquired audio data frame and each set of sensor readings. During subsequent processing, the system aligns the two data streams based on these timestamps to ensure that the acoustic features within any given time period are uniquely associated with the corresponding environmental and motion context features within that time period, thus forming a unified multimodal input data unit.

[0042] like Figure 3 As shown, the specific implementation method for generating the unified context state vector is as follows: The process begins by extracting features from both the audio and sensor data streams. For the audio data stream, it is segmented into multiple consecutive, partially overlapping short frames (e.g., frame length 25 milliseconds, frame shift 10 milliseconds). A window function is applied to each frame, and then it is transformed to the frequency domain using a Fast Fourier Transform. The resulting power spectrum is then filtered through a Mel filter bank and its logarithm is taken to generate a set of Mel filter bank energy (Fbank) features. This process generates an acoustic feature vector for each audio frame, ultimately forming an acoustic feature sequence.

[0043] For sensor data streams, the system extracts a set of contextual features. Specifically, latitude and longitude coordinates obtained from Global Positioning System (GPS) data are mapped into discrete location type labels by comparing them with a pre-defined geographic information database. For accelerometer and gyroscope data, the system calculates statistics such as energy and variance of the three-axis data within a short time window, and inputs these statistics into a pre-trained classifier to output labels representing the current motion state. These location type labels and motion state labels are encoded into numerical vectors, constituting the contextual features.

[0044] After the two feature extractions are completed, the system jointly encodes them to generate a unified context state vector. Specifically, for each time step, the acoustic feature vector of that time step is concatenated with the corresponding context feature vector to form a combined feature vector with a wider dimension. The sequence of these combined feature vectors is input into a joint encoder network. This encoder network can be a multi-layer gated recurrent unit (GRU) network. The GRU network processes the input combined feature vector sequence sequentially and updates its internal hidden state at each time step. After processing a sequence of a preset length, the hidden state vector of the last time step output by the GRU network is used as the unified context state vector that can uniformly represent the current acoustic and contextual information.

[0045] like Figure 4 As shown, to address the prediction uncertainty caused by the ambiguity of passively acquired sensor data, after generating a unified context state vector, the method also includes an uncertainty processing procedure based on active acoustic detection: After the predictive switching controller 120 generates a unified context state vector and calculates the preliminary model weights based on it, the system simultaneously calculates a prediction confidence score. This prediction confidence score can be quantified as the value of the largest weight component in the model weight vector. A lower largest weight value indicates that the controller cannot definitively determine which expert model is the optimal choice for the current scenario, reflecting its uncertainty regarding the current situation.

[0046] The prediction confidence level is compared with a preset numerical threshold (e.g., 0.5). When the confidence level is determined to be lower than the preset threshold, the system triggers the active acoustic detection module 160. This threshold can be preset based on experimental data to achieve a balance between decision reliability and system energy consumption.

[0047] Once triggered, the active acoustic detection module 160 emits a detection signal with a specific frequency and waveform into the surrounding environment via an acoustic transducer. The frequency of this detection signal is set in a band insensitive to the human ear, such as the ultrasonic band above 20kHz, to avoid interfering with the user or the normal speech recognition process. Subsequently, a high-frequency microphone collects the environmental echo formed after the detection signal is reflected and scattered in the environment.

[0048] The system processes the acquired environmental echoes to extract one or more acoustic fingerprint features. For example, by calculating the time required for the echo signal energy to decay by 60 dB, the reverberation time can be obtained, a feature that can effectively distinguish between open and enclosed spaces. Furthermore, the spectral envelope of the echo signal can be analyzed to obtain information about the spatial geometry and surface material.

[0049] The extracted acoustic fingerprint features are quantized into a numerical vector and concatenated with the original contextual features extracted from the sensor data stream. This concatenated enhanced contextual feature vector replaces the original contextual feature vector and, along with the acoustic features, is re-inputted into the joint encoder network to update and generate a more explicit unified contextual state vector. Subsequently, the predictive switching controller 120 re-outputs the model weight vector based on this updated vector, thereby improving the accuracy of the final decision.

[0050] like Figure 5 As shown, the implementation method of the shared context vector injection expert model decoder is as follows: In this step, the system calls multiple expert models contained in the heterogeneous expert automatic speech recognition model library 130 in parallel. The model library is heterogeneous in composition and includes at least: a general model trained on large-scale, multi-domain, standard Mandarin speech data, which has high recognition performance in general scenarios; The noise-resistant model is trained on speech data mixed with various types of noise at different signal-to-noise ratios, making it robust to background noise. Dialect models are trained on accents, vocabulary, and pronunciation paradigms from specific regions to handle dialect recognition tasks.

[0051] These expert models receive audio data streams (or purified audio signals processed by the front-end noise reduction module 150) in parallel as their main inputs.

[0052] To maintain context continuity during dynamic adjustment of model weights, when these expert models are invoked, a unified context state vector generated by the predictive switching controller 120 is synchronously passed to the decoder portion of each expert model in the model library 130 as a shared additional context condition.

[0053] In a specific implementation, if the expert model adopts an encoder-decoder architecture based on a recurrent neural network, this unified context state vector is used as the initial hidden state of the recurrent neural network (such as LSTM or GRU) in the decoder of the model.

[0054] This operation ensures that each expert model's decoder is placed in a state containing complete contextual information before it begins generating word sequences, thus ensuring that all models operate from a unified contextual starting point.

[0055] like Figure 6 As shown, the implementation method of the weighted model fusion decoding process for active acoustic detection is as follows: This step is performed by the weighted model fusion decoder 140. The input to the decoder consists of two parts: one is the model weight vector generated by the predictive switching controller 120, and the other is multiple recognition results output in parallel by the heterogeneous expert automatic speech recognition model library 130.

[0056] In this embodiment, the recognition result of each expert model is specifically the probability distribution of all possible next tokens in the vocabulary at the current decoding time step.

[0057] At each time step of the decoding process, the weighted model fusion decoder 140 first performs a weighted fusion of the word probability distributions output by multiple expert models. This fusion is achieved by weighted summation of the probability distribution vectors, and its calculation process is represented by the following formula: ; In the formula, The total number of expert models; For the model weight vector and the first The weights corresponding to each expert model; For the first The probability distribution of word terms output by an expert model; It serves as a unique index for a specific expert model within the expert automatic speech recognition model library; This represents the final probability distribution obtained after weighted fusion.

[0058] The calculated final word probability distribution is a new probability distribution that integrates the judgments of all expert models and is dynamically weighted according to the current scenario. This distribution is used to guide subsequent text sequence generation.

[0059] Finally, based on the final word probability distribution sequence generated sequentially at each decoding time step, the system employs a beam search algorithm to generate the final speech recognition text. In this algorithm, the system retains the sequence of word probability distributions generated at each decoding step. indivual( The candidate text sequence with the highest probability (i.e., the preset beamwidth).

[0060] For each of these N candidate sequences, we attempt to expand all the lexical units in the vocabulary and use the final lexical unit probability distribution to calculate the cumulative probability of the newly generated longer sequence. Then, we select again from all possible new sequences. The sequence with the highest cumulative probability is selected, and this process is repeated until a complete sentence is generated. The candidate sequence with the highest cumulative probability is selected as the final speech recognition text output.

[0061] In another embodiment, a predictive front-end denoising step may be added before step S220, which is performed by the front-end denoising module 150 in system 100.

[0062] In this step, the front-end noise reduction module 150 receives the raw audio data stream of the current time step from the data acquisition module 110. Simultaneously, it also receives scene information output by the predictive switching controller 120 when processing data from the previous time step. This scene information is a clear identifier of the acoustic environment predicted in the previous time step.

[0063] The front-end noise reduction module 150 has multiple noise reduction parameter sets pre-stored internally. Each parameter set is optimized for a specific acoustic scenario. For example, one parameter set contains specific filter coefficients for suppressing low-frequency engine noise and wind noise; another parameter set may contain algorithm configurations for handling crowd noise and sudden noise.

[0064] The front-end noise reduction module 150 selects and loads a set of noise reduction parameters corresponding to the scene from multiple parameter sets stored internally, based on the received scene information. Subsequently, the module applies the selected parameters to perform noise reduction processing on the raw audio data stream at the current time step. This processing can be based on traditional signal processing algorithms (such as spectral subtraction or Wiener filtering) or on a deep learning speech enhancement model.

[0065] The output of this step is a purified audio signal in which background noise corresponding to the predicted scene has been effectively suppressed. This purified audio signal is then passed to a predictive switching controller 120 (for extracting acoustic features) and a heterogeneous expert automatic speech recognition model library 130 (for performing speech recognition) to replace the original audio data stream as input to these subsequent steps, thereby improving the quality of the input signal before the core recognition process begins.

[0066] The embodiments of the present invention have the following advantages: This application integrates real-time audio streams with contextual data streams from sensors such as accelerometers, gyroscopes, and GPS. Through a predictive switching controller, it not only judges the current scene but, more importantly, predicts possible future scene changes, providing a forward-looking decision-making basis for model switching. This fundamentally solves the problem of decision lag and error susceptibility caused by the reliance on a single audio signal in traditional methods. Abandoning the traditional "hard switching" mechanism, this method runs multiple expert models in parallel and uses dynamic weight vectors generated by the predictive controller to perform real-time weighted fusion of the outputs (such as word probability distributions) of each model. This "soft switching" approach achieves a seamless and smooth transition between models, effectively avoiding recognition interruptions and inconsistent results caused by loss of context during switching. The unified context state vector generated by fusing multimodal data is synchronously injected into all parallel expert model decoders. This ensures that all models share a continuous and unified context starting point when processing audio, fundamentally solving the problem of context information loss caused by hard switching and guaranteeing the high coherence of the final recognized text. When passively acquired data (audio and sensor data) alone is insufficient to make a high-confidence judgment, the system can actively emit specific acoustic signals and analyze their echo characteristics. This ability to actively acquire environmental physical information provides the system with a final means of eliminating ambiguity, greatly enhancing its decision-making robustness and recognition accuracy in ambiguous and boundary-based scenarios.

[0067] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0068] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0069] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0070] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatial relative descriptions used herein will be interpreted accordingly.

[0071] In the detailed description above, reference has been made to the accompanying drawings, which form part of this document. In the drawings, similar symbols typically identify similar parts unless the context otherwise indicates otherwise. The illustrated embodiments described in the detailed specification, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A speech recognition method based on dynamic switching of multiple models, characterized in that, include: Real-time acquisition of audio data streams and corresponding sensor data streams to form a multimodal data stream; Audio features are extracted from the audio data stream and scene features from the sensor data stream, and encoded into a unified context state vector. The audio features are Mel frequency cepstral coefficients or filter bank energy features. The scene features include one or more of geographic location coordinates, region type, or motion state. The audio features and scene features are input into a joint encoder network to generate the unified context state vector, which is used to abstractly represent the comprehensive scene at the current time step. Based on the unified context state vector, scene information for future time steps is predicted and a model weight vector is generated; the prediction of scene information for future time steps and the generation of model weight vectors are represented by the following formula: ； In the formula, This is the model weight vector; To unify the context state vector; and These are a learnable weight matrix and a bias vector, respectively. It is a normalized exponential function; The unified context state vector is injected into several expert models in the expert model library for parallel processing, and each model outputs a word probability distribution. The expert model library is a heterogeneous expert automatic speech recognition model library, which includes several expert models. The expert models include a general model for handling standard environments, a noise-resistant model for handling high-noise environments, and a dialect model for handling specific regional accents. The weight values ​​in the model weight vector correspond one-to-one with the expert models. The word probability distributions are dynamically weighted and summed according to the model weight vector, and the final fused probability distribution is decoded to generate speech recognition text.

2. The speech recognition method based on multi-model dynamic switching according to claim 1, characterized in that, The real-time acquisition of audio data streams and corresponding sensor data streams to form a multimodal data stream includes: The audio data stream is acquired using several microphones; The sensor data stream is acquired using one or more of a Global Positioning System module, an accelerometer, or a gyroscope. The audio data stream and the sensor data stream are time-stamp aligned to generate the multimodal data stream.

3. The speech recognition method based on multi-model dynamic switching according to claim 1, characterized in that, The step of inputting the audio features and the context features into a sequence and then into a joint encoder network to generate the unified context state vector includes: For each time step, the audio features of that time step are concatenated with the corresponding context features to form a combined feature vector. The sequence of combined feature vectors from several time steps is then input into the joint encoder network. The joint encoder network processes the combined feature vector sequence of the input sequentially and updates its internal hidden state at each time step; After processing the combined feature vector sequence of a preset length, the hidden state vector of the last time step output by the joint encoder network is used as the unified context state vector.

4. The speech recognition method based on multi-model dynamic switching according to claim 1, characterized in that, The dynamic weighted summation of the word probability distribution based on the model weight vector is represented by the following formula: ； In the formula, The total number of expert models; For the model weight vector, the one that is related to the first... The weights corresponding to each expert model; For the first The probability distribution of word terms output by an expert model; It serves as a unique index for a specific expert model within the expert automatic speech recognition model library; This represents the final probability distribution obtained after weighted fusion.

5. The speech recognition method based on multi-model dynamic switching according to claim 1, characterized in that, Also includes: Based on the scene information from the previous time step, select the appropriate noise reduction parameters and perform noise reduction processing on the audio data stream.

6. A speech recognition system employing the speech recognition method based on multi-model dynamic switching as described in any one of claims 1 to 5, characterized in that, include: The data acquisition module is used to acquire audio data streams and corresponding sensor data streams in real time, forming a multimodal data stream; A predictive switching controller is used to extract audio features from an audio data stream and scene features from a sensor data stream, encode them into a unified context state vector; and, based on the unified context state vector, predict scene information for future time steps and generate a model weight vector. An expert model library, comprising several expert models, is used to process the unified context state vector in parallel and each model outputs a word probability distribution. The weighted model fusion decoder is used to dynamically weight and sum the word probability distributions according to the model weight vector, and the final fused probability distribution is decoded to generate speech recognition text.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the speech recognition method based on multi-model dynamic switching as described in any one of claims 1-5.

Images