Adaptive speech control system and method for low bandwidth and noisy environments

Abstract

This invention discloses an adaptive voice control system and method for low-bandwidth and noisy environments, belonging to the field of speech recognition and intelligent control technology. It includes: a noise reduction processing module for enhancing the original audio signal; an environmental perception module for detecting speech activity and monitoring network status based on the enhanced noise-reduced signal; a collaborative control module for dynamically and collaboratively deciding on the target encoding bitrate and speech recognition model version by querying a built-in mapping table based on the perceived information; a dynamic encoding module for compressing the enhanced noise-reduced signal at the target bitrate; and a lightweight inference module for loading the model version to recognize the enhanced noise-reduced signal and generate control commands. This invention effectively solves the technical challenge of simultaneously ensuring low latency and high accuracy in voice control systems under dynamically changing low-bandwidth and high-noise environments by introducing real-time collaborative optimization of noise reduction preprocessing and encoding / recognition strategies. It is suitable for complex scenarios such as smart homes and industrial control.

Description

Technical Field

[0001] This invention relates to the fields of speech recognition and intelligent control technology. More specifically, this invention relates to an adaptive speech control system and method for low-bandwidth and noisy environments. Background Technology

[0002] Voice control systems, as an important means of human-computer interaction, have been widely used in smart homes, industrial control, and in-vehicle infotainment. These systems typically include two core processing stages: voice signal compression encoding and transmission, and voice signal recognition and command understanding.

[0003] In real-world deployment environments, especially in scenarios such as the Internet of Things (IoT) and mobile edge computing, voice control systems often face two severe challenges: unstable network transmission conditions and limited bandwidth resources, and background noise interference in the physical environment. These dynamically changing external factors pose a direct threat to the system's real-time performance and accuracy.

[0004] In existing technologies, to address low-bandwidth transmission requirements, audio signals are typically compressed using speech encoders with fixed or limited bitrates (such as OPUS encoders) to reduce data volume. Simultaneously, to operate on resource-constrained edge devices, lightweight speech recognition models (such as Transformer-based models) that have undergone pruning, quantization, and other compression techniques are used for localized inference. However, the design of such systems often suffers from the following inherent drawbacks: First, encoding strategies are typically static or semi-static. The system may choose a fixed encoding bitrate based on an initial network assessment, or allow manual switching between a few preset bitrates. When available network bandwidth fluctuates in real time (which is extremely common in wireless network environments), fixed bitrate encoding can lead to two disadvantages: if the encoding bitrate is set too high, it can cause a large backlog or loss of data packets when bandwidth is tight, increasing transmission latency or even interrupting communication; if the encoding bitrate is set too low, it cannot fully utilize network resources to obtain higher quality audio when bandwidth is sufficient, resulting in resource waste. The root cause is that existing solutions lack continuous, fine-grained real-time awareness of network transmission status, and a dynamic encoding decision-making mechanism based on this awareness.

[0005] Secondly, the selection and loading mechanism of speech recognition models are often fixed. Systems typically deploy a single model that performs optimally on the training set on average. However, different environmental noises (such as steady-state air conditioning noise, non-steady-state keyboard clicks, and sudden human voice interference) mask speech features to varying degrees, and the computational complexity and recognition accuracy of the recognition model differ under different noise levels. Furthermore, variations in network round-trip latency directly affect the system's tolerable backend processing time. Under low-latency requirements (such as when network latency is already high), a less computationally intensive model may be needed for a faster response, but this may sacrifice accuracy; conversely, under good network conditions, a more complex model can be used to pursue a higher recognition rate. Existing solutions struggle to automatically select the most suitable model version under dynamically changing noise environments and network latency constraints, resulting in system performance either failing to meet real-time requirements or experiencing a significant drop in recognition accuracy under harsh acoustic environments.

[0006] A deeper problem lies in the fact that the two core components mentioned above, encoding and recognition, are typically optimized and controlled independently in traditional systems. The encoding module aims to ensure efficient and reliable transmission of speech signals over the network, while the recognition module aims to accurately extract semantics from the received signal. Although there is a sequence to their execution, their control strategies are fragmented. For example, the encoder may use a high compression ratio to adapt to low bandwidth, potentially introducing speech distortion that is precisely the type the currently selected recognition model is insensitive to, thus increasing the recognition error rate. Conversely, the high-quality audio desired to improve the recognition rate may not be achievable due to insufficient network bandwidth. This fragmented and fixed control strategy, when faced with complex operating conditions where network bandwidth and background noise change dynamically simultaneously, makes it difficult to maintain stable overall system performance (manifested as end-to-end response latency and final command recognition accuracy). It easily leads to a "one-for-one" situation, where meeting latency requirements sacrifices accuracy, or ensuring accuracy results in unacceptable latency.

[0007] Previous attempts to solve these problems have faced numerous difficulties. On the one hand, achieving real-time and accurate network status and noise environment perception itself consumes significant computing and communication resources; designing a low-overhead perception mechanism on resource-constrained edge devices is a major challenge. On the other hand, encoding and recognition are two distinct processes, and establishing a collaborative relationship between them requires a unified decision-making framework that balances multiple conflicting objectives (such as bandwidth usage, voice quality, computational complexity, recognition accuracy, and response time). Designing a mechanism that can integrate multi-dimensional real-time information and make globally optimal or near-optimal collaborative decisions, while ensuring that the decision-making process itself is efficient and stable without excessively increasing system complexity, has long been a technical challenge that has not yet been properly resolved. Summary of the Invention

[0008] One object of the present invention is to solve at least the above-mentioned problems and to provide at least the advantages that will be described later.

[0009] To achieve these objectives and other advantages according to the present invention, an adaptive voice control system for low-bandwidth and noisy environments is provided, comprising: The noise reduction processing module receives the original audio signal collected by the audio acquisition device at its input end and outputs the noise-reduced and enhanced audio signal at its output end. An environment perception module, whose input is connected to the output of a noise reduction processing module, is used to receive the noise-reduced and enhanced audio signal. The environment perception module includes a voice activity detection submodule and a network status monitoring submodule. The voice activity detection submodule is used to analyze the audio signal and output a voice activity flag indicating whether the current frame is valid voice. The network status monitoring submodule is used to obtain the round-trip delay and available bandwidth estimate of the current network transmission in real time. The cooperative control module has its first input terminal connected to the output terminal of the environmental perception module. The cooperative control module is used to receive voice activity flags, round-trip time delay, and available bandwidth estimates. The collaborative control module has a built-in bitrate mapping table and a model performance mapping table. The collaborative control module queries the bitrate mapping table based on the voice activity flags and the available bandwidth estimate to determine the target bitrate and generate encoding control instructions. The collaborative control module queries the model performance mapping table based on the round-trip time and available bandwidth estimates to determine the speech recognition model version that needs to be enabled and generates a model selection instruction. The dynamic encoding module has its input end connected to the output end of the noise reduction processing module and the first output end of the collaborative control module; the dynamic encoding module is configured with an OPUS encoder containing multiple bit rate values; the dynamic encoding module is used to select the corresponding target bit rate in the OPUS encoder according to the encoding control command, and to compress and encode the noise-reduced and enhanced audio signal. The lightweight inference module has its input connected to the output of the noise reduction processing module and the second output of the collaborative control module. The lightweight inference module is equipped with a Transformer speech recognition model that has been pruned and quantized. The lightweight inference module is used to load the corresponding speech recognition model according to the model selection instruction, and to perform feature extraction and semantic recognition on the noise-reduced and enhanced audio signal to generate structured control instructions. A communication interface, which is used to send structured control commands to the controlled device.

[0010] Preferably, the collaborative control module also includes: The smoothing filter unit, whose input is connected to the output of the network status monitoring submodule, is used to smooth the continuous available bandwidth estimates output by the network status monitoring submodule to obtain the smoothed bandwidth value. The collaborative control module queries the bitrate mapping table based on the voice activity flag and the smoothed bandwidth value.

[0011] Preferably, it also includes: The privacy-preserving training module, which is connected to the lightweight inference module, is used to train the speech recognition model based on local speech data, obtain local model update parameters, and add random noise to the local model update parameters based on the differential privacy mechanism to generate noisy model parameters. The node selection unit, whose input is connected to the output of the network status monitoring submodule, is used to compare the estimated available bandwidth with a preset bandwidth threshold and the round-trip time with a preset delay threshold. When the estimated available bandwidth is not lower than the bandwidth threshold and the round-trip time does not exceed the delay threshold, the node selection unit marks the current device as a qualified participating node. The privacy protection training module is configured to upload the noisy model parameters to the cloud server when the current device is marked as a qualified participating node. The cloud server is used to aggregate the noisy model parameters from multiple devices, generate global model update parameters, and distribute them. The privacy-preserving training module is also used to receive the distributed global model update parameters to update the speech recognition model in the lightweight inference module.

[0012] Preferably, the noise reduction processing module includes: The adaptive Wiener filter submodule is used to perform the first stage of noise reduction on the original audio signal and output the first noise-reduced signal. A noise classifier, connected to the output of the adaptive Wiener filter submodule, is used to analyze the first noise-reduced signal and identify the dominant noise type. The generative adversarial network (GAN) noise reduction submodule is connected to the output of the adaptive Wiener filter submodule and the noise classifier. The GAN noise reduction submodule is pre-configured with multiple noise reduction generators corresponding to various noise types. The GAN noise reduction submodule is used to switch and enable the corresponding noise reduction generator according to the dominant noise type identified by the noise classifier, perform second-level noise reduction processing on the first noise reduction signal, and output the noise-enhanced audio signal.

[0013] Preferably, the collaborative control module also includes: The coding parameter smoothing control unit is equipped with a state-holding timer and a progressive switching strategy; wherein, when the cooperative control module determines the target bitrate based on the voice activity flag and the available bandwidth estimate, the coding parameter smoothing control unit is configured to perform the following operations: If the target bitrate differs from the current bitrate of the dynamic encoding module, then start the state hold timer; Only after the target bitrate has been maintained for a period of time exceeding the preset stable threshold of the state hold timer will the dynamic encoding module be triggered to gradually switch from the current encoding bitrate to the target bitrate according to the gradual switching strategy. If the target bitrate determined by the collaborative control module changes within the stable threshold time, the state hold timer is reset and the judgment is re-executed based on the newly determined target bitrate.

[0014] Preferably, it also includes: The mapping table online optimization module is connected to the collaborative control module, and is configured as follows: Collect performance feedback data during actual system operation. The performance feedback data includes at least the end-to-end response latency and speech recognition accuracy measured under the control of the collaborative control module when using a specific target bitrate and a specific speech recognition model version combination. Based on the collected performance feedback data, the mapping relationships in the bitrate mapping table and / or model performance mapping table within the collaborative control module are dynamically optimized and updated through regression analysis. The collaborative control module is further configured to determine the target bitrate and speech recognition model version based on the optimized and updated mapping table.

[0015] Preferably, the audio acquisition device includes a multi-microphone array, which is used to acquire multiple raw audio signals and execute a beamforming algorithm to generate a beam of directional enhanced speech signal, which is the audio signal received by the noise reduction processing module.

[0016] Preferably, the lightweight inference module adopts an asynchronous pipeline architecture, which decomposes the speech signal processing flow into multiple stages, including feature extraction, semantic recognition, and structured control instruction generation, and enables these multiple stages to be executed concurrently in time.

[0017] Preferably, it also includes: The cloud-based model building module connects to the lightweight inference module via a communication network and provides a basic speech recognition model to the lightweight inference module. The basic speech recognition model is pre-trained by the cloud server using large-scale general speech data. The edge model adaptation module is built into the lightweight inference module and is equipped with a knowledge distillation engine. The edge model adaptation module is used to: receive the basic speech recognition model from the cloud model building module, and use the local speech dataset to perform knowledge transfer and model compression on the basic speech recognition model through the knowledge distillation engine to generate a lightweight speech recognition model deployed in the lightweight inference module. The online model parameter adjustment module has its input connected to the output of the online mapping table optimization module, and its output connected to the lightweight inference module. The online model parameter adjustment module is used to generate model fine-tuning instructions based on the performance feedback data output by the online mapping table optimization module, and drive the lightweight inference module to make lightweight adjustments to the parameters of the currently loaded speech recognition model.

[0018] Preferably, it also includes: The transmission enhancement module, whose input is connected to the output of the dynamic encoding module, is used to apply forward error correction coding to the encoded data packets and perform error control based on automatic retransmission requests, including detecting data packet loss and triggering retransmission. The instruction mapping module has its input end connected to the output end of the lightweight inference module to receive structured control instructions, and its output end connected to the communication interface. The instruction mapping module has an instruction mapping relationship library and a protocol parsing unit, which is used to query the instruction mapping relationship library according to the identifier of the target controlled device, and convert the structured control instructions into device control instructions that conform to the communication protocol format adopted by the target controlled device through the protocol parsing unit.

[0019] An adaptive voice control method for low-bandwidth and noisy environments is provided, comprising the following steps: Noise reduction processing steps: Perform noise reduction processing on the original audio signal and output the noise-reduced and enhanced audio signal; Environmental awareness steps: Based on the noise-reduced and enhanced audio signal, obtain the speech activity flag indicating whether the current frame is valid speech, and obtain the round-trip delay and available bandwidth estimate of the current network transmission in real time; Collaborative decision-making steps: Based on the speech activity flags and available bandwidth estimates, query the bitrate mapping table to determine the target bitrate and generate encoding control instructions; simultaneously, based on the round-trip time and available bandwidth estimates, query the model performance mapping table to determine the speech recognition model version to be used and generate model selection instructions. Dynamic encoding steps: According to the encoding control instructions, select the corresponding target bit rate in the OPUS encoder and compress and encode the noise-reduced and enhanced audio signal; Lightweight inference steps: Based on the model selection instruction, load the corresponding pruned and integer quantized Transformer speech recognition model, extract features and perform semantic recognition on the noise-reduced and enhanced audio signal, and generate structured control instructions; Command transmission steps: The structured control commands are sent to the controlled device through the communication interface.

[0020] The present invention has at least the following beneficial effects: First, this invention provides a high signal-to-noise ratio input signal to the system by using a noise reduction processing module as a necessary pre-processor. Based on this, the collaborative control module dynamically and collaboratively decides the encoding bitrate and recognition model version based on a comprehensive analysis of the voice activity markers output by the environmental perception module and network status information. This architecture of first denoising and then intelligently coordinating fundamentally improves the signal quality baseline of the system in harsh acoustic environments and enables the strategies of both encoding compression and recognition computation to be globally optimal or near-optimal based on real-time bandwidth and noise conditions. Thus, the system can autonomously adaptively balance transmission efficiency, computational resources, and recognition accuracy, significantly alleviating the contradiction between low latency and high accuracy in dynamic and complex environments, and greatly enhancing overall robustness and scene adaptability.

[0021] Secondly, by processing the estimated available network bandwidth through a smoothing filter unit and managing bitrate switching with a coding parameter smoothing control unit combined with a state-holding timer and a progressive switching strategy, the system can effectively filter out interference from instantaneous network jitter and measurement noise. This avoids unnecessary and frequent switching of the coding bitrate and recognition model due to short-term fluctuations, reduces the resulting signaling overhead and audio quality jitter, ensures the consistency of user experience and the stability of system operation, and mitigates the impact on the network link.

[0022] Third, a two-stage cascaded denoising architecture consisting of adaptive Wiener filtering and a generative adversarial network denoising guided by a noise classifier is adopted, achieving a refined and targeted denoising strategy. This scheme overcomes the shortcomings of single algorithms in generalizing to complex and varied noise, and can more effectively extract and enhance speech features from strong background noise. It not only provides a more efficient signal source for subsequent encoding, but also provides a cleaner and less distorted input for the speech recognition model, laying a solid foundation for maintaining a high recognition rate in low signal-to-noise ratio environments.

[0023] Fourth, by combining a privacy-preserving training module with a node selection unit and an online mapping table optimization module, the system has built a continuous self-evolution capability. While protecting user data privacy, the system utilizes a federated learning framework to optimize the model and dynamically adjusts the collaborative strategy based on actual operational feedback. This enables the system to adapt to long-term changes in the network environment, device performance degradation, and new noise patterns, achieving a leap from static configuration to dynamic adaptation and maintaining optimal long-term performance.

[0024] Fifth, through the transmission enhancement module and command mapping module, the system shields complexity at the communication and control levels. Forward error correction and automatic retransmission mechanisms ensure reliable transmission of commands in unreliable networks; while command mapping and protocol parsing break down protocol barriers between heterogeneous devices. The combination of these two mechanisms ensures the reliable implementation of voice control intentions in various real-world and complex IoT environments, greatly improving the system's practicality and robustness.

[0025] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the architecture and signal flow of the adaptive voice control system for low-bandwidth and noisy environments of the present invention. Figure 2 This is a schematic diagram illustrating the sequential interaction of each module in the adaptive voice control system of the present invention during operation. Figure 3 This is a schematic diagram of the two-stage cascaded noise reduction process of the noise reduction processing module of the present invention; Figure 4 The following is a detailed flowchart of the noise analysis algorithm in this invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0028] To facilitate the measurement of system performance, the "end-to-end response latency" mentioned in this invention specifically refers to the time interval from the moment the audio acquisition device begins capturing an audio frame containing a valid voice command to the moment the communication interface sends the corresponding control command to the controlled device. This latency encompasses all key aspects of signal processing, environmental perception, collaborative decision-making, encoding and compression (if applicable), speech recognition, and command generation and mapping.

[0029] like Figures 1 to 4As shown, this invention provides an adaptive voice control system for low-bandwidth and noisy environments. In one core implementation, the system includes the following modules, which cooperate according to the following signal flow: An audio acquisition device (such as a microphone array) captures the raw audio signal from the environment and sends it to a noise reduction processing module. The noise reduction processing module performs noise reduction and enhancement processing on the raw audio signal, outputting an audio signal with a significantly improved signal-to-noise ratio (hereinafter referred to as the "noise reduction and enhancement signal").

[0030] The output of the noise reduction module is simultaneously fed to three downstream modules: the environmental perception module, the dynamic coding module, and the lightweight inference module.

[0031] The system of this invention adopts a four-layer collaborative architecture to achieve end-to-end optimization from voice acquisition to control execution: Voice acquisition layer: Composed of audio acquisition devices (such as microphone arrays) and their preliminary signal processing (such as beamforming), responsible for capturing and initially enhancing the voice signal in the target direction.

[0032] The data processing layer includes a noise reduction module, an environmental perception module, a collaborative control module, and a lightweight inference module. It is responsible for signal purification, environmental perception, intelligent decision-making, and semantic understanding. Core computation is performed at the edge to ensure low latency and privacy. Within this layer, the lightweight inference module identifies the noise-reduced and enhanced signal and generates structured control commands.

[0033] Narrowband transmission layer: This layer includes a dynamic encoding module and a transmission enhancement module. Its main function is to ensure reliable and low-latency delivery of control commands. The dynamic encoding module can compress and encode audio signals according to a strategy (e.g., for local logging or selective uploading when network conditions permit), while the transmission enhancement module is responsible for applying reliability enhancement mechanisms such as forward error correction and retransmission to the data that needs to be transmitted over the network (mainly structured control commands, and secondly, optional uploaded encoded audio) to cope with low bandwidth and unstable network environments.

[0034] Control execution layer: Includes instruction mapping module and communication interface, responsible for converting the identified intent into instructions that can be executed by specific devices, and sending them across protocols to realize the final implementation of control intent.

[0035] This four-layer architecture, through clear interfaces and signal flow connections, ensures the stable and efficient operation of the system under multiple constraints such as low bandwidth, high noise, low latency, and privacy protection. The main workflow of the system is as follows: speech recognition is completed at the edge and commands are generated, and then the commands are reliably delivered to the controlled device through the narrowband transmission layer and the control execution layer; the compression encoding of the audio signal serves auxiliary functions (such as logging and training data collection) without affecting the real-time performance and low latency characteristics of the main workflow.

[0036] The environmental perception module receives the noise-reduced and enhanced signal and performs two perception tasks: first, it analyzes the signal through its built-in speech activity detection submodule and outputs a speech activity flag indicating whether the current frame is valid speech; second, it obtains the estimated round-trip time and available bandwidth of the current network transmission in real time through its network status monitoring submodule. This perception information is sent to the cooperative control module.

[0037] The collaborative control module has a built-in bitrate mapping table and a model performance mapping table. It queries the bitrate mapping table based on the received speech activity flags and the (smoothed) available bandwidth estimate to determine the target coding bitrate and generate corresponding coding control instructions. Simultaneously, it queries the model performance mapping table based on the round-trip time and available bandwidth estimate to determine the current optimal speech recognition model version and generate a model selection instruction.

[0038] The dynamic encoding module receives the noise reduction and enhancement signal as well as encoding control commands from the collaborative control module. This module is equipped with an OPUS encoder that supports multiple bit rates. Based on the commands, it selects the corresponding target bit rate, compresses and encodes the noise reduction and enhancement signal, and generates encoded data packets suitable for network transmission.

[0039] The lightweight inference module receives the same noise-reduced and enhanced signal as well as model selection instructions from the collaborative control module. This module deploys multiple versions of the Transformer speech recognition model that have undergone pruning and integer quantization. Based on the instructions, it loads the specified model version, performs feature extraction and semantic recognition on the noise-reduced and enhanced signal, and ultimately generates structured device control instructions.

[0040] Structured control commands are sent to the target controlled device through the communication interface to complete the voice control closed loop.

[0041] In one specific embodiment of the present invention, the noise reduction processing module employs a two-stage cascaded noise reduction strategy. The first stage uses adaptive Wiener filtering to perform preliminary broadband noise suppression on the original audio signal, outputting a first noise-reduced signal. The second stage employs a generative adversarial network (GAN) noise reduction submodule. This submodule analyzes the first noise-reduced signal through a noise classifier, identifying the dominant noise type in the current environment (such as steady-state industrial noise, sudden human voice interference, etc.). The core of the noise classifier is to execute a real-time noise analysis process: first, initial noise estimation based on an adaptive quantile method is performed on the input signal to avoid the delay caused by traditional sorting; then, multi-dimensional features such as signal-to-noise ratio, spectral flatness, and spectral difference are calculated; finally, these features are fused through a likelihood ratio test, iteratively updated, and an accurate speech presence probability q is output. This probability q is not only used for noise classification but also directly guides the gain adjustment strategy of the subsequent noise reduction generator (e.g., reducing the suppression intensity to preserve fidelity when the speech probability is high, and increasing the suppression to clean up when the noise probability is high), thereby achieving high-precision speech enhancement in complex and variable noise environments. Based on the identification results, one of the pre-set noise reduction generators specifically trained for different noise types is selected and activated to perform a second-level deep noise reduction process on the first noise-reduced signal, ultimately outputting a noise-enhanced signal. This two-stage targeted noise reduction scheme effectively overcomes the problem of insufficient generalization ability of a single algorithm in complex noise environments, providing a high-quality input signal for subsequent processing.

[0042] The specific process of this noise analysis is as follows: Figure 4 As shown, the process begins with the original audio input signal y(t), goes through the following core steps, and finally outputs the speech presence probability q used to guide noise reduction: The input signal y(t) receives the raw time-domain audio signal from the audio acquisition device.

[0043] Initial noise estimation: The input signal is frequency-domain transformed and a preliminary noise spectrum is estimated to obtain the frequency-domain composite signal Y. k (m) =X k (m) + N k (m), where X k (m) represents the frequency domain speech component, N k (m) represents the frequency domain noise component, providing a basis for subsequent decomposition and feature calculation.

[0044] Signal decomposition: from the frequency domain composite signal Y k The target speech component X is separated from (m). k (m) and noise component N k (m) provides input for multi-dimensional feature calculation.

[0045] Multi-dimensional feature calculation: Calculate the signal-to-noise ratio: based on the separated Xk (m) and N k (m) quantizes the energy ratio of speech to noise in the current frame signal to obtain the feature σ. k (m), ρ k (m).

[0046] Calculate spectral flatness: using noise component N k (m) Analyze the uniformity of the spectral distribution (noise is usually flatter), reflecting the noise characteristics of the signal.

[0047] Calculate the spectral difference: based on noise component N k (m) quantifies the difference between the input signal and the noise template learned by the system by comparing the two.

[0048] Feature fusion (LRT mean feature generation): Based on the likelihood ratio test (LRT), the signal-to-noise ratio features are fused to generate robust LRT mean classification features, which, together with spectral flatness and spectral difference features, provide a basis for decision-making.

[0049] Speech / Noise Probabilistic Model Update: Inputting three types of features—spectral flatness, LRT mean features, and spectral differences—into the probabilistic model, an adaptive update strategy is used to dynamically optimize the model parameters, accurately characterizing the statistical properties of speech and noise.

[0050] Output speech existence probability q: The final speech existence probability q output by the model will be directly fed into the subsequent noise reduction gain control unit (such as Wiener filter) to achieve accurate noise reduction based on probability: when the probability is high, the suppression is weakened to preserve the speech, and when the probability is low, the suppression is strengthened to eliminate noise.

[0051] The algorithm shown in the figure is the core logic of the noise reduction processing module (especially the noise classifier). Through low-latency initial noise estimation and robust decision-making based on multi-feature fusion, it ensures that the system can accurately distinguish between speech and noise in complex and ever-changing high-noise environments, providing a key guarantee for the high noise robustness of the entire system.

[0052] In one specific embodiment of the present invention, the environmental perception module is implemented as follows. The voice activity detection submodule of this module receives a noise-reduced and enhanced signal from the noise reduction processing module. A dual-threshold endpoint detection algorithm based on short-time energy and zero-crossing rate is used to process the signal in frames (e.g., for a 16kHz sampling rate, frame length 256 points, frame shift 160 points), calculating the energy and zero-crossing rate of each frame and comparing them with a preset threshold to determine whether each frame is valid voice and outputting a voice activity flag. Its network status monitoring submodule operates independently, monitoring the congestion window, round-trip time, and packet loss rate of transport layer protocols (such as TCP), using a bandwidth estimation algorithm (such as TCP Westwood) to estimate the available bandwidth of the current network in real time, and measuring the round-trip delay. To suppress the impact of instantaneous network status fluctuations on decision-making, in a preferred embodiment, this submodule also performs smoothing filtering (such as moving average or low-pass filtering) on ​​the continuously measured available bandwidth estimates, outputting the smoothed bandwidth value for use by the collaborative control module.

[0053] In another specific embodiment of the present invention, the collaborative control module makes collaborative decisions by querying a built-in mapping table. The bitrate mapping table defines recommended encoding bitrates under different voice activity states and network bandwidth conditions. Considering the effective operating range of the OPUS encoder used by the dynamic encoding module (e.g., 6kbps to 48kbps), the mapping strategy aims to optimize bitrate allocation within this range. Its core mapping principles are as follows: When the voice activity flag VAD=1 (valid voice), a higher weight is assigned to the target bitrate R to ensure voice quality; when VAD=0 (invalid voice or silent segment), a lower weight is assigned to the target bitrate R to save bandwidth.

[0054] A specific mapping example (bandwidth B within 200kbps) is as follows: Bandwidth B = 25kbps: R = 24kbps when VAD = 1, R = 8kbps when VAD = 0; Bandwidth B = 50kbps: R = 32kbps when VAD = 1, R = 16kbps when VAD = 0; Bandwidth B = 100kbps: R = 48kbps when VAD = 1 (reaching the upper limit set in the example), R = 24kbps when VAD = 0; Bandwidth B >= 150kbps: In the example, R remains at 48kbps when VAD = 1 (fully utilizing OPUS coding quality), and R can remain at a lower level (such as 24kbps) or be further reduced when VAD = 0.

[0055] The bitrate mapping table can be configured according to the actual encoder support range and system optimization goals. When the available network bandwidth far exceeds the audio encoding requirements, the system can maintain the optimal audio quality bitrate and use the redundant bandwidth for reliability enhancement mechanisms such as forward error correction.

[0056] For example, when smoothing bandwidth is sufficient (e.g., >100kbps) and the current frame contains valid speech, a higher bitrate (e.g., 32kbps) is mapped to maintain fidelity; when bandwidth is limited or in a silent segment, a lower bitrate (e.g., 8kbps) is mapped to save bandwidth. The model performance mapping table defines the speech recognition model version to be used under different network round-trip latency and bandwidth conditions. The principle is to prioritize lightweight and fast models under low latency constraints, and switch to high-precision models when conditions permit. The collaborative control module queries these two tables based on real-time perception information to generate encoding control instructions and model selection instructions, respectively.

[0057] Furthermore, in a more optimized implementation, the cooperative control module also includes an encoding parameter smoothing control unit. When this unit detects that a target bitrate switch is needed, it does not execute immediately, but instead starts a state-holding timer. Only when the new target bitrate value has consistently and stably exceeded a preset threshold time (e.g., 200 milliseconds) is the dynamic encoding module triggered to switch according to a preset gradual strategy (e.g., adjusting the bitrate in several steps), thereby effectively avoiding frequent bitrate oscillations caused by instantaneous network fluctuations.

[0058] In one specific embodiment of the invention, the dynamic encoding module is implemented based on the OPUS encoder. The OPUS encoder supports various bitrates from 5 kbps to 48 kbps. This module dynamically configures the target bitrate, frame length (typically 20 ms), and complexity mode of the OPUS encoder according to encoding control instructions issued by the collaborative control module, and compresses and encodes the continuously input noise reduction and enhancement signal. For low bitrate scenarios, the OPUS encoder can adaptively call an optimized CELP codebook to further reduce the bitrate while ensuring speech intelligibility. The encoded data packets are then sent to the transmission enhancement module. This module applies forward error correction (FEC) coding to the data packets and manages automatic retransmission requests. The low bitrate selected by the dynamic encoding module and the redundancy added by FEC (e.g., 10%) are optimized in conjunction with the collaborative control module's assessment of the network status. For example, when network bandwidth is extremely limited, a sufficiently low encoding bitrate is prioritized to ensure connectivity; when bandwidth allows but the packet loss rate is high, FEC redundancy is appropriately increased. This collaborative strategy of encoding and transmission enhancement enables the system to maintain reliable instruction transmission even in weak network environments with packet loss rates as high as 20%.

[0059] In one specific embodiment of the present invention, a lightweight inference module is deployed on an edge device. This module internally stores multiple versions of speech recognition models, all of which are pre-trained models based on the Transformer architecture. These models have undergone pruning (removing redundant attention heads, etc.) and INT8 integer quantization compression, significantly reducing their size. The models are deployed based on the TensorFlow Lite Micro framework, supporting division instructions and featuring two working modes: interrupt handler mode automatically saves the processor state, achieving low-latency interrupt response; thread mode supports multi-task parallel processing, adapting to the limited computing power of edge devices. The module loads the corresponding model parameters from memory into the inference engine according to the model selection instruction. Simultaneously, the module is associated with a dynamic speech database, which supports online learning of user speech habits. It continuously collects speech data of frequently used user commands for incremental updates, optimizing the model's adaptability to specific user speech features (such as accent and speech rate) and improving personalized recognition accuracy. The lightweight inference module employs an asynchronous pipeline architecture to optimize processing latency. This architecture decomposes the speech recognition task into multiple stages, including feature extraction, acoustic model inference, language model decoding, and semantic understanding, and uses multithreading or hardware pipeline technology to allow these stages to overlap in time. For example, when processing feature extraction of the Nth frame of audio, acoustic model inference of the N-1th frame and language model decoding of the N-2th frame can be performed in parallel. This parallelization effectively hides the computational latency of each stage, ensuring that even on resource-constrained edge devices, the end-to-end speech recognition and command generation latency remains consistently below 300ms, meeting the stringent requirements of real-time interaction. This reduces end-to-end processing latency. The model extracts MFCC or FBank features from the input noise-reduced and enhanced signal, outputs recognized text after inference, and then converts it into structured control commands (such as JSON format) through an integrated semantic understanding component.

[0060] In a typical application scenario of this invention, the system is deployed in an industrial workshop. An operator issues a voice command to "turn off the headlights." A multi-microphone array deployed in the workshop acquires the original audio containing strong background noise (approximately 75dB) and initially enhances the voice in the target direction through beamforming. The two-stage noise reduction module improves the signal-to-noise ratio from -5dB to over 15dB. The environmental perception module accurately detects the voice segment in the enhanced noise-reduced signal and measures the current network bandwidth to be approximately 100kbps with a round-trip latency of approximately 80ms. The collaborative control module queries a mapping table, decides to encode the voice at a bitrate of 48kbps, and selects a lightweight recognition model. The dynamic encoding module compresses the clear voice at a bitrate of 48kbps, and the lightweight inference module loads the lightweight model to accurately recognize the "turn off the headlights" text and generate a structured command. The command mapping module converts this command into a Modbus TCP protocol message supported by the workshop headlight controller, sends it through the communication interface, and the controller executes the headlight-off operation, providing feedback that "headlights are off" through the system. In this scenario, the system achieved an end-to-end latency of less than 300ms and a command recognition accuracy of over 91.8%.

[0061] In an extended embodiment of the present invention, the system further includes a privacy-preserving training module and a node selection unit to support the optimization of the speech recognition model using multi-terminal data through a federated learning framework, while strictly protecting user privacy.

[0062] Specifically, the privacy-preserving training module is linked to the lightweight inference module. On the edge device, this module uses locally collected and anonymized speech data to incrementally train or fine-tune the currently loaded speech recognition model in the lightweight inference module, obtaining local model update parameters (such as gradients or weight differences). To prevent privacy leaks, this module, based on a differential privacy mechanism, adds random noise conforming to a specific distribution (such as a Gaussian distribution) to the local model update parameters before uploading them, generating noisy model parameters. The scale of the added noise is precisely controlled by a system-preset privacy budget parameter, thus achieving a balance between contributing model information and protecting the original data.

[0063] Meanwhile, the node selection unit connects to the network status monitoring submodule of the environment awareness module to obtain the estimated available bandwidth and round-trip latency of the current device in real time. This unit compares these two network metrics with preset bandwidth and latency thresholds. Only when the available bandwidth of the current device is not lower than the bandwidth threshold and the round-trip latency does not exceed the latency threshold will the node selection unit mark the device as a qualified federated learning participant node (i.e., a "qualified participating node"). This mechanism ensures that only devices with good network conditions participate in parameter uploading, avoiding training round timeouts or inefficient aggregation caused by poor network conditions.

[0064] The privacy-preserving training module is configured to upload the generated noisy model parameters to the cloud server only when the current device is marked as a qualified participating node by the node selection unit. The cloud server aggregates the noisy model parameters from numerous qualified nodes and uses a secure aggregation algorithm, such as federated averaging, to generate updated global model parameters. Subsequently, the cloud server distributes the global model parameters to the edge devices participating in the learning process. The privacy-preserving training module receives the distributed updated global parameters and uses them to update the speech recognition model stored in the lightweight inference module, thereby continuously improving the model performance of all participating devices without centralizing the original data.

[0065] In an advanced embodiment of the present invention, the system has the ability to learn and evolve on its own, which is achieved through an online optimization module for the mapping table and a cloud-edge collaborative model management system.

[0066] The online mapping table optimization module is connected to the collaborative control module. During system operation, this module continuously collects actual performance feedback data, including at least the end-to-end system response latency and speech recognition accuracy measured after the system adopts each specific combination of target bitrate and speech recognition model version under the decision of the collaborative control module. Based on accumulated historical performance data, this module dynamically analyzes the new relationships between network conditions (bandwidth, latency), environmental states (speech activity), and optimal decisions (bitrate, model version) using regression analysis (such as linear regression or lightweight neural network models). Subsequently, it dynamically optimizes and updates the mapping relationships in the bitrate mapping table and / or model performance mapping table stored within the collaborative control module. In subsequent decisions, the collaborative control module determines the target bitrate and model version based on the optimized and updated mapping table, enabling the system's collaborative strategy to adapt to long-term network environment changes, equipment performance variations, or the impact of newly emerging noise patterns.

[0067] Furthermore, this invention constructs a complete lifecycle management system for a cloud-edge collaborative model. This system includes: The cloud-based model building module connects to the lightweight inference module via a communication network. This module is responsible for providing a basic speech recognition model (e.g., a large-parameter model based on the Transformer architecture) pre-trained on large-scale general-purpose speech data.

[0068] Edge Model Adaptation Module: This module is built into the lightweight inference module and is equipped with a knowledge distillation engine. It receives a base model from the cloud and utilizes a local domain-specific speech dataset on the device. Through knowledge distillation, it transfers the knowledge from the large model to a smaller, more efficient model, while simultaneously performing further pruning and quantization to ultimately generate a lightweight speech recognition model that can be deployed on edge devices.

[0069] The online model parameter adjustment module connects to the output of the online mapping table optimization module at its input and to the lightweight inference module at its output. Based on performance feedback data collected by the online optimization module (e.g., insufficient accuracy of a model version under specific noise conditions), this module generates model fine-tuning instructions, driving the lightweight inference module to perform lightweight parameter adjustments (e.g., bias fine-tuning) on ​​the currently loaded model, enabling rapid scene adaptation.

[0070] This system clarifies the entire process from cloud-based pre-training of the basic model to edge-based personalized knowledge distillation and compression to fine-tuning driven by online performance feedback, solidifying the dynamic optimization process of the model into an executable modular function within the system.

[0071] In one embodiment of the invention to ensure robustness, the system also integrates a transmission enhancement module and an instruction mapping module to cope with unreliable network and heterogeneous device environments.

[0072] The transmission enhancement module is connected to the output of the dynamic coding module. This module applies forward error correction (FEC) coding to the encoded data packets output by the dynamic coding module, for example, by adding a certain proportion (e.g., 10%) of redundant error correction packets. Simultaneously, it executes an error control mechanism based on automatic retransmission requests (ARQ), detecting in real time whether data packets are lost during transmission and automatically triggering a retransmission request when packet loss is detected (the number of retransmissions can be set to an upper limit, such as 3 times). Through the combination of FEC and ARQ, the reliability of encoded voice data or subsequent control commands in network transmission is significantly improved.

[0073] The instruction mapping module connects the output of the lightweight inference module to the communication interface. This module contains an instruction mapping database and a protocol parsing unit. The instruction mapping database is dynamically updated, automatically expanding mapping rules based on newly added controlled device types and user-added control intentions (such as custom instructions), eliminating the need for manual configuration. The database stores mapping rules from structured control instructions to various device-specific control commands. When the instruction mapping module receives a structured control instruction (such as JSON data containing the intention and parameters to "turn on the air conditioner") generated by the lightweight inference module, it first queries the instruction mapping database based on the target controlled device identifier specified in the instruction to find the corresponding device control command template and communication protocol type. Then, the protocol parsing unit fills the parameters from the structured instruction into the template and encapsulates it into a final device control instruction conforming to the communication protocol format used by the target device (such as MQTT / CoAP protocol for smart home scenarios, Modbus TCP messages for industrial control scenarios, or BACnet protocol for building automation), before sending it through the communication interface. This resolves the instruction barriers between devices of different brands and protocols, enabling the system to have broad device compatibility.

[0074] The system described in this invention has good adaptability to hardware platforms. The audio acquisition device can employ a 2- to 6-microphone array supporting beamforming, with an analog-to-digital converter sampling rate of at least 16kHz. The core processor can be an ARM Cortex-A series chip or an equivalent edge computing chip, and support for the INT8 quantization instruction set is recommended to accelerate model inference. At least 512MB of RAM is recommended. For networking, the system is compatible with Wi-Fi 4 / 5, 4G / 5G mobile networks, and low-power wide-area network protocols such as LoRa. The system can run on Linux, real-time operating systems, or embedded operating systems such as Android Things.

[0075] Although the present invention has been described in detail in the above embodiments, those skilled in the art should understand that several modifications or substitutions can be made to the above embodiments without departing from the principles of the present invention, and these modifications or substitutions should also be considered within the scope of protection of the present invention.

Claims

1. An adaptive voice control system for low-bandwidth and noisy environments, characterized in that, include: The noise reduction processing module receives the original audio signal collected by the audio acquisition device at its input end and outputs the noise-reduced and enhanced audio signal at its output end. An environment perception module, whose input is connected to the output of a noise reduction processing module, is used to receive the noise-reduced and enhanced audio signal. The environment perception module includes a voice activity detection submodule and a network status monitoring submodule. The voice activity detection submodule is used to analyze the audio signal and output a voice activity flag indicating whether the current frame is valid voice. The network status monitoring submodule is used to obtain the round-trip delay and available bandwidth estimate of the current network transmission in real time. The cooperative control module has its first input terminal connected to the output terminal of the environmental perception module. The cooperative control module is used to receive voice activity flags, round-trip time delay, and available bandwidth estimates. The collaborative control module has a built-in bitrate mapping table and a model performance mapping table. The collaborative control module queries the bitrate mapping table based on the voice activity flags and the available bandwidth estimate to determine the target bitrate and generate encoding control instructions. The collaborative control module queries the model performance mapping table based on the round-trip time and available bandwidth estimates to determine the speech recognition model version that needs to be enabled and generates a model selection instruction. The dynamic encoding module has its input end connected to the output end of the noise reduction processing module and the first output end of the collaborative control module; the dynamic encoding module is configured with an OPUS encoder containing multiple bit rate values; the dynamic encoding module is used to select the corresponding target bit rate in the OPUS encoder according to the encoding control command, and to compress and encode the noise-reduced and enhanced audio signal. The lightweight inference module has its input connected to the output of the noise reduction processing module and the second output of the collaborative control module. The lightweight inference module is equipped with a Transformer speech recognition model that has been pruned and quantized. The lightweight inference module is used to load the corresponding speech recognition model according to the model selection instruction, and to perform feature extraction and semantic recognition on the noise-reduced and enhanced audio signal to generate structured control instructions. A communication interface, which is used to send structured control commands to the controlled device.

2. The adaptive voice control system for low-bandwidth and noisy environments according to claim 1, characterized in that, The collaborative control module also includes: The smoothing filter unit, whose input is connected to the output of the network status monitoring submodule, is used to smooth the continuous available bandwidth estimates output by the network status monitoring submodule to obtain the smoothed bandwidth value. The collaborative control module queries the bitrate mapping table based on the voice activity flag and the smoothed bandwidth value.

3. The adaptive voice control system for low-bandwidth and noisy environments according to claim 1, characterized in that, Also includes: The privacy-preserving training module, which is connected to the lightweight inference module, is used to train the speech recognition model based on local speech data, obtain local model update parameters, and add random noise to the local model update parameters based on the differential privacy mechanism to generate noisy model parameters. The node selection unit, whose input is connected to the output of the network status monitoring submodule, is used to compare the estimated available bandwidth with a preset bandwidth threshold and the round-trip time with a preset delay threshold. When the estimated available bandwidth is not lower than the bandwidth threshold and the round-trip time does not exceed the delay threshold, the node selection unit marks the current device as a qualified participating node. The privacy protection training module is configured to upload the noisy model parameters to the cloud server when the current device is marked as a qualified participating node. The cloud server is used to aggregate the noisy model parameters from multiple devices, generate global model update parameters, and distribute them. The privacy-preserving training module is also used to receive the distributed global model update parameters to update the speech recognition model in the lightweight inference module.

4. The adaptive voice control system for low-bandwidth and noisy environments according to claim 1, characterized in that, The noise reduction processing module includes: The adaptive Wiener filter submodule is used to perform the first stage of noise reduction on the original audio signal and output the first noise-reduced signal. A noise classifier, connected to the output of the adaptive Wiener filter submodule, is used to analyze the first noise-reduced signal and identify the dominant noise type. The generative adversarial network (GAN) noise reduction submodule is connected to the output of the adaptive Wiener filter submodule and the noise classifier. The GAN noise reduction submodule is pre-configured with multiple noise reduction generators corresponding to various noise types. The GAN noise reduction submodule is used to switch and enable the corresponding noise reduction generator according to the dominant noise type identified by the noise classifier, perform second-level noise reduction processing on the first noise reduction signal, and output the noise-enhanced audio signal.

5. The adaptive voice control system for low-bandwidth and noisy environments according to claim 1, characterized in that, The collaborative control module also includes: The coding parameter smoothing control unit is equipped with a state-holding timer and a progressive switching strategy; wherein, when the cooperative control module determines the target bitrate based on the voice activity flag and the available bandwidth estimate, the coding parameter smoothing control unit is configured to perform the following operations: If the target bitrate differs from the current bitrate of the dynamic encoding module, then start the state hold timer; Only after the target bitrate has been maintained for a period of time exceeding the preset stable threshold of the state hold timer will the dynamic encoding module be triggered to gradually switch from the current encoding bitrate to the target bitrate according to the gradual switching strategy. If the target bitrate determined by the collaborative control module changes within the stable threshold time, the state hold timer is reset and the judgment is re-executed based on the newly determined target bitrate.

6. The adaptive voice control system for low-bandwidth and noisy environments according to claim 1, characterized in that, Also includes: The mapping table online optimization module is connected to the collaborative control module, and is configured as follows: Collect performance feedback data during actual system operation. The performance feedback data includes at least the end-to-end response latency and speech recognition accuracy measured under the control of the collaborative control module when using a specific target bitrate and a specific speech recognition model version combination. Based on the collected performance feedback data, the mapping relationships in the bitrate mapping table and / or model performance mapping table within the collaborative control module are dynamically optimized and updated through regression analysis. The collaborative control module is further configured to determine the target bitrate and speech recognition model version based on the optimized and updated mapping table.

7. The adaptive voice control system for low-bandwidth and noisy environments according to claim 1, characterized in that, The audio acquisition device includes a multi-microphone array, which is used to acquire multiple raw audio signals and execute a beamforming algorithm to generate a beam of directional enhanced speech signal, which is then received by the noise reduction processing module.

8. The adaptive voice control system for low-bandwidth and noisy environments according to claim 6, characterized in that, Also includes: The cloud-based model building module connects to the lightweight inference module via a communication network and provides a basic speech recognition model to the lightweight inference module. The basic speech recognition model is pre-trained by the cloud server using large-scale general speech data. The edge model adaptation module is built into the lightweight inference module and is equipped with a knowledge distillation engine. The edge model adaptation module is used to: receive the basic speech recognition model from the cloud model building module, and use the local speech dataset to perform knowledge transfer and model compression on the basic speech recognition model through the knowledge distillation engine to generate a lightweight speech recognition model deployed in the lightweight inference module. The online model parameter adjustment module has its input connected to the output of the online mapping table optimization module, and its output connected to the lightweight inference module. The online model parameter adjustment module is used to generate model fine-tuning instructions based on the performance feedback data output by the online mapping table optimization module, and drive the lightweight inference module to make lightweight adjustments to the parameters of the currently loaded speech recognition model.

9. The adaptive voice control system for low-bandwidth and noisy environments according to claim 1, characterized in that, Also includes: The transmission enhancement module, whose input is connected to the output of the dynamic encoding module, is used to apply forward error correction coding to the encoded data packets and perform error control based on automatic retransmission requests, including detecting data packet loss and triggering retransmission. The instruction mapping module has its input connected to the output of the lightweight inference module to receive structured control instructions, and its output connected to the communication interface. The instruction mapping module includes an instruction mapping relationship library and a protocol parsing unit. It is used to query the instruction mapping relationship library according to the identifier of the target controlled device, and to convert the structured control instructions into device control instructions that conform to the communication protocol format adopted by the target controlled device through the protocol parsing unit.

10. An adaptive voice control method for low-bandwidth and noisy environments, based on an adaptive voice control system for low-bandwidth and noisy environments as described in any one of claims 1 to 9, characterized in that, Includes the following steps: Noise reduction processing steps: Perform noise reduction processing on the original audio signal and output the noise-reduced and enhanced audio signal; Environmental awareness steps: Based on the noise-reduced and enhanced audio signal, obtain the speech activity flag indicating whether the current frame is valid speech, and obtain the round-trip delay and available bandwidth estimate of the current network transmission in real time; Collaborative decision-making steps: Based on the speech activity flags and available bandwidth estimates, query the bitrate mapping table to determine the target bitrate and generate encoding control instructions; simultaneously, based on the round-trip time and available bandwidth estimates, query the model performance mapping table to determine the speech recognition model version to be used and generate model selection instructions. Dynamic encoding steps: According to the encoding control instructions, select the corresponding target bit rate in the OPUS encoder and compress and encode the noise-reduced and enhanced audio signal; Lightweight inference steps: Based on the model selection instruction, load the corresponding pruned and integer quantized Transformer speech recognition model, extract features and perform semantic recognition on the noise-reduced and enhanced audio signal, and generate structured control instructions; Command transmission steps: The structured control commands are sent to the controlled device through the communication interface.

Images