A real-time speech enhancement method and device for a classroom scenario

Abstract

The application discloses a real-time speech enhancement method and device for a classroom scene, and relates to the technical field of speech signal processing. The method comprises the following steps: acquiring the geometric parameters of a classroom, calculating a theoretical reverberation time prior value and a theoretical direct-to-reverberation ratio prior value; using the prior information as a constraint to estimate the room impulse response through a blind system identification algorithm; adaptively setting the dereverberation algorithm parameters based on a linear prediction algorithm to perform dereverberation processing on an audio signal; suppressing residual distortion through a machine learning model; fusing multi-modal features to detect a target speaker speech segment; applying differential audio processing to different speech segments based on the target speaker speech segment detection result, and applying distance-adaptive compensation to the speech signal according to the estimated distance. The application significantly improves the speech intelligibility in the classroom scene and reduces the speech recognition error rate.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and specifically to a real-time speech enhancement method and apparatus for classroom scenarios. Background Technology

[0002] With the development of educational informatization, interactive devices integrating display, touch, computing, and audio / video acquisition functions have become the core hub of modern classrooms. However, in actual deployment, the voice interaction and acquisition capabilities of these devices are often severely limited by the unique acoustic environment of the classroom.

[0003] First, classrooms typically have poor acoustic characteristics. The large space and the use of hard, highly reflective materials like concrete and ceramic tiles for walls and floors result in excessively long reverberation times, often far exceeding recommended standards for voice communication. Furthermore, under far-field pickup conditions, the energy ratio of direct sound to reverberant sound is extremely low, leading to a significant decrease in the clarity and intelligibility of the speech signal. In addition, interactive equipment is usually mounted close to the wall, introducing strong early reflections that further deteriorate pickup quality.

[0004] Secondly, existing speech enhancement technologies perform poorly in classroom settings. For example, while beamforming-based spatial filtering methods can suppress directional noise, their ability to suppress diffuse late-stage reverberation is limited. De-reverberation techniques based on linear prediction algorithms can handle reverberation, but their general parameter settings are not optimized for the long reverberation characteristics of classrooms, and their convergence speed and processing performance are insufficient to meet real-time requirements. Machine learning-based solutions face the problem of domain offset between training data and the actual classroom environment, leading to decreased generalization performance, and their high computational complexity makes them difficult to run in real-time on embedded heterogeneous SoCs in interactive devices.

[0005] In summary, existing technical solutions generally suffer from insufficient adaptability to specific scenarios, inadequate collaborative optimization by leveraging the hardware characteristics of embedded heterogeneous on-chip systems, and a lack of online adaptive adjustments to changes in the actual environment. Therefore, there is an urgent need for a real-time speech enhancement solution specifically designed for classroom scenarios that can overcome these shortcomings. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention provides a real-time speech enhancement method and apparatus for classroom scenarios. This solution integrates the geometric parameters of the classroom with multimodal perception information, achieving full-process optimization from acoustic environment modeling and adaptive dereverberation to differentiated post-processing. It can run efficiently on embedded heterogeneous system-on-a-chip (SoC) of interactive devices, significantly improving far-field speech quality in classroom environments.

[0007] This invention provides a real-time speech enhancement method for classroom scenarios, implemented using an interactive device, the method comprising: Obtain the geometric parameters of the classroom where the interactive device is located; Based on the geometric parameters, calculate the theoretical reverberation time prior value of the classroom; Based on the geometric parameters, calculate the theoretical direct reverberation ratio prior value for the spatial location within the classroom; Using at least one of the theoretical reverberation time prior value and the theoretical direct reverberation ratio prior value as prior information, the room impulse response is estimated based on the audio signal collected by the microphone array using a blind system identification algorithm; Based on at least one of the room impulse response, theoretical reverberation time prior value, and theoretical direct reverberation ratio prior value, the parameters of the dereverberation algorithm based on the linear prediction algorithm are adaptively set. Based on the reverberation algorithm parameters, the linear prediction algorithm is executed on the audio signal to obtain the reverberated signal. The residual distortion of the reverberation-processed signal is suppressed using a machine learning model. By fusing features from at least two different modalities, target speaker speech segments can be detected. Based on the speech segment detection results of the target speaker, the audio stream is divided into at least two preset speech segment types; Differentiated audio processing is applied to different types of speech segments; the distance between the target speaker and the microphone array is estimated using the room impulse response and the target speaker's speech segments. The speech signal is subjected to distance-adaptive compensation based on the distance.

[0008] Furthermore, the geometric parameters of the classroom can be obtained through various methods, including manual input, using a built-in camera combined with a monocular depth estimation network, or utilizing one or more depth sensors. This approach makes the acquisition of geometric parameters flexible and diverse, lowering the deployment threshold.

[0009] In one embodiment, the linear prediction algorithm is a weighted prediction error algorithm, and the dereverberation algorithm parameters include the linear prediction order, prediction delay, and recursive least squares forgetting factor. Furthermore, in the weighted prediction error algorithm, a spatial regularization term based on the room impulse response is introduced to constrain the structure of the multi-input multi-output prediction matrix. By incorporating acoustic priors into the algorithm, it is made more adaptable to the long reverberation characteristics of classrooms, thus improving the accuracy of dereverberation.

[0010] Optionally, the machine learning model is a convolutional recurrent network for estimating a complex ratio mask; and the method further includes automatically adjusting the size of the first layer convolutional kernel in the encoder of the convolutional recurrent network based on the estimated theoretical reverberation time prior. This adaptive adjustment enables the model to better match the reverberation characteristics of different classrooms, thereby more effectively suppressing residual distortion.

[0011] Furthermore, the features of the at least two different modalities include at least two of the following: acoustic features, visual features based on human pose detection, and interactive context features of the interactive device; and, when the visual features indicate that the target speaker is in a preset area, the detection sensitivity threshold for the target speaker's speech segment is adaptively reduced. Multimodal fusion significantly improves the robustness of speech segment detection in environments with strong reverberation and noise.

[0012] In one implementation, the speech segment types include continuous speech segments by the target speaker, short intermittent speech segments by the target speaker, speech segments by non-target speakers, and silent or noisy segments; the differentiated audio processing includes applying reverberation compression processing of different intensities to different types of speech segments. This scheme ensures the clarity of the target speaker's speech while also taking into account the naturalness of the speech and the preservation of the voices of non-target speakers, making the processing more targeted.

[0013] Optionally, the method further includes: during the idle period of the interactive device, using naturally existing sound sources in the classroom as excitation signals to automatically trigger the re-estimation of the room impulse response, and updating it with a weighted moving average of the historical room impulse response.

[0014] Furthermore, the method operates on an embedded heterogeneous on-chip system and includes: allocating different computing tasks in the method to different hardware units of the embedded heterogeneous on-chip system for execution; and employing a multi-stage pipeline and a zero-copy data transfer mechanism based on direct memory access to achieve data sharing across hardware cores.

[0015] In one embodiment, the method further includes: automatically identifying the current application mode based on the long-term statistical characteristics of the target speaker's speech segment detection results, and adaptively adjusting the dereverberation intensity or processing delay according to the identified mode.

[0016] The present invention also provides a voice enhancement device for a classroom setting, comprising: one or more sensors or input interfaces for acquiring geometric parameters of the classroom in which the device is located; one or more processors; and a memory storing computer-executable instructions that, when executed by the one or more processors, cause the device to perform any of the methods described above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: First, by using classroom geometry parameters to model the acoustic environment and guide subsequent dereverberation and post-processing, this invention achieves a high degree of adaptability to classroom scenarios, significantly improving the speech clarity and intelligibility of far-field sound pickup.

[0018] Secondly, by integrating multimodal information such as acoustics, vision, and interactive context to detect the target speaker's speech segments and implementing differentiated processing strategies, this invention effectively suppresses reverberation while better balancing speech clarity and naturalness.

[0019] Furthermore, the collaborative inference acceleration scheme for embedded heterogeneous on-chip systems proposed in this invention ensures that the entire algorithm can operate stably under strict real-time and power consumption constraints.

[0020] Finally, the online adaptive update mechanism enables the invention to continuously adapt to environmental changes, achieving a "one-time deployment, long-term effectiveness" out-of-the-box effect. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall system architecture of the method in an embodiment of the present invention.

[0022] Figure 2 This is a flowchart illustrating the steps of the method in an embodiment of the present invention.

[0023] Figure 3 This is a block diagram of the hardware deployment architecture of the method of this invention on an embedded heterogeneous system-on-chip (SoC).

[0024] Figure 4 This is a schematic diagram of the multimodal fusion architecture for target speaker speech segment detection in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0026] In the description of this invention, it is important to understand that some core terms used in this invention need to be explained first to ensure that those skilled in the art can accurately understand the technical solutions of this invention. The term "interactive device" refers to an intelligent terminal device that integrates high-definition display, multi-touch, built-in computing, and audio / video capture functions. Its specific form may be an interactive flat panel (IFP), a smart blackboard, a recording and broadcasting host, or a conference flat panel, etc. The term "geometric parameters" refers to parameters used to describe the physical space of a classroom, including but not limited to the classroom's dimensions (length, width, height), the main material types of the walls, ceiling, and floor, and the specific installation location information of the interactive device.

[0027] Furthermore, the term "Room Impulse Response" (RIR) refers to a function describing the acoustic transmission path characteristics from the sound source to the microphone, comprehensively characterizing the combined effects of direct sound, early reflections, and late reverberation. The term "theoretical reverberation time prior" refers to an estimate of the reverberation time (RT60 prior) calculated in advance based on the classroom's geometric parameters and acoustic formulas (e.g., the Eyring formula). Similarly, the term "theoretical direct-to-reverberation ratio prior" refers to an estimate of the distribution of the direct-to-reverberation energy ratio (DRR prior) at different spatial locations within the classroom, calculated in advance based on geometric parameters and acoustic models (e.g., the mirror source method). Although these prior values ​​are theoretically calculated, they provide crucial initial guiding information for subsequent adaptive processing.

[0028] Furthermore, the "linear prediction algorithm" mentioned in this invention refers to a class of algorithms that predict the current signal through a linear combination of historical signals. In the context of this invention, it is mainly used to model and eliminate late reverberation components in audio signals. A specific example is the Weighted Prediction Error (WPE) algorithm. The term "machine learning model" in this invention specifically refers to a data-driven model used for deep processing of the signal after initial déreverberation by the linear prediction algorithm. Its goal is to suppress residual reverberation distortion and background noise. A specific example is a deep neural network used to estimate the Complex Ratio Mask (CRM). The term "target speaker" in a typical teaching scenario is presupposed to be the primary object for speech acquisition and enhancement, usually referring to the teacher. Accordingly, "Target Speaker Vocal Detection" (T-VAD) is used to accurately detect the time period during which the target speaker is speaking in the audio stream. Finally, the term "embedded heterogeneous system-on-a-chip" (SoC) refers to a single-chip system that integrates multiple different types of processing cores (such as CPU big / small cores, NPU, GPU, etc.) and is the core computing unit of interactive devices.

[0029] The core idea of ​​this invention lies in constructing a four-stage technical pipeline: "scene modeling → adaptive dereverberation → scene-adaptive post-processing → efficient edge-side inference." This patent integrates the geometric parameters of the classroom and prior information about the teaching scenario into each stage of speech enhancement, thereby achieving precise responses to the unique acoustic challenges of the classroom. In this way, this invention aims to solve the performance bottleneck problem of existing general-purpose speech enhancement solutions in classroom scenarios due to a lack of scene adaptability, edge-side collaborative optimization, and closed-loop adaptive capabilities.

[0030] Please see Figure 1 , Figure 1 This diagram illustrates the overall system architecture of the method of the present invention, showing the complete four-stage pipeline from classroom geometric parameter acquisition to final enhanced speech output, as well as the module composition and data flow of each stage. The system can be deployed in the interactive device defined above. The system mainly includes: a blind acoustic parameter estimation module 100, a frequency domain adaptive dereverberation and enhancement module 200, a multimodal and adaptive post-processing module 300, and a heterogeneous collaborative inference acceleration module 400. Optionally, the system may also include a teaching scenario adaptive strategy scheduling module. These modules work together to realize a complete processing chain from environmental perception to final speech output.

[0031] Specifically, the acoustic parameter blind estimation module 100 is responsible for sensing the physical acoustic environment of the classroom. It includes a classroom geometry parameter acquisition submodule 110, used to acquire the geometry parameters of the classroom where the interactive equipment is located. Based on these geometry parameters, the acoustic parameter blind estimation module 100 can calculate the theoretical reverberation time prior value and the theoretical direct reverberation ratio prior value, and further use this prior information to guide the blind system identification algorithm to estimate the actual room impulse response (RIR).

[0032] The frequency domain adaptive dereverberation and enhancement module 200 is the core of speech enhancement. It receives multi-channel microphone audio signals and uses information such as RIR output by the acoustic parameter blind estimation module 100 to achieve deep suppression of reverberation and noise through the classroom adaptive filtering submodule 210 and a mask estimation submodule 220.

[0033] The multimodal and adaptive post-processing module 300 then performs refined processing on the enhanced speech. This module, through a trimodal fusion decision submodule 310, combines acoustic, visual, and interactive contextual information to achieve robust detection of the target speaker's (teacher's) speech segments. Based on the detection results, this module applies differentiated post-processing strategies, such as dynamic range compression and gain compensation, to different types of speech segments, thereby improving the naturalness and listenability of the speech while maintaining clarity. Finally, all these computationally intensive tasks are efficiently scheduled and executed by the heterogeneous collaborative inference acceleration module 400, ensuring that the entire processing flow can operate in real-time under the strict resource constraints of the embedded heterogeneous on-chip system. Data flows between the modules through predefined data interfaces, forming an efficient and adaptive speech enhancement system.

[0034] like Figure 2 The diagram shown is a flowchart of the core steps of this invention, describing the algorithmic steps and data interaction relationships between modules in Stage 1 (blind IR estimation), Stage 2 (adaptive dereverberation), and Stage 3 (T-VAD-guided post-processing). Specifically, this invention provides a real-time speech enhancement method for classroom scenarios. A basic embodiment of this method includes the following steps: First, in step S1, the geometric parameters of the classroom where the interactive device is located are obtained, and acoustic prior information, including theoretical reverberation time and theoretical direct reverberation ratio, is calculated based on these parameters. This step provides the entire system with a macroscopic understanding of the acoustic characteristics of the current physical environment and is the foundation for achieving scene adaptation.

[0035] Next, in step S2, the theoretical reverberation time prior value and the theoretical direct reverberation ratio prior value calculated in step S1 are used as regularization constraints or guiding information to estimate the multi-channel room impulse response (RIR) through a blind system identification algorithm. Unlike completely "blind" estimation, using prior information can significantly improve the accuracy and convergence speed of RIR estimation, enabling the system to quickly establish an accurate model of the classroom acoustic transfer function without dedicated measurement equipment.

[0036] Then, in step S3, based on the estimated RIR and prior information from the previous step, the parameters of the dereverberation algorithm based on the linear prediction algorithm are adaptively set. For example, the order of the linear prediction algorithm can be determined based on the estimated reverberation time. After setting the parameters, the linear prediction algorithm is used to perform preliminary dereverberation processing on the audio signal acquired by the microphone array, resulting in a reverberation-processed signal. The main goal of this step is to suppress the late reverberation component with a large energy proportion in the audio signal.

[0037] In step S4, to further improve speech quality, a machine learning model is used to process the reverberation-processed signal output in step S3 to suppress residual reverberation distortion and background noise. This machine learning model is typically trained on large amounts of data and can learn more complex nonlinear mapping relationships than linear prediction models, thereby achieving a more refined speech enhancement effect.

[0038] Subsequently, in step S5, features from at least two different modalities are fused to detect the target speaker's speech segments. In classroom scenarios, relying solely on acoustic features for voice activity detection (VAD) is often unreliable due to strong reverberation and variable noise. By incorporating information from other modalities such as visual and interactive context, the robustness of detection can be greatly improved, ensuring the correct triggering of subsequent processing.

[0039] Based on the speech segment detection results in step S5, step S6 divides the audio stream into at least two preset speech segment types, and applies differentiated audio processing to different types of speech segments. For example, for continuous speech by the target speaker (teacher), stronger processing can be used to maximize clarity; while for student speech or background noise, milder processing can be used to preserve the sense of presence. This differentiated processing strategy allows the final output audio to achieve a better balance between clarity and naturalness.

[0040] Finally, in step S7, the estimated room impulse response and detected target speaker segments are used to back-estimate the distance between the target speaker and the microphone array, and distance-adaptive compensation is applied to the speech signal based on this distance. For example, moderate gain is applied to speech picked up at a distance to ensure that the final output speech signal has a stable and consistent perceived loudness. Through the coordinated work of this series of steps, this method achieves effective enhancement of target speech in complex classroom environments.

[0041] As a specific implementation method, the acquisition of classroom geometric parameters can be flexible and varied. For example, through the configuration interface provided by the interactive device, installers can manually input the approximate dimensions of the classroom (such as length, width, and height) and the main material types of the walls, ceiling, and floor (such as concrete, tile, glass, and wood panels) during device installation. This method is the lowest cost but relies on the accuracy of manual input. To improve automation and accuracy, another approach is to utilize the wide-angle camera typically built into the interactive device, combined with a lightweight monocular depth estimation network (such as a network based on DepthAnything or MiDaS architecture), to photograph the classroom and infer the spatial structure and the location and approximate dimensions of the main reflective surfaces. Furthermore, if the interactive device is equipped with more advanced depth sensors, such as time-of-flight (ToF) sensors or structured light depth sensors, it can directly acquire 3D point cloud data of the classroom, thus obtaining more accurate geometric parameters. These three methods can be used individually or in combination; for example, automatic sensor acquisition can be the primary method, supplemented by manual input for correction, thereby achieving effective acquisition of geometric parameters on devices with different hardware configurations.

[0042] Building upon the above embodiments, this invention further specifies the concrete implementation of the dereverberation and post-processing stages. Specifically, the linear prediction algorithm is a weighted prediction error (WPE) algorithm. The WPE algorithm achieves dereverberation by linearly predicting late-stage reverberation and subtracting it from the observed signal. In this invention, its key parameters are no longer fixed or empirically set, but are adaptively adjusted. For example, the linear prediction order, prediction delay, and recursive least squares (RLS) forgetting factor λ used for iterative updates in the WPE algorithm are automatically set based on the prior values ​​of the theoretical reverberation time and the theoretical direct-to-reverberation ratio estimated in the preceding steps. This adaptive setting ensures that the WPE algorithm operates in an optimal or suboptimal state in classrooms with different levels of reverberation. Furthermore, in order to better utilize the spatial information of the multi-microphone array, this invention also introduces a spatial regularization term based on room impulse response into the WPE algorithm. This regularization term is used to constrain the structure of the multi-input multi-output prediction matrix to make it more consistent with the propagation characteristics of sound waves in physical space (such as an approximate Toeplitz-block structure), thereby improving the dereverberation performance, especially in suppressing early reflections that are close to the direction of the direct sound.

[0043] The machine learning model is preferably a convolutional recurrent network (CRM) for estimating the complex ratio mask (CRM). Compared to models that only estimate real-valued masks (such as amplitude masks), the CRM network can simultaneously adjust the amplitude and phase of the signal, thus theoretically reconstructing cleaner speech more perfectly and effectively avoiding artifacts such as "musical noise." This CRM typically includes an encoder-decoder structure, using recurrent units (such as GRUs or LSTMs) to model the temporal dependencies of the speech signal. Preferably, the parameters of this CRM are also adapted to the classroom acoustic environment: the size of the first layer convolutional kernel in the encoder part of the CRM is automatically adjusted based on the estimated theoretical reverberation time prior. The principle is that a longer reverberation time means a longer tail in the RIR in the temporal domain, requiring a larger convolutional kernel (i.e., a larger receptive field) to capture this long-term correlation. Through this adaptive configuration, the machine learning model itself also possesses the ability to be customized for specific classroom environments, thereby improving its accuracy in suppressing residual distortion and noise.

[0044] As a preferred embodiment, this invention provides a detailed description of target speaker speech segment detection (T-VAD) and differential post-processing. For example... Figure 4The diagram illustrates the T-VAD multimodal fusion architecture, showcasing the fusion process of acoustic, visual, and interactive context features. T-VAD integrates features from at least two, preferably three, different modalities. The first is acoustic feature extraction 510, which may include Mel-frequency cepstral coefficients (MFCC), fundamental frequency (Pitch), spectral centroid, harmonic noise ratio (HNR), and the short-time objective intelligibility (STOI) increment before and after dreverberation. These features collectively describe whether the audio signal itself possesses speech characteristics. The second is visual feature extraction 520, which uses a lightweight human pose detection algorithm (such as MediaPipe Pose) to determine the presence of a human figure, whether that figure is located in a preset lectern area (ROI), and whether it is facing students through head pose estimation. The third is interactive context feature extraction 530, which detects, for example, the teacher using a stylus to write on the whiteboard, turning pages in a PPT, or the system being in "teaching mode." These three types of features are fused through a decision-level weighted fusion module 540, such as weighted voting, to finally output the T-VAD result 550. Preferably, when visual features indicate that the target speaker (teacher) is in the podium area and facing the students, the system adaptively lowers the detection sensitivity threshold of T-VAD. This is based on a strong prior: the probability of the teacher speaking is extremely high at this time, and lowering the threshold helps to capture some weaker speech initiation parts, thereby improving the detection recall.

[0045] After obtaining reliable T-VAD results, the audio stream is segmented into different preset speech segment types. This invention does not limit the specific segmentation type; as an example, the speech segment types are divided into the following four categories: continuous speech segments by the target speaker (e.g., a teacher lecturing for more than 2 seconds), short intermittent speech segments by the target speaker (e.g., a teacher asking a brief question or giving a comment), speech segments by a non-target speaker (e.g., a student answering a question), and silent or noisy segments. For these four different types of speech segments, the system applies reverberation compression processing of varying intensities. For example, for continuous speech segments by the teacher, a high compression ratio (e.g., 4:1), a fast start time (e.g., 5ms), and a moderate release time (e.g., 50ms) dynamic range compression can be used to maximize speech clarity and intelligibility, even at the cost of slightly sacrificing some naturalness. For student speech segments, very mild compression (e.g., 1.2:1) or even no compression is used to preserve their original speech texture and spatial awareness. For silent or noisy segments, the compressor is completely turned off to avoid amplifying background noise. Through this refined and differentiated processing, the present invention improves the clarity of the target speech while preserving the authenticity and naturalness of classroom interaction to the greatest extent.

[0046] To address potential slow changes in the classroom acoustic environment (e.g., seasonal variations in door and window opening / closing, or adjustments to the arrangement of desks, chairs, curtains, etc.), this invention further proposes an online adaptive update mechanism. This method includes a background monitoring program that automatically triggers a room impulse response (RIR) re-estimation process when it detects that interactive devices are idle (e.g., at night or during breaks exceeding a certain duration, such as 30 minutes) and the ambient noise level is below a preset threshold. At this time, the system uses naturally occurring, relatively stable sound sources in the classroom (e.g., the operating noise of air conditioners and fans, or ambient noise from outside the window) as excitation signals to re-execute the aforementioned blind system identification algorithm. The newly estimated RIR is then updated using a weighted moving average with the historically stored RIR. The update formula can be "RIR_new = λ RIR_old + (1-λ) "RIR_estimated", where the forgetting factor λ is typically set to a value close to 1, such as 0.95. This smooth update method ensures system stability and avoids drastic parameter jumps caused by single estimation errors. Simultaneously, the system calculates the spectral deviation between the old and new RIRs. If the deviation exceeds a threshold (e.g., 3dB), it indicates a significant change in the classroom acoustic environment (e.g., the classroom being emptied during holidays). In this case, the system temporarily reduces the forgetting factor λ (e.g., to 0.85) to accelerate adaptation to the new environment. Through this online adaptive calibration mechanism, this invention ensures that the speech enhancement system maintains optimal performance throughout long-term use.

[0047] Furthermore, this invention also includes optimizations for embedded platforms. The core of modern interactive devices is an embedded heterogeneous system-on-a-chip (SoC), such as the Rockchip RK3588, which integrates hardware units with different characteristics and computing power. Figure 3The diagram shows the hardware architecture block diagram of the method of this invention deployed on an interactive flat panel embedded processor (taking RK3588 as an example). This SoC may include a high-performance Cortex-A76 large core 410, a high-efficiency Cortex-A55 small core 420, an NPU 430 dedicated to neural network computation, a DMA controller 440, and shared memory (DDR) 450. To run complex speech enhancement algorithms under strict real-time requirements (e.g., end-to-end latency less than 300ms) and power budget constraints, this invention designs a heterogeneous collaborative inference scheme. First, different computational tasks in the method are rationally allocated: the computationally intensive and highly parallel WPE adaptive filtering algorithm can be vectorized and accelerated using the NEON SIMD instruction set of the Cortex-A76 large core 410; while the inference of the machine learning model (CRM network) is completely offloaded to the NPU 430; relatively lightweight serial tasks such as acoustic feature extraction in T-VAD can be run on the Cortex-A55 small core 420 to reduce overall power consumption.

[0048] To address the data transfer bottleneck between different hardware cores, this invention further employs a multi-stage pipeline and a zero-copy data transfer mechanism based on Direct Memory Access (DMA). For example, a three-stage pipeline can be constructed: when the k-th frame of data is used for CRM network inference on the NPU430, the (k+1)-th frame of data can undergo WPE filtering on the A76 big core 410, while the raw audio data of the (k+2)-th frame can undergo STFT and feature extraction on the A55 little core 420. Data exchange between different processing stages is not performed via memory copying by the CPU, but rather by configuring the DMA controller 440 and utilizing kernel mechanisms such as ION or DMA-BUF, allowing different hardware units to directly access the same shared memory 450. This zero-copy mechanism can significantly reduce data transfer latency (typically saving 2-5ms per frame) and reduce CPU load. Through heterogeneous task allocation and careful data flow design, this invention can efficiently implement complex speech enhancement algorithms on limited embedded resources.

[0049] Furthermore, this invention provides a higher level of scene adaptation capability. By performing long-term statistical analysis on the results of Target Speaker Detection (T-VAD), the system can automatically identify the current application mode. For example, if the system detects that the proportion of teacher speech is very high (e.g., exceeding 80%) in the past few minutes, and the average duration of the speech segments is long, with a low frequency of teacher-student interaction, the system can determine that it is currently in "lecture mode." Conversely, if frequent teacher-student interaction is detected, it is determined to be "interaction mode." If the system is manually set to recording mode, it enters "recording mode." Based on the identified different application modes, the system can adaptively adjust its core parameters. In "lecture mode," the primary optimization goal is the clarity of the students in the back rows, so the maximum intensity of déreverberation and reverberation compression strategies can be adopted, and a slightly higher processing latency (e.g., using the full 300ms budget) can be tolerated. In "interaction mode," to ensure the smoothness of question and answer, it is necessary to reduce processing latency, so the processing frame length can be appropriately shortened, and a medium-intensity processing strategy can be adopted to balance clarity and naturalness. In "recorded playback mode," the goal is to generate high-quality course resources; therefore, the processing should be gentler, preserving an appropriate amount of spatiality (by controlling the wet / dry ratio of the reverberation) to provide a more immersive auditory experience. This application-mode-based adaptive strategy scheduling allows the speech enhancement effects of this invention to better match the actual needs of teaching activities.

[0050] To more comprehensively demonstrate the technical solution of this invention, a complete embodiment will be described below, which integrates the aforementioned technical features. It is assumed that the interactive device uses an RK3588 SoC and deploys the complete voice enhancement system of this invention. The processing flow strictly follows... Figure 2 Perform the steps shown.

[0051] In step S1, after the interactive device is powered on, it uses a wide-angle camera to call an INT8 quantized DepthAnything lightweight network (with a single-frame inference time of less than 15ms on the NPU) to acquire a classroom depth map. Combined with the wall material type (e.g., "concrete") input by the installer in the configuration interface, the geometric parameters of the classroom are calculated. Subsequently, the theoretical reverberation time prior for each 1 / 3 octave band is calculated using the Eyring formula: RT60(f) = 0.161·V / [-S·ln(1-α(f)) + 4mV]. Here, V is the classroom volume, S is the total surface area, α(f) is the average absorption coefficient of frequency band f obtained from a table based on the material type, and m is the air absorption coefficient. Simultaneously, the spatial distribution of the theoretical direct reverberation ratio (DRR) prior value within the classroom is calculated using the mirror source method and stored as a 0.5m × 0.5m grid lookup table.

[0052] In step S2, a 3-5 second ambient background sound segment is acquired and used for blind estimation of the multi-channel room impulse response (RIR). This estimation process is modeled as a nonnegative matrix factorization (NMF) optimization problem with the objective function: min||X-WH||²_F + λ_RT·||E(H)-exp(-γt)||² + λ_DR·||DRR(H)-DRR_prior||². Here, the first term is the standard NMF reconstruction error, the second term is the energy attenuation envelope regularization term based on the RT60 prior, where γ = 3 / RT60_prior, and the third term is the energy ratio regularization term based on the DRR prior. The regularization weights are empirically set to λ_RT = 0.3 and λ_DR = 0.2. By solving this optimization problem, the frequency domain response H_m(f) of each channel's RIR can be obtained, along with more accurate RT60 and DRR estimates derived from it.

[0053] In step S3, the classroom adaptive WPE dredging stage begins. Based on the estimated RT60(f) from the previous step, the linear prediction order L of the WPE is set. p (f) = [RT60(f) / T_hop]·K, where T_hop is the frame shift duration, and the oversampling factor K is 3. The prediction delay τ is adaptively selected within a [2, 8] frame range based on the DRR. The forgetting factor λ used for RLS iteration is set to λ = 1 - 1 / (α·RT60_avg·fs) based on the average reverberation time, where α is 3.0. Furthermore, the spatial regularization intensity η is proportional to (1 - DRR / DRR_max). After performing WPE filtering, the preliminary dereverberated signal Y_wpe_m(t,f) is obtained.

[0054] In step S4, complex ratio mask (CRM) estimation is performed. The real and imaginary parts (2M-dimensional) of the output signal Y_wpe of WPE, the inter-channel phase difference IPD (M(M-1) / 2-dimensional), and the inter-channel level difference ILD (M-dimensional) are concatenated into a joint feature map and input into the CRM network. This network adopts a lightweight convolutional recurrent architecture: the encoder contains 3 layers of depthwise separable convolutions (32 / 64 / 128 channels), the activation function is PReLU, and LayerNorm is used; the temporal modeling layer is a single-layer bidirectional GRU (128 units); and the decoder is a 3-layer transposed convolution. The complex mask G_m(t,f) output by the network is multiplied pointwise with Y_wpe_m(t,f) to obtain the final enhanced spectrum Z_m(t,f), which is then used to reconstruct the temporal speech through iSTFT and Overlap-Add.

[0055] In step S5, the system performs multimodal T-VAD. The acoustic branch extracts a 43-dimensional feature vector, which is input into a 3-layer DNN to obtain the acoustic decision probability p_acoustic. The visual branch runs MediaPipePose at 5fps to determine whether the teacher is within the podium ROI (f_roi) and facing the students (f_facing). The interaction branch monitors touch and application states, outputting an interaction intensity value c_interact in the range [0,1]. The final T-VAD probability is obtained through weighted fusion: p_TVAD = 0.6·p_acoustic + 0.25·(f_roi·f_facing) + 0.15·c_interact. The base decision threshold θ_base is 0.5, but when f_roi·f_facing = 1, this threshold adaptively decreases to 0.35.

[0056] In step S6, differential post-processing is performed based on the T-VAD results. For continuous teacher speech segments (lasting ≥2s), a compression ratio of 4:1 is used, with a start time of 5ms and a release time of 50ms. For short teacher speech segments (<2s), the compression ratio is reduced to 2:1. For student speech segments, the compression ratio is 1.2:1. Simultaneously, the teacher distance d_est = d_ref·10^((DRR_ref - DRR_est) / 20) is calculated based on the DRR, and a smooth distance-adaptive compensation G_dB = min(12, 1.5·d_est) is applied. Finally, the output speech is normalized to -24LKFS loudness according to ITU-R BS.1770-4 and subjected to -1dBTP hard limiting protection.

[0057] In step S7, an online adaptive update is performed in the background. When an idle time exceeding 30 minutes is detected, a 5-second ambient sound sample is re-acquired and the RIR is estimated. If the spectral deviation between the old and new RIRs is greater than 3dB, an accelerated update factor λ = 0.85 is used; otherwise, a regular update factor λ = 0.95 is used. After the update is complete, the relevant parameters of WPE and the size of the first layer convolutional kernel of the CRM network are simultaneously refreshed.

[0058] A real-time voice enhancement device for a classroom setting according to an embodiment of the present invention may further include, from a hardware perspective, one or more sensors or input interfaces for acquiring geometric parameters of the classroom where the device is located; one or more processors; and a memory storing computer-executable instructions, which, when executed by the one or more processors, cause the device to perform operations to implement the method of the aforementioned embodiment.

[0059] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time speech enhancement method for classroom scenarios, characterized in that, Based on an interactive device, the method includes: Obtain the geometric parameters of the classroom where the interactive device is located; Based on the geometric parameters, calculate the theoretical reverberation time prior value of the classroom; Based on the geometric parameters, calculate the theoretical direct reverberation ratio prior value for the spatial location within the classroom; Using at least one of the theoretical reverberation time prior value and the theoretical direct reverberation ratio prior value as prior information, the room impulse response is estimated based on the audio signal collected by the microphone array using a blind system identification algorithm; Based on at least one of the room impulse response, theoretical reverberation time prior value, and theoretical direct reverberation ratio prior value, the parameters of the dereverberation algorithm based on the linear prediction algorithm are adaptively set. Based on the reverberation algorithm parameters, the linear prediction algorithm is executed on the audio signal to obtain the reverberated signal. The residual distortion of the reverberation-processed signal is suppressed using a machine learning model. By fusing features from at least two different modalities, target speaker speech segments can be detected. Based on the speech segment detection results of the target speaker, the audio stream is divided into at least two preset speech segment types; Differentiated audio processing is applied to different types of speech segments; Using the room impulse response and the target speaker's speech segments, the distance between the target speaker and the microphone array is estimated; The speech signal is subjected to distance-adaptive compensation based on the distance.

2. The real-time speech enhancement method for classroom scenarios according to claim 1, characterized in that, The methods for obtaining the geometric parameters of the classroom include: manual input, or using a built-in camera combined with a monocular depth estimation network, or using one or more depth sensors.

3. The real-time speech enhancement method for classroom scenarios according to claim 1, characterized in that, The linear prediction algorithm is a weighted prediction error algorithm, and the parameters of the dereverberation algorithm include the linear prediction order, prediction delay, and recursive least squares forgetting factor; furthermore, in the weighted prediction error algorithm, a spatial regularization term based on room impulse response is introduced to constrain the structure of the multi-input multi-output prediction matrix.

4. The real-time speech enhancement method for classroom scenarios according to claim 1, characterized in that, The machine learning model is a convolutional recurrent network for estimating complex ratio masks; and the method further includes: automatically adjusting the size of the first layer convolutional kernel of the encoder in the convolutional recurrent network based on the estimated theoretical reverberation time prior value.

5. The real-time speech enhancement method for classroom scenarios according to claim 1, characterized in that, The features of the at least two different modalities include at least two of the following: acoustic features, visual features based on human posture detection, and interactive context features of interactive devices; and when the visual features indicate that the target speaker is in a preset area, the detection sensitivity threshold of the target speaker's speech segment detection is adaptively reduced.

6. The real-time speech enhancement method for classroom scenarios according to claim 1, characterized in that, The speech segment types include continuous speech segments by the target speaker, short intermittent speech segments by the target speaker, speech segments by non-target speakers, and silent or noisy segments; the differentiated audio processing includes applying reverberation compression processing of different intensities to different types of speech segments.

7. The real-time speech enhancement method for classroom scenarios according to claim 1, characterized in that, The method further includes: during the idle period of the interactive device, using the natural sound sources in the classroom as excitation signals to automatically trigger the re-estimation of the room impulse response, and updating it with a weighted moving average of the historical room impulse response.

8. The real-time speech enhancement method for classroom scenarios according to claim 1, characterized in that, The method operates on an embedded heterogeneous on-chip system and includes: allocating different computing tasks in the method to different hardware units of the embedded heterogeneous on-chip system for execution; and using a multi-stage pipeline and a zero-copy data transfer mechanism based on direct memory access to achieve data sharing across hardware cores.

9. The real-time speech enhancement method for classroom scenarios according to claim 1, characterized in that, The method further includes: automatically identifying the current application mode based on the long-term statistical features of the target speaker's speech segment detection results, and adaptively adjusting the dereverberation intensity or processing delay according to the identified mode.

10. A real-time speech enhancement device for classroom scenarios, characterized in that, include: One or more sensors or input interfaces are used to acquire the geometric parameters of the classroom where the device is located; One or more processors; And a memory having computer-executable instructions thereon, which, when executed by the one or more processors, cause the device to perform operations to implement the method as described in any one of claims 1 to 9.

Images