A method and apparatus for audio processing based on semantic driving
By employing a semantic-driven audio processing method, utilizing spatial signal decomposition and statistical noise reduction techniques, and combining them with an acoustic semantic perception network, the problem of speech clarity and environmental situational awareness in existing auditory enhancement devices under complex acoustic environments is solved, achieving high-fidelity and low-latency audio processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2026-04-20
- Publication Date
- 2026-07-10
AI Technical Summary
Existing hearing enhancement devices struggle to balance speech clarity and environmental situation awareness in complex acoustic environments, and suffer from sound quality loss and algorithm latency issues, making it difficult to meet users' comprehensive performance requirements.
A semantically driven audio processing method is adopted, which processes the forward target signal and the backward environment signal through spatial signal decomposition, statistical noise reduction and acoustic semantic perception network respectively, and calculates dynamic pass-through weights using semantic complementary weight matrix to achieve weighted fusion of signals.
While suppressing environmental noise, it retains important acoustic information, enhances situational awareness, maintains high audio fidelity, meets the requirements for low-latency edge deployment, and improves users' voice clarity and environmental security awareness.
Smart Images
Figure CN122372918A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio signal processing and smart wearable hearing aids, and particularly to a semantically driven audio processing method and apparatus for hearing aids. Background Technology
[0002] Existing hearing enhancement devices (such as hearing aids, cochlear implants, TWS earphones, and AR glasses) mainly use directional beamforming technology for voice enhancement, which can only provide voice enhancement in the direction of the target in front, making it difficult to meet users' dual needs for both voice clarity and environmental situational awareness in complex acoustic environments.
[0003] Specifically, the existing technology has the following drawbacks:
[0004] First, the ability to perceive the environment is relatively weak. Traditional beamforming technology forcibly suppresses audio signals from the sides and rear by physically cutting off the space. This makes it difficult for users to perceive key acoustic events such as vehicle horns, environmental alarms, or shouts from behind while the voice in front is enhanced. This can lead to the "tunnel hearing" effect, which poses a safety hazard in dynamic environments.
[0005] Secondly, the fidelity of sound quality is easily compromised. End-to-end deep learning-based speech enhancement models are more prone to disrupting the phase integrity of the original speech in low signal-to-noise ratio or non-stationary noise environments, leading to speech tearing or auditory artifacts, resulting in a decline in specific perceptual indicators and affecting the speech recognition ability of hearing-impaired patients.
[0006] Third, the algorithm suffers from poor latency and edge deployment capabilities. Highly complex deep learning models have a large number of parameters, making it difficult to meet the stringent requirements of clinical-grade devices for total round-trip latency (typically required to be within 10 milliseconds) on resource-constrained low-power edge computing platforms or hearing aid DSP chips. The real-time factor (RTF) is also difficult to meet engineering deployment standards.
[0007] In summary, existing technologies struggle to achieve a harmonious balance between strong noise suppression, environmental situational awareness preservation, and ultra-low latency edge deployment, resulting in a poor user experience and failing to meet the comprehensive performance requirements of hearing enhancement devices in real-world application scenarios. Summary of the Invention
[0008] To address the shortcomings of existing technologies, this invention provides a semantically driven audio processing method and apparatus, overcoming the deficiencies of existing technologies.
[0009] To achieve the above objectives, the present invention provides the following technical solution:
[0010] This invention provides a semantically driven audio processing method, comprising the following steps:
[0011] Step S1: Acquire multi-channel audio signals and perform spatial signal decomposition on the multi-channel audio signals to extract forward target signals and backward environment signals;
[0012] Step S2: Perform denoising processing based on statistical models on the forward target signal and the backward environment signal respectively, and calculate the corresponding forward denoising gain and backward denoising gain;
[0013] Step S3: Input the forward target signal and the backward environment signal into the acoustic semantic perception network respectively, and output the corresponding forward semantic probability vector and backward semantic probability vector;
[0014] Step S4: Based on the forward semantic probability vector, the backward semantic probability vector, and the preset semantic complementary weight matrix, calculate the dynamic pass-through weight for the backward environmental signal.
[0015] Step S5: Use the forward noise reduction gain to perform gain processing on the forward target signal, use the backward noise reduction gain and the dynamic pass-through weight to perform gain processing on the backward environmental signal, and then perform weighted fusion of the processed forward and backward signals to output an enhanced audio signal.
[0016] In one embodiment, spatial signal decomposition of the multi-channel audio signal specifically includes:
[0017] Construct two beamforming systems with different null orientations to acquire forward beam signals and backward beam signals respectively;
[0018] Calculate the energy ratio of the forward beam signal to the backward beam signal to generate a spatial beam masking value;
[0019] Based on the spatial beam masking value, a forward target signal with a first beamwidth and a backward environment signal with a second beamwidth are generated respectively.
[0020] In one embodiment, the denoising process based on a statistical model for the forward target signal and the backward environment signal specifically includes:
[0021] An improved minimum controlled recursive averaging algorithm with dual-path deployment is used to independently track the noise power spectral density of the forward target signal and the backward environment signal.
[0022] A logarithmic spectral amplitude estimator is used in conjunction with a decision-oriented method to estimate the prior signal-to-noise ratio, and the forward noise reduction gain and the backward noise reduction gain are calculated respectively.
[0023] In one embodiment, before inputting the forward target signal and the backward environment signal into the acoustic semantic perception network, a low-power pre-processing step is included, the determination step of which is as follows:
[0024] Real-time detection of the sound pressure level of the current audio signal;
[0025] If the current sound pressure level is lower than the preset sound pressure level threshold, the system is configured to skip the network inference step, determine the current semantic category as a quiet environment, and output a preset static pass-through weight to replace the dynamic pass-through weight for gain processing.
[0026] In one embodiment, the dynamic pass-through weight is calculated as follows:
[0027] Calculate the target pass-through weight:
[0028] ,
[0029] in, This is the transpose of the forward semantic probability vector. The pre-defined semantic complementarity weight matrix, The backward semantic probability vector; The elements in the matrix are configured to: increase the weight value of the corresponding element when the semantic category of the backward environmental signal is a significant environmental event;
[0030] The target pass-through weights are smoothed using a first-order recursive filter to obtain the dynamic pass-through weights for the current frame:
[0031] ,
[0032] in, The dynamic pass-through weight for the current frame. This is the preset smoothing coefficient.
[0033] In one embodiment, the method is applied to a heterogeneous computing platform with a central processing unit and a neural network processor:
[0034] The spatial signal decomposition and statistical model-based noise reduction in steps S1 and S2 are performed on the central processing unit.
[0035] The acoustic semantic perception network in step S3, after being quantized with low precision, runs on the neural network processor.
[0036] The present invention also provides a semantic-driven audio processing apparatus for implementing the semantic-driven audio processing method described in any of the preceding claims, the apparatus comprising:
[0037] Spatial decomposition module: used to acquire multi-channel audio signals and perform spatial decomposition to obtain forward target signals and backward environment signals;
[0038] Statistical noise reduction module: used to perform noise tracking and gain estimation on the forward target signal and the backward environment signal respectively, to obtain forward noise reduction gain and backward noise reduction gain;
[0039] Semantic perception module: used to determine the acoustic semantic probabilities of the forward target signal and the backward environment signal respectively, and output the forward semantic probability vector and the backward semantic probability vector;
[0040] Dynamic weighting module: used to calculate dynamic pass-through weights based on the forward semantic probability vector, the backward semantic probability vector, and the preset semantic complementary weight matrix;
[0041] Weighted fusion module: used to receive the forward noise reduction gain, the backward noise reduction gain and the dynamic pass-through weight, and to perform weighted fusion of the processed forward and backward signals to output an enhanced audio signal.
[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the audio processing method described in any of the preceding claims.
[0043] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the audio processing method described in any of the preceding claims.
[0044] This invention provides a semantically driven audio processing method and apparatus, which has the following advantages: by performing spatial signal decomposition on multi-channel audio signals, forward target signals and backward environmental signals are extracted, and the semantic categories of the two signals are determined based on an acoustic semantic perception network. Furthermore, the dynamic transmission weight of the backward environmental signal is calculated through a semantic complementary weight matrix. This mechanism enables the solution to suppress environmental noise while adaptively retaining important acoustic information from the side and rear based on the semantic category of the backward signal (such as significant environmental events such as speech and music), effectively avoiding the "tunneling" effect and improving the user's situational awareness.
[0045] This scheme employs a dual-path architecture of "statistical denoising + semantic gating," performing statistical model-based denoising on both the forward target signal and the backward environment signal. An improved minimum controlled recursive averaging (IMCRA) algorithm and a log-MMSE estimator are used for noise tracking and gain calculation. Compared to end-to-end black-box models, this statistical denoising method better preserves the phase integrity of the original speech, avoiding speech tearing or auditory artifacts caused by nonlinear model processing, thus ensuring high fidelity of the output audio.
[0046] This solution divides the signal processing task and the neural network inference task into heterogeneous computing platforms: spatial signal decomposition and statistical noise reduction run on the central processing unit (CPU), while the acoustic semantic perception network runs on the neural network processor (NPU) after low-precision quantization. This architecture makes full use of the heterogeneous computing resources of edge devices, and at the same time, skips network inference in low sound pressure level scenarios through low-power preprocessing steps, effectively reducing computing power consumption and processing latency, and meeting the strict real-time requirements of hearing aid devices.
[0047] Furthermore, this scheme dynamically adjusts the pass-through weights of the backward environmental signal through a semantic complementary weight matrix. When the backward semantic probability vector points to significant environmental events such as speech or music, the dynamic pass-through weights automatically increase the corresponding weight values; when the backward signal is noise, the weights remain low. This mechanism simulates the selective attention characteristics of the auditory system, achieving intelligent gating of "on-demand pass-through." Simultaneously, this scheme uses a first-order recursive filter to smooth the calculated target pass-through weights, effectively avoiding energy jumps caused by semantic category switching, making the pass-through weight changes smoother, and further improving the user's auditory comfort.
[0048] Ultimately, this solution solves the technical problem of simultaneously achieving noise suppression, environmental awareness, and low-latency deployment in existing technologies. Under stringent hardware computing power constraints, it achieves an organic unity of high-fidelity voice enhancement, environmental situational awareness preservation, and ultra-low-latency edge deployment. Users experience significant improvements in voice clarity, environmental safety awareness, and auditory comfort when using hearing enhancement devices. Attached Figure Description
[0049] Figure 1 This is a schematic diagram of the overall framework of a neural-statistical hybrid architecture audio processing system (SA-OLF) provided in an embodiment of the present invention;
[0050] Figure 2 This is a schematic diagram of beam space polar coordinates in an embodiment of the present invention, wherein (a) is a polar coordinate diagram of the forward target signal and (b) is a polar coordinate diagram of the backward environment signal; Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0052] See attached document Figures 1-2 As shown, in one embodiment, a semantically driven audio processing method includes the following steps:
[0053] Step S1: Acquire multi-channel audio signals and perform spatial signal decomposition on the multi-channel audio signals to extract forward target signals and backward environment signals;
[0054] Specifically, spatial signal decomposition of the multi-channel audio signal includes:
[0055] Construct two beamforming systems with different null orientations to acquire forward beam signals and backward beam signals respectively;
[0056] Calculate the energy ratio of the forward beam signal to the backward beam signal to generate a spatial beam masking value;
[0057] Based on the spatial beam masking value, a forward target signal with a first beamwidth and a backward environment signal with a second beamwidth are generated respectively.
[0058] Specifically, first, a beamforming system with two null points located at 0° and 180° respectively is constructed; the frequency domain expressions of the two signals are as follows:
[0059] ,
[0060] ,
[0061] in The forward beam signal has its null point at 180°. The backbeam signal has its null point at 0°. Indicates frequency point, Indicates the current frame. The system sampling rate, For the spacing between the dual microphone arrays, The number of points in the Fourier transform. The speed of sound in air is approximately ;
[0062] via forward beam signal With backward beam signal ratio Obtain the beam masking matrix The forward target signal is further generated. ,
[0063] ,
[0064] ,
[0065] ,
[0066] Similarly, the backward environment signal is generated. :
[0067] ,
[0068] ,
[0069] ,
[0070] like Figure 2 As shown in (a) and (b), the forward target signal generated in the above manner has a first beamwidth, the backward environment signal has a second beamwidth, and the first beamwidth is narrower than the second beamwidth; the physical robustness of this spatial decomposition overcomes the phase distortion problem of the black box model under low signal-to-noise ratio.
[0071] Step S2: Perform denoising processing based on statistical models on the forward target signal and the backward environment signal respectively, and calculate the corresponding forward denoising gain and backward denoising gain;
[0072] Specifically, the denoising processing of the forward target signal and the backward environment signal based on statistical models includes:
[0073] An improved minimum controlled recursive averaging (IMCRA) algorithm with dual-path deployment is used to independently track the noise power spectral density of the forward target signal and the backward environment signal.
[0074] The Log-MMSE estimator is used in conjunction with a decision-oriented method to estimate the prior signal-to-noise ratio, and the forward denoising gain and the backward denoising gain are calculated respectively.
[0075] Specifically, for the decomposed forward target signal and the backward environment signal, the noise power spectral density is estimated independently using the Improved Minimum Controlled Recursive Average (IMCRA) algorithm. Subsequently, for each signal, a Log-MMSE estimator is used in conjunction with a decision-oriented method to calculate the gain function. The forward noise reduction gain and the backward noise reduction gain are obtained respectively.
[0076] Step S3: Input the forward target signal and the backward environment signal into the acoustic semantic perception network (the acoustic semantic perception network is a feature quantization convolutional neural network, which adopts a residual connection structure combined with a global response normalization mechanism) respectively, and output the corresponding forward semantic probability vector and backward semantic probability vector.
[0077] Before inputting the forward target signal and the backward environment signal into the acoustic semantic perception network, a low-power pre-processing step is included, the determination steps of which are as follows:
[0078] Real-time detection of the sound pressure level of the current audio signal;
[0079] If the current sound pressure level is lower than the preset sound pressure level threshold (e.g., 25 dB SPL), the network inference step is skipped, the current semantic category is determined to be "quiet environment", and the preset static pass-through weight is output to replace the dynamic pass-through weight for gain processing.
[0080] Specifically, in order to reduce the power consumption of edge devices, the system detects the sound pressure level (SPL) of the current audio signal in real time before inputting the forward target signal and the backward environment signal into the acoustic semantic perception network.
[0081] If the current sound pressure level is detected to be lower than a preset sound pressure level threshold, the system is configured to skip subsequent network inference steps, determine the current semantic category as a quiet environment, and output a preset static pass-through weight to replace the dynamic pass-through weight for subsequent gain processing. The preset sound pressure level threshold is set between 20 dB SPL and 30 dB SPL. This mechanism effectively avoids unnecessary semantic misjudgments caused by white noise under extremely low sound pressure conditions and significantly saves NPU computing power.
[0082] Step S4: Based on the forward semantic probability vector, the backward semantic probability vector, and the preset semantic complementary weight matrix, calculate the dynamic pass-through weight for the backward environmental signal.
[0083] The dynamic pass-through weight is calculated as follows:
[0084] Calculate the target pass-through weight:
[0085] ,
[0086] in, This is the transpose of the forward semantic probability vector. The pre-defined semantic complementarity weight matrix, The backward semantic probability vector; The elements in the matrix are configured to increase the weight of the corresponding element when the semantic category of the backward environmental signal is a significant environmental event (such as speech or music).
[0087] To eliminate sudden energy jumps caused by semantic category switching, the target pass-through weights are smoothed using a first-order recursive filter to obtain the dynamic pass-through weights for the current frame:
[0088] ,
[0089] in, The dynamic pass-through weight for the current frame. This is the preset smoothing coefficient.
[0090] Step S5: Use the forward noise reduction gain to perform gain processing on the forward target signal, use the backward noise reduction gain and the dynamic pass-through weight to perform gain processing on the backward environmental signal, and then perform weighted fusion of the processed forward and backward signals to output an enhanced audio signal.
[0091] Specifically, construct a semantic complementarity matrix of dimension N×N. Elements in the matrix Used to represent robust pass-through weights when the current forward channel belongs to category i and the backward channel belongs to category j;
[0092] Utilizing the aforementioned forward noise reduction gain For the forward target signal Gain processing is performed, utilizing the backward noise reduction gain. and the dynamic pass-through weight For the backward environmental signal Gain processing is performed, and the processed forward and backward signals are weighted and fused to output an enhanced audio signal.
[0093] ,
[0094] For example, the The matrix can be configured in the following ways:
[0095] When the forward semantic probability vector points to the speech category and the backward semantic probability vector points to the noise category, the corresponding elements are... Configured to approach 0;
[0096] When the backward semantic probability vector points to a speech category or a music category, the corresponding element will be... Configured to be greater than a preset first threshold;
[0097] When both the forward semantic probability vector and the backward semantic probability vector point to the noise category, the corresponding elements will be... Configured to be between a preset second threshold and a third threshold;
[0098] Based on the semantic complementary weight matrix The transpose of the forward semantic probability vector The backward semantic probability vector The target pass-through weight is calculated. ;
[0099] Furthermore, the target is passed through with weights. A first-order recursive filter is used for smoothing to obtain the final dynamic pass-through weights for the current frame. .
[0100] In a specific embodiment, the method is applied to a heterogeneous computing platform equipped with a central processing unit (CPU) and a neural network processor (NPU):
[0101] The spatial signal decomposition and statistical model-based noise reduction in steps S1 and S2 are performed on the CPU.
[0102] The acoustic semantic perception network in step S3, after low-precision quantization, runs on the NPU.
[0103] The following describes a semantic-driven audio processing device provided by the present invention. The semantic-driven audio processing device described below and the semantic-driven audio processing method described above can be referred to in correspondence.
[0104] In one embodiment, a semantically driven audio processing device includes a spatial decomposition module, a statistical noise reduction module, a semantic perception module, a dynamic weighting module, and a weighted fusion module.
[0105] Spatial decomposition module: used to acquire multi-channel audio signals and perform spatial decomposition to obtain forward target signals and backward environment signals; and output the forward target signals to the statistical noise reduction module and the semantic perception module respectively, and output the backward environment signals to the statistical noise reduction module and the semantic perception module respectively.
[0106] Statistical denoising module: connected to the spatial decomposition module, used to perform noise tracking and gain estimation on the forward target signal and the backward environment signal respectively, to obtain forward denoising gain and backward denoising gain; and output the forward denoising gain and backward denoising gain to the weighted fusion module;
[0107] The semantic perception module is connected to the spatial decomposition module and is used to determine the acoustic semantic probabilities of the forward target signal and the backward environment signal respectively, output the forward semantic probability vector and the backward semantic probability vector, and output the forward semantic probability vector and the backward semantic probability vector to the dynamic weighting module.
[0108] Dynamic weighting module: connected to the semantic perception module, used to calculate dynamic pass-through weights based on the forward semantic probability vector, the backward semantic probability vector and the preset semantic complementary weight matrix;
[0109] Weighted fusion module: connected to the statistical noise reduction module and the dynamic weighting module respectively, used to receive the forward noise reduction gain, the backward noise reduction gain and the dynamic pass-through weight, use the forward noise reduction gain to perform gain processing on the forward target signal, use the backward noise reduction gain and the dynamic pass-through weight to perform gain processing on the backward environmental signal, and perform weighted fusion of the processed forward and backward signals to output an enhanced audio signal.
[0110] In one embodiment, an electronic device, which may be a smart terminal, includes a processor, internal memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a semantic-driven audio processing method, which includes:
[0111] Acquire multi-channel audio signals and perform spatial signal decomposition on the multi-channel audio signals to extract forward target signals and backward environment signals;
[0112] The forward target signal and the backward environment signal are respectively subjected to denoising processing based on statistical models, and the corresponding forward denoising gain and backward denoising gain are calculated.
[0113] The forward target signal and the backward environment signal are respectively input into the acoustic semantic perception network, and the corresponding forward semantic probability vector and backward semantic probability vector are output.
[0114] Based on the forward semantic probability vector, the backward semantic probability vector, and the preset semantic complementary weight matrix, the dynamic pass-through weight for the backward environmental signal is calculated.
[0115] The forward target signal is amplified using the forward noise reduction gain, and the backward environmental signal is amplified using the backward noise reduction gain and the dynamic pass-through weight. The processed forward and backward signals are then weighted and fused to output an enhanced audio signal.
[0116] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements a semantically driven audio processing method, the method comprising:
[0117] Acquire multi-channel audio signals and perform spatial signal decomposition on the multi-channel audio signals to extract forward target signals and backward environment signals;
[0118] The forward target signal and the backward environment signal are respectively subjected to denoising processing based on statistical models, and the corresponding forward denoising gain and backward denoising gain are calculated.
[0119] The forward target signal and the backward environment signal are respectively input into the acoustic semantic perception network, and the corresponding forward semantic probability vector and backward semantic probability vector are output.
[0120] Based on the forward semantic probability vector, the backward semantic probability vector, and the preset semantic complementary weight matrix, the dynamic pass-through weight for the backward environmental signal is calculated.
[0121] The forward target signal is amplified using the forward noise reduction gain, and the backward environmental signal is amplified using the backward noise reduction gain and the dynamic pass-through weight. The processed forward and backward signals are then weighted and fused to output an enhanced audio signal.
[0122] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, it implements a semantically driven audio processing method, the method comprising:
[0123] Acquire multi-channel audio signals and perform spatial signal decomposition on the multi-channel audio signals to extract forward target signals and backward environment signals;
[0124] The forward target signal and the backward environment signal are respectively subjected to denoising processing based on statistical models, and the corresponding forward denoising gain and backward denoising gain are calculated.
[0125] The forward target signal and the backward environment signal are respectively input into the acoustic semantic perception network, and the corresponding forward semantic probability vector and backward semantic probability vector are output.
[0126] Based on the forward semantic probability vector, the backward semantic probability vector, and the preset semantic complementary weight matrix, the dynamic pass-through weight for the backward environmental signal is calculated.
[0127] The forward target signal is amplified using the forward noise reduction gain, and the backward environmental signal is amplified using the backward noise reduction gain and the dynamic pass-through weight. The processed forward and backward signals are then weighted and fused to output an enhanced audio signal.
[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.
[0129] By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0130] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A semantically driven audio processing method, characterized in that, Includes the following steps: Step S1: Acquire multi-channel audio signals and perform spatial signal decomposition on the multi-channel audio signals to extract forward target signals and backward environment signals; Step S2: Perform denoising processing based on statistical models on the forward target signal and the backward environment signal respectively, and calculate the corresponding forward denoising gain and backward denoising gain; Step S3: Input the forward target signal and the backward environment signal into the acoustic semantic perception network respectively, and output the corresponding forward semantic probability vector and backward semantic probability vector; Step S4: Based on the forward semantic probability vector, the backward semantic probability vector, and the preset semantic complementary weight matrix, calculate the dynamic pass-through weight for the backward environmental signal. Step S5: Use the forward noise reduction gain to perform gain processing on the forward target signal, use the backward noise reduction gain and the dynamic pass-through weight to perform gain processing on the backward environmental signal, and then perform weighted fusion of the processed forward and backward signals to output an enhanced audio signal.
2. The audio processing method according to claim 1, characterized in that, The spatial signal decomposition of the multi-channel audio signal specifically includes: Construct two beamforming systems with different null orientations to acquire forward beam signals and backward beam signals respectively; Calculate the energy ratio of the forward beam signal to the backward beam signal to generate a spatial beam masking value; Based on the spatial beam masking value, a forward target signal with a first beamwidth and a backward environment signal with a second beamwidth are generated respectively.
3. The audio processing method according to claim 1, characterized in that, The denoising process based on statistical models for the forward target signal and the backward environment signal specifically includes: An improved minimum controlled recursive averaging algorithm with dual-path deployment is used to independently track the noise power spectral density of the forward target signal and the backward environment signal. A logarithmic spectral amplitude estimator is used in conjunction with a decision-oriented method to estimate the prior signal-to-noise ratio, and the forward noise reduction gain and the backward noise reduction gain are calculated respectively.
4. The audio processing method according to claim 1, characterized in that, Before inputting the forward target signal and the backward environment signal into the acoustic semantic perception network, a low-power pre-processing step is included, the determination steps of which are as follows: Real-time detection of the sound pressure level of the current audio signal; If the current sound pressure level is lower than the preset sound pressure level threshold, the system is configured to skip the network inference step, determine the current semantic category as a quiet environment, and output a preset static pass-through weight to replace the dynamic pass-through weight for gain processing.
5. The audio processing method according to claim 1, characterized in that, The dynamic pass-through weight is calculated as follows: Calculate the target pass-through weight: ， in, This is the transpose of the forward semantic probability vector. The pre-defined semantic complementarity weight matrix, The backward semantic probability vector; The elements in the matrix are configured to: increase the weight value of the corresponding element when the semantic category of the backward environmental signal is a significant environmental event; The target pass-through weights are smoothed using a first-order recursive filter to obtain the dynamic pass-through weights for the current frame: ， in, The dynamic pass-through weight for the current frame. This is the preset smoothing coefficient.
6. The audio processing method according to claim 1, characterized in that, The method is applied to heterogeneous computing platforms equipped with central processing units and neural network processors: The spatial signal decomposition and statistical model-based noise reduction in steps S1 and S2 are performed on the central processing unit. The acoustic semantic perception network in step S3, after being quantized with low precision, runs on the neural network processor.
7. A semantically driven audio processing apparatus for implementing the audio processing method according to any one of claims 1 to 6, characterized in that, include: space Decomposition module: used to acquire multi-channel audio signals and perform spatial decomposition to obtain forward target signals and backward environment signals; Statistical noise reduction module: used to perform noise tracking and gain estimation on the forward target signal and the backward environment signal respectively, to obtain forward noise reduction gain and backward noise reduction gain; Semantic perception module: used to determine the acoustic semantic probabilities of the forward target signal and the backward environment signal respectively, and output the forward semantic probability vector and the backward semantic probability vector; Dynamic weighting module: used to calculate dynamic pass-through weights based on the forward semantic probability vector, the backward semantic probability vector, and the preset semantic complementary weight matrix; Weighted fusion module: used to receive the forward noise reduction gain, the backward noise reduction gain and the dynamic pass-through weight, and to perform weighted fusion of the processed forward and backward signals to output an enhanced audio signal.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the audio processing method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the audio processing method according to any one of claims 1 to 6.