VR large-space active noise reduction and directional speech enhancement system

Abstract

The invention discloses a VR large-space active noise reduction and directional speech enhancement system, particularly relates to the technical field of speech enhancement, and aims to obtain a user attention direction by fusing multi-mode sensor data in user wearing equipment, guide a sound source positioning module to quickly lock a speech interaction main direction in a multi-sound-source interference environment, and improve the speech enhancement efficiency. Non-target speakers are further filtered through voiceprint feature recognition, it is ensured that enhanced voices have individual consistency, through deep anti-reverberation modeling, reflection residual sounds generated by multi-path propagation are effectively stripped, semantic fuzziness caused by pseudo sound source aliasing is avoided, a voice enhancement and active noise reduction module corrects the frequency spectrum phase while enhancing the amplitude spectrum, and the noise reduction effect is improved. The definition and naturalness of output voice are remarkably improved, the enhanced focusing performance of the system is monitored in real time by introducing a target locking drift coefficient and a beam pointing deviation coefficient, and adaptive reconstruction is triggered based on a closed-loop correction index.

Description

Technical Field

[0001] The present invention relates to the technical field of speech enhancement, and more specifically, to a VR large-space active noise reduction and directional speech enhancement system. Background Art

[0002] In a virtual reality (VR) large-space interaction scenario, speech, as a core human-computer interaction channel, its enhancement and noise reduction effects directly affect the immersion and interaction efficiency. Current mainstream directional speech enhancement systems usually rely on microphone arrays for beamforming and sound source tracking, suppressing environmental noise and enhancing the target speech signal through spatial focusing. However, in a VR space with multiple speakers speaking simultaneously, frequent target switching, and complex scene structures, existing solutions expose significant technical bottlenecks.

[0003] Specifically, when the system faces dynamic speakers, there is a response lag at the physical and algorithmic levels in beam tracking, resulting in the beam failing to point to the current target speaker in a timely manner, causing the enhancement target to shift. At the same time, if the system enhances the speech in the wrong direction, it will not only introduce semantic drift but may also strengthen the non-direct sound caused by factors such as reflection and occlusion in the scene, forming a pseudo sound source. Under multi-path propagation conditions, this type of pseudo sound source is mixed with the real speech, which will further mislead the system's direction estimation, causing the subsequent beam to continuously deviate from the target and even fall into a closed-loop state of "wrong enhancement - pseudo sound feedback - wrong tracking", seriously affecting the stability of speech enhancement and the continuity of interaction. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a VR large-space active noise reduction and directional speech enhancement system to solve the problems raised in the above background art.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A VR large-space active noise reduction and directional speech enhancement system, comprising a modality perception module, a sound source localization module, a target speech locking module, a reflected sound recognition and cancellation module, a speech enhancement and active noise reduction module, and a closed-loop correction module;

[0007] The modality perception module is configured to receive multi-modal sensor data from a user-worn device, and obtain the user's attention direction vector based on the multi-modal data, and use the user's attention direction vector as a candidate speech interaction direction;

[0008] The sound source localization module is configured to capture multi-channel sound signals in the space according to a microphone array and combine with the candidate speech interaction direction to obtain the main beam pointing, and generate a dynamic beamforming weight matrix;

[0009] A target voice locking module, which is used to obtain the user voiceprint feature on the main beam pointing direction, match it with the pre-registered user voiceprint feature, and lock the target speaker;

[0010] A reflected sound recognition and cancellation module, which is used to input the voice signal of the target speaker into a deep dereverberation modeling network to remove the spatial reverberation residue and obtain the direct sound with the reflected sound feature stripped;

[0011] A voice enhancement and active noise reduction module, which is used to input the voice signal with the feature stripped into a joint structure of a dual-path recurrent neural network and a complex mask generation network to perform voice spectrum enhancement and active noise reduction output on the language signal;

[0012] A closed-loop correction module, which is used to obtain the drift information of the target lock and the beam deviation information of the beam pointing direction, construct a closed-loop correction mechanism, and trigger adaptive reconstruction.

[0013] In a preferred embodiment, the multimodal data includes:

[0014] An eye movement tracking vector: denoted as A head pose vector: denoted as A gesture focus vector: denoted as

[0015] Construct a comprehensive attention vector according to the three modal vectors: Where represents the comprehensive attention vector, a1, a2, and a3 respectively represent the preset proportionality coefficients of the eye movement tracking vector, the head pose vector, and the gesture focus vector, and a1, a2, and a3 are all greater than 0;

[0016] Judge whether the modal direction is concentrated: calculate the included angle between the eye movement tracking vector and the head pose vector, and the calculation formula is as follows: Where θeh represents the included angle between the eye movement tracking vector and the head pose vector, represents the dot product of the eye movement tracking vector and the head pose vector;

[0017] Compare the included angle between the eye movement tracking vector and the head pose vector with the preset included angle threshold of the eye movement tracking vector and the head pose vector to obtain the user's attention direction vector, specifically as follows:

[0018] If the included angle between the eye movement tracking vector and the head pose vector is less than the included angle threshold of the eye movement tracking vector and the head pose vector, then use the current comprehensive attention vector as the user's attention direction vector, and at the same time use the user's attention direction vector as the candidate voice interaction direction.

[0019] In a preferred embodiment, through a microphone array Mk = {mk1, mk2,..., mk I}, collect the time-domain acoustic signals on each microphone channel in real time, denoted as Xh i (t), where i represents the i-th microphone and t is the time variable;

[0020] Calculate the time delay distribution for any pair of microphones i and j: where XH i (f) represents the Fourier transform of Xh i (t), represents the conjugate of Xh j (t), Xh j (t) represents the Fourier transform of Xh j (t), Fh -1 represents the inverse transform of the frequency-domain signal, back to the time domain, τ ij represents the time delay, Rs ij (τ ij ) represents the time delay distribution from the sound source to microphones i and j;

[0021] By constructing a search network θ for the target direction and θ and respectively represent the azimuth angle and the elevation angle, accumulate the cost function for the time delays of all microphone pairs, and form a sound source focusing response map in the angular space: where represents the expected time delay in the target direction , represents the estimated sound source direction;

[0022] Combine each estimated sound source direction with the comprehensive attention vector to calculate the direction confidence: where ZX represents the confidence, corresponds to a set of estimated sound source directions

[0023] Select the estimated sound source direction with the maximum direction confidence as the main beam pointing;

[0024] Generate the beam weight: where represents the beam weight, XF represents the covariance matrix formed by the microphone array sampling signals, is the steering vector generated for the main beam pointing, represents the conjugate transpose of the steering vector.

[0025] In a preferred embodiment, output the filtered signal on the main beam pointing through a beamformer, input the output filtered signal into a voiceprint feature extraction network to extract the user's voiceprint feature vector, and perform similarity matching between the user's voiceprint feature vector and the pre-registered user voiceprint feature template, using the normalized cosine similarity as the judgment criterion: where sim represents the cosine similarity between the voiceprint feature vector of the user and the pre-registered voiceprint feature vector of the user, represents the voiceprint feature vector of the user, represents the pre-registered voiceprint feature vector of the user

[0026] Compare the cosine similarity with a preset similarity threshold. If the cosine similarity is greater than the similarity threshold, mark the user of the current sound source as the target speaker.

[0027] In a preferred embodiment, the drift information of the target lock includes a target lock drift coefficient, and the beam deviation information of the beam direction includes a beam direction deviation coefficient.

[0028] In a preferred embodiment, the acquisition logic of the target lock drift coefficient is as follows:

[0029] Obtain the cosine similarity sequence {sim(t n )} of the user's voiceprint feature matching from the target voice lock module, where sim(t n ) represents the cosine similarity at the nth time point;

[0030] Calculate the similarity mean value of the cosine similarity sequence: where represents the similarity mean value, n = {1, 2,..., N}, and N is a positive integer;

[0031] Calculate the similarity standard deviation of the cosine similarity sequence: where σsim represents the similarity standard deviation;

[0032] Calculate the target lock drift coefficient: where mbs represents the target lock drift coefficient.

[0033] In a preferred embodiment, the acquisition logic of the beam direction deviation coefficient is as follows:

[0034] Obtain the historical sound source estimation direction sequence where represents the historical sound source estimation direction at the mth time point;

[0035] Perform time series weighting on the historical sound source estimation direction sequence: where is the sound source estimation direction after time series weighting, jq m represents the direction weight of the historical sound source estimation direction at the mth time point, jq m = e -(td-tm) , where td represents the current moment, t mIt represents the m-th time point, where m = {1, 2, ..., M} and M is a positive integer;

[0036] Calculate the direction divergence coefficient: where σθ represents the direction divergence coefficient;

[0037] Calculate the beam pointing deviation angle: where δθ represents the current beam pointing deviation angle, represents the current main beam pointing direction;

[0038] Calculate the beam pointing deviation coefficient: where bsz represents the beam pointing deviation coefficient.

[0039] In a preferred embodiment, a closed-loop correction mechanism is constructed based on the target locking drift coefficient and the beam pointing deviation coefficient to determine whether to trigger adaptive reconstruction, specifically as follows:

[0040] Construct a closed-loop correction model based on the target locking drift coefficient and the beam pointing deviation coefficient, and output the closed-loop correction index zsa. The formula on which the model is based is as follows: zsa = b1 * Qzt + b2 * bsz, where b1 and b2 respectively represent the preset proportionality coefficients of the target locking drift coefficient and the beam pointing deviation coefficient, and both b1 and b2 are greater than 0;

[0041] Compare the closed-loop correction index with the preset closed-loop correction index threshold. If the closed-loop correction index is greater than the closed-loop correction index threshold, trigger adaptive reconstruction; if the closed-loop correction index is less than or equal to the closed-loop correction index threshold, there is no need to trigger adaptive reconstruction.

[0042] The technical effects and advantages of the present invention:

[0043] 1. In the present invention, the modal perception module fuses the multi-modal sensor data in the user's wearable device to obtain the user's attention direction, guides the sound source localization module to quickly lock the main direction of voice interaction in a multi-source interference environment, the target voice locking module further filters out non-target speakers using voiceprint feature recognition to ensure the individual consistency of the enhanced voice, the reflected sound recognition and cancellation module effectively strips the reflected reverberation generated by multi-path propagation through deep dereverberation modeling, avoiding semantic ambiguity caused by pseudo-source aliasing, the voice enhancement and active noise reduction module adopts a collaborative structure of DPRNN and complex mask network to correct the spectral phase while enhancing the amplitude spectrum, significantly improving the clarity and naturalness of the output voice, and the closed-loop correction module introduces the target locking drift coefficient and the beam pointing deviation coefficient to monitor the enhancement focusing performance of the system in real time, and triggers adaptive reconstruction based on the closed-loop correction index, effectively preventing the system from falling into the closed-loop error mode of "mis-enhancement - pseudo-sound feedback - incorrect tracking".

[0044] 2. The present invention can not only improve the extraction accuracy and enhancement quality of the target voice in dynamic and complex scenarios, but also enhance the robustness of the system against interference and errors. It can continuously maintain high immersion, high responsiveness, and high interaction efficiency in VR large-space human-computer interaction, achieve high-precision extraction and dynamic robust enhancement of the target voice, and comprehensively improve the immersion, continuity, and semantic accuracy of voice interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] For the convenience of those skilled in the art to understand, the present invention will be further described below in conjunction with the accompanying drawings;

[0046] Figure 1 It is a flowchart of the system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0048] Embodiment: The present invention provides a VR large-space active noise reduction and directional voice enhancement system as shown in Figure 1 which includes a modality perception module, a sound source localization module, a target voice locking module, a reverberant sound recognition and cancellation module, a voice enhancement and active noise reduction module, and a closed-loop correction module;

[0049] The modality perception module is configured to receive multi-modal sensor data from a user-worn device, and obtain the user's attention direction vector according to the multi-modal data, and use the user's attention direction vector as a candidate voice interaction direction;

[0050] The sound source localization module is configured to capture multi-channel sound signals in space according to a microphone array and combine the candidate voice interaction direction to obtain a main beam direction, and generate a dynamic beamforming weight matrix;

[0051] The target voice locking module is configured to obtain the user's voiceprint feature on the main beam direction and match it with the pre-registered user's voiceprint feature to lock the target speaker;

[0052] The reverberant sound recognition and cancellation module is configured to input the voice signal of the target speaker into a deep dereverberation modeling network to remove the space reverberation residue, and obtain the direct sound with the reverberant sound feature stripped;

[0053] The voice enhancement and active noise reduction module is configured to input the voice signal with the feature stripped into a combined structure of a dual-path recurrent neural network and a complex mask generation network to perform voice spectrum enhancement and active noise reduction on the language signal and output;

[0054] A closed-loop correction module, configured to obtain the drift information of target locking and the beam deviation information of beam pointing to construct a closed-loop correction mechanism to trigger adaptive reconstruction;

[0055] A modality perception module, configured to receive multi-modal sensor data from a user-worn device, obtain a user's attention direction vector according to the multi-modal data, and use the user's attention direction vector as a candidate voice interaction direction;

[0056] The said multi-modal data includes:

[0057] An eye movement tracking vector: denoted as Used to represent the line-of-sight direction of the user's current gaze;

[0058] A head pose vector: denoted as Used to represent the orientation of the user's head;

[0059] A gesture focus vector: denoted as Used to represent the direction of the interaction area controlled by the user through gestures;

[0060] Construct a comprehensive attention vector according to the three modality vectors: Where represents the comprehensive attention vector, a1, a2, and a3 respectively represent the preset proportionality coefficients of the eye movement tracking vector, the head pose vector, and the gesture focus vector, and a1, a2, and a3 are all greater than 0;

[0061] It should be noted that the above formulas are all dimensionless and take their numerical calculations. Common methods for removing dimensions include Min-Max normalization, Z-Score standardization, etc., which will not be elaborated here; a1, a2, and a3 are set according to the actual situation. For example, the expert weighting method is adopted, that is, relevant experts are invited to determine the preset proportionality coefficients of each index through professional opinion surveys and comprehensive evaluations. For example, a1, a2, and a3 can be 0.3, 0.4, 0.3;

[0062] Judge whether the modality directions are concentrated: Calculate the angle between the eye movement tracking vector and the head pose vector, and the calculation formula is as follows: Where θeh represents the angle between the eye movement tracking vector and the head pose vector,

[0063] Compare the angle between the eye movement tracking vector and the head pose vector with the preset angle threshold between the eye movement tracking vector and the head pose vector to obtain the user's attention direction vector, specifically as follows:

[0064] If the angle between the eye movement tracking vector and the head pose vector is less than the angle threshold between the eye movement tracking vector and the head pose vector, it indicates that the user's line of sight and the head direction are relatively concentrated and consistent. The current comprehensive attention vector is used as the user's attention direction vector, and at the same time, the user's attention direction vector is used as the candidate voice interaction direction. If the angle between the eye movement tracking vector and the head pose vector is greater than or equal to the angle threshold between the eye movement tracking vector and the head pose vector, it indicates that the deviation between the user's line of sight and the head direction is relatively large, and the user's multimodal data is re-acquired to repeatedly judge whether the modal pointing is concentrated.

[0065] The sound source localization module is used to capture multi-channel sound signals in space according to the microphone array and obtain the main beam direction in combination with the candidate voice interaction direction, and generate beam weights.

[0066] Through the microphone array Mk = {mk1, mk2,..., mk I},the time-domain sound signals on each microphone channel are collected in real time, denoted as Xh i (t), where i represents the i-th microphone and t is the time variable.

[0067] Calculate the time delay distribution for any pair of microphones i and j: where XH i (f) represents the Fourier transform of Xh i (t), represents the conjugate of Xh j (t), Xh j (t) represents the Fourier transform of Xh j (t), Fh -1 represents the inverse transform of the frequency-domain signal back to the time domain, τ ij represents the time delay, and Rs ij (τ ij ) represents the time delay distribution from the sound source to microphones i and j.

[0068] By constructing a search network θ for the target direction and (θ and represent the azimuth angle and the elevation angle respectively), the time delays of all microphone pairs are accumulated for the cost function to form a sound source focusing response map in the angular space: where represents the expected time delay in the target direction , represents the sound source estimation direction;

[0069] Combine each sound source estimation direction with the comprehensive attention vector to calculate the direction confidence: where ZX represents the confidence, corresponds to a set of sound source estimation directions

[0070] Select the sound source estimation direction with the highest direction confidence as the main beam pointing direction;

[0071] Generate beam weights: where represents the beam weights, XF represents the covariance matrix formed by the microphone array sampling signals, is the steering vector generated for the main beam pointing direction, represents the conjugate transpose of the steering vector;

[0072] It should be noted that the above formulas are all dimensionless and take their numerical values for calculation. Common methods for removing dimensions include Min-Max normalization, Z-Score standardization, etc., which will not be elaborated here;

[0073] The target voice locking module is used to obtain the user's voiceprint features in the main beam pointing direction and match them with the pre-registered user's voiceprint features to lock the target speaker;

[0074] Output the filtered signal in the main beam pointing direction through the beamformer, input the output filtered signal into the voiceprint feature extraction network to extract the user's voiceprint feature vector, and perform similarity matching between the user's voiceprint feature vector and the pre-registered user's voiceprint feature template, using the normalized cosine similarity as the judgment criterion: where sim represents the cosine similarity between the user's voiceprint feature vector and the pre-registered user's voiceprint feature vector, represents the user's voiceprint feature vector, represents the pre-registered user's voiceprint feature vector;

[0075] It should be noted that the above formulas are all dimensionless and take their numerical values for calculation. Common methods for removing dimensions include Min-Max normalization, Z-Score standardization, etc., which will not be elaborated here;

[0076] It should be noted that the voiceprint feature extraction network is a discriminative speech modeling structure based on a deep neural network, and its goal is to extract an embedding representation that is independent of the device, content, and emotion under the premise that the speaker remains unchanged. Common structures such as ECAPA-TDNN (Emphasized Channel Attention, Propagation and Aggregation TimeDelay Neural Network) will not be elaborated here;

[0077] Compare the cosine similarity with the preset similarity threshold. If the cosine similarity is greater than the similarity threshold, mark the user of the current sound source as the target speaker;

[0078] The reflected sound recognition and cancellation module is used to input the speech signal of the target speaker into the deep dereverberation modeling network to remove the spatial reverberation residue, and obtain the direct sound with the reflected sound features stripped;

[0079] Obtain the speech signal of the target speaker in the main beam direction and input it into the deep dereverberation modeling network. The structure of the deep dereverberation modeling network includes an encoder-decoder framework, which combines the attention mechanism and the gated residual path for reverberation modeling. Through the pre-trained deep dereverberation modeling network, the speech signal of the target speaker is output as two sets of components, including the direct sound component and the reflected sound component;

[0080] It should be noted that the direct sound component mainly carries effective semantic information, and the reflected sound component is the reverberation component generated by the reflection of structures such as walls, floors, and headset housings;

[0081] Encoder-decoder framework: Extract and reconstruct the multi-scale time-frequency features of the speech signal;

[0082] Attention mechanism: Enhance the attention weight for the reverberation area;

[0083] Gated residual path: Used to effectively capture and separate non-linear reverberation features to avoid the loss of the main semantic information;

[0084] By canceling the reflected sound component from the input signal, the target speech signal with the main component being the direct sound is obtained: Ym(t) = y(t) - sre(t), where Ym(t) represents the target speech signal with the main component being the direct sound, y(t) represents the speech signal of the target speaker in the main beam direction, and sre(t) represents the reflected sound component;

[0085] It should be noted that the above formulas are all dimensionless and take their numerical calculations. Common methods for removing dimensions include Min-Max normalization, Z-Score standardization, etc., which will not be elaborated here;

[0086] The speech enhancement and active noise reduction module is used to input the speech signal with features stripped into the combined structure of the dual-path recurrent neural network and the complex mask generation network to perform speech spectrum enhancement and active noise reduction output on the speech signal;

[0087] The speech signal with features stripped is framed and then subjected to short-time Fourier transform to obtain the complex spectrum representation: Sp(f,t) = STFT(Ym(t)), where STFT() represents the short-time Fourier transform, Sp(f,t) represents the spectrogram, and the spectrogram is input into the dual-path recurrent neural network to output the enhanced speech spectrogram: where DPRNN() represents the dual-path recurrent neural network, represents the enhanced speech spectrogram;

[0088] The dual-path recurrent neural network is a dual-path context modeling architecture based on component-based modeling and gated recurrent neural units, which can capture both local temporal dependencies within a frame and long-term context information between frames. The implementation steps of the dual-path recurrent neural network are as follows:

[0089] A1, the input spectrogram is divided into multiple time segments after preliminary transformation;

[0090] A2, the first layer of RNN is used within each time segment to capture the intra-frame structure;

[0091] A3, all time segment sequences are fed into the second layer of RNN to capture inter-frame dependencies;

[0092] A4, skip-connections are introduced to improve training stability;

[0093] A5, an enhanced speech spectrogram is output;

[0094] The dual-path recurrent neural network only enhances the magnitude spectrum. To further process the noisy phase in the spectrum, a complex mask network (CMN) is introduced to directly model in the complex domain and predict the complex domain mask of the enhanced speech spectrogram: M(f,t) = Mr(f,t) + j*Mi(f,t), where M(f,t) represents the complex domain mask, Mr(f,t) represents the real part mask, Mi(f,t) represents the imaginary part mask, and j represents the imaginary unit;

[0095] The complex domain mask is applied to the enhanced speech spectrogram: where represents the complex spectrum;

[0096] The complex spectrum is restored to the time-domain waveform using the inverse short-time Fourier transform (iSTFT): where Sout(t) is the time-domain speech signal after spectrum enhancement and noise reduction, and iSTFT() represents the inverse short-time Fourier transform;

[0097] The closed-loop correction module is used to obtain the drift information of target locking and the beam deviation information of beam pointing to construct a closed-loop correction mechanism to trigger adaptive reconstruction;

[0098] The drift information of the target locking includes the target locking drift coefficient, and the beam deviation information of the beam pointing includes the beam pointing deviation coefficient;

[0099] In the present invention, the target locking drift coefficient is used to measure the deviation degree of the voice feature matching between the voiceprint locking output and the target user, and its numerical value reflects the consistency between the current voiceprint recognition result and the pre-registered target template: when the target locking drift coefficient is large, it indicates that there is a significant deviation in the voiceprint features, which may be due to background noise interference, sound source direction change or voiceprint feature attenuation, resulting in a decrease in the locking accuracy of the system for the target speaker and a risk of locking drift; on the contrary, when the drift coefficient is small, it indicates that the current voiceprint matching is stable and reliable, and the system locking state is credible. Based on this coefficient, a closed-loop correction mechanism can dynamically sense the deviation trend of the voiceprint matching state and automatically trigger voiceprint re-matching and beam direction adaptive reconstruction, thereby effectively suppressing the problem of target loss caused by voice feature drift and realizing the continuous locking and self-correction of the system for the target speaker;

[0100] The acquisition logic of the target locking drift coefficient is as follows:

[0101] Obtain the cosine similarity sequence {sim(t n )} of the user's voiceprint feature matching from the target voice locking module, where sim(t n ) represents the cosine similarity at the nth time point; it is used to analyze the fluctuation of voiceprint matching;

[0102] Calculate the similarity mean value of the cosine similarity sequence: where represents the similarity mean value, n = {1, 2,..., N}, and N is a positive integer;

[0103] Calculate the similarity standard deviation of the cosine similarity sequence: where σsim represents the similarity standard deviation;

[0104] Calculate the target locking drift coefficient: where mbs represents the target locking drift coefficient;

[0105] It should be noted that the above formulas are all calculated by taking the numerical values after dimensionless. Common methods for removing dimensions include Min-Max normalization, Z-Score standardization, etc., which will not be elaborated here;

[0106] In the present invention, the beam pointing deviation coefficient is used to measure the angular deviation between the current main lobe direction of the beam and the actual direction of the target sound source. The larger the value, the more significant the angular difference between the main lobe of the beam and the sound source, which may lead to a decrease in the received gain of the sound source signal and inaccurate direction perception. On the contrary, if the deviation coefficient is small, it indicates that the beam has been well aligned with the sound source direction, having good gain concentration effect and spatial separation performance. Therefore, based on the beam pointing deviation coefficient, a closed-loop correction mechanism can be constructed to automatically trigger an adaptive reconstruction strategy when exceeding the dynamic threshold, quickly fine-tune or reconstruct the beam pointing, and continuously maintain efficient pointing and spatial focusing on the target sound source, thereby improving the robustness, clarity and response sensitivity of the whole machine system in a multi-speaker dynamic scenario.

[0107] The acquisition logic of the beam pointing deviation coefficient is as follows:

[0108] Obtain the historical sound source estimated direction sequence where represents the historical sound source estimated direction at the m-th time point;

[0109] Perform time-series weighting on the historical sound source estimated direction sequence: where is the sound source estimated direction after time-series weighting, and jq m represents the direction weight of the historical sound source estimated direction at the m-th time point, where td represents the current moment, and t m represents the m-th time point, m = {1, 2,..., M}, and M is a positive integer;

[0110] Calculate the direction divergence coefficient: where σθ represents the direction divergence coefficient;

[0111] The direction divergence coefficient is used to measure the volatility of the sound source direction sequence;

[0112] Calculate the beam pointing deviation angle: where δθ represents the current beam pointing deviation angle, represents the current main beam pointing direction;

[0113] Calculate the beam pointing deviation coefficient: where bsz represents the beam pointing deviation coefficient;

[0114] It should be noted that the above formulas are all calculated by taking the numerical value after removing the dimension. Commonly used methods for removing the dimension include Min-Max normalization, Z-Score standardization, etc., which will not be elaborated here;

[0115] Construct a closed-loop correction mechanism according to the target locking drift coefficient and the beam pointing deviation coefficient, and judge whether to trigger adaptive reconstruction, as follows:

[0116] A closed-loop correction model is constructed based on the target locking drift coefficient and the beam pointing deviation coefficient, and the closed-loop correction index zsa is output. The formula on which the model is based is as follows: zsa = b1 * Qzt + b2 * bsz, where b1 and b2 respectively represent the preset proportionality coefficients of the target locking drift coefficient and the beam pointing deviation coefficient, and both b1 and b2 are greater than 0;

[0117] It should be noted that the above formulas are all calculated by taking the numerical values after removing the dimensions. Common methods for removing dimensions include Min-Max normalization, Z-Score standardization, etc., which will not be elaborated here; b1 and b2 are set according to the actual situation. For example, the expert weighting method is adopted, that is, experts in related fields are invited to determine the preset proportionality coefficients of each index through professional opinion surveys and comprehensive evaluations. For example, b1 and b2 can be 0.5 and 0.5;

[0118] From the above calculation expressions, it can be seen that the larger the target locking drift coefficient and the beam pointing deviation coefficient, the larger the closed-loop correction index, indicating that there are large offsets in both the voiceprint locking and beam pointing dimensions of the current system. The perceived voice quality of the user may decrease significantly, the accuracy of target tracking may decrease, the focusing ability of the system on the current interaction subject may weaken, and the stronger the need for adaptive reconstruction of the system. It is necessary to trigger the reconstruction mechanism in a timely manner to restore the target focusing performance and signal enhancement accuracy of the voice interaction system, so as to improve the voice recognition accuracy and the stability of the user experience. On the contrary, the smaller the target locking drift coefficient and the beam pointing deviation coefficient, the smaller the closed-loop correction index, indicating that the current system is in a stable state in terms of voiceprint matching and beam alignment, the accuracy of sound source localization and user voice locking is high, the system has effectively focused on the direction of the target speaker, the voice enhancement and interference suppression effects are good, and the necessity of adaptive reconstruction is low. Resource redundancy consumption can be avoided, and the system can maintain the current configuration and operation to ensure the optimal balance of real-time performance and energy efficiency;

[0119] The closed-loop correction index is compared with the preset closed-loop correction index threshold. If the closed-loop correction index is greater than the closed-loop correction index threshold, adaptive reconstruction is triggered; if the closed-loop correction index is less than or equal to the closed-loop correction index threshold, adaptive reconstruction does not need to be triggered;

[0120] It should be noted that when adaptive reconstruction is triggered, the working process of the mode perception module -> sound source localization module -> target voice locking module -> reflected sound recognition and cancellation module -> voice enhancement and active noise reduction module is triggered again;

[0121] The present invention fuses the multi-modal sensor data in the user-worn device through a modal perception module to obtain the user's attention direction, guides the sound source localization module to quickly lock the main direction of voice interaction in a multi-source interference environment, and the target voice locking module further filters non-target speakers by using voiceprint feature recognition to ensure the individual consistency of the enhanced voice. The reflection sound recognition and cancellation module effectively strips the reflection reverberation generated by multi-path propagation through deep dereverberation modeling, avoiding semantic ambiguity caused by pseudo-source aliasing. The voice enhancement and active noise reduction module adopts a cooperative structure of DPRNN and complex mask network, which corrects the spectral phase while enhancing the amplitude spectrum, significantly improving the clarity and naturalness of the output voice. The closed-loop correction module monitors the enhanced focusing performance of the system in real time by introducing a target locking drift coefficient and a beam pointing deviation coefficient, and triggers adaptive reconstruction based on the closed-loop correction index, effectively preventing the system from falling into the closed-loop error mode of "false enhancement - pseudo-sound feedback - wrong tracking".

[0122] The present invention can not only improve the extraction accuracy and enhancement quality of the target voice in a dynamic and complex scenario, but also enhance the robustness of the system to interference and errors. It can continuously maintain high immersion, high responsiveness and high interaction efficiency in VR large-space human-computer interaction, realize high-precision extraction and dynamic robust enhancement of the target voice, and comprehensively improve the immersion, continuity and semantic accuracy of voice interaction.

[0123] The above formulas are all dimensionless and take their numerical calculations. The formula is obtained by collecting a large amount of data for software simulation to get a formula closest to the real situation. The preset parameters in the formula are set by those skilled in the art according to the actual situation.

[0124] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more collections of available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0125] It should be understood that in various embodiments of the present application, the sequence numbers of the above processes do not indicate the order of execution, and the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0126] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs and other media that can store program codes.

[0127] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.

Claims

1. VR large-space active noise reduction and directional voice enhancement system, characterized in that: It includes a modal perception module, a sound source localization module, a target voice locking module, a reflected sound recognition and cancellation module, a voice enhancement and active noise reduction module, and a closed-loop correction module; The modal perception module is used to receive multi-modal sensor data from the user's wearable device, obtain the user's attention direction vector based on the multi-modal data, and use the user's attention direction vector as the candidate voice interaction direction; The sound source localization module is used to capture multi-channel sound signals in space according to the microphone array and obtain the main beam direction in combination with the candidate voice interaction direction, and generate a dynamic beamforming weight matrix; The target voice locking module is used to obtain the user's voiceprint feature on the main beam direction and match it with the pre-registered user's voiceprint feature to lock the target speaker; The reflected sound recognition and cancellation module is used to input the voice signal of the target speaker into the deep dereverberation modeling network to remove the spatial reverberation residue and obtain the direct sound with the reflected sound feature stripped; The voice enhancement and active noise reduction module is used to input the voice signal after feature stripping into the combined structure of the dual-path recurrent neural network and the complex mask generation network to perform voice spectrum enhancement and active noise reduction output on the voice signal; The closed-loop correction module is used to obtain the drift information of the target lock and the beam deviation information of the beam direction to construct a closed-loop correction mechanism and trigger adaptive reconstruction.

2. The VR large-space active noise reduction and directional voice enhancement system according to claim 1, wherein: The multi-modal data described above includes: Eye movement tracking vector: denoted as Head pose vector: denoted as Gesture focus vector: denoted as Construct a comprehensive attention vector based on the three modal vectors: Where represents the comprehensive attention vector, a1, a2, and a3 respectively represent the preset proportionality coefficients of the eye movement tracking vector, the head pose vector, and the gesture focus vector, and a1, a2, and a3 are all greater than 0; Determine whether the modal direction is concentrated: Calculate the angle between the eye-tracking vector and the head pose vector. The calculation formula is as follows: where θeh represents the angle between the eye-tracking vector and the head pose vector, represents the dot product of the eye-tracking vector and the head pose vector; Compare the included angle between the eye movement tracking vector and the head pose vector with the preset included angle threshold between the eye movement tracking vector and the head pose vector to obtain the user's attention direction vector, specifically as follows: If the included angle between the eye movement tracking vector and the head pose vector is less than the included angle threshold between the eye movement tracking vector and the head pose vector, then use the current comprehensive attention vector as the user's attention direction vector, and at the same time use the user's attention direction vector as the candidate voice interaction direction.

3. The VR large-space active noise reduction and directional voice enhancement system according to claim 2, characterized in that: Through the microphone array Mk = {mk1, mk2,..., mk I}, the time-domain acoustic signals on each microphone channel are collected in real time, denoted as Xh i (t), where i represents the i-th microphone and t is the time variable; Calculate the time delay distribution for any pair of microphones i and j: where XH i (f) represents the Fourier transform of Xh i (t), represents the conjugate of Xh j (t), Xh j (t) represents the Fourier transform of Xh j (t), Fh -1 represents the inverse transform of the frequency-domain signal, back to the time domain, τ ij represents the time delay, Rs ij (τ ij ) represents the time delay distribution from the sound source to microphones i and j; By constructing a search network θ in the target direction and θ and respectively represent the azimuth angle and the elevation angle, and accumulate the cost function of the time delay of all microphone pairs to form a sound source focusing response map in the angular space: where represents the expected time delay in the target direction ), and represents the estimated direction of the sound source; Combine each estimated sound source direction with the comprehensive attention vector to calculate the direction confidence: where ZX represents the confidence, corresponding to a set of estimated sound source directions Select the sound source estimation direction with the highest direction confidence as the main beam direction; Generate beam weights: Where represents the beam weights, XF represents the covariance matrix formed by the sampled signals of the microphone array, is the steering vector generated for the main beam direction, represents the conjugate transpose of the steering vector.

4. The VR large-space active noise reduction and directional voice enhancement system according to claim 3, characterized in that: Output the filtered signal in the direction of the main beam through the beamformer, input the output filtered signal into the voiceprint feature extraction network to extract the user's voiceprint feature vector, and perform similarity matching between the user's voiceprint feature vector and the pre-registered user voiceprint feature template. Use the normalized cosine similarity as the judgment criterion: where sim represents the cosine similarity between the user's voiceprint feature vector and the pre-registered user voiceprint feature vector, represents the user's voiceprint feature vector, represents the pre-registered user voiceprint feature vector Compare the cosine similarity with the preset similarity threshold. If the cosine similarity is greater than the similarity threshold, then mark the current sound source user as the target speaker.

5. The VR large-space active noise reduction and directional voice enhancement system according to claim 4, characterized in that: The drift information of the target lock described above includes the target lock drift coefficient, and the beam deviation information of the beam direction includes the beam direction deviation coefficient.

6. The VR large space active noise reduction and directional voice enhancement system according to claim 5, characterized in that: The acquisition logic of the target lock drift coefficient is as follows: Obtain the cosine similarity sequence {sim(t n )} that matches the user's voiceprint feature from the target voice locking module, where sim(t n ) represents the cosine similarity at the nth time point; Calculate the similarity mean of the cosine similarity sequence: where represents the similarity mean, n = {1, 2,..., N}, and N is a positive integer; Calculate the similarity standard deviation of the cosine similarity sequence: where σsim represents the similarity standard deviation; Calculate the target lock drift coefficient: where mbs represents the target lock drift coefficient.

7. The VR large-space active noise reduction and directional voice enhancement system according to claim 6, characterized in that: The acquisition logic of the beam direction deviation coefficient is as follows: Obtain the historical sound source estimation direction sequence where represents the historical sound source estimation direction at the m-th time point; Perform temporal weighting on the historical sound source estimation direction sequence: where is the sound source estimation direction after temporal weighting, and jq m represents the direction weight of the historical sound source estimation direction at the m-th time point, where td represents the current moment, and t m represents the m-th time point, m = {1, 2,..., M}, and M is a positive integer; Calculate the directional divergence coefficient: where σθ represents the directional divergence coefficient; Calculate the beam pointing deviation angle: where δθ represents the current beam pointing deviation angle, represents the current main beam pointing direction; Calculate the beam pointing deviation coefficient: where bsz represents the beam pointing deviation coefficient.

8. The VR large-space active noise reduction and directional voice enhancement system according to claim 5, characterized in that: Construct a closed-loop correction mechanism according to the target lock drift coefficient and the beam direction deviation coefficient, and judge whether to trigger adaptive reconstruction, specifically as follows: Construct a closed-loop correction model according to the target lock drift coefficient and the beam direction deviation coefficient, and output the closed-loop correction index zsa. The formula on which the model is based is as follows: zsa = b1 * Qzt + b2 * bsz, where b1 and b2 respectively represent the preset proportionality coefficients of the target lock drift coefficient and the beam direction deviation coefficient, and both b1 and b2 are greater than 0; Compare the closed-loop correction index with the preset closed-loop correction index threshold. If the closed-loop correction index is greater than the closed-loop correction index threshold, then trigger adaptive reconstruction; if the closed-loop correction index is less than or equal to the closed-loop correction index threshold, then there is no need to trigger adaptive reconstruction.

Images