Sound source localization method and device based on complex value residual shrinkage recurrent neural network

Abstract

The invention provides a sound source localization method and device based on a complex value residual shrinkage recurrent neural network, and the method comprises the steps: constructing a complex value residual shrinkage recurrent neural network model; then, audio signals of a plurality of audio channels are obtained. Then, obtaining a complex value matrix network according to the audio signal; and then, inputting the complex-valued matrix network into a complex-valued residual shrinkage module to obtain a soft threshold corresponding to the audio channel according to the complex-valued matrix, and performing residual shrinkage on the complex-valued matrix according to the soft threshold to obtain a target complex-valued network. And then, inputting the target complex-valued network into a bidirectional complex-valued gating circulation module to obtain a target feature vector. And finally, inputting the target feature vector into an angle prediction module to obtain an azimuth angle and a pitch angle, and positioning a sound source according to the azimuth angle and the pitch angle. Thus, the sound source localization method dynamically suppresses low-energy interference characteristics while effectively extracting high signal-to-noise ratio characteristics in the complex-valued matrix network, thereby significantly improving the estimation precision of sound source localization under high noise and reverberation environment conditions.

Description

Technical Field

[0001] This application relates to the fields of signal processing and deep learning, and more specifically, to a method and apparatus for sound source localization based on complex-valued residual shrinking recurrent neural networks. Background Technology

[0002] Direction of arrival (DOA) estimation is a core task in acoustic signal processing. Most neural network-based DOA estimation methods use signal processing results such as the short-time Fourier transform matrix or covariance matrix as input. These inputs are essentially complex-valued representations, containing both the amplitude and crucial phase information of the acoustic signal. However, traditional neural networks operate in the real-valued domain and cannot directly process such complex data. This fundamental limitation necessitates either decomposing the complex-valued input into two separate real-valued channels (real and imaginary parts, or amplitude and phase) or discarding a component of the complex-valued input, leading to potential information loss.

[0003] Therefore, complex-valued neural networks (CVNNs) are introduced into the field of sound source direction-of-arrival (DOA) estimation. The weights, activation functions, and complex-valued operations of CVNNs are all performed in the complex-valued domain, exhibiting inherent compatibility with complex-valued features. Its complex-valued convolution and other operations mathematically maintain the additive nature of phase processing (compared to the equivalent multiplication operations in the real-valued domain), providing stronger mathematical interpretability for phase-sensitive sound source localization tasks. Furthermore, CVNNs typically exhibit faster convergence speeds and lower overfitting tendencies. However, in complex indoor environments with strong noise and severe reverberation, the estimation accuracy of existing CVNN-based DOA estimation methods drops significantly, making it difficult to achieve accurate sound source localization. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a sound source localization method and apparatus based on a complex-valued residual shrinking recurrent neural network. The aforementioned sound source localization method effectively extracts high signal-to-noise ratio features from the complex-valued matrix network while dynamically suppressing low-energy interference features, thereby significantly improving the estimation accuracy of sound source localization under high-noise and reverberant environmental conditions.

[0005] According to a first aspect of this application, a sound source localization method based on a complex-valued residual shrinking recurrent neural network is provided, applied to a computer device in a sound source localization system. The sound source localization system further includes a plurality of uniformly distributed sound source acquisition devices connected to the computer device. The method includes: A complex-valued residual shrinkage recurrent neural network model is constructed, wherein the complex-valued residual shrinkage recurrent neural network model includes a complex-valued residual shrinkage module, a bidirectional complex-valued gated recurrent module, and an angle prediction module connected in sequence; Acquire audio signals from multiple audio channels detected by the sound source acquisition device; A complex-valued matrix network is obtained based on the audio signal, wherein the complex-valued matrix network is formed by a complex-valued matrix obtained based on the audio signal of each audio channel, and the complex-valued matrix includes a plurality of complex values ​​for characterizing the amplitude and phase of the audio signal; The complex-valued matrix network is input into the complex-valued residual shrinking module to obtain the soft threshold corresponding to the audio channel based on the complex-valued matrix network, and the target complex-valued network is obtained by residual shrinking the complex-valued matrix network based on the soft threshold. The target complex value network is input into the bidirectional complex value gated recurrent module to extract global features from the target complex value network to obtain the target feature vector; The target feature vector is input into the angle prediction module to obtain the azimuth and pitch angles, and the sound source is located based on the azimuth and pitch angles.

[0006] In one possible implementation of the first aspect, the step of inputting the complex-valued matrix network into the complex-valued residual shrinking module to obtain the soft threshold corresponding to the audio channel based on the complex-valued matrix network, and performing residual shrinking on the complex-valued matrix network based on the soft threshold to obtain the target complex-valued network includes: The threshold scaling parameters corresponding to the audio channel are determined based on the complex-valued matrix network. The soft threshold corresponding to the audio channel is obtained based on all complex values ​​in the complex value matrix and the threshold scaling parameter. The magnitude corresponding to the complex value is determined based on the complex value in the complex value matrix; The target complex value is obtained by residual shrinking the complex value based on the soft threshold and the amplitude. The target complex value matrix is ​​obtained based on the target complex value, and the target complex value network is obtained based on the target complex value matrix and the complex value matrix.

[0007] In one possible implementation of the first aspect, the step of obtaining the soft threshold corresponding to the audio channel based on all complex values ​​in the complex value matrix and the threshold scaling parameter includes: The formula for expressing the soft threshold is:

[0008] in, Indicates the soft threshold. Represents complex values. Represents the time dimension in a complex-valued matrix. This represents the frequency dimension in a complex-valued matrix. This represents the audio channels in a complex-valued matrix network. This represents the sum of all complex values ​​in the complex-valued matrix. This represents the threshold scaling parameter.

[0009] In one possible implementation of the first aspect, the step of residual shrinking the complex value based on the soft threshold and the amplitude to obtain the target complex value includes: Compare the soft threshold and the magnitude of the amplitude; When the amplitude is greater than the soft threshold, the amplitude of the complex value is reduced while the phase of the complex value remains unchanged to obtain the target complex value; When the amplitude is less than or equal to the soft threshold, the complex value is assigned the value of 0 to obtain the target complex value.

[0010] In one possible implementation of the first aspect, the step of obtaining the complex-valued matrix network based on the audio signal includes: The audio signal is divided into frames to obtain a framed audio signal, wherein the audio signals of two adjacent frames overlap. The speech activity of the segmented audio signal corresponding to the audio channel at the same time is detected, the segmented audio signal is divided into speech frame audio signal and silence frame audio signal, and the silence frame audio signal is marked as a silence frame. Calculate the root mean square energy and zero-crossing rate of the audio signal of the speech frame; When the root mean square energy of the audio signal of the voice frame is less than or equal to a preset root mean square energy threshold or the zero crossing rate of the audio signal of the voice frame is less than or equal to a preset zero crossing rate threshold, the audio signal of the voice frame is marked as a silent frame. When any of the segmented audio signals in the audio channel at the same time is marked as a silent frame, the segmented audio signal is removed, and the segmented audio signals of the remaining time are spliced ​​together to form a spliced ​​audio signal. The complex-valued matrix is ​​obtained by performing a short-time Fourier transform on the spliced ​​audio signal of each audio channel. The complex-valued matrix network is obtained from the complex-valued matrix.

[0011] In one possible implementation of the first aspect, the step of inputting the target complex value network into the bidirectional complex value gated recurrent module to extract global features from the target complex value network to obtain a target feature vector includes: The target complex value network is integrated into global complex value features according to the time dimension; The target feature vector is obtained by performing average pooling on the global complex features.

[0012] In one possible implementation of the first aspect, the step of inputting the target feature vector into the angle prediction module to obtain the azimuth and elevation angles, and locating the sound source based on the azimuth and elevation angles, includes: The target feature vectors are respectively input into parallel fully connected layer sub-modules to obtain a first target complex value for characterizing the azimuth angle and a second target complex value for characterizing the pitch angle; The azimuth angle is obtained based on the complex value of the first target, and the pitch angle is obtained based on the complex value of the second target; The sound source is located based on the azimuth and elevation angles.

[0013] In one possible implementation of the first aspect, after the steps of inputting the target feature vector into the angle prediction module to obtain the azimuth and elevation angles, and locating the sound source based on the azimuth and elevation angles, the method further includes: The azimuth angle estimation error and the pitch angle estimation error are obtained based on the azimuth angle, the pitch angle, and the pre-predicted azimuth angle and the pitch angle; The azimuth variance is obtained based on the azimuth estimation error, and the pitch variance is obtained based on the pitch estimation error; The azimuth sub-loss function is obtained based on the azimuth angle, and the pitch sub-loss function is obtained based on the pitch angle; A target loss function is constructed based on the azimuth variance, the pitch variance, the azimuth sub-loss function, and the pitch sub-loss function; The complex-valued residual shrinkage recurrent neural network model is adjusted according to the target loss function.

[0014] In one possible implementation of the first aspect, the step of constructing a target loss function based on the azimuth variance, the pitch variance, the azimuth sub-loss function, and the pitch sub-loss function includes: The formula for the target loss function is as follows:

[0015] in, This represents the azimuth variance, which is the azimuth estimation error. variance This represents the pitch angle variance, which is the pitch angle estimation error. variance This represents the azimuth sub-loss function. The formulas for the pitch and azimuth loss functions are as follows: The formula for the pitch loss function is: .

[0016] According to a second aspect of this application, a sound source localization device based on a complex-valued residual shrinking recurrent neural network is provided, applied to a computer device in a sound source localization system. The sound source localization system further includes a plurality of uniformly distributed sound source acquisition devices, the sound source acquisition devices being connected to the computer device. The device comprises: A construction unit is used to construct a complex-valued residual shrinking recurrent neural network model, wherein the complex-valued residual shrinking recurrent neural network model includes a complex-valued residual shrinking module, a bidirectional complex-valued gated recurrent module, and an angle prediction module connected in sequence. The acquisition unit is used to acquire audio signals from multiple audio channels detected by the sound source acquisition device; An input unit is configured to obtain a complex-valued matrix network based on the audio signal, wherein the complex-valued matrix network is formed by a complex-valued matrix obtained based on the audio signal of each audio channel, and the complex-valued matrix includes a plurality of complex values ​​for characterizing the amplitude and phase of the audio signal; The first determining unit is used to input the complex-valued matrix network into the complex-valued residual shrinking module to obtain the soft threshold corresponding to the audio channel based on the complex-valued matrix network, and to perform residual shrinking on the complex-valued matrix network based on the soft threshold to obtain the target complex-valued network; The second determining unit is used to input the target complex value network into the bidirectional complex value gated loop module to extract global features in the target complex value network to obtain a target feature vector. The output unit is used to input the target feature vector into the angle prediction module to obtain the azimuth and pitch angles, and to locate the sound source based on the azimuth and pitch angles.

[0017] Based on any of the above aspects, this application provides a sound source localization method and apparatus based on a complex-valued residual shrinking recurrent neural network. First, a complex-valued residual shrinking recurrent neural network model is constructed. Next, audio signals from multiple audio channels are acquired. Then, a complex-valued matrix network is obtained based on the audio signals. Next, the complex-valued matrix network is input into a complex-valued residual shrinking module to obtain soft thresholds corresponding to the audio channels based on the complex-valued matrix, and residual shrinking is performed on the complex-valued matrix based on the soft thresholds to obtain a target complex-valued network. Then, the target complex-valued network is input into a bidirectional complex-valued gated recurrent module to obtain a target feature vector. Finally, the target feature vector is input into an angle prediction module to obtain the azimuth and elevation angles, and the sound source is located based on these two angles. Thus, the above sound source localization method effectively extracts high signal-to-noise ratio features from the complex-valued matrix network while dynamically suppressing low-energy interference features, thereby significantly improving the estimation accuracy of sound source localization under high-noise and reverberation environments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This embodiment provides a flowchart of the steps for a sound source localization method based on a complex-valued residual shrinking recurrent neural network. Figure 2 This is a flowchart illustrating the sound source localization method provided in this embodiment. Figure 3 for Figure 2 Flowchart of some sub-modules in the complex value residual shrinkage module; Figure 4 for Figure 1 A flowchart illustrating the sub-steps corresponding to step S14; Figure 5 for Figure 1 A flowchart illustrating the steps following step S15; Figure 6 Box plot of the mean absolute error of the azimuth angle provided in the experimental test of this embodiment; Figure 7 Box plot of the mean absolute error of pitch angle in the experimental test provided in this embodiment; Figure 8 The graph shows the test results of the mean absolute error in the ablation experiment provided in this embodiment; Figure 9 The figure shows the test results of the mean absolute error under an unknown signal-to-noise ratio in the generalization performance test provided in this embodiment; Figure 10 The graph shows the test results of the mean absolute error under unknown reverberation in the generalization performance test provided in this embodiment; Figure 11 A graph showing the test results of the mean absolute error in the comparative test provided in this embodiment; Figure 12 This is a schematic diagram of the functional modules of a sound source localization device based on a complex-valued residual shrinking recurrent neural network provided in this embodiment.

[0020] Icons: Complex-valued residual shrinkage module-10; Bidirectional complex-valued gated recurrent module-11; Angle prediction module-12; Convolutional layer submodule-100; First complex-valued residual shrinkage construction submodule-101; Second complex-valued residual shrinkage construction submodule-102; Sound source localization device based on complex-valued residual shrinkage recurrent neural network-20; Construction unit-200; Acquisition unit-210; Output unit-220; First determination unit-230; Second determination unit-240; Output unit-250. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0024] In the description of this application, it should be noted that the terms "upper," "lower," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] It should be noted that, where there is no conflict, different features in the embodiments of this application can be combined with each other.

[0026] The inventors discovered through research that direction of arrival (DOA) estimation is a core task in acoustic signal processing. Most neural network-based DOA estimation methods use signal processing results such as short-time Fourier transform matrices or covariance matrices as input. These inputs are essentially complex-valued representations, containing both the amplitude and crucial phase information of the acoustic signal. However, traditional neural networks operate in the real-valued domain and cannot directly process such complex-valued data. This fundamental limitation necessitates decomposing the complex-valued input into two separate real-valued channels—the real and imaginary parts (or amplitude and phase)—or discarding a component of the complex-valued input, leading to potential information loss.

[0027] Therefore, complex-valued neural networks (CVNNs) have been introduced into the field of sound source direction-of-arrival estimation. The weights, activation functions, and complex-valued operations of CVNNs are all performed in the complex-valued domain, exhibiting inherent compatibility with complex-valued features. Their complex-valued convolution and other operations mathematically preserve the additive nature of phase processing (compared to the equivalent multiplication operations in the real-valued domain), providing stronger mathematical interpretability for phase-sensitive sound source localization tasks. Furthermore, CVNNs typically exhibit faster convergence speeds and lower overfitting tendencies.

[0028] Currently, research on DOA based on CVNNs has made progress. For example, Complex Valued Convolutional Recurrent Neural Networks (CV-CRNN) directly uses the complex form of the STFT as input, while utilizing convolutional layers to extract spatial features and recurrent layers to model temporal context. However, in complex indoor environments with strong noise and severe reverberation, the convolutional kernels in CV-CRNNs, which act as local feature extractors, struggle to detect effective DOA features under extreme interference, leading to a significant decrease in estimation accuracy and making it difficult to achieve accurate sound source localization.

[0029] In order to solve the aforementioned technical problems, the inventors have innovatively designed the following technical solutions, and the specific implementation scheme of this application will be described in detail below with reference to the accompanying drawings.

[0030] Please see Figure 1 and Figure 2 , Figure 1 This embodiment provides a flowchart of the steps for a sound source localization method based on a complex-valued residual shrinking recurrent neural network. Figure 2This is a flowchart illustrating the sound source localization method provided in this embodiment. The sound source localization method is applied to a computer device in a sound source localization system. The sound source localization system also includes multiple evenly distributed sound source acquisition devices connected to the computer device. In this embodiment, the sound source localization system further includes an enclosed space, within which the sound source acquisition devices are evenly distributed. The sound source localization method is used to locate sound sources within the enclosed space. For example, when the enclosed space is a rectangular room, six sound source acquisition devices (e.g., microphones) can be distributed in an octahedral pattern within the enclosed space, with each device located at one of the six vertices of the octahedron, and each device equidistant from the center of the octahedron.

[0031] In this embodiment, the sound source acquisition device can be connected to the computer device via wireless communication (e.g., Bluetooth, WiFi, etc.). For example, the computer device acquires the audio signals of multiple audio channels detected by the sound source acquisition device, analyzes and processes them to obtain the pitch angle and azimuth angle used to locate the sound source, wherein one sound source acquisition device corresponds to one audio channel.

[0032] The following is a detailed introduction to the sound source localization method based on complex-valued residual shrinking recurrent neural network executed by computer equipment.

[0033] Step S11: Construct a complex-valued residual shrinkage recurrent neural network model, wherein the complex-valued residual shrinkage recurrent neural network model includes a complex-valued residual shrinkage module 10, a bidirectional complex-valued gated recurrent module 11, and an angle prediction module 12 connected in sequence.

[0034] In this embodiment, the complex-valued residual shrinking module 10 serves as a feature preprocessing and enhancement layer, primarily extracting local features of the audio signal. Specifically, the "shrinking" mechanism suppresses interfering noise or reverberation components in the complex-valued matrix network while retaining high-energy features useful for sound source localization. The bidirectional complex-valued gated loop module 11 serves as a temporal feature extraction layer, mainly obtaining the target feature vector by capturing the temporal dependencies of the target complex-valued network. The angle prediction module 12 serves as the task output layer, mapping the target feature vector processed by the complex-valued residual shrinking module 10 and the bidirectional complex-valued gated loop module 11 to two complex values ​​representing the azimuth and pitch angles through parallel fully connected sublayers.

[0035] Step S12: Acquire audio signals from multiple audio channels detected by the sound source acquisition device.

[0036] In this step, audio signals from multiple audio channels are acquired within a preset time period detected by the sound source acquisition devices, with one sound source acquisition device corresponding to one audio channel. For example, if the number of sound source acquisition devices is 6, then audio signals from 6 audio channels are acquired.

[0037] It is worth noting that it can acquire audio signals from multiple audio channels within a duration of 1 second, 2 seconds, or 3 seconds detected by the sound source acquisition device. The specific preset duration is not limited here and needs to be selected according to the actual situation.

[0038] Step S13: Obtain a complex value matrix network based on the audio signal. The complex value matrix network is formed by a complex value matrix obtained based on the audio signal of each audio channel. The complex value matrix includes multiple complex values ​​used to characterize the amplitude and phase of the audio signal.

[0039] In this embodiment, since the Complex Value Residual Shrinking Recurrent Neural Network (CV-RSR) model operates in the complex value domain, preprocessing of the audio signal is required to obtain a complex value matrix network that can be input into the CV-RSR. Specifically, firstly, preprocessing is performed on the audio signal of each audio channel to obtain a complex value matrix formed by the frequency and time dimensions. The complex value corresponding to any frequency dimension and any time dimension is used to characterize the amplitude and phase of the audio signal. When the frequency dimension is f and the time dimension is t, if the expression of the corresponding complex value is "a+b*i", the modulus of the complex value is the amplitude of the audio signal at the frequency dimension f and the time dimension t, and the argument of the complex value is the phase of the audio signal at the frequency dimension f and the time dimension t.

[0040] Next, the complex-valued matrices of multiple audio channels form a complex-valued matrix network. For example, if there are 6 audio channels, the complex-valued matrix network is formed by the complex-valued matrices corresponding to the 6 audio channels. Furthermore, the preprocessing operations for the audio signal include frame segmentation, removal of silent frames, and short-time Fourier transform.

[0041] Step S14: Input the complex-valued matrix network into the complex-valued residual shrinkage module 10 to obtain the soft threshold corresponding to the audio channel based on the complex-valued matrix network, and perform residual shrinkage on the complex-valued matrix network based on the soft threshold to obtain the target complex-valued network.

[0042] In this step, the complex-valued residual shrinkage module 10 includes a convolutional layer submodule 100, a first complex-valued residual shrinkage construction submodule 101, and a second complex-valued residual shrinkage construction submodule 102 connected in sequence. First, the complex-valued matrix network is input into the convolutional layer submodule 100, which uses a 6*6 large-kernel convolutional layer to capture large-scale features through an expanded receptive domain, enhancing the coverage of the frequency and time dimensions in the complex-valued matrix network. Next, the complex-valued matrix network processed by the convolutional layer submodule 100 is input into the first complex-valued residual shrinkage construction submodule 101 for initial residual shrinkage, and then input into the second complex-valued residual shrinkage construction submodule 102 for further residual shrinkage to obtain the target complex-valued matrix. The target complex-valued matrix for each audio channel is then concatenated with the complex-valued matrix to form the target complex-valued network.

[0043] In this embodiment, the complex-valued residual shrinking construction submodule adaptively determines the soft threshold and performs residual shrinking on the corresponding complex-valued matrix based on the soft threshold of each audio channel, selectively attenuating the amplitude component while preserving the phase features that may encode Time Difference of Occurrence (TDOA) information to the greatest extent. These phase features are closely related to sound source localization and are processed in the bidirectional complex-valued gated loop module 11. Thus, while effectively extracting high signal-to-noise ratio (SNR) features, noise or reverberation is dynamically suppressed, enhancing robustness in environments with strong noise and reverberation, thereby significantly improving the estimation accuracy of sound source localization. Here, a strong noise and reverberation environment is defined as a signal-to-noise ratio (SNR) less than or equal to 0 dB and a reverberation time (T... 60 An environment condition with a time of 400ms or greater.

[0044] Step S15: Input the target complex value network into the bidirectional complex value gated loop module 11 to extract the global features in the target complex value network to obtain the target feature vector.

[0045] In this embodiment, the bidirectional complex-valued gated loop module 11 can capture the time dependence within the target complex-valued network, thereby correcting the phase distortion caused by reverberation and preventing gradient effects and explosions, forming a target feature vector independent of the time and frequency dimensions.

[0046] Step S16: Input the target feature vector into the angle prediction module 12 to obtain the azimuth and elevation angles, and locate the sound source based on the azimuth and elevation angles.

[0047] In this step, the angle prediction module 12 includes two parallel fully connected layer submodules for mapping the target feature vector to two complex numbers, which are used for the localization of the azimuth and pitch angles, respectively.

[0048] Further, please refer to Figure 3 and Figure 4 , Figure 3 for Figure 2 Flowchart of some sub-modules in the complex value residual shrinkage module. Figure 4 for Figure 1 The flowchart of the sub-step corresponding to step S14 is shown below. Step S14 can be implemented in the following way.

[0049] Sub-step S141: Determine the threshold scaling parameters corresponding to the audio channel based on the complex-valued matrix network.

[0050] In this step, please refer to Figure 3 The complex-valued matrix network, processed by the convolutional sub-module 100 of the complex-valued residual shrinking module 10, is first processed through a first convolutional sub-layer, then batch normalization (BN) and ReLU activation functions are applied, and then processed through a second convolutional sub-layer, after which BN and ReLU are applied again to obtain the processed complex-valued matrix network. The first and second convolutional sub-layers can be 2x2 convolutional layers. The purpose of BN is to normalize the complex-valued matrix network processed by the first or second convolutional sub-layer to accelerate model training convergence. The purpose of ReLU is to introduce nonlinearity into the complex-valued matrix network after BN processing, enabling the complex-valued residual shrinking recurrent neural network to learn and represent complex nonlinear mapping relationships during processing.

[0051] The threshold scaling factor is then determined using the processed complex-valued matrix network. Specifically, firstly, global average pooling is applied to compress the network spatially, resulting in a channel feature vector for each audio channel. Next, this channel feature vector is input into the first fully connected sublayer (FC) to compress the number of audio channels, reducing computational complexity. Then, batch normalization (BN) and ReLU activation functions are applied to the channel feature vector processed by the first fully connected sublayer (FC) to increase non-linearity and stabilize training. Finally, this is input into the second fully connected sublayer (FC) to restore the original number of audio channels, matching the number of audio channels in the processed complex-valued matrix network. Finally, the output value of the second fully connected sublayer is mapped to [0,1] to obtain the threshold scaling parameter for each audio channel.

[0052] Sub-step S142: Obtain the soft threshold corresponding to the audio channel based on all complex values ​​in the complex value matrix and the threshold scaling parameter.

[0053] In this embodiment, the soft threshold for each audio channel is determined based on the complex value matrix and threshold scaling parameters corresponding to each audio channel. If the soft threshold is set too high, effective features may be incorrectly suppressed; if it is set too low, interfering features may not be effectively suppressed. This embodiment enhances robustness in high-noise and reverberant environments by adaptively determining the soft threshold.

[0054] Sub-step S143: Determine the magnitude corresponding to the complex value based on the complex value in the complex value matrix.

[0055] Sub-step S144: Perform residual shrinkage on the complex value based on the soft threshold and amplitude to obtain the target complex value.

[0056] In this step, the target complex value is obtained by residual shrinking based on the comparison of the soft threshold and the amplitude. This can selectively attenuate the amplitude component while preserving the phase component to the greatest extent, which is beneficial to enhancing the estimation accuracy of sound source localization.

[0057] Sub-step S145: Obtain the target complex value matrix based on the target complex value, and obtain the target complex value network based on the target complex value matrix and the complex value matrix.

[0058] In this step, the complex-valued matrix network processed by the convolutional layer submodule 100 is input into the downsampling sublayer to reduce the spatial resolution of the complex-valued matrix network. Then, the complex-valued matrix of the complex-valued matrix network processed by the downsampling sublayer in each audio channel is connected to the target complex-valued matrix by an identity shortcut to form the target complex-valued network. The identity shortcut (i.e., jump connection) can alleviate the difficulty of complex value optimization.

[0059] Furthermore, sub-step S142 can be implemented in the following way.

[0060] The formula for expressing the soft threshold is:

[0061] in, Indicates the soft threshold. Represents complex values. Represents the time dimension in a complex-valued matrix. This represents the frequency dimension in a complex-valued matrix. This represents the audio channels in a complex-valued matrix network. This represents the sum of all complex values ​​in the complex-valued matrix. This represents the threshold scaling parameter.

[0062] In this embodiment, the average complex value of all complex values ​​in the complex value matrix is ​​taken, and then the product of the average complex value and the threshold scaling parameter is moduloed to obtain the soft threshold.

[0063] Furthermore, sub-step S144 can be implemented in the following way.

[0064] First, compare the size of the soft threshold and the amplitude.

[0065] Next, when the amplitude is greater than the soft threshold, the amplitude of the complex value is reduced while keeping the phase of the complex value unchanged to obtain the target complex value.

[0066] Finally, when the amplitude is less than or equal to the soft threshold, the complex value is assigned the value of 0 to obtain the target complex value.

[0067] In this embodiment, the judgment formula corresponding to sub-step S144 is:

[0068] in, Represents the magnitude of a complex value. This represents the complex value after residual shrinkage. This indicates a soft threshold.

[0069] In this embodiment, the amplitude of the complex value can be reduced to the original amplitude minus a soft threshold to achieve noise reduction of the amplitude information while maintaining the original phase information, since the phase information contains TDOA information closely related to sound source localization. This dynamically suppresses low-energy interference (noise or reverberation) while preserving high-noise-ratio regions, thereby improving the estimation accuracy of sound source localization under high-noise or reverberant environments.

[0070] Furthermore, step S13 can be implemented in the following way.

[0071] First, the audio signal is divided into frames to obtain the framed audio signal, where the audio signals of two adjacent frames overlap.

[0072] In this step, the audio signal of each audio channel can be overlapped and framed according to a frame length of 1024 snapshots, with a frame shift of 256 snapshots. Furthermore, during the framing operation, a Hanning window is added to each frame of audio signal to reduce spectral leakage of the speech frame audio signal.

[0073] Next, the speech activity of the segmented audio signal corresponding to the audio channel at the same time is detected, and the segmented audio signal is divided into speech frame audio signal and silence frame audio signal, and the silence frame audio signal is marked as a silence frame.

[0074] In this step, speech activity detection (VAD) is used to detect the speech activity of the framed audio signal.

[0075] Next, the root mean square energy and zero-crossing rate of the audio signal of the speech frame are calculated.

[0076] In this step, the formula for the root mean square energy is: The formula for the zero-crossing rate is: .in, This represents the root-mean-square energy. This indicates the number of sampling points in a single audio frame. This represents the audio signal of the speech frame at the nth sampling point. This represents the audio signal of the speech frame at the (n-1)th sampling point. Indicates the zero-crossing rate.

[0077] In this embodiment, the root mean square energy is used to reflect the average energy or amplitude intensity of a voice frame audio signal, and the zero-crossing rate is used to reflect the frequency characteristics of a voice frame audio signal.

[0078] Then, when the root mean square energy of the audio signal of the speech frame is less than or equal to a preset root mean square energy threshold or the zero-crossing rate of the audio signal of the speech frame is less than or equal to a preset zero-crossing rate threshold, the audio signal of the speech frame is marked as a silent frame.

[0079] In this step, a voice frame is marked as a valid voice frame only if both the root mean square energy and the zero-crossing rate of the audio signal are greater than the preset root mean square energy threshold and zero-crossing rate threshold; otherwise, it is marked as a silent frame.

[0080] Then, when any framed audio signal in the audio channel at the same time is marked as a silent frame, the framed audio signal is removed, and the framed audio signals of the remaining time are spliced ​​together to form the spliced ​​audio signal.

[0081] In this step, if all audio channels are marked as valid speech frames at the same time after all silent frames have been marked, then the speech frame audio signal is retained. Conversely, if a silent frame appears in any audio channel, then the speech frame audio signal of that frame is removed from all audio channels. Furthermore, the spliced ​​audio signal needs to be truncated to a preset sampling point length to ensure that the dimension of the audio signal input to the CV-RSR is consistent. For example, the spliced ​​audio signal can be truncated to 8000 sampling points.

[0082] Next, a short-time Fourier transform is performed on the spliced ​​audio signals of each audio channel to obtain a complex-valued matrix.

[0083] Finally, the complex-valued matrix network is obtained based on the complex-valued matrix.

[0084] Furthermore, step S15 can be implemented in the following way.

[0085] First, the target complex value network is integrated into global complex value features according to the time dimension.

[0086] Next, the global complex-valued features are subjected to average pooling to obtain the target feature vector.

[0087] In this embodiment, by capturing the time dependence of the target complex-valued network to form global complex-valued features, phase distortion caused by reverberation can be corrected, and gradient vanishing and exploding can be prevented. After average pooling in the time dimension, a target feature vector independent of the time and frequency dimensions is formed.

[0088] Furthermore, step S16 can be implemented in the following way.

[0089] First, the target feature vectors are input into parallel fully connected layer submodules to obtain a first target complex value for representing the azimuth angle and a second target complex value for representing the pitch angle.

[0090] Next, the azimuth angle is obtained based on the complex value of the first target, and the elevation angle is obtained based on the complex value of the second target.

[0091] In this step, the complex values ​​of the first and second targets can be mapped onto the complex plane as vectors, with their phase angles representing the azimuth and elevation angles.

[0092] Finally, the sound source is located based on the azimuth and elevation angles.

[0093] Further, please refer to Figure 5 , Figure 5 for Figure 1 The flowchart after step S15 shows that, after step S16, the method further includes: Step S17: Obtain the azimuth estimation error and elevation estimation error based on the azimuth angle, elevation angle, and the pre-predicted azimuth angle and elevation angle.

[0094] In this step, the azimuth estimation error is the difference between the predicted azimuth and the actual azimuth (i.e., the azimuth angle), and the pitch estimation error is the difference between the predicted pitch angle and the actual pitch angle (i.e., the pitch angle).

[0095] Step S18: Obtain the azimuth variance based on the azimuth estimation error, and obtain the pitch variance based on the pitch estimation error.

[0096] Step S19: Obtain the azimuth sub-loss function based on the azimuth angle, and obtain the pitch sub-loss function based on the pitch angle.

[0097] Step S20: Construct the target loss function based on the azimuth variance, pitch variance, azimuth sub-loss function, and pitch sub-loss function.

[0098] Step S21: Adjust the complex-valued residual shrinking recurrent neural network model according to the target loss function.

[0099] In this embodiment, the traditional total loss function is usually expressed as a weighted sum of various sub-loss functions, but the weights require a large number of parameter calls, which may affect the model's performance. This embodiment uses the homoscedastic uncertainty loss function, using the azimuth and pitch variances as learnable parameters, which can dynamically adjust the weights of each module and improve the robustness and generalization ability of CV-RSR in multi-task learning.

[0100] Furthermore, step S20 can be implemented in the following way.

[0101] The objective loss function is expressed as follows:

[0102] in, This represents the azimuth variance, which is the azimuth estimation error. variance This represents the pitch angle variance, which is the pitch angle estimation error. variance This represents the azimuth sub-loss function. The formulas for the pitch and azimuth loss functions are as follows: The formula for the pitch loss function is: .

[0103] In this embodiment, when using the aforementioned sound source localization method based on complex residual shrinking recurrent neural network for sound source localization, especially under high noise and reverberation environmental conditions, the mean absolute error of azimuth and pitch angles is reduced by at least 10%.

[0104] For example, the aforementioned sound source localization method based on complex-valued residual shrinking recurrent neural network was experimentally tested. The test scenario was: a sound source with a size of... A rectangular room. An octahedral microphone array... Centered on the octahedron, the structure consists of M=6 microphones, located at the six vertices of the octahedron, with each microphone 5 cm from the center. Furthermore, a mirror source method is used to simulate the room impulse response (RIR) as a measure of reverberation, and the selected... of This indicates the time it takes for the indoor sound level to decrease by 60 dB, using six different signal-to-noise ratio (SNR) values: -10 dB, 0 dB, 10 dB, and 20 dB.

[0105] The test sample consisted of two voice samples from the male speaker out of 30 clean voice samples from the source speech data, which included both male and female speakers, with a sampling rate of 8kHz.

[0106] Test process and results: Different sound source locations, signal-to-noise ratios, and reverberation levels were combined, and batch simulations were performed using acoustic simulation technology to generate audio signals with noise and reverberation. The aforementioned sound source localization method based on complex-valued residual contraction recurrent neural networks was used to process the audio signals received by the microphone to locate the sound source. The mean absolute error of the estimated azimuth angle was... absolute error of pitch angle Compared to different SNR and The box plots are as follows: Figure 6 and Figure 7 As shown, Figure 6 This embodiment provides a box plot of the mean absolute error of the azimuth angle in the experimental test. Figure 7 This is a box plot showing the mean absolute error of the pitch angle in the experimental tests provided in this embodiment. Figure 6 and Figure 7 It can be observed that when the SNR increases above 0 dB, the MAE decreases significantly, especially in... ms and In milliseconds (ms), the above-described sound source localization method can accurately and precisely locate the sound source even under high noise and reverberation conditions.

[0107] Furthermore, to explore the details of the sound source localization method in this embodiment, ablation experiments were conducted on each module of the complex-valued residual shrinkage recurrent neural network. In the first ablation experiment test, the complex-valued residual shrinkage module was removed, and its average MAE result was as follows: Figure 8 As shown, Figure 8 This is a graph showing the test results of the mean absolute error in the ablation experiment provided in this embodiment. CV-RSR represents a complex-valued residual shrinking recurrent neural network, CV-RSBU-CW represents a complex-valued residual shrinking module, and CV-GRU represents a bidirectional complex-valued gated recurrent module. From... Figure 8 The first and second lines show the estimated values ​​after removing the complex-valued residual shrinkage module. and The increases of 23.1 degrees and 15.3 degrees respectively indicate that the model performance decreased significantly under this condition.

[0108] In the second ablation experiment, the bidirectional complex-gated loop module was removed, from Figure 5 The first and third lines show the estimated value after removing the bidirectional complex-gated loop module. and The increases of 13.2 degrees and 6.9 degrees respectively indicate a significant decrease in model performance. Thus, the complex-valued residual shrinkage module and the bidirectional complex-valued gated recurrent module play crucial roles in the CV-RSR neural network.

[0109] Furthermore, to further test the generalization ability of the sound source localization method, its generalization performance was tested in different scenarios. In the first generalization performance test, for scenarios with unknown SNR, a new set of SNR values ​​{-5 dB, 5 dB, 15 dB} not seen in the above experimental tests was added. The final test results are as follows. Figure 9 As shown, Figure 9 The figure shows the mean absolute error (MAE) test results under unknown signal-to-noise ratio (SNR) conditions in the generalization performance test of this embodiment. It can be observed that the MAE does not increase significantly under all unknown SNR scenarios, regardless of the SNR level. The proposed sound source localization method... and All are less than 3.0 degrees.

[0110] In the second generalization performance test, for the unknown T60 scenario, a new set of T60 values ​​{150 ms, 250 ms, 350 ms, 450 ms, 550 ms} that did not appear in the above experimental tests were added. The final test results are as follows. Figure 10 As shown, Figure 10 This embodiment provides a graph showing the test results of the mean absolute error under unknown reverberation in the generalization performance test. Figure 10 As can be seen, when the signal-to-noise ratio (SNR) is greater than or equal to 0 dB, the mean absolute error (MAE) does not exceed 4.0, indicating good performance. Thus, the sound source localization method of this embodiment also exhibits good generalization ability under unknown SNR conditions.

[0111] Furthermore, a comparative test was conducted between the current CVNN-based sound source localization method and the sound source localization method of this embodiment. The first sound source localization method used in this comparative test is a sound source azimuth angle estimation based on regression-based CV-CRNN, and the second sound source localization method is a sound source localization method based on multi-class real-valued artificial neural networks. This method uses the real and imaginary parts of the spatial covariance matrix of the array signal as input features, and the room space is divided into multiple cubic regions, in which the probability of the source's existence is calculated.

[0112] To demonstrate the effectiveness of the sound source localization method in this embodiment, an additional fully connected layer is integrated into the final part of the CV-CRNN in the first sound source localization method to simultaneously estimate the azimuth and elevation angles. This utilizes various SNR and The three methods were tested in combination, and the test results are as follows: Figure 11 As shown, Figure 11 This is a graph showing the test results of the mean absolute error in the comparative test provided in this embodiment, where CV-CRNN represents the first sound source localization method and ANN represents the second sound source localization method. Figure 11It can be seen that the mean absolute error of the sound source localization method in this embodiment is significantly lower than that of the other two methods. Compared with the first sound source localization method, the proposed CV-RSR-based sound source localization method... and The proposed CV-RSR-based sound source localization method reduced the noise level by 12.8 degrees and 5.4 degrees, respectively, compared to the second sound source localization method. and The temperature dropped by 20.6 degrees and 4.9 degrees.

[0113] Based on the same inventive concept, please refer to Figure 12 , Figure 12 This is a schematic diagram of the functional modules of a sound source localization device 20 based on a complex-valued residual shrinking recurrent neural network provided in this embodiment. This embodiment can divide the sound source localization device 20 based on the complex-valued residual shrinking recurrent neural network into functional modules according to the above method embodiment. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; the actual implementation may have other division methods. For example, in the case of dividing each functional module according to its own function... Figure 12 The sound source localization device 20 based on complex-valued residual shrinking recurrent neural network shown is only a schematic diagram. The sound source localization device 20 based on complex-valued residual shrinking recurrent neural network may include a construction unit 200, an acquisition unit 210, an input unit 220, a first determination unit 230, a second determination unit 240, and an output unit 250. The functions of each module of the sound source localization device 20 based on complex-valued residual shrinking recurrent neural network will be described in detail below.

[0114] The building unit 200 is used to build a complex-valued residual shrinking recurrent neural network model, wherein the complex-valued residual shrinking recurrent neural network model includes a complex-valued residual shrinking module, a bidirectional complex-valued gated recurrent module, and an angle prediction module connected in sequence.

[0115] In this embodiment, the construction unit 200 can be used to execute Figure 1 For a detailed description of the building unit 200, please refer to the description of step S11 shown.

[0116] The acquisition unit 210 is used to acquire audio signals from multiple audio channels detected by the sound source acquisition device.

[0117] In this embodiment, the acquisition unit 210 can be used to perform... Figure 1For a detailed description of the acquisition unit 210, please refer to the description of step S12 shown in step S12.

[0118] The input unit 220 is used to obtain a complex-valued matrix network based on the audio signal, wherein the complex-valued matrix network is formed by a complex-valued matrix obtained based on the audio signal of each audio channel, and the complex-valued matrix includes multiple complex values ​​for characterizing the amplitude and phase of the audio signal.

[0119] In this embodiment, the input unit 220 can be used to perform... Figure 1 For a detailed description of the input unit 220, please refer to the description of step S13 shown.

[0120] The first determining unit 230 is used to input the complex value matrix network into the complex value residual shrinking module to obtain the soft threshold corresponding to the audio channel based on the complex value matrix network, and to perform residual shrinking on the complex value matrix network based on the soft threshold to obtain the target complex value network.

[0121] In this embodiment, the first determining unit 230 can be used to perform... Figure 1 For a detailed description of the first determining unit 230, see the description of step S14 shown.

[0122] The second determining unit 240 is used to input the target complex value network into the bidirectional complex value gated loop module to extract global features in the target complex value network to obtain the target feature vector.

[0123] In this embodiment, the second determining unit 240 can be used to perform... Figure 1 For a detailed description of the second determining unit 240, see the description of step S15 shown.

[0124] The output unit 250 is used to input the target feature vector into the angle prediction module to obtain the azimuth and pitch angles, and to locate the sound source based on the azimuth and pitch angles.

[0125] In this embodiment, the output unit 250 can be used to perform... Figure 1 For a detailed description of the output unit 250, please refer to the description of step S16 shown.

[0126] Based on the same inventive concept, embodiments of this application also provide an electronic device, which includes a processor and a readable storage medium. The processor is connected to the readable storage medium, and the readable storage medium stores a test program or test instructions. The processor is used to execute the test program or test instructions in the readable storage medium to execute multiple sound source localization methods based on complex-valued residual shrinking recurrent neural networks as described in the foregoing embodiments of this application.

[0127] Based on the same inventive concept, embodiments of this application also provide a readable storage medium storing a test program or test instructions. When the test program or test instructions are executed, they implement multiple sound source localization methods based on complex-valued residual shrinking recurrent neural networks as described in the previous embodiments of this application.

[0128] In summary, this application provides a sound source localization method and apparatus based on a complex-valued residual shrinking recurrent neural network. First, a complex-valued residual shrinking recurrent neural network model is constructed. Next, audio signals from multiple audio channels are acquired. Then, a complex-valued matrix network is obtained based on the audio signals. Next, the complex-valued matrix network is input into a complex-valued residual shrinking module to obtain soft thresholds corresponding to the audio channels based on the complex-valued matrix, and residual shrinking is performed on the complex-valued matrix based on the soft thresholds to obtain a target complex-valued network. Then, the target complex-valued network is input into a bidirectional complex-valued gated recurrent module to obtain a target feature vector. Finally, the target feature vector is input into an angle prediction module to obtain the azimuth and elevation angles, and the sound source is located based on these two angles. Thus, the above sound source localization method effectively extracts high signal-to-noise ratio features from the complex-valued matrix network while dynamically suppressing low-energy interference features, thereby significantly improving the estimation accuracy of sound source localization under high-noise and reverberation environments.

[0129] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0130] It should be understood that, although Figure 1 , Figure 4 and Figure 5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders.

[0131] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0132] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A sound source localization method based on a complex-valued residual shrinking recurrent neural network, characterized in that, A computer device used in a sound source localization system, the sound source localization system further comprising a plurality of evenly distributed sound source acquisition devices, the sound source acquisition devices being connected to the computer device, the method comprising: A complex-valued residual shrinkage recurrent neural network model is constructed, wherein the complex-valued residual shrinkage recurrent neural network model includes a complex-valued residual shrinkage module, a bidirectional complex-valued gated recurrent module, and an angle prediction module connected in sequence; Acquire audio signals from multiple audio channels detected by the sound source acquisition device; A complex-valued matrix network is obtained based on the audio signal, wherein the complex-valued matrix network is formed by a complex-valued matrix obtained based on the audio signal of each audio channel, and the complex-valued matrix includes a plurality of complex values ​​for characterizing the amplitude and phase of the audio signal; The complex-valued matrix network is input into the complex-valued residual shrinking module to obtain the soft threshold corresponding to the audio channel based on the complex-valued matrix network, and the target complex-valued network is obtained by residual shrinking the complex-valued matrix network based on the soft threshold. The target complex value network is input into the bidirectional complex value gated recurrent module to extract global features from the target complex value network to obtain the target feature vector; The target feature vector is input into the angle prediction module to obtain the azimuth and pitch angles, and the sound source is located based on the azimuth and pitch angles.

2. The sound source localization method as described in claim 1, characterized in that, The step of inputting the complex-valued matrix network into the complex-valued residual shrinking module to obtain the soft threshold corresponding to the audio channel based on the complex-valued matrix network, and performing residual shrinking on the complex-valued matrix network based on the soft threshold to obtain the target complex-valued network includes: The threshold scaling parameters corresponding to the audio channel are determined based on the complex-valued matrix network. The soft threshold corresponding to the audio channel is obtained based on all complex values ​​in the complex value matrix and the threshold scaling parameter. The magnitude corresponding to the complex value is determined based on the complex value in the complex value matrix; The target complex value is obtained by residual shrinking the complex value based on the soft threshold and the amplitude. The target complex value matrix is ​​obtained based on the target complex value, and the target complex value network is obtained based on the target complex value matrix and the complex value matrix.

3. The sound source localization method as described in claim 2, characterized in that, The step of obtaining the soft threshold corresponding to the audio channel based on all complex values ​​in the complex value matrix and the threshold scaling parameter includes: The formula for expressing the soft threshold is: in, Indicates the soft threshold. Represents complex values. Represents the time dimension in a complex-valued matrix. This represents the frequency dimension in a complex-valued matrix. This represents the audio channels in a complex-valued matrix network. This represents the sum of all complex values ​​in the complex-valued matrix. This represents the threshold scaling parameter.

4. The sound source localization method as described in claim 3, characterized in that, The step of performing residual shrinkage on the complex value based on the soft threshold and the amplitude to obtain the target complex value includes: Compare the soft threshold and the magnitude of the amplitude; When the amplitude is greater than the soft threshold, the amplitude of the complex value is reduced while the phase of the complex value remains unchanged to obtain the target complex value; When the amplitude is less than or equal to the soft threshold, the complex value is assigned the value of 0 to obtain the target complex value.

5. The sound source localization method as described in claim 1, characterized in that, The step of obtaining the complex-valued matrix network based on the audio signal includes: The audio signal is divided into frames to obtain a framed audio signal, wherein the audio signals of two adjacent frames overlap. The speech activity of the segmented audio signal corresponding to the audio channel at the same time is detected, the segmented audio signal is divided into speech frame audio signal and silence frame audio signal, and the silence frame audio signal is marked as a silence frame. Calculate the root mean square energy and zero-crossing rate of the audio signal of the speech frame; When the root mean square energy of the audio signal of the voice frame is less than or equal to a preset root mean square energy threshold or the zero crossing rate of the audio signal of the voice frame is less than or equal to a preset zero crossing rate threshold, the audio signal of the voice frame is marked as a silent frame. When any of the segmented audio signals in the audio channel at the same time is marked as a silent frame, the segmented audio signal is removed, and the segmented audio signals of the remaining time are spliced ​​together to form a spliced ​​audio signal. The complex-valued matrix is ​​obtained by performing a short-time Fourier transform on the spliced ​​audio signal of each audio channel. The complex-valued matrix network is obtained from the complex-valued matrix.

6. The sound source localization method as described in claim 1, characterized in that, The step of inputting the target complex value network into the bidirectional complex value gated recurrent module to extract global features from the target complex value network to obtain the target feature vector includes: The target complex value network is integrated into global complex value features according to the time dimension; The target feature vector is obtained by performing average pooling on the global complex features.

7. The sound source localization method as described in claim 1, characterized in that, The step of inputting the target feature vector into the angle prediction module to obtain the azimuth and elevation angles, and locating the sound source based on the azimuth and elevation angles, includes: The target feature vectors are respectively input into parallel fully connected layer sub-modules to obtain a first target complex value for characterizing the azimuth angle and a second target complex value for characterizing the pitch angle; The azimuth angle is obtained based on the complex value of the first target, and the pitch angle is obtained based on the complex value of the second target; The sound source is located based on the azimuth and elevation angles.

8. The sound source localization method as described in claim 1, characterized in that, After the steps of inputting the target feature vector into the angle prediction module to obtain the azimuth and elevation angles, and locating the sound source based on the azimuth and elevation angles, the method further includes: The azimuth angle estimation error and the pitch angle estimation error are obtained based on the azimuth angle, the pitch angle, and the pre-predicted azimuth angle and the pitch angle; The azimuth variance is obtained based on the azimuth estimation error, and the pitch variance is obtained based on the pitch estimation error; The azimuth sub-loss function is obtained based on the azimuth angle, and the pitch sub-loss function is obtained based on the pitch angle; A target loss function is constructed based on the azimuth variance, the pitch variance, the azimuth sub-loss function, and the pitch sub-loss function; The complex-valued residual shrinkage recurrent neural network model is adjusted according to the target loss function.

9. The sound source localization method as described in claim 8, characterized in that, The step of constructing the target loss function based on the azimuth variance, the pitch variance, the azimuth sub-loss function, and the pitch sub-loss function includes: The formula for the target loss function is as follows: in, This represents the azimuth variance, which is the azimuth estimation error. variance This represents the pitch angle variance, which is the pitch angle estimation error. variance This represents the azimuth sub-loss function. The formulas for the pitch and azimuth loss functions are as follows: The formula for the pitch angle loss function is: .

10. A sound source localization device based on a complex-valued residual shrinking recurrent neural network, characterized in that, A computer device used in a sound source localization system, the sound source localization system further comprising a plurality of evenly distributed sound source acquisition devices, the sound source acquisition devices being connected to the computer device, the device comprising: A construction unit is used to construct a complex-valued residual shrinking recurrent neural network model, wherein the complex-valued residual shrinking recurrent neural network model includes a complex-valued residual shrinking module, a bidirectional complex-valued gated recurrent module, and an angle prediction module connected in sequence. The acquisition unit is used to acquire audio signals from multiple audio channels detected by the sound source acquisition device; An input unit is configured to obtain a complex-valued matrix network based on the audio signal, wherein the complex-valued matrix network is formed by a complex-valued matrix obtained based on the audio signal of each audio channel, and the complex-valued matrix includes a plurality of complex values ​​for characterizing the amplitude and phase of the audio signal; The first determining unit is used to input the complex-valued matrix network into the complex-valued residual shrinking module to obtain the soft threshold corresponding to the audio channel based on the complex-valued matrix network, and to perform residual shrinking on the complex-valued matrix network based on the soft threshold to obtain the target complex-valued network; The second determining unit is used to input the target complex value network into the bidirectional complex value gated loop module to extract global features in the target complex value network to obtain a target feature vector. The output unit is used to input the target feature vector into the angle prediction module to obtain the azimuth and pitch angles, and to locate the sound source based on the azimuth and pitch angles.

Images