Sound source orientation identification method based on three-array-element miniature microphone array

Abstract

The invention belongs to the technical field of sound source positioning, and particularly relates to a sound source orientation identification method based on a three-array-element miniature microphone array. According to the method, sound source localization is converted into a deep learning classification task by extracting a microphone array sound intensity feature map. The method comprises the following steps: firstly, on the basis of a three-array-element differential microphone array, extracting orthogonal sound intensity features and converting the orthogonal sound intensity features into an RGB sound intensity spectrogram; secondly, designing a new lightweight neural network to meet the requirements of sound source localization and deployment; and finally, the horizontal plane sound source orientation is divided into different angle categories, a sound intensity spectrogram data set is made and sent to a neural network for training and testing, and a sound source orientation recognition result is obtained. Compared with the prior art, the feature map and the network model provided by the invention have better robustness in a high reverberation environment, the azimuth recognition angle precision is high, the calculation complexity is low, and the method is not influenced by voice characteristics.

Description

Technical Field

[0001] The present invention belongs to the technical field of sound source localization, and in particular is a sound source orientation recognition method based on a three-element miniature microphone array. Background Art

[0002] Sound source localization plays a vital role in modern communication and sensing systems, such as navigation, human-computer interaction, rescue operations, and intelligent monitoring. In practical applications, sound source localization is usually simplified to DOA arrival direction estimation. Traditional sound source direction estimation methods include: the first sound source DOA estimation method based on time difference of arrival TDOA, such as generalized cross correlation with phase transformation GCC-PHAT; the second beamforming method, such as controllable response power with phase transformation SRP-PHAT; the third high-resolution spectrum estimation technique, such as multiple signal classification MUSIC and signal parameter estimation through rotation invariance technique ESPRIT. However, in indoor environments, the presence of reverberation may cause the performance of these traditional methods to degrade significantly. To solve this problem, a feasible method is to utilize larger arrays and an extended number of microphones, because a larger array means higher spatial diversity. However, in some practical applications, such as mobile portable devices and limited vehicle space, the number of microphones may be limited, and a smaller array configuration needs to be preferred, which poses a challenge to traditional sound source direction estimation methods.

[0003] At the same time, the sound source estimation method based on sound intensity can estimate the sound intensity related to the sound source orientation estimation by simultaneously measuring the sound pressure and particle velocity, which is very beneficial for accurately estimating DOA. Its core is to use the finite difference of the pressure measurement values ​​of two adjacent microphones to approximate the pressure gradient, and construct it as a first-order differential microphone array DMA. Due to the small size characteristics of the first-order DMA array itself, it provides a good solution for sound source orientation estimation under small-size arrays. At present, there have been related research and patent inventions on basic differential microphone small-size arrays and sound source orientation estimation based on sound intensity, but overall the complexity of the sound intensity characteristics obtained is relatively high, and lacks certain lightweight and real-time considerations.

[0004] In recent years, data-driven deep learning technology has shown great potential in sound source localization. It can learn the nonlinear relationship between the information contained in acoustic features and the location of the sound source, thereby improving the accuracy and robustness of sound source localization. Therefore, combining the sound source orientation estimation based on differential microphone sound intensity characteristics with deep learning technology is conducive to promoting the development of sound source localization of small-sized microphone arrays in reverberant environments, and is also a hot direction in fields such as artificial intelligence. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a sound source orientation recognition method based on a three-element micro-microphone array, converts the signals received by the three-element micro-array into a color RGB sound intensity spectrogram to produce a data set, and designs a dedicated lightweight neural network for training and testing the effectiveness of sound source estimation.

[0006] The technical solution of the present invention is:

[0007] Step 1) designing a microphone array with three microphone units and performing sound intensity preprocessing on the received signal;

[0008] Step 2) extract the sound intensity information of the signal and convert it into a designed color sound intensity spectrogram for sound source direction estimation;

[0009] Step 3) Divide the horizontal plane into 72 categories at fixed intervals, and create a sound intensity spectrogram dataset for each category;

[0010] Step 4) Build a lightweight neural network;

[0011] Step 5) Feed the sound intensity spectrogram dataset into the neural network for training and testing to evaluate the sound source localization performance.

[0012] Furthermore, step 1) is specifically as follows:

[0013] 101) Construct a three-element microphone array, whose array structure is an isosceles right triangle. The three microphone units 1, 2, and 3 are denoted as M1, M2, and M3, respectively, and are located at the three vertices of the triangle. Microphone 2 is located at the right-angled vertex of the triangle. The distance between microphone 1 and microphone 2 is denoted as the array diameter, and the array diameter is 4 cm.

[0014] 102) Reusing the right-angle vertex microphone No. 2, forming a differential microphone array through the H axis in the direction of microphones No. 1 and No. 2, and forming a differential microphone array through the R axis in the direction of microphones No. 3 and No. 2, to obtain two orthogonal microphone groups;

[0015] 103) Perform signal preprocessing to obtain the sound pressure and vibration velocity related to the sound source information:

[0016] (1)

[0017] (2)

[0018] (1) In the formula represents the vibration velocity component of each time-frequency point in the H-axis direction where microphones 1 and 2 are located, (2) where It represents the velocity component of each time-frequency point in the R-axis direction where microphone No. 3 and microphone No. 2 are located. is the short-time Fourier change of the sound pressure at microphone i, i=1,2,3, represents the time-frequency point, represents an imaginary number, represents the air density, It means the array size is 4cm;

[0019] 104) The sound intensity component at each time-frequency point is obtained through sound pressure and vibration velocity:

[0020] (3)

[0021] (4)

[0022] (3) In the formula is the component of the sound intensity at each time-frequency point in the H-axis direction at the coordinate of the upper microphone 2, (4) where is the component of the sound intensity at each time-frequency point in the R-axis direction at the coordinate of the upper microphone 2, represents the real part of a complex number;

[0023] 105) Preprocess the obtained sound intensity components, normalize them and map the sound intensity values ​​to the range of [0,255]:

[0024] (5)

[0025] (6)

[0026] (5) In the formula represents the normalized component of the H-axis sound intensity component, (6) where Represents the normalized component of the H-axis sound intensity component;

[0027] (7)

[0028] (8)

[0029] (7) In the formula represents the sound intensity grayscale image of the H-axis sound intensity component, (8) where Represents the grayscale image component of the R-axis sound intensity component, which is used to construct the layer of the RGB sound intensity spectrogram.

[0030] Furthermore, step 2) is specifically as follows:

[0031] 201) Normalize and map the sound intensity component to the value range of [0,255] and The Green and Blue channels placed on the RGB color image channels, referred to as the G and B channels, are combined into an RGB sound intensity spectrogram;

[0032] 202) Furthermore, in order to improve the stability of sound source localization of RGB sound intensity spectrogram in reverberant noise environment, a small value masking layer is designed, and a binary masking function is introduced to refine the selection of reliable TF points:

[0033] (9)

[0034] (9) In the formula is a binary masking function, represents the sound intensity component of the * axis, It represents the average value of the sum of the sound intensities of all time-frequency points of the m×n size numerical matrix after the short-time Fourier transform of the sound intensity component;

[0035] 203) Introducing binary masking into the sound intensity grayscale image, the masked sound intensity spectrogram is expressed as:

[0036] (10)

[0037] (10) In the formula Represents the grayscale image of the masking sound intensity on the H axis;

[0038] 204) Based on the RGB sound intensity spectrogram, the masking sound intensity grayscale map of the H axis is selected and placed in the Red channel of the RGB sound intensity spectrogram, referred to as the R channel, to generate the RGB masking sound intensity spectrogram.

[0039] Furthermore, step 3) is specifically as follows:

[0040] 301) The 360-degree horizontal plane is divided into 72 angle categories with an interval of 5 degrees as the positioning angle categories for sound source azimuth estimation. A total of 300 1-second speech data sets are selected, and the RIR generator software is used to generate array signals after the room impulse response of each angle for the original pure speech. The parameters of the simulated RIR impulse response are configured to control the signal-to-noise ratio SNR in the range of 5dB to 30dB, the reverberation time RT60 in the range of 0.2s to 1.0s, and the room size in the range of 7×6×3 (m). In order to enrich the data set and improve the versatility of azimuth estimation, for each RIR room impulse response, the SNR and RT60 are randomly floated within the basic range, the room size has a random change of ±1m on the basic size, and the sound source position is random in 1m, 2m, and 3m.

[0041] 302) generating RGB sound intensity spectrograms and RGB masking sound intensity spectrograms corresponding to 72 sound source azimuth angles for the simulated array signal;

[0042] 303) The dataset was prepared. There are two types of spectrogram datasets, one is RGB sound intensity spectrogram and the other is RGB masked sound intensity spectrogram. For each type of spectrogram, a sound source localization dataset with 72 angle categories was prepared. Finally, 300 voices × 3 sound source distances × 72 angle categories × 2 types of spectrograms, a total of 129,600 images were obtained as the dataset.

[0043] Furthermore, step 4) is specifically as follows:

[0044] 401) Design a network feature extraction layer based on RepViT to enhance the ability of neural networks to extract local features at different resolutions. At the same time, introduce the MetaFormer structure of RepViT to maintain efficient local feature extraction while capturing global information;

[0045] 402) The downsampling layer with the RepViT structure is fused with the input of the MobileViT network module to form a new lightweight neural network learning module, which is denoted as the RepMobileViT module;

[0046] 403) Reference Figure 6 , by connecting RepViT and RepMobileViT modules in series, configuring the number and structure of modules, and combining them into the designed lightweight network RepMobileViT network; specifically, refer to Figure 7 The detailed submodules of the complete network are shown, namely: (a) RepViT block, including a DW convolution, SE module and 1×1 convolution; (b) RepViT downsampling block, including a RepViT block core, downsampling DW convolution with a stride of 2, 1×1 convolution and FFN module, where B represents the batch size of the input feature, C1 represents the number of input channels, H1 represents the height of the input feature, and W1 represents the width of the input feature; (c) MobileViT-v3 module, including a local representation block composed of DW convolution, a global representation block composed of Transformer and a feature fusion module, where L represents the number of Transformer blocks, h and w represent that the size of the feature block is 2, Cin represents the number of channels of the input feature, Cout represents the number of channels of the output feature, H represents the height of the input feature, and W represents the width of the input feature; use Figure 7 The three submodules in the neural network are designed to achieve Figure 6The complete neural network shown in the figure has a core module implemented by 5 layers: Layer 1 contains a RepViT module; Layer 2 contains a RepViT downsampling block and two RepViT blocks; Layer 3 contains a RepViT downsampling block and a MobileViT-v3 module; Layer 4 contains a RepViT downsampling block and a MobileViT-v3 block; Layer 5 contains a RepViT downsampling block and a MobileViT-v3 block. Finally, the network completes the output Logits of the neural network through a 1×1 convolution, a Global Pool global pooling layer, and a Linear fully connected layer.

[0047] Furthermore, step 5) is specifically as follows:

[0048] 501) Performing neural network training on the RGB sound intensity spectrogram dataset and the RGB masked sound intensity spectrogram respectively, and dividing the dataset into a training set and a test set according to a ratio of 8:2;

[0049] 502) PyTorch is used as the deep learning framework to build and train the RepMobileViT model. The specific equipment and configuration are as follows: Python 3.8 and PyTorch 1.10.1 + cuda11.3; the experimental infrastructure is CPU: Inteli5-13490F; GPU: NVIDIA GeForce RTX 3060 Ti; In addition, the Label Smoothing LossCross-Entropy Loss function and AdamW optimizer are used to update the model parameters; the cosine annealing learning rate scheduler is used, and the initial learning rate is 0.0125;

[0050] 503) Send the training set data set and its category labels to the neural network for training for 200 rounds, and take the weight file best.pt of the best training result as the final weight for testing;

[0051] 504) Use best.pt to test and verify the test set and compare the corresponding indicators; calculate the mean absolute error, accuracy and φ-accuracy to evaluate the performance of the proposed method:

[0052] (11)

[0053] (11) In the formula represents the actual angle of the sound source, represents the estimated angle output by the neural network, Indicates the number of angle estimates;

[0054] (12)

[0055] (12) In the formula Indicates the total number of sound source orientation evaluations. It is the total number of correct sound source direction assessments; φ degree-accuracy indicates the accuracy when the DOA test error is within φ degrees;

[0056] 505) Use the RepMobileViT model to test the performance of RGB sound intensity spectrogram and RGB masked sound intensity spectrogram, and evaluate the performance under different reverberation and different signal-to-noise ratio in the simulated environment;

[0057] 506) Further using the display microphone array and its supporting equipment, the horizontal plane is divided into 12 angle categories with an interval of 30 degrees with the microphone array as the center, and the real environment speech signal is recorded to evaluate the sound source direction estimation performance in the real environment.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] (1) The present invention is a method for identifying the direction of a sound source based on a three-element micro-microphone array. The method extracts the sound intensity information related to the sound source estimation and designs the RGB sound intensity spectrogram and the RGB masked sound intensity spectrogram. Compared with previous studies and patents, the number of microphones in the four-element microphone array in this field is reduced to three, while the complexity of the sound intensity characteristics and the number of channels are reduced, which greatly improves the practicability of the method model.

[0060] (2) This paper constructs a new lightweight neural network RepMobileViT, which provides an efficient and lightweight solution and improves the timeliness and lightweightness of the deep learning sound source direction estimation method from the perspective of network design and optimization;

[0061] (3) Compared with the previous deep learning classification methods for sound source orientation estimation and sound intensity source localization, the present invention further improves the resolution to 5 degrees, that is, 72 angle categories are realized in a 360-degree plane, which can realize sound source orientation estimation in a highly reverberant room. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the system flow of the present invention;

[0063] Figure 2 A conventional four-element microphone array (left) and a three-element microphone array of the present invention (right);

[0064] Figure 3 The RGB sound intensity spectrogram design method proposed by the present invention;

[0065] Figure 4 The RGB masking sound intensity spectrogram design method proposed by the present invention;

[0066] Figure 5 The RGB spectrograms of different angles of the present invention are shown in FIG. (a) 0 degrees; (b) 45 degrees; (c) 90 degrees; (d) 135 degrees; the left side of each group is an RGB sound intensity spectrogram, and the right side is an RGB masking sound intensity spectrogram;

[0067] Figure 6 This is a schematic diagram of the structure of the lightweight neural network RepMobileViT proposed in the present invention;

[0068] Figure 7 Modules used inside the lightweight network: (a) RepViT module; (b) RepViT downsampling module; (c) MobileViT-v3 module;

[0069] Figure 8 The sound source direction estimation accuracy of the proposed method and the commonly used methods in related fields for different reverberation times;

[0070] Fig. 9 Figure 2. Sound source direction estimation accuracy of the proposed method and the commonly used methods in related fields for different signal-to-noise ratios.

[0071] Fig.10 Schematic diagram of the test room. DETAILED DESCRIPTION

[0072] The present invention discloses a sound source orientation recognition method based on a three-element micro-microphone array, which is used to solve the problem of sound source localization in indoor reverberant environments and space-constrained scenes. In addition, the method improves the accuracy and timeliness of the model's sound source orientation estimation in a reverberant environment by proposing an RGB sound intensity spectrogram and a lightweight neural network.

[0073] In order to enable those skilled in the art to better understand the scheme of the present invention, this section further describes the present invention in detail with reference to the accompanying drawings and specific implementation methods. It should be noted that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Specifically, refer to Figure 1 As shown, the sound source orientation recognition method based on a three-element micro-microphone array includes the following steps:

[0075] Step 1) Designing a microphone array with three microphone units and performing sound intensity preprocessing on the received signal includes the following steps:

[0076] Step 101) Refer to Figure 2 (Left) Based on the four-element differential microphone array commonly used in the field, a Figure 2 (Right) The three-element microphone array has an array structure of an isosceles right triangle. Microphones M1, M2, and M3 represent microphones 1, 2, and 3, respectively, and are located at the three vertices of the triangle. Microphone 2 M2 is located at the right-angled vertex of the triangle. The distance between microphone 1 and microphone 2 is recorded as the array diameter, which is 4 cm.

[0077] Step 102) On the basis of the XY coordinate system, an H axis pointing from microphone 2 to microphone 1 is constructed, and an R axis pointing from microphone 2 to microphone 3 is constructed; the right-angle vertex microphone 2 is reused, and a differential microphone array is formed through the H axis in the direction of microphones 1 and 2, and a differential microphone array is formed through the R axis in the direction of microphones 3 and 2, so as to obtain two orthogonal microphone groups;

[0078] Step 103) Preprocess the signal and convert it into a two-dimensional expression in the time-frequency domain through short-time Fourier transform (STFT) to obtain the sound pressure and vibration velocity related to the sound source information:

[0079] (1)

[0080] (2)

[0081] (1) In the formula represents the vibration velocity component of each time-frequency point in the H-axis direction where microphones 1 and 2 are located, (2) where It represents the velocity component of each time-frequency point in the R-axis direction where microphone No. 3 and microphone No. 2 are located. is the short-time Fourier change of the sound pressure at microphone i, i=1,2,3, represents the time-frequency point, represents an imaginary number, represents the air density, It means the array size is 4 cm;

[0082] Step 104) The sound intensity component at each time-frequency point is obtained by using the sound pressure and vibration velocity:

[0083] (3)

[0084] (4)

[0085] (3) In the formula is the component of the sound intensity at each time-frequency point in the H-axis direction at the coordinate of the upper microphone 2, (4) where is the component of the sound intensity at each time-frequency point in the R-axis direction at the coordinate of the upper microphone 2, represents the real part of the complex number; according to the sound intensity components of the H axis and the R axis, a pair of orthogonal sound intensity features can be obtained for subsequent preprocessing;

[0086] Step 105) pre-process the obtained sound intensity components, normalize them and map the sound intensity values ​​to the range of [0,255]:

[0087] (5)

[0088] (6)

[0089] (5) In the formula represents the normalized component of the H-axis sound intensity component, (6) where Represents the normalized component of the H-axis sound intensity component;

[0090] (7)

[0091] (8)

[0092] (7) In the formula represents the sound intensity grayscale image of the H-axis sound intensity component, (8) where Represents the grayscale image component of the R-axis sound intensity component, which is used to construct the layer of the RGB sound intensity spectrogram.

[0093] Step 2) extracting the RGB sound intensity spectrogram, including the following steps:

[0094] Step 201) Refer to Figure 3 , which is normalized and mapped to the sound intensity component in the value range of [0,255] and Placed in the Green channel and Blue channel on the RGB color image channel, referred to as the G channel and R channel, combined into an RGB sound intensity spectrogram;

[0095] Step 202) Further, in order to improve the sound source localization stability of the RGB sound intensity spectrogram in a reverberant noise environment, a small value masking layer is designed, and a binary masking function is introduced to refine the selection of reliable TF points:

[0096] (9)

[0097] (9) In the formula is a binary masking function, represents the sound intensity component of the * axis, It represents the average value of the sum of the sound intensities of all time-frequency points of the m×n size numerical matrix after the short-time Fourier transform of the sound intensity component;

[0098] Step 203) The binary mask is introduced into the sound intensity grayscale image, and the masked sound intensity spectrogram is represented as:

[0099] (10)

[0100] (10) In the formula Represents the grayscale image of the masking sound intensity on the H axis;

[0101] Step 204) Refer to Figure 4 , based on the RGB sound intensity spectrogram, select the masking sound intensity grayscale map of the H axis and place it in the Red channel of the RGB sound intensity spectrogram, referred to as the R channel, to generate the RGB masking sound intensity spectrogram.

[0102] Step 3) Create a DOA label dataset, including the following steps:

[0103] Step 301) The 360-degree horizontal plane is divided into 72 angle categories with an interval of 5 degrees as the positioning angle categories for the sound source azimuth estimation. A total of 300 1-second-long speech data sets are selected, and the RIR generator software is used to generate array signals after the room impulse response of each angle for the original pure speech. The parameters of the simulated RIR impulse response are configured to control the signal-to-noise ratio SNR in the range of 5 dB to 30 dB, the reverberation time RT60 in the range of 0.2 s to 1.0 s, and the room size in the range of 7×6×3 (m). In order to enrich the data set and improve the versatility of azimuth estimation, for each RIR room impulse response, the SNR and RT60 are randomly floated within the basic range, the room size has a random change of ±1 m on the basic size, and the sound source position is randomly set at 1 m, 2 m, and 3 m.

[0104] Step 302) Generate RGB sound intensity spectrograms and RGB masking sound intensity spectrograms corresponding to 72 sound source azimuth angles for the simulated array signal, referring to Figure 5 , shows the RGB sound intensity spectrogram (left) and RGB masked sound intensity spectrogram (right) at 0 degrees (a), 45 degrees (b), 90 degrees (c), and 135 degrees (d);

[0105] Step 303) Complete the dataset production. There are two types of spectrogram datasets in total, one is RGB sound intensity spectrogram, and the other is RGB masked sound intensity spectrogram. For each spectrogram, a sound source localization dataset with 72 angle categories is produced; finally, 300 voices × 3 sound source distances × 72 angle categories × 2 spectrograms, a total of 129,600 images are obtained as the dataset.

[0106] Step 4) Design a lightweight neural network, including the following steps:

[0107] Step 401) Design a network feature extraction layer based on RepViT to enhance the ability of neural network to extract local features at different resolutions, and introduce the MetaFormer structure of RepViT to capture global information while maintaining efficient local feature extraction;

[0108] Step 402) The downsampling layer with the RepViT structure is fused with the input of the MobileViT network module to form a new lightweight neural network learning module, which is denoted as the RepMobileViT module;

[0109] Step 403) Refer to Figure 6 , by connecting RepViT and RepMobileViT modules in series, configuring the number and structure of modules, and combining them into the designed lightweight network RepMobileViT network; specifically, refer to Figure 7 The detailed submodules of the complete network are shown, namely: (a) RepViT block, including a DW convolution, SE module and 1×1 convolution; (b) RepViT downsampling block, including a RepViT block core, downsampling DW convolution with a stride of 2, 1×1 convolution and FFN module, where B represents the batch size of the input feature, C1 represents the number of input channels, H1 represents the height of the input feature, and W1 represents the width of the input feature; (c) MobileViT-v3 module, including a local representation block composed of DW convolution, a global representation block composed of Transformer and a feature fusion module, where L represents the number of Transformer blocks, h and w represent that the size of the feature block is 2, Cin represents the number of channels of the input feature, Cout represents the number of channels of the output feature, H represents the height of the input feature, and W represents the width of the input feature; use Figure 7 The three submodules in the neural network are designed to achieve Figure 6 The complete neural network shown in the figure has a core module implemented by 5 layers: Layer 1 contains a RepViT module; Layer 2 contains a RepViT downsampling block and two RepViT blocks; Layer 3 contains a RepViT downsampling block and a MobileViT-v3 module; Layer 4 contains a RepViT downsampling block and a MobileViT-v3 block; Layer 5 contains a RepViT downsampling block and a MobileViT-v3 block. Finally, the network completes the output Logits of the neural network through a 1×1 convolution, a Global Pool global pooling layer, and a Linear fully connected layer.

[0110] Step 5) Deep learning training and testing, including the following steps:

[0111] Step 501) Performing neural network training on the RGB sound intensity spectrogram dataset and the RGB masked sound intensity spectrogram respectively, and dividing the dataset into a training set and a test set according to a ratio of 8:2;

[0112] Step 502) PyTorch is used as a deep learning framework to build and train the RepMobileViT model. The specific equipment and configuration are as follows: Python 3.8 and PyTorch 1.10.1 + cuda11.3; the experimental infrastructure is CPU: Intel i5-13490F; GPU: NVIDIA GeForce RTX 3060 Ti; In addition, the Label Smoothing Loss Cross-Entropy Loss function and AdamW optimizer are used to update the model parameters; the cosine annealing learning rate scheduler is used, and the initial learning rate is 0.0125;

[0113] Step 503) Send the training set data set and its category labels to the neural network for training for 200 rounds, and take the weight file best.pt of the best training result as the final weight for testing;

[0114] Step 504) Use best.pt to test and verify the test set, and compare the corresponding indicators; calculate the mean absolute error, accuracy and φ-accuracy to evaluate the performance of the proposed method:

[0115] (11)

[0116] (11) In the formula represents the actual angle of the sound source, represents the estimated angle output by the neural network, Indicates the number of angle estimates;

[0117] (12)

[0118] (12) In the formula Indicates the total number of sound source orientation evaluations. It is the total number of correct sound source direction assessments; φ degree-accuracy indicates the accuracy when the DOA test error is within φ degrees;

[0119] Step 505) Using the RepMobileViT model, the performance of the RGB sound intensity spectrogram and the RGB masked sound intensity spectrogram are tested respectively, and the performance under different reverberations and different signal-to-noise ratios in the simulated environment is evaluated respectively. As a reference, the common sound source orientation estimation models in the current research field are compared, including: the GCC-PHAT feature model (GCC-PHAT-CNN) trained by CNN, the normalized phase weighted redundant optimized sound intensity method model (SI-PNR-CNN) trained by LSSVM, the normalized phase weighted sound intensity method model (SI-CNN) trained by CNN, the whitening weighted method (SI-WW) trained by CNN, and the RGB sound intensity spectrogram method and the RGB masked sound intensity spectrogram method trained by the lightweight neural network RepMobileViT network proposed in this paper; in the comparative experiment, when the signal-to-noise ratio SNR is fixed at 10dB and the reverberation RT60 increases from 0.2s to 1.0s, the 5-degree Accuracy comparison results of various methods are referenced Figure 8 , When the reverberation RT60 is fixed at 0.6s and the signal-to-noise ratio SNR increases from 5dB to 30dB, the 5-degree accuracy of various methods is compared. Fig. 9 ;

[0120] Step 506) Further using the display microphone array and its supporting equipment, the horizontal plane is divided into 12 angle categories with an interval of 30 degrees with the microphone array as the center. The room layout is based on Fig.10 , record 20 segments of 0.5 seconds of audio at each angle of the real environment speech signal, and evaluate the sound source direction estimation performance in the real environment. The accuracy of the model is shown in Table 1.

[0121] Table 1

[0122] Sound source angle GCC-PHAT-CNN SI-PNR-CNN SI-CNN SI-WW RGB sound intensity spectrogram RGB masking intensity spectrogram -150° 90% 90% 85% 95% 85% 95% -120° 95% 85% 80% 100% 70% 80% -90° 90% 100% 90% 95% 100% 100% -60° 100% 70% 80% 100% 75% 95% -30° 90% 100% 100% 95% 70% 90% 0° 95% 90% 80% 95% 100% 100% 30° 90% 95% 75% 95% 85% 85% 60° 90% 100% 85% 100% 90% 100% 90° 100% 100% 80% 95% 100% 95% 120° 95% 100% 85% 100% 100% 100% 150° 95% 50% 45% 90% 70% 100% 180° 95% 100% 80% 100% 100% 100% Average accuracy 93.75% 90% 90.42% 96.67% 87.08% 95%

[0123] The present invention proposes RGB sound intensity spectrogram and RGB masked sound intensity spectrogram by extracting the sound intensity information of the three-element microphone array, and designs a lightweight neural network to estimate the sound source direction. The method includes signal sound intensity preprocessing, RGB sound intensity spectrogram design, sound source direction estimation data set production, lightweight neural network design and deep learning training and testing. Finally, the complete system can extract the RGB sound intensity spectrogram in the array received signal and obtain the direction estimation of the sound source through the designed neural network. It achieves good sound source localization accuracy in different reverberation environments and noise environments. In the real environment, the accuracy further shows that the RGB masked sound intensity spectrogram has higher accuracy and robustness in the real environment than the RGB sound intensity spectrogram. Compared with the previous classic algorithms in related fields, the method proposed by the present invention has a lighter feature map design and a lighter and more efficient network design, which can greatly reduce feature calculations while maintaining a higher positioning accuracy, reflecting the innovation and efficiency of the present invention.

[0124] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention is described in detail with reference to the above embodiments, it should be understood by those skilled in the art that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein by equivalents. These modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying the direction of a sound source based on a three-element micro-microphone array, which extracts the sound intensity information of the signal and converts it into a sound intensity spectrogram, and then feeds it into a neural network for training and testing to achieve sound source direction estimation, characterized in that The following steps are involved: Step 1) designing a microphone array with three microphone units and performing sound intensity preprocessing on the received signal; Step 2) extract the sound intensity information of the signal and convert it into a designed color sound intensity spectrogram for sound source direction estimation; Step 3) Divide the horizontal plane into 72 categories at fixed intervals, and create a sound intensity spectrogram dataset for each category; Step 4) Build a lightweight neural network; Step 5) Feed the sound intensity spectrogram dataset into the neural network for training and testing to evaluate the sound source localization performance.

2. A method for identifying the direction of a sound source based on a three-element micro-microphone array according to claim 1, characterized in that The steps of designing a microphone array with three microphone units and performing sound intensity preprocessing on the received signal in step 1) are specifically as follows: 101) Construct a three-element microphone array, whose array structure is an isosceles right triangle. The three microphone units 1, 2, and 3 are denoted as M1, M2, and M3, respectively, and are located at the three vertices of the triangle. Microphone 2 is located at the right-angled vertex of the triangle. The distance between microphone 1 and microphone 2 is denoted as the array diameter, and the array diameter is 4 cm. 102) Reusing the right-angle vertex microphone No. 2, forming a differential microphone array through the H axis in the direction of microphones No. 1 and No. 2, and forming a differential microphone array through the R axis in the direction of microphones No. 3 and No. 2, to obtain two orthogonal microphone groups; 103) Perform signal preprocessing to obtain the sound pressure and vibration velocity related to the sound source information: (1) (2) In the above formula, Represents the velocity component of each time-frequency point in the H-axis direction where microphones 1 and 2 are located. It represents the velocity component of each time-frequency point in the R-axis direction where microphone No. 3 and microphone No. 2 are located. is the short-time Fourier change of the sound pressure at microphone i, i=1,2,3, represents the time-frequency point, represents an imaginary number, represents the air density, It means the array size is 4cm; 104) The sound intensity component at each time-frequency point is obtained through sound pressure and vibration velocity: (3) (4) In the above formula, is the component of the sound intensity at each time-frequency point in the H-axis direction at the coordinate of the upper microphone 2, is the component of the sound intensity at each time-frequency point in the R-axis direction at the coordinate of the upper microphone 2, represents the real part of a complex number; 105) Preprocess the obtained sound intensity components, normalize them and map the sound intensity values ​​to the range of [0,255]: (5) (6) In the above formula, represents the normalized component of the H-axis sound intensity component, Represents the normalized component of the H-axis sound intensity component; (7) (8) In the above formula, The sound intensity grayscale image representing the H-axis sound intensity component, Represents the grayscale image component of the R-axis sound intensity component, which is used to construct the layer of the RGB sound intensity spectrogram.

3. A method for identifying the direction of a sound source based on a three-element micro-microphone array according to claim 1, characterized in that The specific steps of converting the sound intensity information of the extracted signal in step 2) into the designed color sound intensity spectrogram are as follows: 201) Normalize and map the sound intensity component to the value range of [0,255] and The Green and Blue channels placed on the RGB color image channels, referred to as the G and B channels, are combined into an RGB sound intensity spectrogram; 202) Furthermore, in order to improve the stability of sound source localization of RGB sound intensity spectrogram in reverberant noise environment, a small value masking layer is designed, and a binary masking function is introduced to refine the selection of reliable TF points: (9) In the above formula, is a binary masking function, represents the sound intensity component of the * axis, It represents the average value of the sum of the sound intensities of all time-frequency points of the m×n size numerical matrix after the short-time Fourier transform of the sound intensity component; 203) Introducing binary masking into the sound intensity grayscale image, the masked sound intensity spectrogram is expressed as: (10) In the above formula, Represents the grayscale image of the masking sound intensity on the H axis; 204) Based on the RGB sound intensity spectrogram, the masking sound intensity grayscale map of the H axis is selected and placed in the Red channel of the RGB sound intensity spectrogram, referred to as the R channel, to generate the RGB masking sound intensity spectrogram.

4. The method for identifying the direction of a sound source based on a three-element micro-microphone array according to claim 1, characterized in that In step 3), the horizontal plane is divided into 72 categories at fixed intervals, and a sound intensity spectrogram dataset is produced for each category: 301) Divide the 360-degree horizontal plane into 72 angle categories with an interval of 5 degrees, select the speech data set and use the RIRgenerator software to generate the array signal after the room impulse response of each angle for the original clean speech; 302) generating RGB sound intensity spectrograms and RGB masking sound intensity spectrograms corresponding to 72 sound source azimuth angles for the simulated array signal; 303) The dataset production is completed. There are two types of spectrogram datasets in total, one is RGB sound intensity spectrogram, and the other is RGB masked sound intensity spectrogram. For each spectrogram, a sound source localization dataset with 72 angle categories is produced.

5. The method for identifying the direction of a sound source based on a three-element micro-microphone array according to claim 1, characterized in that The construction of the lightweight neural network in step 4) is designed as follows: 401) Design a network feature extraction layer based on RepViT to enhance the ability of neural networks to extract local features at different resolutions. At the same time, introduce the MetaFormer structure of RepViT to maintain efficient local feature extraction while capturing global information; 402) The downsampling layer with the RepViT structure is fused with the input of the MobileViT network module to form a new lightweight neural network learning module, which is denoted as the RepMobileViT module; 403) By connecting RepViT and RepMobileViT modules in series and configuring the number and configuration of modules, the designed lightweight network RepMobileViT network is combined.

6. The method for identifying the direction of a sound source based on a three-element micro-microphone array according to claim 1, characterized in that In step 5), the sound intensity spectrogram dataset is fed to the neural network for training and testing to evaluate the sound source localization performance: 501) Performing neural network training on the RGB sound intensity spectrogram dataset and the RGB masked sound intensity spectrogram respectively, and dividing the dataset into a training set and a test set according to a ratio of 8:2; 502) Use PyTorch as the deep learning framework to build and train the RepMobileViT model; 503) Send the training set data set and its category labels to the neural network for training for 200 rounds, and take the weight file best.pt of the best training result as the final weight for testing; 504) Use best.pt to test and verify the test set and compare the corresponding indicators; calculate the mean absolute error, accuracy and φ-accuracy to evaluate the performance of the proposed method: (11) In the above formula, represents the actual angle of the sound source, represents the estimated angle output by the neural network, Indicates the number of angle estimates; (12) In the above formula, Indicates the total number of sound source orientation evaluations. is the total number of correct sound source location assessments; φ degree-accuracy indicates the accuracy when the DOA test error is within φ degrees; 505) Use the RepMobileViT model to test the performance of RGB sound intensity spectrogram and RGB masked sound intensity spectrogram, and evaluate the performance under different reverberation and different signal-to-noise ratio in the simulated environment; 506) The display microphone array and its supporting equipment are further used to record real environment speech signals and evaluate the sound source direction estimation performance in a real environment.

Images