Sound event positioning and detecting method applied to three-dimensional scene reconstruction

Abstract

The invention discloses a sound event positioning and detecting method applied to three-dimensional scene reconstruction. According to the method, an identification network comprising a channel-space extrusion feature extraction module, a multi-scale pooling fusion module, a Transform network module and a full connection layer is constructed, preprocessed audio data is input into the identification network, and network parameters are optimized by using a mean square error loss function. Wherein a residual network module with an attention mechanism of the channel-space extrusion feature extraction module effectively filters noise and reverberation of a surrounding environment, and extracts a frequency band distribution feature corresponding to a sound event at a current moment; the multi-scale pooling fusion module enhances the feature representation capability, ensures that some important information in the audio is not lost, and can effectively promote the network to analyze the acoustic information of the environment where the network is located. The method provides the ability to identify and locate sound events from audio in the field of three-dimensional scene reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, relates to intelligent sound recognition processing, and specifically to a sound event positioning and detection method applied to three-dimensional scene reconstruction. Background Art

[0002] Sound Event Localization and Detection (SELD) refers to identifying the category of sound events of interest from audio, detecting the start and end time of the sound events of that category, and estimating their spatial location. This technology can effectively describe the occurrence status and activity trajectory of a specific sound event. As one of the tasks of intelligent sound recognition processing, sound event localization and detection has potential use value in many application scenarios.

[0003] In the field of three-dimensional scene reconstruction, sound event positioning and detection technology can be used to analyze the acoustic information of the environment. These acoustic features can be integrated with other sensor data such as vision and laser to provide more comprehensive information support for downstream applications such as three-dimensional scene modeling and environmental perception. For example, in the application of smart homes, sound event positioning and detection technology is combined with video data, and three-dimensional reconstruction technology is used to build a virtual model of the home environment. When a potential threat is detected, the system can quickly generate and display a three-dimensional view of the home environment, mark the source location of the sound event, and help users quickly understand the situation at home and respond. In the field of virtual reality tourism, by simulating the sound of the scenic environment and positioning and visualizing it, users wearing VR devices can walk freely in the virtual environment, hear sounds from all directions, and understand the corresponding events.

[0004] In recent years, deep learning-based methods have brought new opportunities for the realization and development of sound event localization and detection technology. Deep neural network models can automatically extract effective representations from complex acoustic features, greatly improving the recognition and positioning performance of sound events. There are many network structures and algorithms specifically customized for sound event localization and detection tasks, such as SELDnet, ACCDOA, EINv2 and other network models. These models have their own advantages and characteristics in different environments. However, these methods have weak ability to identify specific sound event information in real environments full of noise and reverberation. Part of the reason is that these networks fail to effectively capture key information when extracting features, and in the process of feature dimensionality reduction, due to the use of a single pooling operation method, a lot of important information is ignored. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a sound event localization and detection method for three-dimensional scene reconstruction. Effective information is extracted from the spectrum graph through an attention mechanism, and multi-scale fusion and pooling hybrid methods are used for pooling operations to solve the problem of information loss, thereby improving the accuracy of the model in classifying and positioning sound events in a real acoustic environment.

[0006] A method for locating and detecting sound events applied to three-dimensional scene reconstruction, specifically comprising the following steps: Step 1: Extract the log-mel spectrogram and sound intensity vector from the original audio, and concatenate them in the channel dimension. Finally, get the preliminary features through batch normalization.

[0007] Step 2: Input the preliminary features extracted in step 1 into the channel-space squeezing feature extraction module to extract important information from the preliminary features and strengthen local features.

[0008] Step 3: Input the feature map obtained in step 2 into the multi-scale pooling fusion module, align it with the time dimension of the label in the time dimension, and compress the feature information on the frequency axis through multi-scale fusion and pooling hybrid methods to obtain a fused feature map.

[0009] Step 4: Input the fused feature map obtained in step 3 into the Tranformer network module so that the network can effectively learn the temporal dependency information of the context sequence.

[0010] Step 5: Output the prediction results in the multi-ACCDOA format through the fully connected layer using the features obtained in step 4. Since multiple overlapping sound events of the same type may be predicted at the same time, the track arrangement problem of overlapping sound events of the same type may be encountered when calculating the loss function during training, that is, the predicted track is inconsistent with the label track. To solve this problem, all possible label tracks can be arranged and combined, and then the arrangement with the minimum loss can be found to calculate the loss function, that is, the ADPIT training method is used, and the mean square error loss function (MSE) is used to optimize the network parameters.

[0011] Step 6: Input the audio data to be located into the network trained in step 5 to obtain the categories and occurrence times of various sound events in the audio, as well as the directional coordinates of these sound events in space.

[0012] The present invention has the following beneficial effects: 1. The channel-space squeezing feature extraction module can effectively filter the noise and reverberation of the surrounding environment, extract the frequency band distribution characteristics corresponding to the sound event at the current moment, generate a feature map that emphasizes details, and ensure that sound events are accurately captured in the real environment and the direction and source of such sound events are identified.

[0013] 2. Since average pooling focuses on the information of the overall feature distribution, it can retain more time-frequency information and suppress noise and interference signals to a certain extent, while maximum pooling tends to capture key information in the overall features. Therefore, by mixing the results of average pooling and maximum pooling through the multi-scale pooling fusion module, the advantages of the two pooling operations can be fully utilized to enhance the feature representation capability and improve the robustness of the model. Because the pooling operation is placed after the channel-space compression feature extraction module and a multi-scale fusion operation is performed, the model can fully extract the frequency band distribution characteristics of the sound event from the preliminary features, ensuring that some important information in the audio is not lost. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the method for detecting and locating sound events; Figure 2 This is a schematic diagram of a feature extraction network in an embodiment; Figure 3 Schematic diagram of channel-space squeezing feature extraction block in the embodiment; Figure 4 Schematic diagram of a multi-scale pooling fusion module in an embodiment. DETAILED DESCRIPTION

[0015] The present invention will be further explained below with reference to the accompanying drawings; like Figure 1 As shown, a method for locating and detecting sound events applied in the field of three-dimensional scene reconstruction specifically includes the following steps: Step 1: Extract the log-mel spectrogram and sound intensity vector from the original audio, and concatenate them in the channel dimension. Finally, get the preliminary features through batch normalization. The order of the dimensions of the preliminary features is batch dimension, channel dimension, time dimension, and frequency dimension.

[0016] Step 2: Build Figure 2 The recognition network shown includes a channel-space squeezing feature extraction module, a multi-scale pooling fusion module, a Tranformer network module and a fully connected layer, which are used to further extract audio features and output positioning detection results.

[0017] Step 3: Figure 3As shown, the channel-space squeezing feature extraction module includes three cascaded residual attention modules, each of which includes a residual module and an attention mechanism.

[0018] The residual module performs 3×3 2D convolution, batch normalization, linear rectification, 1×1 2D convolution kernel and batch normalization on the input features in sequence, adds them to the input features, and then inputs them into the attention mechanism.

[0019] The attention mechanism calculates channel attention and spatial attention on the input features respectively, adds the two attentions together, multiplies them with the input features, and obtains the output of the residual attention module.

[0020] The channel attention includes cascaded 2D global average pooling, linear transformation layer and linear rectification, which are used to compress information in the channel dimension; the spatial attention includes swap dimension operation, cascaded 2D global average pooling, linear transformation layer and linear rectification, which are used to compress information in the time dimension.

[0021] Step 4: Figure 4 As shown, the multi-scale pooling fusion module receives the feature map output by the third attention mechanism and performs multi-scale pooling fusion. The multi-scale pooling fusion module includes three parallel branches. The input feature map is processed by the three parallel branches respectively and then added to obtain a fused feature map.

[0022] Each branch includes three cascaded pooling fusion modules with different pooling window sizes. In the first branch, the pooling window sizes of the three pooling fusion modules are 4×1, 4×1, and 2×5, respectively; in the second branch, the pooling window sizes of the three pooling fusion modules are 4×1, 2×1, and 4×5, respectively; in the third branch, the pooling window sizes of the three pooling fusion modules are 2×1, 4×1, and 4×5, respectively.

[0023] The pooling fusion module includes two branches, the first branch is a maximum pooling layer, a 3×3 2D convolution kernel, batch normalization and linear rectification, and the second branch is an average pooling layer, a 3×3 2D convolution kernel, batch normalization and linear rectification. The results of the two branches are added to obtain the output of a pooling fusion module.

[0024] Step 5: Input the fused feature map obtained by the multi-scale pooling fusion module into the Tranformer network module to complete the feature extraction of the audio.

[0025] Step 6: Output the output features obtained in step 5 through the fully connected layer to output the prediction results in the multi-ACCDOA format. During network training, the network is trained using the ADPIT training method, and the mean square error loss function is used to optimize the network; during network inference, the type recognition results of various sound events, the time interval of occurrence, and the direction coordinate annotation of the sound event in space are obtained. Post-process the predicted values, that is, set the length threshold of the predicted coordinate value to optimize the positioning and classification results, further filter out the erroneous estimation of sound event detection caused by the loss function during training, and complete the acoustic information analysis of the environment.

Claims

1. A method for locating and detecting sound events for three-dimensional scene reconstruction, characterized in that: The specific steps include: Step 1: Extract preliminary features from the original audio; Step 2: Input the preliminary features into the channel-space squeezing feature extraction module to obtain a feature map that enhances local features; Step 3: Input the feature map after local feature enhancement into the multi-scale pooling fusion module to compress the feature information on the frequency axis to obtain a fused feature map; Step 4: Input the fused feature map obtained in step 3 into the Tranformer network module to learn the temporal dependency information of the context sequence; then output the prediction result in the multi-ACCDOA format through the fully connected layer; Step 5: Use the ADPIT training method and the mean square error loss function to optimize the network parameters; Step 6: Input the audio data to be located into the network trained in step 5 to obtain the categories and occurrence times of various sound events in the audio, as well as the directional coordinates of these sound events in space.

2. A method for locating and detecting sound events for three-dimensional scene reconstruction as claimed in claim 1, characterized in that: Extract the log-mel spectrogram and sound intensity vector from the original audio, and concatenate the two in the channel dimension; finally, obtain preliminary features through batch normalization; The order of the dimensions of the preliminary features is batch dimension, channel dimension, time dimension, and frequency dimension.

3. A method for locating and detecting sound events for three-dimensional scene reconstruction as claimed in claim 1, characterized in that: The channel-space squeezing feature extraction module includes three cascaded residual attention modules, each of which includes a residual module and an attention mechanism; The attention mechanism calculates channel attention and spatial attention on the input features respectively, adds the two attentions, multiplies them with the input features, and obtains the output of the residual attention module; The channel attention includes cascaded 2D global average pooling, linear transformation layer and linear rectification, which are used to compress information in the channel dimension; the spatial attention includes swap dimension operation, cascaded 2D global average pooling, linear transformation layer and linear rectification, which are used to compress information in the time dimension.

4. A method for locating and detecting sound events for three-dimensional scene reconstruction as claimed in claim 3, characterized in that: The residual module performs 3×3 2D convolution, batch normalization, linear rectification, 1×1 2D convolution kernel and batch normalization on the input features in sequence, adds them to the input features, and then inputs them into the attention mechanism.

5. A method for locating and detecting sound events for three-dimensional scene reconstruction as claimed in claim 1, characterized in that: The multi-scale pooling fusion module includes three parallel branches. The input feature map is processed by the three parallel branches respectively and then added to obtain a fused feature map. Each branch includes three cascaded pooling fusion modules with different pooling window sizes.

6. A method for locating and detecting sound events for three-dimensional scene reconstruction as claimed in claim 5, characterized in that: In the first branch of the multi-scale pooling fusion module, the pooling window sizes of the three pooling fusion modules are 4×1, 4×1, and 2×5, respectively. In the second branch, the pooling window sizes of the three pooling fusion modules are 4×1, 2×1, and 4×5, respectively. In the third branch, the pooling window sizes of the three pooling fusion modules are 2×1, 4×1, and 4×5, respectively.

7. A method for locating and detecting sound events for three-dimensional scene reconstruction as claimed in any one of claims 5 or 6, characterized in that: The pooling fusion module includes two branches, the first branch is a maximum pooling layer, a 3×3 2D convolution kernel, batch normalization and linear rectification, and the second branch is an average pooling layer, a 3×3 2D convolution kernel, batch normalization and linear rectification. The results of the two branches are added to obtain the output of a pooling fusion module.

Images