A multi-sound event detection and positioning method and device based on a neural network model

Abstract

This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for detecting and locating multiple sound events based on a neural network model. The method includes: innovatively designing time-frequency multi-scale residual convolutional blocks, which, together with a Conformer module and a cross-stitch unit module, form a network model to extract features at multiple scales, enhance long sequence modeling, and promote task-based collaborative optimization, thereby improving performance and accuracy; in terms of data processing, pre-emphasis and frame-by-frame windowing improve feature quality, audio channel swapping and spectrum enhancement increase data diversity and reduce overfitting, and SALSA-Lite features are used to enhance feature representation; in terms of training strategy, a multivariate loss function is used to accelerate convergence while considering task requirements, and hyperparameters are flexibly adjusted using a validation set. This makes the method highly efficient in training, has excellent practical performance, strong generalization ability on unknown data, and can accurately cope with complex and ever-changing real-world scenarios, effectively overcoming the shortcomings of traditional methods.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for detecting and locating multiple sound events based on a neural network model. Background Technology

[0002] For smart-related industries, Sound Event Localization and Detection (SELD) technology is one of the core technologies. Whether it's achieving accurate speech recognition and sound source tracking in intelligent conferencing systems, or in smart homes and smart security, effective detection and accurate localization of sound events are indispensable. Sound source localization estimation methods can be divided into parametric methods and deep neural network-based methods. However, traditional parametric methods, such as beamforming, differ in terms of algorithm complexity, array geometric constraints, and acoustic scene modeling assumptions, and suffer from drawbacks such as high computational complexity, high signal-to-noise ratio requirements, and poor real-time performance. Deep neural network-based methods, on the other hand, are more robust to reverberation and low signal-to-noise ratio scenarios. With their powerful feature learning capabilities, they can automatically extract more representative features from the original audio, accurately capturing the essence of the sound and improving detection and localization accuracy. Furthermore, trained on a large amount of data, they can handle complex scenarios such as multiple sound sources and noise interference.

[0003] To solve the problem of detection and localization of multiple sound sources, researchers have proposed a series of methods based on deep neural networks. Among them, convolutional neural networks are used to process sound features, and recurrent convolutional networks are used to capture temporal features. Reference [1] proposes a common learning method for the joint localization and detection of multiple overlapping sound events. First, the method takes multi-channel audio as input and extracts phase and amplitude spectrograms from each audio channel as features. Second, it uses a common convolutional recurrent neural network to map the feature sequence in parallel into two outputs. One is to use sound event detection as a multi-label classification task to determine the category of sound events in each frame; the other is to estimate the three-dimensional Cartesian coordinates of the sound event through a multi-output regression task, thereby achieving localization. Finally, the network output is thresholded to obtain the final result. Reference [2] proposes the ACCDOA method, which uses a single loss function to avoid the problem of balancing objectives. First, the activity state of the sound event is associated with the length of the Cartesian DOA vector to represent the sound event. The activity state is reflected by the vector length, and DOA is represented by the vector direction. Second, a CRNN architecture is used as an embedding network to extract the input audio features, and then a fully connected layer is used to estimate the ACCDOA vector. Finally, mean squared error is used as the loss function, and only the loss of the activity is calculated when there is no event. However, in reference [1], all tasks share the same set of parameters. When the optimal settings of parameters for different tasks are different, it is difficult to meet the needs of all tasks at the same time, resulting in a decrease in performance. Reference [2] trains only based on location information and uses the ACCDOA vector magnitude as the SED activation. Due to the neglect of the key features of sound events, the accuracy in identifying event types will decrease significantly, and it will be difficult to distinguish between different events that occur at similar locations or at the same location in complex scenes, affecting the accuracy of localization and detection. Therefore, how to construct a neural network algorithm to solve the detection and localization of multiple sound events is one of the problems that technicians in this field need to solve. Summary of the Invention

[0004] To address the technical problem of how to construct a neural network algorithm for detecting and locating multiple sound events in existing technologies, this invention provides a method and apparatus for detecting and locating multiple sound events based on a neural network model. The technical solution is as follows:

[0005] On the one hand, a method for detecting and locating multiple sound events based on a neural network model is provided, characterized in that the method includes:

[0006] S1. Construct an audio dataset and split it into a training set, a validation set, and a test set; preprocess the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset;

[0007] S2. Construct a multi-source detection and localization neural network model. Input the SALSA-Lite features of the training set into the multi-source detection and localization neural network model to train sound detection and orientation estimation.

[0008] S3. Train a multi-source detection and localization neural network model using the binary cross-entropy loss function, the mean squared error loss function, and the joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean squared error loss function.

[0009] S4. Use the trained multi-source detection and localization neural network model to predict the SALSA-Lite features of the validation set to obtain sound detection results and orientation estimation results; calculate the evaluation index on the validation set to evaluate the model's performance on the validation set; adjust the model's hyperparameters based on the evaluation results of the validation set to obtain the optimized multi-source detection and localization neural network model.

[0010] S5. Using the optimized multi-source detection and localization neural network model, predict the SALSA-Lite features of the test set to obtain the final sound detection results and location estimation results.

[0011] Optionally, the audio data in the audio dataset is preprocessed to obtain SALSA-Lite features of the audio data in the audio dataset, including:

[0012] The audio data in the training set is expanded by exchanging audio channels.

[0013] The audio data in the audio dataset is pre-emphasized and pre-processed by frame segmentation and windowing to generate SALSA-Lite features.

[0014] Optionally, pre-emphasis is applied to all audio data in the dataset, including:

[0015]

[0016] in, It is the signal after pre-emphasis. It is the original input signal. It is the input signal value from the previous moment. It is the pre-weighting coefficient.

[0017] Optionally, the multi-source detection and localization neural network model includes:

[0018] The design includes a time-frequency multiscale residual convolutional block, a Conformer module, and a cross-stitch unit module for mutual learning between detection and localization tasks in the multi-source detection and localization design.

[0019] The time-frequency multi-scale residual convolution includes a multi-scale convolutional layer that combines time and frequency axis convolution, a ReLU activation function, a regular convolutional layer, and a batch normalization layer. It also includes a downsampling module to handle cases where the number of input and output channels or the stride is inconsistent. Finally, the output is added to the residual branch and output through the ReLU activation function.

[0020] Optionally, during training, data augmentation can be performed on the input SALSA-Lite features of the training set using temporal and frequency masking.

[0021] Optionally, a multi-source detection and localization neural network model is trained using a binary cross-entropy loss function, a mean squared error loss function, and a joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean squared error loss function, including:

[0022] Define the joint loss function:

[0023]

[0024] in, This represents the binary cross-entropy loss function used for sound event detection. This represents the mean squared error loss function used for sound source location estimation. For weights.

[0025] On the other hand, a multi-sound event detection and localization device based on a neural network model is provided. This device is applied to a multi-sound event detection and localization method based on a neural network model. The device includes:

[0026] The feature extraction module is used to construct the audio dataset and split it into training, validation and test sets; it preprocesses the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset;

[0027] The training module is used to build a multi-source detection and localization neural network model. The SALSA-Lite features of the training set are input into the multi-source detection and localization neural network model to train sound detection and orientation estimation.

[0028] The weighting module is used to train a multi-source detection and localization neural network model using the binary cross-entropy loss function, the mean squared error loss function, and the joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean squared error loss function.

[0029] The model tuning module is used to predict the SALSA-Lite features of the validation set using the trained multi-source detection and localization neural network model to obtain sound detection results and orientation estimation results; calculate the evaluation index on the validation set to evaluate the model's performance on the validation set; and adjust the model's hyperparameters based on the evaluation results of the validation set to obtain the tuned multi-source detection and localization neural network model.

[0030] The prediction module is used to predict the SALSA-Lite features of the test set using an optimized multi-source detection and localization neural network model, so as to obtain the final sound detection results and orientation estimation results.

[0031] Optionally, the feature extraction module is used to expand the training set by exchanging audio channels on the audio data of the training set.

[0032] The audio data in the audio dataset is pre-emphasized and pre-processed by frame segmentation and windowing to generate SALSA-Lite features.

[0033] On the other hand, a multi-sound event detection and localization device based on a neural network model is provided. The multi-sound event detection and localization device based on a neural network model includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, they implement any of the methods described above for multi-sound event detection and localization based on a neural network model.

[0034] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for multi-sound event detection and localization based on a neural network model.

[0035] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0036] This invention proposes a multi-sound event detection and localization method based on a neural network model. Compared with existing multi-sound event detection and localization methods, this method has significant advantages. In terms of model structure, an innovative time-frequency (TF) multi-scale residual convolutional block is designed, which, together with the Conformer module and the cross-stitch unit module, forms the network model. Multi-scale feature extraction, enhanced long sequence modeling, and promotion of task collaborative optimization improve performance and accuracy. In terms of data processing, pre-emphasis and frame-by-frame windowing improve feature quality, audio channel swapping and spectrum enhancement increase data diversity and reduce overfitting, and SALSA-Lite features are used to enhance feature representation. In terms of training strategy, a multivariate loss function is used to accelerate convergence while considering task requirements, and hyperparameters are flexibly adjusted using a validation set. This makes the method highly efficient in training, has excellent practical performance, strong generalization ability on unknown data, and can accurately cope with complex and ever-changing real-world scenarios, effectively overcoming the shortcomings of traditional methods. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating a multi-sound event detection and localization method based on a neural network model provided in an embodiment of the present invention;

[0039] Figure 2 This is a model structure diagram of the multi-sound event detection and localization method based on a neural network model provided in an embodiment of the present invention;

[0040] Figure 3 This is a structural diagram of a time-frequency multi-scale residual convolution module provided in an embodiment of the present invention;

[0041] Figure 4 A block diagram of a multi-sound event detection and localization device based on a neural network model provided in an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0044] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0045] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0046] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0047] This invention provides a method for detecting and locating multiple sound events based on a neural network model. This method can be implemented by a device for detecting and locating multiple sound events based on a neural network model, which can be a terminal or a server. Figure 1 The flowchart shown is for a multi-sound event detection and localization method based on a neural network model. Figure 1 As shown, the multi-sound event detection and localization method based on a neural network model proposed in this invention may include the following steps in its processing flow:

[0048] S1. Construct an audio dataset and split it into a training set, a validation set, and a test set; preprocess the audio data in the audio dataset to obtain lightweight Spatial Cue-Augmented Log-Spectrogram (SALSA-Lite) features.

[0049] In one feasible implementation, according to Figure 1An audio dataset was constructed and split into a training set, a validation set, and a test set. In this embodiment, the MIC format audio data and labels from the TAU-NIGENS Spatial Sound Events 2021 dataset provided by the DCASE2021 Task3 competition were selected as the dataset for this embodiment [Politis, Archontis, Sharath Adavanne, and Tuomas Virtanen. "TAU-NIGENS Spatial Sound Events 2021". Zenodo, February 28, 2021.]. The training set contains 400 one-minute audio tracks, and the validation and test sets each contain 100 one-minute audio tracks.

[0050] In one feasible implementation, the audio data in the audio dataset is preprocessed to obtain the SALSA-Lite features of the audio data in the audio dataset, including:

[0051] The audio data in the training set is expanded by exchanging audio channels.

[0052] The audio data in the audio dataset is pre-emphasized and pre-processed by frame segmentation and windowing to generate SALSA-Lite features.

[0053] In one feasible implementation, the training set is expanded by 8 times based on the eight channel swapping modes and their corresponding swapped labels provided by Audio Channel Swapping (ACS) as given in the literature [Qing Wang, Jun Du, Hua-Xin Wu, Jia Pan, Feng Ma, Chin-Hui Lee. “A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection”. IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1251-1264, 2023.].

[0054] One feasible implementation involves using pre-emphasis on all audio data in the dataset, including:

[0055]

[0056] in, It is the signal after pre-emphasis. It is the original input signal. It is the input signal value from the previous moment. It is the pre-weighting coefficient.

[0057] In one feasible implementation, when performing SALSA-Lite audio feature extraction according to the method described in the literature [Thi Ngoc Tho Nguyen, Douglas L. Jones, Karn N. Watcharasupat, Huy Phan, Woon-Seng Gan. “SALSA-Lite: A Fastand Effective Feature for Polyphonic Sound Event Localization and Detection with Microphone Arrays”. ICASSP 2022 - IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 716-720, 2022.], the audio sampling rate is set to 24kHz, and a 1024-point FFT and Hanning window are used for short-time Fourier transform, with a frame shift of 300°. The Hanning window formula is as follows:

[0058]

[0059] in, It is the value of the window function. It is the length of the window. It is the index of the current sampling point.

[0060] The SALSA-Lite feature includes time-frequency representation and spatial information, with a total of 7 channels, and the frequency range is set from 50Hz to 9000Hz.

[0061] S2. Construct a multi-source detection and localization neural network model. Input the SALSA-Lite features of the training set into the multi-source detection and localization neural network model to train sound detection and orientation estimation.

[0062] In one feasible implementation, the multi-sound source detection and localization neural network model includes:

[0063] The design for multi-source detection and localization includes a time-frequency multi-scale residual convolutional block, a Conformer module, and a cross-stitch unit module for mutual learning between detection and localization tasks [Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, Martial Hebert. “Cross-Stitch Networks for Multi-Task Learning”. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994-4003, 2016.]. The time-frequency multi-scale residual convolutional block comprises a multi-scale convolutional layer combining time and frequency axis convolutions, a ReLU activation function, a regular convolutional layer, and a batch normalization layer. A downsampling module is also included to handle cases where the number of input and output channels or the stride is inconsistent. Finally, the output is added to the residual branch and output through the ReLU activation function.

[0064] In one feasible implementation, during training, the input SALSA-Lite features of the training set are augmented using temporal and frequency masking.

[0065] One feasible implementation method is, for example Figure 2 As shown, the SALSA-Lite features from the training set are input into the multi-source detection and localization neural network model for training sound detection and location estimation. Figure 3 As shown, this is a temporal-frequency multi-scale residual convolutional block structure designed for sound event detection and localization tasks. The TF multi-scale convolutional module first uses 3x1 temporal convolutions and 1x3 frequency convolutions to extract features from the input along the time and frequency axes, respectively, capturing time and frequency patterns. The obtained features are then concatenated, processed by a batch normalization layer, and finally fused using a 1x1 convolution. The TF multi-scale convolutional module replaces traditional convolutional layers to extract multi-scale features. After batch normalization and ReLU activation, it undergoes further convolution and batch normalization. Simultaneously, if the number of input and output channels or the stride are inconsistent, downsampling is performed. Finally, the output of the main branch and the output of the residual branch are added and activated by ReLU to obtain the final result, thereby improving the model's feature extraction capability and training stability.

[0066] In one feasible implementation, the TF multi-scale residual convolutional block extracts the temporal dynamics and frequency distribution characteristics of sound features through 3×1 and 1×3 convolutions, respectively. Multi-scale information fusion is achieved through channel concatenation and 1×1 convolution, and residual connections are used to alleviate the gradient vanishing problem. Compared to the single convolution of traditional residual blocks, the first module directly extracts multi-scale time-frequency features, significantly enhancing the utilization of spatial cues in SALSA-Lite features and providing richer time-frequency feature representations for sound event detection and localization. Conventional two-dimensional convolutional layers further extract abstract features based on multi-scale features to uncover potential sound patterns. The Conformer unit integrates CNN local feature extraction and Transformer self-attention mechanism to capture long-short-term dependencies and improve temporal modeling capabilities. The soft parameter sharing mechanism of the cross-stitch unit solves the information competition problem caused by hard parameter sharing in multi-task learning, realizing dynamic feature fusion between SED and DOA branches, balancing task specificity and information sharing requirements.

[0067] S3. Train a multi-source detection and localization neural network model using the binary cross-entropy loss function, the mean squared error loss function, and the joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean squared error loss function.

[0068] In one feasible implementation, a multi-source detection and localization neural network model is trained using a binary cross-entropy loss function, a mean squared error loss function, and a joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean squared error loss function, including:

[0069] Define the joint loss function:

[0070]

[0071] in, This represents the binary cross-entropy loss function used for sound event detection. This represents the mean squared error loss function used for sound source location estimation. For weights.

[0072] S4. Use the trained multi-source detection and localization neural network model to predict the SALSA-Lite features of the validation set to obtain sound detection results and orientation estimation results; calculate the evaluation index on the validation set to evaluate the model's performance on the validation set; adjust the model's hyperparameters (such as learning rate, number of iterations, etc.) based on the evaluation results of the validation set to obtain the optimized multi-source detection and localization neural network model.

[0073] In one feasible implementation, the trained multi-source detection and localization neural network model predicts the SALSA-Lite features of the validation set, and outputs the sound event category probability and three-dimensional coordinate localization results through the SED branch and DOA branch of the model, respectively. After sigmoid and tanh activation, these results are converted into the final sound detection results and orientation estimation results.

[0074] The evaluation system adopted is the "macro-average" approach from DCASE 2021 Task 3, with specific metrics including: SED-related error rate (ER). ≤20° ), calculate the proportion of incorrectly predicted events in all categories to the total number of events, and consider it valid only if the predicted category is correct and the DOA bias is ≤20°, ER ≤20° The lower the value, the fewer the detection errors; SED-related F score (F ≤20° The higher the overall detection accuracy and recall rate (F≤20°), the better the detection precision and completeness; the positioning error (LE) CD ): Calculate the mean angular error between the predicted DOA and the true value across all categories, LE CD The lower the retrieval rate, the higher the positioning accuracy; the positioning recall rate (LR) CD ): Calculate the ratio of correctly located events to actual events, LR CD The higher the value, the better the positioning integrity. SELDerror is the overall error that combines SED and DOA; the lower the value, the better the performance. The formula is:

[0075]

[0076] Based on the validation set evaluation results, the learning rate was set to 0.0003, the batch size to 32, the loss function weight to 0.5, both tasks were considered equally important, and the maximum number of training epochs was set to 100. Training was terminated early when the validation set SELDerror no longer decreased for 10 consecutive epochs to avoid overfitting. The model with the best overall performance (SELDerror) on the test set during training was saved. The optimized model was then tested on the validation set to achieve ER. ≤20° =0.454, F ≤20° =64.4%, LE CD =12.275°, LR CD =72.7%, SELDerror=0.288.

[0077] S5. Using the optimized multi-source detection and localization neural network model, predict the SALSA-Lite features of the test set to obtain the final sound detection results and location estimation results.

[0078] In one feasible implementation, the optimized multi-source detection and localization neural network model described above is used to predict the SALSA-Lite features of the test set. ER is then implemented on the test set. ≤20° =0.471, F ≤20° =63.2%, LE CD =11.979°, LR CD =72.1%, SELDerror=0.296. This demonstrates a certain degree of generalization ability.

[0079] The results of different methods on the test set were compared using the same experimental setup and conditions, as shown in Table 1.

[0080] Table 1. Comparison of results of different methods on the test set under the same experimental setup and conditions.

[0081]

[0082] This method was compared with existing algorithms for sound event detection and direction of arrival estimation tasks, respectively. Experimental results show that, thanks to the innovative design of multi-scale residual blocks, the proposed method exhibits strong robustness in the face of unknown data and complex acoustic environments. Its multi-scale feature fusion mechanism overcomes the shortcomings of existing technologies in single feature extraction, accelerates model convergence through optimized structure, and demonstrates good performance.

[0083] This invention proposes a multi-sound event detection and localization method based on a neural network model. Compared with existing multi-sound event detection and localization methods, this method has significant advantages. In terms of model structure, an innovative time-frequency multi-scale residual convolutional block is designed, which, together with the Conformer module and the cross-stitch unit module, forms a network model. This allows for multi-scale feature extraction, enhanced long sequence modeling, and improved task-based collaborative optimization, thereby improving performance and accuracy. Regarding data processing, pre-emphasis and frame-by-frame windowing improve feature quality, audio channel swapping and spectrum enhancement increase data diversity and reduce overfitting, and SALSA-Lite features are used to enhance feature representation. In terms of training strategy, a multivariate loss function is used to accelerate convergence while considering task requirements, and hyperparameters are flexibly adjusted using a validation set. This results in high training efficiency, excellent practical performance, strong generalization ability on unknown data, and the ability to accurately handle complex and ever-changing real-world scenarios, effectively overcoming the shortcomings of traditional methods.

[0084] Figure 4 This is a block diagram of a multi-sound event detection and localization device 300 based on a neural network model, according to an exemplary embodiment. The device 300 is used in a multi-sound event detection and localization method based on a neural network model. (Refer to...) Figure 4 The device includes a feature extraction module 310, a training module 320, a weighting module 330, a model tuning module 340, and a prediction module 350. Wherein:

[0085] The feature extraction module 310 is used to construct an audio dataset and split the audio dataset into a training set, a validation set, and a test set; it preprocesses the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset;

[0086] Training module 320 is used to build a multi-source detection and localization neural network model. The SALSA-Lite features of the training set are input into the multi-source detection and localization neural network model to train sound detection and orientation estimation.

[0087] The weighting module 330 is used to train a multi-source detection and localization neural network model using the binary cross-entropy loss function, the mean square error loss function, and the joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean square error loss function.

[0088] The model tuning module 340 is used to predict the SALSA-Lite features of the validation set using the trained multi-source detection and localization neural network model to obtain sound detection results and orientation estimation results; calculate the evaluation index on the validation set to evaluate the performance of the model on the validation set; and adjust the hyperparameters of the model according to the evaluation results of the validation set to obtain the tuned multi-source detection and localization neural network model.

[0089] The prediction module 350 is used to predict the SALSA-Lite features of the test set through the optimized multi-source detection and localization neural network model, so as to obtain the final sound detection results and orientation estimation results.

[0090] Optionally, the feature extraction module 310 is used to expand the training set by exchanging audio channels on the audio data of the training set.

[0091] The audio data in the audio dataset is pre-emphasized and pre-processed by frame segmentation and windowing to generate SALSA-Lite features.

[0092] Optionally, pre-emphasis is applied to all audio data in the dataset, including:

[0093]

[0094] in, It is the signal after pre-emphasis. It is the original input signal. It is the input signal value from the previous moment. It is the pre-weighting coefficient.

[0095] Optionally, the multi-source detection and localization neural network model includes:

[0096] The design includes a time-frequency multiscale residual convolutional block, a Conformer module, and a cross-stitch unit module for mutual learning between detection and localization tasks in the multi-source detection and localization design.

[0097] The time-frequency multi-scale residual convolution includes a multi-scale convolutional layer that combines time and frequency axis convolution, a ReLU activation function, a regular convolutional layer, and a batch normalization layer. It also includes a downsampling module to handle cases where the number of input and output channels or the stride is inconsistent. Finally, the output is added to the residual branch and output through the ReLU activation function.

[0098] Optionally, during training, data augmentation can be performed on the input SALSA-Lite features of the training set using temporal and frequency masking.

[0099] Optionally, a multi-source detection and localization neural network model is trained using a binary cross-entropy loss function, a mean squared error loss function, and a joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean squared error loss function, including:

[0100] Define the joint loss function:

[0101]

[0102] in, This represents the binary cross-entropy loss function used for sound event detection. This represents the mean squared error loss function used for sound source location estimation. For weights.

[0103] In this embodiment of the invention, compared with existing multi-sound event detection and localization methods, the device has significant advantages. In terms of model structure, an innovative time-frequency multi-scale residual convolutional block is designed, forming a network model with the Conformer module and the cross-stitch unit module. This allows for multi-scale feature extraction, enhanced long sequence modeling, and improved task collaborative optimization, thus improving performance and accuracy. Regarding data processing, pre-emphasis and frame-by-frame windowing improve feature quality, audio channel swapping and spectrum enhancement increase data diversity and reduce overfitting, and SALSA-Lite features are used to enhance feature representation. In terms of training strategy, a multivariate loss function is used to accelerate convergence while considering task requirements, and hyperparameters are flexibly adjusted using a validation set. This makes the method highly efficient in training, has excellent practical performance, strong generalization ability on unknown data, and can accurately cope with complex and ever-changing real-world scenarios, effectively overcoming the shortcomings of traditional methods.

[0104] Figure 5 This is a schematic diagram of the structure of a multi-sound event detection and localization device based on a neural network model provided in an embodiment of the present invention, as shown below. Figure 5As shown, a multi-sound event detection and localization device based on a neural network model may include the above-mentioned... Figure 4 The illustrated multi-sound event detection and localization device is based on a neural network model. Optionally, the multi-sound event detection and localization device 410 based on a neural network model may include a first processor 2001.

[0105] Optionally, the multi-sound event detection and localization device 410 based on a neural network model may also include a memory 2002 and a transceiver 2003.

[0106] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0107] The following is combined Figure 5 A detailed description of each component of the multi-sound event detection and localization device 410 based on a neural network model is provided below:

[0108] The first processor 2001 is the control center of the multi-sound event detection and localization device 410 based on a neural network model. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0109] Optionally, the first processor 2001 can execute various functions of the multi-sound event detection and positioning device 410 based on a neural network model by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0110] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 5 CPU0 and CPU1 are shown in the diagram.

[0111] In a specific implementation, as one example, the multi-sound event detection and localization device 410 based on a neural network model may also include multiple processors, for example... Figure 5The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0112] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0113] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the multi-sound event detection and positioning device 410 based on a neural network model. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0114] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0115] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 5 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0116] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the multi-sound event detection and positioning device 410 based on a neural network model. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0117] It should be noted that, Figure 5 The structure of the multi-sound event detection and localization device 410 based on the neural network model shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0118] Furthermore, the technical effects of the multi-sound event detection and localization device 410 based on the neural network model can be referred to the technical effects of the multi-sound event detection and localization method based on the neural network model described in the above method embodiments, and will not be repeated here.

[0119] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0120] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0121] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensors. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0122] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0123] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0124] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0127] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0128] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting and locating multiple sound events based on a neural network model, characterized in that, The method includes: S1. Construct an audio dataset and split it into a training set, a validation set, and a test set; preprocess the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset; S2. Construct a multi-source detection and localization neural network model. Input the SALSA-Lite features of the training set into the multi-source detection and localization neural network model to train sound detection and orientation estimation. Multi-source sound detection and localization neural network models include: The design includes a time-frequency multiscale residual convolutional block, a Conformer module, and a cross-stitch unit module for mutual learning between detection and localization tasks in the multi-source detection and localization design. The time-frequency multi-scale residual convolution includes a multi-scale convolutional layer that combines time and frequency axis convolution, a ReLU activation function, a regular convolutional layer, and a batch normalization layer. It also includes a downsampling module to handle cases where the number of input and output channels or the stride is inconsistent. Finally, the output is added to the residual branch and output through the ReLU activation function. S3. Train a multi-source detection and localization neural network model using the binary cross-entropy loss function, the mean squared error loss function, and the joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean squared error loss function. S4. Use the trained multi-source detection and localization neural network model to predict the SALSA-Lite features of the validation set to obtain sound detection results and orientation estimation results; calculate the evaluation index on the validation set to evaluate the model's performance on the validation set; adjust the model's hyperparameters based on the evaluation results of the validation set to obtain the optimized multi-source detection and localization neural network model. S5. Using the optimized multi-source detection and localization neural network model, predict the SALSA-Lite features of the test set to obtain the final sound detection results and location estimation results.

2. The method according to claim 1, characterized in that, The audio data in the audio dataset is preprocessed to obtain SALSA-Lite features, including: The audio data in the training set is expanded by exchanging audio channels. The audio data in the audio dataset is pre-emphasized and pre-processed by frame segmentation and windowing to generate SALSA-Lite features.

3. The method according to claim 2, characterized in that, Pre-emphasis was applied to all audio data in the dataset, including: ； in, It is the signal after pre-emphasis. It is the original input signal. It is the input signal value from the previous moment. It is the pre-weighting coefficient.

4. The method according to claim 3, characterized in that, During training, temporal and frequency masking are used to augment the input SALSA-Lite features of the training set.

5. The method according to claim 4, characterized in that, A multi-source detection and localization neural network model is trained using a binary cross-entropy loss function, a mean squared error loss function, and a joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean squared error loss function, including: Define the joint loss function: ； in, This represents the binary cross-entropy loss function used for sound event detection. This represents the mean squared error loss function used for sound source location estimation. For weights.

6. A multi-sound event detection and localization device based on a neural network model, wherein the multi-sound event detection and localization device based on a neural network model is used to implement the multi-sound event detection and localization method based on a neural network model as described in any one of claims 1-5, characterized in that, The device includes: The feature extraction module is used to construct the audio dataset and split it into training, validation and test sets; it preprocesses the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset; The training module is used to build a multi-source detection and localization neural network model. The SALSA-Lite features of the training set are input into the multi-source detection and localization neural network model to train sound detection and orientation estimation. The weighting module is used to train a multi-source detection and localization neural network model using the binary cross-entropy loss function, the mean squared error loss function, and the joint loss function. The joint loss function is a weighted combination of the binary cross-entropy loss function and the mean squared error loss function. The model tuning module is used to predict the SALSA-Lite features of the validation set using the trained multi-source detection and localization neural network model to obtain sound detection results and orientation estimation results; calculate the evaluation index on the validation set to evaluate the model's performance on the validation set; and adjust the model's hyperparameters based on the evaluation results of the validation set to obtain the tuned multi-source detection and localization neural network model. The prediction module is used to predict the SALSA-Lite features of the test set using an optimized multi-source detection and localization neural network model, so as to obtain the final sound detection results and orientation estimation results.

7. The apparatus according to claim 6, characterized in that, The feature extraction module is used to expand the training set by exchanging audio channels on the audio data of the training set. The audio data in the audio dataset is pre-emphasized and pre-processed by frame segmentation and windowing to generate SALSA-Lite features.

8. A multi-sound event detection and localization device based on a neural network model, the multi-sound event detection and localization device based on a neural network model comprising: processor; A memory storing computer-readable instructions, which, when executed by the processor, implement any one of the methods in the multi-sound event detection and localization method based on a neural network model as described in any one of claims 1-5.

9. A computer-readable storage medium storing at least one instruction, the at least one instruction being loaded and executed by a processor to implement any one of the methods in the multi-sound event detection and localization method based on a neural network model as described in any one of claims 1-5.

Images