Multi-sound event detection positioning method and device based on neural network model

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a multi-sound event detection positioning method and device based on a neural network model. The method comprises the following steps: innovatively designing a time-frequency multi-scale residual error convolution block, forming a network model with a Conformer module and a cross-stitch unit module, extracting features in a multi-scale manner, enhancing long sequence modeling, promoting task collaborative optimization, and improving performance and accuracy; in the aspect of data processing, pre-emphasis and framing windowing improve feature quality, audio channel exchange and spectrum enhancement increase data diversity and reduce overfitting, and SALSA-Lite features are adopted to enhance feature expression. In the training strategy, a multivariate loss function is applied to accelerate convergence in consideration of task requirements, and hyper-parameters are flexibly adjusted by means of a verification set. Therefore, the method is high in training efficiency, excellent in actual performance, high in generalization ability on unknown data, capable of accurately coping with complex and changeable actual scenes and capable of effectively overcoming the defects of a traditional method.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for detecting and locating multiple sound events based on a neural network model. Background Art

[0002] Sound Event Localization and Detection (SELD) technology is a core technology for the intelligent industry. Whether it's intelligent conferencing systems that achieve accurate speech recognition and sound source tracking, or smart homes and intelligent security, effective detection and precise location of sound events are essential. Sound source location estimation methods can be categorized into parametric approaches and deep neural network-based approaches. However, traditional parametric methods, such as beamforming, differ in terms of algorithmic complexity, array geometry constraints, and acoustic scene modeling assumptions. They also suffer from high computational complexity, high signal-to-noise ratio requirements, and poor real-time performance. Deep neural network-based approaches, on the other hand, are more robust to reverberant and low signal-to-noise ratio scenarios. Leveraging powerful feature learning capabilities, they automatically extract more representative features from raw audio, accurately capturing the essence of sound and improving detection and location accuracy. Furthermore, through extensive data training, they can handle complex scenarios involving multiple sound sources and noise interference.

[0003] In order to solve the problem of detecting and locating multiple sound sources, researchers have proposed a series of methods based on deep neural networks, among which convolutional neural networks are used to process sound features, and recurrent convolutional networks are used to capture the combination of temporal features. Reference [1] proposes a joint learning method for the joint localization and detection of multiple overlapping sound events. First, the method takes multi-channel audio as input and extracts phase and amplitude spectra from each audio channel as features. Second, a common convolutional recurrent neural network is used to map the feature sequence into two outputs in parallel. One is to treat sound event detection as a multi-label classification task to determine the category of the sound event in each frame; the other is to estimate the three-dimensional Cartesian coordinates of the sound event through a multi-output regression task to achieve localization. Finally, the network output is thresholded to obtain the final result. The ACCDOA method proposed in reference [2] uses a single loss function to avoid the problem of balancing the target. First, the activity state of the sound event is associated with the length of the Cartesian DOA vector to represent the sound event. The activity state is reflected by the vector length, and the DOA is represented by the vector direction. Second, the CRNN architecture is used as the embedding network to extract the input audio features, and then a fully connected layer is used to estimate the ACCDOA vector. Finally, the mean square error is used as the loss function, and only the loss of activity is calculated when there is no event. However, in the literature [1], all tasks share a set of parameters. When the optimal settings of the parameters for different tasks vary greatly, it is difficult to meet the requirements of all tasks at the same time, resulting in performance degradation. The literature [2] only trains based on position information and uses the ACCDOA vector amplitude as SED activation. This will ignore the key features of the sound event, resulting in a significant decrease in accuracy when identifying the event type. It is also difficult to distinguish different events that appear in similar locations or at the same location in complex scenes, affecting positioning and detection accuracy. Therefore, how to construct a neural network algorithm to solve the problem of multiple sound event detection and positioning is one of the problems that technicians in this field urgently need to solve. Summary of the Invention

[0004] In order to solve the technical problem of how to construct a neural network algorithm for detecting and locating multiple sound events in the prior art, the embodiments of the present invention provide a method and device for detecting and locating multiple sound events based on a neural network model. The technical solution is as follows: On the one hand, a method for detecting and locating multiple sound events based on a neural network model is provided, characterized in that the method includes: S1. Construct an audio dataset and split the audio dataset into a training set, a validation set, and a test set; preprocess the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset; S2. Build a multi-sound source detection and localization neural network model, input the SALSA-Lite features of the training set into the multi-sound source detection and localization neural network model, and perform sound detection and direction estimation training; S3. Training a multi-sound source detection and localization neural network model through a binary cross entropy loss function, a mean square error loss function, and a joint loss function, where the joint loss function is a weighted combination of the binary cross entropy loss function and the mean square error loss function; S4. Use the trained multi-sound source detection and localization neural network model to predict the SALSA-Lite features of the validation set to obtain sound detection results and direction estimation results; calculate evaluation indicators on the validation set to evaluate the performance of the model on the validation set; based on the evaluation results of the validation set, adjust the model's hyperparameters to obtain a tuned multi-sound source detection and localization neural network model; S5. The SALSA-Lite features of the test set are predicted through the tuned multi-sound source detection and localization neural network model to obtain the final sound detection results and direction estimation results.

[0005] Optionally, preprocessing the audio data in the audio dataset to obtain SALSA-Lite features of the audio data in the audio dataset includes: Use audio channel swapping on the audio data of the training set to expand the training set; The audio data in the audio dataset is pre-processed by pre-emphasis, frame division and windowing to generate SALSA-Lite features.

[0006] Optionally, use pre-emphasis on all audio data in the dataset, including:

[0007] in, is the pre-emphasized signal, is the original input signal, is the input signal value at the previous moment, is the pre-emphasis coefficient.

[0008] Optionally, the multi-sound source detection and localization neural network model includes: The time-frequency multi-scale residual convolution block and Conformer module designed for multi-sound source detection and localization, as well as the cross-stitch unit module for mutual learning between detection and localization tasks; Among them, the time-frequency multi-scale residual convolution includes a multi-scale convolution layer that combines the time axis and frequency axis convolution, a ReLU activation function, a normal convolution layer and a batch normalization layer. At the same time, a downsampling module is provided to handle the situation where the number of input and output channels or the step size are inconsistent. Finally, the output is added to the residual branch and output through the ReLU activation function.

[0009] Optionally, during training, temporal masking and frequency masking are used to perform data augmentation on the input training set SALSA-Lite features.

[0010] Optionally, the multi-sound source detection and localization neural network model is trained by a binary cross entropy loss function, a mean square error loss function, and a joint loss function, where the joint loss function performs a weighted combination of the binary cross entropy loss function and the mean square error loss function, including:

[0011] Define the joint loss function:

[0012] in, represents the binary cross entropy loss function used for sound event detection, represents the mean square error loss function used for sound source direction estimation, is the weight.

[0013] On the other hand, a multi-sound event detection and positioning device based on a neural network model is provided, which is applied to a multi-sound event detection and positioning method based on a neural network model, and the device includes: The feature extraction module is used to construct an audio dataset and split the audio dataset into a training set, a validation set, and a test set; preprocess the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset; The training module is used to build a neural network model for multi-sound source detection and localization. The SALSA-Lite features of the training set are input into the neural network model for multi-sound source detection and localization to perform sound detection and direction estimation training. A weighted module is used to train a multi-sound source detection and localization neural network model using a binary cross entropy loss function, a mean square error loss function, and a joint loss function. The joint loss function is a weighted combination of the binary cross entropy loss function and the mean square error loss function. The model tuning module uses the trained multi-sound source detection and localization neural network model to predict the SALSA-Lite features of the validation set, obtaining sound detection and direction estimation results. It also calculates evaluation metrics on the validation set to assess the model's performance on the validation set. Based on the evaluation results of the validation set, it adjusts the model's hyperparameters to obtain a tuned multi-sound source detection and localization neural network model. The prediction module is used to predict the SALSA-Lite features of the test set through the tuned multi-sound source detection and localization neural network model to obtain the final sound detection results and direction estimation results.

[0014] Optionally, a feature extraction module is used to perform audio channel swapping on the audio data of the training set to expand the training set; The audio data in the audio dataset is pre-processed by pre-emphasis, frame division and windowing to generate SALSA-Lite features.

[0015] On the other hand, a multi-sound event detection and positioning device based on a neural network model is provided, and the multi-sound event detection and positioning device based on a neural network model includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the multi-sound event detection and positioning methods based on the neural network model is implemented.

[0016] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned multiple sound event detection and positioning methods based on a neural network model.

[0017] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: In an embodiment of the present invention, a method for detecting and locating multiple sound events based on a neural network model is proposed. Compared with existing methods for detecting and locating multiple sound events, this method has significant advantages. In terms of model structure, an innovative time-frequency (TF) multi-scale residual convolution block is designed to form a network model with a Conformer module and a cross-stitch unit module. Multi-scale feature extraction, strengthening long sequence modeling, and promoting task collaborative optimization improve performance and accuracy; in terms of data processing, pre-emphasis and frame windowing improve feature quality, audio channel exchange and spectrum enhancement increase data diversity and reduce overfitting, and the SALSA-Lite feature is used to enhance feature expression; in terms of training strategy, a multivariate loss function is used to accelerate convergence while taking into account task requirements, and hyperparameters are flexibly adjusted with the help of a validation set. This makes this method highly efficient in training, has excellent actual performance, has strong generalization ability on unknown data, can accurately respond to complex and changing actual scenarios, and effectively overcomes the shortcomings of traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a method for detecting and locating multiple sound events based on a neural network model provided by an embodiment of the present invention; Figure 2 A model structure diagram of a method for detecting and locating multiple sound events based on a neural network model provided by an embodiment of the present invention; Figure 3 A structural diagram of a time-frequency multi-scale residual convolution module provided by an embodiment of the present invention; Figure 4 A block diagram of a multi-sound event detection and positioning device based on a neural network model provided by an embodiment of the present invention; Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0021] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0022] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0023] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0024] The embodiment of the present invention provides a method for detecting and locating multiple sound events based on a neural network model. The method can be implemented by a device for detecting and locating multiple sound events based on a neural network model. The device for detecting and locating multiple sound events based on a neural network model can be a terminal or a server. Figure 1 The flowchart of the multi-sound event detection and positioning method based on the neural network model is shown in FIG. Figure 1 As shown, the present invention proposes a method for detecting and locating multiple sound events based on a neural network model. The processing flow of the method may include the following steps: S1. Construct an audio dataset and split it into a training set, a validation set, and a test set. Preprocess the audio data in the audio dataset to obtain a lightweight version of the Spatial Cue-Augmented Log-Spectrogram (SALSA-Lite) features of the audio data in the audio dataset. In a feasible implementation mode, according to Figure 1 , construct an audio dataset, and split the audio dataset into a training set, a validation set, and a test set. In this embodiment of the present invention, the MIC-formatted audio data and labels in the TAU-NIGENS Spatial Sound Events 2021 dataset provided by the DCASE2021 Task 3 competition were selected as the dataset for this embodiment of the present invention [Politis, Archontis, Sharath Adavanne, and Tuomas Virtanen. "TAU-NIGENS Spatial Sound Events 2021". Zenodo, February 28, 2021.]. The training set contains 400 one-minute audios, and the validation set and test set each contain 100 one-minute audios.

[0025] In a feasible implementation, preprocessing the audio data in the audio dataset to obtain SALSA-Lite features of the audio data in the audio dataset includes: Use audio channel swapping on the audio data of the training set to expand the training set; The audio data in the audio dataset is pre-processed by pre-emphasis, frame division and windowing to generate SALSA-Lite features.

[0026] In one feasible implementation, the training set is expanded eightfold based on the eight channel swapping modes and their corresponding swapped labels provided by Audio Channel Swapping (ACS) given in the literature [Qing Wang, Jun Du, Hua-Xin Wu, Jia Pan, Feng Ma, Chin-Hui Lee. “A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection”. IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1251-1264, 2023.].

[0027] In one possible implementation, pre-emphasis is applied to all audio data in the data set, including:

[0028] in, is the pre-emphasized signal, is the original input signal, is the input signal value at the previous moment, is the pre-emphasis coefficient.

[0029] In one feasible implementation, when performing SALSA-Lite audio feature extraction according to the literature [ Thi Ngoc Tho Nguyen, Douglas L.Jones, Karn N. Watcharasupat, Huy Phan, Woon-Seng Gan. “SALSA-Lite: A Fast and Effective Feature for Polyphonic Sound Event Localization and Detectionwith Microphone Arrays”. ICASSP 2022 - IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 716-720, 2022.], the audio sampling rate is set to 24kHz, a 1024-point FFT and Hanning window are used for short-time Fourier transform, and a frame shift of 300. The Hanning window formula is as follows:

[0030] in, is the value of the window function, is the length of the window, is the index of the current sampling point.

[0031] The SALSA-Lite feature contains time-frequency representation and spatial information, with a total of 7 channels, where the frequency range is set from 50Hz to 9000Hz.

[0032] S2. Build a multi-sound source detection and localization neural network model, input the SALSA-Lite features of the training set into the multi-sound source detection and localization neural network model, and perform sound detection and direction estimation training; In one feasible implementation, the multi-sound source detection and localization neural network model includes: Multi-sound source detection and localization is designed using a time-frequency multi-scale residual convolution block, a Conformer module, and a cross-stitch unit module for mutual learning between detection and localization tasks [Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, Martial Hebert. “Cross-Stitch Networks for Multi-TaskLearning”. Proceedings of the IEEE Conference on Computer Vision and PatternRecognition, pp. 3994-4003, 2016.]. The time-frequency multi-scale residual convolution includes a multi-scale convolution layer that combines time and frequency axis convolutions, a Relu activation function, a normal convolution layer, and a batch normalization layer. A downsampling module is also provided to handle the situation where the number of input and output channels or the step size are inconsistent. Finally, the output is added to the residual branch and output through the Relu activation function.

[0033] In one feasible implementation, during the training process, time masking and frequency masking are used to perform data augmentation on the input training set SALSA-Lite features.

[0034] In a feasible implementation, Figure 2 As shown, the SALSA-Lite features of the training set are input into the multi-sound source detection and localization neural network model to perform sound detection and direction estimation training. Figure 3 Figure 2 shows the time-frequency multi-scale residual convolutional block structure designed for sound event detection and localization. The TF multi-scale convolution module first extracts input features using 3x1 temporal convolution and 1x3 frequency convolution on the time and frequency axes, capturing temporal and frequency patterns. The resulting features are then concatenated and processed through a batch normalization layer, followed by feature fusion using a 1x1 convolution. The TF multi-scale convolution module replaces traditional convolutional layers to extract multi-scale features. After batch normalization and ReLU activation, further convolution is performed, followed by batch normalization. Furthermore, if the number of input and output channels or stride lengths differ, downsampling is performed. Finally, the output of the main branch is added to the output of the residual branch and activated through a ReLU to obtain the final result, thereby improving the model's feature extraction capabilities and training stability.

[0035] In one feasible implementation, the TF multi-scale residual convolutional block extracts the temporal dynamics and frequency distribution characteristics of sound features through 3×1 and 1×3 convolutions, respectively. Multi-scale information fusion is achieved through channel concatenation and 1×1 convolution, and residual connections are combined to mitigate the vanishing gradient problem. Compared to the single convolution of traditional residual blocks, the first module directly extracts multi-scale time-frequency features, significantly enhancing the ability to utilize spatial cues in SALSA-Lite features and providing richer time-frequency feature representations for sound event detection and localization. Conventional two-dimensional convolutional layers further extract abstract features based on multi-scale features to explore potential sound patterns. The Conformer unit integrates CNN local feature extraction and the Transformer self-attention mechanism to capture long-term and short-term dependencies and enhance temporal modeling capabilities. The soft parameter sharing mechanism of the cross-stitch unit addresses the information competition problem caused by hard parameter sharing in multi-task learning, enabling dynamic feature fusion between the SED and DOA branches, balancing task specificity and information sharing requirements.

[0036] S3. Training a multi-sound source detection and localization neural network model through a binary cross entropy loss function, a mean square error loss function, and a joint loss function, where the joint loss function is a weighted combination of the binary cross entropy loss function and the mean square error loss function; In one feasible implementation, a multi-sound source detection and localization neural network model is trained by a binary cross entropy loss function, a mean square error loss function, and a joint loss function. The joint loss function performs a weighted combination of the binary cross entropy loss function and the mean square error loss function, including: Define the joint loss function:

[0037] in, represents the binary cross entropy loss function used for sound event detection, represents the mean square error loss function used for sound source direction estimation, is the weight.

[0038] S4. Use the trained multi-sound source detection and localization neural network model to predict the SALSA-Lite features of the validation set to obtain sound detection results and direction estimation results; calculate evaluation metrics on the validation set to evaluate the model's performance on the validation set; based on the evaluation results of the validation set, adjust the model's hyperparameters (such as learning rate and number of iterations) to obtain a tuned multi-sound source detection and localization neural network model; In one feasible implementation, the trained multi-sound source detection and localization neural network model predicts the SALSA-Lite features of the validation set, and outputs the sound event category probability and three-dimensional coordinate positioning results through the SED branch and DOA branch of the model respectively, which are converted into the final sound detection results and direction estimation results after sigmoid and tanh activation.

[0039] The “macro-average” evaluation system of DCASE 2021 Task 3 is used. The specific indicators include: SED related error rate (ER ≤20° ), calculate the ratio of the number of predicted errors in all categories to the total number of events, and only when the predicted category is correct and the DOA deviation is ≤20° is considered valid, ER ≤20° The lower the value, the fewer the detection errors; SED related F score (F ≤20° ) Comprehensive detection accuracy and recall rate, the higher F≤20°, the better the detection accuracy and completeness; positioning error (LE CD ): Calculate the average angular error between the predicted DOA and the true value in all categories, LE CD The lower the position, the higher the positioning accuracy; the positioning recall rate (LR CD ): Calculate the ratio of the number of correctly positioned events to the number of true events, LR CD The higher it is, the better the positioning integrity. SELDerror combines the overall error of SED and DOA. The lower the value, the better the performance. The formula is:

[0040] Based on the evaluation results of the validation set, the learning rate was set to 0.0003, the batch size was 32, the loss function weight was set to 0.5, the two tasks were given equal importance, the maximum number of training rounds was set to 100, and training was terminated early when the SELDerror of the validation set did not decrease for 10 consecutive rounds to avoid overfitting. The model with the best SELDerror on the test set during training was saved. The tuned model achieved ER on the validation set. ≤20° =0.454, F ≤20° =64.4%,LE CD =12.275°, LR CD =72.7%, SELDerror=0.288.

[0041] S5. The SALSA-Lite features of the test set are predicted through the tuned multi-sound source detection and localization neural network model to obtain the final sound detection results and direction estimation results.

[0042] In a feasible implementation, the SALSA-Lite features of the test set are predicted by the above-mentioned tuned multi-sound source detection and localization neural network model. ≤20° =0.471, F ≤20° =63.2%,LE CD =11.979°, LR CD =72.1%, SELDerror=0.296. This shows a certain generalization ability.

[0043] The results of different methods on the test set are compared using the same experimental settings and conditions, as shown in Table 1.

[0044] Table 1 Comparison of the results of different methods on the test set under the same experimental settings and conditions

[0045] This method was compared with existing algorithms for sound event detection and direction of arrival estimation. The results demonstrate that, thanks to its innovative design of multi-scale residual blocks, its multi-scale feature fusion mechanism demonstrates strong robustness in the face of unknown data and complex acoustic environments, overcoming the limitations of existing techniques in extracting single features. By optimizing the structure, the method accelerates model convergence and achieves good performance.

[0046] In an embodiment of the present invention, a method for detecting and locating multiple sound events based on a neural network model is proposed. Compared with existing methods for detecting and locating multiple sound events, this method has significant advantages. In terms of model structure, an innovative time-frequency multi-scale residual convolution block is designed to form a network model with a Conformer module and a cross-stitch unit module, which extracts features at multiple scales, strengthens long sequence modeling, and promotes task collaborative optimization, thereby improving performance and accuracy. In terms of data processing, pre-emphasis and frame windowing are used to improve feature quality, audio channel exchange and spectrum enhancement increase data diversity, reduce overfitting, and the SALSA-Lite feature is used to enhance feature expression. In terms of training strategy, a multivariate loss function is used to accelerate convergence while taking into account task requirements, and hyperparameters are flexibly adjusted with the help of a validation set. This makes this method highly efficient in training, has excellent actual performance, has strong generalization capabilities on unknown data, can accurately respond to complex and changeable actual scenarios, and effectively overcomes the shortcomings of traditional methods.

[0047] Figure 4 FIG1 is a block diagram of a multi-sound event detection and positioning device 300 based on a neural network model according to an exemplary embodiment. The device 300 is used in a multi-sound event detection and positioning method based on a neural network model. Figure 4 The device includes a feature extraction module 310, a training module 320, a weighting module 330, a model tuning module 340, and a prediction module 350. The feature extraction module 310 is used to construct an audio data set and split the audio data set into a training set, a validation set, and a test set; preprocess the audio data in the audio data set to obtain SALSA-Lite features of the audio data in the audio data set; A training module 320 is used to build a multi-sound source detection and localization neural network model, input the SALSA-Lite features of the training set into the multi-sound source detection and localization neural network model, and perform sound detection and direction estimation training; A weighting module 330 is used to train the multi-sound source detection and localization neural network model using a binary cross entropy loss function, a mean square error loss function, and a joint loss function, where the joint loss function performs a weighted combination of the binary cross entropy loss function and the mean square error loss function; The model tuning module 340 is configured to use the trained multi-sound source detection and localization neural network model to predict the SALSA-Lite features of the validation set to obtain sound detection results and direction estimation results; calculate evaluation indicators on the validation set to evaluate the performance of the model on the validation set; and adjust the model's hyperparameters based on the evaluation results of the validation set to obtain a tuned multi-sound source detection and localization neural network model. The prediction module 350 is used to predict the SALSA-Lite features of the test set using the tuned multi-sound source detection and localization neural network model to obtain the final sound detection results and direction estimation results.

[0048] Optionally, the feature extraction module 310 is configured to perform audio channel swapping on the audio data of the training set to expand the training set; The audio data in the audio dataset is pre-processed by pre-emphasis, frame division and windowing to generate SALSA-Lite features.

[0049] Optionally, use pre-emphasis on all audio data in the dataset, including:

[0050] in, is the pre-emphasized signal, is the original input signal, is the input signal value at the previous moment, is the pre-emphasis coefficient.

[0051] Optionally, the multi-sound source detection and localization neural network model includes: Multi-sound source detection and localization: time-frequency multi-scale residual convolution blocks, Conformer modules, and cross-stitch unit modules for mutual learning between detection and localization tasks; Among them, the time-frequency multi-scale residual convolution includes a multi-scale convolution layer that combines the time axis and frequency axis convolution, a ReLU activation function, a normal convolution layer and a batch normalization layer. At the same time, a downsampling module is provided to handle the situation where the number of input and output channels or the step size are inconsistent. Finally, the output is added to the residual branch and output through the ReLU activation function.

[0052] Optionally, during training, temporal masking and frequency masking are used to perform data augmentation on the input training set SALSA-Lite features.

[0053] Optionally, the multi-sound source detection and localization neural network model is trained by a binary cross entropy loss function, a mean square error loss function, and a joint loss function, where the joint loss function performs a weighted combination of the binary cross entropy loss function and the mean square error loss function, including:

[0054] Define the joint loss function:

[0055] in, represents the binary cross entropy loss function used for sound event detection, represents the mean square error loss function used for sound source direction estimation, is the weight.

[0056] In the embodiment of the present invention, compared with the existing multi-sound event detection and positioning method, this device has significant advantages. In terms of model structure, an innovative time-frequency multi-scale residual convolution block is designed to form a network model with the Conformer module and the cross-stitch unit module, which extracts features at multiple scales, strengthens long sequence modeling, and promotes task collaborative optimization, thereby improving performance and accuracy. In terms of data processing, pre-emphasis and frame windowing improve feature quality, audio channel exchange and spectrum enhancement increase data diversity, reduce overfitting, and the SALSA-Lite feature is used to enhance feature expression. In terms of training strategy, a multivariate loss function is used to accelerate convergence while taking into account task requirements, and hyperparameters are flexibly adjusted with the help of a validation set. This makes this method highly efficient in training, has excellent actual performance, has strong generalization capabilities on unknown data, can accurately respond to complex and changing actual scenarios, and effectively overcomes the shortcomings of traditional methods.

[0057] Figure 5 FIG. 1 is a structural diagram of a multi-sound event detection and positioning device based on a neural network model provided by an embodiment of the present invention. Figure 5 As shown, the multi-sound event detection and positioning device based on the neural network model may include the above Figure 4 The apparatus for detecting and locating multiple sound events based on a neural network model is shown. Optionally, the apparatus for detecting and locating multiple sound events based on a neural network model 410 may include a first processor 2001 .

[0058] Optionally, the multiple sound event detection and positioning device 410 based on a neural network model may further include a memory 2002 and a transceiver 2003 .

[0059] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0060] The following combination Figure 5 The components of the multi-sound event detection and positioning device 410 based on the neural network model are described in detail: The first processor 2001 is the control center of the neural network model-based multiple sound event detection and positioning device 410, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0061] Optionally, the first processor 2001 can perform various functions of the multiple sound event detection and positioning device 410 based on the neural network model by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0062] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0063] In a specific implementation, as an embodiment, the multi-sound event detection and positioning device 410 based on the neural network model may also include multiple processors, such as Figure 5 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0064] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0065] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and access the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0066] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0067] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0068] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and detect the location of the device 410 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0069] It should be noted that Figure 5 The structure of the multi-sound event detection and positioning device 410 based on the neural network model shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0070] In addition, the technical effects of the multi-sound event detection and positioning device 410 based on the neural network model can refer to the technical effects of the multi-sound event detection and positioning method based on the neural network model described in the above method embodiment, and will not be repeated here.

[0071] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0072] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0073] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensor. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0074] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0075] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0076] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0077] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0078] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0079] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for detecting and locating multiple sound events based on a neural network model, characterized in that: The method comprises: S1. Construct an audio dataset and split the audio dataset into a training set, a validation set, and a test set; preprocess the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset; S2. Build a multi-sound source detection and localization neural network model, input the SALSA-Lite features of the training set into the multi-sound source detection and localization neural network model, and perform sound detection and direction estimation training; S3. Training a multi-sound source detection and localization neural network model through a binary cross entropy loss function, a mean square error loss function, and a joint loss function, where the joint loss function is a weighted combination of the binary cross entropy loss function and the mean square error loss function; S4. Use the trained multi-sound source detection and localization neural network model to predict the SALSA-Lite features of the validation set to obtain sound detection results and direction estimation results; calculate evaluation indicators on the validation set to evaluate the performance of the model on the validation set; based on the evaluation results of the validation set, adjust the model's hyperparameters to obtain a tuned multi-sound source detection and localization neural network model; S5. The SALSA-Lite features of the test set are predicted through the tuned multi-sound source detection and localization neural network model to obtain the final sound detection results and direction estimation results.

2. The method according to claim 1, characterized in that Preprocess the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset, including: Use audio channel swapping on the audio data of the training set to expand the training set; The audio data in the audio dataset is pre-processed by pre-emphasis, frame division and windowing to generate SALSA-Lite features.

3. The method according to claim 2, characterized in that Use pre-emphasis on all audio data in the dataset, including: in, is the pre-emphasized signal, is the original input signal, is the input signal value at the previous moment, is the pre-emphasis coefficient.

4. The method according to claim 2, characterized in that Multiple sound source detection and localization neural network model, including: The time-frequency multi-scale residual convolution block and Conformer module designed for multi-sound source detection and localization, as well as the cross-stitch unit module for mutual learning between detection and localization tasks; Among them, the time-frequency multi-scale residual convolution includes a multi-scale convolution layer that combines the time axis and frequency axis convolution, a ReLU activation function, a normal convolution layer and a batch normalization layer. At the same time, a downsampling module is provided to handle the situation where the number of input and output channels or the step size are inconsistent. Finally, the output is added to the residual branch and output through the ReLU activation function.

5. The method according to claim 3, characterized in that During the training process, the input training set SALSA-Lite features are augmented using time masking and frequency masking.

6. The method according to claim 4, characterized in that The multi-sound source detection and localization neural network model is trained through the binary cross entropy loss function, the mean square error loss function and the joint loss function. The joint loss function is a weighted combination of the binary cross entropy loss function and the mean square error loss function, including: Define the joint loss function: in, represents the binary cross entropy loss function used for sound event detection, represents the mean square error loss function used for sound source direction estimation, is the weight.

7. A multi-sound event detection and positioning device based on a neural network model, wherein the multi-sound event detection and positioning device based on a neural network model is used to implement the multi-sound event detection and positioning method based on a neural network model according to any one of claims 1 to 6, characterized in that: The device comprises: The feature extraction module is used to construct an audio dataset and split the audio dataset into a training set, a validation set, and a test set; preprocess the audio data in the audio dataset to obtain the SALSA-Lite features of the audio data in the audio dataset; The training module is used to build a neural network model for multi-sound source detection and localization. The SALSA-Lite features of the training set are input into the neural network model for multi-sound source detection and localization to perform sound detection and direction estimation training. A weighted module is used to train a multi-sound source detection and localization neural network model using a binary cross entropy loss function, a mean square error loss function, and a joint loss function. The joint loss function is a weighted combination of the binary cross entropy loss function and the mean square error loss function. The model tuning module uses the trained multi-sound source detection and localization neural network model to predict the SALSA-Lite features of the validation set, obtaining sound detection and direction estimation results. It also calculates evaluation metrics on the validation set to assess the model's performance on the validation set. Based on the evaluation results of the validation set, it adjusts the model's hyperparameters to obtain a tuned multi-sound source detection and localization neural network model. The prediction module is used to predict the SALSA-Lite features of the test set through the tuned multi-sound source detection and localization neural network model to obtain the final sound detection results and direction estimation results.

8. The device according to claim 7, characterized in that The feature extraction module is used to perform audio channel swapping on the audio data of the training set to expand the training set; The audio data in the audio dataset is pre-processed by pre-emphasis, frame division and windowing to generate SALSA-Lite features.

9. A multi-sound event detection and positioning device based on a neural network model, the multi-sound event detection and positioning device based on a neural network model comprising: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, any one of the multiple sound event detection and positioning methods based on a neural network model as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing at least one instruction, wherein the at least one instruction is loaded and executed by a processor to implement any one of the methods for detecting and locating multiple sound events based on a neural network model as described in any one of claims 1 to 6.

Images