A data-driven method for intelligent classification and recognition of multiple sound sources
Through the TranSFormer structure and characteristic fusion technology in deep learning networks, the existing gun sound recognition technology has the lack of real -time and accuracy, improves the sound information of gun sound, and improves recognition accuracy.
Patent Information
- Application Number
- CN202111602264.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-24
- Publication Date
- 2025-01-24
- Estimated Expiration
- 2041-12-24
AI Technical Summary
The existing gunshot recognition technology is difficult to ensure real-time and accuracy at the same time under complex conditions on the battlefield, and the sound preprocessing method loses phase information, affecting the recognition accuracy.
Deep learning networks including gun sound characteristics extraction network and gun sound classification network are used to extract depth feature information through the Transformer structure encoder group, and combined with the phase information of the gun sound to combine the Merl spectrum and the time -phase spectrum chart.
The real -time and accuracy of gun recognition has been improved, and the problem that the RNN network memory occupies the unable to capture global information by the RNN network memory occupation, and improves the elapse characteristics.
Smart Images
Figure CN114299992B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of sound source recognition, and in particular relates to a data-driven multi-sound source intelligent classification and recognition method. Background Art
[0002] In conventional warfare, peacekeeping and counter-terrorism operations, snipers can efficiently kill enemy personnel and damage key equipment, causing huge casualties and psychological panic to the army, and their threats are increasing. In order to alleviate the threat posed by snipers, it is necessary to study an efficient anti-sniper detection system. Among them, sniper gunshot category recognition technology is one of the key technologies of the anti-sniper detection system.
[0003] With the development of artificial intelligence, gunshot recognition technology mainly draws on deep neural networks for sound source classification, such as conventional urban noise classification, mixed gunshot event detection, and medical heart sound classification diagnosis. The following problems exist when applying the above deep neural networks to gunshot recognition:
[0004] 1. The existing network model cannot guarantee both strong real-time performance and high accuracy when identifying gunshots.
[0005] Under complex battlefield conditions, gunshots are strongly interfered by noise, and traditional methods cannot accurately judge the type of gunshots. Deep learning can effectively solve the noise interference problem, but the inherent sequential properties of existing recurrent neural networks hinder the parallelization of training samples. For long sequences, memory limitations will hinder the batch processing of gunshot samples. Convolutional neural networks place too much emphasis on the texture of gunshot spectrograms and cannot imitate humans to identify information through global contours, resulting in the inability to further improve the accuracy of gunshot recognition.
[0006] 2. The existing sound preprocessing method loses phase information, resulting in incomplete gunshot feature information.
[0007] Among the series of preprocessing steps for one-dimensional gunshot signals, there is a mapping power spectrum operation - taking the square of the modulus of the result after Fourier transform. This operation will cause the phase information of the gunshot to be lost, affecting the accuracy of gunshot recognition. Summary of the invention
[0008] The present invention provides a data-driven multi-sound source intelligent classification and recognition method to solve the above problems.
[0009] In order to solve the above technical problems, the present invention provides a data-driven multi-sound source intelligent classification and recognition method, which is characterized by adopting a deep learning network including a gunshot feature extraction network and a gunshot classification network, and specifically includes the following steps:
[0010] S1: using the data acquisition module to collect gunshot information for a certain period of time, preprocessing the one-dimensional gunshot signal, and generating a logarithmic Mel spectrogram and a phase spectrogram;
[0011] S2, inputting the logarithmic Mel spectrogram and the phase spectrogram obtained in S1 into the gunshot feature extraction network to extract the deep feature information in the input information;
[0012] The gunshot feature extraction network includes multiple Transformer structure encoder groups. First, the logarithmic Mel spectrogram and phase spectrogram obtained in S1 are split and cut into multiple sub-graphs. Then, the sub-graphs of these two feature graphs are paired and input into the linear layer in parallel. Each sub-graph is transformed into a one-dimensional vector, and position information is added to each vector. The vector is input into multiple serially connected Transformer structure encoder groups in parallel for processing. Finally, the logarithmic Mel feature and the phase feature are fused to obtain the deep feature vector of the gunshot information.
[0013] S3. Input the deep feature information of the gunshot information extracted by S2 into the gunshot discrimination network to obtain the gunshot category information: the gunshot discrimination network is composed of an average pooling layer, a linear layer and a tanh activation layer set in sequence. The deep feature information is first average pooled on the channel dimension by the average pooling layer. The result is transformed into the output dimension size by the linear layer, and then processed by the tanh activation layer to obtain the gunshot category prediction result.
[0014] Beneficial effects:
[0015] 1. The present invention creatively uses the Transformer structure for the recognition and classification of gunshots. This structure is far superior to the RNN network in terms of parallel computing, solves the problem of large memory usage of the RNN network and inability to batch process training samples, and improves the real-time performance of gunshot recognition. Compared with the CNN structure, it can better capture global information. In the recognition of gunshot spectrograms, the type is no longer judged by local texture, but by the contour information of the gunshot in the spectrogram, which improves the accuracy of gunshot recognition.
[0016] 2. The present invention adds the phase spectrum information of gunshots to the input features, supplements the lost phase information, improves the characteristic information of gunshots, and further improves the accuracy of gunshot recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 System flow chart of the present invention;
[0018] Figure 2 Schematic diagram of deep feature extraction network
[0019] Figure 3 Transformer structure encoder group structure diagram
[0020] Figure 4 Schematic diagram of the multi-head self-attention mechanism module structure
[0021] Figure 5 Schematic diagram of the scaled dot product attention mechanism structure
[0022] Figure 6 Schematic diagram of the gunshot discrimination network. DETAILED DESCRIPTION
[0023] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below.
[0024] The present invention proposes a data-driven multi-sound source intelligent classification and recognition method, which designs a deep learning network, including a gunshot feature extraction network and a gunshot classification network: first, the one-dimensional gunshot signal collected by the sound sensor is preprocessed, and the preprocessed data is input into the feature extraction network, and the deep feature information in the input data is extracted through the Transformer module, and the obtained deep feature information is input into the gunshot discrimination network, and finally the gunshot category information is obtained. The specific steps are as follows:
[0025] S1: Use the data acquisition module to collect 5S gunshot information at a sampling frequency of 44.1KHz, perform data preprocessing on the one-dimensional gunshot signal (time series information), and generate a logarithmic Mel spectrogram and a time phase spectrogram, as follows:
[0026] 1) Pre-emphasis: The gunshot signal is filtered through a high-pass filter to balance the spectrum and improve the signal to avoid numerical problems during the Fourier transform operation.
[0027] 2) Framing: After pre-emphasis, the signal is divided into short time frames of 50ms in length, and each frame shift length is 1024 points. The gunshot signal is non-stationary, and it is meaningless to perform Fourier transform on the entire signal because the frequency contour of the signal is lost over time.
[0028] 3) Windowing: After dividing the signal into frames, each frame is multiplied by a window function, such as a Hamming window, to increase the continuity of the left and right ends of the frame.
[0029] 4) Calculate the signal energy spectrum: Perform a 2048-point fast Fourier transform operation on each frame of the signal to obtain the spectrum, collect the vector angles, and obtain a phase spectrum with a dimension size of [64, 216], where 64 is the phase scale and 216 is the time dimension. Then take the square of the modulus of the spectrum to obtain the spectral line energy of the gunshot signal.
[0030] 5) Mel filtering: The Mel filter bank is calculated and the power spectrum is passed through a set of 64 filters with equal high Mel scale triangular filters to extract the frequency bands.
[0031] 6) Take the logarithm of the obtained Mel spectrogram to obtain an approximate homomorphic transformation effect, and the output dimension size is (64, 216) logarithmic Mel spectrogram, where 64 is the Mel scale and 216 is the time dimension.
[0032] S2: Input the logarithmic Mel spectrogram and phase spectrogram obtained in S1 into the gunshot feature extraction network to extract the deep feature information in the input information;
[0033] The gunshot feature extraction network consists of multiple Transformer structure encoder groups. The specific network structure is as follows: Figure 2 As shown in the figure, the log-Mel spectrogram and phase spectrogram obtained in S1 are first segmented and cut into 9 sub-images with a dimension size of (7, 24), and then the 18 sub-images of the two feature maps are paired and passed through the linear layer in parallel. Each sub-image is transformed into a one-dimensional vector with a dimension of 168, and position information is attached to each vector; then these 9 groups of vectors are combined into a two-dimensional image (with a shape of [9, 168, 2]) and input into 6 serially connected Transformer structure encoder groups for processing. Finally, the log-Mel and phase features are fused to obtain the deep feature vector of the input information.
[0034] Transformer structure encoder group Figure 3 As shown in the figure, the phase spectrogram and the mel spectrogram are independently input to two encoders. In a single encoder, the input vector is processed in two ways. One way first enters the normalization layer for batch normalization, and then enters the multi-head self-attention module. The result is added to the input vector of the other way to form a residual structure; finally, the features obtained from the residual structure are divided into two ways, one way is batch normalized and input to the feedforward neural network layer for processing, and the obtained features are added to the features of the other way to form a residual structure, which is the output of the module. The multi-head self-attention module is used to extract the global information of the vector; the feedforward neural network layer is used to further optimize the global information.
[0035] The module structure of the multi-head attention mechanism is as follows Figure 4 As shown in the figure, the input features of Query (Q), Key (K), and Value (V) are all themselves. The three paths are mapped to 16 linear layers to obtain 16 input features. These 16 feature vectors are input into 16 scaled dot product attention mechanisms in parallel, and they are combined together to form 16 channels. After being processed by the linear layer, the output features are obtained.
[0036] The structure of the scaled dot product attention mechanism is as follows Figure 4As shown in the figure, the Query (Q) vector and the Key (K) vector are dot-producted. In order to prevent the dot-product result from being too large, a scaling is performed, and the vector after the current time is covered by a mask. After the SoftMax processing, the result is dot-producted with the Value (V) vector to obtain the output of the module.
[0037] After the logarithmic Mel spectrogram and phase spectrogram pass through 6 serially connected Transformer structure encoder groups, 2 corresponding deep Mel feature vectors and deep phase feature vectors are obtained. These two feature vectors are fused in series in the feature dimension to obtain the final deep feature vector of the gunshot information.
[0038] S3: Input the deep feature information of the gunshot information extracted by S2 into the gunshot discrimination network to obtain the gunshot category information.
[0039] The gunshot discrimination network consists of an average pooling layer, a linear layer, and a tanh activation layer, such as Figure 6 As shown in the figure, the deep feature information is first averaged by the average pooling layer to average the channel dimension, and the result is transformed into the output dimension size by the linear layer, and then processed by the tanh activation layer to obtain the gunshot category prediction result.
[0040] Training of deep learning networks:
[0041] The present invention uses pytorch as a framework to build a deep learning environment. Three 1080Ti GPUs are used to accelerate the image processing process in combination with the parallel computing architecture CUDA. The loss function of the deep learning network uses cross entropy; first, pre-training is performed on the AudioSet dataset of a million-level sample data set to obtain a deep learning pre-training model, and then 8,000 groups of gunshot data collected by the acoustic sensor are input into the above model as training samples for transfer learning, iterating 1,000 rounds, taking 256 batches of samples for training at a time during the training process, the initial learning rate is 0.001, and the learning rate is reduced once every 400 rounds. The corresponding deep learning network parameters are obtained, and finally a trained deep learning network model is obtained.
[0042] The 2000 sets of sound data collected by the sound sensor are processed in step S1 to obtain the logarithmic Mel spectrogram and the time phase spectrum, which are input into the network model trained in S4 for testing. After the test, the network accuracy can reach 88%.
[0043] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A data-driven multi-sound source intelligent classification and recognition method, characterized in that: A deep learning network including a gunshot feature extraction network and a gunshot classification network is used, specifically including the following steps: S1: using the data acquisition module to collect gunshot information for a certain period of time, preprocessing the one-dimensional gunshot signal, and generating a logarithmic Mel spectrogram and a phase spectrogram; S2, inputting the logarithmic Mel spectrogram and the phase spectrogram obtained in S1 into the gunshot feature extraction network to extract the deep feature information in the input information; The gunshot feature extraction network includes multiple Transformer structure encoder groups. First, the logarithmic Mel spectrogram and phase spectrogram obtained in S1 are split and cut into multiple sub-graphs. Then, the sub-graphs of these two feature graphs are paired and input into the linear layer in parallel. Each sub-graph is transformed into a one-dimensional vector, and position information is added to each vector. The vector is input into multiple serially connected Transformer structure encoder groups in parallel for processing. Finally, the logarithmic Mel feature and the phase feature are fused to obtain the deep feature vector of the gunshot information. S3. Input the deep feature information of the gunshot information extracted by S2 into the gunshot discrimination network to obtain the gunshot category information: the gunshot discrimination network is composed of an average pooling layer, a linear layer and a tanh activation layer set in sequence. The deep feature information is first average pooled on the channel dimension by the average pooling layer. The result is transformed into the output dimension size by the linear layer, and then processed by the tanh activation layer to obtain the gunshot category prediction result.
2. According to the data-driven multi-sound source intelligent classification and recognition method of claim 1, it is characterized in that: In S1, the data preprocessing steps are as follows: 1) Pre-emphasis: Filter the gunshot signal through a high-pass filter; 2) Framing: After pre-emphasis, the signal is divided into multiple short-time frames; 3) Windowing: After dividing the signal into frames, each frame is multiplied by a window function; 4) Calculate the signal energy spectrum: Perform a fast Fourier transform operation on each frame of the signal to obtain the frequency spectrum and time-phase spectrogram of the gunshot signal, and then take the square of the modulus of the frequency spectrum to obtain the power spectrum of the gunshot signal; 5) Mel filtering: extract the frequency bands of the power spectrum through a set of triangular filters with equal Mel scales to obtain a Mel spectrogram; 6) Take the logarithm of the obtained Mel-spectrogram to obtain a logarithmic Mel-spectrogram.
3. The data-driven multi-sound source intelligent classification and recognition method according to claim 2 is characterized in that: The output dimension size of the logarithmic Mel spectrogram is (64, 216), where 64 is the Mel scale and 216 is the time dimension.
4. The data-driven multi-sound source intelligent classification and recognition method according to claim 1 is characterized in that: In S2, the Transformer structure encoder group includes a batch normalization layer, a multi-head attention mechanism module, a batch normalization layer, and a feedforward neural network layer arranged in sequence; The phase spectrogram and Mel spectrogram are independently input into two encoder groups. In each encoder group, the input vector is processed in two ways. One way first enters the batch normalization layer for batch normalization processing, and then it is input into the multi-head attention mechanism module. The result is added to the input vector of the other way to form a residual structure; finally, the features obtained from the residual structure are divided into two ways. One way is batch normalized and input into the feedforward neural network layer for processing. The features obtained are added to the features of the other way to form a residual structure. The result is the output feature vector of the module.
5. The data-driven multi-sound source intelligent classification and recognition method according to claim 4 is characterized in that: In S2, the logarithmic Mel spectrogram and the phase spectrogram pass through 6 serially connected Transformer structure encoder groups to obtain 2 corresponding deep Mel feature vectors and deep phase feature vectors. These two feature vectors are fused in series in the feature dimension to obtain the final deep feature vector.
6. A data-driven multi-sound source intelligent classification and recognition method according to any one of claims 1 to 4, characterized in that: The loss function of the deep learning network uses cross entropy. First, pre-training is performed on the AudioSet dataset with millions of samples to obtain a deep learning pre-training model. Then, multiple groups of gunshot data collected by the acoustic sensor are input into the pre-training model as training samples for transfer learning. After multiple rounds of iterations, the corresponding deep learning network parameters are obtained.
7. The data-driven multi-sound source intelligent classification and recognition method according to claim 6 is characterized in that: During the training process, 256 batches of samples are taken for training at a time, the initial learning rate is 0.001, and the learning rate is reduced every 400 rounds.