A method and system for classifying acoustic signals of large marine mammals, and electronic equipment

Abstract

The present invention provides a large marine mammal sound signal classification method and system, and electronic equipment, comprising: acquiring sound signal data of different marine mammals, and preprocessing the sound signal data; obtaining original waveform features according to the preprocessed sound signal data, and extracting features from the original waveform features using Mel and IMel feature extraction methods to obtain time-frequency features of the sound signal data; inputting the time-frequency features and the original waveform features into a hybrid attention network and feature fusion model to extract global features, and using the global features to classify the sound signal data, thereby solving the current technical problem of lacking the fusion of two different features, the time-frequency feature spectrogram and the waveform feature, greatly improving the classification efficiency and classification accuracy of the sound signal, and greatly improving the intelligence, usability and reliability of the present invention.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater acoustic signal processing, and in particular to a method and system for classifying acoustic signals of large marine mammals, and electronic equipment. Background Art

[0002] At present, there are two main methods for studying the acoustic signals of large marine mammals. One is to classify using the waveform features of the original audio signal, and the other is to classify based on the time-frequency feature diagram. The time-frequency analysis of the signal through the time-frequency diagram emphasizes the importance of frequency distribution, but loses the phase information, which will lead to inaccurate problems when using the features to restore the sound. For the original waveform features, it has the most original characteristics of the sound signal and retains its own time scale, which can better reflect the time domain characteristics of the sound signal.

[0003] Currently, there are few methods for the fusion of two different features, the time-frequency feature spectrogram and the waveform feature, in the research of sound signals. In terms of classification methods, they are mainly concentrated on network models such as convolutional neural network (CNN), long short-term memory network (LSTM) and Transformer. These methods analyze from different angles, each with its own advantages and disadvantages. A simple CNN model can only extract local feature information from the sound signal. However, for sound signals, only using CNN will lose the long-term dependency of the sound signal, and it is difficult to properly consider the relationship between mutation signals. Although LSTM can handle time series problems, it is slow to process problems with large amounts of data.

[0004] Therefore, the prior art needs to be further developed. Summary of the invention

[0005] The purpose of the present invention is to overcome the above technical deficiencies and provide a large marine mammal sound signal classification method and system, and electronic equipment to solve the problems existing in the prior art.

[0006] To achieve the above technical objectives, according to a first aspect of the present invention, the present invention provides a method for classifying acoustic signals of large marine mammals, comprising:

[0007] S100, acquiring acoustic signal data of different marine mammals, and preprocessing the acoustic signal data;

[0008] S200, obtaining original waveform features according to the preprocessed acoustic signal data, and extracting features from the original waveform features using Mel and IMel feature extraction methods to obtain time-frequency features of the acoustic signal data;

[0009] S300, inputting the time-frequency features and the original waveform features into a hybrid attention network and feature fusion model to extract global features, and using the global features to perform signal classification on the acoustic signal data.

[0010] Specifically, the Mel and IMel feature extraction method is used to extract the original waveform features to obtain the time-frequency features of the acoustic signal data, including:

[0011] The Mel feature extraction method is used to extract the time-frequency features of the original waveform features. The Mel feature extraction method is based on short-time Fourier transform and Mel filter group, and performs time-frequency analysis on the frequency of the original waveform features to obtain the Mel spectrum of the original waveform features.

[0012] Specifically, the extracting the original waveform features by using Mel and IMel to obtain the time-frequency features of the acoustic signal data also includes:

[0013] The IMel feature extraction method is used to extract time-frequency features of the original waveform features. The IMel feature extraction method is to add empirical mode decomposition on the basis of Mel feature extraction to obtain the IMel spectrum of the original waveform.

[0014] Specifically, the S300 includes:

[0015] S310, using a multimodal feature extractor to respectively extract original waveform features and time-frequency features of the acoustic signal data;

[0016] S320, inputting the original waveform features and time-frequency features extracted by the multimodal feature extractor into a hybrid attention module for feature fusion to obtain hybrid features;

[0017] S330: Use a multilayer perceptron to extract global features from the mixed features, and use the global features to classify the acoustic signal data.

[0018] Specifically, the extracting original waveform features and time-frequency features of the acoustic signal data respectively using a multimodal feature extractor includes:

[0019] The multimodal feature extractor includes a spectrogram network and a waveform network;

[0020] The spectrogram network is used to extract the features of Mel spectrum and IMel spectrum, and the waveform network is used to extract the features of original waveform.

[0021] Specifically, the original waveform features and time-frequency features extracted by the multimodal feature extractor are input into the hybrid attention module for feature fusion, including:

[0022] The original waveform features and time-frequency features are locally fused using the multi-head attention mechanism of the hybrid attention module.

[0023] Specifically, the step of inputting the original waveform features and time-frequency features extracted by the multimodal feature extractor into a hybrid attention module for feature fusion further includes:

[0024] Based on the local feature fusion of the original waveform features and time-frequency features, the global relationship of the features is extracted using a multi-head attention mechanism to perform global feature fusion.

[0025] Specifically, the method of using a multilayer perceptron to extract global features from the mixed features and performing signal classification on the global features includes:

[0026] The number of multilayer perceptron blocks in the multilayer perceptron is a first preset value, the multilayer perceptron blocks are used to extract global features from the mixed features, and the global features are input into a classifier to classify the acoustic signal data.

[0027] According to a second aspect of the present invention, there is provided a large marine mammal acoustic signal classification system, comprising:

[0028] Acquisition module: used to acquire acoustic signal data of different marine mammals;

[0029] Control module: used to preprocess the acoustic signal data, obtain the original waveform features according to the preprocessed acoustic signal data, and use Mel and IMel to extract the features of the original waveform to obtain the time-frequency features of the acoustic signal data; or used to input the time-frequency features and the original waveform features into a hybrid attention network and feature fusion model to extract global features, and use the global features to perform signal classification on the acoustic signal data.

[0030] According to a third aspect of the present invention, there is provided an electronic device, comprising: a memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned large marine mammal sound signal classification method is implemented.

[0031] Beneficial effects:

[0032] The present invention uses Mel and IMel feature extraction methods to extract features from original waveform features to obtain time-frequency features of acoustic signal data, which can effectively capture the features of acoustic signals in the frequency domain and further improve the accuracy of classification; the time-frequency features and original waveform features are input into a hybrid attention network and feature fusion model to perform global feature extraction, and the global features are used to perform signal classification on the acoustic signal data, which solves the current technical problem of lack of fusion of two different features, the time-frequency feature spectrogram and the waveform feature, greatly improves the classification efficiency and accuracy of acoustic signals, and greatly improves the intelligence, usability and reliability of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of a method for classifying acoustic signals of large marine mammals provided in a specific embodiment of the present invention;

[0034] Figure 2 Schematic diagram of the system composition of the large marine mammal sound signal classification system provided in a specific embodiment of the present invention;

[0035] Figure 3 It is a flow chart of large marine mammal sound signal classification based on hybrid attention network and feature fusion provided in a specific embodiment of the present invention;

[0036] Figure 4 It is a multi-head calculation process diagram provided in a specific embodiment of the present invention;

[0037] Figure 5 is a functional flow chart of a hybrid attention module provided in a specific embodiment of the present invention;

[0038] Figure 6 is a comparison chart of overall accuracy under different models provided in a specific embodiment of the present invention;

[0039] Figure 7 It is a T-SNE graph on the DeepShip dataset under different models provided in the specific implementation manner of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings of the present invention. Based on the embodiments in this application, other similar embodiments obtained by ordinary technicians in this field without making creative work should all fall within the scope of protection of this application. In addition, the directional words mentioned in the following embodiments, such as "up", "down", "left", "right", etc., are only reference to the directions of the accompanying drawings. Therefore, the directional words used are used to illustrate rather than limit the invention.

[0041] The present invention will be further described below in conjunction with the accompanying drawings and preferred embodiments.

[0042] See also Figure 1 This embodiment provides a large marine mammal acoustic signal classification method, including:

[0043] S100, acquiring acoustic signal data of different marine mammals, and preprocessing the acoustic signal data;

[0044] It should be noted that a first preset value is pre-set before step S100, and the present invention does not limit the specific value of the first preset value, as long as the large marine mammal sound signal classification method proposed in the present invention can be applied.

[0045] Preferably, the present invention sets the first preset value to 2. The above setting is obtained by technicians in this field after a large number of experimental tests. It can effectively extract the global features in the mixed features of the original waveform features and time-frequency features in the sound signal data, which helps to improve the quality of the global features finally input to the classifier, thereby improving the accuracy of sound signal classification.

[0046] It can be understood that the acoustic signal data of different marine mammals can be preprocessed through wavelet denoising, framing, windowing and other methods to remove the complex noise in the marine environment, maintain the effective part of the original signal, reduce the impact of noise on classification, and use the stratified k-cross-validation method to divide the preprocessed acoustic signal data of different marine mammals into training set, test set and validation set.

[0047] S200, obtaining original waveform features according to the preprocessed acoustic signal data, and extracting features from the original waveform features using Mel and IMel feature extraction methods to obtain time-frequency features of the acoustic signal data.

[0048] Specifically, the Mel and IMel feature extraction method is used to extract the original waveform features to obtain the time-frequency features of the acoustic signal data, including:

[0049] The Mel feature extraction method is used to extract the time-frequency features of the original waveform features. The Mel feature extraction method is based on short-time Fourier transform and Mel filter group, and performs time-frequency analysis on the frequency of the original waveform features to obtain the Mel spectrum of the original waveform features.

[0050] It should be noted here that Mel-spectrogram is a time-frequency characteristic spectrum, which is mainly used to represent the change of the frequency components of the signal over time. Mel feature extraction is based on short-time Fourier transform and Mel filter bank to perform time-frequency analysis on the frequency of the audio signal to generate a spectrum feature diagram suitable for human ear perception.

[0051] Specifically, the extracting the original waveform features by using Mel and IMel to obtain the time-frequency features of the acoustic signal data also includes:

[0052] The IMel feature extraction method is used to extract time-frequency features of the original waveform features. The IMel feature extraction method is to add empirical mode decomposition on the basis of Mel feature extraction to obtain the IMel spectrum of the original waveform.

[0053] It should be noted here that the IMel spectrum (Improve-Mel) is an improved Mel spectrum, which is also a time-frequency characteristic spectrum. The IMel spectrum is based on the Mel spectrum and adds empirical mode decomposition (EMD) to capture more dynamic features, that is, empirical mode decomposition (EMD) is added before the short-time Fourier transform to capture the dynamic characteristics of the signal and make up for the short-time Fourier transform, which has a fixed frequency resolution and cannot capture the change of the signal frequency over time for non-stationary signals (such as marine mammal sound signals, which often have certain non-stationarity); and the empirical mode decomposition (EMD) can decompose complex non-stationary signals into multiple intrinsic mode functions (IMFs), each of which has different frequency characteristics. Adding EMD before Fourier transform can first decompose the signal into IMFs with different frequency components, and then perform Fourier transform on each IMF to form IMel, so that appropriate frequency resolution can be used for analysis of signals in different frequency ranges, thereby making up for the deficiency of Fourier transform in frequency resolution for non-stationary signal analysis.

[0054] Furthermore, the key difference between the IMel spectrum and the Mel spectrum is that the IMel spectrum better captures the dynamic changes of the signal through EMD, while the traditional Mel spectrum is only based on Fourier transform and does not perform additional processing on the dynamic characteristics of complex signals.

[0055] S300, inputting the time-frequency features and the original waveform features into a hybrid attention network and feature fusion model to extract global features, and using the global features to perform signal classification on the acoustic signal data.

[0056] It is understandable that classifying the sound signal data of marine mammals by using the fusion features of the time-frequency feature spectrogram and the original waveform features can effectively avoid the technical problem of using network models such as CNN, LSTM and Transformer in the existing technology to analyze from the time-frequency feature spectrogram or the original feature waveform respectively, each with its own advantages and disadvantages, and not being able to integrate the time-frequency features and the original waveform features well, thus resulting in poor classification accuracy, which further improves the accuracy of sound signal data classification.

[0057] Specifically, the S300 includes:

[0058] S310, using a multimodal feature extractor to respectively extract original waveform features and time-frequency features of the acoustic signal data;

[0059] S320, inputting the original waveform features and time-frequency features extracted by the multimodal feature extractor into a hybrid attention module for feature fusion to obtain hybrid features;

[0060] S330: Use a multilayer perceptron to extract global features from the mixed features, and use the global features to classify the acoustic signal data.

[0061] Specifically, the extracting original waveform features and time-frequency features of the acoustic signal data respectively using a multimodal feature extractor includes:

[0062] The multimodal feature extractor includes a spectrogram network and a waveform network;

[0063] The spectrogram network is used to extract the features of Mel spectrum and IMel spectrum, and the waveform network is used to extract the features of original waveform.

[0064] It should be further explained that the multimodal feature extractor (MFE) is used to extract time-frequency features and original waveform features. In order to give full play to the advantages of the two features, two sub-networks, Spectrogram Net and WaveNet, are designed. The Spectrogram Net is used to extract the features of Mel spectrum and IMel spectrum, and the Wave Net is used to extract the original waveform features. These two features are then sent to the Hybrid Attention Network (HANet) for feature fusion. The core is to use the multi-head attention mechanism in the hybrid attention network for dimensional fusion. Finally, the global features are extracted by the fusion feature extractor to achieve the classification effect. The main steps of extracting the time-frequency feature spectrum and the original waveform features using the Spectrogram Net and Wave Net are as follows:

[0065] 1. Spectrogram Net design and feature extraction process of Mel spectrum and IMel spectrum

[0066] The goal of Spectrogram Net is to extract information from Mel spectrum features and IMel spectrum features (real-time spectrogram) to capture the frequency and time domain characteristics of the input signal. The design of this network mainly considers the two-dimensional structure of the spectrogram, so a two-dimensional convolutional neural network (2D-CNN) is used for processing.

[0067] Spectrogram Net design process:

[0068] (1) After preprocessing, the input acoustic signal data is subjected to time-frequency feature extraction to obtain Mel spectrogram and IMel spectrogram, which reflect the time-frequency information of the signal;

[0069] (2) Use 2D-CNN to perform convolution processing on the input Mel spectrum and IMel spectrum, and extract local features in the frequency and time domains from the Mel spectrum and IMel spectrum features, that is, extract the frequency distribution features therein.

[0070] (3) The 2D-CNN network structure is stacked layer by layer, including convolutional layers, pooling layers, and activation layers, which gradually extracts features of different depths for easy integration in subsequent attention modules.

[0071] The process of extracting time-frequency features using Spectrogram Net:

[0072] (1) The Mel spectrum feature map and the IMel spectrum feature map are input into Spectrogram Net respectively, and high-order spectrum features are extracted through a series of convolution operations and pooling layers. The specific steps include: using a two-dimensional convolutional layer (Conv2D) to extract local features in the frequency and time domains from the Mel spectrum and IMel spectrum features, using the ReLU activation function, and then using a two-dimensional maximum pooling layer (MaxPool2D) to reduce the dimension and extract key features. After stacking layer by layer, the global average pooling layer (FAP) is used to take the mean of all feature maps as the output to reduce the feature dimension;

[0073] (2) The final output of the convolutional network is the time-frequency feature , this feature It is a high-level expression of the Mel spectrum and IMel spectrum information. As the input of the feature fusion module, it is further processed and fused.

[0074] 2. Wave Net design and feature extraction process of original waveform

[0075] The purpose of designing Wave Net is to directly process the original waveform features to retain the time domain characteristics of the signal and extract feature information related to the original waveform.

[0076] Wave Net design process:

[0077] Since the original waveform has a large amount of data and obvious timing characteristics, Wave Net uses a one-dimensional convolutional neural network (1D-CNN) to fully utilize the time series structure of the original signal.

[0078] (1) The network learns the frame-by-frame features of waveform data through layer-by-layer extraction of multiple layers of 1D convolutions;

[0079] (2) In order to reduce the amount of data, pooling layers or stride convolutions are appropriately introduced to extract more generalized temporal features layer by layer.

[0080] The process of extracting original waveform features using Wave Net:

[0081] (1) The original waveform features are input into Wave Net, and the frame-by-frame temporal features are extracted through 1D-CNN. The specific steps include: using a one-dimensional convolutional layer (Conv1D) to extract the local features of the original waveform signal, performing batch normalization (BN) on the original waveform signal to accelerate model training and improve stability, using the ReLU activation function to increase nonlinear capabilities, and then using a one-dimensional maximum pooling layer (MaxPool1D) to perform dimensionality reduction and key feature extraction. After stacking layer by layer, the features of the original waveform are extracted;

[0082] (2) The original waveform characteristics of the final output It can retain the main information of the original waveform and is an important feature expression in the time domain. Original waveform features It will also be used as the input of the feature fusion module and the output time-frequency features of Spectrogram Net To merge.

[0083] Specifically, the original waveform features and time-frequency features extracted by the multimodal feature extractor are input into the hybrid attention module for feature fusion, including:

[0084] The original waveform features and time-frequency features are locally fused using the multi-head attention mechanism of the hybrid attention module.

[0085] Specifically, the step of inputting the original waveform features and time-frequency features extracted by the multimodal feature extractor into a hybrid attention module for feature fusion further includes:

[0086] Based on the local feature fusion of the original waveform features and time-frequency features, the global relationship of the features is extracted using a multi-head attention mechanism to perform global feature fusion.

[0087] It is understandable that for local feature fusion, the multi-head attention mechanism in the hybrid attention network focuses more on the feature fusion between channels, aiming to eliminate the influence of the number of channels and strengthen the local correlation between different modal features; for global feature fusion, on the basis of local feature fusion, a higher-level global feature fusion is performed, and the attention mechanism is used to deeply extract the global relationship of features and strengthen the long-distance dependence between different temporal and spatial features. The two-step fusion is progressive. Local feature fusion solves the fusion problem of local features through the multi-head attention mechanism, while global feature fusion further abstracts and extracts global deep-level features on this basis. Both rely on the attention mechanism to achieve feature fusion, but the focus is different. Local feature fusion emphasizes local channel correlation, while global feature fusion further extracts global deep semantic information. Finally, through these two progressive feature fusion steps, the feature fusion model can fully tap the useful information in different modal features and improve the understanding and classification performance of complex signals.

[0088] Specifically, the method of using a multilayer perceptron to extract global features from the mixed features and performing signal classification on the global features includes:

[0089] The number of multilayer perceptron blocks in the multilayer perceptron is a first preset value, the multilayer perceptron blocks are used to extract global features from the mixed features, and the global features are input into a classifier to classify the acoustic signal data.

[0090] It can be understood that after the two-step feature fusion of local feature fusion and global feature fusion, the multi-layer perceptron (MLP) is used to extract global features. The purpose of the two-step fusion is to integrate features from different networks at the local and global levels, and then further capture the global dependencies between features through MLP, so as to obtain a more comprehensive feature representation for subsequent classification or recognition tasks. The specific implementation process is as follows:

[0091] 1. Use a multimodal feature extractor to extract the original waveform features and time-frequency features of the acoustic signal data respectively;

[0092] The original waveform features are extracted through the multimodal feature extractor (MFE) and time-frequency characteristics They are respectively regarded as the Q and K vectors in the attention mechanism. The Q vector, i.e. the query vector, is used to find relevant information from other features, and the K vector, i.e. the key vector, is used to calculate the importance weight of the feature. Original waveform features and time-frequency Align in the time domain, where C is the number of channels, N is the time dimension, and R is a set of real numbers. Then a multi-head attention mechanism is proposed to perform channel-level feature fusion.

[0093] 2. Input the original waveform features and time-frequency features extracted by the multimodal feature extractor into the hybrid attention module for feature fusion to obtain hybrid features. The specific process is as follows:

[0094] like Figure 5 As shown, and After feature extraction by the multimodal feature extractor (MFE), the size of , where B is the batch number, C is the number of channels, and T is the time. Right now Figure 5 In ;

[0095] (1) Local fusion

[0096] and After multiple decomposition, the size is , where H is the size of the convolution kernel; and After transposing, we get and , and then and The dot product calculation is performed to obtain the similarity matrix, and the obtained similarity matrix is ​​normalized by the Softmax function to obtain the size of the attention matrix: , that is, the fusion of local features is completed;

[0097] (2) Global Fusion

[0098] To obtain and The mixed features are connected in the channel dimension to obtain a size of Features , and the size is obtained by projecting MLP Mixed features , the mixed features As a Value vector, through The dot product with the attention matrix gives a size of The features of are then pooled globally to obtain a size of The features of are used to achieve the fusion of global features. The final channel weighted result is expressed as , this process can be expressed by the formula:

[0099] ;

[0100] ;

[0101] in, is the Softmax function, Represents the number of multi-head attention heads.

[0102] 3. Use multi-layer perceptron to extract global features from mixed features, and use global features to classify acoustic signal data;

[0103] The Transformer model can be used as a preferred model for the hybrid attention network. The Transformer model takes the multi-head self-attention mechanism as its core. It assigns weights to different features through multiple parallel attention heads and fuses multiple information sources, so that the model has a strong global feature modeling capability when processing sequence data and extracting features. Feature fusion is achieved by introducing a multi-head attention module based on the Transformer structure to integrate the multimodal characteristics of time-frequency features and waveform features. Therefore, this design can be regarded as using the Transformer's multi-head attention as the core of the hybrid attention model for feature extraction and fusion, thereby improving classification accuracy and model adaptability.

[0104] In the Transformer model, each attention calculation unit is called an attention head. In order to enhance the expressiveness of the model, Transformer introduces the concept of multi-head attention. Multi-head attention enables the model to better capture information of different positions and relationships by operating multiple attention heads, that is, independent attention calculation units, in parallel.

[0105] Figure 4 is the multi-head attention calculation process, such as Figure 4 As shown in the figure, the multi-layer attention mechanism provides each attention head with an independent Q vector, K vector, and V vector. The V vector, i.e., the value vector, indicates that the feature content to be fused uses independent linear devices and weight parameters (the weights of the Q, K, and V matrix multiplication operations). These matrices are combined with the following attention calculation formula to obtain the final attention value AttentionScore:

[0106] ;

[0107] Among them, Q, K, and V represent the Q, K, and V matrices obtained by independent linear layers, respectively, representing the dimensions of the attention head. The attention score is normalized by the Softmax function and then multiplied by V to obtain the attention output of each head. The attention score is an indicator to measure the importance of the relationship between different elements in the input sequence. The attention score is obtained by calculating a certain similarity measure between the Q vector and the K vector. These scores reflect the contribution of each key vector to the current query vector, that is, how much attention the model should give to each key vector when calculating the weighted representation.

[0108] The multi-layer perceptron block in this embodiment includes a global fusion feature extractor of two multi-layer perceptron blocks (MLP Blocks). The task of the extractor is to capture the overall properties of the audio features. Specifically, the feature extractor receives the input signal and processes it through two MLPBlocks. Each MLPBlock includes a series of layers responsible for extracting features. In each MLPBlock, the input is first channel-normalized through each layer to speed up the training process. This step helps to make the functions of different channels work better together during training. By introducing MLPBlock, more comprehensive global feature information can be extracted from the audio feature quantity, providing a more representative feature quantity representation for subsequent tasks.

[0109] ;

[0110] in, Represents the processing result of layer normalization, which is a normalization technique used to normalize the input of each layer of the neural network, thereby improving the stability and convergence speed of the model. and Represent the mean and variance of the input respectively, Set to -5 to prevent division by zero; and are the weights and biases of the affine transformation used to scale the output values, and then The two-layer MLP is sent to global feature extraction, the GeLU nonlinear activation function is used for the first layer MLP, the output of the second layer MLP is added to the input in a residual connection manner, and the final output is obtained, such as Figure 5 As shown, the whole process can be expressed as follows:

[0111] ;

[0112] ;

[0113] in, is the global feature extracted by two layers of MLP, is the output of the second layer MLP, is the output of the first layer of MLP, is a Gaussian random variable with zero mean and unit variance, is the probability, Mixed features The channel weighted result.

[0114] See also Figure 3 The working principle of the present invention is described below by using specific examples:

[0115] This example is based on two public marine mammal audio databases, and selects the sound signals of five large marine mammals, including sperm whales, pilot whales, humpback whales, bowhead whales, and bottlenose dolphins;

[0116] In the preprocessing, the Mel feature extraction was improved to make up for the shortcomings of FFT in capturing dynamic features. EMD was added before Fourier transform (FFT). The feature map was obtained by feature extraction of Mel and IMel to form a training data set, which was divided into training set, test set and validation set using the stratified k-cross validation method.

[0117] In order to enhance contrast training, the original features of the audio are directly extracted by the 1D-CNN method and then classified. Then, the Mel and IMel graph features are classified using 2D-CNN and CNN-LSTM. Finally, by combining the advantages of CNN and attention network, a classification method based on hybrid attention network and feature fusion is proposed.

[0118] This embodiment introduces a multimodal feature extractor, which aims to comprehensively utilize the advantages of the Mel spectrum and the original waveform, send the extracted features to the hybrid attention module, use the multi-head attention network to fuse the features and eliminate the influence of the number of channels, and finally use the multi-layer perceptron to extract global features and realize signal classification. The method includes the following steps:

[0119] (1) Propose a hybrid attention network and feature fusion model

[0120] The specific steps are as follows:

[0121] Step 1: Use the stratified k-cross validation method to divide the training set, test set, and validation set from one category each time, and then put the training set, test set, and validation set of all categories together to form the total training set, test set, and validation set. For example, if there are 1,000 sperm whale data, 200 data are selected for the test set each time. When k=5, exactly 1,000 data can be used for training, testing, and validation, so that the role of the data set can be fully utilized;

[0122] Step 2: Use Sym7, which has the characteristics of compact support, biorthogonality and approximate symmetry, to perform wavelet transform. The mother wavelet function is Symlet wavelet, the mother wavelet order is 7, the wavelet decomposition layer number is 5, and the threshold is 0.05;

[0123] Step 3: Use pre-emphasis to enhance the high-frequency components to make them more suitable for subsequent processing and analysis. The transfer function of pre-emphasis is for:

[0124] ;

[0125] in, represents the coefficient, The range is between 0 and 1, and is selected when pre-emphasis is performed. , the moment The voice input signal With the transfer function Multiply and get the output result As shown below:

[0126] ;

[0127] Step 4: Cut the signal after wavelet denoising into frames, and add a Hamming window to make the space between the two frames smooth, thereby reducing the spectrum leakage problem caused by the frame division.

[0128] Step 5: Use empirical mode decomposition (EMD) to compensate for the shortcomings of FFT in capturing dynamic features on the preprocessed (wavelet denoising, framing and windowing) data, and then perform fast Fourier transform (FFT), Mel filtering and logarithmic compression to obtain the characteristics of the signal's Mel spectrum and IMel spectrum;

[0129] Step 6: The multimodal feature extractor consists of two sub-networks, Spectrogram Net and Wave Net, which extract the time-frequency features. and the original waveform characteristics Extract it;

[0130] Step 7: Extract the and are respectively regarded as Q (query) and K (key) vectors in the attention mechanism, and the original waveform features and time-frequency characteristics Aligned in the time domain, where C is the number of channels and N is the time dimension;

[0131] Step 8: and The same size after feature extraction through MFE , and then through multi-head decomposition, we get The size of and After transposing, we get and , and then and The dot product calculation is performed to obtain the similarity matrix, and the attention matrix is ​​further obtained through the Softmax function. The size of the attention matrix is , that is, the fusion of local features is completed;

[0132] Step 9: Concatenate them in the channel dimension to get a size of of , and the size is obtained by projecting MLP Mixed features As a Value vector. The dot product with the attention matrix gives a size of The features are then obtained through global average pooling. , the final channel weighted result is , realizing the fusion of global features;

[0133] Step 10: MFE Receives Input Signal , and processed by two MLP Blocks. The GeLU nonlinear activation function is used for the first layer MLP. The output of the second layer MLP is added to the input in a residual connection manner to extract global features and obtain the final output.

[0134] (2) Other classification model designs

[0135] In order to enhance contrast training, different deep learning models are designed and selected to classify sound signals. The original features of the audio are directly extracted by the 1D-CNN method, and then classified. Then, 2D-CNN and CNN-LSTM are used to classify Mel and IMel graph features. The specific steps are as follows:

[0136] 1. Use one-dimensional convolutional neural network (1D-CNN) to classify the original waveform signal, which focuses on extracting local features of waveform data;

[0137] 2. Use two-dimensional convolutional neural network (2D-CNN) to classify Mel spectrum and IMel spectrum features, which mainly processes time-frequency feature maps and captures the spatial characteristics of the signal;

[0138] 3. Use the convolutional neural network and long short-term memory network hybrid model (CNN-LSTM) to classify the time-frequency features extracted from the Mel spectrum and IMel spectrum. Combine the features extracted by CNN with LSTM to further capture the time series dependency in the sound signal. Use CNN to extract spatial features and use LSTM to process the time dependency to finally complete the classification of the sound signal.

[0139] (3) Performance comparison

[0140] See also Figure 6 and Figure 7, the proposed hybrid attention network model is compared with other models to verify the superiority of the model, where Waveform represents the original waveform feature and Hybrid represents the hybrid feature. Each model is used to predict five different large marine mammals, including bowhead whales, bottlenose dolphins, humpback whales, pilot whales, and sperm whales. The comparison results are shown in Table 1:

[0141] Table 1 Comparison of the classification results of acoustic signals of five large marine mammals using different models

[0142]

[0143] Figure 6 The accuracy value in is used to measure the similarity between the model prediction and the real observation. The closer the value is to 1, the more accurate the prediction is. Figure 6 It can be seen that different models are used to predict the acoustic signal data of five large marine mammals. Among them, the hybrid attention network model based on mixed features has the most accurate prediction and the highest classification accuracy of acoustic signals.

[0144] In the test set, the classification accuracy of 1D-CNN for the original waveform features was 80.15%, the accuracy of 2D-CNN for Mel and IMel features was 82.1% and 85.2% respectively, and the accuracy of CNN-LSTM reached 84.3% and 87.1%;

[0145] like Figure 7 The T-SNE graph corresponding to the hybrid attention network in the DeepShip dataset can be used to observe the distribution of features learned by different models, such as Figure 7 As shown in the figure, compared with the classification results of residual network 18, regularized convolutional network, dynamic artificial intelligence network and other models, the classification accuracy of the hybrid attention network model reached 95.39%, which is better than traditional methods.

[0146] Furthermore, the present invention addresses the problems of inaccuracy and slow data processing speed when using features to restore sound. On the premise of improving the classification accuracy of different species of marine organisms, it combines the advantages of CNN and Transformer and the method of feature fusion, implements a parallel processing method, improves the classification processing efficiency, and greatly improves the accuracy of sound signal classification.

[0147] The present invention provides a method for classifying the acoustic signals of large marine mammals based on a hybrid attention network and feature fusion. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

[0148] It should be noted here that the present invention uses Mel and IMel feature extraction methods to extract features from the original waveform features to obtain the time-frequency features of the sound signal data, which can effectively capture the characteristics of the sound signal in the frequency domain and further improve the accuracy of classification; the time-frequency features and the original waveform features are input into the hybrid attention network and feature fusion model for global feature extraction, and the global features are used to classify the sound signal data, which solves the current technical problem of lacking the fusion of two different features, the time-frequency feature spectrogram and the waveform feature, greatly improves the classification efficiency and classification accuracy of the sound signal, and greatly improves the intelligence, usability and reliability of the present invention.

[0149] See also Figure 2 This embodiment provides a large marine mammal sound signal classification system, the system comprising:

[0150] Acquisition module 100: used to acquire acoustic signal data of different marine mammals;

[0151] Control module 200: used to preprocess the acoustic signal data, obtain original waveform features based on the preprocessed acoustic signal data, and use Mel and IMel to extract features from the original waveform features to obtain time-frequency features of the acoustic signal data; or used to input the time-frequency features and original waveform features into a hybrid attention network and feature fusion model to perform global feature extraction, and use the global features to perform signal classification on the acoustic signal data.

[0152] It should be noted here that the present invention uses Mel and IMel feature extraction methods to extract features from the original waveform features to obtain the time-frequency features of the sound signal data, which can effectively capture the characteristics of the sound signal in the frequency domain and further improve the accuracy of classification; the time-frequency features and the original waveform features are input into the hybrid attention network and feature fusion model for global feature extraction, and the global features are used to classify the sound signal data, which solves the current technical problem of lacking the fusion of two different features, the time-frequency feature spectrogram and the waveform feature, greatly improves the classification efficiency and classification accuracy of the sound signal, and greatly improves the intelligence, usability and reliability of the present invention.

[0153] In a preferred embodiment, the present application further provides an electronic device, the electronic device comprising:

[0154] A memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method for classifying large marine mammal sound signals is implemented. The computer device can be broadly a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, a memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the computer device can be used to provide necessary computing, processing and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory can provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the computer device can be used to connect and communicate with external devices via a network. When the computer program is executed by the processor, the steps of the method of the present invention are performed.

[0155] The present invention may be implemented as a computer-readable storage medium having a computer program stored thereon, which causes the steps of the method of an embodiment of the present invention to be executed when executed by a processor. In one embodiment, the computer program is distributed on a plurality of computer devices or processors coupled to a network so that the computer program is stored, accessed and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be performed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be performed by one or more computer devices or processors, and one or more other method steps / operations may be performed by one or more other computer devices or processors. One or more computer devices or processors may perform a single method step / operation, or perform two or more method steps / operations.

[0156] It should be noted here that the present invention uses Mel and IMel feature extraction methods to extract features from the original waveform features to obtain the time-frequency features of the sound signal data, which can effectively capture the characteristics of the sound signal in the frequency domain and further improve the accuracy of classification; the time-frequency features and the original waveform features are input into the hybrid attention network and feature fusion model for global feature extraction, and the global features are used to classify the sound signal data, which solves the current technical problem of lacking the fusion of two different features, the time-frequency feature spectrogram and the waveform feature, greatly improves the classification efficiency and classification accuracy of the sound signal, and greatly improves the intelligence, usability and reliability of the present invention.

[0157] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0158] The specific implementation of the present invention described above does not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for classifying acoustic signals of large marine mammals, characterized in that: include: S100, acquiring acoustic signal data of different marine mammals, and preprocessing the acoustic signal data; S200, obtaining original waveform features according to the preprocessed acoustic signal data, and extracting features from the original waveform features using Mel and IMel feature extraction methods to obtain time-frequency features of the acoustic signal data; S300, inputting the time-frequency features and the original waveform features into a hybrid attention network and feature fusion model to extract global features, and using the global features to perform signal classification on the acoustic signal data; The method of extracting the original waveform features by using the Mel and IMel feature extraction method to obtain the time-frequency features of the acoustic signal data includes: The Mel feature extraction method is used to extract the time-frequency features of the original waveform features. The Mel feature extraction method is based on short-time Fourier transform and Mel filter group to perform time-frequency analysis on the frequency of the original waveform features to obtain the Mel spectrum of the original waveform features; The method of extracting the original waveform features by using Mel and IMe1 to obtain the time-frequency features of the acoustic signal data further includes: The IMel feature extraction method is used to extract the time-frequency features of the original waveform features. The IMel feature extraction method is to add empirical mode decomposition on the basis of Mel feature extraction to obtain the IMel spectrum of the original waveform; The S300 includes: S310, using a multimodal feature extractor to respectively extract original waveform features and time-frequency features of the acoustic signal data; S320, inputting the original waveform features and time-frequency features extracted by the multimodal feature extractor into a hybrid attention module for feature fusion to obtain hybrid features; S330, using a multilayer perceptron to extract global features from the mixed features, and using the global features to classify the acoustic signal data; The method of extracting the original waveform features and the time-frequency features of the acoustic signal data by using a multimodal feature extractor comprises: The multimodal feature extractor includes a spectrogram network and a waveform network; The spectrogram network is used to extract the features of Mel spectrum and IMel spectrum, and the waveform network is used to extract the features of original waveform; The step of inputting the original waveform features and time-frequency features extracted by the multimodal feature extractor into a hybrid attention module for feature fusion includes: The original waveform features and the time-frequency features are locally fused using the multi-head attention mechanism of the hybrid attention module; The inputting the original waveform features and time-frequency features extracted by the multimodal feature extractor into the hybrid attention module for feature fusion also includes: Based on the local feature fusion of the original waveform features and time-frequency features, the global relationship of the features is extracted using a multi-head attention mechanism to perform global feature fusion.

2. The method for classifying large marine mammal sound signals according to claim 1, characterized in that: The method of using a multilayer perceptron to extract global features from the mixed features and performing signal classification on the global features includes: The number of multilayer perceptron blocks in the multilayer perceptron is a first preset value, the multilayer perceptron blocks are used to extract global features from the mixed features, and the global features are input into a classifier to classify the acoustic signal data.

3. A large marine mammal acoustic signal classification system, characterized in that: The method for classifying large marine mammal sound signals according to claim 1 or 2 comprises: Acquisition module: used to acquire acoustic signal data of different marine mammals; Control module: used to preprocess the acoustic signal data, obtain the original waveform features according to the preprocessed acoustic signal data, and use Mel and IMel to extract the features of the original waveform to obtain the time-frequency features of the acoustic signal data; or used to input the time-frequency features and the original waveform features into a hybrid attention network and feature fusion model to extract global features, and use the global features to perform signal classification on the acoustic signal data.

4. An electronic device, characterized in that: include: Memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method for classifying large marine mammal sound signals according to claim 1 or 2 is implemented.

Images