Marine mammal sound classification method for carrying out transfer learning based on human voice data

Abstract

Marine mammal sound lacks training data with annotations, so that a large-scale neural network model trained in a supervised mode is difficult to use at present. Inspired by the overlapping frequency of human sound and marine mammal sound, the Man2Marine method is invented, a large amount of unannotated human sound is used for pre-training a self-supervised large model, and then the sound of the marine mammal is used for fine tuning of the self-supervised large model. Meanwhile, in order to solve the problems of excessive parameterization and difficulty in application in a pre-training self-supervision large model, a knowledge distillation technology is used for compressing model parameters. According to the method, the cost of acquiring the sound data of the marine mammals is obviously reduced, and a method with practical application value is provided for research on the sound of the marine mammals.

Description

Technical field:

[0001] Underwater mammal sound classification is a specific application area in the field of underwater sound recognition, focusing on identifying and classifying the sounds made by mammals in underwater environments, such as whales, dolphins, etc. This field has important scientific, environmental protection and ecological research value, and also plays a key role in marine biology and marine ecology. Acoustic methods such as passive acoustic monitoring systems are widely used for sampling and monitoring ecosystems, measuring the range and seasonal occurrence of whales, and estimating the population density of certain marine animals. The large amount of acoustic data generated by passive acoustic monitoring systems is quite time-consuming and expensive if it relies on manual annotation. Therefore, it is the most basic and main task to automatically and accurately detect the calls of marine animals and classify them into the correct species.

[0002] Underwater sound signals are usually very complex and diverse, including different kinds of sounds, noise, various interference factors in the underwater environment, and the characteristics of sound signals changing over time. Deep learning methods are able to process these complex and diverse data, automatically extract features and capture patterns in the data, while traditional methods Traditional sound classification methods usually require manual selection and design of features, which can be very time-consuming and difficult when dealing with diverse and large-scale sound data. At the same time, deep learning methods support end-to-end learning, which means that the original sound data can be input into the model, and then the model can automatically complete feature extraction, classification, and decision-making without the need for manual feature engineering processes. This simplifies the entire sound classification process and improves efficiency. Therefore, deep learning networks have become the most important model for solving the task of marine mammal sound classification. Background technology:

[0003] Most of the current research in this field focuses on detecting specific target marine mammals, and has made considerable progress in detecting the sounds of specific single marine mammals. However, progress has been slow in the task of identifying multiple species of marine mammals. This is mainly because, first, the sound frequency range of different species of marine organisms is large, and traditional marine mammal sound classification methods find it difficult to find common features to characterize the signals of different marine mammals for classification, resulting in poor performance. Second, the acoustic data of marine mammals requires long-term surveys to obtain, and high-quality annotation requires deep expertise. The cost of labeling is often very expensive, resulting in scarce data in the field of bioacoustics. The large model in deep learning is data-driven. If the large model is used directly, it is very easy to fall into underfitting and it is difficult to achieve the desired effect. Therefore, these two factors have hindered the development of classification of sound signals of multiple species of marine mammals.

[0004] In recent years, many methods have been proposed to identify marine mammal calls, and most of the focus is on single target detection of specific species. So far, all methods can be divided into two categories: methods based on target feature extraction and image recognition methods based on spectrograms.

[0005] The first method is based on the method of target feature extraction, using a specific algorithm for feature or parameter extraction. This type of method does not extract the same features, but tries to make a unique parameter table for the animal signal of interest to match the known parameters. This type of method focuses on finding signals of interest in a characteristic bandwidth, but simply looks for signal energy surges in specific frequency bands according to different species, such as detecting the relative amplitude and shape of signals in certain frequency bands to detect specific organisms. The simplicity of this type of method makes them have a higher false alarm rate and poor reliability in noisy environments. In addition, expert knowledge is required to adjust the frequency of the filter for the detection of different organisms, which has poor flexibility.

[0006] The second type of method is the spectrogram-based recognition method, which is the current mainstream method in the field of marine mammal acoustic signal recognition. This type of method is based on the characteristics of different contours of marine organisms' acoustic signals after being converted into spectrograms. First, the audio is cut into segments of fixed length, then converted into spectrograms, and the image detection method is used to detect specific marine organism signals.

[0007] Compared with the acoustic signal recognition of marine mammals, speaker recognition has been more widely studied. In recent years, thanks to the emergence of high-quality human voice datasets and the optimization of model structures, end-to-end deep learning models can generate highly abstract embedding features, and their recognition accuracy has gradually exceeded that of traditional machine learning methods, becoming the mainstream method for speaker recognition. Among them, X-vector, Ecapa-tdnn and TitaNet are the mainstream baseline models in the field of speaker recognition.

[0008] X-vector is a deep neural network architecture commonly used in the field of voiceprint recognition. Thanks to the statistics pooling layer in its network, X-VECTORS can accept input of any length and convert it into a fixed-length feature expression; in addition, data enhancement strategies including noise and reverberation are introduced in training, and it is also robust to noise and language differences.

[0009] Ecapa-tdnn is an improvement on the model network of X-vector. The core idea is to aggregate and propagate features at different levels, pay more attention to the attention between channels, the propagation and aggregation of information, and add the Attention mechanism to the statictics pooling. In this way, it can more effectively capture the long-term changes of sound characteristics.

[0010] TitaNet is a speaker representation model proposed by NVIDIA. TitaNet introduces the global context Squeeze-and-Excitation (SE) layer and the channel attention pooling layer to better capture the global characteristics of the speaker. It uses 1D depthwise separable convolution to extract speaker representation, which has better performance and smaller model size than ECAPA-TDNN.

[0011] It is worth noting that the model in the field of human voiceprint is a data-driven model that requires a large amount of high-quality data. However, due to the size and quality of marine biological acoustic data, it is difficult to achieve the expected performance by directly applying the model in the field of human voiceprint to the recognition of marine biological acoustic signals. Summary of the invention:

[0012] In order to solve the problem of scarce data in the field of marine biological acoustics, one solution is self-supervision and transfer learning technology. In recent years, self-supervised learning has achieved great success in the field of speech. Using a large amount of unlabeled audio data to learn a general audio representation can benefit almost all audio downstream tasks through fine-tuning, reducing the data requirements for downstream tasks. Considering that the characteristics of some common marine biological audio and human voice have certain similarities (similar vibration frequencies), we considered using a large number of human voice datasets to self-supervise the model, and then fine-tuning it on the marine biological dataset, using self-supervision and transfer learning methods to solve the problem of using large models in the field of marine biological acoustic signal recognition. The main contributions are as follows:

[0013] (1) Migrate human voice data to efficiently solve the problem of marine life voiceprint recognition; the cost of collecting human voice data is much lower than that of marine life sound data, which can effectively reduce the cost of model training and obtain accurate recognition accuracy under the condition of smaller marine life data samples;

[0014] (2) The model has strong generalization ability. When new marine biological categories appear, this model can be quickly trained and accurately identified;

[0015] (3) By adopting knowledge distillation technology, the model achieves a balance between accuracy and speed. This balance has practical value for deployment and application in the field. Description of the drawings:

[0016] Figure 1 This is the overall flow chart of the proposed method framework;

[0017] Figure 2 The structure of the pre-trained model;

[0018] Figure 3 A flow chart of the method for voice knowledge transfer;

[0019] Figure 4 Schematic diagram of the knowledge distillation method. Specific implementation method:

[0020] The proposed system consists of three main steps: first, pre-training using a large amount of human speech data; second, transferring human voice knowledge to marine bioacoustics using transfer learning methods; and third, using knowledge distillation methods. Specifically, a self-supervised model is used in the initial step, which is pre-trained using a large amount of human speech data. In the second step, the pre-trained model is fine-tuned using a small marine mammal sound dataset to adapt the model to the task of marine mammal sound classification. Finally, the third step addresses the challenge of the over-parameterized nature of the pre-trained model, which hinders its practical applicability. To alleviate this issue, we adopt knowledge distillation techniques to transfer knowledge from the over-parameterized model to a leaner model, thereby achieving model compression and reducing computational requirements during inference. The subsequent subsections will elaborate on these three stages in more detail.

[0021] Step 1: Pre-training with human voice data

[0022] In the initial stage of the method, we adopt the speech self-supervised model WavLm to pre-train a large amount of human speech data. WavLm is a large-scale, self-supervised pre-training model designed for full-stack speech processing of human speech. WavLm improves on the Huber architecture of the speech self-supervised model and adopts the pre-training scheme of Denoising MaskedSpeech Modeling. As shown in the figure below, the WavLM model contains a CNN Encoder and a Transformer Encoder. The convolutional encoder has seven layers, each of which contains a time-domain convolution layer, a layer normalization layer, and a GELU activation function layer. In the Transformer encoder, a gated relative position bias is used to introduce relative positions into the calculation of the attention network in order to better model local information. During the training phase, WavLM randomly transforms the input wav, such as by mixing two wavs or adding background noise. Then, about 50% of the audio signal is randomly masked, and the label corresponding to the masked position is predicted at the output. WavLM follows the idea proposed by HuBERT, converts continuous signals into discrete labels through the Kmeans method, and models discrete labels as targets. Formally, given an input speech X, we first extract discrete labels z t ∈[C], then X is noised and masked to generate The mask prediction model is parameterized as as follows:

[0023]

[0024] in represents the hidden output of the masked speech frame, W is the projection matrix, and e c is the discrete label corresponding to z t The embedding of f It is necessary to predict the probability distribution of each frame in the masked set on the codebook based on the masked audio. The loss function used in pre-training is the mask prediction loss function as follows:

[0025]

[0026] Where M and Z represent the set of masked frames and corresponding labels, respectively. By optimizing this objective function, the model learns the ability to robustly represent speech during pre-training, which is beneficial to downstream tasks. In this study, the WavLm model was pre-trained using 94,000 hours of English speech data, making it the largest open source self-supervised speech model available. Unlike most speech self-supervised models that rely on e-book audiobook data, the training data used by the WavLm model contains more content and background noise from real-world scenes, thereby enhancing its generalization ability and providing greater robustness. It is worth noting that with the improvements in data augmentation and gated relative position deviation, the WavLm model will implicitly learn the ability to model speaker information during pre-training. Compared with other speech self-supervised models, it is more conducive to downstream tasks related to voiceprints.

[0027] Step 2: Transferring human vocal knowledge to marine bioacoustics

[0028] To adapt the task of marine mammal signal recognition, we fine-tuned the WavLm model in two main steps: audio signal data preprocessing and downstream task fine-tuning.

[0029] In the preprocessing stage, the selection of a standard duration is crucial. If the selected duration is too long, the audio will contain too much irrelevant information, resulting in inefficient use of training resources. On the contrary, if the duration is too short, the audio may lose a lot of useful content information, hindering the optimal extraction of audio features and negatively affecting recognition accuracy. Therefore, we analyzed the duration settings of all audio clips in the training and finally selected the median of 4 seconds as our standard duration. Next, we considered resampling the audio. The sampling rate refers to the sampling frequency of the audio signal saved per second. A higher sampling rate means more sampling points, allowing each audio signal detail to be captured with higher accuracy, thereby reducing any potential delays and distortions. In order to enable the neural network to obtain more comprehensive characteristic audio sequences of marine mammals and facilitate neural network processing, we selected a unified sampling rate signal of 16kHz for bioacoustics. Specifically, we used Wav2vec.FeatureExtractor to automatically extract features and preprocess marine bioacoustic signals. This includes completing resampling and cropping, and finally obtaining a 64,000-dimensional audio vector, which is used as the input of the WavLm model.

[0030] It is worth noting that the experiments study the contribution of different layers of WavLm to various tasks. The experiments show that the lower layers of WavLm are more influential in modeling speaker information while the upper layers are more beneficial for semantically related tasks. In view of this finding, instead of using the output of the last layer of WavLm as input in the downstream model, we introduce learnable weights "W". After assigning weights and summing the hidden representations according to different Transformer layers of WavLm, we determine the input of the downstream model.

[0031] In the downstream task fine-tuning stage, we adopt the model used in the classic TDNN (Time Delay Neural Network) voiceprint technology. The TDNN model includes a frame-level encoder for extracting biological information, and a pooling layer for converting said information into a fixed-dimensional representation. The final prediction is then obtained through a fully connected layer and AMsoftmax. Considering the size difference between WavLm and TDNN, different learning rates are assigned. Cross entropy loss is used to fine-tune the pre-trained WavLm model and the downstream TDNN model.

[0032] Step 3: Knowledge distillation compresses model parameters

[0033] Knowledge distillation is a technique in machine learning where a large, complex model, called a "teacher" model, is used to train a smaller, simplified model, called a "student" model. The goal of knowledge distillation is to transfer the knowledge and expertise of the teacher model to the student model so that the student model can perform with similar accuracy but with reduced complexity and computation.

[0034] In order to maximize the acquisition of knowledge from teacher demonstrations and improve generalization ability, for smaller models, it is necessary to set an appropriate distillation temperature, denoted as T distill . T distill The choice of T is crucial to the effectiveness of distillation and has a significant impact on the model’s category prediction. distill Then the category prediction probability of sample i is obtained through softmax operation:

[0035]

[0036] To optimize our knowledge distillation approach, we enforce certain constraints during the model training and distillation process. Specifically, if the value of the loss function keeps decreasing without any concomitant improvement in the accuracy of the training set, which indicates a potential overfitting scenario, we immediately terminate the training process. Furthermore, to fully exploit and leverage our informative dataset, we first train a student model using the training set. Next, we use the trained model as a new student model to further refine the knowledge:

[0037] Loss = (1-α) Loss soft +αLoss hard .

[0038] Subsequently, we transform the output of the teacher model to obtain the corresponding soft labels, which are combined with the hard labels to form the final distillation loss function. There are two key hyperparameters that need to be set during the distillation process: distillation temperature and label weight. The choice of distillation temperature is crucial to achieve effective distillation results. Too low a temperature cannot capture enough information from the soft labels, while too high a temperature reduces the utility of soft label information. In addition, different label weights have different effects on the training of the loss function. If the weight assigned to the soft label is too large, it means a stronger desire to train a closely related student model to approximate the teacher model. However, this may damage the robustness of the teacher model and make it difficult to achieve the desired distillation results. Similarly, if the weight assigned to the hard label is too prominent, it weakens the influence of the soft label, resulting in a distillation result similar to repeated training. In order to determine the optimal distillation temperature and label weight, the distillation temperature and weight need to be set here.

Claims

1. A marine mammal sound classification method based on transfer learning of human voice data, which includes pre-training using human voice data, transferring human voice knowledge to marine biological acoustics, and knowledge distillation compression model parameters, characterized by: Pre-training of human voice data includes the following six steps: (1) Collect 94,000 hours of human voice data, mainly including English speech data, including real-world scene content and background noise; (2) The continuous signal is converted into discrete labels through the Kmeans method and modeled as the classification label of the target audio; (3) Before the pre-training process, the input audio is enhanced and noise / overlapping speech is manually simulated as input; (4) The pre-training uses the WavLm model, which includes a CNN Encoder and a Transformer Encoder. The convolutional encoder has seven layers, each of which includes a temporal convolution layer, a layer normalization layer, and a GELU activation function layer. In the Transformer encoder, a gated relative position bias is used to introduce the relative position into the calculation of the attention network in order to better model local information. (5) During the pre-training process, formally, given the input speech X, we first extract the discrete label z t ∈[C] is a C-class categorical variable, and then X is noised and masked to generate The mask prediction model is parameterized as as follows: in represents the hidden output of the masked speech frame, W is the projection matrix, and e c is the discrete label corresponding to z t Embedding (6) Use the mask prediction loss function to optimize the model. The loss function used is the mask prediction loss function as follows: Where M and Z represent the set of masked frames and corresponding labels, respectively; The transfer of human vocal knowledge to marine bioacoustics involves the following two steps: (7) The sampling rate and duration of the acoustic audio collected by the underwater equipment are not consistent with the human voice audio during the pre-training process, so the underwater audio is first pre-processed; the duration settings of all audio clips in the training are analyzed and the median of 4 seconds is finally selected as the standard duration. At the same time, the data is downsampled to 16 kHz as the WavLm input audio data for the downstream fine-tuning stage; (8) In the fine-tuning stage, the downstream classification model adopts the classic TDNN (Time Delay Neural Network); the TDNN model includes a frame-level encoder for extracting biological information, and a pooling layer for converting the information into a fixed-dimensional representation; the final prediction is then obtained through a fully connected layer and AM softmax; considering the size difference between WavLm and TDNN, different learning rates are assigned to TDNN; cross entropy loss is used to fine-tune the pre-trained WavLm model and the downstream TDNN model; Knowledge distillation compresses model parameters in two steps: (9) In order to maximize the acquisition of teacher knowledge and improve generalization ability, for smaller models, it is necessary to set an appropriate distillation temperature T. distill , class probability prediction q i It can be expressed as: (10) During the distillation process, the output of the teacher model is transformed to obtain the corresponding soft label, which is combined with the hard label to form the final distillation loss function, which can be expressed as: Loss=(1-α)Loss soft +αLoss hard 。 2. The method for classifying marine mammal sounds by transfer learning based on human voice data according to claim 1, characterized in that: In the step (3) of pre-training the human voice data, the input audio is enhanced and noise / overlapping speech is manually simulated as input. The specific steps are as follows: (21) Randomly select some utterances from each training batch and mix them with randomly selected noise audio or secondary utterances; the noise audio and secondary utterances are randomly selected from the same batch of sounds, randomly cropped, and scaled according to the random source energy ratio, while ensuring that the overlapping area is less than 50%, and the speaker from the first utterance is taken as the main speaker; (22) Then, the data enhancement method is used, which mainly includes randomly transforming the input audio, adding background noise, and randomly shielding about 50% of the audio signal.

3. The method for classifying marine mammal sounds by transfer learning based on human voice data according to claim 1, characterized in that: The sampling rate and duration of the acoustic audio collected by the underwater equipment in the step (1) of migrating the human voice knowledge to marine bioacoustics are not consistent with the human voice audio in the pre-training process, so the underwater audio is first pre-processed, and the specific steps are as follows: (31) Experiments show that the lower layers of the pre-trained WavLM model are more influential in modeling speaker information, while the upper layers are more beneficial for semantic-related tasks. Therefore, in the downstream task fine-tuning stage, a learnable weight "W" is introduced to determine the input of the downstream model instead of using the output of the last layer of WavLM; (32) To enable the neural network to obtain more comprehensive characteristic audio sequences of marine mammals and facilitate neural network processing, we selected a uniform sampling rate signal of 16 kHz for bioacoustics. Specifically, we used Wav2vec.FeatureExtractor to automatically extract features and preprocess the marine bioacoustic signals, which included resampling and cropping, and finally obtained a 64,000-dimensional audio vector, which was used as the input of the WavLm model.

Images