A sperm whale call enhancement method based on VQ-MAE network

Abstract

The present invention relates to the field of audio signal processing, and specifically to a method for enhancing sperm whale calls based on a VQ-MAE network, including: Step 1: noise reduction processing of sperm whale call audio signals, and adjustment of the amplitude range of the audio signals; Step 2: processing spectrum images, and using the extracted feature vectors as part of the MAE network input; Step 3: processing the audio signals through an LSTM network, and extracting the temporal features of the original audio signals; Step 4: designing and training a multimodal mask encoder MAE network, and training fusion weights of different modes; Step 5: performing masking, and selecting the strategy with the best masking effect to train the network for data enhancement. The present invention innovatively introduces the time domain, frequency domain, and spatial domain features of the multimodal network fusion sound signal, and more accurately and deeply captures and enhances the semantic details in the sperm whale calls, providing strong support for the classification and recognition of the calls.

Description

Technical Field

[0001] The present invention relates to the field of audio signal processing, and in particular to a sperm whale call enhancement method based on a VQ-MAE network. Background Art

[0002] In recent years, with the development of deep learning technology, artificial intelligence-based sound signal processing methods have gradually become a hot topic of research, but there are still some key issues that need to be solved. First, many existing audio processing technologies rely heavily on large annotated data sets, which not only increases the cost of data collection and preprocessing stages, but in some cases, it is very difficult to obtain sufficient annotated data. Secondly, the processing of audio signals may perform well on specific data sets in existing models, but their generalization ability is often limited when faced with new and unseen data, which limits the effectiveness of the model in practical applications. Therefore, the combination of VQ-GAN and multimodal MAE network can effectively play their respective characteristics and achieve better results. The discretization characteristics of VQ perform well in learning discrete potential representations, which helps to capture the complex characteristics of audio signals, so it is more suitable for processing audio. VQ-GAN benefits from its efficient encoding and decoding mechanism, adversarial training strategy, diversified loss function, and powerful image generation and processing capabilities, so it is more suitable for processing images. When processing audio and signals, how to choose the appropriate enhancement strategy is undoubtedly a critical and complex issue. As a self-supervised learning method, the Masked Autoencoder MAE can effectively extract and compress image features without manual supervision, and can perfectly avoid the choice of strategy. It not only reduces the cost of data collection and preprocessing, but also uses the transformer-based model for pre-training to obtain a higher level of data representation, which is helpful for any downstream tasks.

[0003] It is difficult to obtain the calls of dolphins or sperm whales. In order to analyze and enhance the calls of sperm whales more deeply, an effective method is to convert the sound signal into a spectrogram. The spectrogram visualizes the time-frequency characteristics of the sound signal, making the key features such as the frequency distribution and harmonic structure in the call more intuitive and easier to process. Compared with directly processing one-dimensional sound signals, the spectrogram provides a richer information dimension, which helps the model capture the complex structure and dynamic changes in the call. Therefore, multimodality is used to fuse different types of data, and audio embedding and spectrogram are fused for cross-modal learning and reconstruction.

[0004] When using the above method, there are problems such as the uneven distribution of different types of sperm whale calls in the data set and the relatively limited data set. Summary of the invention

[0005] In view of the problems existing in the prior art, the purpose of the present invention is to provide a comprehensive framework based on a vector quantization generative adversarial network (VQ-GAN) and a multi-modal masked autoencoder (MAE), aiming to significantly improve the enhancement effect of sperm whale calls.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A sperm whale call enhancement method based on a VQ-MAE network comprises the following steps:

[0008] Step 1: Perform noise reduction processing on the sperm whale call audio signal, segment the audio signal, divide the continuous audio stream into short segments suitable for further analysis, and adjust the amplitude range of the audio signal through normalization processing for analyzing each call signal one by one;

[0009] Step 2: Use the pre-trained VQ-GAN model to process the input spectrum image, divide the image into multiple small blocks, and map these small blocks into a high-dimensional vector space to extract the feature vector of the input image. The extracted feature vector is used as part of the input of the subsequent multimodal mask encoder MAE network;

[0010] Step 3: Process the sperm whale call audio signal through the LSTM network, input the real part and imaginary part of the original audio signal into two independent LSTM layers to form a backbone network, and extract the time series features of the real part and imaginary part of the original audio signal respectively;

[0011] Step 4: Design and train a multimodal mask encoder MAE network, which accepts the feature vector extracted in step 2 and the temporal features extracted in step 3 as input, and trains the fusion weights of different modalities to maximize the fusion performance;

[0012] Step 5: Use different masking strategies for masking. Different masking strategies are adopted for different types of sperm whale calls, and a variable masking rate is used to train the multimodal mask encoder. The strategy with the best masking effect is selected to train the network for data enhancement.

[0013] In the above-mentioned sperm whale call enhancement method based on VQ-MAE network, step 1 comprises:

[0014] Step 1-1: Use mean filtering to take the average value of the neighborhood around the sperm whale call audio signal for noise reduction processing. For the nth discrete sample value signal , mean filtering can be expressed as: ,in, is the filtered signal, is the input discrete sample value signal exist The value at a time point, The value range is from arrive , is the size of the filter window, is half the width of the window;

[0015] Step 1-2: Pre-emphasize the audio signal to emphasize the high-frequency part. The pre-emphasis formula is: ,in, Represents the tth sampling point of the audio data, ɑ is a constant;

[0016] Step 1-3: Cut the denoised continuous audio stream into samples of 1 second in length, and adjust the amplitude range of the audio signal through normalization so that it falls within a uniform range. The formula is: , where X is the original data, is the normalized data, is the maximum value of the original data set, is the minimum value of the original data set.

[0017] In the above-mentioned sperm whale call enhancement method based on the VQ-MAE network, in step 2, the VQ-GAN model includes an encoder, a codebook and a decoder, the encoder and the decoder adopt a UNet structure, and the codebook is responsible for discretizing the continuous feature vector. During the training process, the encoder, the decoder and the codebook are updated together to minimize the reconstruction loss and the codebook loss;

[0018] Spectrum Image is a dimension of The input image to be processed, the encoder inputs the spectrum image Extract intermediate feature variables through CNN multi-layer convolutional layers and pooling layers Convert to continuous feature representation: ,in, represents the encoder, Represents the process of extracting features through multiple layers of convolutional layers and pooling layers;

[0019] A codebook is introduced to discretize the extracted image features. The image is sent to the decoder for image reconstruction.

[0020] In the sperm whale call enhancement method based on the VQ-MAE network, the codebook is a code containing A set of codewords, each codeword is a dimensional vector, codebook Its dimension is ,in is the number of vectors in the codebook, is the dimension of each vector;

[0021] The introducing of a codebook comprises:

[0022] Step a: Generate a discrete value codebook in advance: , ;

[0023] Step b: For feature variables Perform quantization operation to obtain the quantized feature vector , , yes The closest vector in the codebook;

[0024] Step c: Define the reconstruction loss : ,in, Representation Generator Accepts a quantized latent vector, as input and generate corresponding output.

[0025] In the above-mentioned sperm whale call enhancement method based on VQ-MAE network, step 2 comprises:

[0026] Step 2-1: Segment the sperm whale call audio signal into frames, determine the frame length, sampling rate and number of sampling points of each frame, and use overlapping frame sampling to ensure the smoothness of acoustic feature parameters;

[0027] Step 2-2: Apply the window function to each frame. The windowing calculation method is: ,in is a window function, is the sampling point within the frame;

[0028] Step 2-3: Fourier transform converts the time domain signal into the frequency domain signal. The basic formula is as follows: ,in, is a time domain signal, is the frequency domain signal, is the frequency;

[0029] Step 2-4: Convert the minimum and maximum frequencies to Mel-scale frequencies:

[0030] , for the power spectrum of each frame, calculate the energy of each Mel filter and add the energy in each filter;

[0031] Step 2-5: Take the logarithm of the filtered signal energy to obtain the logarithmic energy: ,in, is the energy of the filter bank;

[0032] Step 2-6: Perform discrete cosine transform on the logarithmic energy signal to obtain Mel frequency cepstral coefficients MFCC for better feature extraction: .

[0033] In the above-mentioned sperm whale call enhancement method based on VQ-MAE network, step 3 comprises:

[0034] Step 3-1: Convert the complex audio signal Decompose into real part and the imaginary part , each row represents a time step, and the dimension of the feature vector for each time step is , each column represents a feature dimension, the input matrix The dimension is ,in is the number of time steps;

[0035] Step 3-2: Get the real part of the input matrix as: , the input matrix of the imaginary part is: ;

[0036] Step 3-3: Use two independent LSTM layers that have been trained to process the real and imaginary parts of the sperm whale call signal respectively. The real part LSTM layer receives the normalized real part signal sequence as input, and the imaginary part LSTM layer receives the normalized imaginary part signal sequence, and outputs the hidden state of each time step respectively. The hidden state captures the timing characteristics of each signal.

[0037] In the above-mentioned sperm whale call enhancement method based on VQ-MAE network, step 4 comprises:

[0038] Step 4-1: Build a MAE network consisting of an encoder and a decoder. The encoder part consists of multiple fully connected layers to map multimodal features to the latent space, and the decoder part is used to reconstruct the original features from the latent space.

[0039] Step 4-2: Design a masking mechanism to randomly mask part of the input features during training, forcing the network to learn the ability to reconstruct complete features from partially visible features;

[0040] Step 4-3: A dynamic Gaussian mask strategy is introduced to train the multimodal mask encoder MAE network. The dynamic Gaussian mask strategy randomly samples the mask ratio from a truncated Gaussian distribution and randomly masks the input token sequence with this ratio. The selected token will be replaced with a learnable mask token. , a dynamic Gaussian mask strategy is used to dynamically adjust the mask ratio during training. The formula is: ,in represents a normal distribution, is the mean of the normal distribution, is the random sampling mask ratio in Gaussian distribution, Subordinate to the mean , the variance is Normal distribution of

[0041] Step 4-4: Input the masked token sequence into the MAE network and use the unmasked token information to reconstruct the masked token;

[0042] Step 4-5: Use mean square error MSE as the loss function and use the gradient descent algorithm to optimize the MAE network. , is the reconstruction loss, which represents the total loss of the entire batch of data, is the number of samples in the batch, It is The true value of the samples, It is The predicted value of samples, Represents the square of the Euclidean norm. In each iteration, the multimodal features are input into the network, the loss function is calculated, and the network weights are updated through back propagation.

[0043] In the above-mentioned sperm whale call enhancement method based on VQ-MAE network, in step 5, the types of sperm whale calls include concave calls, convex calls, sinusoidal calls, upward sweep calls, downward sweep calls and constant frequency calls. The concave calls, convex calls and sinusoidal calls adopt a path random masking strategy, and the upward sweep calls, downward sweep calls and constant frequency calls adopt a time level masking strategy.

[0044] In the sperm whale call enhancement method based on the VQ-MAE network, the path random masking strategy is to randomly walk on the spectrum graph from the initial time point to form a frequency sequence path connecting adjacent time points until a preset walk length is reached. ,path It is expressed as: ,in is the initial time point, For time point, is the frequency point;

[0045] The path is calculated using the Canny edge detection algorithm, including:

[0046] Step d: Use Gaussian filtering to smooth the image to reduce noise;

[0047] Step e: The algorithm calculates the gradient magnitude and direction of the image. The gradient represents the rate of change of the image intensity. The edge corresponds to the local maximum of the gradient. The gradient is calculated by applying the Sobel operator in the horizontal and vertical directions. The horizontal gradient of each pixel in the image is estimated by calculating the weighted difference of the adjacent pixels in the horizontal direction:

[0048] Vertical Gradient:

[0049] ,in, is the image at point The intensity value, that is, the pixel value of the point, Represents the horizontal gradient of the image, that is, the rate of change of the image intensity in the horizontal direction. Represents the vertical gradient of the image, that is, the rate of change of the image intensity in the vertical direction. and Respectively represent the image intensity in Direction and The partial derivative in the direction, that is, the rate of change of the gradient in these two directions, Indicates that the image is at point The pixel value of the current point is the pixel value to the upper left of the current point. Indicates that the image is at point The pixel value of , that is, the pixel value to the lower right of the current point;

[0050] Step f: Calculate the gradient magnitude and the gradient direction , ,

[0051] ,

[0052] Measured in The total amount of image intensity change at point, Indicates the direction of the edge;

[0053] Step g: At each pixel point, only the local maximum value in the gradient direction is retained, and other values ​​are suppressed. After calculating the gradient magnitude and gradient direction, check whether the gradient of each pixel point is the largest in its gradient direction. If so, the gradient of the point will be retained; if not, it will be suppressed;

[0054] Step h: Use two thresholds, a high threshold and a low threshold, to determine strong edges and weak edges. A strong edge is an edge whose gradient amplitude is higher than the high threshold, while a weak edge is an edge whose gradient amplitude is between the two thresholds.

[0055] Step i: Track edges through connectivity, connect strong edges and weak edges. If the gradient magnitude of a pixel is above a high threshold, it is determined to be an edge; if it is below a low threshold, it is not an edge; if it is in between, it depends on whether it is connected to a strong edge;

[0056] Step j: After determining the path, randomly mask the pixels on the path.

[0057] In the sperm whale call enhancement method based on the VQ-MAE network, the temporal masking strategy is to mask in a specific time or frequency region to highlight the key features of the call type. The masked time or frequency region can be expressed as:

[0058] in, Represents a collection of time points A specific point in time, Represents a set of frequency points A specific frequency point in is the set of time points selected for masking, is the set of frequency points selected for masking.

[0059] The beneficial effect of the sperm whale call enhancement method based on the VQ-MAE network of the present invention is that the method of the present invention shows extraordinary adaptability and superiority when processing limited or unbalanced dolphin data sets, especially providing strong support for the research of bionic covert underwater acoustic communication. VQ, with its unique discretization representation ability, can accurately capture the key features in the audio signal, while VQ-GAN combines the advantages of VQ and GAN, cleverly extending this advantage to the field of audio spectrum analysis, and using its powerful generation ability to delicately restore and enhance the detailed information in the spectrum. Through the integration and interaction of cross-modal information, the multimodal MAE framework not only improves the effect of single-modal processing, but also promotes the model's comprehensive understanding and response capabilities to complex acoustic environments, thereby achieving improvements in the specific task of sperm whale call enhancement.

[0060] The present invention effectively solves the problem that the distribution of different types of sperm whale calls in the data set may be uneven and the data set is relatively limited. The present invention uses the VQ-GAN network to extract semantic tags from images. It can accurately extract semantic tags from images and innovatively introduces a multimodal network to fuse the time domain, frequency domain, and spatial domain features of sound signals, thereby more accurately and deeply capturing and enhancing the semantic details in sperm whale calls, providing strong support for the classification and recognition of calls. Different masking strategies can be used for different call signals to better learn the features of the image and improve the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0062] Figure 2 It is the technical roadmap of the path random mask strategy of the present invention;

[0063] Figure 3 is a mask strategy diagram in one embodiment of the present invention;

[0064] Figure 4 1 is a structural diagram of a VQ-MAE network encoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood by those skilled in the art, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] Example 1

[0067] like Figure 1 As shown, a sperm whale call enhancement method based on a VQ-MAE network is used to preprocess sperm whale call samples; the audio data is processed through an LSTM network to obtain the time-frequency characteristics of the call signal; the VQ-GAN network divides the corresponding spectrum image into multiple small blocks and maps them to the vector space, extracts the features of the input image, and obtains the semantic labeling of the image; trains a multimodal mask encoder network; adopts different masking strategies for different call signals; and converts the sound signal into an image for reprocessing, which can intuitively and richly display the characteristics of the sound signal, including the following steps.

[0068] Step 1: The sperm whale call audio signal is subjected to noise reduction processing to eliminate interference from environmental noise and non-target sounds. Next, the audio signal is segmented to divide the continuous audio stream into shorter segments suitable for further analysis. Through normalization processing, the amplitude range of the audio signal is adjusted so that each call signal can be analyzed one by one.

[0069] Step 2: Use the pre-trained VQ-GAN model to process the input spectrum image, divide the image into multiple small blocks, and map these small blocks into a high-dimensional vector space to extract the feature vector of the input image. The extracted feature vector is used as part of the subsequent multimodal mask encoder MAE network input.

[0070] In the discretization representation of VQ-GAN images, the spectrogram is used as an input image to be processed" " , CNN extracts features through multiple layers of convolutional layers and pooling layers: ,in, represents the encoder, It represents the process of extracting features through multiple convolutional layers and pooling layers. The extracted image features need to be discretized through the codebook. , whose intermediate features The vector closest to its encoding position will be found in the codebook to obtain the quantized latent space , .

[0071] Step 3: Process the sperm whale call audio signal through the LSTM network, input the real part and imaginary part of the original audio signal into two independent LSTM layers to form a backbone network, and extract the time series features of the real part and imaginary part of the original audio signal respectively.

[0072] In this process, each row represents a time step and each column represents a feature dimension.

[0073] Decompose into real part and the imaginary part ,in, is the number of time steps, and the dimension of the feature vector for each time step is , the input matrix The dimension is . Therefore, the input matrices for the real and imaginary parts are:

[0074] ,

[0075] .

[0076] Step 4: Design and train a multimodal mask encoder MAE network, which accepts the feature vector extracted in step 2 and the temporal features extracted in step 3 as input, and trains the fusion weights of different modalities to maximize the fusion performance.

[0077] Step 5: Use different masking strategies for masking. Different masking strategies are adopted for different types of sperm whale calls, and a variable masking rate is used to train the multimodal mask encoder. The strategy with the best masking effect is selected to train the network for data enhancement.

[0078] In the masking strategy, masking processing is performed on different types of sperm whale calls. Sperm whale call types can be divided into upward sweeping, downward sweeping, concave, convex, constant frequency and sinusoidal shapes. Different masking strategies are selected for different call types. The upward sweeping, downward sweeping and sinusoidal shapes use the path random masking strategy, while the concave, convex and constant frequency types use the time level masking strategy.

[0079] As a structural masking strategy, the path random masking strategy uses the path as the basic processing unit in the sampling process, where the path can be defined as a series of continuous time points, each time point corresponds to a frequency distribution. This path can be regarded as a series of continuous spectrum "slices" on the time series.

[0080] By randomly walking on the spectrum graph from the initial time point, a frequency sequence path connecting adjacent time points is formed until the preset walking length is reached. , in order to enhance the model's ability to recognize call features.

[0081] Horizontal masking strategies mask in specific time or frequency regions to highlight key features of these call types.

[0082] Example 2

[0083] This embodiment provides a method for enhancing sperm whale calls based on a VQ-MAE network, which shows extraordinary adaptability and superiority when processing limited or unbalanced dolphin data sets, especially providing strong support for the research of bionic covert underwater acoustic communication. VQ, with its unique discretization representation ability, can accurately capture the key features in audio signals, while VQ-GAN combines the advantages of VQ and GAN, cleverly extending this advantage to the field of audio spectrum analysis, and using its powerful generation ability to delicately restore and enhance the detailed information in the spectrum. Through the integration and interaction of cross-modal information, the multimodal MAE framework not only improves the effect of single-modal processing, but also promotes the model's comprehensive understanding and response capabilities to complex acoustic environments, thereby achieving improvements in the specific task of sperm whale call enhancement.

[0084] The entire network structure is as follows Figure 1 As shown, the specific implementation is as follows.

[0085] First, the sperm whale call audio signal is subjected to noise reduction processing to eliminate interference from environmental noise and non-target sounds. Next, the audio signal is segmented to divide the continuous audio stream into shorter segments suitable for further analysis. Through normalization processing, the amplitude range of the audio signal is adjusted so that each call signal can be analyzed one by one.

[0086] In order to eliminate the interference of environmental noise and non-target sounds, the sperm whale call audio signal is first subjected to noise reduction processing. This step can be achieved through a variety of filtering techniques, such as mean filtering, median filtering, Gaussian filtering, etc. The present invention adopts mean filtering, and its basic idea is to reduce noise by taking the average value of the neighborhood around the audio signal. , mean filtering can be expressed as: ,in, is the filtered signal, is the input discrete sample value signal exist The value at a time point, The value range is from arrive , N is the size of the filter window, and M is half the width of the window.

[0087] Pre-emphasize the audio signal to emphasize the high frequency part. The pre-emphasis formula is:

[0088] ,in, Indicates the audio data sampling points, is a constant, usually taking a value between (0.95, 0.99).

[0089] The denoised continuous audio stream is cut into 1s samples suitable for further analysis. The amplitude range of the audio signal is adjusted through normalization so that it falls within a uniform range of [-1,1]. The formula is:

[0090] ,in, is the original data, is the normalized data, and are the maximum and minimum values ​​of the original data set, respectively.

[0091] Second, the input spectrum image is processed using the pre-trained VQ-GAN model, the image is divided into multiple small blocks, and these small blocks are mapped into a high-dimensional vector space to extract the feature vector of the input image as part of the subsequent multimodal mask encoder MAE network input.

[0092] The VQ-GAN network structure consists of three main parts: encoder (CNN Encoder), codebook (Codebook) and decoder (CNN Decoder). The encoder and decoder usually adopt the UNet structure, while the codebook is responsible for discretizing the continuous feature vector.

[0093] Spectrum Image is an input image to be processed, whose dimension is ,in is the image height, is the image width, is the number of color channels.

[0094] The encoder takes the input spectrum image Extract intermediate feature variables through CNN multi-layer convolutional layers and pooling layers Convert to continuous feature representation: ,in, represents the encoder, Represents the process of extracting features through multiple convolutional layers and pooling layers. After that, the extracted image features need to be discretized through a codebook. A codebook is introduced. The image is sent to the decoder for image reconstruction.

[0095] Codebook is a A set of codewords, each codeword is a dimensional vector. Define a codebook , whose dimensions are ,in is the number of vectors in the codebook, is the dimension of each vector.

[0096] The specific approach is to pre-generate a codebook of discrete values: , ,in, represents the codebook, is a vector in the codebook. Perform quantization operation to obtain the quantized feature vector , yes The closest vector in the codebook:

[0097] .

[0098] In order to make the encoded and decoded image similar to the original image, the reconstruction loss is defined .

[0099] ,in, Representation Generator Accepts a quantized latent vector as input and generate corresponding output.

[0100] During training, the encoder, decoder, and codebook of VQ-GAN are jointly updated to minimize the reconstruction loss and codebook loss. A Straight-Through Estimator (STE) is used to handle the non-differentiable problem of the quantization operation, so that the gradient can be passed from the decoder back to the encoder.

[0101] First, the audio signal is divided into frames, each frame is 25ms long, the sampling rate is 16kHz, that is, there are 400 sampling points. And the overlapping frame method is adopted, that is, there are overlapping parts between adjacent frames to ensure the smoothness of the acoustic feature parameters.

[0102] Apply a window function, such as a Hamming window or a Hanning window, to each frame to reduce spectral leakage. The windowing calculation is:

[0103] ,in is a window function, is the sampling point in the frame.

[0104] Fourier transform converts time domain signals into frequency domain signals. The basic formula is as follows:

[0105] ,in, is a time domain signal, is the frequency domain signal, It's the frequency.

[0106] Mel filter banks are used to convert frequency domain signals into Mel frequency domain signals. First, the minimum and maximum frequencies need to be converted into Mel-scale frequencies:

[0107] ,in, is the frequency in Hertz, is the corresponding Mel frequency. For each frame of the power spectrum, calculate the energy of each Mel filter and add the energy in each filter.

[0108] Taking the logarithm of the filtered signal energy, we get the logarithmic energy:

[0109] ,in, is the energy of the filter bank.

[0110] Perform discrete cosine transform on the logarithmic energy signal to obtain MFCC: ,in, is a logarithmic energy signal.

[0111] 3. The sperm whale call signal is processed through the LSTM network. The real and imaginary parts of the original signal are input into two independent LSTM layers to form a backbone network, and the temporal features of the real and imaginary parts of the signal are extracted respectively.

[0112] The extracted acoustic features are arranged in the order of frames to form sequence data. The sperm whale call signal is processed through the LSTM network. The real and imaginary parts of the original signal are input into two independent LSTM layers to form a backbone network, and the real and imaginary signals are normalized. Two independent LSTM networks are designed to process the real and imaginary sequences respectively. The two networks have no shared parameters, and the input layer of each LSTM network receives the normalized real or imaginary signal sequence.

[0113] In this process, each row represents a time step and each column represents a feature dimension.

[0114] Decompose into real part and the imaginary part , the dimension of the feature vector at each time step is , the input matrix The dimension is ,in is the number of time steps.

[0115] Therefore, the input matrices for the real and imaginary parts are:

[0116] ,

[0117] Two independent LSTM layers that have been trained are used to process the real and imaginary parts of sperm whale call signals respectively. The real part LSTM layer receives the normalized real part signal sequence as input, while the imaginary part LSTM layer receives the normalized imaginary part signal sequence. The hidden states of each time step are output respectively, and these hidden states capture the temporal characteristics of each signal. These features contain not only local patterns, but also long-term dependencies in the signal.

[0118] Fourth, design and train a multimodal mask encoder MAE network, such as Figure 4 As shown in Figure 1, the network accepts the discrete token sequence and quantized semantic tokens extracted in step 2 and the temporal features extracted in step 3 as input, and trains the fusion weights of different modalities to maximize the fusion performance.

[0119] Then, a MAE network consisting of an encoder and a decoder is constructed. The encoder part consists of multiple fully connected layers and is used to map multimodal features to the latent space. The decoder part is used to reconstruct the original features from the latent space. In addition, a mask mechanism is designed to randomly mask part of the input features during training, forcing the network to learn the ability to reconstruct complete features from partially visible features.

[0120] When training the Multimodal Mask Encoder (MAE) network, we introduce a dynamic Gaussian masking strategy, which randomly masks the input token sequence by randomly sampling the mask ratio from a truncated Gaussian distribution. The range of mask ratio is set to 0.5 to 1. First, the mask ratio is randomly sampled from a Gaussian distribution centered at 0.55, truncated at 0.5 on the left, and truncated at 1 on the right. This ratio determines how much of the input data will be masked. A dynamic Gaussian masking strategy is used to dynamically adjust the masking ratio during training. The formula is as follows:

[0121] , , given the input sequence length , according to the mask ratio obtained by sampling tokens to be masked. These selected tokens will be replaced with a learnable mask token The masked token sequence is input into the MAE network. The goal of the network is to reconstruct the masked token using the unmasked token information. represents a normal distribution, is the mean of the normal distribution, is the random sampling mask ratio in Gaussian distribution, Subordinate to the mean , the variance is The normal distribution of .

[0122] The mean square error (MSE) is used as the loss function, and the gradient descent algorithm is used to optimize the MAE network.

[0123] .in, is the reconstruction loss, which represents the total loss of the entire batch of data, is the number of samples in the batch, It is The true value of the samples, It is The predicted value of the sample, that is, the network output, Represents the square of the Euclidean norm, which is the sum of the squares of the differences of the vector elements.

[0124] In each iteration, the multimodal features are input into the network, the loss function is calculated, and the network weights are updated through back propagation. In order to improve the generalization ability of the model, an L2 regularization term is added to the loss function.

[0125] 5. Use different masking strategies for masking. Different masking strategies are adopted for different types of calls, and a variable masking rate is used to train the multimodal mask encoder. The strategy with the best masking effect is selected to train the network for data enhancement.

[0126] The corresponding masking strategy is selected according to the call type, and the multimodal mask encoder is trained with a variable masking rate to select the best masking strategy for data enhancement. The path random masking strategy is used for concave, convex and sinusoidal calls, and the time level masking strategy is used for upsweep, downsweep and constant frequency calls, such as Figure 3 As shown, different masking strategies are selected for different call types.

[0127] The random path mask strategy process is as follows Figure 2 As shown in Figure 2, the path random mask strategy forms a frequency sequence path connecting adjacent time points by randomly walking on the spectrum graph from the initial time point until the preset walking length is reached. , is the set of paths on the spectrum graph, For path , For time point , Frequency point .path It is expressed as:

[0128] ,in, is the initial time point, is the preset walk length.

[0129] The paths in the path random mask strategy can use algorithms such as the Canny edge detection algorithm, taking it as an example.

[0130] First, the image is smoothed using a Gaussian filter to reduce noise. Next, the algorithm calculates the magnitude and direction of the image's gradient. The gradient represents the rate at which the image intensity changes, and the edge usually corresponds to the local maximum of the gradient. The calculation of the gradient is usually done by applying the Sobel operator in the horizontal and vertical directions. The horizontal gradient and vertical gradient are as follows:

[0131] The horizontal gradient of each pixel in the image is estimated by calculating the weighted difference of its adjacent pixels in the horizontal direction:

[0132] ,

[0133] Vertical Gradient:

[0134] ,in, is the image at point The intensity value, that is, the pixel value of the point, Represents the horizontal gradient of the image, that is, the rate of change of the image intensity in the horizontal direction. Represents the vertical gradient of the image, that is, the rate of change of the image intensity in the vertical direction. and Respectively represent the image intensity in Direction and The partial derivative in the direction, that is, the rate of change of the gradient in these two directions, Indicates that the image is at point The pixel value of the current point is the pixel value to the upper left of the current point. Indicates that the image is at point The pixel value of , that is, the pixel value below and to the right of the current point.

[0135] Calculate the gradient magnitude and the gradient direction , the gradient magnitude represents the length of the gradient vector, and the gradient direction represents the angle between the gradient vector and the horizontal axis.

[0136] ,

[0137] .

[0138] Perform non-maximum suppression on the gradient amplitude in the gradient direction. Measured in The total amount of image intensity change at point, The direction of the edge is indicated. The Canny algorithm further ensures that the edge is thin-line-like through non-maximum suppression and double threshold detection. At each pixel, only the local maximum value in the gradient direction is retained, and other values ​​are suppressed. After calculating the gradient magnitude and direction, check whether the gradient of each pixel is the largest in its gradient direction. If so, the gradient of the point will be retained; if not, it will be suppressed. Two thresholds (high threshold and low threshold) are used to determine strong edges and weak edges. Strong edges are those with a gradient magnitude higher than the high threshold, while weak edges are those with a gradient magnitude between the two thresholds. Tracking edges through 8-connectivity connects strong edges and weak edges. If the gradient magnitude of a pixel is higher than the high threshold, it is determined to be an edge; if it is lower than the low threshold, it is not an edge; if it is between the two, it depends on whether it is connected to a strong edge. 8-connectivity is a concept in image processing that describes the connection relationship between pixels.

[0139] After the path is determined, the pixels on the path are randomly masked, while the temporal masking strategy is to mask in a specific time or frequency region to highlight the key features of the call type. The masked time or frequency region can be expressed as:

[0140] ,in, Represents a collection of time points A specific point in time, Represents a set of frequency points A specific frequency point in is the set of time points selected for masking, is the set of frequency points selected for masking.

[0141] The present invention provides a sperm whale call enhancement method based on a multimodal VQ-MAE network, aiming to solve the problem of obtaining sperm whale call data in the prior art. The method first preprocesses the sperm whale call samples, and then processes the audio data through an LSTM network to obtain the time-frequency characteristics of the call signal. Next, the VQ-GAN network divides the corresponding spectrum image into multiple small blocks and maps them to the vector space to extract the semantic tags of the image. On this basis, we train a multimodal mask encoder network and adopt different masking strategies for different call signals. By converting the sound signal into an image for reprocessing, this method can intuitively and richly display the characteristics of the sound signal, providing a new perspective for the analysis and enhancement of sperm whale calls.

[0142] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0143] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A sperm whale call enhancement method based on VQ-MAE network, characterized in that: The following steps are involved: Step 1: Perform noise reduction processing on the sperm whale call audio signal, segment the audio signal, divide the continuous audio stream into short segments suitable for further analysis, and adjust the amplitude range of the audio signal through normalization processing for analyzing each call signal one by one; Step 2: Use the pre-trained VQ-GAN model to process the input spectrum image, divide the image into multiple small blocks, and map these small blocks into a high-dimensional vector space to extract the feature vector of the input image. The extracted feature vector is used as part of the input of the subsequent multimodal mask encoder MAE network; Step 3: Process the sperm whale call audio signal through the LSTM network, input the real part and imaginary part of the original audio signal into two independent LSTM layers to form a backbone network, and extract the time series features of the real part and imaginary part of the original audio signal respectively; Step 4: Design and train a multimodal mask encoder MAE network, which accepts the feature vector extracted in step 2 and the temporal features extracted in step 3 as input, and trains the fusion weights of different modalities to maximize the fusion performance; Step 5: Use different masking strategies for masking. Different masking strategies are adopted for different types of sperm whale calls, and a variable masking rate is used to train the multimodal mask encoder. The strategy with the best masking effect is selected to train the network for data enhancement.

2. The method for enhancing sperm whale calls based on a VQ-MAE network according to claim 1, characterized in that: The step 1 comprises: Step 1-1: Use mean filtering to take the average value of the neighborhood around the sperm whale call audio signal for noise reduction processing. For the nth discrete sample value signal , mean filtering can be expressed as: ,in, is the filtered signal, is the input discrete sample value signal exist The value at a time point, The value range is from arrive , is the size of the filter window, is half the width of the window; Step 1-2: Pre-emphasize the audio signal to emphasize the high-frequency part. The pre-emphasis formula is: ,in, Represents the tth sampling point of the audio data, ɑ is a constant; Step 1-3: Cut the denoised continuous audio stream into samples of 1 second in length, and adjust the amplitude range of the audio signal through normalization so that it falls within a uniform range. The formula is: , where X is the original data, is the normalized data, is the maximum value of the original data set, is the minimum value of the original data set.

3. The method for enhancing sperm whale calls based on a VQ-MAE network according to claim 1, characterized in that: In step 2, the VQ-GAN model includes an encoder, a codebook and a decoder. The encoder and the decoder adopt a UNet structure. The codebook is responsible for discretizing the continuous feature vector. During the training process, the encoder, the decoder and the codebook are updated together to minimize the reconstruction loss and the codebook loss. Spectrum Image is a dimension of The input image to be processed, where H is the image height, W is the image width, and C is the number of color channels. The encoder converts the input spectrum image Extract intermediate feature variables through CNN multi-layer convolutional layers and pooling layers Convert to continuous feature representation: ,in, represents the encoder, Represents the process of extracting features through multiple layers of convolutional layers and pooling layers; A codebook is introduced to discretize the extracted image features. The image is sent to the decoder for image reconstruction.

4. The method for enhancing sperm whale calls based on a VQ-MAE network according to claim 3, characterized in that: The codebook is a A set of codewords, each codeword is a dimensional vector, codebook Its dimension is ,in is the number of vectors in the codebook, is the dimension of each vector; The introducing of a codebook comprises: Step a: Generate a discrete value codebook in advance: , ; Step b: For feature variables Perform quantization operation to obtain the quantized feature vector , , yes The closest vector in the codebook; Step c: Define the reconstruction loss : ,in, Representation Generator Accepts a quantized latent vector, as input and generate corresponding output.

5. The method for enhancing sperm whale calls based on a VQ-MAE network according to claim 4, characterized in that: The step 2 comprises: Step 2-1: Segment the sperm whale call audio signal into frames, determine the frame length, sampling rate and number of sampling points of each frame, and use overlapping frame sampling to ensure the smoothness of acoustic feature parameters; Step 2-2: Apply the window function to each frame. The windowing calculation method is: ,in is a window function, is the sampling point within the frame; Step 2-3: Fourier transform converts the time domain signal into the frequency domain signal. The basic formula is as follows: ,in, is a time domain signal, is the frequency domain signal, is the frequency; Step 2-4: Convert the minimum and maximum frequencies to Mel-scale frequencies: , for the power spectrum of each frame, calculate the energy of each Mel filter and add the energy in each filter; Step 2-5: Take the logarithm of the filtered signal energy to obtain the logarithmic energy: ,in, is the energy of the filter bank; Step 2-6: Perform discrete cosine transform on the logarithmic energy signal to obtain Mel frequency cepstral coefficients MFCC for better feature extraction: .

6. The method for enhancing sperm whale calls based on a VQ-MAE network according to claim 1, characterized in that: The step 3 comprises: Step 3-1: Convert the complex audio signal Decompose into real part and the imaginary part , each row represents a time step, and the dimension of the feature vector for each time step is , each column represents a feature dimension, the input matrix The dimension is ,in is the number of time steps; Step 3-2: Get the real part of the input matrix as: , the input matrix of the imaginary part is: ; Step 3-3: Use two independent LSTM layers that have been trained to process the real and imaginary parts of the sperm whale call signal respectively. The real part LSTM layer receives the normalized real part signal sequence as input, and the imaginary part LSTM layer receives the normalized imaginary part signal sequence, and outputs the hidden state of each time step respectively. The hidden state captures the timing characteristics of each signal.

7. The method for enhancing sperm whale calls based on a VQ-MAE network according to claim 1, characterized in that: The step 4 comprises: Step 4-1: Build a MAE network consisting of an encoder and a decoder. The encoder part consists of multiple fully connected layers to map multimodal features to the latent space, and the decoder part is used to reconstruct the original features from the latent space. Step 4-2: Design a masking mechanism to randomly mask part of the input features during training, forcing the network to learn the ability to reconstruct complete features from partially visible features; Step 4-3: A dynamic Gaussian mask strategy is introduced to train the multimodal mask encoder MAE network. The dynamic Gaussian mask strategy randomly samples the mask ratio from a truncated Gaussian distribution and randomly masks the input token sequence with this ratio. The selected token will be replaced with a learnable mask token. , a dynamic Gaussian mask strategy is used to dynamically adjust the mask ratio during training. The formula is: ,in represents a normal distribution, is the mean of the normal distribution, is the random sampling mask ratio in Gaussian distribution, Subordinate to the mean , the variance is Normal distribution of Step 4-4: Input the masked token sequence into the MAE network and use the unmasked token information to reconstruct the masked token; Step 4-5: Use mean square error MSE as the loss function and use the gradient descent algorithm to optimize the MAE network. , is the reconstruction loss, which represents the total loss of the entire batch of data, is the number of samples in the batch, It is The true value of the samples, It is The predicted value of a sample, Represents the square of the Euclidean norm. In each iteration, the multimodal features are input into the network, the loss function is calculated, and the network weights are updated through back propagation.

8. The method for enhancing sperm whale calls based on a VQ-MAE network according to claim 1, characterized in that: In step 5, the types of sperm whale calls include concave calls, convex calls, sinusoidal calls, upward sweep calls, downward sweep calls and constant frequency calls. The concave calls, convex calls and sinusoidal calls use a path random masking strategy, and the upward sweep calls, downward sweep calls and constant frequency calls use a time level masking strategy.

9. The method for enhancing sperm whale calls based on a VQ-MAE network according to claim 8, characterized in that: The path random masking strategy is to randomly walk on the spectrum graph from the initial time point to form a frequency sequence path connecting adjacent time points until the preset walk length is reached. ,path It is expressed as: ,in is the initial time point, For time point, is the frequency point; The path is calculated using the Canny edge detection algorithm, including: Step d: Use Gaussian filtering to smooth the image to reduce noise; Step e: The algorithm calculates the gradient magnitude and direction of the image. The gradient represents the rate of change of the image intensity. The edge corresponds to the local maximum of the gradient. The gradient is calculated by applying the Sobel operator in the horizontal and vertical directions. The horizontal gradient of each pixel in the image is estimated by calculating the weighted difference of the adjacent pixels in the horizontal direction: Vertical Gradient: ,in, is the image at point The intensity value, that is, the pixel value of the point, Represents the horizontal gradient of the image, that is, the rate of change of the image intensity in the horizontal direction. Represents the vertical gradient of the image, that is, the rate of change of the image intensity in the vertical direction. and Respectively represent the image intensity in Direction and The partial derivative in the direction, that is, the rate of change of the gradient in these two directions, Indicates that the image is at point The pixel value of the current point is the pixel value to the upper left of the current point. Indicates that the image is at point The pixel value of , that is, the pixel value to the lower right of the current point; Step f: Calculate the gradient magnitude and the gradient direction , , ， Measured in The total amount of image intensity change at point, Indicates the direction of the edge; Step g: At each pixel point, only the local maximum value in the gradient direction is retained, and other values ​​are suppressed. After calculating the gradient magnitude and gradient direction, check whether the gradient of each pixel point is the largest in its gradient direction. If so, the gradient of the point will be retained; if not, it will be suppressed; Step h: Use two thresholds, a high threshold and a low threshold, to determine strong edges and weak edges. A strong edge is an edge whose gradient amplitude is higher than the high threshold, while a weak edge is an edge whose gradient amplitude is between the two thresholds. Step i: Track edges through connectivity, connect strong edges and weak edges. If the gradient magnitude of a pixel is above a high threshold, it is determined to be an edge; if it is below a low threshold, it is not an edge; if it is in between, it depends on whether it is connected to a strong edge; Step j: After determining the path, randomly mask the pixels on the path.

10. The method for enhancing sperm whale calls based on a VQ-MAE network according to claim 8, characterized in that: The temporal level masking strategy is to perform masking in a specific time or frequency region to highlight the key features of the call type. The masked time or frequency region can be expressed as: in, Represents a collection of time points A specific point in time, Represents a set of frequency points A specific frequency point in is the set of time points selected for masking, is the set of frequency points selected for masking.

Images