Visual Transform-based birdsong recognition method

Abstract

The invention discloses a birdsong recognition method based on a visual Transform, and the method comprises the steps: S1, carrying out the preprocessing of an original birdsong audio signal, and generating a Mel spectrogram; s2, constructing a birdsong recognition model and training the birdsong recognition model; s3, inputting the Mel spectrogram into a trained birdsong recognition model, extracting features and classifying the features, and outputting probability distribution of bird species; and S4, according to the probability distribution of the bird species, determining the identified bird species and the confidence coefficient thereof, and performing visual output and report generation. According to the method, on the basis of keeping the global modeling advantage of the visual Transform, a novel twitter recognition model which is stronger in detail perception, more intelligent in feature fusion and more efficient in calculation is constructed, and the recognition performance and practicability are comprehensively improved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and bioacoustics, and in particular to a method for bird call recognition based on a visual Transformer. Background Technology

[0002] The Visual Transformer has achieved revolutionary success in image recognition by segmenting images into patches and globally modeling contextual information using a self-attention mechanism. Inspired by this, existing techniques have emerged that convert audio signals into time-frequency maps and input them as two-dimensional images into the Visual Transformer for recognition. This approach transforms the audio classification problem into an image classification problem and has initially demonstrated its potential to outperform traditional RNN and CNN models.

[0003] However, directly applying visual Transformer models designed for natural images to bird song time-frequency map recognition has a series of inherent drawbacks. First, the patch embedding process of the standard visual Transformer is essentially a downsampling operation, which causes subtle features crucial for species differentiation in the time-frequency map (such as start / end edges of bird songs, harmonic structures, etc.) to be smoothed out early in the modeling process. The standard visual Transformer lacks an explicit mechanism to enhance and preserve these key low-level features, resulting in a decline in the model's ability to distinguish species with similar acoustic features. Second, different levels of the visual Transformer capture features at different levels of abstraction. Shallow features contain rich details and edge information but are sensitive to noise, while deep features contain high-level semantic information but have low spatial resolution and lose details. Existing methods lack an adaptive, non-linear fusion mechanism to optimize the use of full-scale feature information from low to high, limiting the model's overall understanding of complex song structures. Furthermore, the computational complexity of the standard self-attention mechanism is proportional to the square of the sequence length, which is extremely computationally expensive for high-resolution time-frequency maps. In addition, a large number of image patches with background noise or silent regions participate in redundant attention calculations, which distracts the model from focusing on key acoustic events and reduces the model's efficiency and generalization performance. Summary of the Invention

[0004] To address the inherent limitations of existing visual Transformer-based bird call recognition methods when processing time-frequency images, this invention proposes a visual Transformer-based bird call recognition method. This method effectively enhances the extraction capabilities of details and edge features, adaptively fuses multi-scale features, and intelligently focuses on key information to reduce computational redundancy, thereby achieving high-precision and high-efficiency automatic bird call recognition.

[0005] This application discloses a bird call recognition method based on visual Transformer, including the following steps: S1. Preprocess the original bird song audio signal to generate a Mel spectrogram; S2. Construct and train a bird call recognition model. The bird call recognition model includes a backbone layer, a hierarchical structure and a classification head connected in sequence. The hierarchical structure includes multiple stages. An embedding layer connects two adjacent stages. The input of the first stage is connected to the backbone layer, and the output of the last stage is connected to the classification head. Each stage includes multiple improved Transformer blocks. S3. Input the Mel spectrogram into the trained bird call recognition model, extract features and classify them, and output the probability distribution of bird species; S4. Based on the probability distribution of the bird species, determine the identified bird species and their confidence levels, and generate visualization output and reports.

[0006] Preferably, generating the Mel spectrogram includes the following steps: The original bird song audio signal is subjected to noise reduction and enhancement processing, which includes one or more random combinations of the following operations: gain adjustment, adding Gaussian noise, adding short noise, adding background noise, and low-pass filtering; The audio signal after noise reduction and enhancement is framed and windowed, and then subjected to short-time Fourier transform to obtain the spectrogram. The spectrogram is mapped to a Mel scale using a Mel filter bank to generate a Mel spectrogram. The Mel spectrogram is standardized and normalized, and the single-channel Mel spectrogram is expanded into a three-channel spectrogram.

[0007] Preferably, each of the improved Transformer blocks includes the following workflow: The input features are linearly transformed to generate query vectors. Key vector Sum value vector ; For query vector and key vector Spatial attention and channel attention are applied, and the initial context feature representation is obtained through an additive similarity function; The initial contextual feature representation is input into the multi-scale edge enhancement module to enhance details and edge features in the time-frequency map; The output of the multi-scale edge enhancement module is input in parallel to the dynamic feature fusion module and the Top-k sparse attention module; The output of the dynamic feature fusion module is concatenated with the output of the Top-k sparse attention module, and after convolutional fusion, it is residually connected with the output of the multi-scale edge enhancement module to obtain the enhanced contextual feature representation. Enhanced contextual feature representation with value vector Combined, the final processed features are output.

[0008] Preferably, after the multiple improved Transformer blocks are stacked, higher-level features are extracted step by step to finally obtain the feature output of the current stage; Spatial downsampling and channel upsampling are performed between adjacent stages through embedding layers. Through hierarchical downsampling and feature transformation, the final hierarchical structure outputs global features.

[0009] Preferably, the classification head passes global features through GeM pooling and multi-layer Dropout averaging before entering a fully connected layer to output the probability of bird species.

[0010] Preferably, the workflow of the multi-scale edge enhancement module includes: Local features of the input features are extracted through convolutional layers; Features of different scales are generated through average pooling and convolution operations; An edge enhancer is applied to the features at each scale. The edge enhancer enhances the edge response by differencing the features at a certain scale with the average pooling result and then processing them through convolution and activation functions. The local features are concatenated with all features processed by the edge enhancer, and then fused through convolution to finally output the enhanced features.

[0011] Preferably, the dynamic feature fusion module includes a channel dynamic selection submodule and a spatial dynamic selection submodule, and the workflow of the dynamic feature fusion module includes: Multi-scale downsampling is performed on the enhanced features output by the multi-scale edge enhancement module to obtain feature maps at different scales; The feature maps of different scales are input into the channel dynamic selection submodule, the channel dimensions are spliced, the spatial dimension is compressed by global average pooling, channel-level statistical information is generated, channel weights are generated, and finally the channel weights are calibrated and dimensionality reduced to obtain the dimensionality-reduced channel weights. Input feature maps of different scales into the spatial dynamic selection submodule to generate spatial weights; The spatial weights output by the spatial dynamic selection submodule and the dimensionality-reduced channel weights output by the channel dynamic selection submodule are weighted element-wise to generate the final fused features.

[0012] Preferably, the Top-k sparse attention module includes a dynamic k-value selection submodule and a multi-scale fusion submodule. The workflow of the Top-k sparse attention module includes: Calculate query vector With key vector The similarity matrix; For each query row in the similarity matrix, only the top k similarity values ​​in that row are retained, and the rest are set to zero to generate a sparse attention matrix; sparse attention matrix and value vector Weighted aggregation is performed to obtain sparse attention output; By fusing the sparse attention output with the input through residual connections, an enhanced representation is obtained.

[0013] Preferably, the training steps of the bird call recognition model include: Prepare the training dataset: Collect bird call audio with bird species labels and generate the corresponding Mel spectrograms; Set training parameters: Use the AdamW optimizer, adjust the learning rate using linear warm-up and cosine annealing strategies, and use the cross-entropy loss function or the labeled smooth cross-entropy loss function. Training process: Input training data into the bird song recognition model for forward computation, calculate the loss value through the loss function, update the model parameters through backpropagation, and use gradient clipping, Dropout and exponential moving average strategies to stabilize training. Validation and evaluation: Evaluate the model performance on an independent validation set and save the weights of the best-performing model.

[0014] Preferably, it is implemented based on a bird call recognition system, the bird call recognition system comprising: The data preprocessing module is used to convert the input raw bird song audio signal into a standardized Mel spectrogram; The model inference module is loaded with a trained bird call recognition model, which is used to extract features and classify the Mel spectrogram, and output the probability distribution of bird species. The application output module is used to determine the identified bird species and their confidence levels based on the probability distribution, and to generate visualizations and reports. The application output module includes: A graphical user interface for uploading audio files and displaying recognition results; A sliding window detection unit is used for automatic segmentation and recognition of long audio files; Visualization units are used to mark the time periods and species labels of bird song events on the time-frequency graph; The report generation unit is used to export the recognition results into a structured report file.

[0015] The beneficial effects of this invention are: (1) Higher recognition accuracy: Through multi-scale edge enhancement and dynamic feature fusion, the ability to distinguish subtle acoustic features is significantly improved.

[0016] (2) Stronger model adaptability: The dynamic fusion mechanism enables the model to adaptively process different kinds of bird calls.

[0017] (3) Higher computational efficiency: The Top-k sparse attention mechanism significantly reduces computational complexity and improves inference speed.

[0018] (4) Better noise resistance: The sparse attention mechanism enables the model to focus on key acoustic events and reduce background noise interference.

[0019] (5) Stronger generalization ability: It avoids manual feature engineering, and the model learns the most effective features directly from the data. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the bird call recognition method based on visual Transformer according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the bird call recognition model structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the improved Transformer block structure according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the multi-scale edge enhancement module structure according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the edge enhancer structure according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the dynamic feature fusion module structure according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the Top-K sparse attention module structure according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments.

[0022] This application discloses a bird call recognition method based on visual Transformer, the process of which is as follows: Figure 1 As shown, it includes the following steps: S1. Preprocess the original bird song audio signal to generate a Mel spectrogram.

[0023] S11. Denoising and enhancement processing is performed on the original bird song audio signal to improve the signal-to-noise ratio. The processing includes one or more random combinations of the following operations: gain adjustment, adding Gaussian noise, adding short noise, adding background noise, and low-pass filtering. In this embodiment, the following denoising algorithms are combined to significantly increase the diversity of training data, helping the model better generalize to various audio conditions in real-world environments. Each operation has a certain probability of being applied, making the generated enhanced samples more diverse. For example, in this embodiment, the probability of each operation being applied is 20%.

[0024] Gain Adjustment (20% probability): Randomly selects one of two gain adjustment methods: random gain adjustment and gain transition. Random gain adjustment directly modifies the amplitude of the audio signal, simulating different audio input volumes by randomly selecting a gain value between -15dB and 15dB and applying it to the entire audio. Gain transition achieves a smooth transition by dynamically adjusting the audio gain, with the gain value varying between -24dB and 6dB. The transition duration is randomly selected from 0.2 to 6 seconds to simulate natural volume changes during audio playback.

[0025] Gaussian noise addition (probability 20%): Randomly select one of two Gaussian noise methods: adding Gaussian noise and adding Gaussian signal-to-noise ratio noise. Adding Gaussian noise generates random Gaussian-distributed noise and superimposes it onto the audio to simulate background noise from electronic devices. Adding Gaussian signal-to-noise ratio noise generates and superimposes Gaussian noise based on the signal-to-noise ratio, allowing for more precise control of noise intensity and simulating recording environments of varying quality.

[0026] Short Noise Addition (20% probability): Loads short noise clips from a specified folder and inserts them into the audio. Specifically, it loads noise from the short noise folder, with the signal-to-noise ratio controlled between 0-3dB, to simulate sudden, brief noise in a real-world environment.

[0027] Background noise addition (20% probability): Loads long background noise and mixes it with the original audio. Specifically, it loads background noise from the background noise folder, with the signal-to-noise ratio controlled between 0-3dB, to simulate persistent environmental background noise.

[0028] Low-pass filtering (20% probability): Low-frequency signals are allowed to pass through while high-frequency signals are attenuated, simulating the frequency response changes of audio after passing through different devices (such as telephones and speakers).

[0029] S12. The audio signal after noise reduction and enhancement is framed and windowed. The continuous audio signal is divided into short-term overlapping frames, and windowing is performed using functions such as Hamming window to reduce spectral leakage.

[0030] S13. Perform time-frequency transformation: Perform a short-time Fourier transform on each frame of audio to convert the signal from the time domain to the frequency domain and obtain a spectrum.

[0031] S14. Perform Mel-scale transformation, which maps the linear spectrum to a Mel scale that better matches the characteristics of human hearing, generating a Mel spectrogram. This step is crucial; it transforms the audio signal into a high-quality "image." The detailed steps for generating a Mel spectrogram are as follows: Mel filtering: This method uses a set of triangular filters (Mel filter bank) based on the Mel scale to filter the spectrum. The Mel scale is a non-linear scale that simulates the human ear's perception of different frequencies of sound, and can better reflect the characteristics of human hearing, ultimately yielding the Mel spectrum.

[0032] Decibels conversion: Converts the amplitude values ​​of the Mel spectrum to decibel (dB) scale, compresses the dynamic range through logarithmic operations to better match the human ear's perception of sound intensity, and sets a threshold to cut off extreme values.

[0033] Through these steps, the original audio waveform is converted into a Mel spectrogram that reflects the time-frequency characteristics of sound and conforms to human auditory perception.

[0034] S15. Perform normalization and channel dimension supplementation. Standardize and normalize the Mel spectrogram, and expand the single-channel Mel spectrogram into a three-channel one. This includes the following steps: Standardization (Z-score transformation): For the decibel-converted Mel spectrogram, first calculate the global mean and standard deviation for each sample in the frequency dimension (128 Mel filters) and the time dimension (all time frames). Then, transform the data using the formula (X - mean) / (standard deviation + minimum value) to adjust the data to a distribution with a mean close to 0 and a standard deviation close to 1. This step eliminates the magnitude differences between different samples, bringing the features to a uniform scale. The addition of the minimum value avoids cases where the denominator is 0.

[0035] Min-Max scaling (mapping to the [0,1] range): After standardization, the global minimum and maximum values ​​of the data are further calculated. For samples with a dynamic range greater than a set threshold, the standardized results are mapped to a fixed interval of [0,1] using the formula (X - minimum) / (maximum - minimum). This step adds explicit numerical boundaries to the data, making the spectral feature distribution of different samples more consistent and more suitable as input for subsequent networks.

[0036] Network adaptation: After normalization to obtain the Mel spectrogram in the range [0,1], its original data format is "number of samples - number of Mel filters - number of time frames". This format only has three dimensions and lacks the "channel dimension" required by the subsequent backbone network. Then, by inserting a single-channel dimension between "number of samples" and "number of Mel filters", the data shape is adjusted to "number of samples - 1 (single channel) - number of Mel filters - number of time frames". This process only expands the dimensions and does not change the numerical range and features of the spectrogram. Finally, the data format conforms to the general input format of "number of samples - number of channels - feature height - feature width" of visual Transformer-type networks, adapting to the backbone network's requirements for input dimensions.

[0037] S2. Construct and train a bird call recognition model. For example... Figure 2 As shown, the bird call recognition model consists of a main layer, a hierarchical structure, and a classification head connected in sequence. Among them, Indicates the height of the feature map, This represents the width of the feature map. The model's input is a preprocessed three-channel time-frequency spectrum, which first passes through the backbone layer to obtain the initial representation. The hierarchical structure includes multiple stages. In this embodiment, the hierarchical structure consists of four stages and patch embeddings stacked alternately. Patch embeddings connect adjacent stages. The backbone layer is connected to the input of stage 1, and the output of stage 4 is connected to the classification head, finally outputting the result. Each stage includes multiple improved Transformer blocks for feature extraction at the current resolution. The patch embedding layer is used for spatial downsampling and channel upsampling to advance to the next stage. The expression for the above process is as follows:

[0038]

[0039] in, Indicates the first The output feature map of the stage Indicates the first The downsampled feature map of the first stage is obtained by performing embedding layer operations on the second stage. The output is obtained by downsampling the feature map of the stage. Indicates the first The feature extraction module for each stage Indicates the first The image segmentation and embedding module for this stage.

[0040] A single improved Transformer block structure, such as Figure 3 As shown in the figure Represents the number of channels. Representing the sequence length, the input features are first transformed linearly to obtain the query vector. (Query), key vector (Key) and value vector (Value).

[0041] For query vector and key vector Spatial attention processing is performed. A 3×3 depthwise convolution and sigmoid activation are applied to capture local spatial dependencies. The lightweight design of the depthwise convolution (only channel dimensions are computed independently) significantly reduces the number of parameters, while the sigmoid function normalizes the output to the [0,1] interval, generating a spatial weight map. The expression is shown below:

[0042] in, , κ For convolution and size (e.g.) ), Spatial weights; Output range compressed to Generate a spatial weighted graph. Representing feature maps, This represents the nth feature map covered by the convolution kernel. A local area.

[0043] Then, channel attention is processed. First, global average pooling is applied to the spatial attention result to compress the spatial dimension to 1×1. Then, two fully connected layers and a sigmoid activation are used to dynamically adjust the importance of the channel dimension. This "squeeze-activation" structure ensures that the model can adaptively allocate channel resources. The expression is shown below:

[0044] in, Output . This represents the row index on the feature map. Represents the column index on the feature map. This represents the channel index on the feature map.

[0045] The initial context feature representation is obtained through an additive similarity function. and The context scores are added together, avoiding matrix operations and preserving feature dimensions. Compared to multiplicative attention, additive operations not only reduce computational complexity but also avoid the numerical instability of Softmax. The expression is as follows:

[0046] in, Represents the query vector and key vector The additive similarity features between them are the initial context feature representations calculated using the additive similarity function; It is a query vector The score after preprocessing and context modeling It is a key vector The score after preprocessing and context modeling.

[0047] The initial contextual feature representation is input into the Multi-scale Edge Enhancement Module (MEEM) to enhance details and edge features in the time-frequency map. Contextual information from different receptive fields is extracted through multi-scale convolution, and the details and edge information of the spectral map are enhanced through the Edge Enhancer (EE) branch. The final output... This serves as input for subsequent processing. The expression is as follows:

[0048] in, This represents a multi-scale edge enhancement module.

[0049] Multi-scale edge enhancement module structure such as Figure 4 As shown, it internally contains an edge enhancer designed to enhance the edges of features at each scale, highlighting the boundaries and details of objects. It differs a feature at a given scale from its average pooling result, then processes it through convolution and activation functions to enhance the edge response. The edge enhancer structure is as follows: Figure 5 As shown, its workflow is as follows: Input features at a certain scale. Then, low-frequency background information is obtained by smoothing features through average pooling, and edges are extracted by element-wise subtraction. Next, edge features application Convolution, batch normalization, and sigmoid activation are used to enhance edge responses, resulting in enhanced features. While preserving the original features, it highlights the edges. The edge enhancer effectively highlights edges and improves the accuracy of the results through differential operations and residual enhancement.

[0050] The multi-scale edge enhancement module aims to provide the model with rich detail information through multi-scale edge feature extraction and fusion. Its structure and workflow are as follows: Original Features Through average pooling and Convolution generates features at different scales Then for the features at each scale Apply an edge enhancer to highlight the edges. Finally, combine the original features... and enhanced features , , Assemble by channel, then through Convolutional integration ultimately outputs enhanced features. .

[0051] The output of the multi-scale edge enhancement module Parallel inputs are fed into the Dynamic Feature Fusion (DFF) module and the Top-k Spare Attention (TKSA) module.

[0052] DFF outputs MEEM Multi-scale downsampling is performed, and features at different scales are weighted and fused using adaptive weights to form dynamically fused features. Final output. This serves as input for subsequent processing. The expression is as follows:

[0053] in, This indicates the dynamic feature fusion module.

[0054] The dynamic feature fusion module aims to solve the static problem in multi-scale feature fusion, and its structure is as follows: Figure 6 As shown, traditional methods cannot dynamically select important features according to task requirements, leading to redundancy or loss of key information. DFF achieves adaptive optimization of feature fusion through a dynamic selection mechanism guided by global information, improving the model's ability to segment complex structures. The DFF module includes two sub-modules: channel dynamic selection and spatial dynamic selection, which enhance the effectiveness of feature fusion from the channel and spatial dimensions, respectively.

[0055] Dynamic channel selection: Input multi-scale feature maps and , Indicates the first The layers are then concatenated along their channel dimensions, and then the spatial dimensions are compressed using global average pooling to generate channel-level statistics, followed by the generation of channel weights. Finally, feature calibration and dimensionality reduction .

[0056] Spatial dynamic selection: First, spatial weights are generated. After channel selection Pixel-wise weighting to generate Enhance the characteristic response of important spatial locations.

[0057] The Top-K sparse attention module performs Top-K filtering on the MEEM output to achieve sparse attention computation, focusing only on the most relevant contextual information. Final output This serves as input for subsequent processing. The expression is as follows:

[0058] in, This represents a Top-k sparse attention module.

[0059] The Top-k sparse attention module aims to reduce redundancy and improve the effectiveness of feature aggregation by dynamically selecting key attention scores. Its structure is as follows: Figure 7 As shown, it includes two sub-modules: dynamic selection of k-value and multi-scale fusion.

[0060] Dynamically select the k value: The TKSA module can dynamically select the k value within a range. Instead of fixing the value of k, we control the value of k. We then determine the specific value of k by weighted averaging different proportions, balancing sparsity and global information preservation. The specific process is as follows: First, the input features pass through a normalization layer, then a 1×1 convolution adjusts the channel dimension, followed by a 3×3 depthwise separable convolution to capture local spatial relationships, ultimately generating a query vector that meets computational requirements. Key vector Sum value vector For the generated and First, a multiplication operation is performed, followed by a mask operation to initially filter or mark some elements, preparing for subsequent Top-k filtering. Then, the similarity matrix is ​​calculated. ,in , yes and The channel dimension is then determined. For each query row, the top k largest scores are selected, and their corresponding position indices are generated. The rest are set to zero. Softmax is applied to the filtered matrix to obtain a sparse attention matrix. The final feature aggregation is as follows: ,in Choose the Top-k function. The normalized matrix and... After performing multiplication, reshaping, and 1×1 convolution, the output of this part is finally obtained.

[0061] Multi-scale fusion: To fully capture broader scale variation information in images, and addressing the limitations of handling objects or features at extreme scales, we add more depthwise convolutional paths at various scales to enhance multi-scale information mining capabilities. The specific process involves dynamically selecting the k-value, adding the output and input feature residuals, then adding more depthwise convolutional paths at 3×3, 5×5, and 7×7 depthwise separable convolutions, followed by ReLU activation and concatenation. Finally, a 1×1 convolution is used to compress the channels of the concatenated features, restoring their dimensionality to an appropriate level.

[0062] Output DFF With TKSA output The channels are concatenated and then merged using a 1×1 convolution. This is then compared with the MEEM output. The residuals are summed to obtain an enhanced contextual feature representation. This preserves edge enhancement information while fusing dynamic features and sparse attention information, and then it is combined with... Combined output The final processed features are output. The expression is as follows:

[0063]

[0064]

[0065]

[0066] in, The concatenated feature map is represented by the feature map... and feature map The new feature map obtained after the splicing operation; The fused feature map represents the stitched feature map. The final feature fusion result obtained after applying 1×1 convolution; The processed feature map is obtained by fusing the feature maps. With another feature map The result obtained after element-wise addition; Representing vectors and The output of the dot product operation between the two.

[0067] Multiple improved Transformer blocks are stacked to progressively extract higher-level features, ultimately yielding the feature output for the current stage. Spatial downsampling and channel upsampling are performed between adjacent stages through embedding layers. Through hierarchical downsampling and feature transformation, a hierarchical structure is ultimately outputting global features.

[0068] The expression is as follows:

[0069] in, This represents the output feature of the b-th block in the s-th stage; Indicates regularization; This indicates a multilayer perceptron; Representation layer normalization.

[0070] The global features are input into the classification head to obtain the predicted score for each category. After GeM pooling and multi-layer Dropout averaging, the global features are fed into a fully connected layer, outputting the probability of bird species. The expression is as follows:

[0071] in, `` is the Sigmoid function, which makes the output fall within (0,1), representing the probability value. `GeM` is the generalized average pooling function, and the result vector... The length of is the number of categories C, and each dimension represents the model's predicted probability for a certain bird species. The weight matrix representing the classification head, The feature map representing the output of the model at a certain stage. This represents the bias vector of the classification head.

[0072] The bird call recognition model in this embodiment is trained through the following steps: Prepare the training dataset: Collect bird song audio with bird species labels and generate corresponding Mel spectrograms. Resample the collected bird song audio data to 32000Hz. Divide the audio into fixed-length segments (5 seconds), padding with zeros for shorter segments and splitting excessively long segments into multiple segments. Perform a Short-Time Fourier Transform (STFT, window size 1024, frame shift 320) on each segment, passing it through 128 Mel filters to obtain the Mel spectrogram, and take the logarithm to form the log-Mel spectrum. Normalize the mean and variance of the spectrogram to eliminate differences in recording conditions. Duplicate the single-channel spectrogram three times to form a three-channel input, used to adapt the model input. Use Mixup data to enhance details and improve the model's robustness in noisy environments.

[0073] Set training parameters: Use the AdamW optimizer, and set the parameters as follows: , , The weights decay by 1e-6. The initial learning rate is... The first 20 training rounds used linear warm-up, followed by cosine annealing to decay to [the desired temperature]. The training run consisted of 300 epochs with a batch size of 2048. Gradient accumulation could be used when hardware was insufficient. For the loss function, weighted binary cross-entropy loss was used for multi-label recognition tasks, while cross-entropy loss combined with label smoothing (smoothing coefficient 0.1) was used for single-label classification tasks. Regularization and stabilization measures included Dropout (scale 0.1), gradient clipping (threshold 1.0), and exponential moving average (momentum 0.9995), and mixed-precision training was used to improve efficiency.

[0074] Training process: Input training data into the bird song recognition model for forward computation, calculate the loss value through the loss function, update the model parameters through backpropagation, and use gradient clipping, Dropout and exponential moving average strategies to stabilize training. Validation and Evaluation: After each training round, model performance is evaluated on an independent validation set, and the weights of the best-performing model are saved. For multi-label tasks, the evaluation metrics are macro-average F1 score, micro-average F1 score, and mean accuracy (mAP). For single-label tasks, the evaluation metrics are Top-1 and Top-5 classification accuracy. For multi-label tasks, an optimal threshold is searched in the range [0.1, 0.9] to improve the F1 score.

[0075] S3. Input the Mel spectrogram into the trained bird call recognition model, extract features and classify them, and output the probability distribution of bird species.

[0076] S4. Based on the probability distribution of the bird species, determine the identified bird species and their confidence levels, and generate visualization output and reports.

[0077] The category probabilities output for each audio window are summarized to obtain the bird identification result for that window. A threshold is used to determine whether a bird call event exists, and adjacent events are merged into a complete segment.

[0078] The software provides a graphical user interface for visualizing results. Users can upload an audio clip, and the software can perform sliding window detection on long recordings, automatically segmenting and recognizing segments containing birdsong. The software interface lists the names of the identified bird species and their confidence probabilities, sorted from highest to lowest confidence. Detected birdong time periods are marked with rectangles or highlights on the time-frequency graph, along with the corresponding species labels and confidence levels. Users can click on event markers to play the corresponding audio segment or zoom in and out of the spectrogram to view details. The software supports exporting recognition results as reports, which include: basic audio information (filename, duration, sampling rate, etc.), model version and inference parameters, a list of detected birds (name, start and end times, confidence levels), and statistical information (number of occurrences for each birdong type, total duration, highest confidence level). Reports can be exported in CSV or JSON format for easy archiving and subsequent analysis.

[0079] In another specific embodiment, a bird call recognition system is disclosed to implement the above-mentioned visual Transformer-based bird call recognition method, comprising: The data preprocessing module is used to convert the input raw bird song audio signal into a standardized Mel spectrogram.

[0080] The model inference module loads the trained bird call recognition model, which is used to extract features and classify the Mel spectrogram, and output the probability distribution of bird species.

[0081] The application output module is used to determine the identified bird species and their confidence levels based on probability distributions, and to generate visualizations and reports.

[0082] The application output module includes: A graphical user interface for uploading audio files and displaying recognition results.

[0083] The sliding window detection unit is used for automatic segmentation and recognition of long audio files.

[0084] Visualization units are used to mark the time periods and species labels of bird call events on a time-frequency graph.

[0085] The report generation unit is used to export the recognition results into a structured report file.

[0086] This application overcomes the inherent shortcomings of existing visual Transformer-based bird call recognition methods when processing time-frequency images. Through innovative improvements at the architectural level, it provides a solution with higher recognition accuracy and better computational efficiency. Specifically, this application enhances detail perception, optimizes the feature fusion mechanism, and improves computational efficiency and focusing ability. While maintaining the advantages of global modeling with the visual Transformer, it constructs a novel bird call recognition model with stronger detail perception, more intelligent feature fusion, and higher computational efficiency, comprehensively improving recognition performance and practicality.

[0087] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for bird call recognition based on visual Transformer, characterized in that, Includes the following steps: S1. Preprocess the original bird song audio signal to generate a Mel spectrogram; S2. Construct and train a bird call recognition model. The bird call recognition model includes a backbone layer, a hierarchical structure and a classification head connected in sequence. The hierarchical structure includes multiple stages. An embedding layer connects two adjacent stages. The input of the first stage is connected to the backbone layer, and the output of the last stage is connected to the classification head. Each stage includes multiple improved Transformer blocks. S3. Input the Mel spectrogram into the trained bird call recognition model, extract features and classify them, and output the probability distribution of bird species; S4. Based on the probability distribution of the bird species, determine the identified bird species and their confidence levels, and generate visualization output and reports.

2. The bird call recognition method based on visual Transformer according to claim 1, characterized in that, The generation of the Mel spectrogram includes the following steps: The original bird song audio signal is subjected to noise reduction and enhancement processing, which includes one or more random combinations of the following operations: gain adjustment, adding Gaussian noise, adding short noise, adding background noise, and low-pass filtering; The audio signal after noise reduction and enhancement is framed and windowed, and then subjected to short-time Fourier transform to obtain the spectrogram. The spectrogram is mapped to a Mel scale using a Mel filter bank to generate a Mel spectrogram. The Mel spectrogram is standardized and normalized, and the single-channel Mel spectrogram is expanded into a three-channel spectrogram.

3. The bird call recognition method based on visual Transformer according to claim 2, characterized in that, Each of the improved Transformer blocks includes the following workflow: The input features are linearly transformed to generate query vectors. Key vector Sum value vector ; For query vector and key vector Spatial attention and channel attention are applied, and the initial context feature representation is obtained through an additive similarity function; The initial contextual feature representation is input into the multi-scale edge enhancement module to enhance details and edge features in the time-frequency map; The output of the multi-scale edge enhancement module is input in parallel to the dynamic feature fusion module and the Top-k sparse attention module; The output of the dynamic feature fusion module is concatenated with the output of the Top-k sparse attention module, and after convolutional fusion, it is residually connected with the output of the multi-scale edge enhancement module to obtain the enhanced contextual feature representation. Enhanced contextual feature representation with value vector Combined, the final processed features are output.

4. The bird call recognition method based on visual Transformer according to claim 3, characterized in that, After stacking the multiple improved Transformer blocks, higher-level features are extracted step by step, and finally the feature output of the current stage is obtained. Spatial downsampling and channel upsampling are performed between adjacent stages through embedding layers. Through hierarchical downsampling and feature transformation, the final hierarchical structure outputs global features.

5. The bird call recognition method based on visual Transformer according to claim 4, characterized in that, The classification head processes global features through GeM pooling and multi-layer Dropout averaging before entering a fully connected layer, outputting the probability of bird species.

6. The bird call recognition method based on visual Transformer according to claim 5, characterized in that, The workflow of the multi-scale edge enhancement module includes: Local features of the input features are extracted through convolutional layers; Features of different scales are generated through average pooling and convolution operations; An edge enhancer is applied to the features at each scale. The edge enhancer enhances the edge response by differencing the features at a certain scale with the average pooling result and then processing them through convolution and activation functions. The local features are concatenated with all features processed by the edge enhancer, and then fused through convolution to finally output the enhanced features.

7. The bird call recognition method based on visual Transformer according to claim 6, characterized in that, The dynamic feature fusion module includes a channel dynamic selection submodule and a spatial dynamic selection submodule. The workflow of the dynamic feature fusion module includes: Multi-scale downsampling is performed on the enhanced features output by the multi-scale edge enhancement module to obtain feature maps at different scales; The feature maps of different scales are input into the channel dynamic selection submodule, the channel dimensions are spliced, the spatial dimension is compressed by global average pooling, channel-level statistical information is generated, channel weights are generated, and finally the channel weights are calibrated and dimensionality reduced to obtain the dimensionality-reduced channel weights. Input feature maps of different scales into the spatial dynamic selection submodule to generate spatial weights; The spatial weights output by the spatial dynamic selection submodule and the dimensionality-reduced channel weights output by the channel dynamic selection submodule are weighted element-wise to generate the final fused features.

8. The bird call recognition method based on visual Transformer according to claim 7, characterized in that, The Top-k sparse attention module includes a dynamic k-value selection submodule and a multi-scale fusion submodule. The workflow of the Top-k sparse attention module includes: Calculate query vector With key vector The similarity matrix; For each query row in the similarity matrix, only the top k similarity values ​​in that row are retained, and the rest are set to zero to generate a sparse attention matrix; sparse attention matrix and value vector Weighted aggregation is performed to obtain sparse attention output; By fusing the sparse attention output with the input through residual connections, an enhanced representation is obtained.

9. The bird call recognition method based on visual Transformer according to claim 8, characterized in that, The training steps for the bird call recognition model include: Prepare the training dataset: Collect bird call audio with bird species labels and generate the corresponding Mel spectrograms; Set training parameters: Use the AdamW optimizer, adjust the learning rate using linear warm-up and cosine annealing strategies, and use the cross-entropy loss function or the labeled smooth cross-entropy loss function. Training process: Input training data into the bird song recognition model for forward computation, calculate the loss value through the loss function, update the model parameters through backpropagation, and use gradient clipping, Dropout and exponential moving average strategies to stabilize training. Validation and evaluation: Evaluate the model performance on an independent validation set and save the weights of the best-performing model.

10. The bird call recognition method based on visual Transformer according to claim 9, characterized in that, This is based on a bird call recognition system, which includes: The data preprocessing module is used to convert the input raw bird song audio signal into a standardized Mel spectrogram; The model inference module is loaded with a bird call recognition model obtained by the training steps as described in claim 9, which is used to extract features and classify the Mel spectrogram and output the probability distribution of bird species. The application output module is used to determine the identified bird species and their confidence levels based on the probability distribution, and to generate visualizations and reports. The application output module includes: A graphical user interface for uploading audio files and displaying recognition results; A sliding window detection unit is used for automatic segmentation and recognition of long audio files; Visualization units are used to mark the time periods and species labels of bird song events on the time-frequency graph; The report generation unit is used to export the recognition results into a structured report file.

Images