Bird voiceprint target detection method and system

Abstract

The invention relates to the technical field of bird voiceprint processing, in particular to a bird voiceprint target detection method and system. The method comprises the following steps: converting an audio signal into time-frequency domain representation by using short-time Fourier transform to obtain a time-frequency graph, and carrying out scale augmentation and spectrum enhancement on the time-frequency graph to enhance the diversity of voiceprint features. According to the method, technologies such as multi-scale spectrum feature augmentation, voiceprint time-frequency positioning, semi-supervised pre-training and weak supervised pre-training are introduced, so that the precision of bird voiceprint target detection is improved. The bird voiceprint detection method can effectively cope with a complex and changeable natural environment, can also remarkably improve the generalization ability of the model while keeping high precision, and provides a reliable bird voiceprint detection solution for applications such as biodiversity protection and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of bird voiceprint processing, and in particular to a bird voiceprint target detection method and system. Background Art

[0002] Bird voiceprint recognition technology is often used in the fields of biodiversity conservation, species monitoring, and biological sample surveys. Traditional bird voiceprint recognition methods mainly rely on manually annotated data sets and single-scale spectral feature extraction. This method has obvious limitations when dealing with bird voiceprints in complex and changeable natural environments. Although existing deep learning methods have improved the accuracy of bird voiceprint recognition to a certain extent, most of these methods are based on supervised learning frameworks and rely heavily on a large amount of high-quality manually annotated data, which not only consumes a lot of time and manpower costs, but also has difficulty in coping with the challenges of large-scale data sets. Especially when faced with complex voiceprint patterns and diverse bird voiceprint features, the adaptability and generalization capabilities of existing methods still need to be improved.

[0003] In addition, when processing bird soundprints, traditional methods often ignore the diversity of features at different frequencies and time scales, resulting in reduced recognition accuracy of the model in complex environments. Therefore, the technical problem to be solved by the present invention is: how to improve the generalization ability of the bird soundprint target detection model by utilizing the potential information in unlabeled data under limited artificially labeled data, so that it has higher recognition accuracy in complex and changeable natural environments. Summary of the invention

[0004] The present invention provides a bird voiceprint target detection method and system to improve the recognition ability and generalization performance of the model under complex voiceprint patterns by integrating multi-scale feature learning, semi-supervised learning and large-scale pre-training technology.

[0005] The technical solution of the present invention to solve the above technical problems is as follows: On the one hand, a method and system for detecting bird soundprint targets are provided, the method comprising the steps of: The audio signal is converted into a time-frequency domain representation using a short-time Fourier transform to obtain a time-frequency graph, and the time-frequency graph is scaled and spectrum enhanced to enhance the diversity of voiceprint features; Construct a bird soundprint target detection model based on soundprint time-frequency positioning technology, and extract global time-frequency features through a self-attention mechanism neural network; Conduct semi-supervised pre-training on large-scale datasets and use unlabeled data to improve model performance; Weakly supervised pre-training is performed based on the generated semi-supervised dataset to improve the pre-trained model’s understanding of complex voiceprint patterns; The pre-trained model is fine-tuned downstream to optimize the model's task performance.

[0006] Furthermore, the short-time Fourier transform divides the audio signal into frames, calculates the Fourier transform of each frame, and obtains a frequency-time relationship diagram, which is expressed as: , where x[n] is the original audio signal, w(t−n) is the window function, and X(t,f) is each frame of the time-frequency graph.

[0007] Furthermore, discrete wavelet transform is used to scale the time-frequency diagram, and spectrum features at different levels are extracted by adjusting the scales of the frequency and time windows. The expression is: , where a controls the scale of the wavelet, b controls the translation, and ψ(t) is the mother wavelet function.

[0008] Furthermore, the spectrum enhancement simulates different propagation characteristics in a real environment by adjusting the amplitude of the spectrum at each scale, and the expression is: , where S(f,t) is the original spectrum and α(f,t) is the enhancement coefficient, which is dynamically adjusted based on the local changes of frequency f and time t.

[0009] Furthermore, the global time-frequency features are extracted by the self-attention mechanism neural network, and the detection accuracy is improved by combining the time-frequency positioning technology; in the process of self-attention mechanism processing, the time-frequency map X∈R is input N×d ; Transformed by embedding layer, the input features are mapped to a higher dimensional space; Position encoding PE∈R N×d is a matrix that matches the dimension of the input time-frequency graph and represents the position information of each position. For each position i and each dimension k, the position encoding is defined as: , where N is the number of positions in the time-frequency graph and d is the feature dimension of each position.

[0010] Furthermore, the input representation of the self-attention mechanism includes query, key and value; the input matrix X is mapped into three different vectors: query Q, key K and value V, expressed as: Q=XW Q , K = XW K , V = XW V , where W Q ∈R d×dk , W K ∈R d×dk , W V ∈R d×dv , where dk and dv are the dimensions of the key and value respectively, X∈R N×d Represents the input time-frequency graph, Q, K, V∈R N×dkRepresent the query, key, and value obtained through linear transformation respectively.

[0011] Furthermore, the expression of the self-attention mechanism is: , where QK T ∈R N×N It is the dot product of the query and the key, which represents the similarity measure between each position in the time-frequency graph. The dot product result is scaled and activated using the softmax function to obtain the attention weight of each position; the attention weight is multiplied by the value vector V to obtain the weighted output.

[0012] Furthermore, the step of performing semi-supervised pre-training on a large-scale dataset includes: Use manually annotated data for basic supervised learning to learn the basic characteristics and rules of bird voiceprints. Annotate the data set, which has N annotated samples, each sample x i The label is y i , where x i ∈R d represents the time-frequency graph feature vector, y i ∈R c is the bird category label corresponding to the sample, C is the number of bird categories, and the loss function of supervised learning uses cross entropy loss, which is expressed as: ,in, is the model for sample x i The predicted category, the probability distribution obtained through the softmax layer; Based on the trained model, unlabeled samples Make predictions and obtain the category prediction probability distribution of each sample , according to the threshold τ, the prediction with high confidence is selected as the pseudo label, that is: , where τ represents a hyperparameter that controls the confidence threshold of the pseudo-label; After each batch of training, gradually expand the scope of inference, select at least twice as many unlabeled samples for inference, gradually add pseudo-label data, adjust hyperparameters based on new pseudo-label samples, and update the confidence of each new pseudo-label sample. In the semi-supervised learning process, a weighted loss function is used to suppress the influence of low-quality pseudo labels. The weighted loss function expression is: ,in, Represents the confidence of the pseudo-label. For pseudo-labels with lower confidence, a smaller weight is given to reduce their negative impact on the training process.

[0013] Furthermore, the step of performing weakly supervised pre-training based on the generated semi-supervised dataset includes: Using the InfoNCE (Information Contrastive Loss) method for contrastive learning, each pair of audio samples (x i , x j ) is mapped to an embedding space through the neural network model f θ Get their feature representation: z i = f θ (x i ) and z j = f θ (x j ), calculate the similarity between each pair of positive and negative samples. The goal is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The cosine similarity is used to calculate the similarity: ; Construct a contrast loss function and use InfoNCE loss to maximize the similarity of positive samples and minimize the similarity of negative samples. The expression is: , where τ is the temperature parameter that controls the smoothness of the distribution, N is the number of negative samples, and z k is the feature representation of negative samples; By defining pre-tasks to automatically generate supervision signals, we design self-supervised learning tasks based on time-frequency graphs. Given a time-frequency graph X∈R N×D , where N is the number of frames of the time-frequency graph, D is the feature dimension of each frame, and the time-frequency graph is cut into K blocks, each block is X k ∈R n×d , where n is the number of frames in each small block, d is the feature dimension of each small block, and a prediction task is designed to learn the global structure of the time-frequency graph by maximizing the similarity between blocks. The self-supervised loss function is defined as: , where f(X k ) represents the block f(X k ), is the cosine similarity between blocks; Combining contrastive learning and self-supervised learning, the optimization model captures the diversity of bird voiceprints in a multi-scale and multi-level feature space to obtain the final loss function: , where λ is a hyperparameter that balances the two losses.

[0014] On the other hand, a bird voiceprint target detection system is provided, which is used to perform the bird voiceprint target detection method as described above, and the system comprises: A data preprocessing module for receiving an audio signal and converting it into a time-frequency domain representation using a short-time Fourier transform; Multi-scale feature enhancement module, used to perform multi-scale augmentation on the time-frequency graph, including scale augmentation and spectrum enhancement; The target detection module is based on voiceprint time-frequency positioning technology and uses a self-attention mechanism neural network to extract global time-frequency features; Semi-supervised learning module, used for semi-supervised pre-training on large-scale datasets; Weakly supervised pre-training module, used for weakly supervised pre-training based on the generated semi-supervised dataset; The model fine-tuning module is used to perform downstream fine-tuning on the pre-trained model to optimize the performance of the model on specific tasks.

[0015] The beneficial effects of the present invention are: The present invention improves the accuracy of bird voiceprint target detection by introducing technologies such as multi-scale spectral feature augmentation, voiceprint time-frequency localization, semi-supervised pre-training and weakly supervised pre-training. Specifically, multi-scale spectral feature augmentation can extract rich bird voiceprint features at different frequencies and time scales, enhancing the model's adaptability to diverse voiceprint patterns. Through semi-supervised learning methods, the present invention can effectively pre-train on large-scale unlabeled data sets and automatically generate high-quality pseudo-labels, thereby greatly reducing reliance on manually labeled data and reducing time and labor costs. In addition, large-scale pre-training makes full use of the Internet and monitoring data to explore the patterns and characteristics of bird voiceprints in real-world scenarios, further improving the generalization ability of the model.

[0016] Based on the above technical means, the present invention can not only effectively cope with complex and changeable natural environments, but also significantly improve the generalization ability of the model while maintaining high accuracy, providing a reliable bird voiceprint detection solution for applications such as biodiversity protection. By combining multi-scale spectral feature augmentation and semi-supervised pre-training, the present invention achieves efficient and accurate bird voiceprint target detection under limited labeled data conditions, solving the problem of insufficient recognition accuracy of existing technologies in complex voiceprint modes.

[0017] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flow chart of a bird soundprint target detection method according to an embodiment of the present invention; Figure 2 A schematic diagram of short-time Fourier transform in one embodiment of the present invention; Figure 3This is a schematic diagram of scale augmentation in one embodiment of the present invention; Figure 4 Schematic diagram of embedding a time-frequency graph based on Transformer in one embodiment of the present invention; Figure 5 This is a flow chart of multi-scale augmentation of voiceprint spectrum features in one embodiment of the present invention; Figure 6 A flowchart of constructing a bird soundprint target detection model in one embodiment of the present invention; Figure 7 A flowchart of semi-supervised pre-training on a large-scale dataset in one embodiment of the present invention; Figure 8 This is a flowchart of weakly supervised pre-training in one embodiment of the present invention; Fig. 9 Flow chart of downstream fine-tuning of a pre-trained model in one embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The term "comprise" and any variation thereof in the specification of the present application are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In addition, the use of "and / or" in the specification represents at least one of the connected objects, such as A and / or B, which means that A alone, B alone, and A and B exist in three cases.

[0021] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0022] The present invention provides the following preferred embodiments.

[0023] Embodiment 1: In order to improve the generalization ability of the bird soundprint target detection model by utilizing the potential information in the unlabeled data under the condition of limited manually labeled data, so that it has higher recognition accuracy in complex and changeable natural environments. This embodiment proposes a bird soundprint target detection method based on semi-supervised pre-training of multi-scale spectral features, such as Figure 1 As shown, the specific steps of the method include:

[0024] S100, using short-time Fourier transform to convert the audio signal into a time-frequency domain representation to obtain a time-frequency graph, and then performing scale augmentation and spectrum enhancement on the time-frequency graph to enhance the diversity of voiceprint features.

[0025] S200. Build a bird soundprint target detection model based on soundprint time-frequency positioning technology, and extract global time-frequency features through a self-attention mechanism neural network.

[0026] S300, conduct semi-supervised pre-training on large-scale datasets and use unlabeled data to improve model performance.

[0027] S400, performing weakly supervised pre-training based on the generated semi-supervised dataset to improve the pre-trained model's understanding of complex voiceprint patterns.

[0028] S500, perform downstream fine-tuning on the pre-trained model to optimize the model's task performance.

[0029] Step S100: multi-scale augmentation of voiceprint spectrum features: like Figure 2 As shown, the audio signal is converted into a time-frequency domain representation using short-time Fourier transform (STFT), and the relationship between the audio signal and time of each frame is analyzed using the time-frequency diagram. In order to enhance the diversity of voiceprint features and improve the adaptability of the model, this step uses multi-scale augmentation technology. Specifically, the multi-scale spectral feature enhancement of the input bird sound is performed by the following methods: scale augmentation and spectrum enhancement. Specifically, scale augmentation performs discrete wavelet transforms (DWT) of different scales on the original spectrum map, and obtains multi-level spectral features from low frequency to high frequency, such as Figure 3 As shown in the figure. The spectral features corresponding to each level contain different time window sizes, ensuring that features of different frequency and time scales can be effectively extracted. In the spectrum diagram, spectrum enhancement simulates the different propagation characteristics of bird sounds in real environments by enhancing and suppressing the amplitude of the spectral features of each scale. The enhanced amplitude control parameters can be automatically adjusted by maximizing information entropy.

[0030] Step S200: construct a bird voiceprint target detection model based on voiceprint time-frequency positioning technology: In this step, a method based on voiceprint time-frequency localization technology can be used to improve the accuracy of the target detection model. This technology relies on the combination of time localization and frequency localization to accurately locate the time-frequency position of bird voiceprints. In order to locate the bird voiceprint target in the audio, the time-frequency graph is first annotated for target detection, and the global time-frequency features are extracted through the Transformer neural network. By introducing the end-to-end target detection algorithm of Transformer (such as SwinTransformer, DETR, ViT, etc.), bird voiceprints can be automatically located and identified in the time-frequency graph. Specifically, the target detection model inputs the time-frequency graph into the network, and after feature extraction, target localization and classification, it can finally automatically locate and identify bird voiceprints in the time-frequency graph. The long-distance dependencies in the time-frequency graph are captured through the attention mechanism (such as the self-attention mechanism in the Transformer), thereby improving the recognition ability of complex voiceprint patterns. After training, the network can accurately identify the characteristics of bird voiceprints in the time-frequency graph, such as Figure 4 As shown, the bird species category and time-frequency position of each bird soundprint are output, thereby realizing the most basic automated bird soundprint target detection method.

[0031] Step S300, performing semi-supervised pre-training on a large-scale dataset; The goal of this step is to use a large amount of unlabeled data through semi-supervised learning methods to improve the performance and generalization ability of the model. First, use manually labeled data for basic supervised training to ensure that the model can learn the basic characteristics and rules of bird voiceprints. Then, semi-supervised learning is gradually carried out through the trained basic model. Specifically, a large amount of unlabeled data is automatically inferred using a trained model, and the inference results with higher confidence are selected and added to the training set as pseudo-labels. In order to improve the quality of labeling, a batch-by-batch inference method is adopted, and the inference range of each batch is twice as large as the previous batch. In this way, in the process of gradually expanding the inference range, the quality of the pseudo-labels can be ensured to be gradually improved, and the premature use of low-quality pseudo-labels can be avoided to affect the model performance.

[0032] Step S400, based on the semi-supervised data set finally generated, weakly supervised pre-training is performed: In this step, weakly supervised pre-training is performed using the generated semi-supervised training dataset. The purpose of this pre-training phase is to further improve the model's ability to understand and recognize complex voiceprint patterns by learning effective patterns and potential features in the data. The specific operation is to use weakly supervised learning methods such as contrastive learning or self-supervised learning to extract potential patterns and features by minimizing the contrast loss of the data or learning some weak supervision signals.

[0033] Step S500, downstream fine-tuning of the pre-trained model: In this step, the semi-supervised training data set generated in step S400 and the pre-trained model obtained by pre-training in step S500 are applied to the bird voiceprint target detection model based on voiceprint time-frequency positioning technology constructed in step S200, and the performance of the model on specific tasks is optimized by fine-tuning. First, the target detection task is fine-tuned using the parameters of the pre-trained model. During fine-tuning, a layer-by-layer freeze and thaw strategy is adopted to ensure that the model can quickly adapt to the specific needs of the task and avoid overfitting. Finally, the fine-tuned model is evaluated through the validation set to ensure that it has high adaptability and high precision in practical applications, providing a reliable bird voiceprint detection solution for applications such as biodiversity conservation.

[0034] This embodiment proposes a new bird voiceprint target detection method based on multi-scale spectral feature semi-supervised pre-training. This method fully exploits the multi-scale features of bird voiceprints and performs semi-supervised and self-supervised pre-training on a large-scale dataset, which has the following beneficial effects: 1. Mining the multi-scale features of voiceprint spectrum characteristics to make them close to the complex and changeable application scenarios in the real world, and enhancing the generalization ability and recognition accuracy of neural networks on the basis of fully optimizing neural networks; 2. Semi-supervised learning can obtain massive amounts of labeled data at a lower time and labor cost, creating conditions for large-scale pre-training; 3. Large-scale pre-training, making full use of Internet data and monitoring data to explore the patterns and characteristics of bird voiceprints in real-world scenarios and reduce the difficulty of downstream task training.

[0035] Through the technical means of multi-scale spectral feature augmentation, voiceprint time-frequency positioning, semi-supervised pre-training and unsupervised pre-training, the present invention realizes an efficient and accurate bird voiceprint target detection method. Multi-scale spectral feature augmentation can extract rich bird voiceprint features at different frequencies and time scales, enhancing the adaptability and generalization ability of the model. At the same time, semi-supervised learning and large-scale pre-training methods can enable the model to self-optimize on large data sets by automatically generating pseudo-labels, which not only improves the performance of the model in complex environments, but also significantly reduces the dependence on manual labeling. Combining the use of voiceprint time-frequency positioning technology and the Transformer model, not only the target positioning accuracy is improved, but also the complex bird voiceprint patterns can be better captured.

[0036] Embodiment 2: Figure 5 As shown, the specific implementation of step S100 is as follows:

[0037] 1.1. The core goal of this step is to improve the ability of bird voiceprint recognition through multi-scale augmentation. The specific implementation includes: short-time Fourier transform (STFT), scale augmentation and spectrum enhancement.

[0038] Short-time Fourier transform (STFT): This embodiment first converts the audio signals in all data sets into time-frequency domain representation to obtain a two-dimensional time-frequency diagram. In order to reasonably verify the optimization and generalization capabilities of the model, we divide the training set and the test set in a ratio of 8:2. During the training process, the test set also serves as a validation set. In the process of generating a two-dimensional time-frequency diagram, STFT divides the audio signal into frames, calculates the Fourier transform of each frame, and obtains a frequency-time relationship diagram.

[0039] ; Where x[n] is the original audio signal, w(t−n) is the window function, and X(t, f) is each frame of the time-frequency graph.

[0040] 1.2. Scale augmentation: This embodiment uses discrete wavelet transform (DWT) to scale the time-frequency graph, and extracts spectrum features at different levels by adjusting the scale of the frequency and time windows. This can effectively increase the diversity of features and improve the adaptability to a variety of bird soundprints.

[0041] The formula for wavelet transform is: ; Among them, a controls the scale of the wavelet, b controls the translation, and ψ(t) is the mother wavelet function.

[0042] 1.3. Spectrum enhancement: Spectrum enhancement simulates different propagation characteristics in real environments by adjusting the amplitude of the spectrum at each scale. The amplitude control parameters can be automatically adjusted by maximizing information entropy.

[0043] ; Among them, S(f, t) is the original spectrum graph, α(f, t) is the enhancement coefficient, which is dynamically adjusted based on the local changes of frequency f and time t.

[0044] Embodiment 3: Further, as Figure 6 As shown, the specific implementation of step S200 is as follows:

[0045] 2.1. In step S200, the soundprint time-frequency positioning technology is used to accurately locate and detect the bird soundprint target. To achieve this goal, it is first necessary to mark the time-frequency graph for target detection, and then use the Transformer neural network to extract global time-frequency features, and combine the time-frequency positioning technology to further improve the detection accuracy. In this embodiment, the VisionTransformer (ViT) architecture is used.

[0046] Furthermore, after the audio signal is transformed into a time-frequency graph through a short-time Fourier transform (STFT), the target detection and annotation is mainly to determine the location of the bird soundprint in the time-frequency graph. To achieve this goal, the time-frequency graph needs to be manually annotated for target detection. The annotator combines the time-frequency graph with the original audio to identify the potential time-frequency location of the bird soundprint and the bird species to which it belongs.

[0047] 2.2. In practical applications, the input frequency spectrum X is a two-dimensional frequency matrix. In the processing of Transformer, the input matrix X is first transformed through a series of embedding operations through the embedding layer. The role of the embedding layer is to map the input features to a higher dimensional space. Assume that the frequency spectrum in this embodiment is represented as a matrix X∈R N×d , where N is the number of positions in the time-frequency graph (i.e., the number of frames in the time-frequency graph), and d is the feature dimension of each position. For example, the time-frequency graph can provide position information for each position by adding a position encoding. Position encoding PE∈R N×d is a matrix that matches the dimension of the input time-frequency graph and represents the position information of each position. Common position encoding methods are sine and cosine position encoding. For each position i and each dimension k, the position encoding can be defined as: ; In this way, position encoding can provide a unique code for each position, which can be added to the input matrix X through PE to capture the position information of timing and spectral information.

[0048] 2.3. The input representation in the self-attention mechanism consists of a query, a key, and a value. In order to calculate the self-attention of the time-frequency graph in the Transformer, this embodiment maps the input representation X into three different vectors: query Q, key K, and value V. These mappings are obtained by multiplying a trainable weight matrix.

[0049] Assume that the weight matrices are W Q ∈R d×dk , W K ∈R d×dk , W V ∈R d×dv , where dk and dv are the dimensions of the key and value respectively The query, key, and value are: Q=XW Q , K = XW K , V = XW V Where X∈R N×d is the input time-frequency graph, Q, K, V∈R d ×dk are the queries, keys, and values ​​obtained through linear transformation.

[0050] 2.4. The calculation formula of the self-attention mechanism is: ; Among them, QK T ∈R N×N is the dot product of the query and the key, representing the similarity measure between each position in the time-frequency graph. By scaling the dot product result and using the softmax activation function, the attention weights for each position are obtained. These attention weights are then multiplied with the value vector V to obtain the weighted output.

[0051] The output of the self-attention mechanism is: ; The output is a weighted representation of each position in the time-frequency graph, which can capture long-range dependencies and contextual information.

[0052] 2.5. In order to enhance the learning ability of the model, Transformer usually uses a multi-head self-attention mechanism. The query, key, and value are divided into multiple heads, and multiple self-attentions are calculated independently. Then the outputs of these heads are spliced ​​together, and finally the final result is obtained through linear transformation.

[0053] If there are h attention heads, the calculation formula for each head is: ; The final output is: ; Where W O ∈R h·d×d It is a linear transformation matrix, and Concat means concatenating the outputs of all heads together.

[0054] 2.6. To alleviate the gradient vanishing problem of deep networks, Transformer adds residual connections between the input and output of each layer, and applies layer normalization to the output of each layer. Finally, after passing through multiple layers of Transformer encoders, this embodiment obtains the feature embedding of the final time-frequency graph, which will be used for target detection and classification tasks of bird voiceprints.

[0055] 2.7. After obtaining the global features of the time-frequency graph, this embodiment inputs it into the target detection model to perform target positioning and classification. At this time, the network will automatically locate the time-frequency position of the bird soundprint based on the time-frequency features extracted by the attention mechanism and perform classification.

[0056] For target positioning, this embodiment adopts a bounding box regression-based method. Assuming that the detected target is a rectangular area, the model will output the bounding box position of the target [xmin ,y min , x max ,y max ], indicating the position of the bird soundprint in the time-frequency graph. To achieve this goal, this embodiment defines a loss function to optimize the prediction of the bounding box: ; Among them, x i ,y i , w i ,h i Respectively represent the real position and size of the target, , , , is the value predicted by the model.

[0057] For the classification task, the goal is to identify which bird voiceprint category each located target belongs to. In this embodiment, the feature vector of the time-frequency graph can be input into a fully connected layer, and the bird category can be predicted through the softmax layer: ; Among them, W c and b c is the weight and bias corresponding to category c, and X is the feature vector corresponding to the time-frequency graph.

[0058] Embodiment 4: Further, as Figure 7 As shown, the specific implementation of step S300 is as follows:

[0059] 3.1. In step S300, the core goal of semi-supervised pre-training is to further improve the performance and generalization ability of the model by utilizing the potential information in the unlabeled data. Specifically, this embodiment will perform semi-supervised learning in stages and use automatic reasoning to generate pseudo labels to gradually improve the model. The following is the detailed implementation process: 3.2. First, this embodiment uses manually annotated data for basic supervised learning to ensure that the model can learn the basic characteristics and rules of bird voiceprints. For the annotated data set, assume that there are N annotated samples, each sample x i The label is y i , where x i ∈R d represents the feature vector of the time-frequency graph, y i ∈R c is the bird category label corresponding to the sample, and C is the number of bird categories. The loss function of supervised learning usually uses cross entropy loss to train the model: ; in is the model for sample x iThe predicted category is the probability distribution obtained through the softmax layer.

[0060] 3.3. After the initial supervised training is completed, the trained basic model is used to infer a large amount of unlabeled data to generate pseudo labels.

[0061] First, based on the trained model f θ (x), for unlabeled samples Make predictions and obtain the category prediction probability distribution of each sample Then, according to the threshold τ, the prediction with high confidence is selected as the pseudo label, that is: ; Among them, τ is a hyperparameter that controls the confidence threshold of the pseudo label. Only when the confidence of the model prediction is higher than this threshold, its prediction is considered more reliable.

[0062] 3.4. Assume that this embodiment selects M samples from the unlabeled dataset, and each sample has a corresponding pseudo label In this embodiment, these pseudo-label samples are added to the training set to form an expanded data set: ; For the expanded training set D expanded , trained using a loss function similar to basic supervised training: ; in, For the model to pseudo-label samples Through such training, the model will be optimized using both labeled data and pseudo-labeled data.

[0063] 3.5. In order to ensure that the quality of pseudo labels is gradually improved and to prevent low-quality pseudo labels from affecting the performance of the model, this embodiment adopts a batch-by-batch reasoning approach. That is, after each batch of training, the model will gradually expand the reasoning range, select more unlabeled samples for reasoning, and gradually add pseudo-label data. The specific implementation method is as follows: The inference range of each batch will be at least twice that of the previous batch. Assume that the pseudo-label samples currently generated are D pseudo In this embodiment, a part of new samples D is selected from the unlabeled data set. new Perform inference. New pseudo-label dataset D expanded It will contain the pseudo-labels of the previous batch and the new pseudo-label data.

[0064] After each batch of training, the model further adjusts parameters based on new pseudo-label samples and selects higher quality pseudo-labels based on the model update. For each new pseudo-label sample, the model updates the confidence of its label, thereby gradually improving the quality of the pseudo-label.

[0065] 3.6. In the process of semi-supervised learning, using pseudo-labels to expand the data set will cause noise in the model during the learning process, so the quality of pseudo-labels needs to be dynamically adjusted. This embodiment can use a weighted loss function to suppress the influence of low-quality pseudo-labels. Suppose the confidence of the pseudo-label is , then the weighted loss function is: ; in, The confidence of a pseudo label indicates the reliability of the model to the pseudo label. For pseudo labels with lower confidence, this embodiment will assign a smaller weight, thereby reducing its negative impact on the training process.

[0066] Embodiment 5: Further, as Figure 8 As shown, the specific implementation of step S400 is as follows:

[0067] 4.1. The core purpose of step S400 is to use the potential features in the generated semi-supervised dataset to further improve the model's ability to understand and recognize complex voiceprint patterns. Through the weakly supervised learning method, the model can not only better extract the potential features of the voiceprint, but also improve the generalization ability of the model. In this embodiment, a combination of contrastive learning and self-supervised learning is adopted, and the specific steps are as follows: 4.2. This embodiment uses a Laplace embedding or InfoNCE (Information Contrastive Loss) method for contrastive learning. This method first transforms each pair of audio samples (x i , x j ) is mapped to an embedding space through a neural network model f θ , and obtain their feature representation f θ (x i ) and f θ (x j ). For each pair of positive samples and negative samples, this embodiment calculates the similarity between them, with the goal of maximizing the similarity of the positive sample pairs and minimizing the similarity of the negative sample pairs.

[0068] First, define the feature representation: i = f θ (x i ) and z j = f θ (x j ), and then calculate the similarity using cosine similarity: ; Furthermore, a contrast loss function is constructed, and InfoNCE loss is used to maximize the similarity of positive samples and minimize the similarity of negative samples. The expression is as follows: ; Where τ is the temperature parameter that controls the smoothness of the distribution, N is the number of negative samples, and z k is the feature representation of negative samples. The goal of this loss function is to increase the similarity of positive sample pairs and reduce the similarity of negative sample pairs.

[0069] In this way, the model is able to learn a feature space in which the sound characteristics of different birds are effectively distinguished, further improving the model's adaptability and generalization ability.

[0070] 4.3. Self-supervised learning methods do not rely on manual labels, but automatically generate supervision signals by defining certain pre-tasks. Common self-supervised learning tasks include predicting missing parts, generating comparison samples, etc.

[0071] In order to enable the model to learn more abstract feature expressions, this embodiment designs a self-supervised learning task based on time-frequency graphs. In this task, given a time-frequency graph X∈R N×D , where N is the number of frames of the time-frequency graph, and D is the feature dimension of each frame. In this embodiment, the time-frequency graph can be cut into multiple local areas, and the relationship between these areas can be learned through a self-supervised task.

[0072] Specifically, suppose the time-frequency graph is cut into K blocks, each block is X k ∈R n×d , where n is the number of frames of each small block, and d is the feature dimension of each small block. The self-supervised learning goal of this embodiment is to train a network to predict the relationship between the current block and other blocks. To this end, this embodiment designs a prediction task to learn the global structure of the time-frequency graph by maximizing the similarity between blocks.

[0073] The self-supervised loss function is defined as follows: ; Among them, f(X k ) represents the block f(X k ), is the cosine similarity between blocks. This embodiment maximizes the similarity to force the model to learn how to infer the global structure based on the relationship between local blocks, thereby improving the accuracy of the model in capturing complex voiceprint patterns.

[0074] In practical applications, contrastive learning and self-supervised learning can be used in combination to more effectively mine the potential features of voiceprints. Specifically, this embodiment first optimizes the basic feature representation of the model through contrastive learning, and then further captures the global features of the time-frequency graph through self-supervised learning. Through these two learning methods, the model can effectively capture the diversity of bird voiceprints in a multi-scale, multi-level feature space.

[0075] Combining these two methods, the loss function finally obtained in this embodiment is as follows: ; Among them, λ is a hyperparameter that balances the two losses.

[0076] Embodiment 6: Further, as Fig. 9 As shown, the specific implementation of step S500 is as follows:

[0077] 5.1. After completing the weakly supervised pre-training based on contrastive learning and self-supervised learning in step S400, the model has acquired a relatively comprehensive feature representation capability, but in order to better meet the needs of specific tasks, such as bird soundprint target detection tasks, the pre-trained model needs to be fine-tuned. The core purpose of fine-tuning is to further optimize the model in downstream tasks, and to ensure the accuracy and adaptability of specific tasks through the strategy of freezing and unfreezing layer by layer.

[0078] 5.2. At the beginning of fine-tuning, the pre-trained model that has completed weakly supervised learning in step S400 is used as the initial model, denoted as M pre , whose parameter is θ pre At this point, the model has learned the multi-scale features and global relationships of bird voiceprints, but it is not optimized for a specific task.

[0079] Given a downstream dataset , where x i Represents the time-frequency diagram of the input audio, y i is the corresponding true label, and the task goal is to obtain the optimized model parameter θ through training task , so that the model achieves optimal performance on this task.

[0080] 5.3. During the fine-tuning process, in order to prevent the model from overfitting or damaging the pre-trained feature representation, a layer-by-layer freezing and unfreezing strategy is used to adjust the model parameters. Specifically: Assume that the model consists of L layers, and the parameters of each layer are ,in ∈{1, 2, ..., L}. Initially, the first k layers of parameters are frozen , only update the parameters of the last L−k layers After several rounds of training, the previously frozen layers are gradually unfrozen until all layers participate in the training. The freezing and unfreezing process can be expressed by the following formula: ; With this strategy, the model is able to gradually adapt to downstream tasks while retaining pre-trained features.

[0081] The AdamW optimizer is used to optimize the model parameters, and its update formula is: ; Where η is the learning rate, and are the bias-corrected first-order and second-order momentum estimates, respectively, and ϵ is a smoothing term to prevent division by zero.

[0082] The learning rate uses the cosine annealing schedule, and the adjustment formula is: ; Among them, η min and η max are the minimum and maximum values ​​of the learning rate, t is the current training step number, and T is the total training step number.

[0083] During fine-tuning, the model performance is dynamically evaluated through the validation set. The validation indicators include: Classification accuracy: evaluates the proportion of samples that are correctly classified: ; Among them, I(⋅) is the indicator function, is the class predicted by the model.

[0084] Average positioning precision (mAP): measures the average accuracy of bounding box detection, and the calculation expression is: ; Among them, AP c is the average precision of category c.

[0085] F1 score: Comprehensively evaluates the performance of classification tasks: ; Among them, precision (Precision) and recall (Recall) represent: ; ; Among them, TP (True Positive) is the number of samples correctly predicted as positive, FP (False Positive) is the number of samples incorrectly predicted as positive, and FN (False Negative) is the number of samples incorrectly predicted as negative.

[0086] Through these detailed accuracy indicators, the performance of the bird soundprint target detection method based on semi-supervised pre-training of multi-scale spectral features can be comprehensively evaluated to ensure its high adaptability and accuracy in practical applications.

[0087] Model M after fine-tuning task It can accurately identify and locate bird soundprint targets and output the corresponding category and time-frequency position. The output of the model is in the form of: ; where c i Indicates the detected bird category, [x min ,y min , x max ,y max ] are the corresponding bounding box coordinates, and M is the number of detected objects.

[0088] Those skilled in the art will appreciate that the above description is only a preferred embodiment of the present invention and does not limit the present invention. Any modification, equivalent replacement or improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A bird soundprint target detection method, characterized in that: The steps of the method include: The audio signal is converted into a time-frequency domain representation using a short-time Fourier transform to obtain a time-frequency graph, and the time-frequency graph is scaled and spectrum enhanced to enhance the diversity of voiceprint features; Construct a bird soundprint target detection model based on soundprint time-frequency positioning technology, and extract global time-frequency features through a self-attention mechanism neural network; Conduct semi-supervised pre-training on large-scale datasets and use unlabeled data to improve model performance; Weakly supervised pre-training is performed based on the generated semi-supervised dataset to improve the pre-trained model’s understanding of complex voiceprint patterns; The pre-trained model is fine-tuned downstream to optimize the model's task performance.

2. The bird soundprint target detection method according to claim 1, characterized in that: The short-time Fourier transform divides the audio signal into frames and calculates the Fourier transform of each frame to obtain a frequency-time relationship diagram, which is expressed as: , where x[n] is the original audio signal, w(t−n) is the window function, and X(t,f) is each frame of the time-frequency graph.

3. The bird soundprint target detection method according to claim 1, characterized in that: The time-frequency diagram is scaled by using discrete wavelet transform, and the frequency spectrum features at different levels are extracted by adjusting the scales of the frequency and time windows. The expression is: , where a controls the scale of the wavelet, b controls the translation, and ψ(t) is the mother wavelet function.

4. The bird soundprint target detection method according to claim 1, characterized in that: The spectrum enhancement simulates different propagation characteristics in a real environment by adjusting the amplitude of the spectrum at each scale. The expression is: , where S(f,t) is the original spectrum and α(f,t) is the enhancement coefficient, which is dynamically adjusted based on the local changes of frequency f and time t.

5. The bird soundprint target detection method according to claim 1, characterized in that: The self-attention mechanism neural network extracts global time-frequency features and combines the time-frequency positioning technology to improve the detection accuracy; in the self-attention mechanism processing process, the time-frequency map X∈R is input N×d ; Transform through the embedding layer to map the input features to a higher dimensional space; Position encoding PE∈R N×d is a matrix that matches the dimension of the input time-frequency graph and represents the position information of each position. For each position i and each dimension k, the position encoding is defined as: , where N is the number of positions in the time-frequency graph and d is the feature dimension of each position.

6. The bird soundprint target detection method according to claim 5, characterized in that: The input representation of the self-attention mechanism includes query, key and value; the input matrix X is mapped into three different vectors: query Q, key K and value V, expressed as: Q=XW Q , K = XW K , V = XW V , where W Q ∈R d×dk , W K ∈R d×dk , W V ∈R d×dv , where dk and dv are the dimensions of the key and value respectively, X∈R N×d Represents the input time-frequency graph, Q, K, V∈R N×dk Represent the query, key, and value obtained through linear transformation respectively.

7. The bird soundprint target detection method according to claim 1, characterized in that: The expression of the self-attention mechanism is: , where QK T ∈R N×N It is the dot product of the query and the key, which represents the similarity measure between each position in the time-frequency graph. The dot product result is scaled and activated using the softmax function to obtain the attention weight of each position; the attention weight is multiplied by the value vector V to obtain the weighted output.

8. The bird soundprint target detection method according to claim 1, characterized in that: The steps of performing semi-supervised pre-training on a large-scale dataset include: Use manually annotated data for basic supervised learning to learn the basic characteristics and rules of bird voiceprints. Annotate the data set, which has N annotated samples, each sample x i The label is y i , where x i ∈R d represents the time-frequency graph feature vector, y i ∈R c is the bird category label corresponding to the sample, C is the number of bird categories, and the loss function of supervised learning uses cross entropy loss, which is expressed as: ,in, is the model for sample x i The predicted category, the probability distribution obtained through the softmax layer; Based on the trained model, unlabeled samples Make predictions and obtain the category prediction probability distribution of each sample , according to the threshold τ, the prediction with high confidence is selected as the pseudo label, that is: , where τ represents a hyperparameter that controls the confidence threshold of the pseudo-label; After each batch of training, gradually expand the scope of inference, select at least twice as many unlabeled samples for inference, gradually add pseudo-label data, adjust hyperparameters based on new pseudo-label samples, and update the confidence of each new pseudo-label sample. In the semi-supervised learning process, a weighted loss function is used to suppress the influence of low-quality pseudo labels. The weighted loss function expression is: ,in, Represents the confidence of the pseudo-label. For pseudo-labels with lower confidence, a smaller weight is given to reduce their negative impact on the training process.

9. The bird soundprint target detection method according to claim 8, characterized in that: The step of performing weakly supervised pre-training based on the generated semi-supervised dataset includes: Using the InfoNCE (Information Contrastive Loss) method for contrastive learning, each pair of audio samples (x i , x j ) is mapped to an embedding space through the neural network model f θ Get their feature representation: z i = f θ (x i ) and z j = f θ (x j ), calculate the similarity between each pair of positive and negative samples. The goal is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The cosine similarity is used to calculate the similarity: ； Construct a contrast loss function and use InfoNCE loss to maximize the similarity of positive samples and minimize the similarity of negative samples. The expression is: , where τ is the temperature parameter that controls the smoothness of the distribution, N is the number of negative samples, and z k is the feature representation of negative samples; By defining pre-tasks to automatically generate supervision signals, we design self-supervised learning tasks based on time-frequency graphs. Given a time-frequency graph X∈R N×D , where N is the number of frames of the time-frequency graph, D is the feature dimension of each frame, and the time-frequency graph is cut into K blocks, each block is X k ∈R n×d , where n is the number of frames in each small block, d is the feature dimension of each small block, and a prediction task is designed to learn the global structure of the time-frequency graph by maximizing the similarity between blocks. The self-supervised loss function is defined as: , where f(X k ) represents the block f(X k ), is the cosine similarity between blocks; Combining contrastive learning and self-supervised learning, the optimization model captures the diversity of bird voiceprints in a multi-scale and multi-level feature space to obtain the final loss function: , where λ is a hyperparameter that balances the two losses.

10. A bird soundprint target detection system, used to execute the bird soundprint target detection method according to any one of claims 1 to 9, characterized in that: The system comprises: A data preprocessing module for receiving an audio signal and converting it into a time-frequency domain representation using a short-time Fourier transform; Multi-scale feature enhancement module, used to perform multi-scale augmentation on the time-frequency graph, including scale augmentation and spectrum enhancement; The target detection module uses the self-attention mechanism neural network to extract global time-frequency features based on voiceprint time-frequency positioning technology; Semi-supervised learning module, used for semi-supervised pre-training on large-scale datasets; Weakly supervised pre-training module, used for weakly supervised pre-training based on the generated semi-supervised dataset; The model fine-tuning module is used to perform downstream fine-tuning on the pre-trained model to optimize the performance of the model on specific tasks.

Images