Natural environment bird monitoring method and computer device based on multimodal fusion deep learning

Abstract

The present invention relates to the field of bird monitoring, and specifically to a natural environment bird monitoring method and computer device based on multimodal fusion deep learning. The scheme includes: establishing a multimodal bird data set; annotating the multimodal bird data set; preprocessing the annotated data; constructing a multimodal cross-attention Transformer network model; training and optimizing the multimodal cross-attention Transformer network model; jointly analyzing the sound and image data in the multimodal bird data set, using the YOLOv8 target detection model, training the annotated image data, and realizing rapid detection of bird targets; combining the DeepSORT tracking algorithm to track the detected target and obtain the bird's motion trajectory; synchronizing the sound source positioning information with the visual tracking result in time and space to verify the identity and position of the bird. The present invention is suitable for bird monitoring in a natural environment.

Description

Technical Field

[0001] The present invention relates to the field of bird monitoring, and in particular to a natural environment bird monitoring method and a computer device based on multimodal fusion deep learning. Background Art

[0002] Bird monitoring is an important area in ecological research and environmental protection. As an important part of the ecosystem, the health of bird populations reflects the overall health of the ecological environment. Traditional bird monitoring methods mainly rely on manual observation, which is not only inefficient but also difficult to implement in large areas or inaccessible natural environments. Although technological advances in recent years have promoted the development of automated monitoring tools, such as the use of fixed-point cameras and sound recording devices, these methods still have limitations.

[0003] Limited data collection: Traditional monitoring technologies typically only provide limited viewing angles and coverage, making it difficult to fully capture the activities of birds in a wide environment.

[0004] Inadequate data processing: Even if a large amount of data is collected through automated tools, there is often a lack of effective data processing and analysis methods, making it difficult to extract useful information from complex environmental data.

[0005] Poor adaptability to complex environments: Existing automatic monitoring systems often perform poorly under changing natural environmental conditions. For example, their performance decreases significantly in severe weather or in complex background noise.

[0006] Weak real-time monitoring capabilities: Most existing technologies are unable to achieve real-time or near real-time data analysis and feedback, limiting the ability to respond quickly to sudden ecological events.

[0007] In view of these problems, the present invention proposes a new technical solution, which can more efficiently collect and process multimodal data, including visual and sound data, and has stronger environmental adaptability and real-time processing capabilities. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a natural environment bird monitoring method and computer device based on multimodal fusion deep learning, which realizes efficient processing of bird multimodal data and improves the accuracy of bird analysis.

[0009] The present invention adopts the following technical solutions to achieve the above-mentioned purpose. In the first aspect, the present invention provides a natural environment bird monitoring method based on multimodal fusion deep learning, comprising:

[0010] S1. Establish a multimodal bird dataset;

[0011] S2, labeling the multimodal bird dataset;

[0012] S3, preprocessing the labeled data;

[0013] S4. Construct a multimodal cross-attention Transformer network model;

[0014] Perform initial feature extraction on visual and sound data in the multimodal bird dataset;

[0015] For visual data, a pre-trained deep convolutional neural network is used to extract spatial features as follows:

[0016]

[0017] in, is the input image data, are visual features extracted from images;

[0018] For sound data, it is converted into a time-frequency spectrum through short-time Fourier transform, as follows:

[0019]

[0020] in, is the input sound signal, S It is a time spectrum diagram;

[0021] Use a recurrent neural network to extract features from the time-frequency spectrum to obtain sound features. The specific method is as follows:

[0022]

[0023] For sound characteristics;

[0024] The visual features and sound characteristics Convert to sequence data separately through flattening operation:

[0025]

[0026] in, , is the linear embedding matrix, , is the corresponding bias term, is the serialized visual feature, is the serialized sound feature;

[0027] The serialized visual features and sound characteristics They are input into independent Transformer encoders to capture the long-range dependencies within the sequence data, as follows:

[0028]

[0029] For visual features Internal long-range dependencies, For sound characteristics Internal long-range dependencies;

[0030] Introducing a cross-modal attention mechanism, which uses data from one modality as a query and data from another modality as keys and values ​​to calculate cross-modal attention;

[0031] Compute cross-modal attention of visual features to sound features:

[0032]

[0033]

[0034] Compute cross-modal attention of sound features to visual features:

[0035]

[0036]

[0037] in, , , and , , is the learnable projection matrix, is the dimension of the key, represents the cross-modal attention of visual features to sound features. represents the calculation of cross-modal attention of sound features to visual features, Representing visual features Through the projection matrix The resulting query matrix is Indicates sound characteristics Through the projection matrix The resulting key matrix, Indicates sound characteristics Through the projection matrix The resulting value matrix is Indicates sound characteristics Through the projection matrix The resulting query matrix is Representing visual features Through the projection matrix The resulting key matrix, Representing visual features Through the projection matrix The resulting value matrix;

[0038] Fusing the features of the cross-modal attention outputs, including feature concatenation and feedforward neural networks or weighted summation;

[0039] Feature concatenation and feedforward neural network:

[0040] The cross-modal attention results of the two modalities are concatenated and input into the feed-forward network as follows:

[0041]

[0042] Weighted sum:

[0043] The attention outputs of the visual and audio modalities are fused by weighted summation as follows:

[0044]

[0045] in, is the learnable fusion weight, which indicates the relative importance of different modalities in the fusion process, and F represents the fused feature;

[0046] The graph neural network is introduced to model the context of the fusion features. The graph neural network represents bird individuals and their relationships by constructing a graph structure. The node V represents the bird individual, the edge E represents the connection between the bird individuals, and the feature representation of each node In the The layer is updated according to the neighbor node information, the specific method is as follows:

[0047]

[0048] in, is the set of neighbor nodes of node i, is the normalization coefficient, , is the learnable weight matrix, is the activation function, Represents the graph neural network The feature update result of node i in the layer, Indicates that node j is in the graph neural network The feature vector of the layer;

[0049] S5. Train and optimize the multimodal cross-attention Transformer network model;

[0050] S6. Joint analysis of sound and image data in a multimodal bird dataset.

[0051] The generalized cross-correlation phase transform algorithm is used for sound source localization, as follows:

[0052]

[0053] in, and are the Fourier transforms of the sound signals received by the two microphones, is the time variable, is the cross-correlation function, Represents a complex exponential function, which is used to shift the phase of the signal in the frequency domain. f Indicates frequency, J represents an imaginary unit;

[0054] Use the YOLOv8 target detection model to train the labeled image data to achieve rapid detection of bird targets;

[0055] Combined with the DeepSORT tracking algorithm, the detected target is tracked to obtain the movement trajectory of the bird;

[0056] The sound source localization information is synchronized with the visual tracking results in time and space, and the identity and position of the bird are verified through joint analysis.

[0057] Furthermore, step S1 specifically includes:

[0058] The multimodal bird dataset contains data in two modalities: vision and sound;

[0059] Visual data collection, using cameras to collect images and videos of birds in their natural environments, with cameras mounted at different heights and angles to cover a variety of habitats and behavioral scenarios;

[0060] Sound data collection uses a combination of directional microphones and omnidirectional microphones to collect bird calls and environmental sounds. The directional microphone is used to capture bird sounds in a set direction, while the omnidirectional microphone is used to collect background sounds in the environment.

[0061] Combine existing public bird image and sound datasets with the collected data, organize and standardize all the combined data, and unify the data storage format.

[0062] Furthermore, step S2 specifically includes:

[0063] Use labelimg to annotate the image data in the multimodal bird dataset, including the species, age, gender, and behavioral characteristics of the bird target. The annotation results generate files in the corresponding standard format.

[0064] Use Audacity to annotate the sound data in the multimodal bird data set, including the time period, frequency characteristics, and sound source characteristics of the bird sounds, and generate the corresponding standard format files based on the annotated results;

[0065] The annotation of image and sound data is synchronized in time to ensure the consistency of multimodal data in the same time period.

[0066] Furthermore, step S3 specifically includes:

[0067] Image data preprocessing: histogram equalization is performed on each image in the multimodal bird dataset to enhance the contrast of the image. Median filtering and Gaussian filtering are then used to remove noise from the image. The diversity of the data is increased through rotation, scaling, cropping, and color jittering.

[0068] Sound data preprocessing uses frequency domain filtering and noise reduction algorithms to remove background noise from sound data, pre-emphasizes the sound signal, enhances the energy of the high-frequency part, and converts the processed sound signal into Mel-frequency spectrogram and Mel-frequency cepstral coefficient time-frequency features.

[0069] Furthermore, step S5 specifically includes:

[0070] S51, self-supervised pre-training;

[0071] Use unlabeled image and sound data to perform self-supervised pre-training on the visual model and sound model respectively;

[0072] The visual model uses SimCLR or MoCo self-supervised contrastive learning method to learn the potential feature representation of the image. The objective function is:

[0073]

[0074] in, represents the objective function of the visual model, and is the feature representation of the positive sample pair, is the similarity function, is the temperature parameter, 2N represents the total number of sample pairs used in contrastive learning;

[0075] The sound model uses the Contrastive Predictive Coding method to perform self-supervised pre-training on unlabeled sound data;

[0076] S52. After the self-supervised pre-training is completed, the multimodal cross-attention Transformer network model is fine-tuned in a supervised manner based on the labeled dataset.

[0077] By integrating the features of visual and audio modalities, the parameters of the multimodal cross-attention Transformer network model are optimized, and the joint loss function in the training process is optimized as follows:

[0078]

[0079] in, represents the joint loss function, , , are the supervised losses for the visual path, the sound path, and the fusion path, respectively. , , is the loss weight coefficient;

[0080] S53. Set the loss function of the multimodal cross-attention Transformer network model;

[0081] The multimodal cross-attention Transformer network model adopts a multi-task learning framework to simultaneously perform classification, localization, and cross-modal consistency learning. The total loss function consists of the following parts:

[0082] Classification loss:

[0083]

[0084] in, represents the classification loss function, is the true category label, is the predicted probability of the multimodal cross-attention Transformer network model;

[0085] Positioning loss:

[0086]

[0087] in, represents the positioning loss function, and are the true and predicted bounding box positions respectively, using Loss is optimized;

[0088] Cross-modal consistency loss:

[0089]

[0090] in, represents the cross-modal consistency loss function, represents the visual features of the i-th sample, Represents the sound feature corresponding to the i-th visual sample, forming a positive sample pair, represents the jth sound feature, is the cosine similarity function, N represents the number of samples contained in a training batch, is the temperature parameter;

[0091] Total loss function:

[0092]

[0093] Among them, L represents the total loss function, is the weight coefficient of each loss item, which is used to balance the impact between different tasks;

[0094] S54, using reinforcement learning to optimize the multimodal cross-attention Transformer network model strategy;

[0095] Define the reward function R based on the performance of the multimodal cross-attention Transformer network model, including the recognition accuracy A, recall rate R and F1-score indicators, as follows:

[0096]

[0097] in, , , is the weight coefficient, and the policy gradient method is used to optimize the parameter adjustment strategy of the multimodal cross-attention Transformer network model;

[0098] S55. Use grid search and Bayesian optimization methods to search and tune hyperparameters, where the hyperparameters include learning rate, batch size, and network depth.

[0099] In a second aspect, the present invention provides a computer device, including a memory, wherein the memory stores program instructions, and when the program instructions are run, the above-mentioned natural environment bird monitoring method based on multimodal fusion deep learning is executed.

[0100] The beneficial effects of the present invention are:

[0101] The present invention constructs a bird dataset containing high-resolution images and clear bird calls, collects visual and sound data using high-definition cameras and directional and omnidirectional microphones, and expands it with public datasets to provide rich training and verification resources for deep learning models.

[0102] The present invention adopts image enhancement and sound signal enhancement technologies, such as image histogram equalization, sound noise reduction and signal pre-emphasis, to improve data quality and provide optimized input for model training.

[0103] The present invention integrates pre-trained visual and sound models and adopts a dual-path joint modeling network. One path processes visual data and uses a deep convolutional neural network (such as ResNet-50) to extract image features; the other path processes sound data and uses a recurrent neural network (such as LSTM) to process sound waveforms and capture time series dependencies. It combines visual and sound data, improves recognition accuracy through a lightweight attention mechanism, integrates information through a multimodal fusion layer, and fine-tunes on labeled data sets to fine-tune the model to adapt to specific monitoring tasks.

[0104] The present invention defines a reward mechanism based on model performance, uses reinforcement learning to optimize the decision-making process, and adjusts model parameters in real time to adapt to environmental changes.

[0105] The present invention applies the model in different ecological monitoring tasks and evaluates and enhances the generalization ability of the model through transfer learning methods.

[0106] The present invention uses the GCC-PHAT method for sound source localization, combined with advanced target detection algorithms such as YOLOv8 and DeepSORT tracking technology, to achieve real-time dynamic bird monitoring and behavior analysis, thereby improving the accuracy of bird analysis.

[0107] The present invention ensures that the system achieves the best effect in practical applications through hyperparameter adjustment and algorithm optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 It is a flow chart of a natural environment bird monitoring method based on multimodal fusion deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0109] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0110] The present invention provides a natural environment bird monitoring method based on multimodal fusion deep learning, such as Figure 1 As shown, specifically including:

[0111] S1. Establish a multimodal bird dataset;

[0112] The first step of the present invention is to construct a comprehensive multimodal bird dataset covering data in both visual and sound modes. In terms of visual data collection, high-resolution cameras (such as 4K ultra-high-definition cameras) are used to collect images and videos of birds in natural environments such as forests and wetlands. The cameras are installed at different heights and angles to cover various habitats and behavioral scenarios to ensure the diversity and comprehensiveness of the data. In terms of sound data collection, a method combining directional microphones and omnidirectional microphones is used to collect bird calls and environmental sounds. Directional microphones are used to capture bird sounds in specific directions, while omnidirectional microphones are used to collect background sounds in the environment to increase the richness of sound data.

[0113] In order to expand the scale and diversity of the dataset, existing public bird image and sound datasets (such as Xeno-canto, Cornell Lab of Ornithology, etc.) were combined with the collected data. All data were sorted and standardized, and the data storage format was unified, with image data saved in high-resolution JPEG or PNG format and sound data saved in high-quality WAV or FLAC format. The sorted dataset was divided into training set, validation set, and test set in a ratio of 70%, 15%, and 15% to ensure fairness in model training and evaluation.

[0114] S2, labeling the multimodal bird dataset;

[0115] After the data set is established, some data in the training set are finely annotated. In terms of image annotation, professional annotation tools (such as labelimg) are used to annotate the bird targets in the image, including information such as species, age, gender, and behavioral characteristics. The annotation results generate files in standard formats (such as XML or COCO format) for model training and verification. In terms of sound annotation, tools such as Audacity are used to annotate the sound data, annotate the time period, frequency characteristics, and sound source of the bird's song, and generate corresponding label files. In order to achieve effective fusion of multimodal data, the image and sound annotations are synchronized in time to ensure the consistency of multimodal data in the same time period. This requires precise timestamp alignment to ensure the accuracy of subsequent analysis.

[0116] S3, preprocessing the labeled data;

[0117] After the data is labeled, advanced multimodal data preprocessing is performed to improve data quality and provide optimized input for model training. In terms of image data preprocessing, each image is first histogram equalized to enhance the image contrast and make the bird's features more prominent. Then, median filtering, Gaussian filtering and other methods are used to remove noise from the image and improve image quality. Furthermore, data enhancement techniques such as rotation, scaling, cropping, and color jittering are used to increase data diversity and prevent model overfitting.

[0118] In terms of sound data preprocessing, frequency domain filtering and noise reduction algorithms (such as Spectral Gating) are used to remove background noise in sound data and improve the signal-to-noise ratio. The sound signal is pre-emphasized to enhance the energy of the high-frequency part and highlight the characteristics of the bird's song. The processed sound signal is converted into time-frequency features such as Mel-frequency spectrogram and Mel-frequency cepstral coefficient (MFCC) to prepare for subsequent model input. Specifically, the sound signal is transformed by short-time Fourier transform (STFT) to obtain a spectrum:

[0119]

[0120] in, is the original sound signal, is a window function, is the spectrum diagram, represents a complex exponential, used for frequency domain analysis, J represents the imaginary unit, represents the frequency domain component of the signal, A variable representing time, Represents a window function, which is used to intercept or weight the sound signal. Represents the window function at time The signal can be processed in segments to calculate the local spectrum.

[0121] S4. Construct a multimodal cross-attention Transformer network model;

[0122] This paper proposes a new multimodal cross-attention Transformer network framework for effectively fusing visual and sound data to improve the accuracy and robustness of bird monitoring. The design of this framework optimizes the comprehensive feature representation of multimodal data through independent processing of visual and sound modalities, interactive fusion of cross-modal information, and context modeling of graph neural networks.

[0123] First, initial feature extraction is performed on the input visual and audio data. For visual data, a pre-trained deep convolutional neural network (such as ResNet-50) is used to extract spatial features. The specific formula is:

[0124]

[0125] in, is the input image data, is the feature matrix extracted from the image. At the same time, the sound data is converted into a time-frequency spectrum through short-time Fourier transform (STFT), which is expressed as:

[0126]

[0127] in, is the input sound signal. Next, a recurrent neural network (LSTM) is used to extract features from the time-frequency spectrum to obtain the sound features:

[0128]

[0129] Next, in order to process feature information in the Transformer encoder, the visual features and sound characteristics Convert to sequence data separately through flattening operation:

[0130]

[0131] in, , is the linear embedding matrix, , is the corresponding bias term. Next, the serialized visual features and sound characteristics They are input into independent Transformer encoders to capture long-range dependencies within the sequence data:

[0132]

[0133] In order to further enhance the mutual information interaction between visual and sound modalities, a cross-modal attention mechanism is introduced. This mechanism uses the data of one modality as the query (Query) and the data of the other modality as the key (Key) and value (Value) to calculate the cross-modal attention. First, the cross-modal attention of visual features to sound features is calculated:

[0134]

[0135]

[0136] Similarly, we calculate the cross-modal attention of the sound features to the visual features:

[0137]

[0138]

[0139] in, , , and , , is the learnable projection matrix, is the dimension of the key, represents the cross-modal attention of visual features to sound features. represents the calculation of cross-modal attention of sound features to visual features, Representing visual features Through the projection matrix The query matrix obtained by After projection, the present invention can convert the visual features into a feature representation suitable for matching with other modalities (here, sound features). Indicates sound characteristics Through the projection matrix The resulting key matrix, As The matching target is to calculate the similarity between the visual features and the sound features. Indicates sound characteristics Through the projection matrix The obtained value matrix is ​​the value matrix representing the sound features. After multiplying it with the attention weight, we can get the fusion representation of the visual features and the sound features. Indicates sound characteristics Through the projection matrix The query matrix obtained indicates that the present invention wants to find and understand the goal of visual features based on sound features. Representing visual features Through the projection matrix The key matrix obtained is used to help match the sound features to the visual features to calculate the similarity between the two. Representing visual features Through the projection matrix The obtained value matrix is ​​used to describe the value of the visual feature, and is finally multiplied by the calculated attention weight to obtain the fusion representation of the sound feature and the visual feature;

[0140] Through this cross-modal attention mechanism, information between visual and audio modalities is fully interacted and integrated.

[0141] Next, the features of the cross-modal attention output are fused. Two fusion methods are proposed here: feature concatenation and feedforward neural network (FFN) or weighted summation.

[0142] Feature concatenation and feedforward neural network:

[0143] The cross-modal attention results of the two modalities are concatenated and input into the feed-forward network as follows:

[0144]

[0145] Weighted sum:

[0146] The attention outputs of the visual and audio modalities are fused by weighted summation as follows:

[0147]

[0148] in, is the learnable fusion weight, which indicates the relative importance of different modalities in the fusion process, and F represents the fused feature.

[0149] In order to further enhance the model's understanding of the complex relationships between individual birds, a graph neural network (GNN) is introduced to model the context of the fusion features. GNN represents individual birds and their relationships by constructing a graph structure, where the node V represents the individual bird and the edge E represents the connection between individual birds. The feature representation of each node In the The layer is updated according to the neighbor node information, and the formula is as follows:

[0150]

[0151] in, is the set of neighbor nodes of node i, is the normalization coefficient, , is the learnable weight matrix, is the activation function, Represents the graph neural network The feature update result of node i in the layer, Represents the feature vector of node j in the Lth layer of the graph neural network;

[0152] Through GNN modeling, the system can capture the group behavior characteristics and complex spatiotemporal relationships among birds.

[0153] In the overall architecture of the model, the combination of cross-modal Transformer and GNN provides powerful feature representation capabilities for the final bird behavior monitoring and analysis, effectively solving the challenges in multimodal data fusion and individual behavior recognition.

[0154] S5. Train and optimize the multimodal cross-attention Transformer network model;

[0155] After the model framework is built, a multi-task learning framework is used to perform preliminary training and optimization on the model. This framework aims to simultaneously complete classification, localization, and cross-modal consistency learning to fully utilize the joint information of visual and sound data and improve the overall performance of the model.

[0156] Self-supervised pre-training

[0157] First, the visual model and the sound model are pre-trained on a large amount of unlabeled image and sound data. The visual model uses self-supervised contrastive learning methods such as SimCLR or MoCo to learn the potential feature representation of the image. Its objective function is:

[0158]

[0159] in, represents the objective function of the visual model, and is the feature representation of the positive sample pair, is the similarity function, is the temperature parameter. The sound model uses methods such as Contrastive Predictive Coding (CPC) to perform self-supervised pre-training on unlabeled sound data.

[0160] Supervised fine-tuning

[0161] After the self-supervised pre-training is completed, the multimodal cross-attention Transformer network model is fine-tuned in a supervised manner in combination with the labeled data set. By integrating the features of the visual and sound modalities, the parameters of the multimodal cross-attention Transformer network model are further optimized to improve the accuracy of classification and recognition. The joint loss function in the training process is designed as follows:

[0162]

[0163] in, represents the joint loss function, , , are the supervised losses for the visual path, the sound path, and the fusion path, respectively. , , is the loss weight coefficient.

[0164] Multi-task learning and loss function design:

[0165] The model adopts a multi-task learning framework to simultaneously perform classification, localization, and cross-modal consistency learning. The total loss function consists of the following parts.

[0166] Classification loss:

[0167]

[0168] in, represents the classification loss function, is the true category label, is the predicted probability of the model.

[0169] Positioning loss:

[0170]

[0171] in, represents the positioning loss function, and are the true and predicted bounding box positions respectively, using Loss is optimized.

[0172] Cross-modal consistency loss (based on contrastive learning):

[0173]

[0174] in, represents the cross-modal consistency loss function, represents the visual features of the i-th sample, Represents the sound feature corresponding to the i-th visual sample, forming a positive sample pair, represents the jth sound feature, is the cosine similarity function, N represents the number of samples contained in a training batch, is the temperature parameter.

[0175] Total loss function:

[0176]

[0177] Among them, L represents the total loss function, is the weight coefficient of each loss item, which is used to balance the impact between different tasks.

[0178] Reinforcement learning optimization strategy:

[0179] On the basis of multi-task learning, the model also uses reinforcement learning for further optimization, especially for adaptive adjustment in dynamic environments. Define the reward function R based on model performance, including recognition accuracy A, recall rate R and F1-score indicators, the formula is

[0180]

[0181] in, , , is the weight coefficient. Use policy gradient methods (such as Policy Gradient and Actor-Critic) to optimize the model's parameter adjustment strategy so that the model can adaptively adjust itself according to environmental changes.

[0182] Hyperparameter Optimization:

[0183] In order to find the optimal model configuration, the present invention uses grid search and Bayesian optimization methods to search and tune hyperparameters (such as learning rate, batch size, network depth, etc.). By optimizing these hyperparameters, the performance and efficiency of the model are further improved. In order to prevent overfitting, K-fold cross validation is also used to verify the stability and generalization ability of the model under different data set partitions.

[0184] Through the above comprehensive strategy of self-supervised pre-training, supervised fine-tuning, multi-task learning, reinforcement learning and hyperparameter optimization, the model can effectively handle the fusion task of visual and sound modalities, and perform well in the actual application scenarios of bird monitoring and identification.

[0185] S6. Joint analysis of sound and image data in a multimodal bird dataset.

[0186] In the sixth step of the present invention, the joint analysis of sound and image data is realized to improve the accuracy and real-time performance of monitoring. First, the generalized cross-correlation phase transform algorithm is used to locate the sound source. The core calculation is:

[0187]

[0188] in, and are the Fourier transforms of the sound signals received by the two microphones, For time delay, is the cross-correlation function, Represents a complex exponential function, which is used to shift the phase of the signal in the frequency domain. f Indicates frequency, J represents an imaginary unit;

[0189] By calculating the time delay between multiple microphones, the azimuth of the sound source is determined.

[0190] In terms of visual data, the YOLOv8 target detection model is used to train the labeled image data to achieve rapid detection of bird targets. The loss function of the model includes positioning loss and classification loss, which is defined as:

[0191]

[0192] in, is the bounding box positioning loss, is the target classification loss. Combined with the DeepSORT tracking algorithm, the detected target is tracked to obtain the movement trajectory of the bird.

[0193] The sound source localization information is synchronized with the visual tracking results in time and space to ensure the effective fusion of multimodal information. Through joint analysis, the identity and location of the bird can be verified to improve the accuracy of monitoring. After multimodal fusion, real-time monitoring and analysis of bird behavior can be achieved.

[0194] S7, model performance evaluation;

[0195] In the last step, the model performance is comprehensively evaluated and optimized. Quantitative indicators such as accuracy, recall, F1-score, mean average precision (mAP) are used to evaluate the classification and detection capabilities of the model. The classification performance of the model can be expressed as:

[0196]

[0197] in, is the true number of cases, is the number of true negative examples, is the number of false positive cases, is the number of false negative cases.

[0198] The environmental adaptability and anti-interference ability of the model are evaluated through environmental change tests (such as different weather, lighting, and noise conditions) and data perturbation tests (such as adding noise and occlusion). Based on the performance evaluation results, algorithm optimization methods such as model compression and hardware acceleration are used. In terms of model compression, pruning technology is used to remove redundant parameters, and quantization methods are used to reduce model parameters from 32-bit floating point numbers to 8 bits or lower. In terms of hardware acceleration, the parallel computing power of GPU is used to accelerate the training and reasoning process of the model; the model is optimized on embedded devices (such as NVIDIA Jetson Nano) to achieve real-time monitoring in the field.

[0199] Through hyperparameter adjustment and algorithm optimization, the system is ensured to achieve the best results in practical applications, meet the requirements of real-time and accuracy, and provide strong technical support for biodiversity conservation and ecological research.

[0200] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A natural environment bird monitoring method based on multimodal fusion deep learning, characterized in that: include: S1. Establish a multimodal bird dataset; S2, labeling the multimodal bird dataset; S3, preprocessing the labeled data; S4. Construct a multimodal cross-attention Transformer network model; Perform initial feature extraction on visual image data and sound data in the multimodal bird dataset; For visual image data, a pre-trained deep convolutional neural network is used to extract spatial features as follows: in, is the input visual image data, It is a visual feature extracted from visual image data; For sound data, it is converted into a time-frequency spectrum through short-time Fourier transform, as follows: in, is the input sound signal, S It is a time spectrum diagram; Use a recurrent neural network to extract features from the time-frequency spectrum to obtain sound features. The specific method is as follows: For sound characteristics; The visual features and sound characteristics Convert to sequence data separately through flattening operation: in, , is the linear embedding matrix, , is the corresponding bias term, is the serialized visual feature, is the serialized sound feature; The serialized visual features and sound characteristics They are input into independent Transformer encoders to capture the long-range dependencies within the sequence data, as follows: For visual features Internal long-range dependencies, For sound characteristics Internal long-range dependencies; Introducing a cross-modal attention mechanism, which uses data from one modality as a query and data from another modality as keys and values ​​to calculate cross-modal attention; Compute cross-modal attention of visual features to sound features: Compute cross-modal attention of sound features to visual features: in, , , and , , is the learnable projection matrix, is the dimension of the key, represents the cross-modal attention of visual features to sound features. represents the calculation of cross-modal attention of sound features to visual features, Representing visual features Through the projection matrix The resulting query matrix is Indicates sound characteristics Through the projection matrix The resulting key matrix, Indicates sound characteristics Through the projection matrix The resulting value matrix is Indicates sound characteristics Through the projection matrix The resulting query matrix is Representing visual features Through the projection matrix The resulting key matrix, Representing visual features Through the projection matrix The resulting value matrix; Fusing the features of the cross-modal attention outputs, including feature concatenation and feedforward neural networks or weighted summation; Feature concatenation and feedforward neural network: The cross-modal attention results of the two modalities are concatenated and input into the feed-forward network as follows: Weighted sum: The attention outputs of the visual and audio modalities are fused by weighted summation as follows: in, is the learnable fusion weight, which indicates the relative importance of different modalities in the fusion process, and F represents the fused feature; The graph neural network is introduced to model the context of the fusion features. The graph neural network represents bird individuals and their relationships by constructing a graph structure. The node V represents the bird individual, the edge E represents the connection between the bird individuals, and the feature representation of each node In the The layer is updated according to the neighbor node information, the specific method is as follows: in, is the set of neighbor nodes of node i, is the normalization coefficient, , is the learnable weight matrix, is the activation function, Represents the graph neural network The feature update result of node i in the layer, Indicates that node j is in the graph neural network The feature vector of the layer; S5. Train and optimize the multimodal cross-attention Transformer network model; S6. Joint analysis of sound and image data in a multimodal bird dataset. The generalized cross-correlation phase transform algorithm is used for sound source localization, as follows: in, and are the Fourier transforms of the sound signals received by the two microphones, is the time variable, is the cross-correlation function, Represents a complex exponential function, which is used to shift the phase of the signal in the frequency domain. f Indicates frequency, J represents an imaginary unit; Use the YOLOv8 target detection model to train the labeled image data to achieve rapid detection of bird targets; Combined with the DeepSORT tracking algorithm, the detected target is tracked to obtain the movement trajectory of the bird; The sound source localization information is synchronized with the visual tracking results in time and space, and the identity and position of the bird are verified through joint analysis.

2. The natural environment bird monitoring method based on multimodal fusion deep learning according to claim 1 is characterized in that: Step S1 specifically includes: The multimodal bird dataset contains data in two modalities: vision and sound; Visual data collection, using cameras to collect images and videos of birds in their natural environments, with cameras mounted at different heights and angles to cover a variety of habitats and behavioral scenarios; Sound data collection uses a combination of directional microphones and omnidirectional microphones to collect bird calls and environmental sounds. The directional microphone is used to capture bird sounds in a set direction, while the omnidirectional microphone is used to collect background sounds in the environment. Combine existing public bird image and sound datasets with the collected data, organize and standardize all the combined data, and unify the data storage format.

3. The natural environment bird monitoring method based on multimodal fusion deep learning according to claim 1 is characterized in that: Step S2 specifically includes: Use labelimg to annotate the image data in the multimodal bird dataset, including the species, age, gender, and behavioral characteristics of the bird target. The annotation results generate files in the corresponding standard format. Use Audacity to annotate the sound data in the multimodal bird data set, including the time period, frequency characteristics, and sound source characteristics of the bird sounds, and generate the corresponding standard format files based on the annotated results; The annotation of image and sound data is synchronized in time to ensure the consistency of multimodal data in the same time period.

4. The natural environment bird monitoring method based on multimodal fusion deep learning according to claim 1 is characterized in that: Step S3 specifically includes: Image data preprocessing: histogram equalization is performed on each image in the multimodal bird dataset to enhance the contrast of the image. Median filtering and Gaussian filtering are then used to remove noise from the image. The diversity of the data is increased through rotation, scaling, cropping, and color jittering. Sound data preprocessing uses frequency domain filtering and noise reduction algorithms to remove background noise from sound data, pre-emphasizes the sound signal, enhances the energy of the high-frequency part, and converts the processed sound signal into Mel-frequency spectrogram and Mel-frequency cepstral coefficient time-frequency features.

5. The natural environment bird monitoring method based on multimodal fusion deep learning according to claim 1 is characterized in that: Step S5 specifically includes: S51, self-supervised pre-training; Use unlabeled image and sound data to perform self-supervised pre-training on the visual model and sound model respectively; The visual model uses SimCLR or MoCo self-supervised contrastive learning method to learn the potential feature representation of the image. The objective function is: in, represents the objective function of the visual model, and is the feature representation of the positive sample pair, is the similarity function, is the temperature parameter, 2N represents the total number of sample pairs used in contrastive learning; The sound model uses the Contrastive Predictive Coding method to perform self-supervised pre-training on unlabeled sound data; S52. After the self-supervised pre-training is completed, the multimodal cross-attention Transformer network model is fine-tuned in a supervised manner based on the labeled dataset. By integrating the features of visual and audio modalities, the parameters of the multimodal cross-attention Transformer network model are optimized, and the joint loss function in the training process is optimized as follows: in, represents the objective function of the visual model, , , are the supervised losses for the visual path, the sound path, and the fusion path, respectively. , , is the loss weight coefficient; S53. Set the loss function of the multimodal cross-attention Transformer network model; The multimodal cross-attention Transformer network model adopts a multi-task learning framework to simultaneously perform classification, localization, and cross-modal consistency learning. The total loss function consists of the following parts: Classification loss: in, represents the classification loss function, is the true category label, is the predicted probability of the multimodal cross-attention Transformer network model; Positioning loss: in, represents the positioning loss function, and are the true and predicted bounding box positions respectively, using Loss is optimized; Cross-modal consistency loss: in, represents the cross-modal consistency loss function, represents the visual features of the i-th sample, Represents the sound feature corresponding to the i-th visual sample, forming a positive sample pair, represents the jth sound feature, is the cosine similarity function, N represents the number of samples contained in a training batch, is the temperature parameter; Total loss function: Among them, L represents the total loss function, is the weight coefficient of each loss item, which is used to balance the impact between different tasks; S54, using reinforcement learning to optimize the multimodal cross-attention Transformer network model strategy; Define the reward function R based on the performance of the multimodal cross-attention Transformer network model, including the recognition accuracy A, recall rate R and F1-score indicators, as follows: in, , , is the weight coefficient, and the policy gradient method is used to optimize the parameter adjustment strategy of the multimodal cross-attention Transformer network model; S55. Use grid search and Bayesian optimization methods to search and tune hyperparameters, where the hyperparameters include learning rate, batch size, and network depth.

6. A computer device comprising a memory, wherein the memory stores program instructions, characterized in that: When the program instructions are executed, the natural environment bird monitoring method based on multimodal fusion deep learning as described in any one of claims 1 to 5 is executed.

Images