Bird identification method and device based on sound-image multi-modal fusion

Abstract

The invention discloses a bird identification method based on sound-image multi-modal fusion. The bird identification method comprises the following steps: S1, carrying out standardized frame-level preprocessing on bird audio signals; s2, acoustic features are extracted and enhanced, and an acoustic high-level feature vector which highlights birdsong discrimination information and suppresses environmental noise is obtained; s3, visual image standardization preprocessing; s4, performing visual feature extraction and multi-scale fusion to obtain a visual high-level feature vector which enhances correspondence to the bird key form area and inhibits background interference; s5, performing dynamic weighted fusion on the decision-making layer to obtain a bird existence probability; and S6, comparing the bird existence probability with a preset threshold value of the corresponding bird, and judging whether the bird exists or not and the type of the existing bird. Through cross-modal feature enhancement and adaptive fusion, the precision, robustness and real-time performance of bird recognition in a complex orchard environment are significantly improved, and a core technical support is provided for green intelligent bird repelling.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent agricultural equipment technology, specifically a bird recognition method and device based on sound-image multimodal fusion. Background Technology

[0002] The fruit industry is an important component of my country's agricultural economy. Statistics show that my country's orchard area has reached tens of millions of hectares, but over 98% of these orchards suffer from bird damage to varying degrees every year, resulting in fruit yield losses of over 30% and causing huge economic losses. Bird pecking not only directly reduces the commercial value of the fruit but can also spread pests and diseases, further impacting the ecological health of the orchard.

[0003] Currently, commonly used bird deterrence methods mainly include bird repellents, bird nets, and fruit bagging. These methods suffer from low efficiency, high cost, and environmental pollution. In recent years, acoustic and optical bird deterrence technologies have attracted attention due to their environmentally friendly characteristics, such as ultrasonic bird deterrents and laser bird deterrents. However, existing technologies still have significant limitations: (1) Low reliability of single-modal recognition: Acoustic bird deterrents are easily affected by environmental noise, and birds are prone to adapting to fixed sound waves; visual bird deterrents have a sharp drop in recognition accuracy in complex scenarios such as changes in lighting, foliage obstruction, and small targets at a distance. (2) Lack of intelligent and accurate recognition capabilities: Most devices cannot identify bird species and activity patterns, and can only drive away birds indiscriminately, resulting in wasted energy and poor bird deterrence effects. (3) Simple existing fusion methods: Some studies have attempted multi-sensor fusion, but most of them remain at the level of feature splicing or decision voting, failing to achieve adaptive deep fusion based on environmental context.

[0004] Therefore, there is an urgent need for an intelligent bird identification method that can adapt to complex environments, has strong anti-interference capabilities, high recognition accuracy, and can achieve precise bird-based frequency control for driving away birds. Summary of the Invention

[0005] The purpose of this invention is to address the problems of poor robustness and susceptibility to interference in complex orchard environments by single-modal recognition, as well as the insufficient adaptive capability of existing multimodal fusion methods. It provides a bird identification method, device, electronic device, and storage medium based on sound-image multimodal fusion that can efficiently, accurately, and in real-time identify the presence and species of birds. Based on the identification results, it directly matches the corresponding frequency of sound to deter birds, achieving precise bird deterrence by "frequency-based deterrence." This invention is suitable for real-time bird monitoring and intelligent deterrence systems in complex, unstructured environments such as orchards and woodlands, and is of great significance for ensuring agricultural and forestry economic benefits and promoting the development of green and intelligent agriculture.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] As a first aspect of this application, some embodiments of this application provide a fruit posture recognition method, including:

[0008] A bird recognition method based on sound-image multimodal fusion, characterized by the following steps:

[0009] S1. Perform standardized frame-level preprocessing on the collected bird audio signals to ensure the consistency of acoustic features and temporal alignment.

[0010] S2. Acoustic Feature Extraction and Enhancement: For the pre-processed bird audio signal at the frame level, the Mel spectrogram and MFCC features of each frame are extracted to obtain the logarithmic Mel spectrogram. The MFCC features are mapped using spline interpolation to obtain extended MFCC features that match the dimensions and time domain of the logarithmic Mel spectrogram. The logarithmic Mel spectrogram and extended MFCC features are stacked in the channel dimension to form a dual-channel acoustic feature tensor. This dual-channel acoustic feature tensor is input into a convolutional neural network with an embedded SE attention mechanism for feature enhancement and recalibration to obtain a high-level acoustic feature vector that highlights bird call discrimination information and suppresses environmental noise.

[0011] S3. Perform standardized preprocessing on visual images acquired synchronously with bird audio signals to ensure consistency of model input;

[0012] S4. Visual Feature Extraction and Multi-Scale Fusion: The preprocessed visual image is input into the MobileViT backbone network to extract shallow, mid, and deep feature maps. Bilinear interpolation is used to upsample the mid and deep feature maps to the size of the shallow feature map. Shallow, mid, and deep feature maps of the same size are concatenated along the channel dimension and fused using 1×1 convolution to obtain a multi-scale feature map. The multi-scale feature map is then input into the CBAM attention module for channel attention and spatial attention weighting to obtain a visual high-level feature vector that enhances the response to key morphological regions of birds and suppresses background interference. The dimension and temporal domain of the visual high-level feature vector match the acoustic high-level feature vector obtained in step S2.

[0013] S5. Dynamic Weighted Fusion at the Decision Level: The acoustic high-level feature vector obtained in step S2 and the visual high-level feature vector obtained in step S4 are processed by independent classifiers to obtain acoustic modality classification probabilities and visual modality visual probabilities, respectively. The acoustic confidence scores and visual confidence scores are obtained by independent confidence sub-networks, respectively. The acoustic confidence scores and visual confidence scores are concatenated and then input into a Softmax function with controllable temperature parameters for normalization to obtain the dynamic weights for fusing acoustic and visual modalities. The acoustic modality classification probabilities, visual modality visual probabilities, and the dynamic weights for fusing acoustic and visual modalities are weighted and summed to obtain the probability of bird presence.

[0014] S6. Compare the probability of bird presence obtained in step S5 with the preset threshold of the corresponding bird to determine whether there are birds and the types of birds present.

[0015] The standardized frame-level preprocessing method in step S1 is as follows: the original audio is first pre-emphasized to enhance high-frequency components, and then the signal is framed and windowed according to a fixed sampling rate. The frame length and frame shift are set according to the sampling rate and target duration. The parameter configuration is as follows: sampling rate 44100Hz, frame parameters: n_fft=2048, hop_length=512.

[0016] In step S2, the MFCC features are nonlinearly extended from 40 dimensions to 128 dimensions using a cubic spline interpolation function and aligned on the time axis to obtain extended MFCC features that match the dimensions of the logarithmic Mel spectrogram and are aligned in the time domain.

[0017] The formula for fusing Mel spectrograms and MFCC features is:

[0018] X audio =Stack[Mel db [CubicSpline(MFCC)] (1)

[0019] In formula (1): X audio For two-channel acoustic feature tensors; Mel db For the logarithmically transformed Mel spectrum; CubicSpline represents the cubic spline interpolation function.

[0020] The convolutional neural network in step S2 uses a ResNet-SE Backbone backbone network with dual-channel input. In each Bottleneck unit of the ResNet-SE Backbone backbone network, an SE attention mechanism is embedded to adaptively recalibrate the channels.

[0021] The output of the SE attention mechanism is:

[0022] Xse =X·σ(W2·ReLU(W1·GAP(X))) (2)

[0023] In equation (2): σ is the Sigmoid function; GAP is global average pooling; W1 and W2 are the weights of the fully connected layer.

[0024] The visual image standardization preprocessing step in step S3 is as follows: the original image is scaled to a uniform resolution according to the network input requirements to achieve size uniformity, the pixel values ​​are mapped to the [0,1] interval for normalization, and the mean and standard deviation of the image are calculated by channel to obtain channel standardization.

[0025] The fusion formula for the multi-scale feature maps in step S4 is as follows:

[0026] F fused =Conv 1×1 (Concat[F low UpSample(F) mid ),UpSample(F high (3)

[0027] In equation (3): F low F mid F high These represent shallow, medium, and deep feature maps, respectively. UpSample represents the bilinear upsampling operation, Concat represents the channel concatenation operation, and Conv represents the deep feature map. 1×1 This represents a 1×1 convolution fusion operation.

[0028] The output of the dynamic weight generator used for normalization processing of the Softmax function in step S5 is:

[0029]

[0030] In equation (4): c v c is the raw score for visual confidence. a is the raw acoustic confidence score; T is the temperature hyperparameter used to control the smoothness of the weight distribution.

[0031] As a second aspect of this application, some embodiments of this application provide a bird recognition device based on acoustic-image multimodal fusion, comprising:

[0032] Audio signal acquisition equipment: used to acquire bird audio signals and input them into the audio signal preprocessing module;

[0033] Audio signal preprocessing module: Performs standardized frame-level preprocessing on the acquired bird audio signals to ensure the consistency of acoustic features and temporal alignment, and inputs the frame-level preprocessed bird audio signals into the acoustic feature processing module;

[0034] Acoustic Feature Processing Module: For the pre-processed bird audio signal at the frame level, extract the Mel spectrogram and MFCC features of each frame to obtain the logarithmic Mel spectrogram. Then, use spline interpolation to map the MFCC features and obtain extended MFCC features that match the dimensions and time domain of the logarithmic Mel spectrogram. Stack the logarithmic Mel spectrogram and extended MFCC features in the channel dimension to form a dual-channel acoustic feature tensor. Input the dual-channel acoustic feature tensor into a convolutional neural network with an embedded SE attention mechanism for feature enhancement and recalibration to obtain a high-level acoustic feature vector that highlights bird call discrimination information and suppresses environmental noise. Input the vector into the decision fusion module.

[0035] Visual image acquisition device: synchronously acquires visual images of the same environment as the audio signal acquisition device and inputs them into the visual image preprocessing module;

[0036] Visual image preprocessing module: Performs standardized preprocessing on visual images acquired synchronously with bird audio signals to ensure consistency of model input, and inputs the preprocessed visual images into the visual feature processing module;

[0037] The visual feature processing module receives the preprocessed visual image and extracts shallow, mid, and deep feature maps. It uses bilinear interpolation to upsample the mid and deep feature maps to the size of the shallow feature map. Shallow, mid, and deep feature maps of the same size are concatenated along the channel dimension and fused using a 1×1 convolution to obtain a multi-scale feature map. This multi-scale feature map is then input into the CBAM attention module for sequential channel and spatial attention weighting to obtain a visual high-level feature vector that enhances the response to key morphological regions of birds and suppresses background interference. Finally, the visual high-level feature vector, whose dimension and temporal domain match the acoustic high-level feature vector from step S2, is input into the decision fusion module.

[0038] The decision fusion module receives acoustic high-level feature vectors and visual high-level feature vectors. It obtains acoustic modality classification probabilities and visual modality visual probabilities through independent classifiers, and acoustic confidence scores and visual confidence scores through independent confidence sub-networks. The acoustic and visual confidence scores are concatenated and then input into a temperature-controlled Softmax function for normalization. The dynamic weights of the fused acoustic and visual modalities are obtained. The acoustic modality classification probabilities, visual modality visual probabilities, and dynamic weights of the fused acoustic and visual modalities are weighted and summed to obtain the probability of bird presence, which is then input into the decision module.

[0039] Judgment module: Receives the probability of bird presence and compares it with the corresponding preset threshold for birds to determine whether birds exist and the types of birds present.

[0040] As a third aspect of this application, some embodiments of this application provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0041] As a fourth aspect of this application, some embodiments of this application provide a storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any implementation of the first aspect above.

[0042] This invention is implemented through a combination of software and hardware, specifically including an audio sensor, a camera, an embedded processor, and corresponding algorithm modules. The acoustic branch uses tools such as Librosa to extract audio features and selects a pruned ResNet network to process high-level acoustic features. The visual branch uses OpenCV to process images and uses lightweight networks such as Mobile ViT to process high-level visual features. The fusion module combines adaptive weights and a confidence network to output the probability of bird presence. All of the above methods are implemented based on the PyTorch framework. The core of the bird recognition method based on sound-image multimodal fusion lies in innovation at three levels: feature enhancement, multi-scale fusion, and adaptive decision-making. Specifically:

[0043] I. Acoustic Feature Extraction and Enhancement Processing:

[0044] In the acoustics branch, this invention performs standardized frame-level preprocessing on audio signals during the data preprocessing stage to ensure the consistency of acoustic features and timing alignment. The specific process is as follows: The original audio is first pre-emphasized to enhance high-frequency components. Then, the signal is framed and windowed at a fixed sampling rate (the frame length and frame shift are set according to the sampling rate and target duration, with parameters configured as 44100Hz, n_fft=2048, and hop_length=512). For each frame, the Mel spectrogram (n_mels=128) is calculated and the logarithmic scale is taken. At the same time, the MFCC feature (n_mfcc=40) is calculated. In order to achieve the alignment of the Mel spectrogram and the MFCC feature in the channel and time domain, the cubic spline interpolation function is used to nonlinearly extend the MFCC feature from 40 dimensions to 128 dimensions and align them on the time axis, so that the two types of features can be stacked in shape. The aligned extended MFCC and the logarithmic Mel spectrogram are stacked in the channel dimension to obtain a two-channel acoustic feature tensor of the form (2,128,T). The dual-channel acoustic feature tensor input is based on the ResNet acoustic backbone (2 channels). Each Bottleneck unit embeds an SE (Squeeze-and-Excitation) module to adaptively recalibrate the channels, highlighting the channel components containing bird call discrimination information and suppressing environmental noise. Finally, the acoustic high-level feature vector is obtained through adaptive pooling for subsequent discrimination.

[0045] II. Visual Feature Extraction and Multi-Scale Attention Fusion:

[0046] In the visual branch, this invention employs an improved MobileViT backbone network as the visual feature extractor and performs standardization processing on the input image during the data preprocessing stage to ensure the consistency of the model input. Specifically, the original image is first subjected to size unification and normalization processing (scaling pixel values ​​to a uniform range and standardizing them by channel), and then input into the MobileViT backbone. The backbone network retains three layers of features: shallow (Feat2), middle (Feat3), and deep (Feat4). To enhance the small object detection capability, bilinear interpolation is used to upsample the middle and deep features to the shallow layer size. After stacking in the channel dimension, dimensionality reduction and fusion are performed through 1×1 convolution (in the implementation, the number of channels after concatenation is mapped to fusion_out_channels = 128), and the fused representation is input into subsequent MobileViT layers to obtain higher-level semantic representations. The basic convolutional units of the visual backbone adopt ConvLayer with BatchNorm (momentum=0.1) and ReLU activation; at the fusion output of MobileViTBlock, a CBAM attention module is introduced to perform channel and spatial weighting sequentially, which enhances the response to key morphological regions of birds and suppresses background noise such as branches, leaves and lighting, thereby improving the discrimination ability of visual modal under conditions of occlusion, lighting changes and distant small targets. Finally, the high-level visual feature vectors are used for subsequent discrimination.

[0047] III. Dynamic weighted fusion at the decision-making level:

[0048] At the decision-level, this invention employs decision-level adaptive weighted fusion. The visual and acoustic branches are connected to independent prediction heads at their outputs to obtain single-modal logit (v_logit, a_logit), and single-modal probabilities (P_vision, P_audio) are obtained through a sigmoid function. To achieve adaptive weighting, this invention designs two lightweight confidence evaluation subnetworks (each consisting of two fully connected layers with a hidden dimension of 32, ReLU activation, and a single scalar confidence score as the output). The high-level visual and acoustic feature vectors are input into these subnetworks to obtain confidence scalars v_conf and a_conf, respectively. These are then concatenated and normalized using a Softmax normalization method with a temperature parameter T controllable to generate dynamic weights W_visual and W_audio. The final fusion probability is obtained by weighted summation: P_fused = W_audio × P_audio + W_vision × P_vision. To ensure training stability, the present invention preferably adopts a warm-up training strategy: equal-weighted fusion is used for the first 5 rounds of training, and after the warm-up period, the adaptive weighted training described above is switched.

[0049] The present invention has the following advantages over the prior art:

[0050] This invention effectively overcomes the limitations of a single modality by fusing information from two complementary modalities: acoustic and visual. When the acoustic modality is affected by strong noise interference or the visual modality experiences performance degradation due to lighting or occlusion, the other modality can provide effective supplementation, enabling the system to maintain high recognition accuracy (experiments show a recognition rate ≥90%) even in complex orchard environments, significantly enhancing robustness. The SE attention mechanism and CBAM attention module introduced in the feature extraction stage can adaptively focus on key feature channels and spatial regions related to birds, actively filtering environmental noise and redundant information, thereby greatly improving the model's anti-interference and feature selection capabilities. The dynamic weight fusion strategy adopted can intelligently allocate weights based on the prediction confidence of each modality of the input sample, rather than simply averaging, allowing the system to flexibly adapt to various complex scenarios, improving both recognition performance and energy efficiency. In addition, the backbone networks such as MobileViT and ResNet used have excellent lightweight characteristics, facilitating deployment in embedded devices such as Jetson Nano and Raspberry Pi, laying a solid technical foundation for developing low-cost, high-reliability intelligent bird deterrence equipment for fields, and demonstrating significant engineering application value. Attached Figure Description

[0051] Appendix Figure 1 A flowchart illustrating the bird recognition method based on acoustic-image multimodal fusion provided by this invention;

[0052] Appendix Figure 2 A flowchart illustrating the workflow of the acoustic feature processing module provided by this invention;

[0053] Appendix Figure 3 A flowchart illustrating the workflow of the visual feature processing module provided by this invention;

[0054] Appendix Figure 4 The flowchart of the decision fusion module and judgment module provided by the present invention. Detailed Implementation

[0055] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and therefore their detailed description will be omitted.

[0056] The terms “a,” “one,” “the,” and “the” are used to indicate the existence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended meaning of inclusion and that other elements / components / etc. may exist in addition to the listed elements / components / etc.

[0057] like Figure 1-4 A bird recognition method and apparatus based on sound-image multimodal fusion are provided. The bird recognition apparatus includes:

[0058] Audio signal acquisition equipment (selected audio sensor): used to acquire bird audio signals and input them into the audio signal preprocessing module;

[0059] Audio signal preprocessing module: Performs standardized frame-level preprocessing on the acquired bird audio signals to ensure the consistency of acoustic features and temporal alignment, and inputs the frame-level preprocessed bird audio signals into the acoustic feature processing module;

[0060] Acoustic Feature Processing Module: For the pre-processed bird audio signal at the frame level, extract the Mel spectrogram and MFCC features of each frame to obtain the logarithmic Mel spectrogram. Then, use spline interpolation to map the MFCC features and obtain extended MFCC features that match the dimensions and time domain of the logarithmic Mel spectrogram. Stack the logarithmic Mel spectrogram and extended MFCC features in the channel dimension to form a dual-channel acoustic feature tensor. Input the dual-channel acoustic feature tensor into a convolutional neural network with an embedded SE attention mechanism for feature enhancement and recalibration to obtain a high-level acoustic feature vector that highlights bird call discrimination information and suppresses environmental noise. Input the vector into the decision fusion module.

[0061] Visual image acquisition device (camera selected): synchronously acquires visual images of the same environment as the audio signal acquisition device and inputs them into the visual image preprocessing module;

[0062] Visual image preprocessing module: Performs standardized preprocessing on visual images acquired synchronously with bird audio signals to ensure consistency of model input, and inputs the preprocessed visual images into the visual feature processing module;

[0063] The visual feature processing module receives the preprocessed visual image and extracts shallow, mid, and deep feature maps. It uses bilinear interpolation to upsample the mid and deep feature maps to the size of the shallow feature map. Shallow, mid, and deep feature maps of the same size are concatenated along the channel dimension and fused using a 1×1 convolution to obtain a multi-scale feature map. This multi-scale feature map is then input into the CBAM attention module for sequential channel and spatial attention weighting to obtain a visual high-level feature vector that enhances the response to key morphological regions of birds and suppresses background interference. Finally, the visual high-level feature vector, whose dimension and temporal domain match the acoustic high-level feature vector from step S2, is input into the decision fusion module.

[0064] The decision fusion module receives acoustic high-level feature vectors and visual high-level feature vectors. It obtains acoustic modality classification probabilities and visual modality visual probabilities through independent classifiers, and acoustic confidence scores and visual confidence scores through independent confidence sub-networks. The acoustic and visual confidence scores are concatenated and then input into a temperature-controlled Softmax function for normalization. The dynamic weights of the fused acoustic and visual modalities are obtained. The acoustic modality classification probabilities, visual modality visual probabilities, and dynamic weights of the fused acoustic and visual modalities are weighted and summed to obtain the probability of bird presence, which is then input into the decision module.

[0065] Judgment module: Receives the probability of bird presence and compares it with the corresponding preset threshold for birds to determine whether birds exist and the types of birds present.

[0066] like Figure 1-4 A bird recognition method based on sound-image multimodal fusion is provided. The specific steps of this bird recognition method are as follows:

[0067] S1. Standardize the frame-level preprocessing of the collected bird audio signals to ensure the consistency of acoustic features and timing alignment. The standardized frame-level preprocessing method is as follows: first, pre-emphasize the original audio to enhance the high-frequency components, then divide the signal into frames and add windows according to a fixed sampling rate. The frame length and frame shift are set according to the sampling rate and target duration. The parameter configuration is as follows: sampling rate 44100Hz, frame parameters: n_fft=2048, hop_length=512.

[0068] S2. Acoustic Feature Extraction and Enhancement: For the pre-processed bird audio signal at the frame level, the Mel spectrogram and MFCC features of each frame are extracted to obtain the logarithmic Mel spectrogram. The MFCC features are mapped using spline interpolation to obtain extended MFCC features that match the dimensions and time domain of the logarithmic Mel spectrogram. The logarithmic Mel spectrogram and extended MFCC features are stacked in the channel dimension to form a dual-channel acoustic feature tensor. This dual-channel acoustic feature tensor is input into a convolutional neural network with an embedded SE attention mechanism for feature enhancement and recalibration to obtain a high-level acoustic feature vector that highlights bird call discrimination information and suppresses environmental noise.

[0069] The formula for fusing the Mel spectrogram and MFCC features is as follows:

[0070] X audio =Stack[Mel db [CubicSpline(MFCC)] (1)

[0071] In formula (1): X audio For two-channel acoustic feature tensors; Mel db The spectral density is logarithmically transformed into a Mel spectrum; CubicSpline represents the cubic spline interpolation function.

[0072] The convolutional neural network uses a ResNet-SE Backbone with dual-channel input. Each Bottleneck unit of the ResNet-SE Backbone embeds an SE attention mechanism to adaptively recalibrate the channels. The output of the SE attention mechanism is:

[0073] X se =X·σ(W2·ReLU(W1·GAP(X))) (2)

[0074] In equation (2): σ is the Sigmoid function; GAP is global average pooling; W1 and W2 are the weights of the fully connected layer;

[0075] S3. Perform standardization preprocessing on visual images acquired synchronously with bird audio signals to ensure consistency of model input. The standardization preprocessing steps for visual images are as follows: scale the original image to a uniform resolution according to the network input requirements to unify the size, map the pixel values ​​to the [0,1] interval for normalization, and calculate the mean and standard deviation of the image by channel to obtain channel standardization.

[0076] S4. Visual Feature Extraction and Multi-Scale Fusion: The preprocessed visual image is input into the MobileViT backbone network to extract shallow, mid, and deep feature maps. Bilinear interpolation is used to upsample the mid and deep feature maps to the size of the shallow feature map. Shallow, mid, and deep feature maps of the same size are concatenated along the channel dimension and fused using 1×1 convolution to obtain a multi-scale feature map. The multi-scale feature map is then input into the CBAM attention module for channel attention and spatial attention weighting to obtain a visual high-level feature vector that enhances the response to key morphological regions of birds and suppresses background interference. The dimension and temporal domain of the visual high-level feature vector match the acoustic high-level feature vector obtained in step S2.

[0077] The formula for fusing multi-scale feature maps is as follows:

[0078] F fused =Conv 1×1 (Concat[F low UpSample(F) mid ),UpSample(F high (3)

[0079] In equation (3): F low F mid F high These represent shallow, medium, and deep feature maps, respectively. UpSample represents the bilinear upsampling operation, Concat represents the channel concatenation operation, and Conv represents the deep feature map. 1×1 This represents a 1×1 convolution fusion operation;

[0080] S5. Dynamic Weighted Fusion at the Decision Level: The acoustic high-level feature vector obtained in step S2 and the visual high-level feature vector obtained in step S4 are processed by independent classifiers to obtain acoustic modality classification probabilities and visual modality visual probabilities, respectively. The acoustic confidence scores and visual confidence scores are obtained by independent confidence sub-networks, respectively. The acoustic confidence scores and visual confidence scores are concatenated and then input into a Softmax function with controllable temperature parameters for normalization to obtain the dynamic weights for fusing acoustic and visual modalities. The acoustic modality classification probabilities, visual modality visual probabilities, and the dynamic weights for fusing acoustic and visual modalities are weighted and summed to obtain the probability of bird presence.

[0081] The output of the dynamic weight generator used for normalization in the Softmax function is:

[0082]

[0083] In equation (4): c v c is the raw score for visual confidence.a is the raw acoustic confidence score; T is the temperature hyperparameter used to control the smoothness of the weight distribution;

[0084] S6. Compare the probability of bird presence obtained in step S5 with the preset threshold of the corresponding bird to determine whether there are birds and the types of birds present.

[0085] The overall flowchart of the method of this invention is as follows: Figure 1 As shown, the device consists of an audio signal acquisition device, an audio signal preprocessing module, an acoustic feature processing module with an embedded SE attention mechanism, a visual image acquisition device, a visual image preprocessing module, a visual feature processing module with an embedded CBAM attention mechanism, a decision fusion module, and a judgment module. The audio input is processed through feature extraction and a ResNet network incorporating the SE attention mechanism to obtain a high-level acoustic feature vector; the image input is processed through multi-scale fusion and a MobileViT network incorporating the CBAM attention mechanism to obtain a high-level visual feature vector; the classification logits and confidence scores of the two high-level feature vectors are calculated respectively, and the confidence scores are normalized using Softmax to generate dynamic weights. Finally, the classification probabilities are weighted and summed to output the probability of bird presence.

[0086] The bird recognition method based on sound-image multimodal fusion provided by this invention requires each module to be trained before practical application.

[0087] 1. Collect raw audio and visual image data of common birds in orchards and natural environments. Audio data includes calls from different time periods and weather conditions, while image data includes bird images under different lighting conditions, postures, and occlusions. The collected raw audio data underwent preprocessing such as pre-emphasis, frame segmentation, and windowing; the images were then standardized and resized. Subsequently, the overall data was divided into training, validation, and test sets in a 7:2:1 ratio. To improve the model's generalization ability, data augmentation strategies were applied to the training set: audio data underwent random noise addition, speed adjustment, and pitch shifting; image data underwent random horizontal flipping, random rotation, sharpening, and filtering.

[0088] 2. Acoustic feature extraction and enhancement, acoustic feature processing such as... Figure 2As shown, acoustic features were extracted from the preprocessed audio using tools such as Librosa. Specifically, the Mel spectrogram (128-dimensional) and MFCC features (40-dimensional) were calculated for each frame of audio. To achieve feature dimension matching and information complementarity, a cubic spline interpolation function was used to expand the MFCC features from 40-dimensional to 128-dimensional, and stacked with the logarithmic Mel spectrogram in the channel dimension to construct a two-channel acoustic feature tensor of shape (2, 128, T). This tensor was input into a convolutional neural network (ResNet50) with an embedded SE (Squeeze-and-Excitation) attention mechanism module for deep feature extraction and enhancement. The SE attention mechanism module compressed spatial information through global average pooling, and then generated channel weights through two fully connected layers to adaptively calibrate the features, enhance the discriminative features in bird calls and suppress environmental noise, thus obtaining high-level acoustic feature vectors.

[0089] 3. Visual feature extraction and multi-scale fusion; visual multi-scale fusion and attention module structure diagram as shown below. Figure 3 As shown, after the image is input into the Backbone network, three layers of features are extracted: shallow features (Feat2), mid-layer features (Feat3), and deep features (Feat4). The mid-layer and deep features are upsampled and concatenated with the shallow features, then fused through a 1x1 convolution. The fused features are then passed through the channel attention module and the spatial attention module in sequence, outputting a weighted feature.

[0090] 4. Dynamic weighted fusion at the decision-making level, as illustrated in the diagram below. Figure 4 As shown in the diagram, the acoustic high-level feature vector output from step 2 and the visual high-level feature vector output from step 3 are passed through independent classifiers (fully connected layers) to obtain the classification probabilities P_audio for the acoustic modality and P_vision for the visual modality. Simultaneously, these feature vectors are input into a lightweight confidence evaluation sub-network (FC-ReLU-FC), which outputs confidence scores C_audio and C_vision representing the prediction reliability of each modality. The two confidence scores are concatenated and then normalized using a Softmax function with a temperature parameter T, dynamically generating the acoustic modality weight W_audio and the visual modality weight W_vision. Finally, the probability of bird presence is obtained by weighted summation: P_fused = W_audio × P_audio + W_vision × P_vision. The presence of birds is determined based on whether P_fused exceeds a preset threshold, and the category information is output.

[0091] Example

[0092] To more clearly illustrate the specific implementation of this invention, the following focuses on the steps in the year classification method provided by this invention, including image preprocessing and enhancement, audio preprocessing and feature alignment, acoustic backbone and SE integration, model initialization and training strategies, and inference output and post-processing. The numerical values ​​given in the following embodiments are preferred examples and can be appropriately adjusted according to the actual dataset and engineering needs, but should not be construed as limiting the scope of protection of this invention.

[0093] I. Image Preprocessing and Enhancement

[0094] Raw data acquisition: Acquire RGB images of the orchard monitoring area;

[0095] Uniform size: The original image is scaled to a uniform resolution (224×224 pixels) according to the network input requirements;

[0096] Normalization: Maps pixel values ​​to the [0,1] range;

[0097] Channel standardization: Standardize the image by channel using mean and standard deviation;

[0098] Data augmentation during training: random horizontal flipping, random rotation, filtering, sharpening, etc.;

[0099] Input backbone: Input the standardized and enhanced image into the MobileViT backbone, and extract the high-level visual feature vectors for subsequent modules according to the multi-scale stitching-fusion process described in this invention (retaining Feature2, Feature3, Feature4 and stitching after upsampling the middle / deep layers, 1×1 convolution dimensionality reduction);

[0100] II. Audio Preprocessing and Feature Alignment

[0101] Sampling configuration: Sampling rate 44100Hz;

[0102] Pre-emphasis: Pre-emphasis filtering is applied to the original audio to enhance high-frequency components, with a pre-emphasis factor of 0.97;

[0103] Framing and Windowing: The signal is framed according to a fixed frame length and frame shift, and each frame is multiplied by a window function (preferably Hamming window); Example of frame parameters: n_fft = 2048, hop_length = 512;

[0104] Feature extraction: Calculate the Mel spectrum for each frame (n_mels=128) and take the logarithmic scale of the Mel spectrum; at the same time, calculate MFCC (n_mfcc=40);

[0105] MFCC Dimension Expansion and Alignment: Cubic spline interpolation is used to expand the MFCC from 40 dimensions to 128 dimensions and align it with the Mel spectrum on the time axis to ensure that the two types of features have consistent shapes in the channel and time dimensions.

[0106] Channel stacking: The aligned MFCC (128-dimensional) and the logarithmic Mel (128-dimensional) are stacked in the channel dimension to form a two-channel acoustic feature tensor with an example shape of (2,128,T).

[0107] Audio enhancement during training: To improve the robustness of the model in noisy environments, enhancement operations such as random noise superposition, speed variation, and pitch shifting can be performed on the training samples.

[0108] Input backbone: The dual-channel acoustic tensor is input into a ResNet-style acoustic backbone for deep representation learning.

[0109] III. Acoustic Backbone and SE Integration

[0110] Input channels: The first layer of the acoustic backbone network is configured to accept 2 input channels;

[0111] Bottleneck and SE: The SE (Squeeze-and-Excitation) module is embedded in the residual branch of each Bottleneck unit. The reduction ratio of SE is 16. SE achieves adaptive adjustment of channel importance through global average pooling and channel recalibration, thereby highlighting the channel components containing bird call discrimination information and suppressing noise channels.

[0112] Output representation: After passing through several residual layers and downsampling, the acoustic backbone is subjected to adaptive average pooling to obtain a fixed-length acoustic high-level feature vector, which serves as the input to the confidence subnet and the single-modality classification head.

[0113] IV. Model Initialization and Training Strategies

[0114] Parameter initialization: Convolutional layers are initialized using the Kaiming normal method; the linear classification head layer is initialized using the truncated normal method (std = 0.02), with the bias set to zero.

[0115] Warm-up strategy: To ensure the stability of training in each part of the decision fusion module, an equal-weight warm-up strategy (fix_epochs is 5) is adopted in the early stage of training. During the warm-up phase, the single-modal probabilities are fused with equal weight (0.5 / 0.5) for training. After the warm-up period, the training is switched to confidence-driven adaptive weighted fusion training.

[0116] Training loss and optimization: Binary cross-entropy is used for the single-modality classification head; the confidence sub-network and the fusion output jointly participate in the optimization of the total loss, and the training hyperparameters are set.

[0117] V. Inference Output and Post-processing

[0118] Inference output: The model inference returns the following items: bird presence probability P_fused, visual monomodal v_logit, acoustic monomodal a_logit, and current modal weights W_visual and W_audio;

[0119] Judgment and Recording: The presence or absence of birds can be determined based on P_fused and a preset threshold; at the same time, the single-modal logits and weights are recorded for subsequent analysis and threshold adjustment.

[0120] This invention effectively overcomes the limitations of a single modality by fusing information from two complementary modalities: acoustic and visual. When the acoustic modality is affected by strong noise interference or the visual modality experiences performance degradation due to lighting or occlusion, the other modality can provide effective supplementation, enabling the system to maintain high recognition accuracy (experiments show a recognition rate ≥90%) even in complex orchard environments, significantly enhancing robustness. The SE attention mechanism and CBAM attention module introduced in the feature extraction stage can adaptively focus on key feature channels and spatial regions related to birds, actively filtering environmental noise and redundant information, thereby greatly improving the model's anti-interference and feature selection capabilities. The dynamic weight fusion strategy adopted can intelligently allocate weights based on the prediction confidence of each modality of the input sample, rather than simply averaging, allowing the system to flexibly adapt to various complex scenarios, improving both recognition performance and energy efficiency. In addition, the backbone networks such as MobileViT and ResNet used have excellent lightweight characteristics, facilitating deployment in embedded devices such as Jetson Nano and Raspberry Pi, laying a solid technical foundation for developing low-cost, high-reliability intelligent bird deterrence equipment for fields, and demonstrating significant engineering application value.

[0121] In this embodiment of the invention, the term "multiple" refers to two or more, unless otherwise explicitly defined. The terms "install," "connect," and "fix" should be interpreted broadly. For example, "connect" can mean a fixed connection, a detachable connection, or an integral connection. Those skilled in the art can understand the specific meaning of the above terms in this embodiment of the invention based on the specific circumstances.

[0122] In the description of the embodiments of the present invention, it should be understood that the terms "upper" and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or unit referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention.

[0123] In the description of this specification, the terms "an embodiment," "a preferred embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0124] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention. Technologies not covered in this invention can be implemented using existing technologies.

Claims

1. A bird recognition method based on sound-image multimodal fusion, characterized in that: The steps for this bird identification method are as follows: S1. Perform standardized frame-level preprocessing on the collected bird audio signals to ensure the consistency of acoustic features and temporal alignment. S2. Acoustic Feature Extraction and Enhancement: For the pre-processed bird audio signal at the frame level, the Mel spectrogram and MFCC features of each frame are extracted to obtain the logarithmic Mel spectrogram. The MFCC features are mapped using spline interpolation to obtain extended MFCC features that match the dimensions and time domain of the logarithmic Mel spectrogram. The logarithmic Mel spectrogram and extended MFCC features are stacked in the channel dimension to form a dual-channel acoustic feature tensor. This dual-channel acoustic feature tensor is input into a convolutional neural network with an embedded SE attention mechanism for feature enhancement and recalibration to obtain a high-level acoustic feature vector that highlights bird call discrimination information and suppresses environmental noise. S3. Perform standardized preprocessing on visual images acquired synchronously with bird audio signals to ensure consistency of model input; S4. Visual Feature Extraction and Multi-Scale Fusion: The preprocessed visual image is input into the MobileViT backbone network to extract shallow, mid, and deep feature maps. Bilinear interpolation is used to upsample the mid and deep feature maps to the size of the shallow feature map. Shallow, mid, and deep feature maps of the same size are concatenated along the channel dimension and fused using 1×1 convolution to obtain a multi-scale feature map. The multi-scale feature map is then input into the CBAM attention module for channel attention and spatial attention weighting to obtain a visual high-level feature vector that enhances the response to key morphological regions of birds and suppresses background interference. The dimension and temporal domain of the visual high-level feature vector match the acoustic high-level feature vector obtained in step S2. S5. Dynamic weighted fusion of decision-making layers: The acoustic high-level feature vector obtained in step S2 and the visual high-level feature vector obtained in step S4 are respectively passed through independent classifiers to obtain acoustic modality classification probability and visual modality visual probability, respectively passed through independent confidence sub-networks to obtain acoustic confidence score and visual confidence score, respectively, and the acoustic confidence score and visual confidence score are respectively concatenated and input into a Softmax function with controllable temperature parameter for normalization to obtain the dynamic weight of fused acoustic modality and visual modality; The probability of bird presence is obtained by weighting and summing the acoustic modality classification probability, the visual modality visual probability, and the dynamic weights that fuse the acoustic and visual modalities. S6. Compare the probability of bird presence obtained in step S5 with the preset threshold of the corresponding bird to determine whether there are birds and the types of birds present.

2. The bird recognition method based on sound-image multimodal fusion according to claim 1, characterized in that: The standardized frame-level preprocessing method in step S1 is as follows: the original audio is first pre-emphasized to enhance high-frequency components, and then the signal is framed and windowed according to a fixed sampling rate. The frame length and frame shift are set according to the sampling rate and target duration. The parameter configuration is as follows: sampling rate 44100Hz, frame parameters: n_fft=2048, hop_length=512.

3. The bird recognition method based on sound-image multimodal fusion according to claim 1, characterized in that: In step S2, the MFCC features are nonlinearly extended from 40 dimensions to 128 dimensions using a cubic spline interpolation function and aligned on the time axis to obtain extended MFCC features that match the dimensions of the logarithmic Mel spectrogram and are aligned in the time domain. The formula for fusing Mel spectrograms and MFCC features is: X audio =Stack[Mel db ,CubicSpline(MFCC)] (1) In formula (1): X audio For two-channel acoustic feature tensors; Mel db For the logarithmically transformed Mel spectrum; CubicSpline represents the cubic spline interpolation function.

4. The bird recognition method based on sound-image multimodal fusion according to claim 1, characterized in that: The convolutional neural network in step S2 uses a ResNet-SE Backbone network with dual-channel input. Each Bottleneck unit of the ResNet-SE Backbone network embeds an SE attention mechanism to adaptively recalibrate the channels. The output of the SE attention mechanism is: X se =X·σ(W2·ReLU(W1·GAP(X))) (2) In equation (2): σ is the Sigmoid function; GAP is global average pooling; W1 and W2 are the weights of the fully connected layer.

5. The bird recognition method based on sound-image multimodal fusion according to any one of claims 1-4, characterized in that: The visual image standardization preprocessing step in step S3 is as follows: the original image is scaled to a uniform resolution according to the network input requirements to achieve size uniformity, the pixel values ​​are mapped to the [0,1] interval for normalization, and the mean and standard deviation of the image are calculated by channel to obtain channel standardization.

6. The bird recognition method based on sound-image multimodal fusion according to claim 1, characterized in that: The fusion formula for the multi-scale feature maps in step S4 is as follows: F fused =Conv 1×1 (Concat[F low ,UpSample(F mid ),UpSample(F high )]) (3) In equation (3): F low F mid F high These represent shallow, medium, and deep feature maps, respectively. UpSample represents the bilinear upsampling operation, Concat represents the channel concatenation operation, and Conv represents the deep feature map. 1×1 This represents a 1×1 convolution fusion operation.

7. The bird recognition method based on sound-image multimodal fusion according to claim 1, characterized in that: The output of the dynamic weight generator used for normalization processing of the Softmax function in step S5 is: In equation (4): c v c is the raw score for visual confidence. a is the raw acoustic confidence score; T is the temperature hyperparameter used to control the smoothness of the weight distribution.

8. A bird recognition device based on sound-image multimodal fusion, characterized in that: The bird identification device includes: Audio signal acquisition equipment: used to acquire bird audio signals and input them into the audio signal preprocessing module; Audio signal preprocessing module: Performs standardized frame-level preprocessing on the acquired bird audio signals to ensure the consistency of acoustic features and temporal alignment, and inputs the frame-level preprocessed bird audio signals into the acoustic feature processing module; Acoustic Feature Processing Module: For the pre-processed bird audio signal at the frame level, extract the Mel spectrogram and MFCC features of each frame to obtain the logarithmic Mel spectrogram. Then, use spline interpolation to map the MFCC features and obtain extended MFCC features that match the dimensions and time domain of the logarithmic Mel spectrogram. Stack the logarithmic Mel spectrogram and extended MFCC features in the channel dimension to form a dual-channel acoustic feature tensor. Input the dual-channel acoustic feature tensor into a convolutional neural network with an embedded SE attention mechanism for feature enhancement and recalibration to obtain a high-level acoustic feature vector that highlights bird call discrimination information and suppresses environmental noise. Input the vector into the decision fusion module. Visual image acquisition device: synchronously acquires visual images of the same environment as the audio signal acquisition device and inputs them into the visual image preprocessing module; Visual image preprocessing module: Performs standardized preprocessing on visual images acquired synchronously with bird audio signals to ensure consistency of model input, and inputs the preprocessed visual images into the visual feature processing module; The visual feature processing module receives the preprocessed visual image and extracts shallow, mid, and deep feature maps. It uses bilinear interpolation to upsample the mid and deep feature maps to the size of the shallow feature map. Shallow, mid, and deep feature maps of the same size are concatenated along the channel dimension and fused using a 1×1 convolution to obtain a multi-scale feature map. This multi-scale feature map is then input into the CBAM attention module for sequential channel and spatial attention weighting to obtain a visual high-level feature vector that enhances the response to key morphological regions of birds and suppresses background interference. Finally, the visual high-level feature vector, whose dimension and temporal domain match the acoustic high-level feature vector from step S2, is input into the decision fusion module. The decision fusion module receives acoustic high-level feature vectors and visual high-level feature vectors. It obtains acoustic modality classification probabilities and visual modality visual probabilities through independent classifiers, and acoustic confidence scores and visual confidence scores through independent confidence sub-networks. The acoustic and visual confidence scores are concatenated and then input into a temperature-controlled Softmax function for normalization. The dynamic weights of the fused acoustic and visual modalities are obtained. The acoustic modality classification probabilities, visual modality visual probabilities, and dynamic weights of the fused acoustic and visual modalities are weighted and summed to obtain the probability of bird presence, which is then input into the decision module. Judgment module: Receives the probability of bird presence and compares it with the corresponding preset threshold for birds to determine whether birds exist and the types of birds present.

9. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the processors implement the method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images