An automatic field bird species identification system based on multi-modal feature matching

Abstract

The application relates to the field of image recognition and artificial intelligence, and particularly discloses an automatic wild bird species identification system based on multi-modal feature matching. The system comprises a multi-modal data acquisition module, a posture adaptive feature decoupling module, a local-global feature collaborative enhancement module, a multi-modal feature fusion and matching decision module, and a species identification output module. By decoupling posture correlation and species essential features, strengthening key local features by using an attention mechanism, fusing aligned visual and auditory modal information, and combining a hierarchical matching strategy, high-robustness and high-precision automatic identification of wild bird species is realized.

Description

Technical Field

[0001] This invention belongs to the field of image recognition and artificial intelligence technology, specifically relating to an automatic identification system for wild bird species based on multimodal feature matching. Background Technology

[0002] In the fields of biodiversity monitoring and ecological conservation, accurate and efficient automatic identification of wild bird species has significant scientific and applied value. Traditional identification methods mainly rely on expert on-site observation or post-event analysis of image data, which is inefficient and difficult to implement for large-scale, real-time species monitoring. With the development of computer vision and pattern recognition technologies, deep learning-based image recognition methods have become the mainstream technology in the field.

[0003] Automatic bird species identification technology based on visual features aims to achieve automated species classification by analyzing visual information such as morphology, color, and texture in bird images. This technology typically relies on models such as convolutional neural networks to extract discriminative features from the overall image and match them with feature templates of known species.

[0004] Existing technologies face several challenges when processing images of wild birds with varied postures, complex backgrounds, and frequent partial occlusion. Firstly, birds move freely in their natural environment, exhibiting diverse postures and angles, often partially obscured by vegetation, terrain, or other objects. This leads to unstable and insufficiently discriminative global features extracted from single images. Secondly, existing methods tend to focus on recognizing complete, posture-standard images; their performance drops sharply when key local features are lost due to occlusion. Thirdly, different species may exhibit high visual similarity, making accurate differentiation difficult and prone to misclassification based solely on a single visual modality. Therefore, robustly extracting and fusing multi-level discriminative features from wild bird images under suboptimal conditions to address the challenges posed by posture variations and partial occlusion has become a crucial issue for improving the practicality and accuracy of automatic wild bird identification systems. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic identification system for wild bird species based on multimodal feature matching, so as to solve the problem that the existing technology is not robust and accurate enough when facing wild birds with varied postures, complex backgrounds and partial occlusions, due to its reliance on single and unstable global visual features.

[0006] This invention provides an automatic species identification system for wild birds based on multimodal feature matching, comprising: A multimodal data acquisition module is used to simultaneously acquire visual and auditory data streams of target birds. This multimodal data acquisition module includes a high-resolution optical imaging unit and a directional audio acquisition unit. An attitude-adaptive feature decoupling module is connected to the output of the optical imaging unit of the multimodal data acquisition module and is used to perform feature decomposition on the input keyframe image. The attitude-adaptive feature decoupling module includes a pre-trained bird attitude estimation network and a feature decoupling backbone network. A local-global feature co-enhancement module is connected to the output of the pose-adaptive feature decoupling module. It is used to handle the feature loss problem caused by occlusion and enhance discriminative local features. The local-global feature co-enhancement module includes a spatial attention mechanism and a local feature mining sub-network. The multimodal feature fusion and matching decision module is connected to both the output of the local-global feature collaborative enhancement module and the output of the directional audio acquisition unit of the multimodal data acquisition module, and is used to align and fuse features from visual and auditory modalities; the multimodal feature fusion and matching decision module includes a multimodal feature alignment unit and a hierarchical feature fusion network; The species identification output module is connected to the multimodal feature fusion and matching decision module and is used to perform the final species matching and result output.

[0007] Preferably, the optical imaging unit captures a video sequence containing the target bird at a rate of 30 frames per second and extracts keyframe images; The directional audio acquisition unit uses a microphone array with beamforming function to acquire and preprocess the call signals of the target birds while capturing images, and generate corresponding time-spectrum diagrams.

[0008] Preferably, the bird pose estimation network performs key point detection on the bird subject in the input image. The key points include at least 12 key points, including the beak tip, eyes, wing joints, feet and tail feather roots. Based on the two-dimensional coordinates of these key points, a pose encoding vector representing the bird's overall orientation and body extension is calculated. The feature decoupling backbone network receives the original image and the pose encoding vector, and by introducing a pose conditional normalization layer, the pose encoding vector is used as a conditional input to guide the network to separate the feature components that are strongly correlated with the pose and the species essential feature components that are unrelated to the pose during the feature extraction process. The operation process of the pose conditional normalization layer is to dynamically generate a set of scaling and translation parameters using the pose encoding vector, and then perform affine transformation on the channels of the convolutional feature map.

[0009] Preferably, the spatial attention mechanism generates a spatial weighted heatmap based on a pose-independent species essential feature map. The high-weight regions of the spatial weighted heatmap correspond to the most discriminative anatomical structures of birds in the image, including the head, wing patch region, and tail feather region. The local feature mining sub-network receives the original image, spatial weight heatmap, and pose-independent features. It first performs soft cropping on the original image based on the spatial weight heatmap to extract image patches of multiple high-weight regions. Then, it performs deep feature extraction on these image patches through a lightweight convolutional sub-network to generate a set of high-dimensional local discriminative feature vectors. The local-global feature collaborative enhancement module has feature compensation logic. When the spatial attention mechanism detects that the feature response intensity of a certain expected high-weight region is less than a preset threshold, it determines that the high-weight region is occluded and starts the cross-frame feature propagation mechanism to extract the local features of the corresponding region from the adjacent unoccluded frames of the same video sequence to compensate for the missing information of the current frame.

[0010] Preferably, the feature alignment unit receives the pose-independent global feature vector after collaborative enhancement, the set of local discriminative feature vectors, and the acoustic feature vector extracted from the time-spectrum map of the chirping sound. It then calculates the correlation weight between the acoustic feature vector and the visual global feature vector through a cross-modal attention layer, and reconstructs the acoustic features based on this weight to align them with the current visual context in the feature semantic space. The hierarchical feature fusion network is responsible for integrating aligned acoustic features, global visual features, and local visual features. This hierarchical feature fusion network adopts a gating mechanism to learn dynamic weights for each feature modality in order to control the contribution of each modality feature in the final fused feature and output a unified multimodal joint feature representation.

[0011] Preferably, the species identification output module includes a feature matching and classifier based on metric learning, and maintains a species feature template library, which stores multiple multimodal joint feature templates generated by the system for each known bird species; During the identification phase, the species identification output module calculates the cosine similarity between the multimodal joint features of the target to be identified and the template features of all species in the database, and adopts a hierarchical matching strategy: first, coarse-grained matching is performed to select the top 5 candidate species with similarity greater than the first threshold. Then, fine-grained matching is performed, comparing the similarity between the local discriminative feature vector of the target to be identified and the local feature template of the corresponding anatomical part in the candidate species template; The species identification output module integrates coarse-grained similarity and fine-grained local similarity, and outputs the final species identification result and its confidence level through a weighted voting mechanism.

[0012] Preferably, the feature decoupling backbone network adopts an encoder-decoder structure, wherein the encoder part is a deep residual network, and the decoder part includes multiple pose conditional normalization layers and upsampling convolutional layers. The training objective is to ensure that the pose encoding vector can effectively guide feature decoupling by being driven by both reconstruction loss and species classification loss.

[0013] Preferably, the spatial attention mechanism is specifically implemented as a fully convolutional network. This fully convolutional network takes a pose-independent feature map as input, passes through 3 layers of convolution and a sigmoid activation function, and outputs a single-channel weight heatmap with the same spatial size as the input feature map. The value range of each pixel in the heatmap is 0 to 1, which represents the feature discrimination importance of the spatial location of the feature map.

[0014] Preferably, the lightweight convolutional subnetwork in the local feature mining subnetwork is constructed using channel-separated convolution and depth-separable convolution to reduce computational overhead, and its output generates a 256-dimensional local feature vector for each high-weight image patch.

[0015] Preferably, the specific process of the cross-frame feature propagation mechanism is as follows: when the module determines the first frame feature propagation... When a local region of a frame is occluded, the system backtracks the five most recent frames in the video sequence, uses optical flow to track the location of the local region in adjacent frames, and selects the frame with the clearest feature response that was not identified as occluded. It then extracts the local feature vector of the corresponding region and uses it to replace or weight the occluded region in the next frame. In the corresponding local features of the frame.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention introduces a pose-adaptive feature decoupling module to explicitly decompose the features of bird images into pose-related components and pose-independent species-specific components. This design fundamentally weakens the negative impact of the ever-changing bird poses on feature stability, making the subsequent recognition process mainly rely on species-specific features robust to pose changes. This improves the system's adaptability to different shooting angles and body poses, overcoming the problem of recognition failure due to feature mismatch caused by pose changes in existing technologies.

[0017] 2. The local-global feature collaborative enhancement module designed in this invention automatically locates and strengthens the most discriminative local region features through a learnable spatial attention mechanism, and is also equipped with a cross-frame feature compensation mechanism for occlusion. This scheme enables the system to selectively rely on the features of unoccluded key parts for recognition when faced with partial occlusion, and can recover occluded information from the time series, thereby greatly enhancing the robustness of the system in complex field environments and solving the problem of a sharp drop in recognition performance due to the loss of key local features.

[0018] 3. This invention constructs a multimodal feature fusion and matching decision module, innovatively aligning and deeply fusing auditory call features with visual features. The complementarity of visual and auditory features effectively distinguishes visually highly similar species, such as many songbird species. This multimodal fusion strategy provides discrimination capabilities beyond a single visual modality. Through hierarchical fusion and matching, it achieves higher accuracy in species differentiation and reduces the false positive rate.

[0019] 4. This invention employs a hierarchical matching strategy and a dynamically updated species feature template library. Coarse-grained matching quickly filters candidate species, while fine-grained matching focuses on the precise comparison of key local features, improving recognition efficiency and accuracy. The dynamic updating mechanism of the template library enables the system to continuously learn and adapt to morphological differences within the same species and feature performance under different environments, ensuring the system's long-term recognition performance and generalization ability. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention; Figure 2 This is a schematic diagram of the core principle framework of the attitude adaptive feature decoupling module in this invention; Figure 3 This is a logical flowchart of the local-global feature collaborative enhancement module in this invention; Figure 4 This is a schematic diagram of the multi-level interaction relationship and data flow of the multimodal feature fusion and matching decision module in this invention; Figure 5 This is a schematic diagram illustrating the interaction between the hierarchical matching strategy and the template library of the species identification output module in this invention. Detailed Implementation

[0021] Example 1: The overall technical architecture of the automatic identification system for wild bird species based on multimodal feature matching proposed in this invention is shown in the attached figure. Figure 1 As shown in the figures, this system achieves high-precision and robust automatic identification of bird species in complex field environments through the orderly collaboration of five functional units: a multimodal data acquisition module, a posture adaptive feature decoupling module, a local-global feature collaborative enhancement module, a multimodal feature fusion and matching decision module, and a species identification output module. The following will describe in detail the specific implementation methods of each component of the system, with reference to the accompanying figures.

[0022] The multimodal data acquisition module, serving as the front-end input unit of the entire system, is responsible for simultaneously acquiring the visual and auditory information of the target birds. This module integrates a high-resolution optical imaging unit and a directional audio acquisition unit. The optical imaging unit uses a visible light camera with optical image stabilization and autofocus to continuously capture video sequences containing the target birds at a rate of 30 frames per second. During video stream processing, the system uses motion detection algorithms to determine in real time whether there are significant moving targets in the frame. After detecting a valid target, keyframe images are extracted from consecutive frames for subsequent analysis. The keyframe selection strategy is based on a comprehensive evaluation of inter-frame differences and sharpness, ensuring that the selected images have a high signal-to-noise ratio and target integrity.

[0023] Simultaneously, the directional audio acquisition unit starts operating. This unit consists of a ring microphone array of eight omnidirectional microphones, possessing beamforming capabilities. While the optical imaging unit captures images, the audio acquisition unit simultaneously records the call signals emitted by the target birds. The raw audio signal undergoes preprocessing through a bandpass filter to remove environmental noise below 200 Hz and above 12 kHz, retaining the main frequency bands of the bird calls. Subsequently, the system performs a short-time Fourier transform on the preprocessed audio signal to generate a corresponding time-frequency spectrum with a time resolution of 10 milliseconds and a frequency resolution of 50 Hz, ultimately forming a two-dimensional acoustic time-frequency representation as input data for auditory modality. The optical imaging unit and the directional audio acquisition unit ensure strict alignment of their data on the time axis through a hardware-level timestamp synchronization mechanism, with the error controlled within 5 milliseconds, providing a precise time reference for subsequent multimodal feature fusion.

[0024] The attitude-adaptive feature decoupling module is connected to the output of the optical imaging unit of the multimodal data acquisition module. This module performs feature decomposition on the input keyframe image, separating the attitude-affected feature components from the stable feature components reflecting the species' essence. Please refer to the appendix. Figure 2 The workflow of this pose adaptive feature decoupling module is divided into two stages.

[0025] The first stage is bird pose estimation. A pre-trained bird pose estimation network is embedded within the module. This network employs an hourglass network structure based on residual connections and has been trained on over 100,000 images of wild birds annotated with 12 keypoints, covering more than 500 common bird species. These 12 keypoints include the beak tip, left eye, right eye, left shoulder joint, right shoulder joint, left elbow joint, right elbow joint, left hip joint, right hip joint, left foot, right foot, and tail feather base. After receiving the input image, the network outputs the two-dimensional coordinates (x, y) of each keypoint in the image coordinate system. Based on these coordinates, the system calculates a 12-dimensional pose encoding vector P, where each dimension corresponds to the normalized offset of the keypoint relative to the image center. Three geometric parameters are also calculated: wing spread angle, body pitch angle, and tail feather deflection angle, ultimately forming a 15-dimensional pose encoding vector.

[0026] The second stage is feature decoupling. This stage is executed by the feature decoupling backbone network, which adopts an encoder-decoder structure. The encoder part is a deep residual network ResNet-50, responsible for extracting multi-level convolutional features from the original image. The decoder part contains multiple upsampling convolutional layers and pose conditional normalization layers. The pose conditional normalization layer is the core mechanism for achieving feature decoupling, and the operation process is as follows: For the input convolutional feature map... , For feature map height, The width of the feature map. To determine the number of channels in the feature map, we first perform global average pooling along the spatial dimension to obtain the channel statistics. and Subsequently, the pose encoding vector is fed into a two-layer fully connected network to dynamically generate scaling parameters. With translation parameters Ultimately, the feature map is transformed into: ; This represents element-wise multiplication along the channel dimension. Through this operation, the network is guided during feature reconstruction to suppress pose-dependent feature responses while strengthening pose-independent species-discriminating features. The training objective of the entire feature-decoupled backbone network is driven by two loss functions: a pixel-level image reconstruction loss, using the L1 norm to measure the difference between the original and reconstructed images; and a species classification loss, using the cross-entropy loss function to supervise the network's output feature vectors to correctly classify them to their corresponding species. Through joint optimization, the network learns a representation space that effectively separates pose variables from essential species features, ultimately outputting two sets of features: one set of pose-related feature maps and the other set of pose-independent essential species feature maps, the latter being passed to the next module.

[0027] The local-global feature co-enhancement module is connected to the output of the pose-adaptive feature decoupling module. It is specifically designed to address common local occlusion issues in field environments and further enhance the discriminative features of key anatomical sites. Please refer to the appendix. Figure 3 The local-global feature co-enhancement module consists of two parts: a learnable spatial attention mechanism and a local feature mining sub-network. The spatial attention mechanism takes pose-independent species-specific feature maps as input and processes them through a three-layer fully convolutional network. Each layer of this three-layer network contains 32 3×3 convolutional kernels, followed by batch normalization and ReLU activation. The final layer uses a 1×1 convolutional kernel to compress the number of channels to 1 and outputs a single-channel spatial weight heatmap through a sigmoid activation function. In the heatmap, each pixel value represents the discriminative importance of its corresponding spatial location in species identification. Based on extensive prior knowledge of avian morphology, the system presupposes the head region, specific wing patch regions (such as the base of primary flight feathers), and tail feather regions as highly discriminative areas. Therefore, the spatial attention mechanism is guided to prioritize these regions during training.

[0028] The local feature mining subnetwork receives three inputs: the original keyframe image, the spatial weight heatmap, and the pose-independent feature map. The sub-network performs soft cropping on the original image based on the spatial weight heatmap. Specifically, the system sets a threshold. All elements in the heatmap that are greater than Connected regions are considered high-weight regions. For each connected region, its centroid coordinates and bounding box size are calculated, and a fixed-size 64×64 pixel image patch is cropped from the original image using these coordinates as the center. Typically, the system extracts 3 to 5 such image patches. Subsequently, a lightweight convolutional subnetwork extracts depth features from each image patch. This lightweight convolutional subnetwork uses a depthwise separable convolutional structure, containing four convolutional layers with channels of 32, 64, 128, and 256 respectively. Finally, a 256-dimensional local discriminative feature vector is generated through global average pooling. All local feature vectors are concatenated into a set. , This represents the number of image patches extracted.

[0029] This local-global feature co-enhancement module also integrates feature compensation logic to address occlusion. The system monitors the average response intensity of each high-weight region in the spatial weight heatmap in real time. If the average intensity of a region is less than a preset threshold of 0.3, the region is determined to be occluded by leaves, branches, or other objects. At this time, the module initiates a cross-frame feature propagation mechanism. The system backtracks to the five most recent frames in the current video sequence and uses a pre-calculated optical flow field to track the spatial location of the occluded region in adjacent frames. The optical flow field is calculated in real time using the RAFT algorithm, achieving sub-pixel accuracy.

[0030] For each frame, the system checks whether the corresponding local region is also determined to be occluded. If at least one unoccluded image with clear local feature responses is found, the corresponding local feature vector is extracted from it. Finally, the local feature vector of the occluded frame is replaced with the feature vector of the selected frame, or a weighted average strategy is used, with the weights determined by the inter-frame temporal distance and feature sharpness. For example, frames closer to the current frame and with stronger feature responses receive higher weights. This mechanism effectively alleviates the feature loss problem caused by instantaneous occlusion, ensuring the integrity and reliability of local features.

[0031] The multimodal feature fusion and matching decision module is connected to both the output of the local-global feature co-enhancement module and the output of the directional audio acquisition unit of the multimodal data acquisition module. It is responsible for integrating visual and auditory modal information and generating a unified multimodal joint feature representation. Please refer to the appendix. Figure 4 The multimodal feature fusion and matching decision module's processing flow is divided into two stages: feature alignment and hierarchical fusion. In the feature alignment stage, the system first extracts features from the temporal spectrogram of the auditory modality. The temporal spectrogram is fed into a four-layer convolutional neural network, each layer containing 64 3×3 convolutional kernels, followed by batch normalization and ReLU activation. Finally, a 512-dimensional acoustic feature vector is generated through global average pooling. In terms of visual modality, the system possesses pose-independent global feature vectors. (Obtained by global pooling of the final output of the feature decoupling backbone network) and the set of local discriminative feature vectors. The feature alignment unit includes a cross-modal attention layer that employs a scaled dot product attention mechanism. Specifically, the visual global feature vector... Linearly projected into the query vector Acoustic feature vector They are projected as key vectors respectively AND value vector , , , The projection matrix is ​​learnable. Attention weights. Calculated using the following formula: ; The feature dimension is 512 in this case. The aligned acoustic context vector. This process aligns acoustic features semantically with the current visual scene. For example, when a songbird is visually identified, its call features are amplified while background noise is suppressed.

[0032] During the hierarchical integration phase, the system will align the... Visual global features and the aggregated visual local features The input is fed into a gated fusion network. This gated fusion network consists of a multilayer perceptron, containing two hidden layers, each with 256 neurons, and the activation function is ReLU. The network uses... The concatenated vector is taken as input, and the output is the original three-dimensional weight vector. The original weight vector is normalized using the Softmax function to obtain the final fused weights. ; Final multimodal joint feature representation Obtained by weighted summation ; This fusion strategy dynamically adjusts the contribution of each feature based on the modal reliability of the current scene. For example, in well-lit but noisy environments, visual weights are automatically increased; while in scenes with severe visual occlusion but clear vocalizations, auditory weights dominate. This adaptive fusion mechanism greatly improves the system's stability in varying outdoor environments.

[0033] The species identification output module connects to the multimodal feature fusion and matching decision module, and is responsible for performing the final species matching and result output. Please refer to the appendix. Figure 5 The species identification output module includes a feature matching and classifier based on metric learning, and maintains a dynamically updated species feature template library. The template library stores multiple multimodal joint feature templates for each known bird species. Each template is generated by the system under standard conditions (such as no occlusion and clear calls). Vectors. The initial construction of the template library is based on samples from authoritative bird atlas databases, with at least 20 templates for each species. During the recognition phase, the system first calculates the vectors of the target to be identified. Cosine similarity between the vector and templates of all species in the database. Cosine similarity is defined as the cosine of the angle between two vectors, and its value ranges from [-1, 1]. The larger the value, the more similar the features are.

[0034] The identification process employs a hierarchical matching strategy. The first stage is coarse-grained matching: the system selects the top 5 candidate species with a cosine similarity greater than the first threshold of 0.75, forming a candidate set. The second stage is fine-grained matching: for the candidate set... Each species The system compares the local discriminative feature vector of the target to be identified with... The similarity between corresponding local features of all templates in the template library. Finally, the fine-grained matching score. The weighted average yields: ; , , These represent the highest matching scores for the head, wing patches, and tail feathers, respectively. The final recognition result is determined by coarse-grained similarity. With fine-grained fraction The decision is made jointly. The system uses a weighted voting mechanism, with the total score... The candidate species with the highest score was selected as the final identification result, and its corresponding... The value represents the confidence level of identification.

[0035] Furthermore, the species feature template library has dynamic update capabilities. For each successful identification with a confidence level greater than 0.9, the system will update the template used for that identification. New templates are stored as new templates in the template set for the corresponding species. When the number of templates for a species exceeds a preset limit of 50, the system initiates a template simplification process. This process uses a farthest neighbor clustering algorithm: first, templates are randomly selected as initial cluster centers; then, the template with the greatest average distance from the existing centers is iteratively selected as the new center, until 20 representative centers are chosen. The remaining templates are discarded, thus maintaining template diversity while avoiding redundancy. This mechanism enables the system to continuously learn the characteristic changes of the same species under different seasons, ages, sexes, and environments, continuously improving long-term recognition performance.

[0036] In summary, this embodiment constructs an automatic bird species identification system capable of effectively addressing complex challenges in the wild through the close collaboration of the five modules described above. The system not only achieves pose decoupling and local enhancement at the feature level, but also integrates multimodal information and employs a hierarchical matching strategy at the decision level, thereby improving accuracy, robustness, and adaptability.

[0037] Example 2: Building upon Example 1, this example improves the cross-frame feature compensation mechanism in the local-global feature collaborative enhancement module to adapt to more dynamic outdoor scenarios. Specifically, the original example only performed feature compensation by looking back at the most recent 5 frames. However, in scenarios involving high-speed flight or rapid turns, the appearance changes drastically between adjacent frames, potentially leading to feature inconsistencies. Therefore, this example introduces a cross-frame compensation strategy based on trajectory prediction.

[0038] In the early stages of video sequence processing, the system establishes independent tracking trajectories for detected bird targets. These trajectories are maintained by a Kalman filter, and the state vector includes the target's center coordinates, velocity, and scale. When the local-global feature co-enhancement module determines that a local region in frame t is occluded, it is no longer limited to backtracking a fixed number of frames. Instead, it predicts the reliable location of this local region in historical frames based on the target's motion trajectory. Specifically, the system calculates the inverse optical flow from frame t-1 to frame tk (k is at most 10) and, combined with the state prediction from the Kalman filter, performs a spatiotemporal consistency check on the location of the occluded region. Only when the location prediction error of this local region in a historical frame is less than a preset threshold (e.g., 5 pixels) and the local feature response intensity is greater than 0.5, is it included in the candidate compensation source.

[0039] Furthermore, the selection of compensation features no longer relies solely on a single frame, but instead employs a weighted fusion based on motion stability. The system calculates the Structural Similarity Index (SSIM) for the local region between each candidate frame and the current frame. A higher SSIM value indicates better appearance consistency, and its weight is greater. Ultimately, the local feature vector of the occluded region is a weighted average of the features from all valid candidate frames, with the weights determined by the SSIM value and a time decay factor (which decays exponentially with frame intervals). This improvement enhances the feature compensation quality in high-speed motion scenes and avoids noisy features introduced by drastic changes between frames.

[0040] Meanwhile, this embodiment optimizes the gating mechanism in the multimodal feature fusion and matching decision module. In the original embodiment, the gating network only took the feature concatenation vector as input, ignoring the uncertainty between modalities. This embodiment introduces uncertainty-aware gating, that is, adding the confidence estimates of each modality feature to the input. The confidence of the visual global features is determined by the mean confidence of the keypoint detection of the pose estimation network; the confidence of the local features is determined by the maximum response value of the spatial attention heatmap; and the confidence of the acoustic features is determined by the audio signal-to-noise ratio estimate. The gating network takes the extended vector containing confidence information as input and learns more reliable fusion weights. Experiments show that this optimization can improve the recognition accuracy in low-quality input scenarios.

[0041] Through the above improvements, this embodiment further enhances the applicability and stability of the system in extreme dynamic field environments, and is especially suitable for the identification of high-speed flying birds such as raptors and swifts.

Claims

1. An automatic identification system for wild bird species based on multimodal feature matching, characterized in that, include: A multimodal data acquisition module is used to simultaneously acquire visual and auditory data streams of target birds. The multimodal data acquisition module includes a high-resolution optical imaging unit and a directional audio acquisition unit. An attitude-adaptive feature decoupling module is connected to the output of the optical imaging unit of the multimodal data acquisition module and is used to perform feature decomposition on the input keyframe image. The attitude-adaptive feature decoupling module includes a pre-trained bird attitude estimation network and a feature decoupling backbone network. A local-global feature co-enhancement module is connected to the output of the pose-adaptive feature decoupling module. It is used to handle the feature loss problem caused by occlusion and enhance discriminative local features. The local-global feature co-enhancement module includes a spatial attention mechanism and a local feature mining sub-network. The multimodal feature fusion and matching decision module is connected to both the output of the local-global feature collaborative enhancement module and the output of the directional audio acquisition unit of the multimodal data acquisition module, and is used to align and fuse features from visual and auditory modalities; the multimodal feature fusion and matching decision module includes a multimodal feature alignment unit and a hierarchical feature fusion network; The species identification output module is connected to the multimodal feature fusion and matching decision module and is used to perform the final species matching and result output.

2. The automatic identification system for wild bird species based on multimodal feature matching according to claim 1, characterized in that, The optical imaging unit captures video sequences containing the target birds at a rate of 30 frames per second and extracts keyframe images. The directional audio acquisition unit uses a microphone array with beamforming function to acquire and preprocess the call signals of the target birds while capturing images, and generate corresponding time-spectrum diagrams.

3. The automatic identification system for wild bird species based on multimodal feature matching according to claim 2, characterized in that, The bird pose estimation network performs key point detection on the main bird in the input image. The key points include at least 12 key points, including the beak tip, eyes, wing joints, feet and tail feather roots. Based on the two-dimensional coordinates of these key points, a pose encoding vector representing the bird's overall orientation and body extension is calculated. The feature decoupling backbone network receives the original image and the pose encoding vector, and by introducing a pose conditional normalization layer, the pose encoding vector is used as a conditional input to guide the network to separate the feature components that are strongly correlated with the pose and the species essential feature components that are unrelated to the pose during the feature extraction process. The operation process of the pose conditional normalization layer is to dynamically generate a set of scaling and translation parameters using the pose encoding vector, and then perform affine transformation on the channels of the convolutional feature map.

4. The automatic identification system for wild bird species based on multimodal feature matching according to claim 3, characterized in that, The spatial attention mechanism generates a spatial weighted heatmap based on a pose-independent species essential feature map. The high-weight regions of this spatial weighted heatmap correspond to the most discriminative anatomical structures of birds in the image, including the head, wing patch region, and tail feather region. The local feature mining sub-network receives the original image, spatial weight heatmap, and pose-independent features. It first performs soft cropping on the original image based on the spatial weight heatmap to extract image patches of multiple high-weight regions. Then, it performs deep feature extraction on these image patches through a lightweight convolutional sub-network to generate a set of high-dimensional local discriminative feature vectors. The local-global feature collaborative enhancement module has feature compensation logic. When the spatial attention mechanism detects that the feature response intensity of a certain expected high-weight region is less than a preset threshold, it determines that the high-weight region is occluded and starts the cross-frame feature propagation mechanism to extract the local features of the corresponding region from the adjacent unoccluded frames of the same video sequence to compensate for the missing information of the current frame.

5. The automatic identification system for wild bird species based on multimodal feature matching according to claim 4, characterized in that, The feature alignment unit receives the pose-independent global feature vector after collaborative enhancement, the set of local discriminative feature vectors, and the acoustic feature vector extracted from the time-spectrum map of the call. It then calculates the correlation weight between the acoustic feature vector and the visual global feature vector through a cross-modal attention layer, and reconstructs the acoustic features based on this weight to align them with the current visual context in the feature semantic space. The hierarchical feature fusion network is responsible for integrating aligned acoustic features, global visual features, and local visual features. This hierarchical feature fusion network adopts a gating mechanism to learn dynamic weights for each feature modality in order to control the contribution of each modality feature in the final fused feature and output a unified multimodal joint feature representation.

6. The automatic identification system for wild bird species based on multimodal feature matching according to claim 5, characterized in that, The species identification output module includes a feature matching and classifier based on metric learning, and maintains a species feature template library, which stores multiple multimodal joint feature templates generated by this system for each known bird species; During the identification phase, the species identification output module calculates the cosine similarity between the multimodal joint features of the target to be identified and the template features of all species in the database, and adopts a hierarchical matching strategy: first, coarse-grained matching is performed to select the top 5 candidate species with similarity greater than the first threshold. Then, fine-grained matching is performed, comparing the similarity between the local discriminative feature vector of the target to be identified and the local feature template of the corresponding anatomical part in the candidate species template; The species identification output module integrates coarse-grained similarity and fine-grained local similarity, and outputs the final species identification result and its confidence level through a weighted voting mechanism.

7. The automatic identification system for wild bird species based on multimodal feature matching according to claim 6, characterized in that, The feature decoupling backbone network adopts an encoder-decoder structure. The encoder part is a deep residual network, and the decoder part includes multiple pose conditional normalization layers and upsampling convolutional layers. The training objective is to ensure that the pose encoding vector can effectively guide feature decoupling by being driven by both reconstruction loss and species classification loss.

8. The automatic identification system for wild bird species based on multimodal feature matching according to claim 7, characterized in that, The spatial attention mechanism is specifically implemented as a fully convolutional network. This fully convolutional network takes a pose-independent feature map as input, passes through 3 layers of convolution and a sigmoid activation function, and outputs a single-channel weight heatmap with the same spatial size as the input feature map. The value range of each pixel in the heatmap is 0 to 1, which represents the feature discrimination importance of the spatial location of the feature map.

9. The automatic identification system for wild bird species based on multimodal feature matching according to claim 8, characterized in that, The lightweight convolutional subnetwork in the local feature mining subnetwork is constructed using channel-separable convolution and depth-separable convolution to reduce computational overhead. Its output generates a 256-dimensional local feature vector for each high-weight image patch.

10. The automatic identification system for wild bird species based on multimodal feature matching according to claim 9, characterized in that, The specific process of the cross-frame feature propagation mechanism is as follows: When the module determines that a local area of ​​a certain frame is occluded, the system backtracks the 5 most recent frames in the video sequence, uses optical flow to track the position of the local area in the adjacent frames, selects the frame with the clearest feature response that has not been determined to be occluded, extracts the local feature vector of its corresponding area, and uses it to replace or weight average it into the local features corresponding to the frame number.

Images