Wildlife target detection method based on multi-modal feature fusion

Abstract

A kind of few-shot wild animal target detection method based on multi-modal feature fusion, steps include: collecting wild animal pictures, construct wild animal few-shot new class data set, while selecting animal species pictures in public data set as base class data set;Construct few-shot wild animal target detection model;Using base class data set to pre-training model, then fine-tune parameters by few-shot new class data set;Multi-level feature enhancement module HMFE realizes the enhancement and fusion of multi-level image features, improves the detection performance of small size animal target;Multi-branch fusion module MDM decouples the image features extracted by CLIP into semantic and visual related components, realizes the semantic alignment of image-text features, which helps to distinguish animals with similar appearance and animals with camouflage color;Multi-branch fusion module MBF dynamically fuses high-level semantic information and low-level visual features, avoids the class confusion between camouflage animal targets and background, thereby reduces the miss detection rate of camouflage animals.

Description

Technical Field

[0001] This invention relates to a target detection method, and more particularly to a method for detecting wild animal targets in a few-sample environment based on multimodal feature enhancement and fusion. Background Technology

[0002] Wildlife species identification is a prerequisite and key to wildlife research, conservation, and management. Target detection can be used to study population changes, distribution range changes, and migration behavior, thereby assessing the health of ecosystems. For example, it allows for early assessment of the habitats of endangered species by locating and identifying them, providing scientific data support for developing conservation strategies and delineating protected areas. With the increasing demands for ecological and environmental protection, computer vision-based wildlife monitoring technology has become a research hotspot. However, the complex environment in the wild and the scarcity of rare species samples are key factors limiting its monitoring accuracy and application value.

[0003] Existing wildlife monitoring technologies are mainly divided into two categories: traditional electronic device tracking and computer vision detection.

[0004] One type of monitoring is achieved through devices such as positioning collars and sensors, as described in publication CN 116724914 A, "A Low-Power Multifunctional Wildlife Real-Time Positioning Collar and Positioning Monitoring Method." This type of method can cause physiological disturbances and behavioral impacts on animals, and is limited by device battery life and deployment costs, making it difficult to cover a wide range of highly concealed species.

[0005] Another type relies on image data such as visible light and infrared thermal imaging, combined with deep learning algorithms to achieve target detection and tracking. While this type of method has the advantage of being non-invasive, most existing technologies depend on datasets with large numbers of labeled samples for model training. For example, the "A Real-time Wildlife Recognition and Tracking Algorithm Based on Attention Mechanism" published in CN 116798072 A acquires visible light and infrared images using drones and combines them with an attention mechanism to achieve target tracking. However, this method relies on a large number of labeled samples to build a feature database, and the model's generalization ability drops sharply in scenarios with few samples of rare species. The "Small Sample Wildlife Detection Method Based on Improved YOLOv5" published in CN 115393618 A optimizes the YOLOv5 network for scenarios with few samples, but its detection accuracy is insufficient in real and complex wild environments.

[0006] In actual field conservation scenarios, the scarcity of endangered species and their elusive activities result in a very limited number of effectively labeled image samples, making it a typical low-sample detection scenario. Furthermore, complex field scenarios can easily lead to missed or false detections by existing models. Moreover, most model networks are complex and computationally intensive, making them difficult to deploy on resource-constrained field monitoring hardware, thus limiting their practical application in real-time field monitoring.

[0007] In summary, existing technologies have shortcomings in adapting to scenarios with few samples and in robustness to complex environments in the wild. How to improve the accuracy of wildlife detection in complex scenarios when there are insufficient samples is a technical problem that needs to be solved in the field of wildlife detection. Summary of the Invention

[0008] To address the aforementioned issues, this invention utilizes a multimodal feature enhancement and fusion strategy to tackle problems such as low detection accuracy and high false negative rates caused by a lack of labeled wild animal samples, highly camouflaged targets in complex wild environments, high similarity between species, and large variations in target scale, thereby improving target detection accuracy.

[0009] A method for detecting wild animal targets in a few samples based on multimodal feature enhancement and fusion, comprising the following steps:

[0010] Collect images of wild animals in the wild, construct a new few-shot wild animal class dataset WildFS, and generate corresponding text prompts for each species category;

[0011] Meanwhile, animal species images from the public datasets MSCOCO and PASCAL VOC were selected as the base class dataset TrainD;

[0012] A few-shot wildlife target detection model was constructed, comprising a hierarchical multi-level feature enhancement module (HMFE), a modality decoupling module (MDM), and a multi-branch fusion module (MBF).

[0013] The model was pre-trained using the base class dataset TrainD, and then some parameters of the pre-trained model were fine-tuned using the WildFS dataset. Finally, the results of the detection of wild animals in a small number of samples were output.

[0014] in:

[0015] The hierarchical multi-level feature enhancement module HMFE can enhance and fuse multi-level image features extracted by the pre-trained language model CLIP, thereby improving the model's detection performance for small animal targets.

[0016] The Modal Decoupling Module (MDM) decouples the image features extracted by CLIP into semantically relevant components and visually relevant components, achieving semantic alignment of CLIP image-text features. This helps to distinguish animals with similar appearance features and can also assist in identifying camouflaged animal targets.

[0017] The Multi-Branch Fusion Module (MBF) dynamically fuses high-level semantic information with low-level visual features and uses the HMFE-enhanced image features as prior knowledge to avoid category confusion between camouflaged animal targets and the background, thereby reducing the false negative rate of camouflaged animals.

[0018] This invention employs a multimodal feature enhancement and fusion strategy to detect wild animals in limited-sample scenarios. Its main technical contributions include:

[0019] A hierarchical multi-level feature enhancement module (HMFE) is constructed to enhance the ability to extract local detail features and contextual information of small targets, thereby improving the model's ability to detect small-sized animal targets.

[0020] The Modal Decoupling Module (MDM) is designed to achieve semantic alignment of CLIP image-text features, thereby improving the detection capability for highly camouflaged animals and visually similar animals.

[0021] A multi-branch fusion module (MBF) is constructed to avoid category confusion between camouflaged animal targets and the background, thereby reducing the false negative rate of camouflaged animals. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the method for detecting wild animal targets in a small number of samples based on multimodal feature enhancement and fusion according to the present invention;

[0023] Figure 2 Example images of the experimental dataset of this invention;

[0024] Figure 3 This is a schematic diagram of the hierarchical multi-level feature enhancement module HMFE proposed in this invention;

[0025] Figure 4 This is a schematic diagram of the multi-branch fusion module MBF proposed in this invention;

[0026] Figure 5 A comparison of the number of parameters in the few-sample target detection method of the present invention;

[0027] Figure 6 Comparison of 5-shot target detection results;

[0028] Figure 7 Comparison of HMFE module feature activation heatmaps for this invention;

[0029] Figure 8This is a diagram showing the 5-shot target detection results of the present invention in complex outdoor scenarios. Detailed Implementation

[0030] This invention is a method for detecting wild animal targets with few samples based on multimodal feature enhancement and fusion, the steps of which include:

[0031] First, we collect images of wild animals in the wild, construct a new few-sample class dataset for wild animals called WildFS, and generate corresponding text prompts for each species category.

[0032] Meanwhile, animal species images from the public datasets MSCOCO and PASCAL VOC were selected as the base class dataset TrainD to construct a few-sample wildlife target detection model;

[0033] The detection model includes a hierarchical multi-level feature enhancement module (HMFE), a modal decoupling module (MDM), and a multi-branch fusion module (MBF).

[0034] In the detection model:

[0035] The hierarchical multi-level feature enhancement module HMFE can enhance and fuse multi-level image features extracted by the pre-trained language model CLIP, thereby improving the model's detection performance for small animal targets.

[0036] The Modal Decoupling Module (MDM) decouples the image features extracted by CLIP into semantically relevant components and visually relevant components, achieving semantic alignment of CLIP image-text features. This helps to distinguish animals with similar appearance features and can also assist in identifying camouflaged animal targets.

[0037] The Multi-Branch Fusion Module (MBF) dynamically fuses high-level semantic information with low-level visual features and uses the HMFE-enhanced image features as prior knowledge to avoid category confusion between camouflaged animal targets and the background, thereby reducing the false negative rate of camouflaged animals.

[0038] Then, the detection model is pre-trained using the base class dataset TrainD.

[0039] Then, some parameters of the pre-trained model are fine-tuned using the WildFS dataset.

[0040] Finally, using images of wild animals as input to the detection model, the output is the detection results of wild animal targets from a small sample.

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0042] This invention provides Figure 1The method for detecting wild animal targets in a few samples, shown below, is based on multimodal feature enhancement and fusion. The specific steps include:

[0043] (i) Collect images of wild animals in the wild, construct a new dataset of wild animals with few samples, WildFS, and generate corresponding text prompts for each species category;

[0044] (ii) Animal species images from the public datasets MSCOCO and PASCAL VOC were selected as the base class dataset TrainD;

[0045] (III) Constructing a few-sample wildlife target detection model based on multimodal feature enhancement and fusion;

[0046] (iv) Training phase: The few-shot target detection model is trained using the base class dataset TrainD to obtain a pre-trained model;

[0047] (v) Fine-tuning stage: Use the WildFS dataset with few samples to fine-tune some parameters of the pre-trained model obtained in step 4).

[0048] The constructed few-shot wildlife target detection model mainly consists of three modules: a hierarchical multi-level feature enhancement module (HMFE), a modality decoupling module (MDM), and a multi-branch fusion module (MBF). The specific steps for constructing the few-shot wildlife target detection model include:

[0049] 1) The text encoder of the pre-trained language model CLIP is used to encode the constructed text prompt information, and then normalization and average pooling operations are performed to generate the text prototype matrix T, i.e., the text features. The steps involved in generating the text prompt message include:

[0050] 1.1) Construct a scientific name-common name mapping table to convert the Latin scientific name of a species into its English common name;

[0051] 1.2) Construct M different text prompt templates for each species category, and generate text prompt information containing text descriptions of the species category and its physical characteristics;

[0052] 2) Extract image features using the image encoder of the pre-trained language model CLIP. ;

[0053] 3) Use the image features obtained in step 2) After being encoded by the Transformer encoder, a dense feature map is generated. After object query is extracted, it is input into the Transformer decoder to complete the target localization and output candidate target boxes.

[0054] 4) Based on the candidate bounding box coordinates output in step 3), the CLIP image features obtained in step 2) Perform the ROI Align operation to crop and align regions, obtaining the local features of each candidate region. ;

[0055] 5) Construct a hierarchical multi-level feature enhancement module (HMFE). The HMFE enhances the local image features obtained in step 4). Feature enhancement and fusion are performed to obtain multi-scale enhanced features. The execution steps of the HMFE module include:

[0056] 5.1) Extracting multi-level features using the image encoder of the pre-trained language model CLIP. ;

[0057] 5.2) The features obtained in step 5.1) are respectively processed by the corresponding fine-grained feature extraction module (FFE) for feature enhancement to obtain the enhanced features. The execution steps of the FFE module include:

[0058] 5.2.1) Input hierarchical features After processing by the LayerNorm layer, the normalized features are added to the residuals of the input layer features, and then the dimensions are reduced by the Down projection layer.

[0059] 5.2.2) Input the dimensionality-reduced features from step 5.2.1) into three parallel branches. The three branches use dilated convolutions with different dilation rates to obtain receptive fields of different sizes. This expands the receptive field without increasing the number of parameters and avoids the loss of semantic information of small-scale animal targets in the image.

[0060] 5.2.3) Calculate the average value of the output features of the three branches, and then... The convolutional layer performs feature fusion to obtain the fused features; 5.2.4) The fused features are then passed through the activation unit GeLU and the up projection layer to restore them to their original dimensions, and then combined with the input features. By summing the residuals, the enhanced features of a single branch are obtained. ;

[0061] 5.3) Features obtained in step 5.2.4) Adaptive weighted fusion is performed using LearnableWeight Fusion, and then multi-scale enhanced features are output through a linear layer. .

[0062] 6) Design a Modal Decoupling Module (MDM). The MDM module is a structure with two parallel linear projection layers, which will input the characteristics of the MDM module. Decoupling into complementary modal information, i.e., semantically relevant components. Vision-related components The linear layer maintains the feature dimension of the input image features unchanged, and introduces a Frobenius norm orthogonal loss function during training to constrain the semantically related components and the visually related components in order to maintain their discriminability.

[0063] 7) Construct a multi-branch fusion module (MBF) to integrate text features. semantically related components Visual related components and multi-scale enhancement features Perform dynamic fusion and output classification Logits, i.e. The specific steps of the MBF module include:

[0064] 7.1) Text features obtained in step 1) The semantically related components obtained in step 6) The soft-alignment method is used to perform matching in the semantic space to obtain text-related classification logits, i.e. The calculation formula is shown in equation (1):

[0065]

[0066] In the formula, This is a temperature parameter, a hyperparameter greater than 0, used for adjustment. and The sharpness of the soft alignment distribution between them.

[0067] 7.2) and visually relevant components After feature transformation of the input bottleneck structure Bottleneck, the input adapter Adapter1 obtains the visually relevant classification logits, i.e. ;

[0068] 7.3) will and The logits obtained by weighted fusion (i.e. Input adapter Adapter3 to learn the inter-class obfuscation pattern;

[0069] 7.4) The multi-scale enhancement features obtained in step 5.3) Input adapter Adapter2 to learn the prior knowledge that leads to confusion between classes;

[0070] 7.5) Add the output logits of the two adapters Adapter3 and Adapter2, and then process the resulting logits through adapter Adapter4 to fuse the inter-class confusion features learned in steps 7.3) and 7.4);

[0071] 7.6) Using a residual structure, the logits output by adapter4 are compared with those obtained in step 7.3). The connection is used to remove learned inter-class confusion patterns in order to distinguish between animals of different species that look similar.

[0072] 7.7) Multi-scale enhancement features Through the weight generator ( Generate fusion weights ;

[0073] 7.8) Based on the aforementioned fusion weights The output of step 7.4) The logits output by Adapter2 (i.e.) The weighted fusion is performed to obtain the final classification logits. .

[0074] 8) Match the classification Logits output in step 7.8) with the candidate target box coordinates output by the Transformer decoder in step 3), and combine the category prediction results with the localization results to obtain the final few-shot target detection output (including target category and bounding box).

[0075] To verify the performance of the animal tracking method proposed in this invention, a large number of experiments were conducted.

[0076] The dataset comes from two sources. One source is a public dataset, which can be found at the URLs PASCALVOC (http: / / host.robots.ox.ac.uk / pascal / VOC / ) and MSCOCO (http: / / cocodataset.org / #download). The other source is a large number of images of endangered or rare species captured by infrared trap cameras in five protected areas between 2015 and 2017. Figure 2 Example images of the dataset are shown. The software program was written in the PyTorch framework and trained on a computer configured with an NVIDIA GeForce V100 (32GB VRAM).

[0077] To verify the performance of the proposed model in the task of detecting targets in wild animals with few samples, a series of ablation experiments were designed, and the results are shown in Table 1.

[0078] This invention constructs three modules: MDM, HMFE, and MBF. The HMFE module enhances CLIP image feature representation layer by layer through parallel FFE sub-modules and dilated convolutions, improving the detection accuracy of small wild animal targets. The MDM module adopts a text-visual dual-splitter architecture to achieve fine-grained decoupling of modal features, which can enhance the detection capability of highly camouflaged wild animals with similar visual features, achieving a maximum nAP50 improvement of 11.7% in 1 to 5-shot scenarios. The MBF module uses the image features enhanced by HMFE as prior knowledge and alleviates inter-class confusion and reduces the false negative rate through adapter, residual network, and dual-path adaptive fusion mechanism, achieving a maximum nAP50 improvement of 1.7% in 1 to 5-shot scenarios.

[0079] In summary, the collaborative work of the three modules significantly improved the nAP50 index across all scenarios from 1 to 5 shots. Specifically, the nAP50 reached 27.3% in the 1-shot scenario and 59.5% in the 5-shot scenario, effectively enhancing the detection accuracy of wild animals in scenarios with few samples. This model can be widely applied to actual field monitoring, providing data support for wildlife protection and ecological monitoring.

[0080] Table 1 Ablation Experiment Results

[0081]

[0082] To verify the advancement of the proposed method, the MSF-DETR model was compared with mainstream few-shot object detection methods such as TFA and FSDetView on the WildFS dataset. The results are shown in Table 2. In 1-shot, 3-shot, and 5-shot scenarios, the nAP50 of the proposed model reached 27.3%, 55.4%, and 59.5%, respectively, significantly outperforming the comparison methods and improving upon the second-best methods in 3-shot and 5-shot scenarios by 15.2 and 6.5 percentage points, respectively.

[0083] Table 2 Comparison of experimental results

[0084]

[0085] Figure 5The figure shows a comparison of the parameter count and inference speed of the method of the present invention with other few-sample target detection methods. As can be seen from the figure, the few-sample target detection method of the present invention achieves a good balance between accuracy and efficiency, and can be deployed on field mobile monitoring equipment with limited computing resources, thus possessing outstanding practical value.

[0086] Figure 6 The presentation shows a comparison of 5-shot object detection results for different methods. Due to the scarcity of training data for new categories, most of the compared methods suffer from serious false negatives, especially TFA and VFA. Furthermore, FPD and DE-VIT both generate a large number of redundant bounding boxes. In contrast, the model proposed in this invention maintains relatively stable detection performance in complex vegetation backgrounds and nighttime infrared scenes.

[0087] Figure 7 shows the feature activation maps before and after introducing the HMFE module. The results show that when processed by the CLIP model alone, the feature activation map incorrectly focuses on a large amount of irrelevant background and the focus on the animal target is incomplete. After introducing the HMFE module, the feature activation map can significantly focus on the animal target, the high activation region accurately covers the target outline, and background interference is effectively suppressed. This effect confirms that the HMFE module enhances the model's perception and localization capabilities for small and camouflaged wild animal targets through hierarchical multi-scale feature extraction and adaptive fusion strategies.

[0088] Figure 8 illustrates the wildlife detection results of the method of this invention in a complex field setting with a small 5-shot sample size, covering three common and challenging situations in actual animal monitoring: (a) varying lighting environments such as sunlight and shadow, low contrast in night vision infrared, and low light in foggy weather; (b) small camouflaged targets easily confused with the background; and (c) morphologically similar ungulates whose subtle differences in appearance can easily lead to classification confusion. Under these complex conditions, the method can still accurately classify and locate various targets, fully demonstrating its excellent adaptability and robustness in complex field situations.

Claims

1. A method for detecting wild animal targets with few samples based on multimodal feature fusion, characterized in that, First, a few-sample wildlife target detection model based on multimodal feature fusion is constructed; Then, the object detection model is trained using images of wild animal species from a public dataset as the base class dataset, resulting in a pre-trained object detection model. Next, images of wild animals in the wild were collected to construct WildFS, a new dataset of few-shot wild animals, and text prompts were generated for each species category. Finally, the WildFS dataset was used to fine-tune some parameters of the pre-trained object detection model; Using images of wild animals in outdoor scenes as input, an object detection model is used to output object detection results; The target detection model includes: a hierarchical multi-level feature enhancement module (HMFE), a modality decoupling module (MDM), and a multi-branch fusion module (MBF); The processing flow of the object detection model includes: 1) First, the text encoder of the pre-trained language model is used to encode the text prompt information corresponding to the species category; Then, normalization and average pooling operations are performed to generate a text prototype matrix T, which represents the text features. ; 2) An image encoder based on a pre-trained language model is used to extract image features from wildlife images in outdoor scenes. ; 3) Image features The Transformer encoder generates a dense feature map, extracts the object query ObjectQueries, and then inputs it into the Transformer decoder to complete the target localization and output candidate target boxes; 4) Based on the coordinates of the candidate target boxes, the image features obtained in step 2) Perform the Region of Interest (ROIAlign) operation to crop and align the regions, obtaining the local features of each candidate region. ; 5) Construct a hierarchical multi-level feature enhancement module (HMFE); HMFE enhances the local image features obtained in step 4). Feature enhancement and fusion are performed to obtain multi-scale enhanced features. ; 6) Using MDM to Decoupled into semantically related components Vision-related components ; 7) Use MBF to extract text features semantically related components Visual related components and multi-scale enhancement features Perform dynamic fusion and output classified Logits, represented as ; 8) Match the classification Logits obtained in step 7) with the candidate target bounding box coordinates obtained in step 3), and combine the category prediction results with the localization results to obtain the final few-shot target detection output; the target detection output includes the target category and bounding box. The core component of the HMFE is the fine-grained feature extraction module (FFE); within the HMFE: 5.1) Image encoder of pre-trained language model extracts multi-level features ; 5.2) The multi-level features are enhanced by applying the corresponding FFE to obtain the enhanced features. ; The execution process of FFE includes: 5.2.1) Input hierarchical features After processing by the LayerNorm layer, the normalized features are added to the input layer features by residual addition, and then the dimensions are reduced by the Down projection layer. 5.2.2) Input the dimensionality-reduced features into three parallel branches; the three branches use dilated convolutions with different dilation rates to obtain receptive fields of different sizes; 5.2.3) Calculate the average value of the output features of the three branches, and then... Convolutional layers perform feature fusion to obtain fused features; 5.2.4) The fused features are sequentially passed through the activation unit GeLU and the up projection layer to restore them to their original dimensions; then they are combined with the input layer features. By summing the residuals, the enhanced features of a single branch are obtained. ; 5.3) After adaptive weighted fusion using Learnable Weight Fusion, multi-scale enhanced features are output through a linear layer. ; The MDM has two parallel linear projection layer structures that project the input features. Decoupled into complementary modal information, which are semantically related components. Vision-related components ; The MBF processing steps include: 7.1) Text features obtained in step 1) The semantically related components obtained in step 6) The soft-alignment method is used to perform matching in the semantic space to obtain text relevance classification logits, which are represented as follows: ; 7.2) and visually relevant components After performing feature transformation on the input bottleneck structure Bottleneck, and then inputting it into the first adapter Adapter1, visual relevance classification logits are obtained, represented as follows: ; 7.3) will and The logits obtained by weighted fusion are represented as Next, input the third adapter Adapter3 to learn the inter-class obfuscation pattern; 7.4) will The second adapter, Adapter2, is used to obtain logits, which are represented as follows: It is used to learn prior knowledge that leads to confusion between classes; 7.5) The logits obtained by adding the output logits of Adapter3 and Adapter2 are then processed by the fourth adapter Adapter4 to fuse the inter-class obfuscation features learned in steps 7.3) and 7.4); 7.6) Use a residual structure to compare the logits output by Adapter4 with the logits obtained in step 7.3). Connections are used to remove learned inter-class confusion patterns in order to distinguish between animals of different species that look similar. 7.7) Through weight generator Generate fusion weights ; 7.8) Based on weights The output of step 7.4) and We perform weighted fusion to obtain the final classification logits, which are represented as .

2. The method for detecting wild animal targets with few samples based on multimodal feature fusion according to claim 1, characterized in that, Step 1) The steps for generating the text prompt information are as follows: 1.1) Construct a scientific name-common name mapping table to convert the Latin scientific name of a species into its English common name; 1.2) Construct multiple different text prompt templates for each species category to generate text prompt information containing text descriptions of the species category and its physical characteristics.

3. The method for detecting wild animal targets with few samples based on multimodal feature fusion according to claim 1, characterized in that, In step 6), the linear projection layer of the MDM module maintains the feature dimension of the input image features unchanged, and introduces the Frobenius norm orthogonal loss function during training to constrain the semantically related components and the visually related components.

4. The method for detecting wild animal targets with few samples based on multimodal feature fusion according to claim 1, characterized in that, Step 7.1) The soft-alignment method is expressed as follows: ， In the formula, This is a temperature parameter, a hyperparameter greater than 0, used for adjustment. and The sharpness of the soft alignment distribution between them.

5. The method for detecting wild animal targets with few samples based on multimodal feature fusion according to claim 1, characterized in that, Wildlife species images from the public datasets MSCOCO and PASCAL VOC are used as the base class dataset TrainD.

6. The method for detecting wild animal targets with few samples based on multimodal feature fusion according to claim 1, characterized in that, The language model is CLIP.

Images