Wild animal detection model training method and system based on multi-modal feature fusion
By using the CSP module and query-key-value mechanism in wildlife detection, combined with modal consistency loss and learnable weights, the contribution of visible light and infrared features is dynamically adjusted, which solves the shortcomings of multimodal fusion detection methods in terms of robustness and accuracy, and achieves stable detection results in different environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2025-12-17
- Publication Date
- 2026-05-01
AI Technical Summary
Existing multimodal fusion detection methods suffer from low model robustness and poor detection accuracy in wildlife detection. In particular, they are difficult to dynamically balance the advantages of visible light and infrared light modes under different environments, resulting in unstable detection results.
The CSP module is used as the backbone network. Symmetrical cross-attention calculation of visible light and infrared features is realized through query-key-value mechanism. Modality consistency loss and learnable weight coefficients are introduced during feature fusion to dynamically adjust the contribution of the two modalities and construct a feature pyramid to improve detection accuracy.
It improves the robustness and accuracy of wildlife detection, maintains stable performance under different environmental conditions, and avoids performance fluctuations caused by traditional fixed-weight fusion strategies.
Smart Images
Figure CN121963250A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and animal protection technology, specifically to a method and system for training wildlife detection models based on multimodal feature fusion. Background Technology
[0002] Monitoring and protecting wild animals is of great practical significance for protecting biodiversity and maintaining ecosystem balance. When computer vision technology is applied to the identification, counting, and behavioral monitoring of wild animals, compared with traditional manual inspection methods, it not only enables real-time monitoring over a wide area around the clock, but also avoids adverse impacts on the normal activities of wild animals.
[0003] When using machine vision for wildlife detection, it is often based on a single modality. Specifically, it is either based on visible light images or infrared images. However, when using visible light images, it is easily affected by insufficient lighting at night or in bad weather, leading to blurred target features and a high rate of missed detections. While infrared images have the advantages of strong penetration and being unaffected by lighting conditions, they lack texture information and are difficult to effectively distinguish and identify small targets or similar species.
[0004] While some studies employ both visible light and infrared images for multimodal fusion detection, including image stitching and weighted processing of detection results, these existing methods still suffer from the following limitations: image stitching easily leads to feature interference between modalities, causing feature confusion and increasing model complexity; weighted processing of detection results fails to fully exploit the interaction of bimodal features and struggles to handle complex scenarios such as occlusion and background interference; furthermore, fixed weight allocation dynamically balances visible and infrared modes under different environments (such as day-night cycles and rainy / foggy weather). Consequently, existing multimodal fusion detection methods still exhibit low model robustness and poor detection accuracy when used for wildlife detection. Summary of the Invention
[0005] The purpose of this invention is to provide a training method and system for a wildlife detection model based on multimodal feature fusion, so as to solve the technical problems of low model robustness and poor detection accuracy when using existing multimodal fusion detection methods for wildlife detection.
[0006] To achieve the above objectives, the present invention proposes the following technical solution: Firstly, a training method for wildlife detection models based on multimodal feature fusion is provided, including: Acquire several pairs of visible light images and infrared images; input the visible light images into the first branch network to obtain visible light feature maps output at a preset level, and input the infrared images into the second branch network to obtain infrared feature maps output at a preset level; wherein, both the first branch network and the second branch network use CSP modules as backbone networks; After flattening the visible light feature map and the infrared feature map to obtain the corresponding one-dimensional visible light feature map and one-dimensional infrared feature map, feature mapping and query generation operations are performed to obtain the visible light query vector, infrared key vector and infrared value vector corresponding to the one-dimensional visible light feature map, and the infrared query vector, visible light key vector and visible light value vector corresponding to the one-dimensional infrared feature map. The infrared attention of visible light features to infrared features is calculated based on the visible light query vector, infrared key vector, and infrared value vector. The target visible light feature map is obtained by combining the one-dimensional visible light feature map. The visible light attention of infrared features to visible light features is calculated based on the infrared query vector, visible light key vector, and visible light value vector. The target infrared feature map is obtained by combining the one-dimensional infrared feature map. The infrared attention level is as follows: The visible light attention level is: ; where Q vis For visible light query vector, Q inf For infrared query vector, K vis For visible light bond vectors, K inf For infrared bond vectors, V vis For visible light value vector, V inf d is the infrared value vector; k The dimension of the key vector; After obtaining the fused feature map based on the target's visible light feature map and infrared feature map, it is input into the FPN module to construct the feature pyramid. At the same time, the modality consistency loss is introduced into the hybrid loss function for model training optimization. Specifically, when acquiring the fused feature map, corresponding learnable weight coefficients are introduced for the target visible light feature map and the target infrared feature map, respectively; the modal consistency loss is: The CTF() function represents the calculation function of the CTF module. One-dimensional visible light feature map A one-dimensional infrared feature map, i Indicates the position index number in the feature sequence; The learnable weight coefficients are adjusted by the hybrid loss function.
[0007] Furthermore, acquiring several sets of paired visible light images and infrared images includes: Acquire several raw visible light images and raw infrared images; the raw visible light images and raw infrared images correspond to the target animal categories in various scenes; The original visible light image is standardized to eliminate illumination differences, thereby obtaining an intermediate visible light image; the original infrared image is histogram equalization to enhance contrast, thereby obtaining an intermediate infrared image. Data enhancement processing is performed on both the intermediate visible light image and the intermediate infrared image to achieve multimodal pairing enhancement, resulting in several pairs of visible light images and infrared images; wherein, the data enhancement processing includes: random cropping, rotation and flipping, and noise addition.
[0008] Furthermore, the step of obtaining a fused feature map based on the target visible light feature map and the target infrared feature map, and then inputting it into the FPN module to construct a feature pyramid, includes: Channel compression is performed on the fused feature map based on convolutional layers to make the channel dimension of the fused feature map consistent with the channel dimension of the preset layer; The compressed fused feature map is input into the FPN module.
[0009] Furthermore, including: The hybrid loss function is: ; in, L box The bounding box regression loss is calculated using the CIoU() function, where λ1 is the bounding box regression loss coefficient. L obj To determine the existence loss of the target, binary cross-entropy is used to predict whether the target exists, where λ2 is the target existence loss coefficient. L cls For classification loss, cross-entropy is used to classify the target category, λ3 is the classification loss coefficient; λ4 is the modality consistency loss coefficient.
[0010] Furthermore, the model training optimization includes: After freezing the weights of the shallow convolutional layers of the backbone networks in the first and second branch networks, iterative training begins. When the preset training progress value is reached, the weights of each frozen shallow convolutional layer are unfrozen sequentially according to the preset number of training rounds and added to the iterative training in sequence. During the iterative training process, the target visible light feature map and the target infrared feature map are updated through backpropagation, and corresponding learnable weight coefficients are introduced respectively.
[0011] Secondly, a training system for a wildlife detection model based on multimodal feature fusion is provided, including: The image acquisition module is used to acquire several pairs of visible light images and infrared images; and input the visible light images into the first branch network to obtain a visible light feature map output at a preset level, and input the infrared images into the second branch network to obtain an infrared feature map output at a preset level; wherein, both the first branch network and the second branch network use the CSP module as the backbone network; The mapping generation module is used to flatten the visible light feature map and the infrared feature map respectively to obtain the corresponding one-dimensional visible light feature map and one-dimensional infrared feature map, and then perform feature mapping and query generation operations to obtain the visible light query vector, infrared key vector and infrared value vector corresponding to the one-dimensional visible light feature map, and the infrared query vector, visible light key vector and visible light value vector corresponding to the one-dimensional infrared feature map. The feature processing module is used to calculate the infrared attention of visible light features to infrared features based on visible light query vectors, infrared key vectors, and infrared value vectors, and to obtain the target visible light feature map by combining the one-dimensional visible light feature map; and to calculate the visible light attention of infrared features to visible light features based on infrared query vectors, visible light key vectors, and visible light value vectors, and to obtain the target infrared feature map by combining the one-dimensional infrared feature map. The infrared attention level is as follows: The visible light attention level is: ; where Q vis For visible light query vector, Q inf For infrared query vector, K vis For visible light bond vectors, K inf For infrared bond vectors, V vis For visible light value vector, V inf d is the infrared value vector; k The dimension of the key vector; The fusion training module is used to obtain a fused feature map based on the target visible light feature map and the target infrared feature map, and then input it into the FPN module to construct a feature pyramid. At the same time, modality consistency loss is introduced into the hybrid loss function for model training optimization. Specifically, when obtaining the fused feature map, corresponding learnable weight coefficients are introduced into the target visible light feature map and the target infrared feature map respectively. The modal consistency loss is as follows: The CTF() function represents the calculation function of the CTF module. One-dimensional visible light feature map A one-dimensional infrared feature map, i Indicates the position index number in the feature sequence; The learnable weight coefficients are adjusted by the hybrid loss function.
[0012] Furthermore, the image acquisition module includes: The acquisition unit is used to acquire several raw visible light images and raw infrared images; wherein the raw visible light images and raw infrared images correspond to the target animal categories in various scenarios; The first processing unit is used to standardize the original visible light image to eliminate illumination differences and then obtain an intermediate visible light image; and to perform histogram equalization on the original infrared image to enhance contrast and then obtain an intermediate infrared image. The second processing unit is used to perform data enhancement processing on both the intermediate visible light image and the intermediate infrared image to achieve multimodal pairing enhancement, thereby obtaining several pairs of paired visible light images and infrared images; wherein, the data enhancement processing includes: random cropping, rotation and flipping, and noise addition.
[0013] Furthermore, the fusion training module includes: A compression unit is used to perform channel compression on the fused feature map based on the convolutional layer so that the channel dimension of the fused feature map is consistent with the channel dimension of the preset layer; The input unit is used to input the compressed fused feature map into the FPN module.
[0014] Thirdly, an electronic device is provided, including at least one processor coupled to a memory storing a computer program configured to be executed by the processor when run.
[0015] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement the method.
[0016] Beneficial effects: As can be seen from the above technical solutions, the technical solution of the present invention provides a training method for a wildlife detection model based on multimodal feature fusion to solve the defects of existing multimodal feature fusion methods in terms of robustness and detection accuracy.
[0017] This technical solution first considers the impact of the network architecture itself on training efficiency and detection, using the CSP module as the backbone network for both the first and second branch networks. Then, it reduces redundant computation and retains rich gradient information through segmentation, processing, and splicing strategies. Next, it implements symmetrical cross-attention computation of visible light and infrared features through a query-key-value mechanism, ensuring that both the target visible light and infrared feature maps carry the associated visible light-infrared feature information to avoid feature confusion and improve the accuracy of feature information acquisition, thereby enhancing the accuracy of subsequent detection and recognition. Furthermore, during subsequent feature fusion, considering the changing importance of different modes under different environmental conditions, it introduces corresponding learnable weight coefficients for the target visible light and infrared feature maps respectively, and introduces a mode consistency loss into the hybrid loss function. Then, it continuously optimizes the corresponding learnable weight coefficients through backpropagation, thereby dynamically adjusting the contribution of visible light and infrared features. Finally, it endows the model with the ability to dynamically perceive mode contribution, enabling the model to maintain stable performance in scenarios where visible light dominates during the day and infrared dominates at night, effectively avoiding performance fluctuations caused by traditional fixed-weight fusion strategies.
[0018] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other.
[0019] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description
[0020] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of the training method for a wildlife detection model based on multimodal feature fusion described in this embodiment; Figure 2 This is a flowchart illustrating the acquisition of visible light and infrared images in this embodiment; Figure 3 This is a flowchart illustrating the channel compression process in this embodiment; Figure 4 A flowchart illustrating the training optimization process for this embodiment; Figure 5 This is a block diagram of the training system for a wildlife detection model based on multimodal feature fusion as described in this embodiment; Figure 6 This is a structural block diagram of the electronic device described in this embodiment. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art to which this invention pertains.
[0022] The terms "first," "second," and similar words used in this application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "an," "a," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. Terms such as "comprising" or "including" mean that the element or object preceding "comprising" encompasses the features, integrals, steps, operations, elements, and / or components listed following "comprising" or "including," and do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or sets thereof. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0023] In existing technologies, wildlife detection based on machine vision technology mainly relies on a single modality. However, visible light images are easily affected by illumination at night or in bad weather, resulting in blurred target features and a high rate of missed detection. Although infrared images have the advantages of strong penetration and are not affected by illumination, their texture information is insufficient, making it difficult to distinguish small targets or similar species. Traditional multimodal fusion methods have the following problems: (1) Feature confusion: When fusion is based on image stitching, it is easy to cause feature interference between modalities, increasing the complexity of the model; (2) Insufficient complementarity: When fusion is based on result weighting, it is difficult to fully explore the interaction of bimodal features, making it difficult to cope with complex scenarios such as occlusion and background interference; and fixed weight allocation cannot dynamically balance visible light images and infrared images in different environments (such as day and night alternation, rain and fog). Based on this, this embodiment aims to provide a wildlife detection model training method based on multimodal feature fusion to solve the above-mentioned technical defects at the same time.
[0024] The following section, with reference to the accompanying drawings, provides a detailed description of the training method for a wildlife detection model based on multimodal feature fusion as described in this embodiment.
[0025] To execute the method described, model training in this embodiment is specifically based on an NVIDIA GeForce RTX 4090 GPU. This GPU is equipped with 24GB of GDDR6X memory and an AD102 architecture, supporting FP16 mixed-precision computation, which can significantly accelerate the model training process. In terms of software framework, PyTorch 2.0 is used to build the YOLO11 model, CUDA 12.1 is used for accelerated computation, and the torch.compile optimizer is used to improve training efficiency. Regarding image processing tools, OpenCV 4.7 is used for image preprocessing (such as normalization and histogram equalization) and post-processing (such as non-maximum suppression, NMS).
[0026] Specifically, in combination Figure 1 The method includes: Step S202: Obtain several pairs of visible light images and infrared images; input the visible light images into the first branch network to obtain the visible light feature map output at the preset level, and input the infrared images into the second branch network to obtain the infrared feature map output at the preset level.
[0027] In this embodiment, in order to improve the model's generalization ability, and also considering improving the model's robustness and detection accuracy, the following measures are taken: Figure 2 As shown, several pairs of paired visible light and infrared images are acquired in the following manner: Step S20202: Acquire several raw visible light images and raw infrared images.
[0028] This embodiment collects raw visible light and raw infrared images from scenes such as nature reserves and wildlife habitats, covering target categories of wild animals such as mammals and birds. Furthermore, during the data collection process, special attention was paid to sample diversity under different lighting conditions (e.g., daytime and nighttime) and weather conditions (e.g., sunny days and rain / fog).
[0029] Step S20204: Standardize the original visible light image to eliminate illumination differences and obtain an intermediate visible light image; perform histogram equalization on the original infrared image to enhance contrast and obtain an intermediate infrared image.
[0030] Specifically, the original visible light image is subjected to mean-variance standardization to normalize the pixel value range to [0,1], thereby eliminating the lighting differences under different devices or scenes.
[0031] Histogram equalization is used to enhance contrast and highlight thermal radiation features in the original infrared image. The specific formula is as follows: ; in, I These are the original pixel values. I enhanced To enhance pixel values.
[0032] Step S20206: Perform data enhancement processing on both the intermediate visible light image and the intermediate infrared image to achieve multimodal pairing enhancement, resulting in several pairs of visible light images and infrared images.
[0033] In this embodiment, the data augmentation process includes random cropping, rotation and flipping, and noise addition. Random cropping involves cropping a random region centered on the target (cropping ratio ranging from 0.8 to 1.2) to simulate target detection requirements under different field-of-view conditions. Rotation and flipping involve randomly rotating the image from 0° to 360° and flipping it horizontally / vertically to improve the model's adaptability to changes in target pose. Noise addition is used to add Gaussian noise (mean = 0, standard deviation = 0.1) to the image during the training phase to simulate sensor interference in real-world scenarios. The resulting multimodal pairing data exhibits good consistency after these augmentation operations.
[0034] In this embodiment, both the first and second branch networks use the CSP module in the YOLO11 architecture as the backbone network. Its lightweight design and high-precision feature extraction capabilities enable it to perform well in multimodal fusion. The preset layers in both the first and second branch networks are P4 layers (corresponding to 1 / 16 resolution feature maps); simultaneously, to ensure compatibility with subsequent feature fusion, the number of channels in the P4 layers of both branch networks is set to 512.
[0035] Continuing, a CTF module is inserted into the P4 layer, which can perform symmetrical cross-attention calculation of visible light and infrared features through a query-key-value mechanism. Specifically, the following steps are performed: Step S204: After flattening the visible light feature map and the infrared feature map to obtain the corresponding one-dimensional visible light feature map and one-dimensional infrared feature map, perform feature mapping and query generation operations to obtain the visible light query vector, infrared key vector and infrared value vector corresponding to the one-dimensional visible light feature map, and the infrared query vector, visible light key vector and visible light value vector corresponding to the one-dimensional infrared feature map.
[0036] Specifically, the flattening operation facilitates the input of the corresponding feature map into the Transformer Encoder (the core component of the CTF module) for processing. Specifically, the original feature map of size H×W×C is flattened and converted into a sequence of length H×W, with each position corresponding to a C-dimensional feature vector.
[0037] The feature mapping and query generation process is the corresponding bidirectional attention acquisition process.
[0038] Specifically, the visible light query vector, infrared key vector, and infrared value vector corresponding to the one-dimensional visible light feature map are: ; Among them, Q vis For visible light query vector, K inf For infrared bond vectors, V inf For infrared value vectors, V vis One-dimensional visible light feature map V inf A one-dimensional infrared feature map; W Qvis W Kinf W Vinf For the corresponding weights.
[0039] The infrared query vector, visible light bond vector, and visible light value vector corresponding to the one-dimensional infrared feature map are: ; Among them, Q inf For infrared query vector, K vis For visible light bond vector, V vis W is the visible light value vector Qinf W Kvis W Vvis For the corresponding weights.
[0040] Correspondingly, the infrared attention level is: ; Where, d k is the dimension of the key vector.
[0041] The visible light feature map of the target is as follows: ; in, LayerNorm The function is used to normalize the characteristic distribution and prevent numerical instability.
[0042] Correspondingly, the visible light attention level is: .
[0043] The target's infrared signature is as follows: .
[0044] Step S206: After obtaining the fused feature map based on the target visible light feature map and the target infrared feature map, input it into the FPN module to construct the feature pyramid. At the same time, the modality consistency loss is introduced into the hybrid loss function for model training optimization.
[0045] To adapt to changes in modal importance under different environmental conditions, this invention introduces learnable weight parameters and maps these weights to the (0,1) interval using a sigmoid function, thereby dynamically adjusting the contribution of visible light and infrared features. Specifically, as shown in the following formula: ; in, W vis These are learnable weight parameters corresponding to the characteristics of visible light; W inf These are learnable weight parameters corresponding to infrared features. At this point, the model is given dynamic sensing capability regarding modal contributions; when the visible light quality of the input image is better than the infrared quality (e.g., in a well-lit daytime scene), the model automatically reduces... W inf_adj The value of determines whether visible light features dominate the fusion result; conversely, when the quality of infrared images is significantly better than that of visible light (such as at night or in rainy or foggy weather), the model will increase . W inf_adj The value of is used to prioritize the use of infrared features for target detection.
[0046] Continuing on, the fused feature representation corresponding to the final obtained fused feature map is as follows: .
[0047] Therefore, the essence of the fused feature map is to combine the complementarity of two modalities and dynamically balance their weight distribution through learnable parameters. For example, in nighttime scenes, visible light images may blur the target outline due to insufficient lighting, while infrared images can still retain the target shape information through thermal radiation characteristics. In this case, the model will automatically learn through training. W inf_adj →1、 W vis_adj The trend towards →0 makes infrared features the dominant factor; while in daytime scenes, visible light images have richer texture information, which enhances the model. W vis_adj The value of is determined by this mechanism, which enables the model to maintain stable performance in both daytime (dominantly visible light) and nighttime (dominantly infrared light) scenarios, effectively avoiding performance fluctuations caused by fixed-weight fusion.
[0048] Specifically, this adaptive mechanism is achieved through a loss function. During training, modality consistency loss is introduced into the hybrid loss function and the weight parameters are continuously optimized through backpropagation. This allows the model to automatically adjust the modality weights based on the environmental characteristics of the input data (such as light intensity and weather conditions), thereby avoiding performance fluctuations in complex scenarios caused by traditional fixed-weight fusion strategies.
[0049] Specifically, the modal consistency loss is: .
[0050] in, One-dimensional visible light feature map A one-dimensional infrared feature map, i This represents the position index number in the feature sequence. The CTF() function represents the computation function of the CTF module; it is built based on the multi-head attention mechanism of Transformer Encoder, using visible light features and infrared features as inputs, and achieving bidirectional attention interaction through Query-Key-Value projection, ultimately outputting fused features. The position index number is used to traverse the feature vector at each position in the sequence (e.g., i=1 corresponds to the pixel feature at the top left corner of the feature map, i=N corresponds to the pixel feature at the bottom right corner), and then, combined with modality consistency loss, forces the symmetry of the visible light-infrared and infrared-visible fusion results at each position, ensuring the spatial consistency of the CTF module.
[0051] Furthermore, the hybrid loss function is specifically expressed as follows: .
[0052] in, L boxThe bounding box regression loss is calculated using the CIoU() function, where λ1 is the bounding box regression loss coefficient. L obj To determine the existence loss of the target, binary cross-entropy is used to predict whether the target exists, where λ2 is the target existence loss coefficient. L cls For classification loss, cross-entropy is used to classify the target category, λ3 is the classification loss coefficient; λ4 is the modality consistency loss coefficient.
[0053] Specifically, in combination Figure 3 As shown, in step S206, to adapt to the subsequent Feature Pyramid Network (FPN module) of YOLO11, the fused feature map is subjected to the following channel number compression and feature integration processing before being input into the FPN module: Step S20602: Perform channel compression on the fused feature map based on the convolutional layer so that the channel dimension of the fused feature map is consistent with the channel dimension of the preset layer.
[0054] In this embodiment, the convolutional layer is specifically a 1×1 convolutional layer, used to compress the channel dimension of the fused feature map to 512.
[0055] Step S20604: Input the compressed fused feature map into the FPN module.
[0056] At this point, based on steps S20602 to S20604, while preserving spatial details and reducing computational redundancy, the model inference speed is ensured to meet real-time requirements.
[0057] As another specific implementation, in the training optimization process of step S206, the Adam optimizer (initial learning rate η=1e−5, batch size=16) is used for end-to-end training, and the weights of the first few convolutional layers of the backbone network (specifically the YOLO11 feature extraction backbone network) are frozen to prevent gradient explosion. Specifically, the first few convolutional layers refer to the shallow convolutional layers in the backbone network responsible for learning general low-level visual features. Since shallow features are highly consistent across different image modalities (visible light / infrared) and tasks, freezing them avoids destroying pre-trained weights and prevents gradient oscillations in the early stages of training. During training, the backbone network weights are gradually unfrozen, allowing the model to gradually adapt to multimodal feature fusion. Modality-adaptive weight parameters are automatically updated based on the backpropagation algorithm, enabling the model to adaptively adjust the contribution of the two modalities under different environmental conditions. For example, in rainy or foggy weather, the infrared modality weights are automatically increased to compensate for the blurring problem of visible light images.
[0058] Specifically, based on the above design concepts, combined with Figure 4 As shown, it specifically includes: Step S20622: After freezing the weights of the shallow convolutional layers of the backbone networks in the first and second branch networks, iterative training begins.
[0059] Specifically, the operations for the CTF module are as follows: during backpropagation, only the learnable weight parameters and Transformer Encoder parameters of the CTF module are calculated and updated. The operations for the FPN module are as follows: the convolutional layer and upsampling / downsampling layer parameters of the FPN module are updated. The weights of the backbone network remain fixed to prevent gradient explosion.
[0060] Step S20624: When the preset training progress value is reached, the weights of each frozen shallow convolutional layer are unfrozen sequentially according to the preset number of training rounds so that they can be added to the iterative training sequentially.
[0061] Specifically, when training reaches 30%, starting from the 5th layer, one layer is unfrozen every 5 epochs (e.g., unfrozen the 5th layer at epoch 50, unfrozen the 6th layer at epoch 55), so that the backbone network weights are gradually added to the optimization process, avoiding training oscillations caused by sudden unfreezing.
[0062] During the iterative training process in steps S20622 to S20624, the target visible light feature map and the target infrared feature map are updated by backpropagation, and corresponding learnable weight coefficients are introduced respectively.
[0063] In summary, this embodiment achieves high-precision, low-latency detection of wildlife targets and adapts to robustness requirements under complex environmental conditions through steps such as dual-modal input and feature extraction, cross-modal Transformer fusion module (CTF) design, mid-layer feature fusion and channel compression, loss function design and model training.
[0064] The aforementioned program can run in a processor or be stored in memory (or a computer-readable storage medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves.
[0065] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented using different modules, and different steps can be implemented using different modules.
[0066] This embodiment also provides a training system for a wildlife detection model based on multimodal feature fusion. Combined with... Figure 5 As shown, it includes the following functional modules: The image acquisition module is used to acquire several pairs of visible light images and infrared images; and input the visible light images into the first branch network to obtain visible light feature maps output at a preset level, and input the infrared images into the second branch network to obtain infrared feature maps output at a preset level; wherein, both the first branch network and the second branch network use the CSP module as the backbone network.
[0067] The mapping generation module is used to flatten the visible light feature map and the infrared feature map respectively to obtain the corresponding one-dimensional visible light feature map and one-dimensional infrared feature map. Then, it performs feature mapping and query generation operations to obtain the visible light query vector, infrared key vector and infrared value vector corresponding to the one-dimensional visible light feature map, and the infrared query vector, visible light key vector and visible light value vector corresponding to the one-dimensional infrared feature map.
[0068] The feature processing module is used to calculate the infrared attention of visible light features to infrared features based on visible light query vectors, infrared key vectors, and infrared value vectors, and to obtain the target visible light feature map by combining the one-dimensional visible light feature map; and to calculate the visible light attention of infrared features to visible light features based on infrared query vectors, visible light key vectors, and visible light value vectors, and to obtain the target infrared feature map by combining the one-dimensional infrared feature map.
[0069] The infrared attention level is as follows: The visible light attention level is: ; where Q vis For visible light query vector, Q inf For infrared query vector, K vis For visible light bond vectors, K inf For infrared bond vectors, V vis For visible light value vector, V inf d is the infrared value vector; k is the dimension of the key vector.
[0070] The fusion training module is used to obtain a fused feature map based on the target visible light feature map and the target infrared feature map, and then input it into the FPN module to construct a feature pyramid. At the same time, modality consistency loss is introduced into the hybrid loss function for model training optimization. Specifically, when obtaining the fused feature map, corresponding learnable weight coefficients are introduced into the target visible light feature map and the target infrared feature map respectively.
[0071] The modal consistency loss is as follows: The CTF() function represents the calculation function of the CTF module. One-dimensional visible light feature map A one-dimensional infrared feature map, i This represents the position index number in the feature sequence.
[0072] The learnable weight coefficients are adjusted by the hybrid loss function.
[0073] Since the system is built based on the method described above, the points already explained will not be repeated here.
[0074] For example, the image acquisition module includes: The acquisition unit is used to acquire several raw visible light images and raw infrared images; wherein the raw visible light images and raw infrared images correspond to the target animal categories in various scenarios.
[0075] The first processing unit is used to standardize the original visible light image to eliminate illumination differences and thus obtain an intermediate visible light image; and to perform histogram equalization on the original infrared image to enhance contrast and thus obtain an intermediate infrared image.
[0076] The second processing unit is used to perform data enhancement processing on both the intermediate visible light image and the intermediate infrared image to achieve multimodal pairing enhancement, thereby obtaining several pairs of paired visible light images and infrared images; wherein, the data enhancement processing includes: random cropping, rotation and flipping, and noise addition.
[0077] For example, the fusion training module includes: A compression unit is used to perform channel compression on the fused feature map based on the convolutional layer so that the channel dimension of the fused feature map is consistent with the channel dimension of the preset layer.
[0078] The input unit is used to input the compressed fused feature map into the FPN module.
[0079] At the same time, combined Figure 6 As shown, an electronic device is also provided, including at least one processor coupled to a memory storing a computer program configured to be executed by the processor when run.
[0080] Furthermore, a computer-readable storage medium is also provided, on which a computer program is stored, the computer program being executed by a processor to implement the method.
[0081] Since the system, electronic device, and storage medium are all built on or used to implement the method, they have significant technical advantages in terms of robustness and accuracy in practical applications.
[0082] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.
Claims
1. A training method for a wildlife detection model based on multimodal feature fusion, characterized in that, include: Acquire several pairs of visible light and infrared images; The visible light image is input into the first branch network to obtain a visible light feature map output at a preset level, and the infrared image is input into the second branch network to obtain an infrared feature map output at a preset level; wherein, both the first branch network and the second branch network use CSP modules as backbone networks; After flattening the visible light feature map and the infrared feature map to obtain the corresponding one-dimensional visible light feature map and one-dimensional infrared feature map, feature mapping and query generation operations are performed to obtain the visible light query vector, infrared key vector and infrared value vector corresponding to the one-dimensional visible light feature map, and the infrared query vector, visible light key vector and visible light value vector corresponding to the one-dimensional infrared feature map. The infrared attention of visible light features to infrared features is calculated based on the visible light query vector, infrared key vector, and infrared value vector. The target visible light feature map is obtained by combining the one-dimensional visible light feature map. The visible light attention of infrared features to visible light features is calculated based on the infrared query vector, visible light key vector, and visible light value vector. The target infrared feature map is obtained by combining the one-dimensional infrared feature map. The infrared attention level is as follows: The visible light attention level is: ; where Q vis For visible light query vector, Q inf For infrared query vector, K vis For visible light bond vectors, K inf For infrared bond vectors, V vis For visible light value vector, V inf d is the infrared value vector; k The dimension of the key vector; After obtaining the fused feature map based on the target's visible light feature map and infrared feature map, it is input into the FPN module to construct the feature pyramid. At the same time, the modality consistency loss is introduced into the hybrid loss function for model training optimization. Specifically, when acquiring the fused feature map, corresponding learnable weight coefficients are introduced for the target visible light feature map and the target infrared feature map, respectively; the modal consistency loss is: The CTF() function represents the calculation function of the CTF module. One-dimensional visible light feature map A one-dimensional infrared feature map, i Indicates the position index number in the feature sequence; The learnable weight coefficients are adjusted by the hybrid loss function.
2. The method for training a wildlife detection model based on multimodal feature fusion according to claim 1, characterized in that, The acquisition of several pairs of paired visible light and infrared images includes: Acquire several raw visible light images and raw infrared images; the raw visible light images and raw infrared images correspond to the target animal categories in various scenes; The original visible light image is standardized to eliminate illumination differences, thereby obtaining an intermediate visible light image; the original infrared image is histogram equalization to enhance contrast, thereby obtaining an intermediate infrared image. Data enhancement processing is performed on both the intermediate visible light image and the intermediate infrared image to achieve multimodal pairing enhancement, resulting in several pairs of visible light images and infrared images; wherein, the data enhancement processing includes: random cropping, rotation and flipping, and noise addition.
3. The method for training a wildlife detection model based on multimodal feature fusion according to claim 1, characterized in that, The process of obtaining a fused feature map based on the target visible light feature map and the target infrared feature map, and then inputting it into the FPN module to construct a feature pyramid, includes: Channel compression is performed on the fused feature map based on convolutional layers to make the channel dimension of the fused feature map consistent with the channel dimension of the preset layer; The compressed fused feature map is input into the FPN module.
4. The method for training a wildlife detection model based on multimodal feature fusion according to claim 1, characterized in that, include: The hybrid loss function is: ； in, L box The bounding box regression loss is calculated using the CIoU() function, where λ1 is the bounding box regression loss coefficient. L obj To determine the existence loss of the target, binary cross-entropy is used to predict whether the target exists, where λ2 is the target existence loss coefficient. L cls For classification loss, cross-entropy is used to classify the target category, λ3 is the classification loss coefficient; λ4 is the modality consistency loss coefficient.
5. The method for training a wildlife detection model based on multimodal feature fusion according to claim 1, characterized in that, The process of optimizing model training includes: After freezing the weights of the shallow convolutional layers of the backbone networks in the first and second branch networks, iterative training begins. When the preset training progress value is reached, the weights of each frozen shallow convolutional layer are unfrozen sequentially according to the preset number of training rounds and added to the iterative training in sequence. During the iterative training process, the target visible light feature map and the target infrared feature map are updated through backpropagation, and corresponding learnable weight coefficients are introduced respectively.
6. A training system for a wildlife detection model based on multimodal feature fusion, characterized in that, include: The image acquisition module is used to acquire several pairs of visible light images and infrared images; The visible light image is input into the first branch network to obtain a visible light feature map output at a preset level, and the infrared image is input into the second branch network to obtain an infrared feature map output at a preset level; wherein, both the first branch network and the second branch network use CSP modules as backbone networks; The mapping generation module is used to flatten the visible light feature map and the infrared feature map respectively to obtain the corresponding one-dimensional visible light feature map and one-dimensional infrared feature map, and then perform feature mapping and query generation operations to obtain the visible light query vector, infrared key vector and infrared value vector corresponding to the one-dimensional visible light feature map, and the infrared query vector, visible light key vector and visible light value vector corresponding to the one-dimensional infrared feature map. The feature processing module is used to calculate the infrared attention of visible light features to infrared features based on visible light query vectors, infrared key vectors, and infrared value vectors, and to obtain the target visible light feature map by combining the one-dimensional visible light feature map; and to calculate the visible light attention of infrared features to visible light features based on infrared query vectors, visible light key vectors, and visible light value vectors, and to obtain the target infrared feature map by combining the one-dimensional infrared feature map. The infrared attention level is as follows: The visible light attention level is: ; where Q vis For visible light query vector, Q inf For infrared query vector, K vis For visible light bond vectors, K inf For infrared bond vectors, V vis For visible light value vector, V inf d is the infrared value vector; k The dimension of the key vector; The fusion training module is used to obtain a fused feature map based on the target visible light feature map and the target infrared feature map, and then input it into the FPN module to construct a feature pyramid. At the same time, modality consistency loss is introduced into the hybrid loss function for model training optimization. Specifically, when obtaining the fused feature map, corresponding learnable weight coefficients are introduced into the target visible light feature map and the target infrared feature map respectively. The modal consistency loss is as follows: The CTF() function represents the calculation function of the CTF module. One-dimensional visible light feature map A one-dimensional infrared feature map, i Indicates the position index number in the feature sequence; The learnable weight coefficients are adjusted by the hybrid loss function.
7. The wildlife detection model training system based on multimodal feature fusion according to claim 6, characterized in that, The image acquisition module includes: The acquisition unit is used to acquire several raw visible light images and raw infrared images; wherein the raw visible light images and raw infrared images correspond to the target animal categories in various scenarios; The first processing unit is used to standardize the original visible light image to eliminate illumination differences and then obtain an intermediate visible light image; and to perform histogram equalization on the original infrared image to enhance contrast and then obtain an intermediate infrared image. The second processing unit is used to perform data enhancement processing on both the intermediate visible light image and the intermediate infrared image to achieve multimodal pairing enhancement, thereby obtaining several pairs of paired visible light images and infrared images; wherein, the data enhancement processing includes: random cropping, rotation and flipping, and noise addition.
8. The wildlife detection model training system based on multimodal feature fusion according to claim 6, characterized in that, The fusion training module includes: A compression unit is used to perform channel compression on the fused feature map based on the convolutional layer so that the channel dimension of the fused feature map is consistent with the channel dimension of the preset layer; The input unit is used to input the compressed fused feature map into the FPN module.
9. An electronic device, characterized in that, It includes at least one processor coupled to a memory storing a computer program configured to be executed by the processor to perform the method of any one of claims 1-5.
10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which is used to be executed by a processor to implement the method of any one of claims 1-5.