Seal identification method in complex marine environment based on multi-modal fusion

Abstract

The present application belongs to the technical field of image recognition, and particularly relates to a seal identification method in complex marine environment based on multi-modal fusion, comprising: planning an unmanned aerial vehicle operation path based on historical seal activity data, constructing a seal visible light dataset and a thermal infrared dataset; constructing a lightweight seal detection network, training an optimal detection model of the two modalities of visible light and infrared based on the self-built dataset; inputting the synchronously collected seal visible light and infrared test set into the visible light and infrared detection model respectively to obtain the detection results of the model, performing multi-modal fusion on the visible light detection result and the thermal infrared detection result, determining the scene state through environment perception and adaptively adjusting the confidence of each modality to complete the fusion detection of the visible light and thermal infrared detection results; based on the multi-modal fusion detection method, outputting the number of seal population individuals and coordinate information to realize all-weather seal individual accurate counting and population distribution analysis.

Description

Technical Field

[0001] This invention relates to the field of spotted seal conservation and ecological monitoring technology, specifically to a spotted seal identification method based on multimodal fusion in complex marine environments, which enables efficient identification and accurate counting of spotted seals in different environmental scenarios. Background Technology

[0002] As a top indicator species, the spotted seal's population dynamics directly reflect the health of the ecosystem. Monitoring the spotted seal population provides a basis for decision-making regarding its conservation and ecological maintenance. Traditional methods of manual field observation and vessel patrols require a large investment of manpower and are prone to disturbing the spotted seal's normal habitat. Furthermore, they are easily limited by natural conditions such as weather, sea conditions, and sunlight, making it difficult to achieve long-term, continuous monitoring of the spotted seal population's size and activity patterns.

[0003] The development of UAV remote sensing technology has provided new solutions for spotted seal monitoring. Since spotted seals spend most of their lives in water, on coastal reefs, and on ice, they face challenges in complex marine environments such as waves, reflections, changes in lighting, and individual target occlusion. Furthermore, spotted seals often gather in large groups, closely packed together, sometimes even overlapping, forming large, clump-like masses in images. This results in small, dense, occluded, and morphologically variable image data, leading to low recognition efficiency and failing to meet the needs of accurate counting and individual analysis. Meanwhile, single-modal target detection technologies have limitations in complex environments. For example, in visible light scenarios, they perform excellently under good lighting and clear target conditions, but their performance drops sharply in glare, darkness, or under target camouflage. In thermal infrared scenarios, sensing the thermal radiation of spotted seals is unaffected by visible light conditions and can effectively detect targets with thermal characteristics, but it is susceptible to interference from environmental heat sources, leading to false detections. These limitations restrict the effectiveness of spotted seal monitoring and conservation efforts, necessitating the development of multimodal intelligent detection technologies to overcome existing difficulties and utilize intelligent monitoring systems to provide a scientific basis for protected area management and conservation work. Summary of the Invention

[0004] This invention aims to solve the above-mentioned technical problems and provides a method for identifying spotted seals in complex marine environments based on multimodal fusion. Through deep fusion of visible light and thermal infrared modes, it achieves all-weather, high-precision monitoring of spotted seals. Specifically, it includes the following steps: Based on historical activity data of spotted seals, the operation path of the UAV remote sensing platform equipped with visible light and thermal infrared cameras is planned, and the target area is divided into core acquisition area, secondary acquisition area and buffer zone. The UAV is controlled to simultaneously acquire visible light and thermal infrared images of spotted seal habitat during flight, thereby constructing visible light dataset and thermal infrared dataset. Boundary boxes were labeled for spotted seal targets in the visible light dataset and thermal infrared dataset, and data augmentation was performed on the labeled data. The augmented dataset was then divided into training set, validation set and test set. A global multi-scale fusion module is embedded in the backbone and neck network of the YOLOv11 network, and a lightweight spatial channel attention module is introduced into the neck network to construct a lightweight spotted seal detection network. The lightweight spotted seal detection network is trained based on visible light datasets and thermal infrared datasets to obtain the optimal detection model. A scene-adaptive parameter adjustment module was established. Based on the scene-adaptive parameter adjustment module, the brightness characteristics of the input image were analyzed to identify the ambient light conditions. An environment-adaptive decision-level multimodal fusion framework was constructed. The fusion detection of visible light and thermal infrared results was carried out based on comprehensive scene and image quality assessment, complementary saliency mask calibration, spatial clustering enhancement calibration, spatial consistency optimization of Markov random field, and intersection expansion fusion strategy. Based on the fusion detection results, the number and location information of individual spotted seals are output to achieve accurate individual counts and spatial distribution analysis of the spotted seal population.

[0005] Furthermore, the UAV remote sensing platform constructs a two-dimensional planar base map by collecting geographic information of spotted seal habitats, establishes spotted seal behavior patterns based on long-term observation data, and divides the area into a core collection zone, a secondary collection zone, and a buffer zone. In the core collection zone, the UAV's flight altitude is adjusted according to the size and sensitivity of the group, and the path of the UAV is designed around a large ring around the main inhabitant group. In the secondary collection zone, the UAV is controlled to use a bow-shaped path to collect spotted seal data, and the flight altitude is appropriately reduced while meeting the data collection accuracy requirements, while avoiding disturbing the scattered spotted seal individuals at too low an altitude. The buffer zone serves as a transition area for the UAV to approach and withdraw.

[0006] Furthermore, the training and test sets are subjected to data augmentation processing, including random horizontal and vertical flipping, rotation, scaling, and color transformation, wherein the color transformation includes adjusting hue, saturation, and brightness information.

[0007] Furthermore, the YOLOv11-SealLite spotted seal detection network uses YOLOv11 as the baseline network and embeds a GMSF global multi-scale fusion module and an LRSAM lightweight spatial channel attention module into its structure. The GMSF module is introduced into both the backbone and neck networks. This module, while maintaining low computational cost, significantly enhances the network's multi-scale feature representation ability and feature saliency response through multi-branch convolution and global context fusion, thereby improving the model's target detection performance in complex scenes. The introduction of the LRSAM module into the neck network effectively enhances the network's ability to represent the spatial and channel features of targets while maintaining lightweight design, thus improving the detection performance of small targets in complex backgrounds.

[0008] The spotted seal individual dataset was further divided into training, validation, and test sets in an 8:1:1 ratio. 80% of the data served as the training set, providing sample support for the model to learn individual features; 10% served as the validation set for model hyperparameter tuning and overfitting suppression; and the remaining 10% served as an independent test set to objectively evaluate the model's generalization ability. Visible light and thermal infrared spotted seal individual detection models were then trained based on the self-built dataset to obtain spotted seal detection results for the corresponding modalities.

[0009] Furthermore, a scene-adaptive parameter adjustment module is used to analyze the brightness characteristics of visible light and infrared images, identify environmental conditions such as low light, weak light, and normal light, and dynamically adjust the fusion weights accordingly to adapt to different marine environments. In the confidence fusion stage, modal complementarity analysis is performed based on the scene state determined by environmental perception. Simultaneously, an environment-adaptive decision-level multimodal fusion framework is constructed, integrating scene and image quality assessment, complementary saliency mask calibration, spatial clustering enhancement calibration, spatial consistency optimization based on Markov random fields, and intersection expansion fusion strategies to perform fusion detection of visible light and thermal infrared results. Finally, non-maximum suppression is applied to all fusion detection results to generate and save the final fusion detection result.

[0010] Furthermore, in low-light scenes, infrared detection is the primary method. Image enhancement processing is performed on the thermal infrared dataset. A gamma correction algorithm is used to perform gamma calibration enhancement processing on the target area image to improve the contrast between the hot target and the cold background. The enhanced image is then used for re-detection. A detection result fusion strategy is then applied to the infrared image detection results and the enhanced image.

[0011] The global multi-scale fusion module extracts local detail features, context features at different scales, and global information through four parallel branches. It employs depthwise separable convolutions to reduce computational complexity, utilizes SE attention mechanisms to automatically adjust channel weights based on feature content, and mitigates the gradient vanishing problem through residual connections. The global multi-scale fusion module is represented as follows: Let the input features be ,

[0012] The module introduces a lightweight Squeeze-and-Excitation attention mechanism.

[0013] The module uses residual connections.

[0014] in: This represents the channel-level concatenation of four branch features; Indicates fused convolution; For batch normalization; The SiLU activation function is used. This indicates a global average pooling operation. For the Sigmoid function, This indicates element-wise multiplication.

[0015] Furthermore, the lightweight spatial channel attention module captures inter-channel dependencies by constructing channel attention branches through global average pooling, channel compression expansion, and sigmoid activation; it concatenates the channel-dimensional average and maximum values ​​of the input features using the spatial attention branch and extracts spatial saliency features through 7×7 convolution; it multiplies the channel-attention-modulated features with spatial attention weights to achieve synergistic enhancement of both channel and spatial dimensions; and it adaptively controls the fusion ratio of the enhanced features and the original input features through learnable scalar parameters. The lightweight spatial channel attention module is represented as follows: Let the input features be The channel attention branch first obtains channel statistics through global average pooling:

[0016] Channel weight vectors are generated through two layers of 1×1 convolutions and non-linear activation. :

[0017] The first convolutional layer Information is compressed to extract compact features; the second convolutional layer Then restore the information and generate the attention score for each channel. Indicating Sigmoid activation, the channel-weighted feature is:

[0018] The spatial attention branch generates spatial weights by calculating the channel average map and the maximum map, concatenating them, and then performing a convolutional mapping. :

[0019] in This indicates a channel splicing operation. The operation is a 7×7 convolution, and the spatially weighted features are:

[0020] Finally, combined with residual scaling parameters The module output is: .

[0021] This invention proposes a multimodal fusion-based method for identifying spotted seals in complex marine environments. It utilizes a dual-lens camera (visible and thermal infrared) mounted on a drone to simultaneously acquire image data of spotted seals in both visible and infrared environments, constructing a high-quality visible and thermal infrared dataset. Two detection models are trained using the YOLOv11-SealLite model, and adaptive multimodal fusion detection is employed. Based on environmental perception, the scene state is determined, and the weights of each modality are adaptively adjusted. Adaptive cross-modal verification and decision integration are performed, enabling accurate identification and population counting of spotted seals in all weather conditions, providing reliable data support for habitat population health monitoring. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the spotted seal identification method in complex marine environments based on multimodal fusion of the present invention; Figure 2 This is a path planning diagram for the monitoring of the spotted seal unmanned aerial vehicle (UAV) of the present invention; Figure 3 This is an image showing the annotation results of spotted seal individuals in visible and infrared modes according to the present invention; Figure 4This is a diagram of the YOLOv11-SealLite model framework of the present invention; Figure 5 This invention provides a comparison of different evaluation index curves between YOLOv11 and YOLOv11-SealLite. Figure 6 This is the result of multimodal fusion detection under normal lighting conditions according to the present invention; Figure 7 This is the result of multimodal fusion detection in low-light scenes according to the present invention; Figure 8 This is the result of multimodal fusion detection in low-light scenes according to the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] like Figure 1 The method for identifying spotted seals in complex marine environments based on multimodal fusion comprises the following steps: First, we will build a drone remote sensing platform adapted to the marine environment to collect data on spotted seals in a non-invasive and remote manner.

[0027] like Figure 2The specific path planning map for spotted seal detection by UAVs is based on a hierarchical planning strategy, using a constructed environmental model and analyzed spotted seal behavior patterns. It divides the area into three levels: a core collection area, a secondary collection area, and a peripheral area. In the core collection area, flight altitude is increased according to the size of the spotted seal gatherings and the sensitive behavior periods to ensure that noise and shadow interference are below the warning threshold. A large circular flight path is used to cover the main groups, minimizing disturbance to the core habitat. In the secondary collection area, flight altitude can be appropriately reduced, and a bow-shaped flight path can be used to obtain information on individuals or small groups, balancing data accuracy and safety. A buffer zone is used for smooth takeoffs and landings and flight path changes, avoiding direct flight over the core habitat area and reducing impact on the core region.

[0028] The collected data will undergo data processing, including: We collected high-quality images and videos of spotted seals in various marine environments and with different morphologies. The synchronously collected spotted seal data was divided into visible light datasets and thermal infrared datasets to ensure that the visible light data and thermal infrared data corresponded consistently.

[0029] like Figure 3 The YOLOv11-SealLite model framework diagram is as follows: To address the challenges posed by the small scale, weak texture, and low signal-to-noise ratio of spotted seals in marine scenes, as well as the significant suppression of detection accuracy due to environmental factors such as strong reflections, dynamic wave backgrounds, and similarly colored reefs, a YOLOv11-based network was constructed. This network incorporates a Global Multi-Scale Fusion (GMSF) module and a Lightweight Spatial Channel Attention (LRSAM) module. The GMSF module is introduced into both the backbone and neck network. While maintaining low computational cost, this module significantly enhances the network's multi-scale feature representation capabilities and feature saliency response through multi-branch convolutions and global context fusion, thereby improving the model's target detection performance in complex scenes. The LRSAM module, introduced into the neck network, effectively enhances the network's ability to represent the spatial and channel features of targets while maintaining a lightweight design, thus improving the detection performance of small targets in complex backgrounds.

[0030] like Figure 4 The specific results of the bimodal individual identification and annotation of spotted seals and the model training are as follows: First, LabelImg was used to provide bounding box annotations for the visible light and thermal infrared spotted seal datasets. The datasets were then expanded by mirroring and flipping, rotating and scaling, cropping, and adding noise to increase the quantity and diversity of data, thereby providing more training samples and enhancing the model's generalization ability. The spotted seal individual dataset was divided into training, validation, and test sets in an 8:1:1 ratio. Visible light and thermal infrared spotted seal individual detection models were then trained based on these datasets, resulting in the corresponding visible light and thermal infrared models.

[0031] As shown in Table 1, YOLOv11-SealLite significantly reduces model complexity while maintaining high accuracy. Compared to YOLOv11, YOLOv11-SealLite achieves an mAP50-95 of 81.1% in visible light scenes, an improvement of 5.0%, and an mAP50-95 of 86.3% in infrared scenes, an improvement of 3.4%, with a model parameter count of 1.37M. Through multimodal fusion, the detection accuracy of spotted seals in complex marine environments is improved.

[0032] Table 1. Performance Comparison of YOLOv11-SealLite Algorithm in Visible and Infrared Environments.

[0033] For visible light and thermal infrared target detection, detection results and corresponding confidence boxes for two modalities are obtained respectively. The target detection method based on dynamic adaptive multimodal fusion is suitable for intelligent fusion of infrared and visible light images. This method performs multi-factor confidence analysis through modules such as multimodal target detection, scene brightness analysis, self-supervised modal weight adjustment, complementary region saliency analysis, and spatial distribution clustering analysis. The steps include: Multimodal target detection: Pre-trained deep learning object detection models were used to detect objects in visible light and infrared images, respectively, and the detection results were obtained. Each detection box in the detection result contains six parameters: {x1, y1, x2, y2, conf, cls}, where (x1, y1) and (x2, y2) are the coordinates of the top left and bottom right corners of the bounding box, respectively, conf is the confidence score, and cls is the class label.

[0034] Non-maximum suppression (NMS) is performed on the preliminary detection results to eliminate redundant detection boxes. NMS is used to calculate the intersection-over-union (IoU) ratio between detection boxes.

[0035] in, and This represents the region of two bounding boxes. Indicates the area of ​​the region.

[0036] Scene and image quality assessment and analysis: The scene-adaptive parameter adjustment module calculates the average brightness, brightness histogram distribution, and contrast of local bright and dark areas of the input visible light and infrared images, and identifies different imaging environments such as low light, weak light, and normal light based on a preset brightness classification model. The weighting function compresses color-related features into the (0,1) interval through a sigmoid mapping, thereby achieving smooth weighting of different color consistency levels and avoiding the discontinuity problem caused by hard thresholding.

[0037]

[0038]

[0039] in, Indicates input variables; The slope is a control parameter that determines the steepness of the function's transition. The center offset parameter corresponds to the inflection point of the curve. The visible light and infrared weights are applied using complementary normalization to ensure that the sum of the contributions from both modes is 1.

[0040] Complementary saliency mask calibration analysis: To quantify the complementary contributions between modes, a complementary saliency mask is defined. Perform a pixel-by-pixel XOR operation on the binarized detection response region of the two modalities:

[0041] in These are the detection areas for visible light and infrared light, respectively. This identifies regions that were successfully detected by only a single modality. If the bounding box of a target significantly overlaps with this masked region, it is considered to benefit from modal complementarity and is given a complementarity reward. :

[0042] In the formula, As a complementary reward coefficient, The bounding box is the target. This mechanism can effectively improve the confidence of detection results dominated by another modality when one modality fails due to environmental interference.

[0043] Spatial clustering enhancement calibration analysis: This method is used for spotted seal population detection, distinguishing between dense areas and isolated individuals, thus improving sensitivity and accuracy in areas with high seal population density. First, the center coordinates of all detection boxes are extracted to construct a set of target center points. Then, the DBSCAN density clustering algorithm is used to cluster these center points, setting appropriate neighborhood radii and minimum sample sizes. Detections belonging to the same cluster are marked as clustered region detections, while isolated points are marked as isolated detections.

[0044]

[0045]

[0046] in For the set of center points of the detection box, Represents the neighborhood radius. This represents the minimum number of samples. Detection of clustered regions is considered more reliable and will receive the confidence score for spatial clustering rewards in subsequent fusion. .

[0047]

[0048] Combining all the above calibration factors, the final formula for calculating the fusion confidence is obtained:

[0049] To improve the overall consistency and noise resistance of the detection results, a Markov random field is introduced at the end of the fusion process to globally optimize the target confidence. This method treats the detected targets as nodes in a graph and iteratively allows the confidence of spatially adjacent and semantically similar targets to influence each other, achieving consistency propagation. A weighted smoothing strategy is used for confidence updates.

[0050] in, This represents the current confidence value of the i-th detected target at the t-th iteration. For smoothing intensity coefficient, This represents the neighborhood weight. This represents the current confidence level of the j-th target, which is influenced by the spatial similarity and category consistency with target i, thus affecting the confidence level update of target i.

[0051] Neighborhood weighting takes into account spatial distance, class consistency, and confidence similarity.

[0052] In a Markov random field model, the weights determine the strength of the influence of confidence levels between adjacent targets in the control space. They determine the extent to which a target's confidence is affected by other targets in its neighborhood during global smoothing; a larger weight indicates a higher confidence level. confidence level The stronger the smoothing effect. Indicates the first , The center coordinates of the target detection boxes are given, and the spatial distance weighting term represents the Euclidean distance between two targets in the image. This indicates the degree of consistency between the confidence levels of two objectives. The decay rate of the confidence consistency term is controlled. This mechanism can enhance the confidence of real targets affected by occlusion or noise while suppressing spatially isolated false alarms, thereby improving the stability and consistency of detection at the global level.

[0053] Intersection expansion fusion analysis: Intersection-Expansion Fusion (IEF) strategy. This strategy aims to effectively fuse the core consensus region of bimodal information in a geometrically conservative manner, while suppressing noise introduced by inconsistencies, thereby achieving a balance between positioning stability and information complementarity.

[0054] The geometric center of the intersection of the two frames is chosen as the center point of the merged frame, and the calculation formula is as follows:

[0055] in, and These represent the coordinates of the upper left and lower right corners of the intersection region, respectively. Using their center as the center of the fusion box can effectively filter out the local positioning errors that may exist individually for each modality.

[0056] After determining the center point, it is necessary to estimate the target's width and height. First, a confidence-weighted average is performed on the bimodal scale information, and then a shrinkage factor related to the IoU value is introduced for calibration.

[0057]

[0058] in, By analyzing the original widths of the visible and infrared modes... Based on its detection confidence level Perform a weighted summation. This is a shrinkage factor that moderately shrinks the scale after weighted averaging to prevent excessive expansion of the fusion box due to inconsistencies in scale estimation between modalities. The value of this factor increases with increasing IoU, meaning that higher spatial consistency results in weaker shrinkage, thus adaptively balancing the conservatism of the fusion process with the degree of information utilization.

[0059] For independent detections that fail to match in visible light and infrared images, the confidence threshold is dynamically adjusted according to the modality weight; independent detections with dominant modalities are given a lower threshold, while those with inferior modalities are given a higher threshold; for independent detections that pass the threshold screening, complementary region enhancement and clustering enhancement strategies are also applied; the adjusted independent detections are added to the final fusion result set.

[0060] Finally, non-maximum suppression is performed again on all fused detection boxes to ensure that there are no highly overlapping detection boxes in the output. The entire process achieves a complete closed loop from scene analysis and confidence adjustment to result fusion, ensuring high-quality detection results under various lighting conditions.

[0061] Through the above steps, this method achieves intelligent fusion of infrared and visible light, enabling accurate target detection results under various lighting conditions. Finally, through multimodal confidence calibration, it achieves precise counting of spotted seals under different environmental conditions, effectively avoiding missed and false detections, thus enabling accurate health monitoring of the spotted seal population.

[0062] like Figure 5 The results shown are from a multimodal fusion detection under normal lighting conditions. By fusing visible light and infrared data, the average confidence level was improved under a single modality while maintaining a high detection count and high accuracy. This effectively combined the advantages of both modalities, achieving an average confidence level of 95.47% and an average accuracy of 97.56% after fusion, superior to single-modality testing. A total of 41 targets were detected, achieving more reliable and accurate detection results while retaining the advantages of each modality.

[0063] Table 2. Multimodal fusion detection results under normal lighting conditions

[0064] like Figure 6 The image shows the multimodal fusion detection results in a low-light scene: Under low-light conditions, this fusion method analyzes the degradation of images with extremely poor quality, where visible light detection is weak. In contrast, the infrared thermal imaging modality relies on the target's thermal radiation characteristics, and its imaging quality is less affected by ambient light, stably presenting the temperature difference contours between homeothermic animals like spotted seals and the background. Furthermore, a self-supervised mechanism significantly favors the more reliable infrared modality in the fusion weights, assigning it a higher fusion weight. The average accuracy after fusion reaches 93.33%, improving the detection efficiency of spotted seals in low-quality images and overcoming the limitations of a single sensor in extreme environments.

[0065] Table 3. Multimodal fusion detection results in low-light scenes

[0066] like Figure 7 The image shows the multimodal fusion detection results in a low-light scene: In nighttime environments, the visible light mode is almost ineffective, providing little to no visual information. In this case, infrared imaging, based on its thermal radiation imaging principle, can reliably detect spotted seals even in complete darkness. To further improve nighttime image quality, an infrared-dominated fusion strategy is employed in this mode, combined with a gamma correction enhancement algorithm to optimize the contrast of the infrared image, thereby improving target visibility and detection accuracy in nighttime and extreme environments. Specifically, gamma correction, combined with adaptive histogram equalization, achieves adaptive adjustment of image brightness. This processing method effectively suppresses and improves image quality, ensuring good visual performance under various lighting conditions. The gamma function is shown in the equation.

[0067]

[0068] In the formula, R represents the image after gamma correction, X is the original image with pixel values ​​ranging from 0 to 1, c is a constant parameter controlling brightness, and γ represents the gamma value parameter. In this process, the constant parameter c and the gamma value parameter γ play important roles, controlling brightness and adjusting image contrast and color saturation, respectively. After detection, a self-supervised weight fine-tuning mechanism dynamically allocates the dominant weights to the better-performing gamma-corrected infrared mode. Ultimately, while maintaining high detection accuracy, the average precision is improved to 97.41%, a 1.68% improvement compared to the infrared environment, resulting in more accurate detection in nighttime environments.

[0069] Table 4. Multimodal fusion detection results in low-light scenes

[0070] After the aforementioned multimodal fusion decision-making process, a final, validated target list is output. The precise number of spotted seal individuals in the survey is obtained by counting all "true positive" targets in the list. Combined with georeferenced information collected by the drone, the center point coordinates of each identified individual are obtained. Through multiple flight data analyses, the spatiotemporal distribution patterns, habitat preferences, and population density changes of spotted seal populations can be identified, providing crucial data support for assessing habitat health and developing scientific conservation strategies.

[0071] This invention provides a method for spotted seal identification in complex marine environments based on multimodal fusion. It employs two independently trained models for target detection and introduces a multimodal confidence fusion strategy. When a single modality dominates detection, the fusion decision is made based on the contextual quality of the other modality, such as the presence of glare in the visible light image or insufficient thermal contrast in the thermal infrared image. For low-confidence targets in the thermal infrared image at night, gamma-corrected infrared images are used for auxiliary verification to compensate for the limitations of visible light environments at night. This method effectively overcomes the limitations of single-modality detection in complex environments, significantly improves the detection accuracy of spotted seals, and achieves precise counting of seals. This enables more accurate monitoring of the health of the spotted seal population and provides a basis for decision-making in ecological maintenance.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying spotted seals in complex marine environments based on multimodal fusion, characterized in that, Includes the following steps: Based on historical activity data of spotted seals, the operation path of the UAV remote sensing platform equipped with visible light and thermal infrared cameras is planned, and the target area is divided into core acquisition area, secondary acquisition area and buffer zone. The UAV is controlled to simultaneously acquire visible light and thermal infrared images of spotted seal habitat during flight, thereby constructing visible light dataset and thermal infrared dataset. Boundary boxes were labeled for spotted seal targets in the visible light dataset and thermal infrared dataset, and data augmentation was performed on the labeled data. The augmented dataset was then divided into training set, validation set and test set. A global multi-scale fusion module is embedded in the backbone and neck network of the YOLOv11 network, and a lightweight spatial channel attention module is introduced into the neck network to construct a lightweight spotted seal detection network. The lightweight spotted seal detection network is trained based on visible light datasets and thermal infrared datasets to obtain the optimal detection model. A scene-adaptive parameter adjustment module was established. Based on the scene-adaptive parameter adjustment module, the brightness characteristics of the input image were analyzed to identify the ambient light conditions. An environment-adaptive decision-level multimodal fusion framework was constructed. The fusion detection of visible light and thermal infrared results was carried out based on comprehensive scene and image quality assessment, complementary saliency mask calibration, spatial clustering enhancement calibration, spatial consistency optimization of Markov random field, and intersection expansion fusion strategy. Based on the fusion detection results, the number and location information of individual spotted seals are output to achieve accurate individual counts and spatial distribution analysis of the spotted seal population.

2. The method for identifying spotted seals in complex marine environments based on multimodal fusion according to claim 1, characterized in that, Based on long-term observation data, a behavioral pattern of spotted seals was established, and core collection areas, secondary collection areas, and buffer zones were defined. In the core collection area, the flight altitude of the drone was adjusted according to the size and sensitivity of the group, and the drone's path was designed around a large circle around the main habitat group. In the secondary collection area, the drone was controlled to use a bow-shaped path to collect spotted seal data. The flight altitude was appropriately reduced while meeting the data collection accuracy requirements, and the altitude was kept low to avoid disturbing the scattered spotted seal individuals. The buffer zone served as a transition area for the drone to approach and withdraw.

3. The method for identifying spotted seals in complex marine environments based on multimodal fusion according to claim 1, characterized in that: The visible light dataset and thermal infrared dataset are augmented with random horizontal and vertical flipping, rotation, scaling, and color transformation. The color transformation includes adjusting hue, saturation, and brightness information. Individual datasets of spotted seals in visible light and thermal infrared are obtained through data augmentation.

4. The method for identifying spotted seals in complex marine environments based on multimodal fusion according to claim 1, characterized in that: The global multi-scale fusion module extracts local detail features, context features at different scales, and global information through four parallel branches. It employs depthwise separable convolutions to reduce computational complexity, utilizes SE attention mechanisms to automatically adjust channel weights based on feature content, and mitigates the gradient vanishing problem through residual connections. The global multi-scale fusion module is represented as follows: Let the input features be , The module introduces a lightweight Squeeze-and-Excitation attention mechanism. The module uses residual connections. in: This represents the channel-level concatenation of four branch features; Indicates fused convolution; For batch normalization; The SiLU activation function is used. This indicates a global average pooling operation. For the Sigmoid function, This indicates element-wise multiplication.

5. The method for identifying spotted seals in complex marine environments based on multimodal fusion according to claim 1, characterized in that: The lightweight spatial channel attention module captures inter-channel dependencies by constructing channel attention branches through global average pooling, channel compression expansion, and sigmoid activation. It then uses the spatial attention branch to concatenate the channel-dimensional average and maximum values ​​of the input features and extracts spatial saliency features through 7×7 convolution. The channel-attention-modulated features are multiplied by spatial attention weights to achieve synergistic enhancement of both channel and spatial dimensions. A learnable scalar parameter adaptively controls the fusion ratio between the enhanced features and the original input features. The lightweight spatial channel attention module is represented as follows: Let the input features be The channel attention branch first obtains channel statistics through global average pooling: Channel weight vectors are generated through two layers of 1×1 convolutions and non-linear activation. : The first convolutional layer Information is compressed to extract concise features; Second convolutional layer Then restore the information and generate the attention score for each channel. Indicating Sigmoid activation, the channel-weighted feature is: The spatial attention branch generates spatial weights by calculating the channel average map and the maximum map, concatenating them, and then performing a convolutional mapping. : in This indicates a channel splicing operation. The operation is a 7×7 convolution, and the spatially weighted features are: Finally, combined with residual scaling parameters The module output is: 。 6. The method for identifying spotted seals in complex marine environments based on multimodal fusion according to claim 1, characterized in that: Using the scene adaptive parameter adjustment module, the brightness characteristics of visible light and infrared images are analyzed to identify environmental conditions such as dark light, weak light and normal light, and the fusion weights are dynamically adjusted accordingly to adapt to the weight allocation in different marine environments. In the confidence fusion stage, modal complementarity analysis is performed based on the scene state determined by environmental perception. The fusion detection of visible light and thermal infrared results is carried out by integrating scene and image quality assessment, complementary saliency mask calibration, spatial clustering enhancement calibration, spatial consistency optimization based on Markov random field, and intersection expansion fusion strategy. Non-maximum suppression is performed on all fusion detection results to generate and save the final fusion detection results.

7. The method for identifying spotted seals in complex marine environments based on multimodal fusion according to claim 1, characterized in that, In low-light scenarios, infrared detection is the primary method. Image enhancement processing is performed on the thermal infrared dataset. A gamma correction algorithm is used to perform gamma calibration enhancement processing on the target area image to improve the contrast between the spotted seal and the environment. The enhanced image is then used for re-detection. Finally, a detection fusion strategy is employed to combine the infrared image detection results with the enhanced image.

Images