All-weather cross-modal ship detection method and device based on YOLO11 and electronic equipment
By combining the YOLO11 network with thermal imaging and visible light images, a cross-modal ship detection method was developed, which solved the problem of poor ship detection performance in extreme environments. This method enables high-precision and robust ship detection in water conservancy hubs around the clock, with significantly improved recognition performance, especially in complex environments such as nighttime, rainy days, and foggy days.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGDONG HUANAN HYDROPOWER HIGH-TECH DEV CO LTD
- Filing Date
- 2026-04-17
- Publication Date
- 2026-06-05
AI Technical Summary
Existing ship inspection methods perform poorly in extreme environments, lack cross-modal training data, and have low accuracy in detecting small-scale ships, failing to meet the needs of all-weather intelligent safety management for water conservancy projects.
A YOLO11 network is used to construct an all-weather, cross-modal ship detection method. Visible light and thermal imaging images are acquired, and physically consistent images are generated by fine-tuning the LoRA module in the Stable Diffusion backbone network. Image enhancement and annotation are performed to construct a cross-modal dataset. A cross-modal attention module is embedded in the YOLO11 backbone network, and a composite loss function is used to optimize the model to achieve multi-scale ship target detection.
It achieves high-precision and robust ship detection in complex environments such as fog, rain, and snow, as well as in day-night cycles, significantly improving the recall rate and identification accuracy of small-scale ships and reducing system maintenance costs.
Smart Images
Figure CN122156597A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of smart water conservancy emergency monitoring and early warning technology, specifically to an all-weather cross-modal ship detection method, device, and electronic equipment based on YOLO11. Background Technology
[0002] As infrastructure projects integrating flood control, navigation, power generation, and water resource allocation, the dam area and upstream and downstream waters of water conservancy projects are key safety control areas. However, due to surrounding human activities and regulatory blind spots, vessel traffic in the reservoir waters is frequent, especially illegal navigation at night and unauthorized entry into prohibited navigation areas, which seriously threaten the operational safety of the project, public safety, and water management order. Currently, relevant management departments mainly rely on traditional methods such as manual patrols and on-site verbal warnings for control. These methods are not only costly in terms of manpower and slow in response, but also fail to achieve all-weather, full-coverage dynamic monitoring, and can no longer meet the current needs of intelligent and refined safety management of water conservancy projects.
[0003] At the technical level, existing automatic ship identification methods are mostly based on visible light video surveillance and target detection algorithms. However, these methods suffer from significant image quality degradation in low-light, strong glare, or nighttime scenes, leading to blurred target features and insufficient detection confidence. In adverse weather conditions such as fog, rain, and snow, image contrast and visibility further deteriorate, resulting in significantly higher false positive and false negative rates. To improve environmental adaptability, recent studies have introduced cross-modal fusion strategies of thermal imaging and visible light, utilizing thermal radiation information to compensate for the perception loss of visible light under limited lighting conditions. However, these methods still have significant limitations: on the one hand, synchronously acquired visible light-thermal imaging paired data in real-world scenarios is scarce, and cross-modal sample annotation relies heavily on manual intervention, resulting in high costs and long cycles, making it difficult to support effective training of deep neural networks; on the other hand, existing fusion mechanisms mostly employ shallow strategies such as channel splicing or static weighting, failing to fully explore the deep correlation between the two modalities in terms of semantic consistency and structural complementarity, leading to insufficient fusion feature representation capabilities and limiting the detection accuracy and robustness of the model in complex and variable aquatic environments. Summary of the Invention
[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide an all-weather, cross-modal ship detection method, device, and electronic equipment based on YOLO11. This invention addresses the technical problems of existing ship detection methods, such as poor detection performance in extreme environments, scarcity of cross-modal training data, and low detection accuracy of small-scale ships. It enables all-weather, high-precision, and highly robust intelligent detection of ships in water conservancy hubs, meeting the needs of intelligent safety management in water conservancy hubs.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] In a first aspect, the present invention provides an all-weather, cross-modal ship inspection method based on YOLOv11, comprising the following steps:
[0007] S1. Collect all-weather visible light images of water conservancy hubs and paired images of thermal imaging ships. After annotation, construct a training dataset for the generative model. Freeze the parameters of the Stable Diffusion backbone network and fine-tune the LoRA module to obtain a generative model that can controllably generate physically consistent visible light-thermal imaging ship paired images. Use this generative model to generate augmented images. Merge the augmented images with the original paired images to form a cross-modal ship dataset.
[0008] S2. Perform image augmentation on the cross-modal ship dataset and annotate to generate ship bounding boxes and category labels;
[0009] S3. The enhanced cross-modal ship dataset is mapped to feature embeddings with the same spatial resolution and number of channels by an independent modal encoder. After concatenation along the channel dimension, the embeddings are input into the YOLO11 backbone network. A cross-modal attention module is embedded after the first convolutional downsampling layer of the YOLO11 backbone network to achieve channel-level semantic guidance fusion of thermal imaging modality and visible light modality. After generating fused features, the YOLO11 backbone network is used to generate multi-scale feature maps by downsampling and upsampling at each level. These features are then input into the YOLO11 native detection head to complete ship target detection, thus constructing an end-to-end cross-modal ship detection model.
[0010] S4. For multi-scale ship targets, a composite loss function consisting of zoom classification loss, distributed focus regression loss, and task alignment intersection-over-union loss is used to train and optimize the cross-modal ship detection model. The zoom classification loss is used to differentiate between positive and negative samples to improve the recall rate of small-scale ships. The distributed focus regression loss is used to enhance the robustness of ship target localization. The task alignment intersection-over-union loss is used to suppress the interference of low-quality prediction boxes on model optimization.
[0011] S5. Deploy the optimized cross-modal ship detection model to the field to detect ships in the target area.
[0012] As a preferred technical solution, in step S1, the all-weather visible light image and thermal imaging image are used to characterize the visual appearance and thermal radiation characteristics of ships in low-light, strong glare, limited visibility or complex background scenarios.
[0013] The fine-tuning of the Stable Diffusion model using LoRA technology specifically includes: freezing all parameters of the Stable Diffusion backbone network and training only on the modules embedded in LoRA to preserve the prior knowledge of the pre-trained model; and ensuring that the generated augmented image pairs are physically consistent with the real monitoring scene in terms of scale, pose, and water background.
[0014] As a preferred technical solution, step S2, the preprocessing of the cross-modal ship dataset includes:
[0015] Image enhancement is performed on the acquired visible light-thermal imaging paired images, including left-right flipping, Gaussian noise injection, and mosaic enhancement;
[0016] A semi-automatic annotation method is adopted, combining preliminary screening by a pre-trained detection model with manual verification to generate ship bounding boxes and category labels;
[0017] The labeled dataset is divided into training, validation, and test sets according to a set ratio.
[0018] As a preferred technical solution, in step S3, the cross-modal feature fusion mechanism adopts a single-backbone cross-modal collaborative sensing structure, specifically including:
[0019] The RGB visible light image and the single-channel thermal imaging image are initially feature mapped by independent modal encoders to obtain visible light embedding and thermal imaging embedding with the same spatial resolution and number of channels.
[0020] The visible light embedding and the thermal imaging embedding are spliced along the channel dimension to form a joint feature tensor;
[0021] The joint feature tensor is input into the YOLO11 backbone network, and a cross-modal attention module is embedded after the first convolutional downsampling layer of the YOLO11 backbone network to generate the first-stage fusion feature.
[0022] Using the first stage fusion features as input, the subsequent layers in the YOLO11 backbone network are downsampled and upsampled step by step to generate a multi-scale feature map, and the multi-scale feature map is directly input into the YOLO11 native detection head for ship target detection.
[0023] The entire process of feature fusion, scale transfer, and target detection is completed within the YOLO11 single-backbone architecture.
[0024] As a preferred technical solution, the cross-modal attention module is configured to perform the following operations:
[0025] The joint feature tensor input to this module is split into visible light features and thermal imaging features along the channel dimension;
[0026] The visible light features are applied to a self-attention mechanism: channel statistics are obtained through global average pooling, channel attention weights are generated through two fully connected layers and a nonlinear activation function, and then the weights are multiplied with the visible light features channel by channel.
[0027] Based on the thermal imaging features, a cross-modal guidance signal is constructed: first, the thermal imaging features are subjected to global average pooling to obtain a channel-level response vector; then, a channel modulation weight vector is generated through a fully connected layer. This weight vector is used to weight the visible light features after self-attention enhancement channel by channel to realize cross-modal semantic guidance of the thermal imaging mode to the visible light mode.
[0028] The visible light features enhanced by the above self-attention and thermal imaging-guided modulation are re-stitched with the original thermal imaging features according to the channel dimension to generate the first-stage fusion features.
[0029] As a preferred technical solution, the zoom classification loss employs differentiated calculation methods for positive and negative samples. For positive samples, the intersection-union ratio of the predicted bounding box and the ground truth bounding box is used as a quality label, and logarithmic calculation is performed with the predicted classification confidence. For negative samples, a focus parameter greater than 0 is introduced to exponentially process the classification confidence before combining it with the logarithmic term for calculation. The zoom classification loss improves the recall rate of small-scale ships by assigning higher learning weights to high-quality positive samples.
[0030] As a preferred technical solution, the distributed focus regression loss is calculated for each of the four coordinate dimensions of the ship target bounding box. First, a soft label distribution is generated based on the true coordinate values. Then, the cross-entropy loss is calculated and summed between the coordinate probability distribution output by the model and the soft label distribution. The distributed focus regression loss transforms continuous coordinate regression into discrete probability modeling, which significantly enhances the localization robustness in low-resolution thermal imaging images.
[0031] As a preferred technical solution, the task alignment cross-union ratio loss is calculated only for all positive sample predicted boxes. First, the task alignment metric is calculated by combining the classification confidence of the predicted box, the cross-union ratio between the predicted box and the ground truth box, and two alignment weights greater than 0. Then, a decay exponent greater than or equal to 0 is introduced, and the loss calculation formula is constructed by combining the cross-union ratio. The task alignment cross-union ratio loss applies strong constraints only to predicted boxes with high alignment, so as to suppress optimization interference from low-quality or false-detection boxes.
[0032] Secondly, the present invention provides an all-weather cross-modal ship detection device based on YOLO11, which is applied to the all-weather cross-modal ship detection method based on YOLO11, including an image acquisition module, an image enhancement module, a model building module, a model training module, and a model deployment module;
[0033] The image acquisition module is used to acquire all-weather visible light images of water conservancy hubs and paired images of thermal imaging ships. After annotation, a training dataset for the generative model is constructed. The parameters of the Stable Diffusion backbone network are frozen and only the LoRA module is fine-tuned to obtain a generative model that can controllably generate physically consistent visible light-thermal imaging ship paired images. The generative model is used to generate extended images, and the extended images are merged with the original paired images to form a cross-modal ship dataset.
[0034] The image enhancement module is used to enhance the images of the cross-modal ship dataset and generate ship bounding boxes and category labels.
[0035] The model building module is used to map the enhanced cross-modal ship dataset into feature embeddings with the same spatial resolution and number of channels through an independent modal encoder, and then concatenate them along the channel dimension before inputting them into the YOLO11 backbone network. After the first convolutional downsampling layer of the YOLO11 backbone network, a cross-modal attention module is embedded to achieve channel-level semantic guidance fusion of thermal imaging modality and visible light modality. After generating fused features, they are downsampled and upsampled step by step by the YOLO11 backbone network to generate multi-scale feature maps, which are then input into the YOLO11 native detection head to complete ship target detection, thus constructing an end-to-end cross-modal ship detection model.
[0036] The model training module is used to train and optimize the cross-modal ship detection model for multi-scale ship targets using a composite loss function consisting of zoom classification loss, distributed focus regression loss, and task alignment intersection-over-union loss. The zoom classification loss is used to differentiate between positive and negative samples to improve the recall rate of small-scale ships, the distributed focus regression loss is used to enhance the robustness of ship target localization, and the task alignment intersection-over-union loss is used to suppress the interference of low-quality prediction boxes on model optimization.
[0037] The model deployment module is used to deploy the optimized cross-modal ship detection model to the field to detect ships in the target area.
[0038] Thirdly, the present invention provides an electronic device, the electronic device comprising:
[0039] At least one processor; and,
[0040] A memory communicatively connected to the at least one processor; wherein,
[0041] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the YOLO11-based all-weather cross-modal ship detection method.
[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0043] (1) This invention acquires paired visible light-thermal imaging images of ships at water conservancy hubs, combines this with fine-tuning the LoRA module of the StableDiffusion backbone network to generate physically consistent paired images, and implements left-right flipping, Gaussian noise, and mosaic data enhancement to construct a highly robust training dataset covering all weather conditions and multiple scenarios. This data strategy effectively solves the problem of insufficient ship feature discrimination in low-resolution images, ensures the model's stable recognition ability under complex meteorological conditions such as fog, rain, and snow, as well as day-night alternation scenarios, and significantly improves the model's generalization performance and anti-interference ability in real marine monitoring environments.
[0044] (2) This invention constructs a single-backbone cross-modal structure based on YOLO11. After downsampling in the first layer of the backbone network, a cross-modal attention module is embedded to achieve channel-level guided fusion of visible light in thermal imaging. This mechanism effectively alleviates the recognition bottleneck caused by changes in illumination, strong glare, occlusion, and severe weather by complementing the visible light texture details and thermal imaging thermal signal characteristics, and significantly improves the accuracy of ship recognition. In particular, the recognition performance is significantly optimized in complex environments such as night, rain, and fog, overcoming the limitations of single-modal recognition.
[0045] (3) This invention is designed for small-scale ship detection. It innovatively uses a composite loss function of VFL, DFL and TA-IoU to optimize the feature extraction network, which significantly enhances the model's feature response capability to small targets. This optimization effectively reduces the false negative rate of small-scale ships, accurately improves the detection accuracy of small targets, and fills the technical gap in the field of small ship identification in the existing technology.
[0046] (4) Based on feedback from low-confidence, missed detection, and false detection samples in the field, this invention expands the data and fine-tunes the model, establishing a closed-loop iterative optimization mechanism. This mechanism enables the model to dynamically adapt to the deployment environment, continuously improves the recognition accuracy during long-term operation, significantly reduces maintenance costs, and ensures the high reliability and continuous evolution capability of the system in long-term practical applications. Attached Figure Description
[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a flowchart of an embodiment of the all-weather cross-modal ship inspection method based on YOLO11 according to an embodiment of the present invention;
[0049] Figure 2 This is a flowchart of the Stable Diffusion network of the present invention, as shown in an embodiment of the invention.
[0050] Figure 3 This is a diagram of the YOLO11 cross-modal network architecture according to an embodiment of the present invention;
[0051] Figure 4 This is a flowchart illustrating the cross-modal attention process according to an embodiment of the present invention.
[0052] Figure 5 This is a block diagram of an all-weather, cross-modal ship inspection device based on YOLO11, according to an embodiment of the present invention.
[0053] Figure 6 This is a block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0054] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.
[0055] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.
[0056] like Figure 1 As shown in the figure, this embodiment provides an all-weather, cross-modal ship inspection method based on YOLO11, which includes the following steps:
[0057] S1: Data acquisition and cross-modal image generation;
[0058] First, visible light images and thermal images were collected in the water conservancy hub area, covering various extreme environmental conditions, including low light, strong glare, fog, rain, snow and other scenes with limited visibility or complex backgrounds. The ship position, scale, attitude and water background information in each image were manually annotated to construct an annotated dataset for generating model training.
[0059] Furthermore, to expand the training samples, this embodiment employs a Stable Diffusion-based image generation model combined with LoRA fine-tuning technology to achieve controllable generation of visible light and thermal imaging ship images. The specific implementation steps are as follows:
[0060] S11. Based on the above annotations, construct the training dataset for the visible light-thermal imaging paired images to generate the model;
[0061] S12. Freeze all parameters of the Stable Diffusion backbone network and train only the LoRA module;
[0062] S13. The fine-tuned generation model generates visible light-thermal imaging paired images of ships that are physically consistent with the real monitoring scene in terms of scale, attitude and water background based on text prompts.
[0063] S14. Merge the generated paired images with the original acquired paired images to form a cross-modal ship dataset, which will be used for training the YOLO11 detection model.
[0064] Furthermore, the controllable generation of visible light and thermal imaging ship images is achieved using... Figure 2 The implementation of the StableDiffusion network process is shown below:
[0065] The input text prompts and Gaussian noise vectors are processed by a frozen Stable Diffusion backbone network, and the feature generation process is dynamically adjusted by a fine-tuned LoRA module. The output is a spatially registered visible light-thermal imaging paired image of a ship. The VAE encoder decodes the generated result into a visible light image and a thermal imaging image, which maintain geometric alignment and semantic consistency to ensure the authenticity and usability of cross-modal data.
[0066] S2. Dataset preprocessing and partitioning;
[0067] The collected and generated cross-modal ship dataset is preprocessed, including image enhancement, semi-automatic annotation, and dataset partitioning, as detailed below:
[0068] S21. Perform image enhancement on the visible light-thermal imaging paired image. The enhancement operation includes flipping the image horizontally and horizontally, injecting Gaussian noise with a standard deviation of 0.01-0.05 and mosaic enhancement to simulate various interference factors that may occur in actual monitoring.
[0069] S22. Use the software X-AnyLabeling to automatically screen the acquired images in conjunction with the initially trained detection model, and then manually verify and correct them to generate ship bounding boxes and category labels. The bounding boxes guide the output of prediction boxes during model training, and the category labels guide the classification. They are also used to calculate the accuracy and recall on the validation and test sets to quantitatively evaluate the detection performance of the model.
[0070] The vessel categories labeled in this embodiment include: fishing vessels, sand carriers, mission vessels, passenger ships, and cargo ships. The total dataset contains 12,000 images, including 9,720 images in the training set, 1,080 images in the validation set, and 1,200 images in the test set. The training and validation sets comprise 90% (10,800 images), and the test set comprises 10% (1,200 images). Within the training and validation sets, images are further allocated in a 9:1 ratio.
[0071] S3. Construct an end-to-end cross-modal ship detection model; This embodiment is based on the YOLO11 model architecture and uses a single-backbone cross-modal collaborative sensing structure to construct an end-to-end cross-modal ship detection model. The specific implementation process is as follows:
[0072] S31. Initial feature mapping: Input the RGB visible light image and the single-channel thermal imaging image into independent modal encoders for initial feature mapping to obtain visible light embedding and thermal imaging embedding with the same spatial resolution and number of channels.
[0073] S32, Feature stitching: The visible light embedding and the thermal imaging embedding are stitched together along the channel dimension to form a joint feature tensor;
[0074] S33. Cross-modal feature fusion: The joint feature tensor is input into the YOLO11 backbone network, and a cross-modal attention module is embedded after the first convolutional downsampling layer of the YOLO11 backbone network (see structure). Figure 3 The cross-modal attention module performs the following operations, the process of which is described in [link to flowchart]. Figure 4S331. The joint feature tensor is split into visible light features and thermal imaging features along the channel dimension according to a preset partitioning method. The two features have the same spatial size and their respective set number of channels. S332. Channel self-attention operation is performed on the visible light features: channel statistics are obtained through global average pooling, channel attention weights are generated through two fully connected layers and a nonlinear activation function, and multiplied with the original visible light features channel by channel. S333. The thermal imaging features are obtained by global average pooling to obtain a channel-level response vector. A channel modulation weight vector is generated through a fully connected layer with an output dimension equal to the number of visible light feature channels, and the self-attention-enhanced visible light features are weighted channel by channel. S334. The weighted visible light features and the original thermal imaging features are re-stitched along the channel dimension to output the first-stage fusion features.
[0075] S34. Multi-scale feature extraction and detection: Using the fused features from the first stage as input, the YOLO11 backbone network performs downsampling and upsampling operations at subsequent layers to generate multi-scale feature maps, which are then directly fed into the YOLO11 native detection head to complete ship target detection. The entire feature fusion, scale transfer, and target detection process is implemented within the YOLO11 single backbone architecture, without the need for additional feature fusion modules or post-processing steps.
[0076] S4. Optimize the model using an improved loss function;
[0077] For multi-scale ship targets, especially small-scale ships with an area ratio less than a preset threshold, an improved loss function is used to improve recognition accuracy. This embodiment adopts a composite loss function of task-aligned perception, which consists of three parts: Varifocal Loss (VFL), Distribution Focal Loss (DFL), and Task-Aligned IoU Loss (TA-IoU).
[0078] The total loss expression is:
[0079]
[0080] in, These are the preset non-negative weighting coefficients.
[0081] The specific definitions of each loss function are as follows:
[0082] Defined as:
[0083] in, The classification confidence score predicted by the model. The intersection-union ratio (IoU) of the predicted bounding boxes and the ground truth bounding boxes serves as the quality label for positive samples. This is a focused parameter. The loss improves the recall rate of small-scale ships by assigning higher learning weights to high-quality positive samples.
[0084] Defined as:
[0085] in, This represents the four coordinate dimensions of the bounding box. This is a soft label distribution generated based on the true values (obtained through neighboring interval interpolation). This represents the probability distribution output by the model. Represents cross-entropy loss, The number of discrete intervals is denoted by . This loss transforms continuous coordinate regression into discrete probability modeling, significantly enhancing the robustness of localization in low-resolution thermal imaging images.
[0086] The definition is:
[0087] in, For the set of all positive sample predicted boxes, For the first The intersection-union ratio of each predicted bounding box to its corresponding ground truth bounding box. For task alignment metrics, Classify its confidence level. To align weights, The decay exponent is used. This loss only applies to high alignment (i.e., high classification score and high accuracy). The predicted bounding boxes are subject to strong constraints, which effectively suppress optimization interference from low-quality or false-detection boxes.
[0088] S5. Model Deployment and Iterative Optimization;
[0089] The trained cross-modal ship detection model was deployed at the water conservancy project site to perform real-time inference on the actual monitoring video stream.
[0090] The system collects predictions below 0.3 confidence levels, missed detections, and false positives from the model during field testing, and generates new labeled data based on manual review. This new labeled data is added to the original cross-modal dataset, and the model is incrementally trained or fine-tuned based on this training set. This "deployment-feedback-optimization" process is repeated until the model's ship detection accuracy and robustness in the field testing meet the preset performance thresholds.
[0091] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.
[0092] Based on the same concept as the YOLO11-based all-weather cross-modal ship inspection method in the above embodiments, this invention also provides a YOLO11-based all-weather cross-modal ship inspection device. This system can be used to execute the aforementioned YOLO11-based all-weather cross-modal ship inspection method. For ease of explanation, the structural schematic diagram of the YOLO11-based all-weather cross-modal ship inspection device embodiment only shows the parts related to the embodiments of this invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0093] Please see Figure 5 In another embodiment of this application, an all-weather cross-modal ship inspection device 100 based on YOLO11 is provided. The system includes an image acquisition module 101, an image enhancement module 102, a model building module 103, a model training module 104, and a model deployment module 105.
[0094] The image acquisition module 101 is used to acquire all-weather visible light images of water conservancy hubs and paired images of thermal imaging ships. After annotation, a training dataset for the generative model is constructed. The parameters of the Stable Diffusion backbone network are frozen and only the LoRA module is fine-tuned to obtain a generative model that can controllably generate physically consistent visible light-thermal imaging ship paired images. The generative model is used to generate extended images, and the extended images are merged with the original paired images to form a cross-modal ship dataset.
[0095] The image enhancement module 102 is used to enhance the images of the cross-modal ship dataset and generate ship bounding boxes and category labels.
[0096] The model building module 103 is used to map the enhanced cross-modal ship dataset into feature embeddings with the same spatial resolution and number of channels through an independent modal encoder, and then concatenate them along the channel dimension before inputting them into the YOLO11 backbone network. After the first convolutional downsampling layer of the YOLO11 backbone network, a cross-modal attention module is embedded to realize channel-level semantic guidance fusion of thermal imaging modality and visible light modality. After generating fused features, they are downsampled and upsampled step by step by the YOLO11 backbone network to generate multi-scale feature maps, which are then input into the YOLO11 native detection head to complete ship target detection, thus constructing an end-to-end cross-modal ship detection model.
[0097] The model training module 104 is used to train and optimize the cross-modal ship detection model for multi-scale ship targets using a composite loss function consisting of zoom classification loss, distributed focus regression loss, and task alignment intersection-over-union loss. The zoom classification loss is used to differentiate between positive and negative samples to improve the recall rate of small-scale ships, the distributed focus regression loss is used to enhance the robustness of ship target localization, and the task alignment intersection-over-union loss is used to suppress the interference of low-quality prediction boxes on model optimization.
[0098] The model deployment module 105 is used to deploy the optimized cross-modal ship detection model to the field to detect ships in the target area.
[0099] It should be noted that the YOLO11-based all-weather cross-modal ship inspection device of the present invention corresponds one-to-one with the YOLO11-based all-weather cross-modal ship inspection method of the present invention. The technical features and beneficial effects described in the embodiments of the YOLO11-based all-weather cross-modal ship inspection method described above are applicable to the embodiments of the YOLO11-based all-weather cross-modal ship inspection. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.
[0100] Furthermore, in the embodiments of the YOLO11-based all-weather cross-modal ship inspection device described above, the logical division of each program module is merely an example. In actual applications, the functions described above can be assigned to different program modules as needed, for example, for the sake of hardware configuration requirements or software implementation convenience. That is, the internal structure of the YOLO11-based all-weather cross-modal ship inspection device can be divided into different program modules to complete all or part of the functions described above.
[0101] Please see Figure 6 In one embodiment, an electronic device is provided for implementing an all-weather cross-modal ship detection method based on YOLO11. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as an all-weather cross-modal ship detection program 203 based on YOLO11.
[0102] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal and external storage units of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the YOLO 11-based all-weather cross-modal ship detection program 203, but also to temporarily store data that has been output or will be output.
[0103] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.
[0104] Figure 6 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 6 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0105] The YOLO 11-based all-weather, multi-modal ship detection program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:
[0106] S1. Collect all-weather visible light images of water conservancy hubs and paired images of thermal imaging ships. After annotation, construct a training dataset for the generative model. Freeze the parameters of the Stable Diffusion backbone network and fine-tune the LoRA module to obtain a generative model that can controllably generate physically consistent visible light-thermal imaging ship paired images. Use this generative model to generate augmented images. Merge the augmented images with the original paired images to form a cross-modal ship dataset.
[0107] S2. Perform image augmentation on the cross-modal ship dataset and annotate to generate ship bounding boxes and category labels;
[0108] S3. The enhanced cross-modal ship dataset is mapped to feature embeddings with the same spatial resolution and number of channels by an independent modal encoder. After concatenation along the channel dimension, the embeddings are input into the YOLO11 backbone network. A cross-modal attention module is embedded after the first convolutional downsampling layer of the YOLO11 backbone network to achieve channel-level semantic guidance fusion of thermal imaging modality and visible light modality. After generating fused features, the YOLO11 backbone network is used to generate multi-scale feature maps by downsampling and upsampling at each level. These features are then input into the YOLO11 native detection head to complete ship target detection, thus constructing an end-to-end cross-modal ship detection model.
[0109] S4. For multi-scale ship targets, a composite loss function consisting of zoom classification loss, distributed focus regression loss, and task alignment intersection-over-union loss is used to train and optimize the cross-modal ship detection model. The zoom classification loss is used to differentiate between positive and negative samples to improve the recall rate of small-scale ships. The distributed focus regression loss is used to enhance the robustness of ship target localization. The task alignment intersection-over-union loss is used to suppress the interference of low-quality prediction boxes on model optimization.
[0110] S5. Deploy the optimized cross-modal ship detection model to the field to detect ships in the target area.
[0111] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).
[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0113] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0114] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A method for all-weather, cross-modal ship inspection based on YOLOv11, characterized in that, Includes the following steps: S1. Collect all-weather visible light images of water conservancy hubs and paired images of thermal imaging ships. After annotation, construct a training dataset for the generative model. Freeze the parameters of the Stable Diffusion backbone network and fine-tune the LoRA module to obtain a generative model that can controllably generate physically consistent visible light-thermal imaging ship paired images. Use this generative model to generate augmented images. Merge the augmented images with the original paired images to form a cross-modal ship dataset. S2. Perform image augmentation on the cross-modal ship dataset and annotate to generate ship bounding boxes and category labels; S3. The enhanced cross-modal ship dataset is mapped to feature embeddings with the same spatial resolution and number of channels by an independent modal encoder. After concatenation along the channel dimension, the embeddings are input into the YOLO11 backbone network. A cross-modal attention module is embedded after the first convolutional downsampling layer of the YOLO11 backbone network to achieve channel-level semantic guidance fusion of thermal imaging modality and visible light modality. After generating fused features, the YOLO11 backbone network is used to generate multi-scale feature maps by downsampling and upsampling at each level. These features are then input into the YOLO11 native detection head to complete ship target detection, thus constructing an end-to-end cross-modal ship detection model. S4. For multi-scale ship targets, a composite loss function consisting of zoom classification loss, distributed focus regression loss, and task alignment intersection-over-union loss is used to train and optimize the cross-modal ship detection model. The zoom classification loss is used to differentiate between positive and negative samples to improve the recall rate of small-scale ships. The distributed focus regression loss is used to enhance the robustness of ship target localization. The task alignment intersection-over-union loss is used to suppress the interference of low-quality prediction boxes on model optimization. S5. Deploy the optimized cross-modal ship detection model to the field to detect ships in the target area.
2. The all-weather, cross-modal ship inspection method based on YOLOv11 according to claim 1, characterized in that, In step S1, the all-weather visible light image and thermal imaging image are used to characterize the visual appearance and thermal radiation characteristics of ships in low-light, strong glare, limited visibility or complex background scenarios, respectively. The fine-tuning of the Stable Diffusion model using LoRA technology specifically includes: freezing all parameters of the Stable Diffusion backbone network and training only on the modules embedded in LoRA to preserve the prior knowledge of the pre-trained model; and ensuring that the generated augmented image pairs are physically consistent with the real monitoring scene in terms of scale, pose, and water background.
3. The all-weather, cross-modal ship inspection method based on YOLOv11 according to claim 1, characterized in that, In step S2, the preprocessing of the cross-modal ship dataset includes: Image enhancement is performed on the acquired visible light-thermal imaging paired images, including left-right flipping, Gaussian noise injection, and mosaic enhancement; A semi-automatic annotation method is adopted, combining preliminary screening by a pre-trained detection model with manual verification to generate ship bounding boxes and category labels; The labeled dataset is divided into training, validation, and test sets according to a set ratio.
4. The all-weather, cross-modal ship inspection method based on YOLOv11 according to claim 1, characterized in that, In step S3, the cross-modal feature fusion mechanism adopts a single-backbone cross-modal collaborative sensing structure, specifically including: The RGB visible light image and the single-channel thermal imaging image are initially feature mapped by independent modal encoders to obtain visible light embedding and thermal imaging embedding with the same spatial resolution and number of channels. The visible light embedding and the thermal imaging embedding are spliced along the channel dimension to form a joint feature tensor; The joint feature tensor is input into the YOLO11 backbone network, and a cross-modal attention module is embedded after the first convolutional downsampling layer of the YOLO11 backbone network to generate the first-stage fusion feature. Using the first stage fusion features as input, the subsequent layers in the YOLO11 backbone network are downsampled and upsampled step by step to generate a multi-scale feature map, and the multi-scale feature map is directly input into the YOLO11 native detection head for ship target detection. The entire process of feature fusion, scale transfer, and target detection is completed within the YOLO11 single-backbone architecture.
5. The all-weather cross-modal ship inspection method based on YOLOv11 according to claim 4, characterized in that, The cross-modal attention module is configured to perform the following operations: The joint feature tensor input to this module is split into visible light features and thermal imaging features along the channel dimension; The visible light features are applied to a self-attention mechanism: channel statistics are obtained through global average pooling, channel attention weights are generated through two fully connected layers and a nonlinear activation function, and then the weights are multiplied with the visible light features channel by channel. Based on the thermal imaging features, a cross-modal guidance signal is constructed: first, the thermal imaging features are subjected to global average pooling to obtain a channel-level response vector; then, a channel modulation weight vector is generated through a fully connected layer. This weight vector is used to weight the visible light features after self-attention enhancement channel by channel to realize cross-modal semantic guidance of the thermal imaging mode to the visible light mode. The visible light features enhanced by the above self-attention and thermal imaging-guided modulation are re-stitched with the original thermal imaging features according to the channel dimension to generate the first-stage fusion features.
6. The all-weather, cross-modal ship inspection method based on YOLOv11 according to claim 1, characterized in that, The zoom classification loss employs differentiated calculation methods for positive and negative samples. For positive samples, the intersection-union ratio of the predicted bounding box and the ground truth bounding box is used as a quality label, which is then logarithmically calculated with the predicted classification confidence. For negative samples, a focus parameter greater than 0 is introduced to exponentially process the classification confidence before combining it with the logarithmic term. This zoom classification loss improves the recall rate of small-scale ships by assigning higher learning weights to high-quality positive samples.
7. The all-weather, cross-modal ship inspection method based on YOLOv11 according to claim 1, characterized in that, The distributed focus regression loss is calculated for each of the four coordinate dimensions of the ship target bounding box. First, a soft label distribution is generated based on the true coordinate values. Then, the cross-entropy loss is calculated and summed between the coordinate probability distribution output by the model and the soft label distribution. The distributed focus regression loss transforms continuous coordinate regression into discrete probability modeling, which significantly enhances the localization robustness in low-resolution thermal imaging images.
8. The all-weather cross-modal ship inspection method based on YOLOv11 according to claim 1, characterized in that, The task alignment cross-union ratio loss is calculated only for all positive sample predicted boxes. First, the task alignment metric is calculated by combining the classification confidence of the predicted box, the cross-union ratio between the predicted box and the ground truth box, and two alignment weights greater than 0. Then, a decay exponent greater than or equal to 0 is introduced and combined with the cross-union ratio to construct the loss calculation formula. The task alignment cross-union ratio loss applies strong constraints only to predicted boxes with high alignment to suppress optimization interference from low-quality or false-detection boxes.
9. A YOLOv11-based all-weather cross-modal ship inspection device, characterized in that, The all-weather, multi-modal ship detection method based on YOLO11, applicable to any one of claims 1-7, includes an image acquisition module, an image enhancement module, a model building module, a model training module, and a model deployment module; The image acquisition module is used to acquire all-weather visible light images of water conservancy hubs and paired images of thermal imaging ships. After annotation, a training dataset for the generative model is constructed. The parameters of the Stable Diffusion backbone network are frozen and only the LoRA module is fine-tuned to obtain a generative model that can controllably generate physically consistent visible light-thermal imaging ship paired images. The generative model is used to generate augmented images, and the augmented images are merged with the original paired images to form a cross-modal ship dataset. The image enhancement module is used to enhance the images of the cross-modal ship dataset and generate ship bounding boxes and category labels. The model building module is used to map the enhanced cross-modal ship dataset into feature embeddings with the same spatial resolution and number of channels through an independent modal encoder, and then concatenate them along the channel dimension before inputting them into the YOLO11 backbone network. After the first convolutional downsampling layer of the YOLO11 backbone network, a cross-modal attention module is embedded to achieve channel-level semantic guidance fusion of thermal imaging modality and visible light modality. After generating fused features, they are downsampled and upsampled step by step by the YOLO11 backbone network to generate multi-scale feature maps, which are then input into the YOLO11 native detection head to complete ship target detection, thus constructing an end-to-end cross-modal ship detection model. The model training module is used to train and optimize the cross-modal ship detection model for multi-scale ship targets using a composite loss function consisting of zoom classification loss, distributed focus regression loss, and task alignment intersection-over-union loss. The zoom classification loss is used to differentiate between positive and negative samples to improve the recall rate of small-scale ships, the distributed focus regression loss is used to enhance the robustness of ship target localization, and the task alignment intersection-over-union loss is used to suppress the interference of low-quality prediction boxes on model optimization. The model deployment module is used to deploy the optimized cross-modal ship detection model to the field to detect ships in the target area.
10. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the YOLO11-based all-weather cross-modal ship detection method as described in any one of claims 1-7.