An infrared and visible light-based ship image target detection method, system, device and medium

Abstract

The application provides a ship image target detection method, system, device and medium based on infrared and visible light, belongs to the technical field of ship image detection, and inputs infrared and visible light ship images into a YOLOv10 model; a separation kernel attention mechanism is used to extract texture and contour features in the infrared and visible light ship images; a multi-scale expansion attention mechanism is used to calculate attention weights and pay attention to surrounding environment information of the ship in the infrared and visible light ship images; a loss function based on a boundary box shape is used to analyze and detect target information in the infrared and visible light ship images, and output display is performed. By processing infrared and visible light images at the same time, the infrared image is sensitive to temperature, can provide effective target information at night or under complex weather conditions; the visible light image provides texture and color information in the daytime or under good lighting conditions, and the accuracy and robustness of detection are improved.

Description

Technical Field

[0001] This invention belongs to the field of ship image detection technology, and particularly relates to a method, system, equipment and medium for ship image target detection based on infrared and visible light. Background Technology

[0002] Most ship target detection techniques rely on manually designed features and relatively complex image processing procedures, which are not only time-consuming and labor-intensive, but also often have unsatisfactory detection performance when faced with complex backgrounds, ships of different scales, and unpredictable weather conditions.

[0003] With the development of deep learning technology in the field of target detection, deep learning technology has been applied to ship target detection in infrared and visible light ship images. Some advanced deep learning models such as Faster R-CNN, EfficientDet, and YOLO series have been widely used. Researchers have continuously improved these advanced algorithms and trained them with a large amount of data to continuously optimize model parameters in order to improve the accuracy of ship target detection.

[0004] For example, related technologies have proposed an improved visible light ship image target detection algorithm based on Faster R-CNN, which enhances the model's multi-scale detection capability by introducing a feature pyramid module. Liu Lang et al. proposed an improved ship target detection algorithm, E-EfficientDet, which uses EfficientNet V2 and EfficientNet-B0 networks as feature extraction networks to improve the model's feature extraction capabilities. Simultaneously, an SPPNet network is added to perform max pooling on multi-scale feature maps to avoid information loss. Related technologies have also proposed an edge-information-guided infrared ship target detection network (EGISD-YOLO). To address the interference problem in complex infrared image backgrounds, a deconvolutional channel attention mechanism is designed to acquire local target features. Its edge-guided structure can be fused with deep features, improving the model's feature localization capability. Although the above methods can perform real-time detection of ship targets in infrared and visible light images, they still suffer from low accuracy in detecting multi-scale or small targets. The limitations of single-modality images in ship image detection scenarios result in low detection accuracy. Summary of the Invention

[0005] This invention provides a ship image target detection method based on infrared and visible light. The method can effectively solve the limitations of single-modal images in ship image detection scenarios and improve the accuracy and robustness of detection.

[0006] The methods include: Configure a split kernel attention mechanism in the backbone network of the YOLOv10 model; configure a multi-scale expanded attention mechanism in the neck network; Acquire infrared and visible light images of ships and input these images into the YOLOv10 model; The separable kernel attention mechanism uses depthwise separable convolution to capture individual features between channels and extracts different channel features containing texture and contour in infrared and visible light ship images; Attention weights are calculated using a multi-scale expanded attention mechanism, and attention is paid to the environmental information surrounding the ship in infrared and visible light ship images; The loss function based on bounding box shape is used to analyze and detect target information in infrared and visible light ship images, and the output is displayed.

[0007] It should be further noted that in the YOLOv10 model, the first convolutional layer based on the C2f module processes the infrared and visible light ship images and splits the infrared and visible light ship images into two parts; One part is passed to the Concat block, and the other part is passed to multiple Bottleneck blocks for processing; The Concat block concatenates the two processed parts to obtain the feature map of the fused visible light image, which is then processed by a second convolutional layer to obtain the output feature map of the visible light image.

[0008] It should be further noted that the processing methods of the separation core attention mechanism include: Execute the following convolution function Combine convolution kernel Infrared and visible light ship images Perform convolution operations in the horizontal direction, and use convolution kernels in the vertical direction. Perform convolution operations;

[0009] Features are extracted using spatially dilated convolutional layers. ;

[0010]

[0011] Output of the Separate Kernel Attention Mechanism It is an attention map and input feature map The Hadama product; .

[0012] It should be further explained that the multi-scale dilated attention mechanism in the method utilizes depthwise separable convolution to capture individual features between channels, extracting different channel features containing texture and contour in infrared and visible light ship images. Specifically, it includes the following steps: The feature maps of infrared and visible light ship images are used as input, and the dimensions of the input ship feature maps are defined based on height, width, and number of channels. The infrared ship feature map is defined as S1, and the visible light ship feature map is defined as S2. For the infrared ship feature image S1, a query, key, and value are generated through linear transformation; For the visible light ship feature map S2, a query, key, and value are generated through linear transformation; The query, key, and value of the infrared ship feature map S1 and the channel dimensions of the query, key, and value of the visible light ship feature map S2 are divided into multiple headers; the division is set to n headers; For each head, perform self-attention operations between colored blocks in the window surrounding the red query block; Different dilation rates are used in different heads to sample the feature blocks around the query; For the i-th head in the infrared image; The kernel size is set to 3×3, and the expansion rate r is... i =1, 2, 3; Use depthwise separable convolutions to operate on the query CH, key JH, and value ZH respectively; Depthwise separable convolution includes channel-wise convolution and point-wise convolution; When executing channel-wise convolution, the channel-wise convolution kernel is defined as TDH; Channel-wise convolution is applied to the query, key, and value separately to capture individual features between channels, resulting in: Channel-wise convolution query feature CDH=TDH CH; Channel-wise convolutional bond features JDH=TDH JH; Channel-wise convolutional value feature ZDH=TDH ZH; During pointwise convolution, a 1×1 convolution kernel ZDJ is used to perform pointwise convolution on the result of channel-wise convolution. The number of channels is adjusted as follows: Pointwise convolution query feature CZD=ZDJ·CDH; Pointwise convolutional key features JZD = ZDJ·JDH; Pointwise convolution query feature ZZD = ZDJ·ZDH; Calculate the attention score matrix: DFj = Softmax(CZD·(JZD)) T ); T is the time step; Calculate the weighted value SQ = DFj·ZZD; By connecting the features from different heads, we can obtain the infrared ship feature map after the attention mechanism processing at the i-th head and the visible light ship feature map after the attention mechanism processing at the i-th head.

[0013] It should be further explained that the infrared ship output features processed by the attention mechanism and the visible light ship output features are stitched together to obtain the stitched ship image features Tz. The ship image features Tz are linearly processed and fused into an output feature G that contains ship information at different scales; G = Qz·Tz + Pz; Qz is the information integrator, which weights and combines the features extracted from each head to obtain an output feature matrix containing semantic information at different scales; Pz is the parameter vector. The output feature G contains semantic information at different scales from infrared and visible light images, matches the information of the ship's surrounding environment, and extracts features from different channels using depthwise separable convolution.

[0014] It should be further noted that the methods for parsing and detecting target information in infrared and visible light ship images based on the bounding box shape loss function include: The loss function IoU in an object detection model is defined as:

[0015] Where B is the predicted bounding box region, B gt The actual bounding box area; The Shape-IoU calculation method is as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] in, and These are the weighting coefficients for the horizontal and vertical directions, respectively. It is a scaling factor. and These are the width and height of the actual bounding box area. It is the bounding box loss.

[0022] It should be further noted that the method also includes configuring evaluation indicators for the target detection results to evaluate the status of the target information results; The mean precision is used as the evaluation metric for algorithm performance, including: precision (P) and recall (R), as shown in the following formula:

[0023]

[0024] Where TP is the number of ships whose target is correctly predicted by the model, FP is the number of ships whose target is incorrectly predicted by the model, and FN is the number of ships whose target is incorrectly predicted by the model. The formulas for calculating the average precision (AP) and the average AP value (mAP) are as follows:

[0025]

[0026] Where P(R) is the P value in the PR curve when the horizontal axis is R, and N is the number of ship categories.

[0027] This application also provides a ship image target detection system based on infrared and visible light, the system including: YOLOv10 model, image acquisition module and display module; Configure a separate kernel attention mechanism in the backbone network; configure a multi-scale expanded attention mechanism in the neck network; The image acquisition module acquires infrared and visible light images of ships through a 360° omnidirectional camera and inputs these images into the YOLOv10 model. The separable kernel attention mechanism uses depthwise separable convolution to capture individual features between channels and extracts different channel features containing texture and contour in infrared and visible light ship images; Attention weights are calculated using a multi-scale expanded attention mechanism, and attention is paid to the environmental information surrounding the ship in infrared and visible light ship images; The loss function based on the bounding box shape is used to analyze and detect target information in infrared and visible light ship images, and the output is displayed based on the display module.

[0028] According to another embodiment of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the infrared and visible light-based ship image target detection method.

[0029] According to another embodiment of this application, a storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the infrared and visible light-based ship image target detection method.

[0030] As can be seen from the above technical solutions, the present invention has the following advantages: The ship image target detection method provided in this application, based on infrared and visible light, simultaneously processes infrared and visible light images. Infrared images, being temperature-sensitive, can provide effective target information at night or under complex weather conditions; while visible light images provide texture and color information during the day or in well-lit conditions. This application effectively overcomes the limitations of single-modal images in ship image detection scenarios, improving detection accuracy and robustness.

[0031] This application incorporates a separable kernel attention mechanism into the backbone network of the YOLOv10 model, enabling the capture of individual features between channels and the extraction of different channel features, including texture and contour. A multi-scale expanded attention mechanism is configured in the neck network to calculate attention weights and focus on information about the ship's surrounding environment. The separable kernel attention mechanism employs depthwise separable convolutions, reducing the model's computational burden while maintaining good feature extraction capabilities, making the model more efficient when processing large-scale image data. It can handle targets of different scales, making the model more adaptable to changes in ship size.

[0032] In actual detection processes, because the size of the ship and its distance from the camera will vary, the multi-scale expanded attention mechanism can ensure that the model can accurately detect targets under various conditions. The loss function based on bounding box shape can more accurately parse and detect target information. It can optimize the target localization and the shape of the bounding box, making the detection results more accurate and reducing false positives and false negatives. Attached Figure Description

[0033] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of a ship image target detection method based on infrared and visible light. Figure 2 This is a diagram of the YOLOv10n network structure. Figure 3 This is a diagram of the LSKA network structure. Figure 4 This is a diagram of the MSDA network structure. Figure 5 The improved YOLOv10n network structure diagram; Figure 6 This is a PR curve chart; Figure 7 This is a map showing the detection of large targets in infrared and visible light images; Figure 8 This is a map showing the detection of small targets in an open sea environment. Figure 9 For heatmaps; Figure 10 This is a schematic diagram of an electronic device. Detailed Implementation

[0035] The target detection method for ship images based on infrared and visible light provided in this application is designed for complex background conditions of infrared and visible light ship images. The method incorporates the LSKA module into the YOLOv10 model to improve the model's feature extraction capability.

[0036] The YOLOv10 model in this application involves a backbone network, a neck, and a head, enabling real-time detection of infrared and visible light ship images, demonstrating the characteristics of fast detection speed and high accuracy.

[0037] The backbone network of this application is mainly designed to extract features from infrared and visible light ship images, and enhances the transfer of gradients between different layers through cross-stage connections.

[0038] The neck of this application focuses on feature fusion and multi-scale information extraction from infrared and visible light ship images. Through bottom-up and top-down path aggregation, it achieves effective fusion of features at different scales, improving the model's ability to detect objects of varying sizes. The head is used for classification and position regression of infrared and visible light ship images. Multiple predictions can be generated for each object during training, providing supervision signals and improving the model's accuracy.

[0039] To enhance the model's multi-scale detection capabilities and improve the fusion of multi-scale features from both types of images, this application incorporates an MSDA module into the neck network. Finally, to improve the model's small target detection capabilities, a Shape-IoU loss function is introduced, which considers the influence of inherent properties such as the bounding box's scale on bounding box regression. This application meets the detection requirements for both infrared and visible light targets.

[0040] The specific steps of the ship image target detection method based on infrared and visible light will be described in detail below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0041] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see Figure 1 The diagram shows a flowchart of a ship image target detection method based on infrared and visible light in a specific embodiment. The method includes: S101: Configure a separate kernel attention mechanism in the backbone network of the YOLOv10 model; configure a multi-scale expanded attention mechanism in the neck network.

[0044] The YOLOv10 model in this embodiment uses YOLOv10n as the baseline network model, such as Figure 2 As shown, it can be divided into three parts: the backbone network, the neck network, and the head. The YOLOv10 model has a C2f module, which processes the feature map based on the first convolutional layer and splits the feature map into two parts. One part is passed to the Concat block, and the other part is passed to multiple Bottleneck blocks for processing. Then, the two processed parts are concatenated in the Concat block to obtain the fused feature map. Finally, it is processed by the second convolutional layer to obtain the output feature map.

[0045] In this embodiment, the split operation in the YOLOv10 model C2fCIB and the CIB (CompactInverted Bottleneck) effectively mix spatial and channel features by combining depthwise convolution and pointwise convolution.

[0046] The YOLOv10 model's SCDown (spatial-channel decoupled downsampling) first adjusts the number of channels using a 1×1 point convolution, then performs spatial downsampling using a 3×3 depthwise convolution. SPPF (Spatial Pyramid Pooling Fast) performs multi-scale feature extraction and fusion on the input feature map. PSA (Partial Self-Attention) first divides the input features evenly into two parts using a 1×1 convolution, then feeds one part of the features into an NPSA block composed of a Multi-Head Self-Attention Module (MHSA) and a Feedforward Network (FFN). The two parts are then concatenated and fused using a 1×1 convolution. The Neck integrates low-level detailed information with high-level semantic information. The Head implements functions such as object classification and bounding box size prediction.

[0047] In this embodiment, a separable kernel attention mechanism is configured in the backbone network of the YOLOv10 model for feature extraction and initial downsampling. This embodiment uses depthwise separable convolution instead of standard convolution, performing convolution on each channel separately and then linearly combining them across channels to capture individual features between channels.

[0048] In some specific embodiments, for the input infrared and visible light ship images, each channel is convolved individually, and then pointwise convolution is performed for linear combination across channels.

[0049] After infrared and visible light ship images are preprocessed and fed into the backbone network of the YOLOv10 model, the feature maps sequentially pass through the separation kernel attention mechanism module. This allows for the capture of features both between and within channels. For infrared and visible light images, it enables the effective extraction of different channel features such as texture and contour, tailored to the characteristics of different channels.

[0050] S102: Acquire infrared and visible light images of the ship and input the infrared and visible light images into the YOLOv10 model.

[0051] In some embodiments, infrared and visible light cameras can be fixed on the same rotating platform to capture video images in a 360° omnidirectional manner. The acquired infrared and visible light ship images are derived from different frames in the video sequence, resulting in two types of images: infrared and visible light.

[0052] As one implementation of this application, the acquired images can also be preprocessed, including image normalization, size adjustment and other operations, to meet the input requirements of the YOLOv10 model, and then the preprocessed infrared and visible light images can be simultaneously input into the YOLOv10 model.

[0053] The normalization method involves normalizing the pixel values ​​of infrared and visible light ship images from [0,255] to the range of [0,1]. The normalization formula used is G= .in, These are the original pixel values. It is the mean of the dataset. It is the standard deviation of the dataset. This embodiment also performs scaling processing on infrared and visible light ship images. Bilinear interpolation can be used to adjust the image to the input size preset by the YOLOv10 model, specifically 608×608.

[0054] This embodiment processes infrared images to highlight the thermal radiation characteristics of ships, resulting in excellent performance at night or in low-light environments. It can also process the rich visual information such as texture and color contained in visible light images, improving the accuracy and robustness of target detection.

[0055] S103: The separable kernel attention mechanism uses depthwise separable convolution to capture individual features between channels and extracts different channel features containing texture and contour in infrared and visible light ship images.

[0056] In some embodiments, the Large Separable Kernel Attention (LSKA) mechanism can focus attention on more discriminative feature locations in infrared and visible light images, taking into account the differences in ship target features due to factors such as illumination angle and imaging differences. Furthermore, the LSKA mechanism can extract individual features between channels captured by the separable convolution, extracting features from different channels including texture and contours, providing more targeted feature representations for subsequent attention calculations. This embodiment uses large kernel convolutions to calculate attention weights, enabling the network to consider more contextual feature information and pay attention to the environmental information surrounding the ship in infrared and visible light images, reflecting the importance of different regions to the ship target, thereby improving the model's feature extraction capabilities.

[0057] The structure of the separation of nuclear attention mechanism is as follows Figure 3 As shown, the horizontal and vertical features of the input feature map are first extracted by two convolutional layers. By using convolution operations in different directions, the differences in image features in the two directions are roughly obtained, so that the subsequent network can pay more attention to the key regions of the image.

[0058] This embodiment also uses a spatially dilated convolutional layer to further extract features from the generated initial attention map. The spatially dilated convolutional layer can use different dilation rates and process features more accurately in both horizontal and vertical directions.

[0059] For the separate kernel attention mechanism, new features can be fused through the last convolutional layer to generate a final attention map. This final attention map is multiplied element-wise with the initial input feature map. From then on, each element of the initial feature map is weighted according to the value of the final attention map, making the network pay more attention to the key features of the ship target and suppressing interfering features.

[0060] The separate kernel attention mechanism in this embodiment can divide the two-dimensional convolution of the deep convolutional layer into two cascaded one-dimensional convolutional kernels, allowing LSKA to directly use the deep convolutional layer with large kernel convolution, reducing computational complexity and ensuring the model's feature extraction capability when deployed on hardware devices with limited resources.

[0061] The output of the separate kernel attention mechanism is shown in equations (1) to (3), where * and It is the convolution and Hadamard product.

[0062] (1) (2) (3) in, First use convolution kernel For the input feature map Perform a horizontal convolution operation, followed by a vertical convolution kernel. Perform a convolution operation. This is a further output, the output of LSKA. It is an attention map and input feature map The product of Hadama.

[0063] In this embodiment, for the input feature map Here, C represents the channel dimension. First, a convolution kernel is used. In the horizontal direction Perform a convolution operation. This means applying a convolution kernel of size (2d - 1) × 1 along the horizontal (width) direction of the image to each channel C. Then, on the result of the horizontal convolution, apply a convolution kernel in the vertical direction. Perform a convolution operation. That is, apply a convolution kernel of size (2d - 1) × 1 along the vertical direction (height direction) of the image. The final result is... .

[0064] In infrared and visible light ship image target detection, stepwise convolution in the horizontal and vertical directions can capture local features in different directions. In infrared and visible light ship images, horizontal convolution can capture features such as texture and lines in the horizontal direction, while vertical convolution can capture features in the vertical direction. In this embodiment, the extraction of features in the horizontal and vertical directions helps to more accurately depict the shape of the ship. Furthermore, by controlling 'd', feature extraction can be performed at different scales, adapting to ship targets of different sizes and the features of their surrounding environment.

[0065] For formula (2) in this embodiment, the formula obtained from formula (1) is... For input, first use convolution kernels A convolution operation is performed in the horizontal direction. Here, (|k / d|) represents a scale parameter related to k and d, determining the size of the horizontal convolution kernel. A convolution kernel is then applied to the result of the horizontal convolution in the vertical direction. Perform convolution operations to finally obtain .

[0066] This embodiment further improves the detection of ship targets in infrared and visible light images. By processing the data and performing convolutions again in the horizontal and vertical directions using kernels of different scales, feature information at different scales can be extracted. In this way, the model captures features of the ship and its surrounding environment at different levels of detail. Specifically, at smaller scales, it can capture the local details of the ship; at larger scales, it can capture the overall shape of the ship and its relationship with its surroundings.

[0067] Formula (3) in this embodiment is for Use a 1×1 convolution kernel Perform convolution operations to obtain the attention map. A 1×1 convolutional kernel is mainly used to adjust the channel dimension, and can also fuse information between different channels to generate attention weight distribution.

[0068] This embodiment also includes attention maps Compared with the original input feature map Perform the Hadamard product to obtain the final output. .

[0069] In target detection of infrared and visible light ship images, attention maps generated by 1×1 convolution are used. Different weights can be assigned to different feature channels and spatial locations to highlight features important for ship target detection and suppress irrelevant background information. In infrared images, areas with significant ship thermal radiation are assigned higher weights; in visible light images, areas with clear ship outlines and textures are assigned higher weights. The attention map is combined with the original feature map through the Hadamard product, resulting in a final output... It retains the original feature information while highlighting important features, thus improving the accuracy of target detection in infrared and visible light ship images.

[0070] S104: The attention weights are calculated using a multi-scale extended attention mechanism, and attention is paid to the environmental information around the ship in infrared and visible light ship images.

[0071] The Multi-Scale Dilated Attention (MSDA) mechanism involved in this embodiment can detect ship targets of different sizes and shapes, focusing on both the overall outline and local features of the ship. For small targets at a distance, larger-scale features are needed to capture their approximate location, while for large targets nearby, smaller-scale features are needed to accurately identify their specific structures.

[0072] To enhance multi-scale feature fusion and improve the model's feature representation capabilities, this embodiment incorporates a Multi-Scale Dilated Attention (MSDA) mechanism into YOLOv10. MSDA uses a sliding-window dilated attention module (SWDA) with different dilation rates in different heads to capture multi-scale semantic information. This allows the model to focus on local target features with a small dilation rate and capture semantic features of a wider region with a large dilation rate. MSDA leverages the sparsity of the self-attention mechanism at different scales, reducing computational weights for blocks far from the query block and with low relevance, focusing on the ship target and its surrounding area, thereby reducing unnecessary computation and improving computational efficiency. Furthermore, MSDA separates the feature map channels into multiple heads, each processed in parallel, enhancing the model's learning capabilities. The MSDA network structure diagram is shown below. Figure 4 As shown.

[0073] The MSDA in this embodiment mainly consists of three parts: an input linear projection layer, a multi-scale dilated attention layer, and an output linear fusion layer. The input linear projection layer performs a linear transformation on the input feature map to generate a query (Q), a key (K), and a value (V). These vectors form the basis for subsequent attention calculations, representing the input feature information from different perspectives. The multi-scale dilated attention layer divides the channel dimension of the input feature map into multiple heads, performing self-attention operations between colored blocks in the window surrounding the red query block. Different heads use different dilation rates (r) to sample the feature blocks around the query (Q). Furthermore, features from different heads are concatenated and then passed to the output linear fusion layer. Typically, a 3×3 kernel size is used, with dilation rates r = 1, 2, and 3, and receptive field sizes of 3×3, 5×5, and 7×7 for different heads. The output linear fusion layer fuses the concatenated features into output features containing semantic information at different scales. This comprehensively represents the image content, providing more effective features for subsequent recognition tasks.

[0074] S105: Based on the loss function of bounding box shape, analyze and detect target information in infrared and visible light ship images, and output the results.

[0075] This embodiment addresses the fact that existing bounding box regression methods generally only consider the geometric relationship between the predicted and ground truth bounding boxes, calculating the loss based on the relative position and shape of the bounding boxes, while neglecting the influence of the bounding box's own shape and scale on the regression. This is particularly relevant in infrared and visible light image ship target detection tasks, where the image background is complex and blurry, and small targets are easily obscured by background interference or other objects. Shape-IoU, however, considers the shape and scale of the bounding box itself, thus making the bounding box regression more accurate.

[0076] The loss function IoU in the object detection model in this embodiment is defined as follows: (4) Where B is the predicted bounding box region, B gt This represents the actual bounding box region.

[0077] Shape-IoU is calculated as shown in formulas (5) to (10): (5) (6) (7) (8) (9) (10) in and These are the weighting coefficients for the horizontal and vertical directions, respectively. It is a scaling factor, related to the scale of the target dataset. and These are the width and height of the actual bounding box area. It is the bounding box loss.

[0078] As can be seen, the ship image target detection method based on infrared and visible light provided in this embodiment improves the model's feature extraction capability by adding an LSKA attention module to the backbone network. To address the issue of large size differences between ship targets in infrared and visible light images, three MSDA attention modules are added between the neck network and the detection head. These modules can acquire semantic information at different scales through the self-attention mechanism of different heads, better adapting to ship targets of different scales and improving the model's multi-scale detection capability. Finally, the model introduces a Shape-IoU loss function, focusing on the shape and size of the bounding box itself during detection, enhancing the ability to detect small targets. The improved model structure diagram is shown below. Figure 5 As shown.

[0079] In one embodiment of the present invention, the multi-scale dilated attention mechanism utilizes depthwise separable convolution to capture individual features between channels, extracting different channel features containing texture and contour in infrared and visible light ship images, specifically including the following steps: The feature maps of infrared and visible light ship images are used as input, and the dimensions of the input ship feature maps are defined based on height, width, and number of channels. The infrared ship feature map is defined as S1, and the visible light ship feature map is defined as S2.

[0080] For the infrared ship feature image S1, a query, key, and value are generated through linear transformation.

[0081] For the visible light ship feature map S2, a query, key, and value are generated through linear transformation.

[0082] The query, key, and value of the infrared ship feature map S1 and the channel dimensions of the query, key, and value of the visible light ship feature map S2 are divided into multiple headers; the division is set to n headers.

[0083] For each head, self-attention is performed between colored blocks in the window surrounding the red query block.

[0084] Different dilation rates are used in different headers to sample the feature blocks around the query.

[0085] For the i-th head in the infrared image.

[0086] The kernel size is set to 3×3, and the expansion rate r is... i =1, 2, 3.

[0087] Use depthwise separable convolutions to operate on the query CH, key JH, and value ZH respectively.

[0088] Depthwise separable convolution includes channel-wise convolution and point-wise convolution.

[0089] When executing channel-wise convolution, the channel-wise convolution kernel is defined as TDH.

[0090] Channel-wise convolution is applied to the query, key, and value separately to capture individual features between channels, resulting in: Channel-wise convolution query feature CDH=TDH CH; Channel-wise convolutional bond features JDH=TDH JH; Channel-wise convolutional value feature ZDH=TDH ZH.

[0091] During pointwise convolution, a 1×1 convolution kernel ZDJ is used to perform pointwise convolution on the result of channel-wise convolution. The number of channels is adjusted as follows: Pointwise convolution query feature CZD=ZDJ·CDH; Pointwise convolutional key features JZD = ZDJ·JDH; Pointwise convolution query feature ZZD = ZDJ·ZDH; Calculate the attention score matrix: DFj = Softmax(CZD·(JZD)) T ); T represents the time step.

[0092] Calculate the weighted value SQ = DFj·ZZD.

[0093] By connecting the features from different heads, we can obtain the infrared ship feature map after the attention mechanism processing at the i-th head and the visible light ship feature map after the attention mechanism processing at the i-th head.

[0094] This embodiment also stitches together the infrared ship output features and the visible light ship output features after attention mechanism processing to obtain the stitched ship image features Tz. The ship image features Tz are linearly processed and fused into an output feature G that contains ship information at different scales; G = Qz·Tz + Pz; Qz is the information integrator, which weights and combines the features extracted from each head to obtain an output feature matrix containing semantic information at different scales; Pz is the parameter vector. The output feature G contains semantic information at different scales from infrared and visible light images, matches the information of the ship's surrounding environment, and extracts features from different channels using depthwise separable convolution.

[0095] The target detection method for ships based on infrared and visible light integrates a multi-scale extended attention mechanism into a separable kernel attention mechanism, and uses depthwise separable convolution to extract features from different channels in infrared and visible light images to calculate attention weights and focus on information about the ship's surrounding environment.

[0096] Based on the above embodiments, to further improve the accuracy and robustness of the ship image target detection method based on infrared and visible light provided in the above embodiments, the following is a specific implementation method. In one embodiment, visible light and infrared acquisition devices are used for data acquisition. Infrared and visible light cameras are fixed on the same rotating platform to capture video images from all directions (360°). The experimental dataset comes from different frames in the video sequence, containing both infrared and visible light images. The targets cover seven object classes, including cargo ships, ferries, coast guard vessels, buoys, fishing boats, tugboats, and rescue vessels. The images were obtained under different weather and imaging conditions, and the target sizes vary greatly. In addition to ships of different types and sizes, there are also interference factors such as islands, waves, and reflections. The overall number of images is large, and the infrared image resolution is relatively low. The dataset contains a total of 6018 images, divided into 4212 images for the training set, 1203 images for the validation set, and 603 images for the test set. The dataset information is shown in Table 1.

[0097] Table 1 Dataset Information

[0098] In this embodiment, the electronic device is configured with a Windows 11 system, 24GB of memory, and a 13th Gen Intel® Core CPU. TM The system used an i7-13650HX processor and an NVIDIA GeForce RTX 4060 GPU. The training environment consisted of Python 3.9, PyTorch 2.0.1, CUDA version 12.6, 16 training batches, and 300 training epochs. The input image size was adjusted to 640×640 pixels. The optimizer used the SGD (Stochastic Gradient Descent) algorithm with an initial learning rate of 0.01 and a weight decay coefficient of 0.0005.

[0099] In this embodiment, the mean precision (mAP) with an IoU threshold of 0.5 is used as the evaluation metric for the ship image target detection method based on infrared and visible light. The formulas for precision (P) and recall are as follows:

[0100]

[0101] Wherein, TP is the number of ships whose target is correctly predicted by the model as positive, FP is the number of ships whose target is incorrectly predicted by the model as positive, and FN is the number of ships whose target is incorrectly predicted by the model as negative.

[0102] Mean precision (AP) is the area under the PR curve. The average of the AP calculated for all categories is mAP. The formulas for AP and mAP are as follows:

[0103]

[0104] Wherein, P(R) is the P value on the PR curve when the horizontal axis is R, and N is the number of ship categories. In addition, this application also uses parameters as an evaluation index.

[0105] This embodiment proposes an improved YOLOv10-based algorithm for ship target detection in infrared and visible light images. The algorithm incorporates an LSKA attention mechanism for feature extraction and an MSDA attention mechanism in the neck network, while also introducing a Shape-IoU loss function. To investigate the impact of each module on ship target detection, a series of ablation experiments were designed. A thorough exploration of the role of each module in the overall model provides a reference for further optimization of the algorithm. Specific ablation experiment results are shown in Table 2.

[0106] Table 2 Ablation Experiment Results

[0107] As can be seen from the results in Table 2, the mAP value of YOLOv10n is 83.9%. After adding LSKA, the mAP value increased by 0.2%. This shows that LSKA, which uses spatial dilation convolution with different dilation rates, can better extract key features of the image and improve the feature extraction capability of the network.

[0108] The baseline model's mAP improved by 0.7% after incorporating the MSDA attention mechanism. The MSDA mechanism divides the feature map channels into multiple heads, each processing a different subset of features. This allows the model to focus on features at different scales, and integrating the information from each head yields a richer feature representation. Its parallel feature extraction and fusion method enhances the model's multi-scale detection capability. Introducing the Shape-IoU loss function improved the mAP by 0.4%, indicating that the Shape-IoU loss function provides more discriminative supervision information, enabling the model to learn the shape and position of targets faster and improving its small target detection capability. Furthermore, applying LSKA and MSDA simultaneously improved the mAP by 0.9%, and the algorithm in this application achieved an mAP of 85.1%, a 1.2% improvement over the baseline model. Moreover, the precision and recall of this application's algorithm outperformed most algorithms, demonstrating that the three modules each contribute to the original model's performance and can be fused together, improving the original model's ship target detection capability.

[0109] To verify the effectiveness of the algorithm proposed in this application, the algorithm involved in this embodiment was compared with other advanced algorithms on the same dataset. The results of the comparison are shown in Table 3. As can be seen from Table 3, the algorithm of this application outperforms other advanced algorithms in mAP@0.5. Although the number of parameters is slightly more than the original algorithm, it is still less than the number of parameters in most advanced algorithms, which further demonstrates the feasibility of the algorithm of this application. The PR curve of the algorithm in this embodiment is shown in the figure below. Figure 6 As shown.

[0110] Table 3 Comparison of experimental results

[0111] During testing of the algorithm proposed in this application, it was found that it could accurately detect both large ship targets in infrared and visible light images and small targets in open sea environments. This result fully demonstrates the algorithm's strong multi-scale detection capability, enabling it to effectively identify and locate targets of different scales. Simultaneously, its small target detection capability was also demonstrated, providing an efficient solution for target detection tasks in complex scenes. The test results are shown in the figure below. Figure 7 and Figure 8 As shown.

[0112] In terms of heatmaps, heatmaps can use color intensity to represent information such as target location and confidence level, intuitively showing the model's attention to different targets and detection results. Examples of heatmaps before and after model improvement are shown below. Figure 9 As shown. From Figure 9It can be seen that the improved model's heatmap can basically completely cover the targets when detecting large targets in open sea environments, ship targets in cloudy and foggy weather, and scattered small targets, and can reduce the interference of irrelevant background on the model, demonstrating the feasibility of the model in this application.

[0113] This application addresses the challenges of complex backgrounds, large scale variations in ship targets, and low resolution in infrared and visible light ship images. It proposes an improved target detection method for infrared and visible light ship images based on YOLOv10. The method incorporates an LSKA attention mechanism into the backbone network of the original model to enhance its feature extraction capabilities, adds an MSDA attention mechanism to improve multi-scale detection, and employs a Shape-IoU loss function to improve small target detection. Experimental results demonstrate that the proposed algorithm achieves an mAP of 85.1% on the original dataset, significantly improving performance compared to the baseline model and other state-of-the-art algorithms, thus proving the effectiveness of the proposed algorithm.

[0114] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0115] The following are embodiments of the ship image target detection system based on infrared and visible light provided in this disclosure. This system and the ship image target detection method based on infrared and visible light in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the ship image target detection system based on infrared and visible light, please refer to the embodiments of the ship image target detection method based on infrared and visible light described above.

[0116] The system includes: a YOLOv10 model, an image acquisition module, and a display module.

[0117] Configure a separate kernel attention mechanism in the backbone network; configure a multi-scale expanded attention mechanism in the neck network.

[0118] The image acquisition module acquires infrared and visible light images of ships through a 360° omnidirectional camera and inputs these images into the YOLOv10 model.

[0119] The separable kernel attention mechanism uses depthwise separable convolution to capture individual features between channels and extracts different channel features containing texture and contour in infrared and visible light ship images.

[0120] Attention weights are calculated using a multi-scale extended attention mechanism, and attention is paid to the environmental information surrounding the ship in infrared and visible light ship images.

[0121] The loss function based on the bounding box shape is used to analyze and detect target information in infrared and visible light ship images, and the output is displayed based on the display module.

[0122] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0123] like Figure 10 As shown, this application also provides an electronic device, including a display module 103, a memory 102, a processor 101, and a computer program stored in the memory and executable on the processor 101. When the processor 101 executes the program, it implements the steps of a ship image target detection method based on infrared and visible light.

[0124] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown in this application, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments described and / or claimed in this application.

[0125] In this embodiment, processor 101 may be implemented using at least one of an Application-Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), a Field-Programmable Gate Array (FPGA), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such implementations may be implemented within a controller. For software implementations, implementations such as processes or functions may be implemented with separate software modules that allow the performance of at least one function or operation. The software code may be implemented by a software application (or program) written in any suitable programming language, and the software code may be stored in memory and executed by the controller.

[0126] The display module 103 is used to display information input by the user or information provided to the user. The display module 103 may include a display panel, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0127] The memory 102 can be used to store software programs and various data. The memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0128] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the infrared and visible light-based ship image target detection method.

[0129] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0130] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for ship image target detection based on infrared and visible light, characterized in that the method... include: Configure a split kernel attention mechanism in the backbone network of the YOLOv10 model; Configure multi-scale expanded attention mechanisms in the neck network; Acquire infrared and visible light images of ships and input these images into the YOLOv10 model; The separable kernel attention mechanism uses depthwise separable convolution to capture individual features between channels and extracts different channel features containing texture and contour in infrared and visible light ship images. It extracts horizontal and vertical features of the input feature map based on two convolutional layers and uses convolution operations in different directions to obtain the differences in image features in two directions. Attention weights are calculated using a multi-scale expanded attention mechanism, and attention is paid to the environmental information surrounding the ship in infrared and visible light ship images; The loss function based on bounding box shape is used to analyze and detect target information in infrared and visible light ship images, and the output is displayed. The processing methods for multi-scale extended attention mechanisms include: The feature maps of infrared and visible light ship images are used as input, and the dimensions of the input ship feature maps are defined based on height, width, and number of channels. The infrared ship feature map is defined as S1, and the visible light ship feature map is defined as S2. For the infrared ship feature image S1, a query, key, and value are generated through linear transformation; For the visible light ship feature map S2, a query, key, and value are generated through linear transformation; The query, key, and value of the infrared ship feature map S1 and the channel dimensions of the query, key, and value of the visible light ship feature map S2 are divided into multiple headers; the division is set to n headers; For each head, perform self-attention operations between colored blocks in the window surrounding the red query block; Different dilation rates are used in different heads to sample the feature blocks around the query; For the i-th head in the infrared image; The kernel size is set to 3×3, and the expansion rate ri = 1, 2, 3; Use depthwise separable convolutions to operate on the query CH, key JH, and value ZH respectively; Depthwise separable convolution includes channel-wise convolution and point-wise convolution; When executing channel-wise convolution, the channel-wise convolution kernel is defined as TDH; Channel-wise convolution is applied to the query, key, and value separately to capture individual features between channels, resulting in: channel-wise convolutional query feature CDH = TDH CH; channel-wise convolutional key features JDH = TDH JH; Channel-wise convolution value feature ZDH = TDH ZH; During pointwise convolution, a 1×1 convolution kernel ZDJ is used to perform pointwise convolution on the result of channel-wise convolution. The number of channels is adjusted as follows: Pointwise convolution query feature CZD=ZDJ·CDH; Pointwise convolutional key features JZD = ZDJ·JDH; Pointwise convolution query feature ZZD = ZDJ·ZDH; Compute attention score matrix: DFj= Softmax ); T is the time step; Calculate the weighted value SQ = DFj·ZZD; By connecting the features from different heads, we can obtain the infrared ship feature map after the attention mechanism processing at the i-th head and the visible light ship feature map after the attention mechanism processing at the i-th head. For the generated initial attention map, a spatially dilated convolutional layer is used to further extract features. The spatially dilated convolutional layer uses different dilation rates and processes features in both horizontal and vertical directions. The new features are fused through the last convolutional layer to generate the final attention map. This final attention map is then multiplied element-wise with the initial input feature map, so that each element of the initial feature map is weighted according to the value of the final attention map. This makes the network focus more on the key features of the ship target and suppresses interfering features.

2. The infrared and visible light based ship image target detection method according to claim 1, wherein, In the YOLOv10 model, the first convolutional layer based on the C2f module processes infrared and visible light ship images and splits the infrared and visible light ship images into two parts; One part is passed to the Concat block, and the other part is passed to multiple Bottleneck blocks for processing; The Concat block concatenates the two processed parts to obtain the feature map of the fused visible light image, which is then processed by a second convolutional layer to obtain the output feature map of the visible light image.

3. The infrared and visible light based ship image target detection method according to claim 1, wherein, The processing methods for separating the attention mechanism include: First use convolution kernel For the input feature map Perform a horizontal convolution operation, followed by a vertical convolution kernel. Perform a convolution operation to obtain ; Using spatially dilated convolutional layers to extract features ; the output of the separate nuclear attention mechanism is an attention map and the input feature map hadamard product 。 4. The ship image target detection method based on infrared and visible light according to claim 1, characterized in that, The infrared ship output features processed by the attention mechanism and the visible light ship output features are stitched together to obtain the stitched ship image features Tz. The ship image features Tz are linearly transformed and fused into output features G containing ship information at different scales; G = Qz·Tz + Pz; Qz is the information integrator, which weights and combines the features extracted from each head to obtain an output feature matrix containing semantic information at different scales; Pz is the parameter vector. The output feature G contains semantic information at different scales from infrared and visible light images, matches the information of the ship's surrounding environment, and extracts features from different channels using depthwise separable convolution.

5. The infrared and visible light based ship image target detection method according to claim 1, wherein, Methods for parsing and detecting target information in infrared and visible light ship images based on bounding box shape loss functions include: The loss function IoU in an object detection model is defined as: wherein B is a prediction bounding box region, is a ground truth bounding box region; The Shape-IoU calculation method is as follows: wherein, and are horizontal and vertical weight coefficients, respectively, is a scale factor, and are the width and height of the real box region, is the bounding box loss.

6. The infrared and visible light based ship image target detection method according to claim 1, wherein, The method also includes configuring evaluation indicators for target detection results to evaluate the status of target information results; The mean precision is used as the evaluation metric for algorithm performance, including: precision (P) and recall (R), as shown in the following formula: Where TP is the number of ships whose target is correctly predicted by the model, FP is the number of ships whose target is incorrectly predicted by the model, and FN is the number of ships whose target is incorrectly predicted by the model. The formulas for calculating the average precision (AP) and the average AP value (mAP) are as follows: Where P(R) is the P value in the PR curve when the horizontal axis is R, and N is the number of ship categories.

7. An infrared and visible light based ship image target detection system, characterized by The system is used to implement the ship image target detection method based on infrared and visible light as described in any one of claims 1 to 6; The system includes: a YOLOv10 model, an image acquisition module, and a display module; Configure a separate kernel attention mechanism in the backbone network; configure a multi-scale expanded attention mechanism in the neck network; The image acquisition module acquires infrared and visible light images of ships through a 360° omnidirectional camera and inputs these images into the YOLOv10 model. The separable kernel attention mechanism uses depthwise separable convolution to capture individual features between channels and extracts different channel features containing texture and contour in infrared and visible light ship images; first, two convolutional layers extract the horizontal and vertical features of the input feature map, and then use convolution operations in different directions to obtain the differences in image features in the two directions; Attention weights are calculated using a multi-scale expanded attention mechanism, and attention is paid to the environmental information surrounding the ship in infrared and visible light ship images; The loss function based on the bounding box shape is used to analyze and detect target information in infrared and visible light ship images, and the output is displayed based on the display module. The processing methods for multi-scale extended attention mechanisms include: The feature maps of infrared and visible light ship images are used as input, and the dimensions of the input ship feature maps are defined based on height, width, and number of channels. The infrared ship feature map is defined as S1, and the visible light ship feature map is defined as S2. For the infrared ship feature image S1, a query, key, and value are generated through linear transformation; For the visible light ship feature map S2, a query, key, and value are generated through linear transformation; The query, key, and value of the infrared ship feature map S1 and the channel dimensions of the query, key, and value of the visible light ship feature map S2 are divided into multiple headers; the division is set to n headers; For each head, perform self-attention operations between colored blocks in the window surrounding the red query block; Different dilation rates are used in different heads to sample the feature blocks around the query; For the i-th head in the infrared image; The kernel size is set to 3×3, and the expansion rate ri = 1, 2, 3; Use depthwise separable convolutions to operate on the query CH, key JH, and value ZH respectively; Depthwise separable convolution includes channel-wise convolution and point-wise convolution; When executing channel-wise convolution, the channel-wise convolution kernel is defined as TDH; Channel-wise convolution is applied to the query, key, and value separately to capture individual features between channels, resulting in: channel-wise convolutional query feature CDH = TDH CH; channel-wise convolutional key features JDH = TDH JH; Channel-wise convolution value feature ZDH = TDH ZH; During pointwise convolution, a 1×1 convolution kernel ZDJ is used to perform pointwise convolution on the result of channel-wise convolution. The number of channels is adjusted as follows: Pointwise convolution query feature CZD=ZDJ·CDH; Pointwise convolutional key features JZD = ZDJ·JDH; Pointwise convolution query feature ZZD = ZDJ·ZDH; Compute attention score matrix: DFj= Softmax ); T is the time step; Calculate the weighted value SQ = DFj·ZZD; By connecting the features from different heads, we can obtain the infrared ship feature map after the attention mechanism processing at the i-th head and the visible light ship feature map after the attention mechanism processing at the i-th head. For the generated initial attention map, a spatially dilated convolutional layer is used to further extract features. The spatially dilated convolutional layer uses different dilation rates and processes features in both horizontal and vertical directions. The new features are fused through the last convolutional layer to generate the final attention map. This final attention map is then multiplied element-wise with the initial input feature map, so that each element of the initial feature map is weighted according to the value of the final attention map. This makes the network focus more on the key features of the ship target and suppresses interfering features.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the infrared and visible light based ship image target detection method according to any one of claims 1 to 6 when executing the program.

9. A storage medium having stored thereon a computer program, characterized in that The computer program implements the steps of the infrared and visible light based ship image target detection method according to any one of claims 1 to 6 when executed by the processor.

Images