Multi-modal ship target detection method based on asymmetric architecture knowledge distillation

Abstract

The present application relates to the technical field of computer vision and target detection, and specifically provides a multi-modal ship target detection method based on asymmetric architecture knowledge distillation, comprising: constructing a student detection model and a teacher model; in the training stage, inputting the same visible light image and thermal infrared image into the pre-trained teacher model and student detection model; obtaining the trained student detection model; in the inference stage, inputting the visible light image and thermal infrared image of the ship into the trained student detection model; splicing and fusing the visible light multi-scale features and thermal infrared multi-scale features to obtain a fused feature map; inputting the fused feature map into a detection head network to predict the position parameters and category information of the ship target, and obtaining a target detection result. The present application significantly improves the accuracy of ship target detection in complex maritime scenarios by using an asymmetric network structure.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and target detection technology, specifically relating to a multimodal ship target detection method based on asymmetric architecture knowledge distillation. Background Technology

[0002] With the acceleration of global economic integration, the ocean has become a crucial artery connecting the world and supporting economic development, leading to a growing demand for port and waterway monitoring in fields such as marine surveillance and maritime safety. However, the inherent complexity of the marine environment, such as wave morphology disturbances, drastic changes in lighting, and the dynamic interaction of objects at multiple scales, makes the effective detection of marine vessel targets a challenging task.

[0003] Existing ship target detection algorithms are mainly divided into two categories: single-sensor-based methods and multi-modal fusion-based methods. Traditional single-sensor methods, especially those relying on visible light images, while providing rich texture and color information, suffer from extremely low signal-to-noise ratios under adverse weather conditions such as nighttime, dense fog, and glare, leading to soaring false negative and false positive rates. While thermal infrared imaging offers the advantage of all-weather operation, it lacks fine texture, has low resolution, and is susceptible to interference from thermal crosstalk and sea surface clutter. Therefore, designing an efficient multi-modal fusion framework that organically combines the rich details of visible light with the all-weather characteristics of infrared images, fully leveraging the complementary advantages of both modalities, has become a recognized promising research direction in the industry.

[0004] Current mainstream deep learning-based multimodal detection methods typically employ mid-term fusion strategies, utilizing high-performance convolutional neural networks such as ResNet-101 or Transformer architectures to extract features and employing attention mechanisms for modal interaction. However, in practical applications requiring low power consumption and low parameter counts, existing multimodal fusion technologies suffer from three significant problems: First, the sheer number of model parameters makes it difficult to meet real-time deployment requirements. Mainstream high-performance fusion detection models generally have a large number of parameters and high computational complexity. Although lightweight networks such as MobileNet exist, in multimodal tasks, simple backbone replacement or symmetric compression can lead to a sharp decline in feature extraction capabilities, making it difficult to achieve lightweighting while maintaining high accuracy. Second, modal laziness is prevalent in the training of lightweight models. When model capacity is limited, the network tends to seek shortcuts during training, over-relying on visible light modalities with high information content and easily extracted features, while neglecting the learning of infrared modal features. This results in severe performance degradation of the fusion model in scenarios where visible light is unavailable, such as at night, negating the significance of multimodal fusion. Third, knowledge distillation between heterogeneous networks is difficult to perform effectively. To compensate for the accuracy loss caused by lightweighting, knowledge distillation is commonly employed. However, there are significant differences in feature dimensions between high-performance teacher network features and lightweight student network features in knowledge distillation. Existing distillation methods often fail to systematically address this feature adaptation problem between heterogeneous networks, resulting in the ineffective transfer of intermediate layer features containing rich spatial and semantic information. Summary of the Invention

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a multimodal ship target detection method based on asymmetric architecture knowledge distillation to solve the aforementioned technical problems.

[0006] In a first aspect, the present invention provides a multimodal ship target detection method based on asymmetric architecture knowledge distillation, comprising: A student detection model is constructed, which includes a visible light branch network, a thermal infrared branch network, a feature fusion module, and a detection head network. A teacher model is also constructed, which is a pre-trained multimodal ship target detection model. During the training phase, visible light and thermal infrared images of the ship are acquired. The same visible light and thermal infrared images are simultaneously input into the pre-trained teacher model and student detection model. The visible light branch network and thermal infrared branch network in the student detection model are asymmetric network structures. The teacher model and student detection model are based on the input visible light image and thermal infrared image, and output teacher feature information and teacher detection results, and student feature information and student detection results, respectively. Based on the differences between teacher and student feature information and the differences between teacher and student detection results, training constraints are constructed to train the student detection model, resulting in a trained student detection model. During the inference phase, visible light and thermal infrared images of the ship are input into the trained student detection model, and visible light multi-scale features and thermal infrared multi-scale features are obtained based on the visible light branch network and the thermal infrared branch network, respectively. Visible light multi-scale features and thermal infrared multi-scale features are spliced ​​and fused, and the fused features are recalibrated using a coordinate attention mechanism to obtain a fused feature map. The fused feature map is input into the detection head network to predict the position parameters and category information of the ship target, thus obtaining the target detection result.

[0007] In an optional implementation, the step of constructing the visible light branch network and the thermal infrared branch network in the student detection model as asymmetric network structures includes: MobileNetV2 was chosen as the basic network architecture for the asymmetric network structure, and visible light branch basic network structure and thermal infrared branch basic network structure were constructed respectively. Acquire a set of visible light images and a set of thermal infrared images; Pixel value statistics were performed on both the visible light image set and the thermal infrared image set, and the values ​​of each pixel value were statistically analyzed. The probability of appearing in the corresponding image set And calculate the average information entropy of the corresponding image set based on the pixel value probability distribution. The formula for calculating average information entropy is:

[0008] in, The information entropy of image set X is represented by n, where n represents the total number of pixel values. Based on the formula for calculating average information entropy, the average information entropy of the visible light image set and the average information entropy of the thermal infrared image set are obtained respectively. When the average information entropy of the visible light image set is greater than the average information entropy of the thermal infrared image set, it is determined that the visible light image set contains more information than the thermal infrared image set. The visible light branch basic network structure is configured as a feature extraction network with standard channel width to extract color information, high-frequency texture information and complex background information in visible light images; The thermal infrared branch basic network structure is configured as a channel-clipping feature extraction network to extract target contour information and temperature distribution features from thermal infrared images. The number of output channels in each network layer of the thermal infrared branch basic network structure is configured according to the channel clipping formula:

[0009] in, This represents the number of output channels for the corresponding network layer in the thermal infrared branch basic network structure. This represents the number of output channels of the corresponding network layer in the visible light branching basic network structure. Indicates the channel clipping ratio, and ; In the visible light branch and thermal infrared branch basic network structures, standard convolution operations are replaced with depthwise separable convolution operations, so that convolution operations are performed through channel-wise convolution and point-wise convolution respectively. Multi-scale feature output layers are set according to the same network layer position, so that the visible light branch network structure and the thermal infrared branch network structure output multi-scale feature maps with consistent spatial resolution at the corresponding layer, resulting in the visible light branch network structure and the thermal infrared branch network structure in the constructed asymmetric network structure.

[0010] In an optional implementation, the method further includes: training the student network using teacher network training, comprising: Set the number of FPN output channels for the teacher network and the number of FPN output channels for the student network, wherein the student network is an asymmetric network structure; A feature adaptation layer is set up between the teacher network and the student network to align the feature dimensions between the teacher network and the student network. Global average pooling is performed on the visible light image and the thermal infrared image respectively to obtain global feature vectors. The cosine similarity between the two global feature vectors is calculated to obtain the modality alignment loss. The distillation loss function of the feature adaptation layer and the distillation loss function of the Logits output layer are calculated separately. A composite total training loss function is constructed based on the explicit modality alignment loss, the basic detection loss, and the distillation loss function of the student network. Calculate the basic detection loss function; A composite total training loss function is constructed based on explicit modality alignment loss, basic detection loss, and distillation loss function of the student network. The student detection model is then trained based on the composite total training loss function to obtain a well-trained student detection model.

[0011] In an optional implementation, a feature adaptation layer is set between the teacher network and the student network to align the feature dimensions between the teacher network and the student network, including: Set up between teacher network and student network The convolutional module acts as a feature adaptation layer, transforming the low-dimensional feature maps of the student network. Projected to the teacher network:

[0012] in, The distillation loss function represents the feature adaptation layer. Represents the low-dimensional feature map of the student network. Represents the high-dimensional feature graph of the teacher network. express In the convolutional adaptation operation, C represents the channel, H represents the height, and W represents the width.

[0013] In an optional implementation, global average pooling is performed on the visible light image and the thermal infrared image respectively to obtain global feature vectors. The cosine similarity between the two global feature vectors is calculated to obtain the modal alignment loss, including:

[0014] in, Indicates the modal alignment loss. This represents the global feature vector of the visible light modes. This represents the global feature vector of the infrared mode.

[0015] In an optional implementation, the distillation loss function of the feature adaptation layer is calculated, including: In the feature adaptation layer, feature layers P3, P4, and P5 are selected. The distance between the student network and the teacher network after feature adaptation is calculated, and the distillation loss function of the feature adaptation layer is obtained:

[0016] in, The distillation loss function represents the feature adaptation layer. Represents the feature adaptation layer. The teacher network is in the first The feature map output from the feature layer Indicates the student network in the first Feature map of the feature layer after feature adaptation; Calculate the distillation loss function of the Logits output layer, including:

[0017] in, This represents the distillation loss function of the Logits output layer. This represents the Logits vector output by the teacher network in the detection head classification branch. This represents the Logits vector output by the student network in the classification branch of the detection head, where T represents the distillation temperature. This represents the Softmax function.

[0018] In one optional implementation, calculating the basic detection loss function includes:

[0019]

[0020]

[0021]

[0022]

[0023] in, This represents the basic detection loss function. This represents the bounding box regression loss function. Represents the classification loss function. The weight coefficients represent the bounding box loss. The weighting coefficients represent the classification loss. This represents the intersection-union ratio (IU) between the predicted bounding box and the ground truth bounding box, measuring the degree of overlap between them. Indicates the center point of the prediction box. Represents the center point of the true bounding box. This represents the Euclidean distance between two center points. This represents the diagonal length of the smallest bounding rectangle that simultaneously contains the predicted bounding box and the ground truth bounding box, used to normalize the center point distance. This represents the weighting parameter, used to control the impact of the aspect ratio penalty term. This represents a parameter used to measure aspect ratio consistency. This indicates the width and height of the prediction box. This represents the width and height of the actual bounding box. This represents the total number of positive samples, where positive samples include candidate bounding boxes for ship targets. Let represent the true class label of the i-th sample. If the sample belongs to a specific ship category, then... ,otherwise , This represents the probability value that the model predicts the i-th sample belongs to this category, ranging from 0 to 1.

[0024] In an optional implementation, a composite total training loss function is constructed based on the explicit modality alignment loss, the basic detection loss, and the distillation loss function of the student network, including:

[0025] in, This represents the basic detection loss function. Indicates the weights of the basic detection loss function. This represents the feature adaptation layer distillation loss function. The weights of the feature adaptation layer distillation loss function are represented. Indicates the modal alignment loss. Indicates the modal alignment loss weights. This represents the distillation loss function of the Logits output layer. This represents the weights of the distillation loss function in the Logits output layer.

[0026] In an optional implementation, visible light multi-scale features and thermal infrared multi-scale features are spliced ​​and fused, and the fused features are recalibrated using a coordinate attention mechanism to obtain a fused feature map, including: For any scale layer L, features of the same scale are fused along the channel dimension to obtain fused input features. :

[0027] in, This represents the fused input features of the scale layer L. This represents the multi-scale characteristics of visible light. Represents the multi-scale characteristics of thermal infrared radiation; Global average pooling is performed along both the width and height directions on the fused input features to obtain directional aggregated features:

[0028]

[0029] Where c represents the channel, h represents the height, and w represents the width. and These represent the height and width of the scale layer L, respectively. Indicates height-direction features, Indicates width-direction features; The height and width features are concatenated to obtain attention weights, and the fused input features are then processed based on these attention weights. Weighting yields the fused feature map. :

[0030] in, This represents the fused feature map of scale layer L. This represents the height-oriented attention weight. This represents the attention weight in the width direction.

[0031] In an optional implementation, the fused feature map is input into a pre-built detection head network to predict the position parameters and category information of the ship target, thereby obtaining the target detection result, including: The fused feature maps at various scales are input into a pre-constructed detection head network to form a candidate detection result set. The candidate detection result set includes the location parameters, confidence score, and class probability vector of each candidate detection result.

[0032] in, Let N represent the set of candidate detection results, and let N represent the number of candidate targets. This represents the bounding box position parameter of the i-th candidate target. Let represent the confidence score of the i-th candidate target. This represents the category probability vector of the i-th candidate target; The candidate detection result set is filtered based on a pre-set reliability threshold to obtain the initial candidate set:

[0033] in, This indicates a preset confidence threshold. Represents the initial candidate set; Calculate the intersection-union ratio (CUI) between candidate bounding boxes in the initial candidate set. For any two candidate bounding boxes... and The intersection-union ratio is:

[0034] in, Represents candidate bounding boxes and The area of ​​the intersection region, Represents candidate bounding boxes and The area of ​​the union region; When the intersection-union ratio of two candidate bounding boxes is greater than a preset threshold, a set of target detection results is obtained.

[0035] The beneficial effects of this invention are as follows: The multimodal ship target detection method based on asymmetric architecture knowledge distillation provided by this invention acquires visible light and thermal infrared images of ships in the same maritime scenario, and inputs them into the visible light branch network structure and thermal infrared branch network structure of the student network pre-trained using the teacher network for multi-scale feature extraction. This allows the model to simultaneously utilize the rich texture, color, and structural information in the visible light image, as well as the stable target contour and temperature distribution features in the thermal infrared image, thereby effectively improving the robustness of ship target detection under conditions such as nighttime, backlight, fog, and complex sea conditions. Through the asymmetric dual-stream feature extraction structure, different modal branches perform differentiated feature modeling based on their respective imaging characteristics, reducing model computational redundancy while ensuring detection accuracy. This improves the model's applicability in lightweight and practical deployment scenarios. By outputting visible light multi-scale features and thermal infrared multi-scale features, the network can take into account both long-distance small-scale ships and short-distance large-scale ships, enhancing the detection capability of ships of different scales. A coordinate attention mechanism is introduced to fuse multimodal and multi-scale features, enabling the network to simultaneously focus on the spatial location information and channel semantic information of the features during the fusion process. This highlights the relevant areas of the ship target, suppresses sea surface background noise, and improves the discrimination capability of the fused feature map for ship targets. The fused feature map is then input into the detection head network to predict the position parameters and category information of the ship target, effectively improving the detection results in terms of target positioning accuracy and category recognition accuracy, meeting the application needs of ship target detection in complex maritime scenarios.

[0036] This application also introduces a lightweight feature adaptation layer after the FPN feature output of the student network, projecting the low-dimensional features of the student network onto a high-dimensional feature space consistent with that of the teacher network. This eliminates the mathematical inconsistency caused by the different number of channels between the teacher and student networks, enabling feature layer distillation to be performed stably and effectively. By introducing distillation constraints simultaneously in the multi-scale feature adaptation layer and the detection head classification output layer, the student network not only approximates the teacher network at the intermediate feature representation level, but also learns the soft label distribution of the teacher network at the final classification decision level. This significantly improves the detection accuracy of the lightweight model without significantly increasing the number of model parameters and computational complexity.

[0037] In this application, the teacher network, feature adaptation layer, and distillation and alignment loss are only used in the training phase and are all removed in the inference phase. Only the lightweight asymmetric two-stream student network is retained for object detection, thereby achieving a significant improvement in detection performance without affecting the actual deployment efficiency, and has good engineering application value. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0040] Figure 2 This is a schematic flowchart of a method according to another embodiment of the present invention.

[0041] Figure 3 This is a schematic flowchart of a method according to another embodiment of the present invention.

[0042] Figure 4 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0043] Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0046] The multimodal ship target detection method based on asymmetric architecture knowledge distillation provided in this embodiment of the invention is executed by a computer device, and correspondingly, the multimodal ship target detection system based on asymmetric architecture knowledge distillation runs on the computer device.

[0047] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be a multimodal ship target detection system based on asymmetric architecture knowledge distillation. Depending on different requirements, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0048] like Figure 1 and Figure 2 As shown, the method includes: S1. Construct a student detection model, which includes a visible light branch network, a thermal infrared branch network, a feature fusion module, and a detection head network. Construct a teacher model, which is a pre-trained multimodal ship target detection model.

[0049] S2. During the training phase, visible light and thermal infrared images of the ship are acquired. The same visible light and thermal infrared images are simultaneously input into the pre-trained teacher model and student detection model. The visible light branch network and thermal infrared branch network in the student detection model are asymmetric network structures.

[0050] S3, the teacher model and the student detection model, are based on the input visible light image and thermal infrared image, and output teacher feature information and teacher detection results, and student feature information and student detection results, respectively.

[0051] S4. Based on the differences between teacher and student feature information and the differences between teacher and student detection results, training constraints are constructed to train the student detection model, resulting in a trained student detection model.

[0052] S5. During the inference phase, the visible light image and thermal infrared image of the ship are input into the trained student detection model, and visible light multi-scale features and thermal infrared multi-scale features are obtained based on the visible light branch network and thermal infrared branch network, respectively.

[0053] S6. The visible light multi-scale features and the thermal infrared multi-scale features are spliced ​​and fused, and the fused features are recalibrated using the coordinate attention mechanism to obtain the fused feature map.

[0054] S7. Input the fused feature map into the detection head network to predict the position parameters and category information of the ship target and obtain the target detection result.

[0055] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0056] Acquiring visible light and thermal infrared images of ships involves: collecting a paired maritime vessel dataset (PoLaRIS dataset) containing both visible light and thermal infrared images; ensuring strict spatial pixel registration between the visible light and thermal infrared images through feature point matching; uniformly resizing the images and dividing them into training and testing sets at an 8:2 ratio; and performing preprocessing such as Mosaic enhancement and random flipping on the images. Pairs of visible light and infrared images are then used as input to the designed model for training. Specifically, considering the synchronization of multimodal data, the registered visible light images are used... and infrared images Pairs of input tensors During the training phase, paired input tensors are fed into both the teacher and student networks simultaneously; during the inference phase, only the lightweight student network is fed in.

[0057] The steps for constructing the visible light branch network and thermal infrared branch network in the student detection model as asymmetric network structures include: MobileNetV2 was chosen as the basic network architecture for the asymmetric network structure. Visible light branch and thermal infrared branch basic network structures were constructed separately. Visible light image sets and thermal infrared image sets were obtained; pixel value statistics were performed on each set of visible light and thermal infrared images, and the statistics for each type of pixel value were calculated. The probability of appearing in the corresponding image set And calculate the average information entropy of the corresponding image set based on the pixel value probability distribution. The formula for calculating average information entropy is:

[0058] in, The information entropy of image set X is represented by n, where n represents the total number of pixel values. Based on the formula for calculating average information entropy, the average information entropy of the visible light image set and the average information entropy of the thermal infrared image set are obtained respectively. When the average information entropy of the visible light image set is greater than that of the thermal infrared image set, it is determined that the amount of information contained in the visible light image set is higher than that contained in the thermal infrared image set. The visible light branch basic network structure is configured as a feature extraction network with standard channel width to extract color information, high-frequency texture information, and complex background information in the visible light images. The thermal infrared branch basic network structure is configured as a channel-clipping feature extraction network to extract target contour information and temperature distribution features in the thermal infrared images. The number of output channels of each network layer in the thermal infrared branch basic network structure is configured according to the channel clipping formula.

[0059] in, This represents the number of output channels for the corresponding network layer in the thermal infrared branch basic network structure. This represents the number of output channels of the corresponding network layer in the visible light branching basic network structure. Indicates the channel clipping ratio, and In the visible light branch and thermal infrared branch basic network structures, standard convolution operations are replaced with depthwise separable convolution operations, so that convolution operations are performed through channel-wise convolution and point-wise convolution respectively. Multi-scale feature output layers are set according to the same network layer position, so that the visible light branch network structure and the thermal infrared branch network structure output multi-scale feature maps with consistent spatial resolution at corresponding layers, resulting in the visible light branch network structure and the thermal infrared branch network structure in the constructed asymmetric network structure.

[0060] Specifically, this step is the core of achieving lightweight model and eliminating computational redundancy. Unlike traditional symmetric two-stream architectures, this method uses an asymmetric design based on the physical entropy difference theory in information theory. In maritime scenarios, visible light images contain color, high-frequency textures, and light and shadow variations, with a wide distribution of histograms and high information entropy. High. Thermal infrared images primarily reflect the temperature field of an object, have poor texture, concentrated histograms, and mainly provide low-frequency contour information, with low information entropy. Lower. Preliminary experiments show... Much larger Based on the principle that network capacity should match the amount of data, the infrared branch does not need to have the same feature extraction capacity as the visible light branch.

[0061] Based on the above theory, an asymmetric dual-stream backbone network is constructed. The visible light main branch uses a lightweight network with standard width, including but not limited to MobileNetV2-1.0x. This branch retains the full number of channels and feature extraction capabilities, responsible for capturing complex sea surface backgrounds and fine ship textures. It includes a complete inverse residual structure and a linear bottleneck layer, with the number of channels configured to standard settings. The thermal infrared auxiliary branch uses a channel-pruned version of the network, including but not limited to MobileNetV2-0.5x, where the number of channels in the convolutional kernels of each layer is halved. In both branches, all standard... Convolution is replaced with depthwise separable convolution. Compared to the symmetric design, the number of infrared branch parameters is reduced, the computational load is decreased, and computational redundancy is eliminated when processing low-entropy infrared data.

[0062] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner. The steps for training student networks using teacher-led network training include: like Figure 3As shown, the number of FPN output channels for the teacher network and the student network are set, with the student network being an asymmetric network structure. A feature adaptation layer is set between the teacher and student networks to align the feature dimensions between them. Global average pooling is performed on the visible light image and the thermal infrared image respectively to obtain global feature vectors. The cosine similarity between the two global feature vectors is calculated to obtain the modality alignment loss. The distillation loss function of the feature adaptation layer and the distillation loss function of the Logits output layer are calculated respectively. Based on the distillation loss function of the feature adaptation layer and the distillation loss function of the Logits output layer, the total distillation loss function of the student network is obtained. The basic detection loss function is calculated. A composite total training loss function is constructed based on the explicit modality alignment loss, the basic detection loss, and the distillation loss function of the student network. The student detection model is trained based on the composite total training loss function to obtain the trained student detection model.

[0063] Specifically, to perform knowledge distillation, a high-performance teacher network needs to be constructed. ResNet-50 is chosen as the backbone of the teacher network. This network has powerful feature extraction capabilities, and although it has a large number of parameters, it is only used to guide the student network during the training phase. The SE module is used in the teacher network for multimodal fusion to ensure that it can fully utilize visible light and infrared information. The teacher network needs to be fully trained on the target dataset, the PoLaRIS dataset, until convergence is achieved, and its weights are frozen. The mAP value of the teacher network will serve as the upper limit benchmark for the performance of the student network.

[0064] A feature adaptation layer is set up between the teacher network and the student network to align the feature dimensions between the two networks, including: Set up between teacher network and student network The convolutional module acts as a feature adaptation layer, transforming the low-dimensional feature maps of the student network. Projected to the teacher network:

[0065] in, The distillation loss function represents the feature adaptation layer. Represents the low-dimensional feature map of the student network. Represents the high-dimensional feature graph of the teacher network. express In the convolutional adaptation operation, C represents the channel, H represents the height, and W represents the width.

[0066] Specifically, this step aims to address the heterogeneous network problem, specifically the mismatch in feature dimensions between the ResNet teacher network and the MobileNet student network, which prevents direct distillation. The ResNet-50 teacher network has a fixed 256 output channels for its FPN layer to ensure rich feature representation. However, the lightweight MobileNetV2 student network designed in this method has its FPN output channels compressed to 96 to control parameter count. Directly calculating the distance between the two feature maps presents a mathematical obstacle due to the dimensionality inconsistency. Therefore, after the student network's FPN feature output and before calculating the distillation loss, a lightweight... The convolutional module acts as a feature adaptation layer, which is to adapt the low-dimensional feature maps of the student network. Projected into the same high-dimensional space as the teacher network.

[0067] This adaptor layer participates only in gradient backpropagation during the training phase to align feature dimensions, thereby allowing the computation of mean squared error loss and forcing the student network to learn the high-dimensional feature distribution pattern of the teacher network in a low-dimensional space. During the testing phase, this adaptor layer is removed.

[0068] Global average pooling is performed on both the visible light image and the thermal infrared image to obtain global feature vectors. The cosine similarity between the two global feature vectors is calculated to obtain the modality alignment loss, including:

[0069] in, Indicates the modal alignment loss. This represents the global feature vector of the visible light modes. This represents the global feature vector of the infrared mode.

[0070] Specifically, this step aims to mitigate the modal laziness problem in lightweight models. During joint training, due to model capacity limitations, the network tends to learn and rely only on the visible light mode, which performs well in most scenarios, gradually abandoning the learning of the infrared mode. This leads to vanishing gradients in the infrared branch, turning it into a noise bypass. Therefore, to alleviate the modal laziness problem in lightweight models, an explicit modal alignment loss is designed to force the network to align heterogeneous features in the semantic direction. This loss is positioned after the output of the dual-stream backbone network and before the input of the multimodal fusion module, specifically for the visible light feature maps. and infrared feature map Perform global average pooling to obtain global feature vectors for both modalities, and then calculate the modal alignment loss. , is defined as the negative value of the cosine similarity between the two.

[0071] This constraint allows the feature vectors of two modalities to differ in modulus length, but mandates that their orientations in the high-dimensional semantic space converge. That is, regardless of whether the input is a visible light or thermal infrared image, as long as it is the same target, its feature vectors should point to the same semantic category. This fundamentally prevents the separation of feature spaces, ensuring that the fusion module receives semantically aligned input.

[0072] The calculation of the distillation loss function for the feature adaptation layer includes: In the feature adaptation layer, feature layers P3, P4, and P5 are selected. The distance between the student network and the teacher network after feature adaptation is calculated, and the distillation loss function of the feature adaptation layer is obtained:

[0073] in, The distillation loss function represents the feature adaptation layer. Represents the feature adaptation layer. The teacher network is in the first The feature map output from the feature layer Indicates the student network in the first Feature map of the feature layer after feature adaptation.

[0074] Calculate the distillation loss function of the Logits output layer, including:

[0075] in, This represents the distillation loss function of the Logits output layer. This represents the Logits vector output by the teacher network in the detection head classification branch. This represents the Logits vector output by the student network in the classification branch of the detection head, where T represents the distillation temperature. This represents the Softmax function.

[0076] The calculation of the basic detection loss function includes:

[0077]

[0078]

[0079]

[0080]

[0081] in, This represents the basic detection loss function. This represents the bounding box regression loss function. Represents the classification loss function. The weight coefficients represent the bounding box loss. The weighting coefficients represent the classification loss. This represents the intersection-union ratio (IU) between the predicted bounding box and the ground truth bounding box, measuring the degree of overlap between them. Indicates the center point of the prediction box. Represents the center point of the true bounding box. This represents the Euclidean distance between two center points. This represents the diagonal length of the smallest bounding rectangle that simultaneously contains the predicted bounding box and the ground truth bounding box, used to normalize the center point distance. This represents the weighting parameter, used to control the impact of the aspect ratio penalty term. This represents a parameter used to measure aspect ratio consistency. This indicates the width and height of the prediction box. This represents the width and height of the actual bounding box. This represents the total number of positive samples, where positive samples include candidate bounding boxes for ship targets. Let represent the true class label of the i-th sample. If the sample belongs to a specific ship category, then... ,otherwise , This represents the probability value that the model predicts the i-th sample belongs to this category, ranging from 0 to 1.

[0082] A composite total training loss function is constructed based on explicit modality alignment loss, basic detection loss, and distillation loss function of the student network, including:

[0083] in, This represents the basic detection loss function. Indicates the weights of the basic detection loss function. This represents the feature adaptation layer distillation loss function. The weights of the feature adaptation layer distillation loss function are represented. Indicates the modal alignment loss. Indicates the modal alignment loss weights. This represents the distillation loss function of the Logits output layer. This represents the weights of the distillation loss function in the Logits output layer.

[0084] In one embodiment of the present invention, based on step S6, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0085] Visible light multi-scale features and thermal infrared multi-scale features are spliced ​​and fused, and the fused features are recalibrated using a coordinate attention mechanism to obtain a fused feature map, including: For any scale layer L, features of the same scale are fused along the channel dimension to obtain fused input features. :

[0086] in, This represents the fused input features of the scale layer L. This represents the multi-scale characteristics of visible light. Represents the multi-scale characteristics of thermal infrared radiation; Global average pooling is performed along both the width and height directions on the fused input features to obtain directional aggregated features:

[0087]

[0088] Where c represents the channel, h represents the height, and w represents the width. and These represent the height and width of the scale layer L, respectively. Indicates height-direction features, Indicates width-direction features; The height and width features are concatenated to obtain attention weights, and the fused input features are then processed based on these attention weights. Weighting yields the fused feature map. :

[0089] in, This represents the fused feature map of scale layer L. This represents the height-oriented attention weight. This represents the attention weight in the width direction.

[0090] Specifically, to obtain accurate detection boxes, a lightweight detection head is constructed. The input to this module is the feature maps at three scales fused in step 3. The output is the predicted object detection results at the three scales. A lightweight, decoupled detection head based on anchor boxes is adopted, predicting three anchor boxes per grid. Each anchor box contains four location coordinates (cx, cy, w, h), one confidence score, and one class probability. The detection head also uses depthwise separable convolutions to maintain its lightweight nature.

[0091] In one embodiment of the present invention, based on step S7, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0092] The fused feature map is input into a pre-built detection head network to predict the position parameters and category information of the ship target, resulting in target detection results, including: The fused feature maps at various scales are input into a pre-constructed detection head network to form a candidate detection result set. The candidate detection result set includes the location parameters, confidence score, and class probability vector of each candidate detection result.

[0093] in, Let N represent the set of candidate detection results, and let N represent the number of candidate targets. This represents the bounding box position parameter of the i-th candidate target. Let represent the confidence score of the i-th candidate target. This represents the category probability vector of the i-th candidate target; The candidate detection result set is filtered based on a pre-set reliability threshold to obtain the initial candidate set:

[0094] in, This indicates a preset confidence threshold. Represents the initial candidate set; Calculate the intersection-union ratio (CUI) between candidate bounding boxes in the initial candidate set. For any two candidate bounding boxes... and The intersection-union ratio is:

[0095] in, Represents candidate bounding boxes and The area of ​​the intersection region, Represents candidate bounding boxes and The area of ​​the union region; when the intersection-union ratio of two candidate bounding boxes is greater than a preset threshold, the target detection result set is obtained.

[0096] In one possible implementation, this method is implemented within the PyTorch deep learning framework. The input image size is set to... The training batch size was set to 16, and the total number of training epochs was set to 300. Adam was used as the model optimizer, with a momentum parameter set to 0.9 and weight decay set to... The initial learning rate was set to 0.001, and a cosine annealing strategy was used to dynamically adjust the learning rate to avoid the model getting trapped in local optima. To overcome the training difficulties that the lightweight asymmetric architecture might bring, a step-by-step initialization strategy was adopted. First, network initialization was performed, using the SeaShips pre-trained visible light branch to improve the main branch's ability to extract ship texture features. To prevent the infrared branch from underfitting due to the asymmetric design, the infrared branch was initialized using weights pre-trained on ImageNet. Then, joint training was performed, feeding the input data into the entire model, and forward propagation was used to calculate the total loss. The parameters of the student network backbone, fusion layer, detection head, and feature adaptation layer are updated simultaneously through backpropagation, while the teacher network parameters remain frozen. Finally, during the actual inference phase, the teacher network, feature adaptation layer, and alignment loss calculation module are removed, leaving only the lightweight asymmetric student network. Visible light and thermal infrared images are input, and the output is the category and location coordinates of the ship target.

[0097] As can be seen, this application constructs an asymmetric two-stream lightweight architecture based on physical entropy difference and a coordinate attention fusion module. Existing multimodal detection models typically employ a symmetric two-stream structure, ignoring the fundamental difference in information content between visible light and infrared images. Since infrared images mainly reflect low-frequency thermal contours, their information entropy is much lower than that of visible light images. Symmetrical designs lead to significant computational redundancy in the infrared branch, and simple compression results in a decrease in feature extraction capabilities. Compared to traditional symmetric frameworks, this method eliminates computational redundancy in the infrared branch at the source through an asymmetric configuration of visible light branch standardization and infrared branch pruning. Simultaneously, to address the potential loss of spatial information due to pruning, a coordinate attention mechanism is introduced to simultaneously capture cross-channel information and long-distance spatial dependencies. This allows the model to lock onto ship targets while reducing the number of parameters and computational load, achieving a balance between detection speed and accuracy.

[0098] This application introduces an explicit modal alignment constraint mechanism based on cosine similarity. In the training process of lightweight multimodal models, modal laziness exists; that is, due to limited model capacity, the network tends to over-rely on the information-rich visible light modality while neglecting the learning of infrared features. This leads to performance degradation in scenarios where visible light is unavailable, such as at night, thus negating the significance of multimodal fusion. This method forces visible light and infrared features to maintain semantic consistency by calculating the cosine similarity of global feature vectors. This explicit constraint does not rely on complex generative adversarial networks, prevents feature space separation with low computational cost, ensures that the model can fully utilize the complementary information of the infrared branch, and improves detection robustness under all-weather conditions.

[0099] This application designs a systematic knowledge distillation framework including a feature adaptation layer. Traditional knowledge distillation methods, when dealing with heterogeneous networks, cannot effectively transfer intermediate layer features containing rich spatial and semantic information due to differences in feature channel dimensions. This method overcomes the limitation of dimensionality differences by designing a lightweight 1×1 convolutional feature adaptation layer to losslessly project the low-dimensional features of the student network to the high-dimensional space of the teacher network. Combining the main distillation of the FPN multi-scale feature layer with the auxiliary distillation of the output layer Logits, this method successfully transfers the spatial localization and semantic discrimination capabilities of high-performance large models to lightweight small models, recovering the accuracy loss caused by model compression.

[0100] The model in this application can obtain efficient feature representations based on an asymmetric lightweight backbone network, thereby improving inference speed and meeting the deployment requirements of lightweight devices. It can also utilize feature adaptation and systematic distillation to enable the lightweight model to have detection accuracy close to that of a high-performance teacher network. By introducing explicit modality alignment loss to train the network, it learns as consistent a semantic direction as possible for features of the same target in different modalities, thereby solving the modality laziness problem and ultimately achieving all-weather multimodal ship target detection.

[0101] In some embodiments, the multimodal ship target detection system based on asymmetric architecture knowledge distillation may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the multimodal ship target detection system based on asymmetric architecture knowledge distillation may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality of multimodal ship target detection based on asymmetric architecture knowledge distillation.

[0102] In this embodiment, the multimodal ship target detection system based on asymmetric architecture knowledge distillation can be divided into multiple functional modules according to its functions, such as... Figure 4 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0103] The image acquisition module is used to acquire visible light and thermal infrared images of ships in the same maritime scenario.

[0104] The feature output module is used to input visible light images and thermal infrared images into the visible light branch network structure and thermal infrared branch network structure of the student network pre-trained by the teacher network, respectively, and output visible light multi-scale features and thermal infrared multi-scale features.

[0105] The feature fusion module is used to stitch together visible light multi-scale features and thermal infrared multi-scale features, and uses a coordinate attention mechanism to recalibrate the fused features to obtain a fused feature map.

[0106] The target detection module is used to input the fused feature map into the pre-built detection head network, predict the position parameters and category information of the ship target, and obtain the target detection result.

[0107] Figure 5 The multimodal ship target detection method based on asymmetric architecture knowledge distillation provided in this application embodiment can be applied to devices. Those skilled in the art will understand that the device structure involved in the embodiments of this invention does not constitute a limitation on the device; the device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0108] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0109] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is able to perform some or all of the steps in the above method embodiments.

[0110] The processor 310 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0111] The communication unit 330 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0112] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0113] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0114] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0115] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0116] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0118] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A multimodal ship target detection method based on asymmetric architecture knowledge distillation, characterized in that, include: A student detection model is constructed, which includes a visible light branch network, a thermal infrared branch network, a feature fusion module, and a detection head network. A teacher model is also constructed, which is a pre-trained multimodal ship target detection model. During the training phase, visible light and thermal infrared images of the ship are acquired. The same visible light and thermal infrared images are simultaneously input into the pre-trained teacher model and student detection model. The visible light branch network and thermal infrared branch network in the student detection model are asymmetric network structures. The teacher model and student detection model are based on the input visible light image and thermal infrared image, and output teacher feature information and teacher detection results, and student feature information and student detection results, respectively. Based on the differences between teacher and student feature information and the differences between teacher and student detection results, training constraints are constructed to train the student detection model, resulting in a trained student detection model. During the inference phase, visible light and thermal infrared images of the ship are input into the trained student detection model, and visible light multi-scale features and thermal infrared multi-scale features are obtained based on the visible light branch network and the thermal infrared branch network, respectively. Visible light multi-scale features and thermal infrared multi-scale features are spliced ​​and fused, and the fused features are recalibrated using a coordinate attention mechanism to obtain a fused feature map; The fused feature map is input into the detection head network to predict the position parameters and category information of the ship target, thus obtaining the target detection result.

2. The method according to claim 1, characterized in that, The steps for constructing the visible light branch network and thermal infrared branch network in the student detection model as asymmetric network structures include: MobileNetV2 was chosen as the basic network architecture for the asymmetric network structure, and visible light branch basic network structure and thermal infrared branch basic network structure were constructed respectively. Acquire a set of visible light images and a set of thermal infrared images; Pixel value statistics were performed on both the visible light image set and the thermal infrared image set, and the values ​​of each pixel value were statistically analyzed. The probability of appearing in the corresponding image set And calculate the average information entropy of the corresponding image set based on the pixel value probability distribution. The formula for calculating average information entropy is: in, The information entropy of image set X is represented by n, where n represents the total number of pixel values. Based on the formula for calculating average information entropy, the average information entropy of the visible light image set and the average information entropy of the thermal infrared image set are obtained respectively. When the average information entropy of the visible light image set is greater than the average information entropy of the thermal infrared image set, it is determined that the visible light image set contains more information than the thermal infrared image set. The visible light branch basic network structure is configured as a feature extraction network with standard channel width to extract color information, high-frequency texture information and complex background information in visible light images; The thermal infrared branch basic network structure is configured as a channel-clipping feature extraction network to extract target contour information and temperature distribution features from thermal infrared images. The number of output channels in each network layer of the thermal infrared branch basic network structure is configured according to the channel clipping formula: in, This represents the number of output channels for the corresponding network layer in the thermal infrared branch basic network structure. This represents the number of output channels of the corresponding network layer in the visible light branching basic network structure. Indicates the channel clipping ratio, and ; In the visible light branch and thermal infrared branch basic network structures, standard convolution operations are replaced with depthwise separable convolution operations, so that convolution operations are performed through channel-wise convolution and point-wise convolution respectively. Multi-scale feature output layers are set according to the same network layer position, so that the visible light branch network structure and the thermal infrared branch network structure output multi-scale feature maps with consistent spatial resolution at the corresponding layer, resulting in the visible light branch network structure and the thermal infrared branch network structure in the constructed asymmetric network structure.

3. The method according to claim 1, characterized in that, Also includes: The steps for training student networks using teacher-led network training include: Set the number of FPN output channels for the teacher network and the number of FPN output channels for the student network, wherein the student network is an asymmetric network structure; A feature adaptation layer is set up between the teacher network and the student network to align the feature dimensions between the teacher network and the student network. Global average pooling is performed on the visible light image and the thermal infrared image respectively to obtain global feature vectors. The cosine similarity between the two global feature vectors is calculated to obtain the modality alignment loss. The distillation loss function of the feature adaptation layer and the distillation loss function of the Logits output layer are calculated separately. A composite total training loss function is constructed based on the explicit modality alignment loss, the basic detection loss, and the distillation loss function of the student network. Calculate the basic detection loss function; A composite total training loss function is constructed based on explicit modality alignment loss, basic detection loss, and distillation loss function of the student network. The student detection model is then trained based on the composite total training loss function to obtain a well-trained student detection model.

4. The method according to claim 3, characterized in that, A feature adaptation layer is set up between the teacher network and the student network to align the feature dimensions between the two networks, including: Set up between teacher network and student network The convolutional module acts as a feature adaptation layer, transforming the low-dimensional feature maps of the student network. Projected to the teacher network: in, The distillation loss function represents the feature adaptation layer. Represents the low-dimensional feature map of the student network. Represents the high-dimensional feature graph of the teacher network. express In the convolutional adaptation operation, C represents the channel, H represents the height, and W represents the width.

5. The method according to claim 3, characterized in that, Global average pooling is performed on both the visible light image and the thermal infrared image to obtain global feature vectors. The cosine similarity between the two global feature vectors is calculated to obtain the modality alignment loss, including: in, Indicates the modal alignment loss. This represents the global feature vector of the visible light modes. This represents the global feature vector of the infrared mode.

6. The method according to claim 4, characterized in that, The calculation of the distillation loss function for the feature adaptation layer includes: In the feature adaptation layer, feature layers P3, P4, and P5 are selected. The distance between the student network and the teacher network after feature adaptation is calculated, and the distillation loss function of the feature adaptation layer is obtained: in, The distillation loss function represents the feature adaptation layer. Represents the feature adaptation layer. The teacher network is in the first The feature map output from the feature layer Indicates the student network in the first Feature map of the feature layer after feature adaptation; Calculate the distillation loss function of the Logits output layer, including: in, This represents the distillation loss function of the Logits output layer. This represents the Logits vector output by the teacher network in the detection head classification branch. This represents the Logits vector output by the student network in the classification branch of the detection head, where T represents the distillation temperature. This represents the Softmax function.

7. The method according to claim 3, characterized in that, The calculation of the basic detection loss function includes: in, This represents the basic detection loss function. This represents the bounding box regression loss function. Represents the classification loss function. The weight coefficients represent the bounding box loss. The weighting coefficients represent the classification loss. This represents the intersection-union ratio (IU) between the predicted bounding box and the ground truth bounding box, measuring the degree of overlap between them. Indicates the center point of the prediction box. Represents the center point of the true bounding box. This represents the Euclidean distance between two center points. This represents the diagonal length of the smallest bounding rectangle that simultaneously contains the predicted bounding box and the ground truth bounding box, used to normalize the center point distance. This represents the weighting parameter, used to control the impact of the aspect ratio penalty term. This represents a parameter used to measure aspect ratio consistency. This indicates the width and height of the prediction box. This represents the width and height of the actual bounding box. This represents the total number of positive samples, where positive samples include candidate bounding boxes for ship targets. Let represent the true class label of the i-th sample. If the sample belongs to a specific ship category, then... ,otherwise , This represents the probability value that the model predicts the i-th sample belongs to this category, ranging from 0 to 1.

8. The method according to claim 3, characterized in that, A composite total training loss function is constructed based on explicit modality alignment loss, basic detection loss, and distillation loss function of the student network, including: in, This represents the basic detection loss function. Indicates the weights of the basic detection loss function. This represents the feature adaptation layer distillation loss function. The weights of the feature adaptation layer distillation loss function are represented. Indicates the modal alignment loss. Indicates the modal alignment loss weights. This represents the distillation loss function of the Logits output layer. This represents the weights of the distillation loss function in the Logits output layer.

9. The method according to claim 1, characterized in that, Visible light multi-scale features and thermal infrared multi-scale features are spliced ​​and fused, and the fused features are recalibrated using a coordinate attention mechanism to obtain a fused feature map, including: For any scale layer L Features of the same scale are fused along the channel dimension to obtain fused input features. : in, This represents the fused input features of the scale layer L. This represents the multi-scale characteristics of visible light. Represents the multi-scale characteristics of thermal infrared radiation; Global average pooling is performed along both the width and height directions on the fused input features to obtain directional aggregated features: Where c represents the channel, h represents the height, and w represents the width. and These represent the height and width of the scale layer L, respectively. Indicates height-direction features, Indicates width-direction features; The height and width features are concatenated to obtain attention weights, and the fused input features are then processed based on these attention weights. Weighting yields the fused feature map. : in, This represents the fused feature map of scale layer L. This represents the height-oriented attention weight. This represents the attention weight in the width direction.

10. The method according to claim 1, characterized in that, The fused feature map is input into a pre-built detection head network to predict the position parameters and category information of the ship target, resulting in target detection results, including: The fused feature maps at various scales are input into a pre-constructed detection head network to form a candidate detection result set. The candidate detection result set includes the location parameters, confidence score, and class probability vector of each candidate detection result. in, Let N represent the set of candidate detection results, and let N represent the number of candidate targets. This represents the bounding box position parameter of the i-th candidate target. Let represent the confidence score of the i-th candidate target. This represents the category probability vector of the i-th candidate target; The candidate detection result set is filtered based on a pre-set reliability threshold to obtain the initial candidate set: in, This indicates a preset confidence threshold. Represents the initial candidate set; Calculate the intersection-union ratio (CUI) between candidate bounding boxes in the initial candidate set. For any two candidate bounding boxes... and The intersection-union ratio is: in, Represents candidate bounding boxes and The area of ​​the intersection region, Represents candidate bounding boxes and The area of ​​the union region; When the intersection-union ratio of two candidate bounding boxes is greater than a preset threshold, a set of target detection results is obtained.

Images