Multi-source information small target ship identification and abnormity study and judgment method and device

Abstract

The invention relates to the technical field of computer vision and deep learning, in particular to a multi-source information small target ship identification and anomaly study and judgment method and device, and aims to improve the ship detection performance under a high-resolution remote sensing image by introducing a small target detection algorithm model based on an optical remote sensing image. The invention provides an improved YOLOv8x-CA-CFAR model and further provides a novel unsupervised domain adaptive classification model Multi-CDT in order to solve the problems that small target ship pixels in an optical remote sensing image are few and the features are not obvious, so that the transfer learning ability of different source data is enhanced, and the classification performance under the condition of lacking annotated data is improved. Finally, by constructing a complete multi-source information fusion framework, more accurate small target ship detection and classification are realized, the situation of matching failure of the optical remote sensing image and the AIS data can be effectively handled, and support for anomaly study and judgment is provided.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a method and device for identifying small target ships and abnormality analysis based on multi-source information. Background Art

[0002] Traditional ship identification relies on Automatic Identification System (AIS) data. Although AIS can provide real-time location information of ships, it has some obvious limitations. AIS equipment can be turned off or data can be modified manually, resulting in the loss of some ship information, especially in illegal activities. In addition, some small ships do not install AIS equipment, further exacerbating the problem of data incompleteness. This makes it difficult to effectively cover all ships by monitoring methods that rely solely on AIS. In order to make up for the shortcomings of AIS data, shore-based radar has been widely used in monitoring nearshore areas. Although radar systems can provide supplementary detection for offshore ships, their effective range is limited to nearshore areas, while offshore monitoring still relies on satellite remote sensing images. In addition, radar signals are easily affected by external conditions such as weather and sea conditions, and data quality may be unstable, resulting in reduced monitoring accuracy. This instability increases the complexity in actual operations and limits the application of radar technology in a wider range of fields.

[0003] Multi-source information fusion methods based on optical remote sensing images and AIS data provide stronger capabilities for ship detection. However, these methods are highly dependent on the successful matching of images and AIS data, and in practical applications, matching failures are not uncommon. Existing research lacks further research and judgment mechanisms for such abnormal matching failures, which is a major loophole in the monitoring of potential illegal or abnormal activities, resulting in some abnormal ships not being discovered in time. This is a key problem in this field.

[0004] In terms of ship classification, although the classification technology of SAR images combined with AIS data has made progress, it is still insufficient in the field of optical remote sensing images. In particular, when dealing with unlabeled data, existing supervised learning methods are difficult to apply. Although unsupervised domain adaptation technology provides a solution to this problem, its application in multi-source heterogeneous data fusion still faces challenges. Although traditional manual feature extraction methods can process multi-source data, they are highly complex and have poor adaptability, making it difficult to cope with the changing situations in actual operations. Based on these problems, how to effectively apply transfer learning on multi-source heterogeneous data and improve the generalization ability of the model has become a key difficulty in this field. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose a method and device for small target ship identification and abnormal analysis based on multi-source information, so as to solve the problems of strong dependence on AIS data, limited monitoring range of nearshore radar, and lack of effective abnormal analysis mechanism after failure of multi-source information matching.

[0006] Based on the above purpose, the present invention provides a method for identifying small target ships and abnormality analysis based on multi-source information, comprising the following steps:

[0007] S1, collect optical remote sensing images and ship automatic identification system data, and perform preprocessing;

[0008] S2. Input the optical remote sensing image into the constructed small target ship detection model YOLOv8x-CA-CFAR based on constant false alarm rate secondary judgment to obtain the small target ship detection result. The small target ship detection model YOLOv8x-CA-CFAR first uses a deep learning algorithm to detect suspicious objects for slicing, and then uses CA-CFAR to distinguish the slices to obtain the small target ship detection result;

[0009] S3, matching the small target ship detection result with the pre-processed ship automatic identification system data, and if the match is successful, outputting the matching result;

[0010] S4. When the matching fails, it is determined that there is an abnormal situation, and the abnormal situation is judged. When it is determined that there is no ship around the ship automatic identification system data, the ship automatic identification system data is considered to be abnormal data and is retained;

[0011] S5. If it is determined that there is no automatic ship identification system data around the ship, the corresponding optical remote sensing image is sliced ​​according to the detection result, and input into the trained unsupervised domain adaptive classification model named Multi-CDT to obtain the predicted category of the abnormal ship;

[0012] S6. The ship information and the ship automatic identification system data in the matching results are retained. For the unmatched ships, the abnormal data obtained in S4 and the coordinates and predicted categories of the abnormal ships obtained in S2 and S5 are retained.

[0013] Preferably, the pretreatment comprises:

[0014] The optical remote sensing image is processed into a 640×640 8-bit RGB image by color depth conversion and image cropping;

[0015] Through time screening, using the updatetime field in the ship automatic identification system data, ensure that the selected ship automatic identification system data is consistent with the shooting time range of the optical remote sensing image;

[0016] Use the offset information in the ship automatic identification system data to perform coordinate correction;

[0017] The fused coordinates are obtained by averaging multiple ship automatic identification system position points after time screening and coordinate correction.

[0018] Preferably, the small target ship detection model YOLOv8x-CA-CFAR temporarily retains the detection box with a confidence level between 0.05 and 0.25, and uses the CA-CFAR algorithm to further analyze the suspicious area. If CA-CFAR confirms the existence of a ship, the detection box is retained, otherwise, it is discarded.

[0019] Preferably, before performing the matching in step S3, the method further includes:

[0020] The pre-processed Automatic Identification System data are spatially screened to exclude the Automatic Identification System data that exceeds the image coverage.

[0021] Preferably, the matching process in step S3 adopts the Hungarian algorithm, and the matching formula is:

[0022]

[0023] Where M represents a set of matching pairs that meet the conditions, which includes valid matching pairs of ship and AIS information, S = {s1, s2, ..., s n} represents a group of ships to be matched, A={a1,a2,...,a n} indicates the corresponding AIS information. The min and arrow at the top of the formula indicate that this is a minimization problem, the goal is to minimize the value of the objective function. Subject to is a constraint, indicating some conditions that must be met in the problem. means that each i must be assigned to only one j, means that each j must also receive only one assignment from i, d ij Indicates i with a j The distance between ij Indicates i Is it related to a j matching, τ represents the distance threshold.

[0024] Preferably, the construction process of the unsupervised domain adaptive classification model named Multi-CDT includes:

[0025] Using the FUSAR dataset, we extract the AIS data features in the dataset, and use these AIS data features in Multi-CDT to train another classifier based on the AIS data features.

[0026] A classifier based on the data features of the automatic ship identification system is used as the teacher model, and distillation learning is used to guide the pre-training of the student model ViT, a classification model based on remote sensing images. The loss functions used in pre-training include classification loss and distillation loss.

[0027] Preferably, the calculation formula for classification loss is:

[0028]

[0029] Where N is the number of samples in the batch, C is the number of categories, x is the log probability of the model output, and x i,j represents the model output of the jth class corresponding to the i-th sample, log represents the natural logarithm, e is the base of the natural logarithm, which is used to calculate the exponent, y i ' ,j represents the smoothed label;

[0030]

[0031] Where C is the number of categories, y is the real label of the one-hot encoding, the label smoothing parameter is ε, and y i ' ,j Represents the smoothed label.

[0032] Preferably, the calculation formula of distillation loss is:

[0033]

[0034] Among them, N is the number of samples (i.e. batch size), K is the number of categories, and Ais_score ik Indicates the probability of the i-th sample in the k-th category given by the AIS information classifier, Image_score ik It represents the probability score of the i-th sample in the k-th category given by the image classifier, and log represents the logarithmic operation.

[0035] Preferably, XGBoost is used as a classifier model based on the data features of the automatic ship identification system.

[0036] The present invention also provides a multi-source information small target ship recognition and abnormality analysis device, which is used to execute any of the multi-source information small target ship recognition and abnormality analysis methods mentioned above.

[0037] Beneficial effects of the present invention:

[0038] This paper constructs a framework for small target ship recognition and abnormal analysis based on optical remote sensing images and AIS information, filling the gap in related research. The framework is generally divided into two steps: detection and classification, and conducts in-depth research and judgment on abnormal situations. The framework improves the scientificity and reliability of abnormal situation handling through a systematic approach.

[0039] In terms of small target ship detection, the present invention proposes the YOLOv8x-CA-CFAR method. First, YOLOv8x is used to detect suspicious objects and perform slice processing, and then CA-CFAR is used to perform secondary discrimination on the slices. This method makes full use of the advantages of deep learning and CFAR technology, makes up for their respective limitations, and improves the performance of the model. The recall rate and F1-score of YOLOv8x-CA-CFAR are 2.9% and 1.13% higher than those of YOLOv8x, respectively.

[0040] In terms of small target ship classification, this study addresses the research gap in small target ship classification in optical remote sensing images, uses SAR-AIS data to migrate to optical remote sensing images through unsupervised domain adaptive learning methods, and constructs a small target ship classification model suitable for optical remote sensing images. Specifically, a Transformer-based multi-source heterogeneous domain adaptive algorithm Multi-CDT is proposed, which effectively integrates SAR images and AIS information, overcomes the limitations of traditional transfer learning, and improves the accuracy and applicability of the model in complex marine environments. Compared with CDTrans, Multi-CDT not only retains more classification categories, but also improves the accuracy of the target domain by 0.32%, indicating that the teacher model based on AIS features successfully helps the student model extract more robust features. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 This is a framework diagram of small target ship recognition and abnormal analysis based on multi-source information fusion according to an embodiment of the present invention;

[0043] Figure 2 This is a visualization diagram of coordinate fusion before and after correction of an embodiment of the invention;

[0044] Figure 3 A comparison diagram of the unsupervised domain adaptation of an embodiment of the invention;

[0045] Figure 4A diagram of the Multi-CDT network architecture of an embodiment of the invention;

[0046] Figure 5 It is a comparison chart of the test results of three models of the embodiment of the invention;

[0047] Figure 6 It is a source domain pre-training trend diagram of an embodiment of the invention;

[0048] Figure 7 It is a trend diagram of unsupervised domain adaptation training in the target domain of an embodiment of the invention;

[0049] Figure 8 An attention map based on a source domain image (SAR remote sensing image) according to an embodiment of the invention;

[0050] Fig. 9 An attention map based on a target domain image (optical remote sensing image) according to an embodiment of the invention;

[0051] Fig.10 A T-SNE diagram based on a target domain image (optical remote sensing image) according to an embodiment of the invention;

[0052] Fig.11 This is a diagram showing the situation where there is no AIS data in the area according to an embodiment of the invention;

[0053] Fig.12 This is a diagram showing the situation where AIS data is available in the area according to an embodiment of the invention. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0055] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0056] like Figure 1As shown, the embodiment of this specification provides a method for identifying small target ships and abnormality analysis based on multi-source information, including the following steps:

[0057] 1. First, preprocess the image and AIS data respectively.

[0058] Image preprocessing:

[0059] Since conventional optical remote sensing images have a large resolution and the number of channels may vary, remote sensing images need to be preprocessed. On the one hand, the color depth needs to be converted, and on the other hand, they need to be cropped. Finally, all input images will be preprocessed into 640×640 8-bit RGB images.

[0060] AIS data preprocessing:

[0061] In practical applications, due to the different sampling characteristics of remote sensing images and AIS information, remote sensing images capture the static instantaneous position of ships, while AIS information is frequently updated to reflect the dynamic trajectory of ships. This difference may lead to the inconsistency of the position of the same ship in the two.

[0062] In order to improve the accuracy in matching, this study conducted multi-faceted preprocessing of AIS data, including time screening, coordinate correction, and coordinate fusion.

[0063] First, through time screening, the updatetime field in the AIS data is used to ensure that the selected AIS data is consistent with the image shooting time range to avoid mismatching due to a large time span. The time screening formula is shown in Equation 1.

[0064] S1={x∈S|T start ≤t(x)≤T end} (1)

[0065] Here, let the time update column be t, that is, t = df[updatetime], t(x) represents the time corresponding to the entry x in the data set, T start Indicates the image start time, T end Indicates the end time of the image, S1 represents the time interval [T start ,T end ] all entries within ].

[0066] Secondly, the offset information in the AIS is used to perform coordinate correction to ensure that the position data is more accurate. The coordinate correction formulas are shown in Equations 2 and 3.

[0067]

[0068] Among them, OffsetA represents the east-west offset of the ship's position reported by AIS, in meters, a positive value indicates that the ship is on the east side, and a negative value indicates that the ship is on the west side. OffsetB represents the north-south offset of the ship's position reported by AIS, in meters, a positive value indicates that the ship is on the north side, and a negative value indicates that the ship is on the south side. Lon, Lat represent longitude and latitude respectively, Lon_corrected and Lat_corrected represent longitude and latitude after coordinate correction respectively. k is an empirical value or calibration factor, which takes into account the magnified meters and converts them into suitable degree units to adjust the longitude and latitude positions, and the value here is 60000.

[0069] Then, by averaging the selected multiple AIS position points, more reliable fusion coordinates are obtained to reduce the impact of time error on matching. The coordinate fusion formulas are shown in Equations 4 and 5.

[0070]

[0071] Among them, n = |S1|, that is, the number of data entries that meet the conditions after time screening, Lon_corrected i and Lon_corrected i Respectively represent the longitude and latitude of the i-th coordinate after coordinate correction, Lon avg and Lat avg Represent the fused longitude and latitude respectively.

[0072] exist Figure 2 In the figure, the fusion coordinates before and after correction are visually evaluated. The yellow dot is the detected ship position, and the green dot is the AIS information position. It can be found that the corrected AIS information position is closer to the position of the ship detected on the image than the uncorrected position. Therefore, it can be considered that the preprocessing method using correction fusion is more reliable than the preprocessing method using fusion alone.

[0073] In general, these preprocessing steps can more accurately match remote sensing images with AIS information, making subsequent analysis more reliable. This processing method not only reduces errors, but also solves the challenges caused by AIS data anomalies or inconsistent image and AIS sampling to a certain extent, providing a more robust basis for ship detection and classification.

[0074] 2. Small target ship detection model YOLOv8x-CA-CFAR based on constant false alarm rate secondary judgment.

[0075] In the task of detecting small-target ships in remote sensing images, there have been studies in the field of SAR images that have combined CFAR with deep learning to improve the accuracy and applicability of the detection algorithm. CFAR can detect small-sized targets based on the amplitude of pixels, which has its unique advantages.

[0076] It should be noted that optical remote sensing images and SAR (synthetic aperture radar) remote sensing images have many differences in imaging principles, data characteristics and weather dependence, so the CFAR method is not directly applicable to optical remote sensing images.

[0077] If the CFAR algorithm is to be used for optical remote sensing images, the difference in statistical characteristics between the two needs to be reduced.

[0078] In optical remote sensing images, we can first use deep learning algorithms to detect suspicious objects and slice them to reduce the difference between SAR images and optical remote sensing images, and then use CA-CFAR to distinguish the slices, so as to complement the limitations of deep learning and CFAR and improve the model performance. In this paper, YOLOv8x is used as the preliminary detection model, and the detection results are secondarily judged, and YOLOv8x-CA-CFAR is proposed.

[0079] The original YOLOv8x algorithm only retains detection frames with a confidence level of more than 0.25, which reduces false detections and may miss small ships with unclear features, affecting the recall rate and detection accuracy. To improve this problem, the YOLOv8x-CA-CFAR algorithm temporarily retains detection frames with a confidence level between 0.05 and 0.25, and uses the CA-CFAR algorithm to further analyze these suspicious areas. If CA-CFAR confirms the existence of a ship, the detection frame is retained; otherwise, it is discarded. This method improves the recognition of small ships and maintains high detection accuracy. The final result combines detection frames with high confidence and confirmed by CA-CFAR, achieving a more comprehensive and accurate detection output.

[0080] 3. The detected small target ship is matched with the AIS information.

[0081] Then the ship detection results and the pre-processed AIS information are input into the matching module for matching. However, before the formal matching, it is necessary to perform spatial screening based on the geographical scope of the image to exclude AIS data that exceeds the image coverage, thereby avoiding the introduction of redundant information and improving matching efficiency. The formula for spatial screening is as follows:

[0082]

[0083] Lon min Indicates the minimum longitude, Lon avg Indicates the fused longitude, Lonmax Indicates the maximum longitude, Lat min Indicates the minimum latitude, Lat avg Indicates the latitude after fusion, Lat max represents the maximum latitude, S1 represents the maximum latitude in a given time interval [T start ,T end ], and S2 represents all the entries remaining after spatial filtering.

[0084] The Hungarian algorithm is used in the matching process because it can find the global optimal solution in the bipartite graph matching problem, ensure the minimization of the total matching distance, and avoid the influence of early decisions on subsequent matching, thereby improving the matching accuracy and overall efficiency. If the greedy algorithm is used, it will lead to suboptimal matching, waste of resources, increase in error matching rate, and complex later adjustments. In addition, based on the difference between remote sensing images and AIS data, it cannot be expected that the fused AIS coordinates are exactly the same as the detected ship coordinates. Therefore, a reasonable matching distance needs to be set, and the recommended maximum tolerance distance is 2 kilometers. If the distance between the ship and the AIS information is within 2 kilometers, it is considered a successful match. Finally, the mathematical representation of the matching part is as follows:

[0085]

[0086] Where M represents a set of matching pairs that meet the conditions, which includes valid matching pairs of ship and AIS information, S = {s1, s2, ..., s n} represents a group of ships to be matched, A={a1,a2,...,a n} indicates the corresponding AIS information. The min and arrow at the top of the formula indicate that this is a minimization problem, the goal is to minimize the value of the objective function. Subject to is a constraint, indicating some conditions that must be met in the problem. means that each i must be assigned to only one j, This means that each j must also receive only one assignment from i. ij Indicates i with a j The distance between ij Indicates i Is it related to a j matching, τ represents the distance threshold.

[0087] 4. Discussion based on matching situation.

[0088] Analyze and judge based on the matching situation. At this time, it can be divided into two major categories. When the match is successful, that is, the normal situation, the matching result is output, and when the match fails, it is considered that there is an abnormal situation. At this time, it can be refined into two small categories, that is, there is no AIS information around the ship or there is no ship around the AIS information. When there is no AIS information around the ship, the ship is considered to be an abnormal ship, and it is sliced ​​according to the detection results, and then input into the classifier trained by AIS-SAR, corresponding to the model Multi-CDT in step 6, and finally the coordinates of the abnormal ship and its predicted category are output; when there is no ship around the AIS information, the AIS information is considered to be abnormal data and retained.

[0089] 5. Multi-CDT, a small target ship classification model based on unsupervised domain adaptation.

[0090] Existing research on unsupervised domain adaptation transfer learning mainly focuses on homogeneous or heterogeneous migration of a single source domain. Even when it involves multiple source domains, it is mostly limited to the migration of multi-source homogeneous data. However, practical applications often face the challenge of multi-source heterogeneous data, and there is a lack of research in this field. In existing research, although unsupervised domain adaptation of multi-source heterogeneous data is also mentioned, it refers to the difference in the feature dimensions of the source domain. Figure 3 In the figure, a visual representation of several situations mentioned in the present invention is given.

[0091] This paper migrates two heterogeneous sources, SAR images and AIS information, to optical remote sensing images, and proposes a Transformer-based multi-source heterogeneous unsupervised domain adaptation model Multi-CDT. The model architecture is as follows Figure 4 shown.

[0092] Specifically, considering that the imaging principles of different remote sensing images are different, which will lead to differences in pixel distribution, the source domain data used in the present invention is the FUSAR dataset. Each SAR remote sensing image has its corresponding AIS information, which contains the length and width information corresponding to the ship on each image. In existing studies, 16 features are designed based on the length and width information, such as geometric features such as circumference, area, and aspect ratio. Considering that these features are not easy to change with different image imaging principles, these AIS features are used in Multi-CDT to train a classifier based on AIS features. Then, the classifier based on AIS features is used as a teacher model. During the pre-training of ViT, distillation learning is used to guide and extract more discriminative and robust features. While improving the performance of the ViT model, it can also improve the performance of the model later migrated to the optical remote sensing image. In the pre-training stage, the loss function used can be divided into two parts. The first part is the classification loss, as shown in Formula 10, and the second part is the distillation loss, as shown in Formula 12.

[0093]

[0094] Where N is the number of samples in the batch, C is the number of categories, x is the log probability of the model output (the softmax output before logarithm), x i,j represents the model output of the jth class corresponding to the i-th sample, log represents the natural logarithm, e is the base of the natural logarithm, which is used to calculate the exponent, y i ' ,j represents the smoothed label, as shown in Equation 11.

[0095]

[0096] Where C is the number of categories, y is the real label of the one-hot encoding, the label smoothing parameter is ε, and y i ' ,j Represents the smoothed label.

[0097] By reducing the reliance on accurate labels and increasing the model's small probability predictions for all categories, it helps the model to be more robust in the face of uncertainty and improves the model's generalization ability.

[0098] Common multi-source information fusion often requires alignment at the feature level, but the data types, distributions, and feature expressions of different data sources may vary greatly, such as text and image data. This heterogeneity may lead to information loss or inaccurate alignment during the alignment process, especially in the absence of explicit associations between data modalities. The distillation loss shown in Equation 12 achieves knowledge transfer at the decision level.

[0099]

[0100] Among them, N is the number of samples (i.e. batch size), K is the number of categories, and Ais_score ik Indicates the probability of the i-th sample in the k-th category given by the AIS information classifier, Image_score ik It represents the probability score of the i-th sample in the k-th category given by the image classifier, and log represents the logarithmic operation.

[0101] Considering that the geometric features extracted from AIS information are structured data, XGBoost is used as a classifier model based on AIS features in Multi-CDT. Other classifier models can also be used here.

[0102] In general, Multi-CDT makes full use of the robust geometric features in AIS information and combines knowledge distillation technology to significantly improve the adaptability and generalization ability of the model in the unlabeled target domain. It provides new ideas for the research of adaptive transfer learning in multi-source heterogeneous domains and for solving practical problems such as classification of small target ships in optical remote sensing images, and demonstrates the great potential of multimodal data fusion in complex scenarios.

[0103] 6. The final result is retained.

[0104] Finally, the matched ship information and AIS information are retained, including the detected ship image and AIS information such as MMSI number, ship category, ship coordinates, ship size, etc. For unmatched ships, the ship image, longitude and latitude coordinates of the ship and the corresponding predicted category result information are retained.

[0105] The embodiments of the present specification also provide a device for identifying and abnormally evaluating small target ships based on multi-source information, which is used to execute the method for identifying and abnormally evaluating small target ships based on multi-source information.

[0106] The method provided in this manual is then evaluated and verified.

[0107] 1. Experimental environment and evaluation indicators

[0108] In the embodiment of the present invention, the training of the target detection and classification networks are both run on a server, whose CPU is Intel(R) Core(TM) i7-7800X CPU@3.50GHz 3.50GHz, GPU is NVIDIA TITAN RTX, and dedicated GPU memory is 24G.

[0109] The optical remote sensing image data of this example is based on a self-built small target ship detection dataset for high-resolution wide-area optical remote sensing images. Specifically, the dataset is constructed based on the remote sensing images of Hainan Satellite 1. Since the resolution of the original image is too large, each image is first split, and three image processing methods are used for data enhancement, and then labelme is used for annotation. Finally, there are 3829 images with a resolution of 1024×1024. The original image dataset is randomly divided into a training set (2679 images), a test set (345 images), and a validation set (805 images) according to a certain ratio through a Python script.

[0110] The radar image data used is based on the public dataset FUSAR, which contains 512×512 resolution ship images, divided into 15 major categories, with 5244 images. In order to construct the optical remote sensing image classification dataset (target domain data), the detection dataset of Hainan Satellite 1 was processed. The ship image was sliced ​​by annotating the box, and the sliced ​​rectangular image was filled with black pixels to make it a square, so as to avoid the distortion of the ship scale in the subsequent resize process, thereby maintaining the classification performance. Finally, 3814 small target ship classification images based on Hainan Satellite 1 were obtained as the unlabeled target domain dataset.

[0111] In the target detection task, the present invention uses P (Precision), R (Recall), and F1-score as comprehensive performance evaluation indicators. The present invention represents true positive, false positive, and false negative as TP, FP, and FN respectively. Then the formulas for Precision and Recall are as follows:

[0112]

[0113] At this time, the calculation formula of F1-score can be expressed as:

[0114]

[0115] The F1 score takes into account both precision and recall and is the harmonic mean of the two. Since precision and recall often constrain each other, the F1 score is used to balance the two in order to find the best model performance between the two. Its value is between 0 and 1. The higher the value, the better the model performance. When both precision and recall are high, the F1 score will also be correspondingly high.

[0116] In the target classification task, since it is difficult to obtain classification labels for high-resolution wide-area optical remote sensing images in engineering practice, the quantitative analysis of this invention mainly relies on source domain labels and target domain pseudo labels. This part of the experiment focuses on qualitative analysis. In the qualitative analysis experiment, Grad-CAM (Gradient-weighted Class Activation Mapping) and T-SNE (t-distributed stochastic neighbor embedding) techniques are used to generate images for analysis.

[0117] 2. Quantitative Analysis

[0118] In the small target ship detection task, in order to verify the effectiveness of the algorithm proposed in this invention, Faster R-CNN, SSD, YOLOv5n, YOLOv7, YOLOV7x, YOLOv7&QFL, YOLOv8n, YOLOv8x, YOLOv8x-LT and the algorithm YOLOv8x-CA-CFAR proposed in this invention are trained and tested on a self-built optical remote sensing image dataset. Among them, in order to better reflect the impact of adding CA-CFAR for secondary judgment, YOLOv8x-LT (simply reducing the threshold, conf=0.05) is further added to the experiment for comparative analysis. The experimental results of these models are shown in Table 1.

[0119] Table 1 Experimental results of different network methods on self-built optical remote sensing image dataset

[0120]

[0121] As can be seen from the table, the performance of different YOLO models varies, among which the YOLOv8 variant performs particularly well. YOLOv8n achieves a high F1-score (88.3%) with its lightweight design, which is suitable for real-time applications with limited resources; YOLOv8x performs well at the default confidence threshold (conf=0.25), with an F1-score of 94.6%, which is suitable for scenarios with high precision requirements; by lowering the confidence threshold to 0.05, YOLOv8x-LT significantly improves the recall rate and detects more targets, but the accuracy decreases slightly, indicating that the looser detection standard introduces more false alarms, and the F1-score is still 94.6%, which is also consistent with the theory; the YOLOv8x-CA-CFAR proposed in the present invention maintains a high recall rate and significantly reduces the false alarm rate under the same low threshold through the CA-CFAR method, achieving the highest F1-score (95.8%).

[0122] In the small target ship classification task, in order to verify the effectiveness of the algorithm proposed in the present invention, DAN, JAN, DANN, AFN, MCD, BSP, CDT and the algorithm Multi-CDT proposed in the present invention use FUSAR as the source domain and the cropped self-built optical remote sensing image dataset as the target domain for training and testing.

[0123] In CDTrans, unsupervised domain adaptation learning is divided into two steps: first, pre-training is performed on the source domain using ViT, and then transfer learning is performed using the trained ViT model. In quantitative experiments, CDTrans and Multi-CDT are trained for 50 epochs on the source domain and target domain respectively, and other models are trained for 100 epochs to ensure fair comparison.

[0124] Table 2 Performance comparison of each model on classification tasks

[0125]

[0126] The experimental results are shown in Table 2. Multi-CDT shows excellent classification ability in both the source domain and the target domain, with an accuracy of 99.82%, which is significantly better than other transfer learning methods. This shows that Multi-CDT extracts more discriminative and robust features through multi-source information fusion, which is suitable for complex and changeable actual scenarios.

[0127] 3. Qualitative analysis

[0128] In order to verify the improvement effect of the present invention on the small target ship detection task, Figure 5 The detection results of YOLOv8x, YOLOv8x-LT and the model YOLOv8x-CA-CFAR proposed in this invention in different scenarios are displayed and compared, where the results marked with red circles are missed detection results, and the results marked with yellow circles are false detection results.

[0129] Depend on Figure 5 From the comparison, we can see that the original YOLOv8x model has some missed detections for some small-sized ships. In engineering applications, more attention is paid to the recall rate of the model. The most direct way is to lower the confidence threshold to improve the recall rate, that is, the YOLOv8x-LT model. However, it can be found from the detection results that this leads to a large number of false detections. The YOLOv8x-CA-CFAR proposed in the present invention detects small-sized ships compared to YOLOv8x, reduces the number of missed detections, and reduces the number of false detections compared to YOLOv8x-LT. At the expense of a smaller accuracy rate, the recall rate is greatly improved, and the overall performance of the model is improved.

[0130] In order to verify the improvement effect of the present invention on the small target ship classification task, Figure 6 , Figure 7 The above gives the training trend charts of Multi-CDT and CDTrans in the source domain pre-training task and the target domain unsupervised domain adaptation training task respectively.

[0131] like Figure 6As shown in the figure, the Multi-CDT model showed higher accuracy and faster improvement on the test set, indicating that it better captured data correlation and characteristics through multi-source information fusion, thereby improving performance. In contrast, the accuracy of the CDTrans model improved more slowly and ultimately performed worse than the Multi-CDT model. At the end of pre-training, Multi-CDT was 1.2% more accurate than CDTrans.

[0132] Figure 7 The performance comparison of Multi-CDT and CDTrans in unsupervised transfer learning tasks is demonstrated. The Multi-CDT model showed high accuracy at the beginning of the experiment and remained above 98% throughout the training process, demonstrating stable and consistent high performance. In contrast, the accuracy of the CDTrans model was low at first, but gradually improved during training and stabilized after about the 20th epoch. The accuracy of Multi-CDT was close to 100% with small fluctuations, while the CDTrans model had a maximum accuracy of 99.5%, but a starting accuracy of less than 90%. Overall, Multi-CDT demonstrated better initial performance and stability, and the multi-source information fusion strategy enabled it to more effectively utilize source domain information to improve target domain performance.

[0133] Next, we will use the models trained in the second step of domain adaptation, CDTrans and Multi-CDT, to generate thermal maps for source domain images (SAR remote sensing images) and target domain images (optical remote sensing images) for qualitative analysis.

[0134] according to Figure 8 and Fig. 9 From the attention map shown, it can be seen that the proposed Multi-CDT method can capture important areas more accurately than CDTrans, pay less attention to the background, and have more hot spots on the target object.

[0135] Next, we use CDTrans and Multi-CDT to extract features from the target domain image (optical remote sensing image) using the models trained in the second step of domain adaptation, and generate T-SNE graphs for qualitative analysis. The obtained T-SNE graphs are shown in the figure below. Fig.10 shown.

[0136] In classification tasks in practical applications, it is important that the model can accurately distinguish the diverse categories of the source domain, rather than simply simplifying it into a few large categories. Considering that there are 15 categories in total, including a class6 representing "others", two T-SNE graphs are used to further analyze the category distribution in detail:

[0137] Fig.10(a) shows three main categories (circle, square, and prism), among which the circle category is densely clustered in the lower half of the figure, suggesting that the model may classify multiple categories as "others". The square and prism categories show good clustering, but are not enough to reflect the category diversity in the source domain.

[0138] Fig.10 (b) shows five main categories (circle, square, prism, triangle, and star), providing a more detailed category division, showing that the model has better recognition ability for different categories and can better reflect the category diversity of the source domain.

[0139] 4. Practical demonstration of the invented device

[0140] In actual combat, the detected ships are marked with indexes, and a circle with a radius of 2 kilometers is drawn around them to indicate the visible matching range. The matched AIS information is represented by a green dot on the image, and the unmatched ones are displayed with a red dot. If there is no AIS information in a certain area, the system will warn "There is no AIS information in this area" and display the location of the abnormal ship and the prediction results of the classifier, such as Fig.11 shown.

[0141] On the contrary, if AIS information is available, the system will try to match it and display the result on the image and in the detection result overview, such as Fig.12 shown.

[0142] The above tests show that the multi-source information fusion anomaly detection framework proposed in the present invention has good robustness and can obtain the required detection results under various practical input conditions.

[0143] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present invention is limited to these examples; under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying small target ships and abnormality analysis based on multi-source information, characterized in that: The following steps are involved: S1, collect optical remote sensing images and ship automatic identification system data, and perform preprocessing; S2. Input the optical remote sensing image into the constructed small target ship detection model YOLOv8x-CA-CFAR based on constant false alarm rate secondary judgment to obtain the small target ship detection result. The small target ship detection model YOLOv8x-CA-CFAR first uses a deep learning algorithm to detect suspicious objects for slicing, and then uses CA-CFAR to distinguish the slices to obtain the small target ship detection result; S3, matching the small target ship detection result with the pre-processed ship automatic identification system data, and if the match is successful, outputting the matching result; S4. When the matching fails, it is determined that there is an abnormal situation, and the abnormal situation is judged. When it is determined that there is no ship around the ship automatic identification system data, the ship automatic identification system data is considered to be abnormal data and is retained; S5. If it is determined that there is no automatic ship identification system data around the ship, the corresponding optical remote sensing image is sliced ​​according to the detection result, and input into the trained unsupervised domain adaptive classification model named Multi-CDT to obtain the predicted category of the abnormal ship; S6. The ship information and the ship automatic identification system data in the matching results are retained. For the unmatched ships, the abnormal data obtained in S4 and the coordinates and predicted categories of the abnormal ships obtained in S2 and S5 are retained.

2. The method for identifying small target ships and judging abnormalities based on multi-source information according to claim 1 is characterized in that: The pre-processing comprises: The optical remote sensing image is processed into a 640×640 8-bit RGB image by color depth conversion and image cropping; Through time screening, using the updatetime field in the ship automatic identification system data, ensure that the selected ship automatic identification system data is consistent with the shooting time range of the optical remote sensing image; Use the offset information in the ship automatic identification system data to perform coordinate correction; The fused coordinates are obtained by averaging multiple ship automatic identification system position points after time screening and coordinate correction.

3. The method for identifying small target ships and judging abnormalities based on multi-source information according to claim 1 is characterized in that: The small target ship detection model YOLOv8x-CA-CFAR temporarily retains the detection frame with a confidence level between 0.05 and 0.25, and uses the CA-CFAR algorithm to further analyze the suspicious area. If CA-CFAR confirms the existence of a ship, the detection frame is retained, otherwise, it is discarded.

4. The method for identifying small target ships and judging abnormalities based on multi-source information according to claim 1 is characterized in that: Before performing the matching in step S3, the method further includes: The pre-processed Automatic Identification System data are spatially screened to exclude the Automatic Identification System data that exceeds the image coverage.

5. The method for identifying small target ships and judging abnormalities based on multi-source information according to claim 1 is characterized in that: The matching process in step S3 adopts the Hungarian algorithm, and the matching formula is: Where M represents a set of matching pairs that meet the conditions, which includes valid matching pairs of ship and AIS information, S = {s1, s2, ..., s n } represents a group of ships to be matched, A={a1,a2,...,a n } indicates the corresponding AIS information. The min and arrow at the top of the formula indicate that this is a minimization problem, the goal is to minimize the value of the objective function. Subject to is a constraint, indicating some conditions that must be met in the problem. means that each i must be assigned to only one j, means that each j must also receive only one assignment from i, d ij Indicates i with a j The distance between ij Indicates i Is it related to a j matching, τ represents the distance threshold.

6. The method for identifying small target ships and judging abnormalities based on multi-source information according to claim 1 is characterized in that: The construction process of the unsupervised domain adaptation classification model named Multi-CDT includes: Using the FUSAR dataset, we extract the AIS data features in the dataset, and use these AIS data features in Multi-CDT to train another classifier based on the AIS data features. A classifier based on the data features of the automatic ship identification system is used as the teacher model, and distillation learning is used to guide the pre-training of the student model ViT, a classification model based on remote sensing images. The loss functions used in pre-training include classification loss and distillation loss.

7. The method for identifying small target ships and judging abnormalities based on multi-source information according to claim 6 is characterized in that: The calculation formula of the classification loss is: Where N is the number of samples in the batch, C is the number of categories, x is the log probability of the model output, and x i,j represents the model output of the jth class corresponding to the i-th sample, log represents the natural logarithm, e is the base of the natural logarithm, which is used to calculate the exponent, y i ' ,j represents the smoothed label; Where C is the number of categories, y is the real label of the one-hot encoding, the label smoothing parameter is ε, and y i ' ,j Represents the smoothed label.

8. The method for identifying small target ships and judging abnormalities based on multi-source information according to claim 6 is characterized in that: The calculation formula of the distillation loss is: Among them, N is the number of samples (i.e. batch size), K is the number of categories, and Ais_score ik Indicates the probability of the i-th sample in the k-th category given by the AIS information classifier, Image_score ik It represents the probability score of the i-th sample in the k-th category given by the image classifier, and log represents the logarithmic operation.

9. The method for identifying small target ships and judging abnormalities based on multi-source information according to claim 6 is characterized in that: XGBoost is used as a classifier model based on the data features of the automatic ship identification system.

10. A multi-source information small target ship recognition and abnormal analysis device, characterized in that: Used to execute the small target ship identification and abnormality analysis method of multi-source information as described in any one of claims 1-9.

Images