A method for detecting small targets at sea based on ViT
By using ViT and Swin-Transformer networks in maritime target detection, combined with PRD strategy and Dynamic head, the shortcomings of traditional ship perception systems in small target detection are solved, efficient small target detection effect is achieved, and detection accuracy and generalization are improved. Metamorphosis.
Patent Information
- Application Number
- CN202410520403.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-28
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-04-28
AI Technical Summary
Traditional ship perception systems perform poorly when detecting small or partially immersed targets, and there are problems with space-time alignment and error calibration during data fusion, which affects the operational efficiency and navigation safety of the ship.
Using Vision Transformer (ViT)-based small-object detection method, the Swin-Transformer backbone network and the sample input weight dynamic update strategy (PRD) for predicting position information and score information is introduced, and the subtle features and global in the image are extracted through the sliding window mechanism. Features and feature fusion and target classification through Dynamic head and DINO head networks.
The detection accuracy and generalization of the model for small targets is improved, the effect of small target detection is improved, and small target detection with high accuracy and high comprehensiveness in complex marine environments is achieved.
Smart Images

Figure CN118505964B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent perception of maritime traffic, and in particular to a method for detecting small maritime targets based on ViT. Background Art
[0002] Traditional ship perception systems such as radar and sonar can monitor the marine environment in real time, but they perform poorly in detecting small or partially submerged targets. In addition, the above systems need to process data from multiple sensors, which not only increases the computational complexity, but also brings about the problems of spatiotemporal alignment and error calibration generated during the data fusion process. These limitations affect the operational efficiency and navigation safety of ships.
[0003] In terms of small target detection, the challenges faced by existing methods include small target size, little feature information, and susceptibility to occlusion and environmental factors. When dealing with small targets, traditional anchor-frame-based target detection methods have difficulty accurately classifying and locating small targets due to the low matching degree between the anchor frame and the target size and shape and the low target attention.
[0004] This paper proposes a method for detecting small targets at sea based on ViT. This method introduces a ViT-based Swin-Transformer backbone network and efficiently extracts subtle features and global features in the image through a sliding window mechanism. Furthermore, in order to solve the problems of low matching between the anchor frame and the target size and shape and low target attention, a sample input weight dynamic update strategy (PRD) based on predicted position information and score information is proposed to improve the model's detection effect on small targets. Summary of the invention
[0005] The purpose of the present invention is to provide a method for detecting small targets at sea based on ViT to solve the problems raised in the above background technology.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] A method for visual detection of small targets at sea based on ViT, comprising the following steps:
[0008] Step 1, select a marine target data set, preprocess the marine target data set, and divide it into a training set, a test set, and a validation set;
[0009] Step 2: Build a small target detection model at sea;
[0010] Step 3, training, testing, and verifying the marine small target detection model through the training set, test set, and validation set to obtain the optimal marine small target detection model;
[0011] Step 4: The image of the small target at sea to be detected is passed into the optimized small target detection model at sea for small target detection.
[0012] Furthermore, the pre-processing in step 1 includes:
[0013] S1, performs various data enhancements on the marine target dataset to expand the dataset samples;
[0014] S2, using the Copy-paste method, increases the number of small target samples, thereby enhancing the weight of small targets.
[0015] Furthermore, the S2 specifically includes:
[0016] S201, extraction: selecting images containing different marine targets (such as ships, buoys, etc.) from the expanded data set, and these images have different environmental conditions (such as lighting, weather, water surface texture, etc.);
[0017] S202, small target capture: performing target detection and segmentation on the image selected in S201, separating the small target object from its original background, and obtaining a pure small target object image;
[0018] S203, background image selection: selecting some background images containing only water surface background from other data sources, and these background images also have different environmental conditions to increase data diversity;
[0019] S204, pasting small target objects: randomly pasting the small target object images extracted in S202 onto the background image selected in S203 to generate a new composite image.
[0020] Furthermore, the S204 also includes: during the pasting process, performing some random transformations on the small target object image, such as scaling, rotating, flipping, etc., to simulate different viewing angles and postures.
[0021] Furthermore, the marine small target detection model includes Swin-Transformer, Dynamic head, PRD strategy, and DINO head network;
[0022] Swin-Transformer is the backbone network of the marine small target detection model, which is mainly responsible for extracting image features. It processes local features in the image by using a windowed self-attention mechanism, and gradually expands the receptive field through a hierarchical structure to capture a wider range of contextual information. This structure makes the network more flexible and efficient in processing images of different sizes;
[0023] The Dynamic head neck network is used to establish connections between feature maps extracted from the backbone network and perform feature fusion. It dynamically adjusts the structure of its modules to adapt to different target sizes and shapes, optimizes the feature transfer path, and thus improves the model's detection accuracy for various targets.
[0024] Through the PRD strategy, the predicted position information and score information are introduced to dynamically update the sample input weight. Through more reasonable sample weight distribution, the weight of small target samples is enhanced, thereby achieving a more detailed small target detection effect;
[0025] DINO head, as the head network, is responsible for the classification of targets and the precise positioning of bounding boxes. It receives the feature maps output by the neck network Dynamic head and uses a series of optimized network layers to determine the category of each candidate region and accurately calculate the position of its bounding box.
[0026] Furthermore, the processing process of the Dynamic head is divided into three attention calculation processes, namely scale attention, spatial attention and channel attention, and the calculation process is:
[0027] W(F)=π C (π S (π L (F)·F)·F)·F
[0028] Among them, W represents the attention calculation function, F represents the input three-dimensional feature tensor, and π L , π S , π C They refer to the level, width-height product, and number of channels of the feature map respectively. The scale attention, spatial attention, and channel attention modules perform attention operations on their specific dimensions respectively.
[0029] Furthermore, the process of the PRD strategy includes:
[0030] S301, the feature map is roughly detected to generate several pre-check boxes B as the basis for subsequent fine detection. In order to ensure the accuracy of the predicted box, a smooth L1 loss function is used for supervised learning. The loss function calculates the difference between the predicted box B and the real labeled box. This process is expressed as:
[0031]
[0032] Among them, B represents the coordinates of the predicted box, G represents the coordinates of the real annotation box, i represents the dimensions of the coordinates, that is, the (x, y) coordinates of the center point and the width w and height h, B i With G i Representing the specific parameters of the bounding box, the specific form of the smooth L1 loss function can be expressed as:
[0033]
[0034] Among them, if|x|<1 means that the absolute value of x is less than 1.
[0035] S302, select the top K pre-inspection boxes according to the confidence level, and generate a feature weight enhancement mask M according to their positions. This process can be:
[0036]
[0037]
[0038] Among them, i and j represent the position of the current pixel, M i,j represents the weight of the corresponding pixel position in the image, B k Indicates the pre-check box and indicates the function Indicates whether the current position is in the pre-check box B K If it is, it is set to 1, otherwise it is set to 0;
[0039] S303, normalize the feature weight enhancement mask M and use it as a weight adjustment feature to improve the response of the feature to the pre-check box area. The process can be expressed as:
[0040]
[0041] Among them, Fm represents the original feature map, Fm' represents the enhanced feature map, α is a hyperparameter used to control the feature response amplification factor, and max(M) is the maximum value in the feature weight enhancement mask M.
[0042] Through this operation, the feature response of the potential target area is amplified by α times, thereby increasing the model's attention to the potential target area, which is beneficial for the model to capture its features and location information.
[0043] Compared with the prior art, the present invention has the following beneficial effects:
[0044] 1) The present invention adopts ViT (Vision Transformer) to extract image features. Its hierarchical Transformer structure can effectively capture features of different scales, which enables it to perform well when processing small targets at a long distance or large targets at a close distance. It adopts a dynamic attention mechanism, which allows the model to dynamically adjust its processing method according to the input data, thereby improving the generalization of the model.
[0045] 2) The present invention uses Dynamic head for efficient feature fusion and effectively improves the model's ability to identify features of different scales through the self-attention mechanism;
[0046] 3) The present invention proposes a PRD strategy, which effectively improves the model's attention to small target areas through progressive detection, thereby improving the detection effect of small targets. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a flow chart of the present invention;
[0048] Figure 2 It is a structural diagram of a small target detection model at sea according to the present invention;
[0049] Figure 3 This is a schematic diagram of the Copy-paste method;
[0050] Figure 4 It is the structural diagram of the Swin-transformer backbone network in the present invention;
[0051] Figure 5 It is a diagram of the Dynamic head network structure in the present invention;
[0052] Figure 6 It is a schematic diagram of the PRD strategy flow in the present invention;
[0053] Figure 7 This is a comparison chart of the detection effects of the small target detection model at sea proposed in the present invention and the baseline model. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] See also Figure 1-Figure 7 ,A method for visual detection of small targets at sea based on ViT (Vision Transformer) includes the following steps:
[0056] Step 1: Select a public maritime target dataset, preprocess the maritime target dataset, and divide it into a training set, a test set, and a validation set.
[0057] Preprocessing operations include:
[0058] S1, performs various data enhancements on the marine target dataset to expand the dataset samples;
[0059] S2, using the Copy-paste method, increases the number of small target samples, thereby enhancing the weight of small targets, including:
[0060] S201, extraction: selecting images containing different marine targets (such as ships, buoys, etc.) from the expanded data set, and these images have different environmental conditions (such as lighting, weather, water surface texture, etc.);
[0061] S202, small target capture: performing target detection and segmentation on the image selected in S201, separating the small target object from its original background, and obtaining a pure small target object image;
[0062] S203, background image selection: selecting some background images containing only water surface background from other data sources, and these background images also have different environmental conditions to increase data diversity;
[0063] S204, pasting small target objects: randomly pasting the small target object images extracted in S202 onto the background image selected in S203 to generate a new composite image.
[0064] Step 2: Build a small target detection model at sea. The small target detection model at sea includes Swin-Transformer, Dynamic head, PRD strategy, and DINO head network.
[0065] Step 3, training, testing, and verifying the marine small target detection model through the training set, test set, and validation set to obtain the optimal marine small target detection model;
[0066] Step 4: The image of the small target at sea to be detected is passed into the optimized small target detection model at sea for small target detection.
[0067] The workflow of the above-mentioned small target detection model at sea is:
[0068] a. First, use the Swin-Transformer backbone network to extract image feature maps and process image data. The image is divided through the Patch Partition layer. Each 4×4 pixel block is divided into a separate patch and expanded on the RGB channels to form a 4×4×3 channel number. Then, these channels are converted into C channels through the Linear Embedding module, and then feature extraction is performed through two Swin-Transformer Block layers to generate a preliminary feature map. After that, the model is further processed through three stages with the same structure. Each stage starts with a Patch Partition layer, downsamples the feature map and adjusts the number of channels, and then processes through several Swin-TransformerBlock layers to output feature maps of different scales.
[0069] b. Feature fusion is performed through the Dynamic head module. The processing process of Dynamic head is divided into three attention calculation processes, namely scale attention, spatial attention and channel attention. The calculation process can be expressed by the following formula:
[0070] W(F)=π C (π S (π L (F)·F)·F)·F
[0071] Among them, W represents the attention calculation function, F represents the input three-dimensional feature tensor, and π L , π S , π C They refer to the level, width-height product, and number of channels of the feature map, respectively. The scale attention, spatial attention, and channel attention modules perform attention operations on their specific dimensions, respectively.
[0072] c. Through the PRD strategy, the weight of small target samples is enhanced. The specific process includes:
[0073] S301, the feature map processed by the backbone network and the neck network is roughly detected to generate a number of pre-check boxes B as the basis for subsequent fine detection. In order to ensure the accuracy of the predicted box, a smooth L1 loss function is used for supervised learning. The loss function calculates the difference between the predicted box B and the ground-truth box. This process can be expressed as follows:
[0074]
[0075] Among them, B represents the coordinates of the predicted box, G represents the coordinates of the real annotation box, i represents the dimensions of the coordinates, that is, the (x, y) coordinates of the center point and the width w and height h, Bi and Gi represent the specific parameters of the bounding box, and the specific form of the smooth L1 loss function is shown in the following formula:
[0076]
[0077] Among them, if|x|<1 means that the absolute value of x is less than 1.
[0078] S302, for potential pre-check boxes B, select the top K of them according to their confidence, and generate feature weight enhancement masks M according to their positions. This process can be expressed as follows:
[0079]
[0080]
[0081] Among them, i and j represent the position of the current pixel, M i,jrepresents the weight of the corresponding pixel position in the image, Bk represents the pre-check box, and the indicator function Indicates whether the current position is within the pre-check box BK. If so, it is set to 1, otherwise it is set to 0.
[0082] S303, normalizing the feature weight enhancement mask M and using it as a weight adjustment feature to improve the response of the feature to the pre-check box area. The process can be expressed as follows:
[0083]
[0084] Among them, Fm represents the original feature map, Fm' represents the enhanced feature map, α is a hyperparameter set by the detection result and is used to control the feature response amplification factor, and max(M) is the maximum value in the feature enhancement mask.
[0085] Through this operation, the feature response of the potential target area is amplified by α times, thereby increasing the model's attention to the potential area, which is beneficial for the model to capture the features and location information of the area.
[0086] d, Classification and regression using the DINO head network.
[0087] The small target detection model at sea proposed in the present invention has an average detection accuracy of 6.9% higher than that of the baseline model DINO on a water surface object detection dataset with a large number of small targets on a server equipped with an Nvidia A40 graphics card, and the small target detection accuracy is improved by 5.9%. In addition, high precision and high recall small target detection results can be achieved in the complex marine environment actually collected, thereby improving the generalization and robustness of the model.
[0088] In a specific implementation case, in order to verify the effectiveness of the model proposed in the present invention, the model of the present invention is compared with the existing advanced model under the same experimental conditions, and the results of the comparison of various performance indicators are shown in the following table.
[0089]
[0090]
[0091] As can be seen from the above table, the model proposed in the present invention has significant advantages over several other advanced target detection models in the marine target detection task. Compared with the mainstream target detection model, the mAP of the model proposed in the present invention reaches 89.5%, indicating that the model proposed in the present invention has the highest detection accuracy in the marine detection task compared with the existing model. At the same time, the model proposed in the present invention also achieved the highest detection score in the small target detection task. The AP_s of the model proposed in the present invention is 36.3%, which is ahead of other mainstream models, indicating that the present invention is effective.
[0092] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art. Although the embodiments of the present invention have been shown and described, it is understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A ViT-based visual detection method for small targets at sea. It is characterized in that The following steps are involved: Step 1, select a marine target data set, preprocess the marine target data set, and divide it into a training set, a test set, and a validation set; Preprocessing includes: S1, performs various data enhancements on the marine target dataset to expand the dataset samples; S2, using the Copy-paste method, increases the number of small target samples, including: S201, extraction: selecting images containing different marine targets from the expanded data set, and the images have different environmental conditions; S202, small target capture: performing target detection and segmentation on the image selected in S201, separating the small target object from its original background, and obtaining a pure small target object image; S203, background image selection: selecting some background images containing only water surface background from other data sources, and these background images also have different environmental conditions to increase data diversity; S204, pasting small target objects: randomly pasting the small target object images extracted in S202 onto the background image selected in S203 to generate a new composite image. During the pasting process, the small target object images are randomly transformed to simulate different viewing angles and postures. Step 2, constructing a small target detection model at sea, which includes Swin-Transformer, Dynamic head, PRD strategy, and DINO head network; using Swin-Transformer as the backbone network of the small target detection model at sea to extract feature maps; using Dynamic head to fuse image features; using PRD strategy to enhance the weight of small target samples; and using DINO head network for classification and regression; The processing process of the Dynamic head is divided into three attention calculation processes, namely scale attention, spatial attention and channel attention, and the calculation process is as follows: W(F)=π C (p S (p L (F)·F)·F)·F Among them, W represents the attention calculation function, F represents the input three-dimensional feature tensor, and π L , π S , π C They refer to the level, width-height product, and number of channels of the feature map respectively. The scale attention, spatial attention, and channel attention modules perform attention operations on their specific dimensions respectively; The process of the PRD strategy includes: S301, perform a rough detection on the feature map to generate several pre-check boxes B as the basis for subsequent fine detection. In order to ensure the accuracy of the predicted box, a smooth L1 loss function L is used. smoothL1 For supervised learning, the loss function calculates the difference between the predicted box B and the true labeled box. This process is expressed as: L smoothL1 (B,G)=∑ i∈{x,y,w,h} smooth L1 (B i -G i ) Among them, B represents the coordinates of the predicted box, G represents the coordinates of the real annotation box, i represents the dimensions of the coordinates, that is, the (x, y) coordinates of the center point and the width w and height h, B i With G i Representing the specific parameters of the bounding box, the specific form of the smooth L1 loss function can be expressed as: S302, select the top K pre-inspection boxes according to the confidence level, and generate a feature weight enhancement mask M according to their positions. This process can be: Among them, i and j represent the position of the current pixel, M i,j represents the weight of the corresponding pixel position in the image, B k Represents the pre-check box, indicating function I BK (x) indicates whether the current position is in the pre-check box B K If it is, it is set to 1, otherwise it is set to 0; S303, normalize the feature weight enhancement mask M and use it as a weight adjustment feature to improve the response of the feature to the pre-check box area. The process can be expressed as: Among them, Fm represents the original feature map, Fm' represents the enhanced feature map, α is a hyperparameter set by the detection result, which is used to control the feature response amplification factor, and max(M) is the maximum value in the feature weight enhancement mask M; Step 3, training, testing, and verifying the marine small target detection model through the training set, test set, and validation set to obtain the optimal marine small target detection model; Step 4: The image of the small target at sea to be detected is passed into the optimized small target detection model at sea for small target detection.
