Real-time marine ship target detection method based on improved DETR

Abstract

The invention belongs to the field of ship detection, and provides a real-time marine ship target detection method based on an improved DETR. By making a sea-sky-line small target detection data set, the problem of low precision of small target marine ship target detection in marine ship target detection is solved. And secondly, by using a coding layer and a non-cascade decoding layer based on feature pyramid decoupling, computing resources are reduced, and the marine ship target detection speed is increased. And finally, a fusion compensation module is introduced, and a plurality of auxiliary detection heads are introduced into the fusion compensation module, so that richer supervision information is provided, the expression ability of the feature map is improved, and the identification precision of the model on the marine ship is improved. Meanwhile, the fusion compensation module is only used in the training stage, so that the precision of marine ship target detection is improved, and the detection speed of the model in the reasoning stage can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of ship detection, and in particular to a real-time marine ship target detection method based on improved DETR. Background Art

[0002] In the complex and ever-changing marine environment, ship target detection is a vital task. It is not only related to maritime traffic safety and marine resource management, but also directly affects many fields such as national border security. By monitoring the position of ships at sea in real time, it helps to optimize waterway management, avoid ship collisions, and ensure navigation safety; at the same time, it is of great significance to maintain marine order, ensure shipping safety, and promote the development of the marine economy.

[0003] At present, the target detection technology used in marine ships is mainly YoloV5, but its target detection accuracy and detection speed depend on the design of the anchor frame. When designing and training for the target detection task of marine ships, the deviation of the anchor frame design will lead to extremely poor target detection results, which is not conducive to the compatibility deployment of mainstream development boards on the market. At the same time, YoloV5 relies on manually set NMS and other post-processing operations, and the post-processing operation time increases with the increase in the number of target detections, and the average time required for post-processing operations is twice the model inference time, which seriously affects the target detection speed of marine ships. The DETR model based on the Transform architecture can well solve the problems of the above-mentioned traditional YoloV5 model, but the traditional DETR has a large number of parameters and has certain requirements for computing resources, which is not conducive to the deployment of inference on the development board. The Transform architecture in the DETR model is to associate the global features before and after, resulting in poor target detection accuracy for small target ships with sparse features. Summary of the invention

[0004] Aiming at the problem that the DETR model in the existing technology in the field of ship detection has a large number of parameters and high computing requirements, the reasoning speed of the DETR model is slow after deployment on the development board, which cannot meet the needs of real-time marine ship target detection; at the same time, the DETR model has poor detection accuracy for small marine ship targets with sparse features. A real-time marine ship target detection method based on improved DETR is proposed, which reduces the number of parameters and computing requirements of the model, realizes real-time marine ship target detection, and improves the detection accuracy of small marine ship targets.

[0005] The specific plan is as follows:

[0006] A real-time marine ship target detection method based on improved DETR, comprising:

[0007] S1: Collect the video stream data of ships sailing at sea, and capture pictures of the video stream data of ships sailing at sea at a set frequency; integrate the captured pictures into a marine ship image dataset;

[0008] S2: Use the Hough transform model to detect the coordinates (x1, y1, x2, y2) of the upper left corner and lower right corner of the sea-sky line of all images in the S1 marine ship image dataset, and cut the image according to the coordinates (x1, y1, x2, y2), and save the image within the coordinates; summarize the cut images to obtain the sea-sky line small target dataset;

[0009] S3: Manually annotate the S1 marine ship image dataset and the S2 sea-sky line small target dataset, and mix the two manually annotated datasets into the final marine ship image dataset; the manual annotation is to annotate the (x, y) coordinate information of the upper left corner and the lower right corner of the ship in the image;

[0010] S4: Perform data augmentation operations on the images in the final marine ship image dataset in S3;

[0011] S5: Build a real-time DETR training model with a fusion compensation module, including: a backbone network based on ResNet50, an encoding layer based on pyramid decoupling, a non-cascaded decoding layer, and a fusion compensation module;

[0012] S6: Using the data set after data enhancement in S4, train the real-time DETR training model with fusion compensation module in S5 and save the model weights;

[0013] S7: Build a real-time DETR model without fusion compensation module, including: backbone network based on ResNet50, encoding layer based on pyramid decoupling, and non-cascaded decoding layer;

[0014] S8: Import the model weights in S6 into the real-time DETR model constructed in S7 to perform real-time marine ship target detection on the marine ship video stream.

[0015] Preferably, the data enhancement operation described in S4 includes: contrast enhancement, image rotation, image flipping, affine transformation image expansion, and HSV data enhancement.

[0016] Preferably, the real-time DETR training model method with a fusion compensation module described in S5 is:

[0017] S51: converting the input image into an RGB three-dimensional feature matrix; adjusting the RGB three-dimensional feature matrix into a three-dimensional feature matrix of a fixed size;

[0018] S52: Input the fixed-size three-dimensional feature matrix into the backbone network based on ResNet50 for feature extraction to obtain the features of each stage in the backbone network;

[0019] S53: Input the features {S3, S4, S5} of the last three stages in S52 into the encoding layer based on pyramid decoupling to obtain the encoded features;

[0020] S54: Input the encoded features of S53 into the non-cascade decoding layer to obtain the category and bounding box of the marine ship;

[0021] S55: Input the potential features of S54 into a fusion compensation module to obtain a marine ship target detection result, including: a bounding box, a confidence level, and a category; the fusion compensation module comprises a multi-scale adapter and K auxiliary branches; the auxiliary branch comprises an auxiliary detection head and a non-cascaded decoding layer.

[0022] Preferably, the specific method of encoding the layer based on pyramid decoupling described in S53 is:

[0023] S531: Flatten the feature S5 of the last stage of S52 to obtain Q, V, K;

[0024] S532: Perform data shaping operation on Q, V, K to obtain a new feature F5 with the same shape as the feature S5;

[0025] S533: Input the low-level features S3, S4 and the new feature F5 into the cross-scale feature fusion module to obtain fused features, and use the fused features as encoded features;

[0026] The specific process can be summarized into the following calculation formula:

[0027] Q=K=V=Flatten(S5) (1)

[0028] F5=Reshape(Attn(Q,K,V)) (2)

[0029] Output =CFM({S3, S4, F5}) (3)

[0030] Among them, Flatten represents the flattening operation; Q is the Query vector as part of the request information, which is used to find relevant information in the input sequence; K is the Key vector that provides information about the input sequence and is used to match the Query vector; V is the Value vector containing the actual input data information. When the match with the Query vector is successful, the information will be extracted from V; Att represents the attention mechanism; CFM represents the cross-scale feature fusion module; Reshape represents the data reshaping operation.

[0031] Preferably, the execution steps in the non-cascaded decoding layer in S54 are:

[0032] S541: Input the features encoded in S53 into the Region Proposal Network (RPN) to obtain two-dimensional reference points, including classification scores and bounding boxes;

[0033] S542: Sort the reference points according to the classification scores in S541, set and select the first n reference points as the output features corresponding to the features after encoding in S53, and use them as the initial object query of the non-cascaded decoding layer, i.e., objectqueries;

[0034] S543: Input the encoded features of S53 and the initial object queries of S542 into the non-cascaded decoding layer for fusion to obtain the fused result;

[0035] S544: Input the fused result of S543 into the FFN layer to obtain the category and bounding box of the detected marine vessel.

[0036] Preferably, the specific method of the fusion compensation module described in S55 includes:

[0037] S551: Use the multi-scale adapter to transform the encoded features of S53 into a feature pyramid {F1, ···, F J};

[0038] S552: Setting the auxiliary detection heads of K auxiliary branches and the corresponding label allocation method A K ; {F1,···,F J}Sent to the auxiliary detection head of the i-th auxiliary branch to obtain the target detection result of the marine ship Includes: bounding box coordinates, classification and confidence; using the label assignment method A in the i-th auxiliary branch i calculate Supervision objectives for positive and negative samples;

[0039] The process can be described as:

[0040]

[0041] where {pos} and {neg} represent the i Determine the positive samples and negative samples; positive samples refer to the areas containing the target object; negative samples refer to the areas that do not contain the target object; G represents the true set; j represents {F1,····,F J} in feature index; is the set of positive coordinates in space; P i{pos} and P i {neg} is the supervision target in the corresponding coordinates, including the category and regression offset;

[0042] S553: ​​The spatial positive coordinate set of the auxiliary detection head of the i-th auxiliary branch obtained in S552 Generate additional custom positive queries All queries in the auxiliary branch are considered positive queries; they can be generated as follows:

[0043]

[0044] Where Linear represents a linear operation; Mi is the number of positive samples, PE(·) represents the position encoding, and the corresponding feature is selected from E(·) according to the index pair (j, positive or negative coordinate in Fj); E(·) represents the output of the encoder; F * Represents a feature pyramid;

[0045] S554: Input the customized positive query generated in S553 to the non-cascaded decoding layer of the auxiliary branch to obtain the marine ship target detection result Including: classification scores and bounding boxes; the non-cascaded decoding layer of the auxiliary branch shares parameters with the non-cascaded decoding layer of S54.

[0046] Preferably, the training model is iteratively updated by the total target loss function. The specific formula for calculating the total target loss function is as follows:

[0047] Calculate the loss function of the non-cascaded decoding layer:

[0048]

[0049] in represents the predicted value; y represents the true value; y = {c, b} where c and b represent the true category and bounding box; and Represents the predicted category and bounding box; IoU represents the overlap between the real box and the predicted box; L box represents the bounding box loss function; L cls represents the category loss function;

[0050] Calculate the loss function of the auxiliary detection head of the i-th auxiliary branch in the fusion compensation module:

[0051]

[0052] The regression loss of negative samples is discarded during the calculation process, and the training objective of the auxiliary detection head optimization is as follows:

[0053]

[0054] The loss function of the i-th auxiliary branch in the fusion compensation module can be expressed as:

[0055]

[0056] in, represents the output prediction of the non-cascaded decoding layer of the i-th auxiliary branch; P i {pos} represents the supervision target in positive coordinates;

[0057] Calculate the total objective loss function:

[0058]

[0059] Among them, λ1 and λ2 are the coefficients of balancing loss.

[0060] Preferably, the specific implementation method of the real-time DETR reasoning model without the fusion compensation module is:

[0061] S71: converting the input image into an RGB three-dimensional matrix; adjusting the RGB three-dimensional feature matrix into a feature matrix of a fixed size;

[0062] S72: Input the fixed-size feature matrix into the backbone network based on ResNet50 for feature extraction to obtain the features of each stage;

[0063] S73: Input the features of the last three stages in S72 into the encoding layer based on pyramid decoupling to obtain the encoded features

[0064] S74: Process the features encoded in S73 through PRN to obtain two-dimensional reference points, which include classification scores and bounding boxes. Sort the reference points according to their classification scores, select the first n reference points as the output features of the corresponding input, and use them as the initial object queries of the non-cascaded decoding layer;

[0065] S75: Input the features encoded by S73 and the initial object queries of S74 into the non-cascade decoding layer for fusion to obtain the fused result;

[0066] S76: Input the fused result of S75 into the FFN layer to obtain the category and bounding box of the detected marine ship;

[0067] S77: Draw the ship bounding box detected by S76 on the image input by S71.

[0068] Beneficial effects:

[0069] The present invention proposes a real-time marine ship target detection method based on improved DETR, and realizes marine ship target detection by optimizing the DERT model. By producing a sea-sky-line small target detection dataset, the problem of low accuracy of marine ship target detection model for small target marine ship targets is solved. Secondly, by using a coding layer and a non-cascaded decoding layer based on feature pyramid decoupling, the computing resources are reduced and the marine ship target detection speed is improved. Finally, the fusion compensation module is introduced only in the training stage, which not only improves the accuracy of marine ship target detection, but also ensures the detection speed of the model in the reasoning stage. Specifically:

[0070] 1) The real data of ships sailing at sea are used to create a sea-sky-line small target dataset, and the sea-sky-line small target dataset is used to train the marine ship target detection model, so as to train the marine ship target detection model to better identify and locate these difficult-to-detect targets, and further improve the accuracy of small target ship target detection near the sea-sky line.

[0071] 2) By using the encoding layer based on feature pyramid decoupling, the computing resources are greatly reduced while ensuring the processing of the output features of the backbone network, effectively improving the speed of marine ship detection. By combining features of different scales through a cross-scale fusion mechanism, an overall understanding of marine ships of different sizes is achieved, which helps to integrate feature information from different scales, enabling the model to better handle marine ship targets of different sizes and improve the comprehensiveness and accuracy of marine ship target detection.

[0072] 3) By using non-cascaded decoding layers, the number of decoding layers can be effectively reduced. The non-cascaded decoding layer structure can significantly reduce the computational complexity and memory requirements, thereby improving the training and inference speed of the marine ship target detection model. While reducing computing resources, it does not affect the fusion of the encoding layer output features that are decoupled from object queries and feature pyramids. Compared with the traditional multi-layer decoder structure, the non-cascaded decoding layer is easier to reach a convergence state.

[0073] 4) In the training phase, the bounding box IoU in RPN is used as the basis for initializing object queries. At the same time, the bounding box IoU is used to correct the decoding loss function to accelerate the convergence speed of the marine ship target detection model.

[0074] 5) A fusion compensation module is used in the training phase to improve the accuracy without affecting the inference speed. Multiple auxiliary branches are introduced in the fusion compensation module. The auxiliary detection head in the auxiliary branch helps to improve the expression ability of the feature map, and at the same time can provide richer supervision information, thereby improving the accuracy of the model for marine ship target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 Flowchart of a real-time marine ship target detection method based on improved DETR.

[0076] Figure 2 Schematic diagram of the cross-scale feature fusion module. DETAILED DESCRIPTION

[0077] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0078] like Figure 1 As shown, the specific steps of the present invention include:

[0079] S1: Collect the video stream data of ships sailing at sea, and capture pictures of the video stream data of ships sailing at sea at a set frequency; integrate the captured pictures into a marine ship image dataset;

[0080] S2: Use the Hough transform model to detect the coordinates (x1, y1, x2, y2) of the upper left corner and lower right corner of the sea-sky line of all images in the S1 marine ship image dataset, and cut the image according to the coordinates (x1, y1, x2, y2), and save the image within the coordinates; summarize the cut images to obtain the sea-sky line small target dataset;

[0081] S3: Manually annotate the S1 marine ship image dataset and the S2 sea-sky line small target dataset, and mix the two manually annotated datasets into the final marine ship image dataset; the manual annotation is to annotate the (x, y) coordinate information of the upper left corner and the lower right corner of the ship in the image;

[0082] S4: Perform data augmentation operations on the images in the final marine ship image dataset in S3;

[0083] S5: Build a real-time DETR training model with a fusion compensation module, including: a backbone network based on ResNet50, an encoding layer based on pyramid decoupling, a non-cascaded decoding layer, and a fusion compensation module;

[0084] S6: Using the data set after data enhancement in S4, train the real-time DETR training model with fusion compensation module in S5 and save the model weights;

[0085] S7: Build a real-time DETR model without fusion compensation module, including: backbone network based on ResNet50, encoding layer based on pyramid decoupling, and non-cascaded decoding layer;

[0086] S8: Import the model weights in S6 into the real-time DETR model constructed in S7 to perform real-time marine ship target detection on the marine ship video stream.

[0087] Specific content includes:

[0088] 1. Dataset creation

[0089] (1) First, the real data of ships sailing at sea is obtained, differentiated according to weather conditions, and captured at a frequency of one frame per second to obtain image data under different weather conditions. The sea-sky line in the image data is identified using the Hough transform model, and the coordinates of the upper left corner and lower right corner of the sea-sky line in the image area (x1, y1, x2, y2) are obtained. The image is cut according to the coordinates, and the image within the coordinates is saved in JPEG format to obtain the sea-sky line image data.

[0090] (2) The image data under different weather conditions are integrated to obtain a ship image dataset, and the sea-sky line image data are integrated to obtain a sea-sky line small target dataset.

[0091] (3) The sea-skyline small target dataset and the ship image dataset are mixed and shuffled, and randomly divided into training set and validation set in a ratio of 4:1.

[0092] (4) The dataset is further enhanced by performing data enhancement operations, including contrast enhancement, image rotation, image flipping, image affine transformation, and HSV data enhancement, to obtain a dataset with diverse features.

[0093] (5) The dataset is labeled with real boxes according to the format of the COCO dataset label to obtain a labeling file in JSON format. The labeling information is the (x, y) coordinate information of the upper left corner and lower right corner of the ship in the image. The labeled results will be used as input information for model training and information for verifying the quality of the model.

[0094] (6) During training, the data is read as a tensor matrix, the image data is transformed into an image matrix of size 640*640*3, and normalized according to the mean variance of the image.

[0095] 2. Build a real-time DETR training model with a fusion compensation module

[0096] (1) Select the ResNet50 network as the backbone network; input the image matrix into the backbone network features, construct a feature pyramid, and mark the last three layers of the feature pyramid as {S3, S4, S5}. The number of feature channels in each layer is 512, 1024, and 2048, respectively. The resolution of each layer of feature maps is (H / 8)*(W / 8), (H / 16)*(W / 16), and (H / 16)*(W / 16), respectively.

[0097] (2) The deep feature S5 is flattened to stretch it into a one-dimensional vector and copied three times as Q, V, and K of the multi-head attention. These three variables are exactly the same and are used as the input of the multi-head self-attention. The parameters of the multi-head self-attention mechanism are as follows:

[0098] Table 1 Multi-head attention mechanism parameter settings

[0099] Parameter name Specific parameters Activation Function gelu Hidden layer dimensions 256 Number of multi-head self-attention heads 8 Fully connected layer dimensions 1024 Dropout: 0

[0100] (3) Use the following code to multi-scale the features obtained by the backbone network:

[0101] nn.Sequential(nn.Conv2d(in_channel,hidden_dim,kernel_size=1,

[0102] bias=False),nn.BatchNorm2d(hidden_dim))

[0103] (4) Initialize the multi-head self-attention mechanism. The initialization code is as follows:

[0104] TransformerEncoderLayer(hidden_dim,nhead=nhead,dim_feedforward=dim_feedforward, dropout=dropout, activation=enc_act)

[0105] (5) {S3, S4, S5} is input into the cross-scale feature fusion module. The module structure is as follows: Figure 2 As shown, including:

[0106] Fusion module: The fusion module has two inputs. After the concatenation operation, the two inputs are sent to the 1*1 convolution kernel and the two branches consisting of the 1*1 convolution kernel and N layers of RepBlock respectively. The results calculated by the two branches are fused together through the element-wise addition operation and finally converted into a 512-dimensional vector through the Flatten operation.

[0107] S1 module: outputs the features after Conv1x1, BN, and SiLU operations;

[0108] S2 module: Output the features after Conv3x3, BN, and SiLU operations.

[0109] The calculation process of the cross-scale feature fusion module is as follows:

[0110] 1) Perform a Reshape operation on the Flatten transformed feature S5 to obtain a feature F5 with the same shape as feature S5;

[0111] 2) First, process feature F5 and input it into module S1 to get output F 5S1 The result is sent to two fusion modules marked as R1 and R2 respectively;

[0112] 3) R1 module receives feature F 5S1 and S4, and then the output R is obtained after fusion 11 , R 11 Input S1 module to get R 11S1 ;

[0113] 4) R 11S1 Combine S5 with the input of a new fusion module, and mark the fusion module as R3;

[0114] 5) Output R of R3 31 Perform S2 module processing and mark the processed result as R 31S2 ;

[0115] 6) R 31S2 and R 11S1 Input R1 fusion module to get the fusion result R 12 ;

[0116] 7) Fusion result R 12 After processing by S2 module, R 12S2 ;

[0117] 8) R 12S2 With F 5S1 Together they are used as the input of the fusion module R2, and after fusion, the output result R 21 ;

[0118] 9) R 21 , R 12 , R 31 The concatenate process is performed to obtain the output characteristics of the improved coding layer of the present invention.

[0119] (6) The output features of the coding layer are divided into two branches for operation, namely the non-cascaded decoding layer branch and the fusion compensation branch:

[0120] Non-cascaded decoding layer branch:

[0121] 1) Input the features of the encoding layer into the Region Proposal Network (RPN) to obtain two-dimensional reference points, which include classification scores and bounding boxes. Select the output features of the encoding layer corresponding to the first 100 reference points based on the bounding box scores and use them as the initial object queries of the non-cascaded decoding layer.

[0122] 2) Get the initial object queries through nn.Embedding(100,256);

[0123] 3) Define the non-cascaded decoding layer by the following code:

[0124] decoder_layer=TransformerdecoderLayer(hidden_dim,nhead,dim_feedforward,dropout,activation,num_levels,num_decoder_points)

[0125] Transformer decoder(hidden_dim,decoder_layer,num_decoder_layers,eval_idx)

[0126] The parameters of the non-cascaded decoding layer are set as follows:

[0127] hidden_dim=256, nhead=8, dim_feedforward=1024, dropout=0.,activaton='relu',num_levels=3,num_decoder_points=4,num_decoder_layers=1,eval_idx=-1

[0128] 4) Make object queries have high IoU values ​​while having high classification scores. In order to achieve the selection of high IoU and high scores for object queries, the loss function of the decoding is as follows:

[0129]

[0130] Fusion compensation module: The fusion compensation module transforms the output of the encoding layer, inputs the transformed result into the traditional target detection head such as ATSS and Faster-RCNN, and inputs the obtained prediction result as a positive sample into the non-cascade decoding layer to improve the effect of the non-cascade decoding layer and the convergence of training. The specific steps are as follows:

[0131] 1) Use the Multi-scale adapter to transform the output features of the encoding layer into a feature pyramid {F1,···,F J} where J represents a 2 2+J The downsampled feature map with the stride.

[0132] Specifically, the output features from the encoding layer are continuously down-sampled (3×3 convolution with a stride of 2) or up-sampled to generate a feature pyramid.

[0133] 2) Setting K (K=2) auxiliary detection heads of auxiliary branches and corresponding label allocation method A K ; {F1,···,F J} is sent to the auxiliary detection head of the i-th auxiliary branch to obtain the prediction Includes: bounding box coordinates, classification, confidence.

[0134] 3) Label allocation method A in the i-th auxiliary branch i calculate The supervision target of positive and negative samples, A i To calculate P i The supervision target of positive and negative samples in the . Let G be the true set, and the process can be expressed as:

[0135]

[0136] where {pos} and {neg} represent the i Determine the positive and negative samples; positive samples refer to the area containing the target object; negative samples refer to the area not containing the target object; G represents the true set; j represents {F1,····,F J} in feature index; is the set of positive coordinates in space; P i {pos} and P i {neg} is the supervised target in the corresponding coordinates, including the category and regression offset.

[0137] 4) The detailed information of the two auxiliary detection heads used are as follows:

[0138] Table 2 Detailed information of auxiliary detection head

[0139]

[0140] 5) The loss function is defined as:

[0141]

[0142] Note that the regression loss of negative samples is discarded, and the training objectives optimized by the two auxiliary detection heads are as follows:

[0143]

[0144] 6) Generate enough customized positive queries for each auxiliary branch to improve the learning efficiency of cross-attention in decoding. Specifically, given the positive coordinate set in the auxiliary detection head of the i-th auxiliary branch Where Mi is the number of positive samples, additional custom positive queries It can be generated by:

[0145]

[0146] Where PE(·) represents positional encoding, and we select the corresponding feature from E(·) according to the index pair (j, positive or negative coordinate in Fj).

[0147] 7) 3 groups of queries contribute to a single one-to-one set matching branch and 2 branches with one-to-many label assignment. The auxiliary one-to-many label assignment branch shares the same parameters as the non-cascaded decoding layer in the original main branch. All queries in the auxiliary branch are considered as positive queries and the matching process is discarded. The i-th auxiliary branch loss function of the non-cascaded decoding layer can be expressed as:

[0148]

[0149] in, represents the output prediction of the decoding layer of the i-th auxiliary detection head.

[0150] 8) The final training goal is:

[0151]

[0152] 9) After determining the total loss function, training is performed with batch_size = 16, epoch = 100 as the training iteration and batch size parameters.

[0153] 3. Build a real-time DETR detection model without fusion compensation module

[0154] (1) The construction of the encoding part is exactly the same as the construction of the encoding part in the real-time DETR training model with fusion compensation module

[0155] (2) For the initialization of object queries, the first 100 dimensional features of the encoded output are selected as object queries. The construction code is as follows:

[0156] nn.Embedding(100,hidden_dim)

[0157] (3) For the decoding part, a layer of non-cascaded decoding construction code is constructed as follows:

[0158] decoder_layer=TransformerdecoderLayer(hidden_dim,nhead,dim_feedforward,dropout,activation,num_levels,num_decoder_points)

[0159] Transformerdecoder(hidden_dim,decoder_layer,num_decoder_layers,eval_idx)

[0160] The parameters in the non-cascade decoding code are set as follows:

[0161] hidden_dim=256, nhead=8, dim_feedforward=1024, dropout=0.,activaton='

[0162] relu',num_levels=3,num_decoder_points=4,num_decoder_layers=1,

[0163] eval_idx = -1

[0164] (4) The encoded output features and object queries are fused, and the fused results are passed through FFN. The construction code of FFN is as follows:

[0165]

[0166] (5) Processing to obtain predicted classification scores and bounding boxes;

[0167] (6) Use the OpenCV image drawing tool to draw the results on the graph.

[0168] Specifically, in order to prove the effectiveness of the improved DETR for marine ship target detection, it is compared with the traditional industrial target detection model YOLOv5. The experimental results are shown in Table 1:

[0169] Table 3 Experimental comparison results of different models for marine ship target detection

[0170] Model Name <![CDATA[FPS bs＝1 ]]> Model size AP YOLOv5-X 54 86 50.7 Improved DETR 108 42 60.1

[0171] From the data in Table 3, it can be seen that when the improved DETR detects targets on marine ships, the image refresh frequency is twice that of YOLOv5-X, indicating that the detection speed of the improved DETR has been significantly improved; although the scale of the improved DETR model is reduced by half compared with YOLOv5-X, the detection accuracy is improved by 9.4%, indicating that the improved DETR can ensure the improvement of detection performance while reducing the model volume.

[0172] The above specific implementation modes are preferred embodiments of the present invention and cannot be used to limit the present invention. Any other changes or other equivalent replacement methods that do not deviate from the technical solution of the present invention are included in the protection scope of the present invention.

Claims

1. A real-time marine ship target detection method based on improved DETR, characterized in that: S1: Collecting video stream data of ships sailing at sea, and capturing pictures of the video stream data of ships sailing at sea at a set frequency; Integrate captured images into a marine vessel image dataset; S2: Use the Hough transform model to detect the coordinates (x1, y1, x2, y2) of the upper left corner and lower right corner of the sea-sky line of all images in the S1 marine ship image dataset, and cut the image according to the coordinates (x1, y1, x2, y2), and save the image within the coordinates; summarize the cut images to obtain the sea-sky line small target dataset; S3: Manually annotate the S1 marine ship image dataset and the S2 sea-sky line small target dataset, and mix the two manually annotated datasets into the final marine ship image dataset; the manual annotation is to annotate the (x, y) coordinate information of the upper left corner and the lower right corner of the ship in the image; S4: Perform data augmentation operations on the images in the final marine ship image dataset in S3; S5: Build a real-time DETR training model with a fusion compensation module, including: a backbone network based on ResNet50, an encoding layer based on pyramid decoupling, a non-cascaded decoding layer, and a fusion compensation module; S6: Using the data set after data enhancement in S4, train the real-time DETR training model with fusion compensation module in S5 and save the model weights; S7: Build a real-time DETR model without fusion compensation module, including: backbone network based on ResNet50, encoding layer based on pyramid decoupling, and non-cascaded decoding layer; S8: Import the model weights in S6 into the real-time DETR model constructed in S7 to perform real-time marine ship target detection on the marine ship video stream.

2. A real-time marine ship target detection method based on improved DETR according to claim 1, characterized in that: The data enhancement operations described in S4 include: contrast enhancement, image rotation, image flipping, affine transformation image expansion, and HSV data enhancement.

3. A real-time marine ship target detection method based on improved DETR according to claim 1, characterized in that: The real-time DETR training model method with fusion compensation module described in S5 is: S51: Convert the input image into an RGB three-dimensional feature matrix; Adjust the RGB three-dimensional feature matrix into a three-dimensional feature matrix of fixed size; S52: Input the fixed-size three-dimensional feature matrix into the backbone network based on ResNet50 for feature extraction to obtain the features of each stage in the backbone network; S53: Input the features {S3, S4, S5} of the last three stages in S52 into the encoding layer based on pyramid decoupling to obtain the encoded features; S54: Input the encoded features of S53 into the non-cascade decoding layer to obtain the category and bounding box of the marine ship; S55: inputting the potential features of S54 into a fusion compensation module to obtain a marine ship target detection result, including: a bounding box, a confidence level, and a category; the fusion compensation module includes a multi-scale adapter and K auxiliary branches; The auxiliary branch includes an auxiliary detection head and a non-cascaded decoding layer.

4. A real-time marine ship target detection method based on improved DETR according to claim 3, characterized in that: The specific method of encoding layers based on pyramid decoupling described in S53 is: S531: Flatten the feature S5 of the last stage of S52 to obtain Q, V, K; S532: Perform data shaping operation on Q, V, K to obtain a new feature F5 with the same shape as the feature S5; S533: Input the low-level features S3, S4 and the new feature F5 into the cross-scale feature fusion module to obtain fused features, and use the fused features as encoded features; The specific process can be summarized into the following calculation formula: Q=K=V=Flatten(S5) (1) F5=Reshape(Attn(Q,K,V)) (2) Output =CFM({S3, S4, F5}) (3) Among them, Flatten represents the flattening operation; Q is the Query vector as part of the request information, which is used to find relevant information in the input sequence; K is the Key vector that provides information about the input sequence and is used to match the Query vector; V is the Value vector containing the actual input data information. When the match with the Query vector is successful, the information will be extracted from V; Att represents the attention mechanism; CFM represents the cross-scale feature fusion module; Reshape represents the data reshaping operation.

5. A real-time marine ship target detection method based on improved DETR according to claim 3, characterized in that: In S54, the execution steps at the non-concatenated decoding layer are: S541: Input the features encoded in S53 into the region proposal network to obtain two-dimensional reference points, including classification scores and bounding boxes; S542: Sort the reference points according to the classification scores in S541, set and select the first n reference points as the output features corresponding to the features after encoding in S53, and use them as the initial object query of the non-cascaded decoding layer, i.e., objectqueries; S543: Input the encoded features of S53 and the initial object queries of S542 into the non-cascaded decoding layer for fusion to obtain the fused result; S544: Input the fused result of S543 into the FFN layer to obtain the category and bounding box of the detected marine vessel.

6. A real-time marine ship target detection method based on improved DETR according to claim 3, characterized in that: The specific method of the fusion compensation module described in S55 includes: S551: Use the multi-scale adapter to transform the encoded features of S53 into a feature pyramid {F1, ···, F J }; S552: Setting the auxiliary detection heads of K auxiliary branches and the corresponding label allocation method A K ; {F1,···,F J }Sent to the auxiliary detection head of the i-th auxiliary branch to obtain the target detection result of the marine ship Includes: bounding box coordinates, classification and confidence; using the label assignment method A in the i-th auxiliary branch i calculate Supervision objectives for positive and negative samples; The process can be described as: where {pos} and {neg} represent the i Determine the positive samples and negative samples; positive samples refer to the area containing the target object; negative samples refer to the area that does not contain the target object; G represents the true set; j represents {F1,····,F J } in feature index; is the set of positive coordinates in space; P i {pos} and P i {neg} is the supervision target in the corresponding coordinates, including the category and regression offset; S553: ​​The spatial positive coordinate set of the auxiliary detection head of the i-th auxiliary branch obtained in S552 Generate additional custom positive queries All queries in the auxiliary branch are considered positive queries; they can be generated as follows: Where Linear represents a linear operation; Mi is the number of positive samples, PE(·) represents the position encoding, and the corresponding feature is selected from E(·) according to the index pair (j, positive or negative coordinate in Fj); E(·) represents the output of the encoder; F * Represents a feature pyramid; S554: Input the customized positive query generated in S553 to the non-cascaded decoding layer of the auxiliary branch to obtain the marine ship target detection result Including: classification scores and bounding boxes; the non-cascaded decoding layer of the auxiliary branch shares parameters with the non-cascaded decoding layer of S54.

7. A real-time marine ship target detection method based on improved DETR according to claim 1, characterized in that: The training model is to iteratively update the model through the total target loss function. The specific formula for calculating the total target loss function is as follows: Calculate the loss function of the non-cascaded decoding layer: in represents the predicted value; y represents the true value; y = {c, b} where c and b represent the true category and bounding box; and Represents the predicted category and bounding box; IoU represents the overlap between the real box and the predicted box; L box represents the bounding box loss function; L cls represents the category loss function; Calculate the loss function of the auxiliary detection head of the i-th auxiliary branch in the fusion compensation module: The regression loss of negative samples is discarded during the calculation process, and the training objective of the auxiliary detection head optimization is as follows: The loss function of the i-th auxiliary branch in the fusion compensation module can be expressed as: in, represents the output prediction of the non-cascaded decoding layer of the i-th auxiliary branch; P i {pos} represents the supervision target in positive coordinates; Calculate the total objective loss function: Among them, λ1 and λ2 are the coefficients of balancing loss.

8. A real-time marine ship target detection method based on improved DETR according to claim 1, characterized in that: The specific implementation method of the real-time DETR reasoning model without the fusion compensation module is as follows: S71: converting the input image into an RGB three-dimensional matrix; adjusting the RGB three-dimensional feature matrix into a feature matrix of a fixed size; S72: Input the fixed-size feature matrix into the backbone network based on ResNet50 for feature extraction to obtain the features of each stage; S73: Input the features of the last three stages in S72 into the encoding layer based on pyramid decoupling to obtain the encoded features S74: Process the features encoded in S73 through PRN to obtain two-dimensional reference points, which include classification scores and bounding boxes. Sort the reference points according to their classification scores, select the first n reference points as the output features of the corresponding input, and use them as the initial object queries of the non-cascaded decoding layer; S75: Input the features encoded by S73 and the initial object queries of S74 into the non-cascade decoding layer for fusion to obtain the fused result; S76: Input the fused result of S75 into the FFN layer to obtain the category and bounding box of the detected marine ship; S77: Draw the ship bounding box detected by S76 on the image input by S71.

Images