A real-time offshore ship target detection method based on improved detr

Abstract

The application belongs to the field of ship detection, and proposes a real-time offshore ship target detection method based on improved DETR. Through the preparation of sea-sky small target detection data set, the problem of low accuracy of offshore ship target detection for small target offshore ship target detection is solved. Secondly, by using the encoding layer and non-cascading decoding layer based on feature pyramid decoupling, the calculation resources are reduced, and the offshore ship target detection speed is improved. Finally, the fusion compensation module is introduced, a plurality of auxiliary detection heads are introduced in the fusion compensation module, more rich supervision information is provided, which is helpful to improve the expression ability of the feature map and improve the recognition accuracy of the model for offshore ships. At the same time, the fusion compensation module is only used in the training stage, so as to improve the accuracy of offshore ship target detection and ensure the detection speed of the model in the inference stage.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of ship detection, and in particular to a real-time offshore ship target detection method based on improved DETR. BACKGROUND

[0002] In the complex and changeable marine environment, ship target detection is a crucial task. It is not only related to maritime traffic safety, marine resource management, but also directly affects national border security and other fields. By monitoring the position of ships at sea in real time, it helps to optimize channel management, avoid ship collisions and ensure safe navigation; at the same time, it is of great significance to maintain marine order, ensure maritime safety and promote the development of marine economy.

[0003] The current target detection technology applied to offshore ships is YoloV5, but its target detection accuracy and detection speed depend on the design of anchor frames. When designing and training for the task of offshore ship target detection, the deviation of anchor frame design will result in extremely poor target detection results, which is not conducive to the compatibility deployment of mainstream development boards on the market. At the same time, YoloV5 relies on manual setting of NMS and other post-processing operations, and the post-processing operation time increases with the increase in the number of target detection, and the average time required for post-processing operation is twice the model inference time, which seriously affects the target detection speed of offshore ships. The DETR model based on the Transform architecture can well solve the above problems existing in the traditional YoloV5 model, but the traditional DETR has a large number of parameters and requires certain computing resources, which is not conducive to deployment and inference on development boards. The Transform architecture in the DETR model is related to the front and back for global features, resulting in poor target detection accuracy for small target ships with few features. SUMMARY

[0004] In view of the problems in the prior art of the ship detection field that the DETR model has a large number of parameters and high computing requirements, resulting in slow inference speed of the DETR model after deployment on the development board, and the DETR model has poor target detection accuracy for small target offshore ships with few features, a real-time offshore ship target detection method based on improved DETR is proposed, which reduces the parameter quantity and computing requirements of the model, realizes real-time offshore ship target detection, and improves the target detection accuracy for small target offshore ships.

[0005] The specific scheme is as follows:

[0006] A real-time offshore ship target detection method based on improved DETR, comprising:

[0007] S1: collect ship offshore navigation video stream data, and capture pictures of ship offshore navigation video stream data at a set frequency; integrate the captured pictures into offshore ship image data set;

[0008] S2: detect the coordinates (x1, y1, x2, y2) of the upper left corner and the lower right corner of the sea-sky line in all pictures in the S1 offshore ship image data set using a Hough transform model, cut the image according to the coordinates (x1, y1, x2, y2), and save the image within the coordinates; and summarize the cut images to obtain a sea-sky line small target data set;

[0009] S3: artificially label the S1 offshore ship image data set and the S2 sea-sky line small target data set, and mix the two data sets after artificial labeling into a final offshore ship image data set; the artificial labeling is labeling the (x, y) coordinate information of the upper left corner and the lower right corner of the ship in the image;

[0010] S4: perform data enhancement operation on the pictures in the S3 final offshore ship image data set;

[0011] S5: construct a real-time DETR training model with a fusion compensation module, including: a backbone network based on ResNet50, an encoding layer based on pyramid decoupling, a non-cascaded decoding layer, and a fusion compensation module;

[0012] S6: train the real-time DETR training model with a fusion compensation module in S5 using the data set after data enhancement in S4, and save the model weight;

[0013] S7: construct a real-time DETR model without a fusion compensation module, including: a backbone network based on ResNet50, an encoding layer based on pyramid decoupling, and a non-cascaded decoding layer;

[0014] S8: import the model weight in S6 into the real-time DETR model constructed in S7, and perform real-time offshore ship target detection on the offshore ship video stream.

[0015] Preferably, the data enhancement operation in S4 includes contrast enhancement, rotating the image, flipping the image, affine transformation to expand the image, and HSV data enhancement.

[0016] Preferably, the real-time DETR training model with a fusion compensation module in S5 is:

[0017] S51: convert the input picture into an RGB three-dimensional feature matrix; and adjust the RGB three-dimensional feature matrix into a fixed-size three-dimensional feature matrix;

[0018] S52: input the fixed-size three-dimensional feature matrix into the backbone network based on ResNet50 for feature extraction to obtain the features of each stage in the backbone network;

[0019] S53: input the features of the last three stages {S3, S4, S5} in S52 into the encoding layer based on pyramid decoupling to obtain the encoded features;

[0020] S54: input the encoded features in S53 into the non-cascaded decoding layer to obtain the category and bounding box of the marine vessel;

[0021] S55: input the latent features in S54 into the fusion compensation module to obtain the marine vessel target detection result, including the bounding box, confidence and category; the fusion compensation module includes a multi-scale adapter and K auxiliary branches; the auxiliary branch includes an auxiliary detection head and a non-cascaded decoding layer.

[0022] Preferably, the specific method of the encoding layer based on pyramid decoupling in S53 is:

[0023] S531: perform flattening operation on the feature S5 of the last stage in S52 to obtain Q, V and K;

[0024] S532: perform data reshaping operation on Q, V and K to obtain a new feature F5 with the same shape as S5;

[0025] S533: input the low-level features S3, S4 and the new feature F5 into the cross-scale feature fusion module to obtain the fused features, and use the fused features as the encoded features;

[0026] The specific process can be summarized as the following calculation formula:

[0027] Q = K = V = Flatten(S5) (1)

[0028] F5 = Reshape(Attn(Q, K, V)) (2)

[0029] Output = CFM({S3, S4, F5}) (3)

[0030] Wherein, Flatten represents flattening operation; Q is the Query vector as part of the request information, used to find relevant information in the input sequence; K is the Key vector providing information about the input sequence, used to match with the Query vector; V is the Value vector containing actual input data information, which will extract information from V when matched with the Query vector successfully; Att represents attention mechanism; CFM represents cross-scale feature fusion module; Reshape represents data reshaping operation.

[0031] Preferably, the execution steps in S54 of the non-concatenated decoding layer are as follows:

[0032] S541: Input the S53 encoded features into the Region Proposal Network (RPN) to obtain two-dimensional reference points, including classification scores and bounding boxes;

[0033] S542: Sort the reference points according to the classification scores in the reference points of S541, and select the first n reference points as the output features of the corresponding S53 encoded features, and use them as the initial object queries of the non-cascaded decoding layer.

[0034] S543: Input the S53 encoded features and the initial S542 object queries into the non-concatenated decoding layer for fusion to obtain the fused result;

[0035] S544: Input the fused result of S543 into the FFN layer to obtain the category and bounding box of the detected ships at sea.

[0036] Preferably, the specific method of the fusion compensation module described in S55 includes:

[0037] S551: Using a multi-scale adapter, the features encoded by S53 are transformed into a feature pyramid {F1, ..., F...} J};

[0038] S552: Set the auxiliary detection heads for K auxiliary branches and the corresponding label allocation method A K ; set {F1,···,F J The auxiliary detection head of the i-th auxiliary branch is sent to obtain the target detection results of the ships at sea. This includes: bounding box coordinates, classification, and confidence score; and label assignment method A from the i-th auxiliary branch. i calculate The supervisory targets for positive and negative samples;

[0039] This process can be described as follows:

[0040]

[0041] Where {pos} and {neg} represent the result of A i Defined positive and negative samples; positive samples refer to regions containing the target object; negative samples refer to regions that do not contain the target object; G represents the true set; j represents {F1, ..., F...} J Feature index in}; It is the set of positive coordinates in space; P i{pos} and P i {neg} are supervised targets in corresponding coordinates, including class and regression offset;

[0042] S553: obtaining the set of spatial positive coordinates in the auxiliary detection head of the i-th auxiliary branch according to S552 generate additional customized positive queries All queries in the auxiliary branch are considered as positive queries; Specifically, they can be generated in the following way:

[0043]

[0044] where Linear represents linear operation; Mi is the number of positive samples, PE(·) represents position encoding, and the corresponding features are selected from E(·) according to the index (positive coordinates or negative coordinates in Fj); E(·) represents the output of the encoder; F * represents a feature pyramid;

[0045] S554: inputting the customized positive queries generated in S553 into the non-cascaded decoding layer of the auxiliary branch to obtain the marine ship target detection result including: classification score and bounding box; The non-cascaded decoding layer of the auxiliary branch shares parameters with the non-cascaded decoding layer of S54.

[0046] Preferably, the training model is updated iteratively by the total target loss function, and the formula for calculating the total target loss function is as follows:

[0047] Calculate the non-cascaded decoding layer loss function:

[0048]

[0049] where represents the predicted value; y represents the true value; y={c,b} where c and b represent the real class and bounding box; and represent the predicted class and bounding box; IoU represents the overlap degree of the real box and the predicted box; L box represents the bounding box loss function; L cls represents the class loss function;

[0050] Calculate the loss function of the auxiliary detection head of the i-th auxiliary branch in the fusion compensation module:

[0051]

[0052] In the calculation process, the regression loss of the negative sample is discarded, and the training target optimized by the auxiliary detection head is as follows:

[0053]

[0054] The loss function of the i-th auxiliary branch in the fusion compensation compensation module can be expressed as:

[0055]

[0056] Wherein, represents the output prediction of the non-cascaded decoding layer of the i-th auxiliary branch; P i {pos} represents the supervised target in the positive coordinate;

[0057] The total target loss function is calculated as:

[0058]

[0059] Wherein, λ1 and λ2 are coefficients for balancing loss.

[0060] Preferably, the real-time DETR inference model without fusion compensation module is implemented as follows:

[0061] S71: The input picture is converted into an RGB three-dimensional matrix; the RGB three-dimensional feature matrix is adjusted into a fixed size feature matrix;

[0062] S72: The fixed size feature matrix is input into the backbone network based on ResNet50 for feature extraction to obtain the features of each stage;

[0063] S73: The features of the last three stages in S72 are input into the encoding layer based on pyramid decoupling to obtain the encoded features

[0064] S74: The encoded features in S73 are processed by PRN to obtain two-dimensional reference points, the reference points contain classification scores and bounding boxes, and the reference points are sorted according to the classification scores, and the first n reference points are selected as the output features corresponding to the input, which are used as the initial object queries of the non-cascaded decoding layer;

[0065] S75: The encoded features in S73 and the initial object queries in S74 are input into the non-cascaded decoding layer for fusion to obtain the fused results;

[0066] S76: The fused results in S75 are input into the FFN layer to obtain the detected categories and bounding boxes of the marine vessels;

[0067] S77: The detected ship bounding boxes in S76 are drawn on the input picture in S71.

[0068] Beneficial effects:

[0069] The application proposes a real-time offshore ship target detection method based on improved DETR. The target detection of offshore ships is realized by optimizing the DERT model. Through the preparation of the sea-sky small target detection data set, the problem of low accuracy of the offshore ship target detection model in detecting small target offshore ships is solved. Secondly, by using the encoding layer and non-cascading decoding layer based on feature pyramid decoupling, the calculation resources are reduced and the offshore ship target detection speed is improved. Finally, only the fusion compensation module is introduced in the training stage, which not only improves the accuracy of offshore ship target detection, but also ensures the detection speed of the model in the inference stage. Specifically:

[0070] 1) Use the obtained real data of ship navigation at sea to prepare a sea-sky small target data set, and use the sea-sky small target data set to train the offshore ship target detection model, so as to train the offshore ship target detection model to better identify and locate these difficult to detect targets, and further improve the small target ship target detection accuracy near the sea-sky.

[0071] 2) By using the encoding layer based on feature pyramid decoupling, the calculation resources are greatly reduced while ensuring the processing of the output features of the backbone network, effectively improving the offshore ship target detection speed. Through the cross-scale fusion mechanism, different scale features are merged to realize the overall understanding of offshore ships of different sizes, which helps to integrate feature information from different scales, so that the model can better process offshore ship targets of different sizes, and improve the comprehensiveness and accuracy of offshore ship target detection.

[0072] 3) By using the non-cascading decoding layer, the number of decoding layers can be effectively reduced, and the non-cascading decoding layer structure can significantly reduce the calculation complexity and memory requirement, thereby improving the training and inference speed of the offshore ship target detection model. While reducing the calculation resources, it does not affect the fusion of object queries and feature pyramid decoupled encoding layer output features. Compared with the traditional multi-layer decoder structure, the non-cascading decoding layer is easier to reach a convergence state.

[0073] 4) In the training stage, the boundary box IoU in the RPN is used as the initialization basis for object queries, and the boundary box IoU is used to correct the loss function of decoding, to speed up the convergence speed of the offshore ship target detection model.

[0074] 5) The fusion compensation module is used in the training stage to improve the accuracy without affecting the inference speed. In the fusion compensation module, multiple auxiliary branches are introduced. The auxiliary detection head in the auxiliary branch helps to improve the expression ability of the feature map, and at the same time can provide more rich supervision information, and improve the accuracy of the model in detecting offshore ship targets. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 A flow chart of a real-time offshore ship target detection method based on improved DETR.

[0076] Figure 2 Schematic diagram of a cross-scale feature fusion module. DETAILED DESCRIPTION

[0077] The application will be further described below in conjunction with the drawings and examples.

[0078] As Figure 1 shown, the specific steps of the application include:

[0079] S1: Collecting ship offshore navigation video stream data, and capturing pictures of the ship offshore navigation video stream data at a set frequency; integrating the captured pictures into an offshore ship image dataset;

[0080] S2: Detecting the coordinates (x1, y1, x2, y2) of the upper left corner and the lower right corner of the horizon line in all pictures in the S1 offshore ship image dataset using a Hough transform model, cutting the image according to the coordinates (x1, y1, x2, y2), and saving the image within the coordinates; and summarizing the cut images to obtain a horizon line small target dataset;

[0081] S3: Artificially labeling the S1 offshore ship image dataset and the S2 horizon line small target dataset, and mixing the two artificially labeled datasets into a final offshore ship image dataset; the artificial labeling is labeling the (x, y) coordinate information of the upper left corner and the lower right corner of the ship in the image;

[0082] S4: Performing a data enhancement operation on the pictures in the S3 final offshore ship image dataset;

[0083] S5: Constructing a real-time DETR training model with a fusion compensation module, including: a backbone network based on ResNet50, an encoding layer based on pyramid decoupling, a non-cascaded decoding layer, and a fusion compensation module;

[0084] S6: Training the real-time DETR training model with the fusion compensation module in S5 using the dataset enhanced in S4, and saving the model weight;

[0085] S7: Constructing a real-time DETR model without a fusion compensation module, including: a backbone network based on ResNet50, an encoding layer based on pyramid decoupling, and a non-cascaded decoding layer;

[0086] S8: Importing the model weight in S6 into the real-time DETR model constructed in S7 to perform real-time offshore ship target detection on the offshore ship video stream.

[0087] The specific content includes:

[0088] 1. Dataset making

[0089] (1) First, the real data of ship sea navigation is obtained, which is divided according to the weather condition, and captured at a frequency of one frame per second to obtain image data under different weather conditions. The horizon line in the image data is identified by using the Hough transform model to obtain the coordinates (x1, y1, x2, y2) of the upper left corner and the lower right corner of the horizon line in the image. According to the coordinates, the image is cut, and the image in the coordinates is saved in JPEG format to obtain the horizon line image data.

[0090] (2) The image data under different weather conditions is integrated to obtain a ship image dataset, and the horizon line image data is integrated to obtain a horizon line small target dataset.

[0091] (3) The horizon line small target dataset and the ship image dataset are mixed and shuffled, and randomly divided into a training set and a validation set according to a ratio of 4:1.

[0092] (4) Further data augmentation processing is performed on the dataset, which includes contrast enhancement, rotating the image, flipping the image, affine transformation to expand the image, and HSV data enhancement to obtain a dataset with diversified features.

[0093] (5) The dataset is labeled with real boxes according to the format of the COCO dataset label to obtain a JSON format annotation file. The annotation information is the (x, y) coordinate information of the upper left corner and the lower right corner of the ship in the image. The labeled result will be used as input information during model training and information for verifying the pros and cons of the model.

[0094] (6) During training, the data is read as a tensor matrix, the image data is transformed into a 640*640*3 size image matrix, and normalization processing is performed according to the mean and variance of the image.

[0095] 2. Constructing a real-time DETR training model with a fusion compensation module

[0096] (1) Selecting ResNet50 network as the backbone network; inputting the image matrix into the backbone network feature to construct a feature pyramid, marking the last three layers of the feature pyramid as {S3, S4, S5}, and the feature channel numbers of each layer are 512, 1024 and 2048 respectively. The resolution of each layer of feature map is (H / 8)*(W / 8), (H / 16)*(W / 16), (H / 16)*(W / 16) respectively.

[0097] (2) Deep features S5 are transformed by Flatten to stretch them into one-dimensional vectors and copied three times as the Q, V, and K of multi-head attention. The three variables are exactly the same and are used as the input of multi-head self-attention. The multi-head self-attention mechanism parameters are as follows:

[0098] Table 1 Multi-head attention mechanism parameter settings

[0099] Parameter name Specific parameter Activation function gelu Hidden layer dimension 256 Multi-head self-attention head number 8 Fully connected layer dimension 1024 Dropout: 0

[0100] (3) The following code can be used to perform multi-scale on the features obtained by the backbone network:

[0101] nn.Sequential(nn.Conv2d(in_channel, hidden_dim, kernel_size = 1,

[0102] bias = False), nn.BatchNorm2d(hidden_dim))

[0103] (4) The multi-head self-attention mechanism is initialized, and the initialization code is as follows:

[0104] TransformerEncoderLayer(hidden_dim, nhead = nhead, dim_feedforward = dim_feedforward, dropout = dropout, activation = enc_act)

[0105] (5) Input {S3, S4, S5} into the cross-scale feature fusion module, which has the structure as shown in Figure 2 , including:

[0106] Fusion module: The fusion module has two inputs. After the Concatenate operation, the two inputs are sent to a 1*1 convolution kernel and two branches composed of a 1*1 convolution kernel and N layers of RepBlock. The results calculated by the two branches are fused together by the Element-wise Addition operation, and finally the Flatten operation is performed to convert them into a 512-dimensional vector.

[0107] S1 module: The feature is output after the Conv1x1, BN, and SiLU operations.

[0108] S2 module: The feature is output after the Conv3x3, BN, and SiLU operations.

[0109] The cross-scale feature fusion module calculation process is as follows:

[0110] 1) Reshape operation is performed on the feature S5 after the Flatten transformation, to obtain a feature F5 with the same shape as the feature S5;

[0111] 2) The feature F5 is first processed, and the processed feature is input into the S1 module to obtain an output F 5S1 The result is sent to two fusion modules marked as R1 and R2, respectively;

[0112] 3) The R1 module receives the feature F 5S1 and the feature S4, and the fusion result is an output R 11 The R 11 is input into the S1 module to obtain an output R 11S1 ;

[0113] 4) The R 11S1 is input into a new fusion module together with the feature S5, and the fusion module is marked as R3;

[0114] 5) The output R 31 of the R3 is processed by the S2 module, and the processed result is marked as R 31S2 ;

[0115] 6) The R 31S2 and the R 11S1 are input into the R1 fusion module to obtain a fusion result R 12 ;

[0116] 7) The fusion result R 12 is processed by the S2 module to obtain an output R 12S2 ;

[0117] 8) The R 12S2 is input into the fusion module R2 together with the feature F 5S1 , and the fusion result is an output result R 21 ;

[0118] 9) The R 21 , the R 12 , and the R 31 are concatenated to obtain the output feature of the improved encoding layer.

[0119] (6) The output feature of the encoding layer is divided into two branches for operation, which are a non-cascaded decoding layer branch and a fusion compensation branch, respectively:

[0120] Non-cascaded decoding layer branch:

[0121] 1) The obtained feature of the encoding layer is input into the region proposal network (RPN) to obtain two-dimensional reference points, and the reference points contain classification scores and boundary boxes. According to the boundary box score, the top 100 reference points corresponding to the output features of the encoding layer are selected as the initial object queries of the non-cascaded decoding layer.

[0122] 2) Obtain initial object queries through nn.Embedding(100, 256);

[0123] 3) Define the non-cascaded decoding layer through the following code:

[0124] decoder_layer = TransformerdecoderLayer(hidden_dim, nhead, dim_feedforward, dropout, activation, num_levels, num_decoder_points)

[0125] Transformer decoder(hidden_dim, decoder_layer, num_decoder_layers, eval_idx)

[0126] The parameter settings of the non-cascaded decoding layer are as follows:

[0127] hidden_dim = 256, nhead = 8, dim_feedforward = 1024, dropout = 0., activaton = ’relu’, num_levels = 3, num_decoder_points = 4, num_decoder_layers = 1, eval_idx = -1

[0128] 4) Make the object queries have high IoU values while having high classification scores. In order to realize the selection of object queries with high IoU and high score scores, the loss function of the decoding is as follows:

[0129]

[0130] The fusion compensation module is obtained by transforming the output results of the encoding layer, and then inputting the transformed results into traditional target detection heads such as ATSS and Faster-RCNN. The obtained prediction results are input into the non-cascaded decoding layer as positive samples, which is used to improve the effect of the non-cascaded decoding layer and the convergence of the training. The specific steps are as follows:

[0131] 1) Use a multi-scale adapter to transform the output features of the encoding layer into a feature pyramid {F1, ..., F2}. J} where J represents having 2 2+J Downsampling feature map of step size.

[0132] Specifically, the feature pyramid is generated by using continuous downsampling (3×3 convolution with a stride of 2) or upsampling operations on the output features from the coding layer.

[0133] 2) Set up auxiliary detection heads for K (K=2) auxiliary branches and the corresponding label allocation method A K ; set {F1,···,F J The auxiliary detection head is sent to the i-th auxiliary branch to obtain the prediction. Includes: bounding box coordinates, classification, and confidence score.

[0134] 3) Label assignment method A in the i-th auxiliary branch i calculate The supervisory targets for positive and negative samples, A i Used to calculate P i The supervision targets for positive and negative samples. Let G represent the true set; this process can be expressed as:

[0135]

[0136] Where {pos} and {neg} represent the result of A i Defined positive and negative samples; positive samples are regions containing the target object; negative samples are regions that do not contain the target object; G represents the true set; j represents {F1, ..., F...} J Feature index in}; It is the set of positive coordinates in space; P i {pos} and P i {neg} It refers to the supervised target in the corresponding coordinates, including the category and regression offset.

[0137] 4) Details of the two auxiliary detection heads used are shown in the table below:

[0138] Table 2. Detailed information about the auxiliary detection head.

[0139]

[0140] 5) The loss function is defined as:

[0141]

[0142] Note that the regression loss of negative samples is discarded, and the training objectives of the two auxiliary detection heads are as follows:

[0143]

[0144] 6) Generate enough customized positive queries for each auxiliary branch to improve the learning efficiency of cross-attention in decoding. Specifically, given the positive coordinate set in the auxiliary detection head of the ith auxiliary branch Where Mi is the number of positive samples, and additional customized positive queries Can be generated as follows:

[0145]

[0146] Where PE(·) represents position encoding, and we select the corresponding features from E(·) according to the index pair (j, positive coordinates or negative coordinates in Fj).

[0147] 7) The three groups of queries help the single one-to-one set matching branch and the two branches with one-to-many label assignment. The auxiliary one-to-many label assignment branch shares the same parameters with the non-cascaded decoding layer in the original main branch. All queries in the auxiliary branch are considered as positive queries, and the matching process is discarded. The loss function of the ith auxiliary branch of the non-cascaded decoding layer can be represented as:

[0148]

[0149] Where, represents the output prediction of the decoding layer of the ith auxiliary detection head.

[0150] 8) The final training objective is:

[0151]

[0152] 9) After determining the total loss function, train with batch_size = 16, epoch = 100 as the training iteration and batch size parameters.

[0153] 3. Construct a real-time DETR detection model without fusion compensation module

[0154] (1) The encoding part is constructed exactly the same as in the construction of the encoding part of the real-time DETR training model with the fusion compensation module

[0155] (2) For the initialization of object queries, select the first 100-dimensional features of the encoding output as object queries, and the construction code is as follows:

[0156] nn.Embedding(100,hidden_dim)

[0157] (3) For the decoding part, a non-cascaded decoding construction code is constructed as follows:

[0158] decoder_layer = TransformerdecoderLayer (hidden_dim, nhead, dim_feedforward, dropout, activation, num_levels, num_decoder_points)

[0159] Transformerdecoder (hidden_dim, decoder_layer, num_decoder_layers, eval_idx)

[0160] The parameter settings in the code of non-cascaded decoding are as follows:

[0161] hidden_dim = 256, nhead = 8, dim_feedforward = 1024, dropout = 0., activaton = '

[0162] relu', num_levels = 3, num_decoder_points = 4, num_decoder_layers = 1,

[0163] eval_idx = -1

[0164] (4) The features output by the encoding are fused with the object queries, and the fused results are passed through an FFN. The construction code of the FFN is as follows:

[0165]

[0166] (5) The processing is performed to obtain the predicted classification score and the bounding box;

[0167] (6) The results are drawn on the graph using the opencv image drawing tool.

[0168] Specifically, in order to prove the effectiveness of the improved DETR in detecting marine ship targets, a comparison is made with the traditional industrial target detection model YOLOv5, and the experimental results are shown in Table 1:

[0169] Table 3 Comparison of experimental results of different models for marine ship target detection

[0170] Model name FPS bs＝1 ]]> Model size AP YOLOv5-X 54 86 50.7 Improved DETR 108 42 60.1

[0171] From the data in Table 3, it can be seen that the improved DETR has a refresh frequency of the image that is twice that of YOLOv5-X when performing target detection on a ship at sea, indicating that the detection speed of the improved DETR has been significantly improved; although the size of the model of the improved DETR is reduced by half relative to YOLOv5-X, the detection accuracy is improved by 9.4%, indicating that the improved DETR can reduce the model volume while still ensuring the improvement of the detection performance.

[0172] The above specific embodiments are the preferred embodiments of the present application, and cannot limit the present application, and any changes or other equivalent replacement manners made without departing from the technical solutions of the present application are included in the protection scope of the present application.

Claims

1. An improved DETR-based real-time offshore ship target detection method, characterized in that, S1: Collecting ship offshore navigation video stream data, and capturing pictures of ship offshore navigation video stream data at a set frequency; Integrating the captured pictures into an offshore ship image dataset; S2: Detecting the coordinates (x1, y1, x2, y2) of the upper left corner and the lower right corner of the sea-sky line in all pictures in the S1 offshore ship image dataset using a Hough transform model, cutting the image according to the coordinates (x1, y1, x2, y2), and saving the image within the coordinates; and summarizing the cut images to obtain a sea-sky line small target dataset; S3: Artificially labeling the S1 offshore ship image dataset and the S2 sea-sky line small target dataset, and mixing the two artificially labeled datasets to form a final offshore ship image dataset; the artificial labeling is labeling the (x, y) coordinate information of the upper left corner and the lower right corner of the ship in the image; S4: Performing data enhancement operations on the pictures in the S3 final offshore ship image dataset; S5: Constructing a real-time DETR training model with a fusion compensation module, including: a backbone network based on ResNet50, an encoding layer based on pyramid decoupling, a non-cascaded decoding layer, and a fusion compensation module; S6: Training the real-time DETR training model with a fusion compensation module in S5 using the data set after data enhancement in S4, and saving the model weight; S7: Constructing a real-time DETR model without a fusion compensation module, including: a backbone network based on ResNet50, an encoding layer based on pyramid decoupling, and a non-cascaded decoding layer; S8: Importing the model weight in S6 into the real-time DETR model constructed in S7 to perform real-time offshore ship target detection on the offshore ship video stream.

2. The real-time offshore ship target detection method based on improved DETR according to claim 1, characterized in that, The data enhancement operations in S4 include: contrast enhancement, rotating the image, flipping the image, affine transformation to expand the image, and HSV data enhancement.

3. The real-time offshore ship target detection method based on improved DETR according to claim 1, characterized in that, The real-time DETR training model with a fusion compensation module in S5 is: S51: Converting the input picture into an RGB three-dimensional feature matrix; Adjusting the RGB three-dimensional feature matrix into a fixed-size three-dimensional feature matrix; S52: Inputting the fixed-size three-dimensional feature matrix into the backbone network based on ResNet50 for feature extraction to obtain the features at each stage in the backbone network; S53: features of the last three stages in S52 input the encoding layer based on pyramid decoupling to obtain the encoded features; S54: Inputting the encoded features in S53 into the non-cascaded decoding layer to obtain the class and bounding box of the offshore ship; S55: Inputting the latent features in S54 into the fusion compensation module to obtain the offshore ship target detection results, including the bounding box, confidence, and class; the fusion compensation module includes a multi-scale adapter and K auxiliary branches; The auxiliary branch includes an auxiliary detection head and a non-cascaded decoding layer.

4. The real-time offshore ship target detection method based on improved DETR according to claim 3, characterized in that, The specific method of the encoding layer based on pyramid decoupling in S53 is: S531: features from the last stage of S52 perform a flattening operation to obtain ; S532: obtaining the data shaping operation performing a data shaping operation to obtain and a new feature with the same characteristic shape ; S533: input the low-level features and the new features into the cross-scale feature fusion module to obtain fused features, and take the fused features as the encoded features. The specific process is summarized as follows: (1) (2) (3) Wherein, Flatten represents a flattening operation; Q is a Query vector as part of the request information for finding relevant information in the input sequence; K is a Key vector providing information about the input sequence for matching with the Query vector; V is a Value vector containing actual input data information, which will be extracted from V when matching with the Query vector is successful; Att represents an attention mechanism; CFM represents a cross-scale feature fusion module; Reshape represents a data reshaping operation.

5. The real-time offshore ship target detection method based on improved DETR according to claim 3, characterized in that, The execution step of the non-cascaded decoding layer in S54 is: S541: input the encoded features in S53 into the region proposal network to obtain two-dimensional reference points, including classification scores and bounding boxes; S542: sort the reference points according to the classification scores in the reference points, and set the top n reference points as the output features corresponding to the encoded features in S53, which are also the initial object queries of the non-cascaded decoding layer; S543: input the encoded features in S53 and the initial object queries in S542 into the non-cascaded decoding layer for fusion to obtain the fused results; S544: input the fused results in S543 into the FFN layer to obtain the category and bounding box of the detected marine vessel.

6. The real-time offshore ship target detection method based on improved DETR according to claim 3, wherein, The specific method of the fusion compensation module in S55 includes: S551: converting the S53 encoded features into a feature pyramid using a multi-scale adapter ; S552: set up the auxiliary detection head of K auxiliary branches and the corresponding label assignment mode ; send to the auxiliary detection head of the i-th auxiliary branch to obtain the target detection result of the marine vessel , including: bounding box coordinates, classification and confidence; using the label assignment mode in the i-th auxiliary branch Calculate the supervised target of positive and negative samples in ​ The process is described as: (4) wherein and {neg} represents the positive and negative samples determined by the positive sample refers to a region containing a target object; and the negative sample refers to a region not containing a target object; is represented as a true set; j represents a feature index in the feature set; is a set of spatial positive coordinates; and is a supervised target in the corresponding coordinates, including a class and a regression offset; S553: a set of spatial positive coordinates in the auxiliary detection head of the ith auxiliary branch obtained according to S552 generating additional custom positive queries ; all queries in the auxiliary branch are considered as positive queries; generated by the following way: (5) where Linear denotes a linear operation; M1is the number of positive samples, PE(·) represents position encoding, which selects the corresponding feature from E(·) according to the index pair (j, positive or negative coordinate in Fj); E(·) represents the output of the encoder; denotes a feature pyramid; S554: input the customized positive query generated in S553 into the non-cascaded decoding layer of the auxiliary branch to obtain a marine vessel target detection result , including: a classification score and a bounding box; the non-cascaded decoding layer of the auxiliary branch shares parameters with the non-cascaded decoding layer of S54.

7. The real-time offshore ship target detection method based on improved DETR according to claim 6, characterized in that, The training model is updated iteratively by the total target loss function, and the formula for calculating the total target loss function is as follows: The loss function of the non-cascaded decoding layer is calculated: (6) wherein represents a predicted value; represents a true value; , wherein and represent a true class and a bounding box; and represent a predicted class and a bounding box; represents an overlap degree of a true box and a predicted box; represents a bounding box loss function; represents a class loss function; The loss function of the auxiliary detection head of the i-th auxiliary branch in the fusion compensation module is calculated: (7) In the calculation process, the regression loss of the negative sample is discarded, and the training target optimized by the auxiliary detection head is as follows: (8) The loss function of the i-th auxiliary branch in the fusion compensation module is calculated as: (9) wherein, represents the output prediction of the non-cascaded decoding layer of the i-th auxiliary branch; represents the supervised target in the positive coordinate; The total target loss function is calculated as: (10) Wherein, λ1 and λ2 are coefficients for balancing the loss.

8. The real-time offshore ship target detection method based on improved DETR according to claim 1, wherein The specific implementation method of the real-time DETR inference model without the fusion compensation module is: S71: input the picture into an RGB three-dimensional matrix; adjust the RGB three-dimensional feature matrix into a fixed-size feature matrix; S72: input the fixed-size feature matrix into the backbone network based on ResNet50 for feature extraction to obtain the features at each stage; S73: input the features at the last three stages in S72 into the encoding layer based on pyramid decoupling to obtain the encoded features S74: process the encoded features in S73 through PRN to obtain two-dimensional reference points, which include classification scores and bounding boxes; sort the reference points according to the classification scores, and select the top n reference points as the output features corresponding to the input, which are also the initial object queries of the non-cascaded decoding layer; S75: input the encoded features in S73 and the initial object queries in S74 into the non-cascaded decoding layer for fusion to obtain the fused results; S76: input the fused results in S75 into the FFN layer to obtain the category and bounding box of the detected marine vessel. S77: draw the ship bounding box detected in S76 on the picture input in S71.

Images