Marine small target detection method and system based on dynamic task alignment

Abstract

The invention discloses an offshore small target detection method and system based on dynamic task alignment, and relates to the technical field of image detection, and the method comprises the steps: obtaining an initial feature map, carrying out the cutting and grouping of a convolution channel of a yolov8 detection model based on a convolution kernel, and obtaining a feature extraction unit; processing the initial feature map through a feature extraction unit to obtain a deep feature map, extracting deep features of the deep feature map by using shared convolution, and combining the deep features with the initial feature map to obtain joint features; sending the joint features into a task alignment detection head to obtain a classification prediction map and a regression prediction map; and performing prediction frame alignment operation, obtaining a falling target type of the initial feature map based on the classification prediction map, and obtaining a falling target position of the initial feature map based on the regression prediction map. According to the invention, search and rescue are carried out on the sea through the detection head, a drowning target or ship is rapidly positioned based on the acquired sea surface image, and the detection precision and the detection speed of a small target object on the sea are improved.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and more specifically to a method and system for detecting small targets at sea based on dynamic task alignment. Background Art

[0002] The harsh marine environment and complex and changeable weather conditions pose great challenges to the feature perception, imaging quality, and recognition efficiency of marine search and rescue targets: the marine environment is complex and changeable, and targets are easily affected by wind, waves, fog, backlight, etc.; the field of vision at sea is wide, and the search range is large, which requires higher accuracy and timeliness in the recognition of small targets. Traditional target detection algorithms have difficulty capturing the feature information of small targets and are more susceptible to noise, making it difficult to identify small targets.

[0003] Therefore, how to quickly capture the characteristic information of small targets at sea is an urgent problem that technicians in this field need to solve. Summary of the invention

[0004] In view of this, the present invention provides a method and system for detecting small targets at sea based on dynamic task alignment, which can quickly capture the characteristic information of small targets at sea.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A method for detecting small targets at sea based on dynamic task alignment, comprising:

[0007] Get the initial feature map, cut and group the convolution channel of the yolov8 detection model based on the convolution kernel, and get the feature extraction unit;

[0008] Processing the initial feature map by the feature extraction unit to obtain a deep feature map, extracting deep features of the deep feature map by using shared convolution, and combining the deep features with the initial feature map to obtain a joint feature;

[0009] The joint features are sent to the task alignment detection head for task decomposition to obtain classification prediction graphs and regression prediction graphs;

[0010] The classification prediction map and the regression prediction map are subjected to a prediction frame alignment operation, the falling-in-water target type of the initial feature map is obtained based on the classification prediction map, and the falling-in-water target position of the initial feature map is obtained based on the regression prediction map.

[0011] Preferably, obtaining the joint features includes extracting the feature layers P3, P4, and P5 of the yolov8 detection model, processing the initial feature map using the feature layers to obtain a deep feature map, and then obtaining the joint features through two 3x3 GroupNormalization shared convolutions. The formula is as follows:

[0012]

[0013] Among them, F is the feature layers P3, P4, and P5 extracted by feature fusion in the backbone network of the yolov8 detection model; and For two shared convolutions, a 3×3 convolution kernel is used; F union is the joint feature obtained through residual connection.

[0014] Preferably, the classification prediction map and the regression prediction map are obtained through the regression branch, the classification branch and the task decomposition module, the regression branch is a generator, and the mask and offset corresponding to the deformable convolution v2 are generated by the generator; the classification branch successively uses a convolution layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 3x3 for convolution, and after two convolutions, uses the ReLU activation function and the Sigmoid activation function to obtain the feature map D respectively; the task decomposition module decomposes the joint features into a prediction map A and a prediction map B, obtains the regression prediction map according to the mask and offset corresponding to the deformable convolution v2 and the prediction map A, and obtains the regression prediction map according to the feature map D and the prediction map B.

[0015] Preferably, the generating of the mask and offset corresponding to the deformable convolution specifically includes:

[0016] C M =G(F union );

[0017] C O =G(F union );

[0018] Among them, C M Mask, C O is Offset and G is the generator.

[0019] Preferably, the step of obtaining the characteristic graph D specifically includes:

[0020] D = σ(conv2(δ(conv1(F union ))));

[0021] Where D is the feature map D extracted by the classification branch, conv1 uses a convolution kernel of size 1×1, δ is the ReLU activation function, conv2 uses a convolution kernel of size 3×3, and σ is the Sigmoid activation function.

[0022] Preferably, the task decomposition module specifically includes: pooling the joint features into an average pooling feature F of size 1×1 by using the adaptive average pooling method in Pytorch avg, while keeping the depth of the feature map unchanged, and then through the two 1×1 convolutions conv3 and conv4 of the layer attention module, respectively connected to the ReLU and Sigmoid activation functions, to obtain the layer attention weight L; then calculate the weighted convolution kernel, and reshape the layer attention weight and the initial weight W of the convolution kernel conv Combined to get the new convolution weight W; input feature F at the same time union It is also reshaped to match the dimension of the weighted convolution kernel, and finally a batch normalization matrix multiplication operation is performed to multiply the weighted convolution kernel and the input feature. The result is processed by group normalization and activation function, and the processed prediction feature map F is finally output. p .

[0023] Preferably, the predicted feature map F after the final output processing p Specifically include:

[0024] L = σ(conv2(δ(conv1(P avg (F union )))));

[0025] W=L×W conv ;

[0026] F p =S(GN(W×F union ));

[0027] Among them, conv3 and conv4 are both convolution layers with a convolution kernel size of 1×1, δ is the ReLU activation function, σ is the Sigmoid activation function, and P avg is the adaptive average pooling layer, L represents the layer attention weight, L is the layer attention weight calculated in the previous formula, and W conv is the convolution weight, F union is the joint feature of the initial input of the module, W is the new convolution weight calculated by the previous formula, GN is the group normalization processing, S represents the SiLU activation function, and F p is a prediction feature map, and the prediction feature map F p They are respectively referred to as prediction graph A and prediction graph B.

[0028] A marine small target detection system based on dynamic task alignment, comprising:

[0029] Multi-scale grouped convolution module, obtains the initial feature map, cuts and groups the convolution channel of the yolov8 detection model based on the convolution kernel, and obtains the feature extraction unit;

[0030] A feature extraction module, which processes the initial feature map through the feature extraction unit to obtain a deep feature map, extracts deep features of the deep feature map using shared convolution, and combines the deep features with the initial feature map to obtain a joint feature;

[0031] The task alignment detection head module sends the joint features to the task alignment detection head for task decomposition to obtain classification prediction graphs and regression prediction graphs;

[0032] The target acquisition module performs prediction frame alignment operation on the classification prediction map and the regression prediction map, obtains the falling target type of the initial feature map based on the classification prediction map, and obtains the falling target position of the initial feature map based on the regression prediction map.

[0033] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for detecting small targets at sea based on dynamic task alignment. By using a detection head to search and rescue at sea, it can quickly locate fallen targets or ships based on the collected sea surface images, thereby improving the detection accuracy and speed of small target objects at sea, and greatly improving the efficiency of search and rescue missions at sea. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0035] Figure 1 A diagram of the method steps provided by the present invention;

[0036] Figure 2 The yolov8 detection framework diagram provided by the present invention;

[0037] Figure 3 A structural diagram of a feature extraction module provided by the present invention;

[0038] Figure 4 A structural diagram of the task alignment detection head provided by the present invention;

[0039] Figure 5 The task decomposition module structure diagram provided by the present invention;

[0040] Figure 6 A structural diagram of the multi-scale grouped convolution module provided by the present invention;

[0041] Figure 7 A comparison effect diagram of the model provided by the present invention;

[0042] Figure 8 A comparison diagram of the model images provided by the present invention;

[0043] Fig. 9 A comparison diagram of the receptive fields of the models provided by the present invention;

[0044] Fig.10 This is a thermal comparison diagram provided by the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The embodiment of the present invention discloses a method for detecting small targets at sea based on dynamic task alignment, such as Figure 1 As shown, including:

[0047] Get the initial feature map, cut and group the convolution channel of the yolov8 detection model based on the convolution kernel, and get the feature extraction unit;

[0048] The initial feature map is processed by the feature extraction unit to obtain a deep feature map, the deep features of the deep feature map are extracted by shared convolution, and the deep features are combined with the initial feature map to obtain joint features;

[0049] The joint features are sent to the task alignment detection head for task decomposition to obtain classification prediction graphs and regression prediction graphs;

[0050] The classification prediction map and the regression prediction map are aligned with each other, the type of the falling target in the initial feature map is obtained based on the classification prediction map, and the position of the falling target in the initial feature map is obtained based on the regression prediction map.

[0051] In a specific embodiment, the basic framework used is the yolov8 detection framework, such as Figure 2 and Figure 3 As shown, similar to other single-stage detectors, yolov8 also has a backbone network-feature pyramid-detection head structure. Considering the model size and parameter quantity, the joint features include extracting the feature layers P3, P4, and P5 of the yolov8 detection model, using the feature layers to process the initial feature map to obtain a deep feature map, and then obtaining the joint features through two 3x3 Group Normalization shared convolutions. The formula is as follows:

[0052]

[0053] Among them, F is the feature layers P3, P4, and P5 extracted by feature fusion in the backbone network of the yolov8 detection model; and For two shared convolutions, a 3×3 convolution kernel is used; F union is the joint feature obtained through residual connection. union It is the joint feature obtained through residual connection. Because it is a shared convolution, the number of parameters can be greatly reduced, which makes the model lighter, especially on resource-constrained devices. After the residual connection, the new joint feature is obtained, and the new joint feature will be sent to the task alignment detection head module for detection.

[0054] Yolov8 is used as the backbone network of the model, the training batch is 32, the training epoch is 300, and a comparison is made for different models. The results are as follows Figure 7 As shown in the figure, the dynamic task alignment detection head achieved an AP@.5 of 30.9% on the TinyPerson test set compared to other model methods, which is a 3.3% performance improvement over the original model.

[0055] like Figure 8 As shown in the figure, when small targets gather in the scene, the conventional model is difficult to distinguish, but our proposed method can effectively detect small targets. In addition, after adding multi-scale group convolution, the model effect can be slightly improved compared to using only the dynamic task alignment detection head, and the model parameters and calculation amount are also reduced to a certain extent.

[0056] In a specific embodiment, the classification prediction map and the regression prediction map are obtained through the regression branch, the classification branch and the task decomposition module. The regression branch is a generator, and the mask and offset corresponding to the deformable convolution v2 are generated by the generator (the mask and offset required by the deformable convolution v2 are obtained from the feature layer, and this feature layer is C, which is obtained by the generator); the classification branch successively uses the convolution layer with a convolution kernel size of 1x1 and the convolution layer with a convolution kernel size of 3x3 for convolution, and uses the ReLU activation function and the Sigmoid activation function to obtain the feature map D after two convolutions; the task decomposition module decomposes the joint feature into a prediction map A and a prediction map B, and obtains the regression prediction map according to the mask and offset corresponding to the deformable convolution v2 and the prediction map A, and obtains the regression prediction map according to the feature map D and the prediction map B. In the task alignment detection head, the joint feature obtains the regression prediction map A and the classification prediction map B through the task decomposition module, and two new feature maps C and D are generated in the branch line to further align and correct A and B. In addition, in order to deal with the problem of inconsistent scales of targets detected by each detection head, a scaling layer is used to scale the features, such as Figure 5 shown.

[0057] In a specific embodiment, the generator is a convolution layer composed of a 3x3 convolution kernel. Since the deformable convolution v2 module needs to be input later, and the convolution kernel size of the deformable convolution v2 is 3×3, the dimension of the mask is 9, and the offset will act in both the X and Y directions, so the dimension of the offset is 18, so the number of generator output channels is 27. These offsets and masks will dynamically adjust the regression box at each position to ensure that it is the best prediction box around. This part is to allow the model to dynamically learn the feature information that needs to be extracted from the feature tensor and suppress unnecessary feature information. The mask and offset corresponding to the deformable convolution are specifically generated as follows:

[0058] C M =G(F union );

[0059] C O =G(F union );

[0060] Among them, C M Mask, C O is Offset, G is the generator. Then it and the regression prediction map A are sent to the deformable convolution v2 to learn the new regression prediction map.

[0061] In a specific embodiment, in the classification branch, a 1x1 convolution kernel and a 3x3 convolution kernel are used for convolution successively, and ReLU and Sigmoid activation functions are used respectively after the two convolutions. The classification prediction box is dynamically adjusted according to the joint features, because it is calculated from the joint feature map, so it is possible to learn to align the prediction boxes of the two tasks in spatial positions. Obtaining the feature map D specifically includes:

[0062] D = σ(conv2(δ(conv1(F union ))));

[0063] Where D is the feature map D extracted by the classification branch, conv1 uses a convolution kernel of size 1×1, δ is the ReLU activation function, conv2 uses a convolution kernel of size 3×3, and σ is the Sigmoid activation function.

[0064] In a specific embodiment, the task decomposition module specifically includes: pooling the joint features into an average pooling feature F of size 1×1 by using the adaptive average pooling method in Pytorch avg, while keeping the depth of the feature map unchanged, and then through the two 1×1 convolutions conv3 and conv4 of the layer attention module, respectively connected to the ReLU and Sigmoid activation functions, to obtain the layer attention weight L; then calculate the weighted convolution kernel, and reshape the layer attention weight and the initial weight W of the convolution kernel conv Combined to get the new convolution weight W; input feature F at the same time union It is also reshaped to match the dimension of the weighted convolution kernel, and finally a batch normalization matrix multiplication operation is performed to multiply the weighted convolution kernel and the input feature. The result is processed by group normalization and activation function, and the processed prediction feature map F is finally output. p .

[0065] In a specific embodiment, the final output processed prediction feature map F p Specifically include:

[0066] L = σ(conv4(δ(conv3(P avg (F union )))));

[0067] W=L×W conv ;

[0068] F p =S(GN(W×F union ));

[0069] Among them, conv3 and conv4 are both convolution layers with a convolution kernel size of 1×1, δ is the ReLU activation function, σ is the Sigmoid activation function, and P avg is the adaptive average pooling layer, L represents the layer attention weight, L is the layer attention weight calculated in the previous formula, and W conv is the convolution weight, which is obtained by random initialization and is assigned a random value; F union is the joint feature of the initial input of the module, W is the new convolution weight calculated by the previous formula, and the weight of each convolution layer is adjusted according to its corresponding attention weight. The above operation realizes the attention mechanism, allowing the network to dynamically adjust the weights of different convolution layers. GN is group normalization processing, S represents SiLU activation function, and F p To predict the feature map, the predicted feature map F p They are respectively referred to as prediction graph A and prediction graph B.

[0070] In a specific embodiment, the convolution channels of the yolov8 detection model are cut and grouped based on the convolution kernel, specifically including: deep convolution extracts features for each channel, but there is a lack of interaction between different feature channels, and the model is weak in extracting features of target objects of different scales under a single-sized convolution kernel. The convolution kernel is the basic computing unit in the convolutional neural network, which is mainly used to extract and express feature maps. Its size directly affects the receptive field of the input data that it can determine, that is, the range of local information that the convolution kernel can capture. Therefore, we cut and group the channels of the model during the convolution process, and use convolution kernels of different sizes to convolve and extract feature information for different groups (such as Figure 6 As shown in the figure, the images are divided into 4 groups and convolved with 1x1, 3x3, 5x5 and 7x7 convolution kernels respectively. Then, the feature information extracted from each group is fused through a 1x1 convolution kernel. The advantages of multi-scale group convolution are fully utilized to strengthen the information exchange between different channels.

[0071] Experiments show that multi-scale grouped convolution not only improves the detection of small objects, but also has fewer model parameters than the original model. Compared with ordinary 3x3 convolution, assuming that its input channel and output channel are the same and both are C, then C Cx3x3 convolution kernels are required, and the parameter calculation is:

[0072] C×3×3×C=9C 2

[0073] During the convolution process, we divide them into four groups, each of which uses convolution kernels of 1x1, 3x3, 5x5, and 7x7 sizes for convolution, and the parameter quantity is:

[0074]

[0075] Afterwards, information is exchanged through a 1x1 convolution kernel, and its parameter quantity is:

[0076] C×1×1×C=C 2 ;

[0077] The total number of parameters is 5.25C2+C2=6.25C2, which is nearly 1 / 3 less than that of ordinary convolution.

[0078] like Figure 7 As shown in the figure, when small targets gather in the scene, the conventional model is difficult to distinguish, but the present invention can effectively detect small targets. In addition, after adding multi-scale group convolution, compared with using only the dynamic task alignment detection head, the model effect can be slightly improved and the model parameters and calculation amount can also be reduced to a certain extent.

[0079] Fig. 9 and Fig.10These are comparative experiments on the model receptive field and heat map. Compared with other model methods, the method proposed in this invention has a larger receptive field, thanks to the use of multi-scale grouped convolution to replace the original C2f convolution module, which gives the model a larger receptive field. In terms of heat map performance, our proposed method also has a good effect, which means that the model can focus more on small target objects during training.

[0080] A marine small target detection system based on dynamic task alignment, comprising:

[0081] Multi-scale grouped convolution module, obtains the initial feature map, cuts and groups the convolution channel of the yolov8 detection model based on the convolution kernel, and obtains the feature extraction unit;

[0082] The feature extraction module processes the initial feature map through the feature extraction unit to obtain a deep feature map, uses shared convolution to extract deep features of the deep feature map, and combines the deep features with the initial feature map to obtain joint features;

[0083] The task alignment detection head module sends the joint features to the task alignment detection head for task decomposition to obtain classification prediction graphs and regression prediction graphs;

[0084] The target acquisition module performs prediction frame alignment operations on the classification prediction map and the regression prediction map, obtains the falling target type of the initial feature map based on the classification prediction map, and obtains the falling target position of the initial feature map based on the regression prediction map.

[0085] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0086] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting small targets at sea based on dynamic task alignment, characterized in that: include: Get the initial feature map, cut and group the convolution channel of the yolov8 detection model based on the convolution kernel, and get the feature extraction unit; Processing the initial feature map by the feature extraction unit to obtain a deep feature map, extracting deep features of the deep feature map by using shared convolution, and combining the deep features with the initial feature map to obtain a joint feature; The joint features are sent to the task alignment detection head for task decomposition to obtain classification prediction graphs and regression prediction graphs; The classification prediction map and the regression prediction map are subjected to a prediction frame alignment operation, the falling-in-water target type of the initial feature map is obtained based on the classification prediction map, and the falling-in-water target position of the initial feature map is obtained based on the regression prediction map.

2. According to claim 1, a method for detecting small targets at sea based on dynamic task alignment is characterized in that: The joint feature is obtained by extracting the feature layers P3, P4, and P5 of the yolov8 detection model, using the feature layers to process the initial feature map to obtain a deep feature map, and then obtaining the joint feature through two 3x3 Group Normalization shared convolutions. The formula is as follows: Among them, F is the feature layers P3, P4, and P5 extracted by feature fusion in the backbone network of the yolov8 detection model; and For two shared convolutions, a 3×3 convolution kernel is used; F union is the joint feature obtained through residual connection.

3. According to the method of claim 1, a method for detecting small targets at sea based on dynamic task alignment is characterized in that: The classification prediction map and the regression prediction map are obtained through the regression branch, the classification branch and the task decomposition module. The regression branch is a generator, and the mask and offset corresponding to the deformable convolution v2 are generated by the generator; the classification branch successively uses a convolution layer with a convolution kernel size of 1x1 and a convolution layer with a convolution kernel size of 3x3 for convolution, and after two convolutions, the ReLU activation function and the Sigmoid activation function are used respectively to obtain the feature map D; the task decomposition module decomposes the joint features into a prediction map A and a prediction map B, obtains the regression prediction map according to the mask and offset corresponding to the deformable convolution v2 and the prediction map A, and obtains the regression prediction map according to the feature map D and the prediction map B.

4. A method for detecting small targets at sea based on dynamic task alignment according to claim 3, characterized in that: The mask and offset corresponding to the generated deformable convolution specifically include: C M =G(F union ); C O =G(F union ); Among them, C M Mask, C O is Offset and G is the generator.

5. The method for detecting small targets at sea based on dynamic task alignment according to claim 3 is characterized in that: The obtaining of the feature map D specifically includes: D=σ(conv2(δ(conv1(F union )))); Where D is the feature map D extracted by the classification branch, conv1 uses a convolution kernel of size 1×1, δ is the ReLU activation function, conv2 uses a convolution kernel of size 3×3, and σ is the Sigmoid activation function.

6. The method for detecting small targets at sea based on dynamic task alignment according to claim 3 is characterized in that: The task decomposition module specifically includes: using the adaptive average pooling method in Pytorch to pool the joint features into an average pooling feature F of size 1×1 avg , while keeping the depth of the feature map unchanged, and then through the two 1×1 convolutions conv3 and conv4 of the layer attention module, respectively connected to the ReLU and Sigmoid activation functions, to obtain the layer attention weight L; then calculate the weighted convolution kernel, and reshape the layer attention weight and the initial weight W of the convolution kernel conv Combined to get the new convolution weight W; input feature F at the same time union It is also reshaped to match the dimension of the weighted convolution kernel, and finally a batch normalization matrix multiplication operation is performed to multiply the weighted convolution kernel and the input feature. The result is processed by group normalization and activation function, and the processed prediction feature map F is finally output. p .

7. A method for detecting small targets at sea based on dynamic task alignment according to claim 6, characterized in that: The final output processed predicted feature map F p Specifically include: L=σ(conv2(δ(conv1(P avg (F union ))))); W=L×W conv ; F p =S(GN(W×F union )); Among them, conv3 and conv4 are both convolution layers with a convolution kernel size of 1×1, δ is the ReLU activation function, σ is the Sigmoid activation function, and P avg is the adaptive average pooling layer, L represents the layer attention weight, L is the layer attention weight calculated in the previous formula, and W conv is the convolution weight, F union is the joint feature of the initial input of the module, W is the new convolution weight calculated by the previous formula, GN is the group normalization processing, S represents the SiLU activation function, and F p is a prediction feature map, and the prediction feature map F p They are respectively referred to as prediction graph A and prediction graph B.

8. A marine small target detection system based on dynamic task alignment, applying a marine small target detection method based on dynamic task alignment as described in any one of claims 1 to 7, characterized in that: include: Multi-scale grouped convolution module, obtains the initial feature map, cuts and groups the convolution channel of the yolov8 detection model based on the convolution kernel, and obtains the feature extraction unit; A feature extraction module, which processes the initial feature map through the feature extraction unit to obtain a deep feature map, extracts deep features of the deep feature map using shared convolution, and combines the deep features with the initial feature map to obtain a joint feature; The task alignment detection head module sends the joint features to the task alignment detection head for task decomposition to obtain classification prediction graphs and regression prediction graphs; The target acquisition module performs prediction frame alignment operation on the classification prediction map and the regression prediction map, obtains the falling target type of the initial feature map based on the classification prediction map, and obtains the falling target position of the initial feature map based on the regression prediction map.

Images