A marine ship target detection method based on attention mechanism under occlusion conditions

Abstract

The present invention discloses an unmanned boat maritime ship target detection method based on an attention mechanism under occlusion conditions, comprising the following steps: obtaining a data set of unmanned boat maritime ship targets under occlusion conditions to be processed; preprocessing the images in the data set; dividing the preprocessed data set into a training set and a test set according to a ratio; constructing an AodeMar model for feature extraction, feature fusion and prediction of the location and category of the target; training the AodeMar model based on the training set to obtain a trained AodeMar model; inputting the test set into the trained AodeMar model to achieve detection of unmanned boat maritime ship targets under occlusion conditions. The present invention adopts residual connection to fuse shallow and high-level features to enhance the representation ability of local features, suppress features that are invalid for detection and enhance features that are effective for detection. The problem of accurately locating the edge of the target in the occluded area is effectively solved, and the expression ability and detection accuracy of the model are improved.

Description

Technical Field

[0001] The present invention belongs to the field of fully automated products and relates to a method for detecting marine ship targets under occlusion conditions based on an attention mechanism. Background Art

[0002] my country has vast sea areas and rich marine resources, and the development and utilization of the ocean has always been a key issue of concern. Unmanned boats have played a very important role in marine survey and drilling, maritime rescue, underwater salvage, marine environmental monitoring, maritime traffic control, offshore geophysical exploration and other fields. In the application of unmanned boats, due to the special horizontal viewing angle of the ship-borne camera, the mutual occlusion of ships in the collected images is relatively serious, especially when there are many ships around the unmanned boat, which brings great challenges to the target detection of unmanned boats based on image information. Therefore, it is particularly important to study the ship target detection algorithm under mutual occlusion conditions for unmanned boats.

[0003] Although deep learning has promoted the development of object detection, in real-world scenarios, object detection is still a challenging task due to the existence of occlusion. The human visual system can infer the attributes of objects through the contours in the scene even when part of the object information is occluded or lost. However, it is still very difficult for computer vision systems based on deep learning to effectively detect occluded objects.

[0004] At present, the object detection algorithms for occlusion are mainly divided into three categories: data enhancement, improved detectors, and some semantic-based algorithms. The detection method based on data enhancement refers to adding noise to the original image to simulate the occlusion situation, so that the detector can learn various occlusion situations, thereby improving the robustness of the network in the occlusion scene. The method based on improved detectors focuses on improving the detection performance of the network under occlusion, such as introducing multi-scale feature fusion, object relationship modeling, or designing adaptive anchors to solve the occlusion problem. Some semantic-based algorithms use some semantic information related to occlusion for object detection, such as detecting different parts of the target or identifying targets with different postures. The above methods can solve the occlusion problem to a certain extent, but they are inconsistent with the actual occlusion environment. Therefore, most of the research focuses on optimizing the network structure to identify and locate occluded objects.

[0005] At present, a series of attention modules have been proposed one after another, which can be easily embedded into deep learning models as plug-and-play modules. The model of attention mechanism in computer vision can be divided into three attention domains, namely spatial domain, channel domain and hybrid domain. In the spatial domain, the spatial information in the image is converted into the corresponding spatial domain to extract key information. In the channel domain, it is similar to adding weights to the signal of each channel to represent the correlation between the channel and the key information. The larger the weight, the higher the correlation. In the hybrid domain, the focus of the spatial domain is to ignore the information in the channel domain and treat the image features in each channel equally so that the channel domain and the spatial domain can be used in combination. One of the most classic attention modules is the Squeeze-and-Excitation Network (SE) [70, Squeeze-and-Excitation Networks] channel attention. It uses a simple compression method to process each two-dimensional feature map to effectively construct the dependency between channels. However, SE attention ignores the position information that is crucial for visual tasks. Bottleneck Attention Module (BAM) and Convolutional Block Attention Module (CBAM)

[71] attempt to insert position information by globally pooling channels, but this approach can only capture local information and cannot capture long-range dependencies. CBAM uses the maximum and average values ​​of multiple channels at each position as weighting coefficients, but it only considers information within a local range. Coordinate Attention (CA)

[30] is a lightweight and efficient attention mechanism that can encode long-range dependencies and position information in horizontal and vertical spatial directions into channel attention. The self-attention mechanism is also an attention mechanism that was originally used in natural language processing (NLP). Its core is to capture the association between different positions in a single sequence. The representation of the sequence is calculated by learning the association between each word and all other words. It has been successfully applied to various tasks in the field of NLP. Thanks to the success of NLP, many works have attempted to combine CNN with self-attention, such as Vision Transformer (ViT) and Swin Transformer (STR) [41, Swin Transformer: Hierarchical Vision Transformer using Shifted Windows], to obtain the features of targets that appear in the same scene at the same time, and then complete the interaction of cross-scale features.

[0006] Currently, in the detection of marine ship targets in actual occlusion environments:

[0007] 1) Compared with pedestrian and vehicle detection, the correlation between the local features of a ship and the overall target is weaker, that is, it is difficult to accurately predict the category of a ship based on a certain part of the ship. On the one hand, there are strong similarities between different types of ships, such as general cargo ships, bulk carriers, tankers and container ships, fireboats and tugboats. On the other hand, in the near-shore environment, the characteristics of the superstructure of the ship are similar to those of the shore buildings, resulting in a high false detection rate of ship target detection under occlusion conditions.

[0008] 2) Occlusion can affect the precise positioning of targets. For example, two mutually occluded targets with similar features are easily identified as one target and predicted into one detection frame, which affects the accuracy of positioning. Summary of the invention

[0009] In order to solve the above problems, the technical solution adopted by the present invention is: a method for detecting unmanned boat marine ship targets under occlusion conditions based on an attention mechanism, comprising the following steps:

[0010] Obtain a data set of unmanned boat and marine ship targets under occlusion conditions to be processed;

[0011] Preprocessing the images in the data set;

[0012] Divide the preprocessed data set into training set and test set according to the proportion;

[0013] Construct an AodeMar model for feature extraction, feature fusion, and prediction of the location and category of the target;

[0014] The AodeMar model is trained based on the training set to obtain a trained AodeMar model;

[0015] The test set is input into the trained AodeMar model to realize the detection of unmanned boat and maritime ship targets under occlusion conditions.

[0016] Further: the AodeMar model includes a backbone network for extracting features from images, a neck network for fusing the extracted features, and a detection head network for predicting the location and category of the target;

[0017] The backbone network, the neck network, and the detection head network are connected in cascade in sequence.

[0018] Further: the backbone network includes 9 CBS modules, namely a first CBS module, a second CBS module, a third CBS module, a fourth CBS module, a fifth CBS module, a sixth CBS module, a seventh CBS module, an eighth CBS module and a ninth CBS module;

[0019] The five C3 modules are the first C3 module, the second C3 module, the third C3 module, the fourth C3 module and the fifth C3 module; the first C3 module, the second C3 module, the third C3 module, the fourth C3 module and the fifth C3 module are all conducive to extracting more and more detailed feature information

[0020] The RCAC3 module and the SPPF module focus on processing edge information of overlapping occluded areas by applying the CA method; the SPPF module mainly fuses multi-scale feature information; the RCAC3 module is to further enhance feature extraction capabilities;

[0021] The first CBS module, the second CBS module, the first C3 module, the third CBS module, the second C3 module, the fourth CBS module, the third C3 module, the fifth CBS module, the RCAC3 module, the SPPF module, the sixth CBS module, the fourth C3 module, the seventh CBS module, the fifth C3 module, the eighth CBS module and the eighth CBS module are connected in sequence.

[0022] Further: the CBS module is composed of a convolutional layer, a batch normalization layer and an activation function (SiLU). The main function of the CBS module is to upsample, reduce or increase the dimension and normalize the feature map;

[0023] There are three maximum pooling layers in SPPF, and each pooling layer operates differently on the input feature map; residual connections are added to SPPF.

[0024] The RCAC3 module is formed by embedding the RCAB module into the C3 module, which consists of three convolutions and a CSP Botleneck.

[0025] The RCAB module consists of two consecutive 2-D convolutional layers, batch normalization and SiLU activation functions, a CA module, and a residual connection.

[0026] Further: The process of the backbone network extracting features from the image is as follows:

[0027] Assume that the output after passing through two CBS modules is First, in order to embed the coordinate information, two independent attention feature maps are generated in the horizontal and vertical directions respectively. and They are represented as:

[0028]

[0029] in: and are the outputs of the cth channel in height and width, respectively. h and ω represent the height and width of the kth (k=1, 2, ···, c) channel, respectively. m and n represent the dimensions of the pooling kernel, and (,) represents the position coordinates.

[0030] Equations (1) and (2) represent average pooling in the X and Y directions, respectively;

[0031] The second step is the CA generation transformation process. First, the above two feature maps are cascaded and then sent to a 1×1 convolution F and a nonlinear activation function δ; at the same time, the spatial information can be encoded into the feature map G, that is:

[0032]

[0033] in: l = 1, 2, ···, c / r and r represents the reduction factor. Then the feature map G l Decomposed along the position direction into and Then, two 1×1 convolution functions L are used h and L ω Respectively and Transformed into Tensors with the same channels. Output with exact coordinate information can be output as:

[0034]

[0035]

[0036] Among them, γ represents the sigmoid function, As the attention weight input, the output of the CA block can be expressed as:

[0037]

[0038] In the RCAB module, two CBS modules are used consecutively before CA, and a residual connection is added; in this case, RCAB It can be calculated as:

[0039]

[0040] In the above case, the output of the RCAB module can be expressed as:

[0041]

[0042]

[0043] in, and Represent the CBS operations with 1×1 and 3×3 convolution kernels respectively.

[0044] and They are the input and output feature maps of RCAB, Is the shape and Same coefficient matrix. Element-wise addition.

[0045] The RCAB module enhances the feature extraction capability and improves the representation of the region of interest;

[0046] The RCAB module is embedded into the C3 module to form the RCAC3 module; the output of the fifth CBS module will pass through two paths at the same time, one of which passes through the CBS module and the RCAB module, and the other only passes through the CBS module; the fused output will be operated by the CBS module;

[0047] Similar to RCAB, the position enhancement module RCAC3 input can be expressed as:

[0048]

[0049] in, and Represent the input and output feature maps of RCAC3 respectively. Join operation in channel dimension.

[0050] Further: the neck network includes a SP-STR module;

[0051] The SP-STR module is formed by inserting the SPP module as an additional module into the STR self-attention block;

[0052] The input feature map of SP-STR is Here and Represent the number of channels and dimensions in the width and height directions respectively. First, a 1×1 convolution operation CBS block is used to encode each channel of the input feature map to reduce the number of channels, which is defined as:

[0053]

[0054] in Represents a CBS operation with a convolution kernel size of 1×1;

[0055] In the SPP module, the output of the previous concatenation layer is first subjected to a maximum pooling operation with kernel sizes of 1×1, 5×5, 9×9, and 13×13 to fuse global and local feature information, which is defined as:

[0056]

[0057]

[0058] in represents the maximum pooling operation with different pooling kernel sizes (i.e. ). Extract the corresponding feature map It represents the concatenated feature map.

[0059] Finally, the output channels of SPP are restored to the same level as the input channels through a CBS block with a 1×1 convolution operation. The same size can be expressed as:

[0060]

[0061] in,

[0062] The STR module contains two consecutive STR self-attention modules, where each block contains two sub-layers, namely, window-based multi-head self-attention (W-MSA) and multi-layer perceptron (MLP) layers. Specifically, layer normalization (LayerNorm) is used to normalize the different features of a single sample in the channel dimension;

[0063] Therefore, W-MSA is used to build information correlation in the window. In order to alleviate the vanishing gradient problem, the residual connection combines the forward complete information into the layer for local global semantic feature learning. Each layer of the W-MSA layer and the MLP layer of the STR self-attention module is normalized before the layer. The sub-layer output in the l-1-th block is expressed as:

[0064]

[0065] in, and Indicates W-MSA and LayerNorm operations. Please note

[0066] After that, the MLP layer is deployed to learn the feature weights of global perception, which are expressed as:

[0067]

[0068] in, Represents an MLP operation.

[0069] In order to further reduce the calculation of MSA and establish information connection across windows, similarly, the shifted W-MSA (SW-MSA) and the corresponding MLP module are calculated, and the expression is:

[0070]

[0071]

[0072] in, and Indicates SW-MSA operation;

[0073] In this context, multi-scale feature semantic-related information is embedded to enhance the feature fusion capability and detection performance of objects of different sizes. In the proposed SP-STR module, two consecutive STR self-attention blocks are used, so l = 2.

[0074] The present invention provides an unmanned boat marine ship target detection method based on an attention mechanism under occlusion conditions. Considering that in the marine ship target detection under actual occlusion environments, compared with the detection of pedestrians and vehicles, the correlation between the local features of the ship and the overall target is weaker, that is, it is difficult to accurately predict the category to which it belongs based on only a certain part of the ship. Based on this, the present invention proposes a multi-scale feature semantic association module (SP-STR module) to help learn the multi-scale feature semantic correlation. In addition, because there is a strong similarity between the maximum features of certain types of ships, such as tankers and container ships. In a near-shore environment, the features of the superstructures of most ships are similar to those of the shore buildings. Two targets with similar features will be identified as one target, resulting in a high false detection rate of ship target detection under occlusion conditions. Based on this, the present invention proposes a position enhancement module (RCAC3 module) to enhance the features that are effective for detection and suppress invalid features, establish a long-range dependency relationship between features, and accurately locate the edge of the target in the occluded area.

[0075] A position enhancement module called RCAC3 is created using residual connections and CA mechanisms, which can provide 3D feature information in terms of position, channel, and space. RCAC3 makes full use of shallow information and high-level semantics through residual connections and establishes long-range dependencies of features through CA, thus providing accurate bounding boxes for occluded ships.

[0076] By combining SPP with the STR self-attention encoder block, a multi-scale feature semantic association module, named SP-STR, is designed to enhance the classification ability of occluded ships in both global and local senses.

[0077] An attention-aware occlusion detection scheme for marine vessels, named AodeMar, is developed by replacing the convolutional layers of the backbone network and the neck network with RCAC3 and SP-STR modules, respectively, which can fully enhance the accuracy and robustness of occlusion detection.

[0078] The present invention proposes a multi-scale feature semantic association module (SP-STR module), which effectively solves the problem of poor detection performance caused by weak correlation between local features and overall features of a ship in an occluded scene, and improves the detection accuracy.

[0079] The present invention proposes a position enhancement module (RCAC3 module), which uses the position attention mechanism to establish long-range dependencies between features, and uses residual connections to fuse shallow and high-level features to enhance the representation ability of local features, suppressing features that are invalid for detection while enhancing features that are effective for detection. It effectively solves the problem of accurately locating the edge of the target in the occluded area and improves the expression ability and detection accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0081] Figure 1 This is the overall structure diagram of the AodeMar model;

[0082] Figure 2 It is the SPPF structure diagram;

[0083] Figure 3 It is the connection structure diagram of CBS, C3, RCAC3, SPPF modules;

[0084] Figure 4 This is the SP-STP module structure diagram;

[0085] Figure 5 This is the structure diagram of the CA attention mechanism;

[0086] Figure 6 This is the RCAB module structure diagram;

[0087] Figure 7 This is the RCAC3 module structure diagram;

[0088] Figure 8 It is the PR curve of the detection results of each method for four types of ships;

[0089] Fig. 9It is the PR curve of the detection results of each method for nine types of ships;

[0090] Fig.10 These are the detection results under four natural occlusion scenarios;

[0091] Fig.11 These are the occlusion detection result images under four harsh environments. DETAILED DESCRIPTION

[0092] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0093] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0094] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0095] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, the numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be clear that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The technology, methods and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be regarded as part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0096] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention: the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0097] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below their position devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0098] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0099] The present invention relates to a method for detecting unmanned boat marine vessels under occlusion conditions based on an attention mechanism; the method comprises the following steps:

[0100] S1: Obtain a data set of unmanned boat and marine ship targets under occlusion conditions to be processed;

[0101] S2: Preprocess the images in the data set; the preprocessing is performed by cutting them into a required size; the size requirement may be: 640×640×3;

[0102] S3: Divide the preprocessed data set into a training set and a test set according to the ratio;

[0103] S4: Construct an AodeMar model for feature extraction, feature fusion, and prediction of the location and category of the target;

[0104] S5: Train the AodeMar model based on the training set to obtain a trained AodeMar model;

[0105] S6: Input the test set into the trained AodeMar model to detect unmanned boat and maritime ship targets under occlusion conditions.

[0106] Steps S1 / S2 / S3 / S4 / S5 / S6 are executed in sequence, or S4 / S1 / S2 / S3 / S5 / S6 are executed in sequence;

[0107] like Figure 1 As shown in the figure, the overall network architecture is designed and implemented by the above two modules: position enhancement module and multi-scale feature semantic association module.

[0108] The AodeMar model includes a backbone network for extracting features from images, a neck network for fusing the extracted features, and a detection head network for predicting the location and category of the target (Cargo, Container, Bulker, Tanker, Sailingboat, Tug, Fishing, Drill, Firefighting, Containership, Warship, Submarine, thirteen categories);

[0109] The backbone network, the neck network, and the detection head network are connected in cascade in sequence.

[0110] The backbone network includes 9 CBS modules, namely a first CBS module, a second CBS module, a third CBS module, a fourth CBS module, a fifth CBS module, a sixth CBS module, a seventh CBS module, an eighth CBS module and a ninth CBS module;

[0111] The 5 C3 modules are the first C3 module, the second C3 module, the third C3 module, the fourth C3 module and the fifth C3 module; the first C3 module, the second C3 module, the third C3 module, the fourth C3 module and the fifth C3 module are all conducive to extracting more and more detailed feature information. At the same time, a residual structure is introduced in the C3 module to avoid the gradient disappearance problem.

[0112] The RCAC3 module and SPPF module focus on processing edge information of overlapping occluded areas by applying the CA method; the SPPF module (fast spatial pyramid pooling) is mainly used to fuse multi-scale feature information; the RCAC3 module is to further enhance the feature extraction capability;

[0113] The first CBS module, the second CBS module, the first C3 module, the third CBS module, the second C3 module, the fourth CBS module, the third C3 module, the fifth CBS module, the RCAC3 module, the SPPF module, the sixth CBS module, the fourth C3 module, the seventh CBS module, the fifth C3 module, the eighth CBS module and the eighth CBS module are connected in sequence.

[0114] The CBS module consists of a convolutional layer (Covn), a batch normalization layer (BN) and an activation function (SiLU). Its main function is to upsample, reduce or increase the dimension and normalize the feature map.

[0115] The CBS module consists of a convolutional layer, a batch normalization layer, and an activation function (SiLU). The main function of the CBS module is to upsample, reduce or increase the dimension, and normalize the feature map.

[0116] There are three maximum pooling layers in SPPF, and each pooling layer operates differently on the input feature map; residual connections are added to SPPF. Figure 2 It is the SPPF structure diagram;

[0117] The RCAC3 module is formed by embedding the RCAB module into the C3 module, which consists of three convolutions and a CSP Botleneck.

[0118] The RCAB module consists of two consecutive 2-D convolutional layers, batch normalization and SiLU activation functions, a CA module, and a residual connection.

[0119] The C3 module consists of three convolutions and a CSP Bottleneck, where the RCAC3 block focuses on processing edge information in overlapping occluded areas by applying the CA method;

[0120] The RCAB module consists of two consecutive 2-D convolutional layers, batch normalization and SiLU activation functions, a CA module, and a residual connection.

[0121] Figure 3 It is the connection structure diagram of CBS, C3, RCAC3 and SPPF modules;

[0122] The process of the backbone network extracting features from an image is as follows:

[0123] Assume that the output after passing through two CBS modules is First, in order to embed the coordinate information, two independent attention feature maps are generated in the horizontal and vertical directions respectively. and They are represented as:

[0124]

[0125] in: and are the outputs of the cth channel in height and width, respectively. h and ω represent the height and width of the kth (k=1, 2, ···, c) channel, respectively. m and n represent the dimensions of the pooling kernel, and (,) represents the position coordinates.

[0126] Equations (1) and (2) represent average pooling in the X and Y directions, respectively;

[0127] The second step is the CA generation transformation process. First, the above two feature maps are cascaded and then sent to a 1×1 convolution F and a nonlinear activation function δ; at the same time, the spatial information can be encoded into the feature map G, that is:

[0128]

[0129] in: l = 1, 2, ···, c / r and r represents the reduction factor. Then the feature map G l Decomposed along the position direction into and Then, two 1×1 convolution functions L are used h and L ω Respectively and Transformed into Tensors with the same channels. Output with exact coordinate information can be output as:

[0130]

[0131]

[0132] Among them, γ represents the sigmoid function, As the attention weight input, the output of the CA block can be expressed as:

[0133]

[0134] In the RCAB module, two CBS modules are used consecutively before CA and a residual connection is added; in this case, RCAB It can be calculated as:

[0135]

[0136] In the above case, the output of the RCAB module can be expressed as:

[0137]

[0138]

[0139] in, and Represent the CBS operations with 1×1 and 3×3 convolution kernels respectively.

[0140] and They are the input and output feature maps of RCAB, Is the shape and Same coefficient matrix. Element-wise addition.

[0141] The RCAB module enhances the feature extraction capability and improves the representation of the region of interest;

[0142] The RCAB module is embedded into the C3 module to form the RCAC3 module; the output of the fifth CBS module will pass through two paths at the same time, one of which passes through the CBS module and the RCAB module, and the other only passes through the CBS module; the fused output will be operated by the CBS module;

[0143] Similar to RCAB, the position enhancement module RCAC3 input can be expressed as:

[0144]

[0145] in, and Represent the input and output feature maps of RCAC3 respectively. Join operation in channel dimension.

[0146] The neck network includes an SP-STR module; Figure 4 This is the SP-STP module structure diagram

[0147] The SP-STR module is formed by inserting the SPP module as an additional module into the STR self-attention block;

[0148] The input feature map of SP-STR is Here and Represent the number of channels and dimensions in the width and height directions respectively. First, a 1×1 convolution operation CBS block is used to encode each channel of the input feature map to reduce the number of channels, which is defined as:

[0149]

[0150] in Represents a CBS operation with a convolution kernel size of 1×1;

[0151] In the SPP module, the output of the previous concatenation layer is first subjected to a maximum pooling operation with kernel sizes of 1×1, 5×5, 9×9, and 13×13 to fuse global and local feature information, which is defined as:

[0152]

[0153] in Indicator There are max pooling operations with different pooling kernel sizes (i.e. ). Extract the corresponding feature map Notice It represents the concatenated feature map.

[0154] Finally, the output channels of SPP are restored to the same level as the input channels through a CBS block with a 1×1 convolution operation. The same size can be expressed as:

[0155]

[0156] in,

[0157] The STR module contains two consecutive STR self-attention modules, where each block contains two sub-layers, namely, window-based multi-head self-attention (W-MSA) and multi-layer perceptron (MLP) layers. Specifically, layer normalization (LayerNorm) is used to normalize the different features of a single sample in the channel dimension;

[0158] Therefore, W-MSA is used to build information correlation in the window. In order to alleviate the vanishing gradient problem, the residual connection combines the forward complete information into the layer for local global semantic feature learning. Each layer of the W-MSA layer and the MLP layer of the STR self-attention module is normalized before the layer. The sub-layer output in the l-1-th block is expressed as:

[0159]

[0160] in, and Indicates W-MSA and LayerNorm operations. Please note

[0161] After that, the MLP layer is deployed to learn the feature weights of global perception, which are expressed as:

[0162]

[0163] in, Represents an MLP operation.

[0164] In order to further reduce the calculation of MSA and establish information connection across windows, similarly, the shifted W-MSA (SW-MSA) and the corresponding MLP module are calculated, and the expression is:

[0165]

[0166]

[0167] in, and Indicates SW-MSA operation;

[0168] In this context, multi-scale feature semantic-related information is embedded to enhance the feature fusion capability and detection performance of objects of different sizes. In the proposed SP-STR module, two consecutive STR self-attention blocks are used, so l = 2.

[0169] In the SPP module, the output of the previous concatenation layer is first subjected to a maximum pooling operation with kernel sizes of 1×1, 5×5, 9×9, and 13×13.

[0170] Specifically, layer normalization (LayerNorm) is used to normalize the different features of a single sample in the channel dimension. Therefore, W-MSA is used to build information correlation in the window. In order to alleviate the vanishing gradient problem, the residual connection combines the forward full information to the layer used for local global semantic feature learning. Each layer of the STR self-attention module (W-MSA layer, MLP layer) is preceded by layer normalization.

[0171] In the Neck network part, a top-down and bottom-up feature fusion module is adopted, in which the convolution modules of the last two fusion layers are replaced by a newly designed multi-scale feature semantic association module (SP-STR). In SP-STR, SPP provides multi-scale feature semantic information, and the STR self-attention encoder block is used to learn the correlation between each feature.

[0172] Considering that in the detection of marine ship targets under actual occlusion environment, the correlation between the local features of the ship and the overall target is weaker than that of pedestrian detection and vehicle detection, that is, it is difficult to accurately predict the category based on only a certain part of the ship. Based on this, the present invention proposes a multi-scale feature semantic association module (SP-STR module) to help learn the multi-scale feature semantic correlation. In addition, because there is a strong similarity between the maximum features of certain types of ships, such as tankers and container ships. In a near-shore environment, the features of the superstructures of most ships are similar to those of the shore buildings. Two targets with similar features will be identified as one target, resulting in a high false detection rate of ship target detection under occlusion conditions. Based on this, the present invention proposes a position enhancement module (RCAC3 module) to enhance the features that are effective for detection and suppress invalid features, establish a long-range dependency relationship between features, and accurately locate the edge of the target in the occluded area.

[0173] Three detection heads are used to output feature maps of three different scales, with sizes of 20×20×54, 40×40×54, and 80×80×54 respectively.

[0174] In order to accurately locate the edge of the target in the occluded area, the features that are invalid for detection are suppressed, the features that are valid for detection are enhanced, and the shallow and high-level features are fused. Based on this, the present invention designs the RCAC3 module based on the CA attention mechanism. The RCAC3 module is composed of the RCAB module (such as Figure 3 ) is embedded into the C3 module, wherein the C3 module (as shown Figure 1 As shown in Figure 2, it consists of three convolutions and a CSP Bottleneck. The RCAB module is a CA module (as shown in Figure 2). Figure 5 ), which consists of two consecutive 2-D convolutional layers, batch normalization and SiLU activation functions, CA modules and residual connections (as shown in Figure 6 As shown in the figure, the 2-D convolutional layer extracts local features of the image, while CA can enhance effective features and suppress invalid features, and establish long-range dependencies between features to accurately locate the edge of the target in the occluded area. In order to enhance the representation ability of local features, residual connections are used to fuse shallow and high-level features. Figure 6 This is the RCAB module structure diagram; Figure 7 This is the RCAC3 module structure diagram;

[0175] In order to obtain multi-scale feature interactions and enhance the relevance of local semantics, SPP is inserted into the STR self-attention block as an additional module to form the SP-STR block. It should be noted that the hierarchical architecture in SP-STR has the flexibility to model at various scales and has linear computational complexity with respect to the image size. Therefore, SP-STR is designed to replace the last two convolution operations in the Neck network to learn the semantic relevance between each feature block and assign different attention to each feature block. Since the feature map at the end of the network has a lower resolution, applying Transformer on the low-resolution feature map can reduce computational and memory costs.

[0176] like Figure 4 As shown in the figure, in the SPP module, the output of the previous concatenation layer is first subjected to the maximum pooling operation with kernel sizes of 1×1, 5×5, 9×9 and 13×13 to enhance the local relationship of the features. After fusion, they are used as the input of the STR block. STR contains two consecutive STR self-intentional modules, each of which consists of a layer of window-based multi-head self-attention (W-MSA), translation W-MSA (SW-MSA) and multi-layer perceptron (MLP) modules. In STR, each layer of these modules is preceded by layer normalization (Layer Norm).

[0177] Experimental content and evaluation indicators

[0178] In order to verify the effectiveness and superiority of the proposed AodeMar model, this paper compares it with several SOTA methods, such as SSD, YOLOv3, YOLOv4, and YOLOv5s on the MVDD13 dataset.

[0179] The experimental data uses the MVDD13 dataset, which contains a variety of occlusion situations. In all experiments, there are 25,541 training set images, 2,838 validation set images, and 7,095 test set images.

[0180] The present invention uses Average Precision (AP), Category Average Precision (mAP) and Frames Per Second (FPS) as quantitative evaluation indicators.

[0181] Experimental results analysis

[0182] A. Ablation Experiment

[0183] In order to illustrate the effectiveness and superiority of the model proposed in this invention, 10 groups of comparative experiments were carried out. The detailed experimental results are shown in Table 1.

[0184] Table 1

[0185]

[0186]

[0187] Note: The bold numbers in each row represent the optimal values ​​of mAP@.5, mAP@[0.5:0.95] and AP of each category, and the underlined numbers represent the suboptimal values.

[0188] The experiments are evaluated using quantitative indicators such as AP, mAP@.5, and mAP@[0.5:0.95]. In each case, different components are combined in sequence to train the detection scheme, and the best detection accuracy is selected from 300 iterations as the evaluation output. Therefore, the impact of each component on the detection scheme is listed in Table 2.

[0189] Compared with the baseline model, the use of the RCAC3 block does not increase the total number of layers of the model, but improves the mAP@.5 by 1.15% and the mAP@[0.5:0.95] by 3.18%. Compared with the residual attention blocks based on SE and CBAM attention mechanisms (named RSEC3 and RCBAMC3), RCAC3 shows significant advantages in mAP@.5 and mAP@[.5:.95].

[0190] In addition, combined with the SP-STR module, RCAC3 has further improvements over RSEC3 while RCBAMC3 has decreased. The reason is that the use of CA helps to locate densely distributed objects. Using the STR encoder block increases mAP@.5 and mAP@[0.5:0.95] by 0.76% and 1.79% respectively compared to the baseline model, because the two consecutive sub-layers and residual connections enhance the ability to capture local information at different layers. It should be noted that after fusing SPP, SP-STR increases slightly in mAP@.5.

[0191] Combined with the RCAC3 module, SP-STR shows a clear advantage in mAP@.5 and mAP@[.5:.95], as well as in most of the APs in each category, especially for categories such as fishing boats, warships, and sailboats. This shows that SPP and STR can promote the improvement of detection performance. The RCAC3 module is added to the backbone network at the same time, and two SP-STR blocks are used in the last two fusion layers of the neck network to construct the AodeMar model.

[0192] Through experiments, we can see that although the algorithm proposed in this paper does not obtain the best AP value in all categories, AodeMar performs better than other models in mAP@.5 and mAP@[0.5:0.95]. In this case, combined with RCAC3 and SP-STR modules, the AodeMar model achieves superior detection performance.

[0193] B. Comparative Experiments with Classical Models

[0194] In order to fully reflect the superiority of the proposed AodeMar model, the present invention considers one-stage detection methods with good real-time performance suitable for USVs, including Faster RCNN, SSD, YOLOv3, YOLOv4 and YOLOv5s. It should be noted that the above schemes have been pre-trained on the PASCAL VOC dataset and then fine-tuned on the MVDD13 dataset. Therefore, the comparison results are shown in Table 2. It can be seen that the proposed AodeMar model can achieve better AP performance in categories such as bulk carriers, tankers, fishing boats, drilling ships, container ships, warships, submarines, etc., while achieving suboptimal performance in the two categories of sailboats and fireboats.

[0195] Table 2

[0196]

[0197] Note: The bold numbers in each row indicate mAP@.5, mAP@[0.5:0.95 ], the optimal values ​​of FPS and AP for each category.

[0198] Without the RCAC3 and SP-STR modules, the YOLOv5s scheme cannot achieve the same performance as the proposed AodeMar model except for FPS, where mAP@.5 and mAP@[.5:.95] are 1.35% and 3.32% lower than the proposed AodeMar model, respectively. In terms of detection speed, the FPS of the proposed AodeMar scheme is much better than other models and meets real-time performance, although it is slower than YOLOv5s.

[0199] Figure 8 It is the PR curve of the detection results of each method for four types of ships;

[0200] Fig. 9 It is the PR curve of the detection results of each method for nine types of ships;

[0201] By plotting the precision-recall (PR) curve at IoU = 0.5, the test performance evaluation of different detectors can be vividly described, such as Figure 8 and Fig. 9As shown. When using RCAC3 and SP-STR modules, the AodeMar model proposed in the present invention shows a better balance between precision and recall. The detection performance shown in categories such as submarines, tankers, warships, fishing boats, drilling ships and container ships is significantly better than that of the other four methods. For types such as general cargo ships, cruise ships, tugboats and fireboats, although the detection performance cannot achieve the best results, a higher recall rate can be achieved when the precision is greater than 0.5. Therefore, the proposed AodeMar model has better detection performance.

[0202] C. Verification experiment under natural occlusion scene

[0203] Fig.10 These are the detection results under four natural occlusion scenes; AodeMar model Fig.10 The proposed AodeMar model performs well in the four real-world scenarios shown. Specifically, in the first column, the single target on the right is easily identified, but the two overlapping targets of different categories on the left are difficult. The detection performance of SSD, YOLOv3, and YOLOv4 is poor, with SSD misidentifying the tanker as a bulk carrier and YOLOv5s missing the small tugboat attached to the stern of the tanker. In this case, the proposed AodeMar model is able to accurately detect the boundaries of the small tugboat in the occluded area, demonstrating its excellent performance in locating the edges of overlapping targets.

[0204] In addition, as shown in the second column, while YOLOv4 can correctly detect all objects with high confidence, the proposed AodeMar model can provide more accurate bounding boxes. In densely occluded scenes, SSD, YOLOv3, YOLOv4, and YOLOv5s all have a large number of missed detections, and multiple objects are even bound to one bounding box. As shown in the third column, since the feature pyramid operation can fuse low-level and high-level features, SSD can achieve detection performance comparable to that of the proposed AodeMar model. As shown in the fourth column, the proposed AodeMar model significantly outperforms other corresponding models in terms of location and confidence scores, because the SP-STR block can capture semantically relevant features of objects of different scales that appear in the same scene at the same time. Through the above analysis, it can be concluded that the proposed AodeMar model shows significantly superior performance in positioning and classification, and can be applied in practical application scenarios.

[0205] D. Verification experiment under harsh occlusion scene

[0206] In order to verify the robustness of the proposed AodeMar model to environmental interference, SOTA detection frameworks such as SSD, YOLOv3 YOLOv4 and YOLOv5 were selected for comparison. Experiments were carried out in four typical scenarios: dense fog occlusion (first column), strong light and tail wave occlusion (second column), small targets at a long distance (third column) and color distortion (fourth column).

[0207] Fig.11 These are the occlusion detection results in four harsh environments. Fig.11 It can be easily observed that the proposed AodeMar algorithm performs better in these four scenes, especially in the case of strong light and trail occlusion and color distortion, all targets can be detected. It is worth noting that although two targets are inevitably missed due to blurred pixels in the sea fog and long-distance small target occlusion scenes, the proposed AodeMar algorithm is still better than other models. In this case, it is shown that the proposed AodeMar model can maintain robust detection performance in extreme marine environments. The following table, i.e., Table 3, is the abbreviation and full name of the present invention.

[0208] E. Quantitative comparison of OCCLUDED and UNOCCLUDEDCAS.

[0209]

[0210] To further verify the detection performance of occluded objects, 1173 occluded ships (OS) in 625 images are specifically marked as “occluded=1” in the entire test dataset. In this case, the test dataset is divided into 6470 unoccluded images and 625 occluded images, namely UI and OI. Therefore, as shown in Table V, the mAP indicators under different IoU thresholds are deployed to evaluate the detection capabilities of OS and unoccluded ships (US) respectively to show that AodeMar outperforms OS in both OS and US detection. In the UI sub-dataset, AodeMar retains a higher MAP compared to other typical detectors, thus showing its advantage in unoccluded scenes. On the OI sub-dataset, AodeMar significantly outperforms SSD and YOLOs with a mAP improvement of ~3%, and recognizes a maximum of 738 OS in a total of 1173 samples. It can be clearly seen that AodeMar has higher accuracy and success rate and can detect OS and US evenly.

[0211]

[0212]

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting unmanned boat targets at sea based on an attention mechanism under occlusion conditions, characterized in that: The following steps are involved: Obtain a data set of unmanned boat and marine ship targets under occlusion conditions to be processed; Preprocessing the images in the data set; Divide the preprocessed data set into training set and test set according to the proportion; Construct an AodeMar model for feature extraction, feature fusion, and prediction of the location and category of the target; The AodeMar model is trained based on the training set to obtain a trained AodeMar model; The test set is input into the trained AodeMar model to detect unmanned boats and marine vessels under occlusion conditions. The AodeMar model includes a backbone network for extracting features from images, a neck network for fusing the extracted features, and a detection head network for predicting the location and category of the target; The backbone network, the neck network, and the detection head network are connected in cascade in sequence; The backbone network includes 9 CBS modules, namely a first CBS module, a second CBS module, a third CBS module, a fourth CBS module, a fifth CBS module, a sixth CBS module, a seventh CBS module, an eighth CBS module and a ninth CBS module; The five C3 modules are the first C3 module, the second C3 module, the third C3 module, the fourth C3 module and the fifth C3 module; the first C3 module, the second C3 module, the third C3 module, the fourth C3 module and the fifth C3 module are all conducive to extracting more and more detailed feature information The RCAC3 module and the SPPF module focus on processing edge information of overlapping occluded areas by applying the CA method; the SPPF module mainly fuses multi-scale feature information; the RCAC3 module is to further enhance feature extraction capabilities; The first CBS module, the second CBS module, the first C3 module, the third CBS module, the second C3 module, the fourth CBS module, the third C3 module, the fifth CBS module, the RCAC3 module, the SPPF module, the sixth CBS module, the fourth C3 module, the seventh CBS module, the fifth C3 module, the eighth CBS module and the eighth CBS module are connected in sequence; The CBS module consists of a convolutional layer, a batch normalization layer, and an activation function. The main function of the CBS module is to upsample, reduce or increase the dimension, and normalize the feature map; There are three maximum pooling layers in SPPF, and each pooling layer operates differently on the input feature map; residual connections are added to SPPF; The RCAC3 module is formed by embedding the RCAB module into the C3 module, which consists of three convolutions and a CSP Botleneck. The RCAB module consists of two consecutive 2-D convolutional layers, batch normalization and SiLU activation functions, a CA module, and a residual connection; the backbone network extracts features from the image as follows: Assume that the output after passing through two CBS modules is First, in order to embed the coordinate information, two independent attention feature maps are generated in the horizontal and vertical directions respectively. and They are represented as: in: and are the outputs of the Cth channel in height and width, respectively. h and ω represent the height and width of the kth channel, respectively, k = 1, 2, ..., c; m and n represent the dimensions of the pooling kernel, (,) represents the position coordinates; the above two formulas represent the average pooling in the X and Y directions, respectively; The second step is the CA generation transformation process. First, the above two feature maps are cascaded and then sent to a 1×1 convolution F and a nonlinear activation function δ; at the same time, the spatial information can be encoded into the feature map G, that is: in: And r represents the reduction factor; then the feature map G l Decomposed along the position direction into and Then, two 1×1 convolution functions L are used h and L ω Respectively and Transformed into Tensors with the same channels; output with exact coordinate information can be output as: Among them, γ represents the sigmoid function, As the attention weight input, the output of the CA block can be expressed as: In the RCAB module, two CBS modules are used consecutively before CA, and a residual connection is added; in this case, the input of RCAB is: In the above case, the RCAB module The output can be expressed as: in, and Represents CBS operations with 1×1 and 3×3 convolution kernels respectively; and They are the input and output feature maps of RCAB, Is the shape and The same coefficient matrix, Element-wise addition; The RCAB module enhances the feature extraction capability and improves the representation of the region of interest; The RCAB module is embedded into the C3 module to form the RCAC3 module; the output of the fifth CBS module will pass through two paths at the same time, one of which passes through the CBS module and the RCAB module, and the other only passes through the CBS module; the fused output will be operated by the CBS module; Similar to RCAB, the position enhancement module RCAC3 input can be expressed as: in, and Respectively represent the input and output feature maps of RCAC3, Join operation in channel dimension.

2. According to claim 1, a method for detecting unmanned boat targets at sea based on an attention mechanism under occlusion conditions, characterized in that: The neck network includes an SP-STR module; The SP-STR module is formed by inserting the SPP module as an additional module into the STR self-attention block; The input feature map of SP-STR is Here and Represent the number of channels and dimensions in the width and height directions respectively. First, a 1×1 convolution operation CBS block is used to encode each channel of the input feature map to reduce the number of channels, which is defined as: in Represents a CBS operation with a convolution kernel size of 1×1; In the SPP module, the output of the previous concatenation layer is first subjected to a maximum pooling operation with kernel sizes of 1×1, 5×5, 9×9, and 13×13 to fuse global and local feature information, which is defined as: in Represents the maximum pooling operation with different pooling kernel sizes, extracting the corresponding feature maps It represents the concatenated feature map; note Finally, the output channels of SPP are restored to the same level as the input channels through a CBS block with a 1×1 convolution operation. The same size can be expressed as: in, The STR module contains two consecutive STR self-attention modules, each of which contains two sub-layers, namely, window-based multi-head self-attention W-MSA and multi-layer perceptron MLP layer. Specifically, layer normalization LayerNorm is used to normalize different features of a single sample in the channel dimension; Therefore, W-MSA is used to build information correlation in the window. In order to alleviate the vanishing gradient problem, the residual connection combines the forward complete information into the layer for local global semantic feature learning. Each layer of the W-MSA layer and the MLP layer of the STR self-attention module is normalized before the layer. The sub-layer output in the l-1-th block is expressed as: in, and Indicates W-MSA and LayerNorm operations. Please note After that, the MLP layer is deployed to learn the feature weights of global perception, which are expressed as: in, represents the MLP operation; In order to further reduce the calculation of MSA and establish information connection across windows, similarly, the shifted W-MSA and the corresponding MLP module are calculated, and the expression is: in, and Indicates SW-MSA operation; In this context, multi-scale feature semantic-related information is embedded to enhance the feature fusion capability and detection performance of objects of different sizes. In the proposed SP-STR module, two consecutive STR self-attention blocks are used, so l = 2.

Images