A sea surface weak and small target detection method based on semantic association

Abstract

The application relates to a sea surface small target detection method based on semantic association, which comprises the following steps: S1, acquiring a sea surface image containing a small target; S2, extracting a feature map F from the sea surface image by using a backbone feature extraction network; S3, acquiring a local context feature map from the feature map F by using a cavity convolution network, inputting the local context feature map into a self-attention network to acquire a global context feature map, fusing the local context feature map and the global context feature map to obtain a global-local context semantic feature map; meanwhile, acquiring a target association semantic feature map from the feature map F by using a graph convolution network; S4, fusing the global-local context semantic feature map and the target association semantic feature map in S3, classifying and regressing the fused feature map, and obtaining the category and position of the detected sea surface small target. Compared with the prior art, the application uses semantic association information to enhance and supplement the information of the small target, and improves the accuracy of the sea surface small target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, in particular to a sea surface small target detection method based on semantic association. BACKGROUND

[0002] The most widely used sea surface target detection method is for target detection under normal scenes, and these detection methods have good target detection effect in sunny days without occlusion. However, in complex scenes such as overcast, fog, rain and snow or occlusion, the target is not easy to observe, and the weak feature characteristics are presented, which aggravates the difficulty of small target detection.

[0003] In the process of small target detection, the current mainstream detection methods are mainly image level, instance level and pixel level alignment. The image level alignment method pays more attention to the global feature information of the whole image, and lacks attention to the context semantic information of the image. The instance level alignment method pays more attention to the appearance shape information of the target, and lacks attention to the associated semantic information between targets. The pixel level alignment method pays more attention to the fine-grained pixel information of the target, and lacks attention to the semantic information of the target.

[0004] In summary, in complex scenes such as image degradation caused by bad weather and target occlusion, the current method cannot obtain enough information for target recognition, thereby affecting the performance of small target detection in complex scenes. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a sea surface small target detection method based on semantic association, which uses semantic association information to enhance and supplement the information of small targets, and improves the accuracy of sea surface small target detection.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] The present application provides a sea surface small target detection method based on semantic association, which comprises the following steps:

[0008] Step S1, obtaining a sea surface image containing a small target;

[0009] Step S2, extracting a feature map F from the sea surface image using a backbone feature extraction network;

[0010] Step S3, obtaining a local context feature map N from the feature map F using a hollow convolution network, inputting the local context feature map into a self-attention network to obtain a global context feature map M', and fusing the local context feature map and the global context feature map to obtain a global-local context semantic feature map F t ; at the same time, a graph convolution network is used to obtain a target associated semantic feature map Fs ;

[0011] Step S4, fusing the global-local context semantic feature map F in step S3 t and the target associated semantic feature map F s , and regression, to obtain the category and position of the detected small sea target.

[0012] Preferably, the sea surface image containing a small target in step S1 adopts a high-resolution sea surface image.

[0013] Preferably, the backbone feature extraction network in step S2 is a backbone feature extraction network based on a first-order full convolution target detection FCOS.

[0014] Preferably, in step S3, a local context feature map is obtained from the feature map F using a dilated convolution network, which specifically includes the following sub-steps:

[0015] Step S301, obtaining a single local context feature map: the feature map F obtained by step S2 is operated using four dilated convolutions with the same parameters and expansion rates k1, k2, k3 and k5, to obtain four local context feature maps F1, F2, F3 and F5, wherein F, F1, F2, F3 and F5∈R C*H*W , C represents the channel dimension of the feature map, and H and W represent the length and width of the feature map, respectively;

[0016] Step S302, splicing to obtain a local context feature map: the obtained four local context feature maps F1, F2, F3 and F5 are spliced to obtain a local context feature map N, wherein N∈R 4C*H*W .

[0017] Preferably, the size of the dilated convolution is 3x3, and the expansion rates are set as k1=1, k2=2, k3=3 and k5=5, respectively.

[0018] Preferably, in step S3, the local context feature map is input into a self-attention network to obtain a global context feature map, which specifically includes the following sub-steps:

[0019] Step S311, for the local context feature map N, respectively using convolution layers W s and W D convert it into semantic spaces S and D, wherein {S, D}∈R C′*H*W , C' is the channel dimension of the converted feature map;

[0020] Step S312, capturing global context semantic information using a correlation matrix K, whose calculation formula is as follows:

[0021] K=ST D

[0022] wherein K∈R H*W*H*W ;

[0023] Step S313, normalize the correlation matrix K using sigmoid function and average pooling to construct a global context information mapping matrix K', wherein K'∈R (H*W*1) ; meanwhile use a convolution layer W M convert the feature map F into a value vector form M;

[0024] Step S314, perform element-wise multiplication on K' and the feature M to obtain a global context feature map M'.

[0025] Preferably, the step S3 fuses the local context feature map and the global context feature map to obtain a global-local context semantic feature map, specifically:

[0026] concatenate the local context feature map N and the global context feature map M' to obtain a local-global related context feature map N', and use a convolution layer to reduce the dimension of the feature to obtain a global-local context semantic feature map F t ; wherein N'∈R 5C*H*W , F t ∈R (H*W*C) , C represents the channel dimension of the feature map, and H and W represent the length and width of the feature map, respectively.

[0027] Preferably, the step S3 uses a graph convolution network to obtain a target associated semantic feature map from the feature map F, specifically including the following sub-steps:

[0028] Step S321, extract a semantic feature V0 of a specific class from the feature map F:

[0029] Each class v c is expressed as a weighted sum of the feature map F, and the resulting v c selectively aggregates features related to its specific class C, and the calculation formula is as follows:

[0030]

[0031] wherein is the semantic weight of the cth class at the feature map (i, j), and n c is the semantic weight of the cth class, and F i，j is the feature vector at the feature map (i, j);

[0032] Step S322, the class semantic feature V0 is used as the input of the graph convolution network to construct a class semantic relation graph and obtain a class associated node V1;

[0033] Step S323, the category association node V1 is upgraded to a category association semantic weight F' with the same dimension as the feature map F, then multiplied with the feature map F, and fused by using a convolution layer to obtain a target association semantic feature map F s ; wherein, F' ∈ R H*W*C .

[0034] Preferably, the calculation expression of the category association node V1 is:

[0035] V1 = δ (AV0W)

[0036] In the formula, A is an association matrix, W is a linear transformation matrix, δ(·) is an activation function Sigmoid(.), and V1 ∈ R C2*D , C2 is the number of graph node categories, and D is the number of graph vector dimensions.

[0037] Preferably, the step S4 specifically comprises: fusing the feature map F with the context semantics t and the feature map F with the target association semantics s , using feature splicing and 1x1 convolution to fuse to form a feature map F c , and finally classifying and regressing to obtain the category and position of the detected small target;

[0038] The target category classifier and regressor are used to obtain a target category probability vector p and a position (x, y, w, h), wherein x and y represent the center coordinates of the rectangular frame, and w and h represent the width and height of the rectangular frame.

[0039] Compared with the prior art, the present application has the following advantages:

[0040] 1) The method of the present application introduces target semantic association information, enhances and supplements the lost information of the small target on the sea surface, and further improves the accuracy of the small target detection on the sea surface; especially in the case of complex scenes such as overcast, fog, rain, snow or occlusion, the target is not easy to observe, and presents weak feature characteristics, which aggravates the detection difficulty of the small target

[0041] 2) The method of the present application increases the context semantic information to enhance the context semantic features related to the scene, so as to improve the accuracy of the small target detection. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flow chart of the method of the present application;

[0043] Figure 2 is a small target image on the sea surface input to the method of the present application;

[0044] Figure 3 is a small target detection result image on the sea surface obtained by the method of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort should fall within the protection scope of the present application.

[0046] Embodiment 1

[0047] The embodiment provides a sea surface small target detection method based on semantic association, which comprises the following steps.

[0048] Step S1, acquiring a sea surface image containing a small target;

[0049] Step S2, extracting a feature map F from the sea surface image by using a backbone feature extraction network;

[0050] Step S3, acquiring a local context feature map N from the feature map F by using a hole convolution network, inputting the local context feature map into a self-attention network to acquire a global context feature map M ′ , fusing the local context feature map and the global context feature map to obtain a global-local context semantic feature map F t ; meanwhile, acquiring a target association semantic feature map F s from the feature map F by using a graph convolution network;

[0051] Step S4, fusing the global-local context semantic feature map F t and the target association semantic feature map F s in step S3, and performing classification and regression on the fused feature map to obtain the category and position of the detected sea surface small target.

[0052] Embodiment 2

[0053] The embodiment provides a sea surface small target detection method based on semantic association, which comprises the following steps.

[0054] Step S1, acquiring a sea surface image containing a small target; in the embodiment, the sea surface image is acquired by using a high-resolution camera, as shown in FIG. 1. Figure 2

[0055] Step S2, extracting a feature map F of the whole image from a backbone feature extraction network; the backbone feature extraction network in the embodiment is a backbone feature extraction network based on a first-order full convolution target detection FCOS, and the specific implementation process in the embodiment is as follows.

[0056] ​A feature map F of the whole image is extracted from a backbone feature extraction network of a first-order full convolution target detection FCOS, where F e R C*H*W , C represents a channel dimension value of the feature map being 3, H and W represent a length value of the feature map being 1024 and a width value of the feature map being 1024 respectively.

[0057] Step S3, the feature map obtained in step S2 is used to obtain a local context feature map by using a dilated convolution; the obtained local context feature map is input into a self-attention network to obtain a global context feature map; the local context feature map and the global context feature map are fused to obtain a global-local context semantic feature map; and the feature map obtained in step S2 is used to obtain a target association semantic feature map by using a graph convolution network.

[0058] Step S4, finally, the feature map with the context semantics and the target association semantics is fused, and the fused feature map is classified and regressed to obtain the category and position of the detected small target.

[0059] Next, the method in the embodiment is described in detail.

[0060] In the embodiment, referring to Figure 1 , the feature map obtained in step S2 is used to obtain a local context feature map in step 3. The process is as follows:

[0061] Step S301, a single local context feature map: the feature map F obtained in step 2 is operated by using four 3x3 dilated convolutions with the same parameters and expansion rates of 1, 2, 3 and 5 respectively to obtain four local context feature maps F1, F2, F3 and F5, and the specific calculation formula is as follows:

[0062] F1, F2, F3, F5 = Conv 3×3 (F) (1，2，3，5 ) (1)

[0063] Wherein, F1, F2, F3, F5 e R C*H*W , C represents a channel dimension value of the feature map being 3, H and W represent a length value of the feature map being 1024 and a width value of the feature map being 1024 respectively.

[0064] Step S302, local context feature map: the four local context feature maps F1, F2, F3 and F5 obtained by the formula (1) are spliced to obtain a local context feature map N, and the specific calculation formula is as follows:

[0065] N = Concat (F1, F2, F3, F5) (2)

[0066] Wherein, N e R 4C*H*W, C represents the channel dimension value of the feature map is 3, H, W represent the length value of the feature map is 1024, the width value is 1024.

[0067] In this embodiment, referring to Figure 1 , the local context feature map obtained in step 3 is input into the self-attention network to obtain the global context feature map, and the process is as follows:

[0068] Step S311, for the local context feature map N, respectively adopt convolution layer W s and W D Convert it to semantic space S, D, the calculation formula is as follows:

[0069]

[0070]

[0071] Where, {S, D} ∈ R C′*H*W , C' represents the channel dimension value of the feature map is 512, H, W represent the length value of the feature map is 1024, the width value is 1024.

[0072] Step S312, use the correlation matrix K to capture the global context semantic information, the calculation formula is as follows:

[0073] K = S T D (5)

[0074] Where, K ∈ R H*W*H*W , H, W represent the length value of the feature map is 1024, the width value is 1024.

[0075] Step S313, use the sigmoid function and average pooling to normalize the correlation matrix K, construct a global context information mapping matrix K', where K' ∈ R (H*W*1) , H, W represent the length value of the feature map is 1024, the width value is 1024. At the same time, use the convolution layer W M Convert the feature map F to the value vector form M, the calculation formula is as follows:

[0076]

[0077] Where, M ∈ R (H*W*C) , C represents the channel dimension value of the feature map is 3, H, W represent the length value of the feature map is 1024, the width value is 1024.

[0078] Step S314, element-wise multiplication of K' and feature M, get the global context feature map representation M', the calculation formula is as follows:

[0079] M' = K'*M (7)

[0080] where M' ∈ R (H*W*C) , C represents the channel dimension value of the feature map as 3, H and W represent the length and width values of the feature map as 1024 respectively.

[0081] In this embodiment, referring to Figure 1 , the fusion of the local context feature map and the global context feature map in step 3 is used to obtain a global-local context semantic feature map.

[0082] The local-global context feature map N' ∈ R 5C*H*W is obtained by splicing the local context feature map N obtained by formula (2) and the global related context M'. t 1x1 convolution operation is used to reduce the dimension of the feature, and finally a global-local context semantic feature map F t is obtained, and the calculation formula is as follows:

[0083] F t = Conv 1×1 (concat(N, M')) (8)

[0084] where F t ∈ R (H*W*C) , C represents the channel dimension value of the feature map as 3, H and W represent the length and width values of the feature map as 1024 respectively.

[0085] In this embodiment, referring to Figure 1 , the feature map obtained in step 2 is used in step 3 to obtain a target associated semantic feature map using a graph convolution network, and the process is as follows:

[0086] Step S321, semantic feature of a specific class: the class-related semantic feature extraction is performed on the feature map F output in step 2 to form a semantic feature V0 of a specific class, and each class v c is expressed as a weighted sum of F, and the v c thus generated can selectively aggregate features related to its specific class C, and the calculation formula is as follows:

[0087]

[0088] where represents the semantic weight of the cth class in the feature map (i, j), nc represents the semantic weight of the cth class, and F i，j represents the feature vector in the feature map (i, j). Here, c can represent the liner, other ship class, i and j represent values from 0 to 1024 respectively.

[0089] Step S322, category semantic relationship graph construction: the category semantic features V0 are taken as the input of graph convolution, a single-layer graph convolution is used to construct a category semantic relationship graph, so as to obtain category association node representation V1. The correlation matrix A reflects the relationship between the features of each node. In the reasoning process, the correlation matrix A first diffuses the relevant information to all nodes, and then each node receives all the necessary information and updates its state through a linear transformation W. The calculation formula is as follows:

[0090] V1=δ(AV0W) (10)

[0091] Wherein, A is randomly initialized, W is learned in training; δ(·) represents the activation function Sigmoid(.), which makes the whole operation nonlinear, V1∈R C2*D , C2 is the number of graph node categories 14, and D is the number of graph vector dimensions 512.

[0092] Step S323, feature map of category association semantic information: the dimension of the category association node V1 is upgraded to form a category association semantic weight F' with the same dimension as the feature map F output in step 2. H*W*C Then multiply it with the feature map F, and use 1x1 convolution to fuse it to form a feature map F2 containing category association semantic information, and the calculation formula is as follows:

[0093] F s =Conv 1×1 (F·F′) (11)

[0094] Wherein, F s ∈R H*W*C , C represents the channel dimension value of the feature map, which is 3, H and W represent the length and width of the feature map respectively, which are 1024 and 1024.

[0095] In this embodiment, referring to Figure 1 , the feature map F t with context semantic information in step S4 is fused with the feature map F s with target association semantic information, feature splicing and 1x1 convolution are used to fuse the feature map F c , and finally the feature map F c is classified and regressed to obtain the category and position of the detected small target, and the feature map F c , and the calculation formula is as follows:

[0096] F c =Conv 1×1 (concat(F t ,F s )) (12)

[0097] Wherein, F c ∈RH*W*C C represents a characteristic map channel dimension value of 3, H and W represent a characteristic map length value of 1024 and a characteristic map width value of 1024, respectively.

[0098] The target category classifier and regressor are used to obtain a target category probability vector p and a position (x, y, w, h), wherein x and y represent the center coordinates of the rectangular frame, and w and h represent the width and height of the rectangular frame. The target probability vector p of the small target detection category civilian ship and the position are as shown in Figure 3 .

[0099] In summary, the sea small target detection method based on semantic association in the embodiment, (1) input the image obtained by the high-resolution camera as shown in Figure 2 ; (2) extract the feature map of the whole image from the backbone feature extraction network of FCOS; (3) use the feature map obtained in step (2) to obtain a local context feature map by using a hole convolution; (4) input the local context feature map obtained in step (3) into a self-attention network to obtain a global context feature map; (5) fuse the local context feature map and the global context feature map to obtain a global-local context semantic feature map; (6) use the feature map obtained in step (2) to obtain a target association semantic feature map by using a graph convolution network; (7) fuse the feature maps with context semantics and target association semantics, classify and regress the fused feature map, and obtain the category and position of the detected small target, as shown in Figure 3 . Compared with the traditional method, the method of the embodiment uses semantic association information, which is more likely to detect small targets on the sea surface and improves the accuracy of sea small target detection.

[0100] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting a small target on sea surface based on semantic correlation, characterized in that, The method comprises the following steps: Step S1, acquiring a sea surface image containing a weak and small target; Step S2, extracting a feature map from the sea surface image by using a backbone feature extraction network ; Step S3: Use a dilated convolutional network to extract features from the feature map. Obtaining local context feature maps The local context feature map is input into the self-attention network to obtain the global context feature map. By fusing local context feature maps and global context feature maps, a global-local context semantic feature map is obtained. Simultaneously, a graph convolutional network is used to extract features from the feature map. Obtain target-related semantic feature map ; Step S4, fusing the global-local context semantic feature maps in step S3 and the target associated semantic feature map Classifying and regressing the fused feature map to obtain the category and position of the detected small target on the sea surface.

2. The weak small target detection method based on semantic association on sea surface according to claim 1, characterized in that, The sea surface image containing a weak and small target in the step S1 is a high-resolution sea surface image.

3. The weak small target detection method based on semantic association on sea surface according to claim 1, characterized in that, The backbone feature extraction network in the step S2 is a backbone feature extraction network based on a first-order full convolution target detection FCOS.

4. The weak small target detection method on sea surface based on semantic association according to claim 3, characterized in that, The step S3 employs a cavity convolution network to obtain the feature map from the feature map Obtaining a local context feature map, specifically comprising the following sub-steps: Step S301, obtaining a single local context feature map: the feature map obtained by step S2 , four local context feature maps are obtained by using four same parameters, and the dilation rates are respectively set as k1, k2, k3 and k5 , wherein, , indicates the channel dimension of the feature map, respectively indicates the length and width of the feature map; Step S302, splicing to obtain a local context feature map: splicing the four obtained local context feature maps Splicing is performed to obtain a local context feature map wherein, .

5. The weak small target detection method on sea surface based on semantic association according to claim 4, characterized in that, The size of the hole convolution is 3x3, and the expansion rates are respectively k1=1, k2=2, k3=3 and k5=5.

6. The weak small target detection method based on semantic association on sea surface according to claim 1, characterized in that, In the step S3, the local context feature map is input into a self-attention network to acquire a global context feature map, and the step S3 specifically comprises the following substeps: Step S311, for local context feature map , respectively, using convolutional layers and convert them into semantic space and , wherein, , is the converted feature map channel dimension, respectively represent the length and width of the feature map; Step S312, using the correlation matrix Capturing global context semantic information, whose calculation formula is as follows: In the formulae, ; Step S313, normalizing the correlation matrix using a sigmoid function and average pooling Normalization is performed to construct a global context information mapping matrix Wherein ; while using a convolution layer Convert the feature map F into a value vector form ; Step S314, element-wise multiplication is performed on and the features to obtain the global context feature map .

7. The weak small target detection method based on semantic association on sea surface according to claim 1, characterized in that, In the step S3, the local context feature map and the global context feature map are fused to obtain a global-local context semantic feature map, and the step S3 specifically comprises the following substeps: Spliced local context feature map With global context feature map , get local-global related context feature map , using convolution layer to reduce dimension of features, get global-local context semantic feature map ; wherein, , , Indicate the channel dimension of the feature map, Indicate the length and width of the feature map respectively.

8. The weak small target detection method based on semantic association on sea surface according to claim 1, characterized in that, The step S3 adopts a graph convolution network to obtain a target correlation semantic feature map from the feature map The step S3 adopts a graph convolution network to obtain a target correlation semantic feature map from the feature map Step S321, extracting semantic features from the feature map extracting semantic features of a specific class : Each class The expression is for feature maps The weighted sum produces Selectively aggregate according to its specific category The relevant features and calculation formulas are as follows: In the formula, Represented as the first Each class in the feature map semantic weights, Represented as the first Semantic weight of each class, Represented as in the feature map eigenvectors, These represent the length and width of the feature map, respectively. Step S322, category semantic feature As the input of the graph convolution network, a category semantic relation graph is constructed, and a category correlation node V1 is obtained. Step S323, upgrade the category association node V1 to a category association semantic weight with the same dimension as the feature map F , and then multiplied by the feature map , and fused by a convolution layer to obtain a target association semantic feature map ; wherein, , indicates the channel dimension of the feature map.

9. The weak small target detection method on sea surface based on semantic association according to claim 8, characterized in that, The calculation expression of the category correlation node V1 is: In the formula, A is an association matrix, W is a linear transformation matrix, δ(·) is an activation function Sigmoid(.), , is the number of graph node categories, is the number of graph vector dimensions.

10. The weak small target detection method on sea surface based on semantic association according to claim 1, characterized in that, The step S4 specifically includes: fusing the feature map with context semantics The feature map associated with the target semantics The feature map is fused by using feature splicing and 1x1 convolution Finally, classification and regression are performed to obtain the category and position of the detected small target A target category classifier and a regressor are used to obtain a category probability vector p and a position (x, y, w, h) of the target, wherein x and y represent the center coordinates of the rectangular frame, and w and h represent the width and height of the rectangular frame.

Images