A ship target detection and heading discrimination method based on bow point auxiliary of remote sensing image

Abstract

The application provides a ship head point auxiliary-based remote sensing image ship target detection and heading discrimination method.The method is as follows: a remote sensing image of a target ship is acquired, a ship center point estimation sub-network is used to extract features of the remote sensing image target ship, a center point position of the ship target, a center point regression offset, and a width and a height of the ship target are predicted, a ship head point estimation sub-network is used to extract ship head features of the remote sensing image target ship, a ship head point position of the ship target, a ship head point regression offset are predicted, and a final ship head point position of the ship target is determined, the predicted ship head point position and the center point position are connected, a rotation angle of the ship target is determined, and the sailing direction of the ship target is discriminated.The method of the application converts a discontinuous angle regression problem into a continuous ship head point and center point estimation problem of the ship target, and realizes ship target detection and heading discrimination.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a remote sensing image ship target detection and heading discrimination method based on a ship head point auxiliary. BACKGROUND

[0002] With the launch of many high-resolution remote sensing satellites, the ability to obtain sea surface ship target data has been greatly improved, which provides important support for building a maritime power. At the same time, the ship target detection and heading recognition method based on remote sensing images is the basis for judging the formation, situation, tracking and other downstream tasks of the fleet. How to further improve the performance of remote sensing image ship target detection and realize the discrimination of ship heading is a hot research topic at present.

[0003] At present, the labeling method of the target in the air-to-ground detection and recognition method is divided into a rotating frame and a horizontal frame. The horizontal frame can be used for labeling general target detection, but this method is not suitable for ship targets with arbitrary direction, dense arrangement and complex background. The large overlap of the horizontal frame labeled for dense ships will lead to redundant background information and reduce the detection accuracy, and cannot effectively reflect the heading of the ship head and tail. Using a rotating frame to mark the remote sensing ship target can extract more accurate features, positions and directions.

[0004] With the development of deep learning technology, remote sensing image target detection, as a major branch, has also attracted much attention. Initially, some scholars directly migrated the detection method of general targets to remote sensing image ship target detection, avoiding the traditional complex feature selection and extraction, and improving the detection ability and speed of ship targets in complex scenes compared with manual feature extraction. However, these works use horizontal frame labeling and do not consider the characteristics of ship targets, such as extreme aspect ratio and multi-directional rotation, which will introduce excessive background noise and limit the detection accuracy. To solve the limitations of horizontal frame labeling of ship targets, angle information is introduced into the feature extraction network, and an angle prediction branch is added to rotate the original horizontal frame according to the predicted angle to fit the extreme aspect ratio of the ship target. However, the extreme aspect ratio of the ship target is very sensitive to the change of the angle, and the change of the angle has periodicity, which causes boundary problems in angle regression, making it difficult to train the angle regression. Some scholars have proposed to convert the angle information into a classification problem, which to some extent alleviates the loss function discontinuity problem caused by the periodicity of the angle, but there is still a small range of boundary oscillation. The current research direction of remote sensing image ship target is mainly concentrated on detection and recognition, and the research on ship heading and formation shape discrimination is relatively less, which is also due to the lack of directionality of the rotating frame label, which cannot directly reflect the heading of the ship head and tail. SUMMARY

[0005] The technical problem to be solved by the present invention is to provide a method for ship target detection and heading determination based on bow point assistance in remote sensing images, which addresses the shortcomings of the prior art. This method transforms the discontinuous angle regression problem into a continuous key point estimation problem, thereby realizing ship target detection and heading determination.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for ship target detection and course determination based on bow point-assisted remote sensing images, characterized in that it includes:

[0007] S1. Use a rotating frame with a bow point to mark ship targets arranged in any direction. Ship targets are represented by (c x ,c y ,w,h,h x ,h y ) indicates that (h x ,h y (c) represents the coordinates of the bow point of the ship target. x ,c y ) represents the coordinates of the center point of the ship target's rotating frame, w represents the width of the ship target, and h represents the height of the ship target;

[0008] S2. Introduce an orientation-invariant model and a SimAM parameterless attention mechanism into the feature extraction network to extract rotation-invariant bow and center point feature maps.

[0009] S3. Select the vertices of the center point feature map as the center point to form a ship center point estimation sub-network, predict the center point position of the ship target, the center point regression offset, and the width and height of the ship target.

[0010] S4. Extract potential head points from the bow point feature map to form a bow point estimation sub-network, predict the bow point position of the ship target, the bow point regression offset, and determine the final bow point position of the ship target.

[0011] S5. Connect the predicted bow position and center position to determine the ship target's sailing direction, determine the ship target's rotation angle, and combine this with the predicted ship target's width and height.

[0012] Preferably, S2 introduces a direction-invariant model and a SimAM parameterless attention mechanism into the feature extraction network; specifically including:

[0013] Step S201: Generate orientation-invariant feature maps using the orientation-invariant model.

[0014] The direction-invariant model is introduced into the DLA34 backbone network, so that the ship target obtains angle and position information, and has complete rotational invariance; the direction-invariant model mainly includes a rotation filter for generating a feature map with direction and coding information, and a response pooling for generating a rotation-invariant deep feature by discarding direction information through pooling, a new angle feature map is obtained by performing convolutional transformation on the feature map of the last layer by using the filter corresponding to the angle, and the finally output rotation-invariant feature map is a combination of feature maps of all angles;

[0015] In step S202, the SimAM parameter-free attention mechanism enhances the attention degree of the ship target:

[0016] The SimAM attention mechanism does not have a parameter burden on the network, and can infer 3-D target attention weights of the feature map; inspired by human vision, in order to obtain spatial and channel attention at the same time, the SimAM attention mechanism is designed so that each neuron has its own weight.

[0017] Preferably, in S3, the top point of the center point feature map is selected as the center point to form a ship center point estimation sub-network, which predicts the center point position, center point regression offset, width and height of the ship target; specifically including:

[0018] In step S301, the ship center point estimation sub-network predicts the center point position:

[0019] The target ship remote sensing image is subjected to a ship target feature extraction network to generate an image center point heat map, and a Gaussian kernel function is used to map the center point of the ship target rotation frame to the heat map; since the positive samples in the heat map are only the center points, and the negative samples are too many, focal loss is used for training:

[0020]

[0021] Where N is the number of detected targets, and α and β are constraint coefficients, Y xyc is a predicted center point heat map, Y xyc is a real heat map;

[0022] In step S302, the ship center point estimation sub-network predicts the center point regression offset:

[0023] The input target ship remote sensing image is subjected to a feature network for 4 times of down-sampling; when the feature is mapped to the original image again, an error will be generated; it is assumed that all classes share the same offset prediction, and the training process uses an L1 loss function to optimize the position offset of the center point, specifically:

[0024]

[0025] Where O pa predicted offset for the center point, a predicted value for the center point position coordinate, an offset for the center point position coordinate.

[0026] Step S303, the ship center point estimation sub-network predicts the width and height of the ship target:

[0027] The width and height of the ship are predicted at the center point of the ship target using an L1 distance loss function:

[0028]

[0029] where S cwh is the estimated width and height of the ship target, S wh is the width and height of the original target.

[0030] Preferably, the potential head point in the ship head point feature map is extracted in S4, a ship head point estimation sub-network is formed, the ship head point position of the ship target is predicted, the ship head point regression offset is determined, and the final ship head point position is determined; specifically including:

[0031] Step S401, the ship head point estimation sub-network predicts the target ship head point:

[0032] Assuming the coordinates of the ship head point are h k =(h x ,h y ), after 4 times of down-sampling of the input data, the key point corresponding to the ship head point is Further, the heat map of the ship head point and the offset is obtained by using the Gaussian kernel function, and the target function is continuously optimized:

[0033]

[0034] where Exyc is the predicted ship head point heat map, E xyc is the true heat map;

[0035] Step S402, the ship head point estimation sub-network regresses the ship head point offset to obtain the final ship head point position:

[0036] The offset at the ship head point is optimized using the L1 loss function:

[0037]

[0038] where O OH is the offset of the predicted ship head point, headk / R is the feature after R times of down-sampling of the data, head is the predicted ship head point position, (head k / R-head) is the offset of the final coordinate position of the ship head point.

[0039] Preferably, the connection of the predicted bow point position and the center point position in S5 determines the sailing direction of the ship target, determines the rotation angle of the ship target, and combines the predicted width and height of the ship target; Specifically, it includes:

[0040] The resulting rotation frame coordinates of the ship target are (x+δ x ,y+δ y ,w,h,x+ζ x ,h y +ζ y ), wherein (x,y) is the estimated center point of the ship target, (δ x ,δ y ) is the center point offset, w and h are the width and height of the ship target respectively, (ζ x ,ζ y ) is the bow point offset, and further connecting the center point and the bow point can determine the heading of the ship target, realizing the discrimination of the ship heading.

[0041] Compared with the prior art, the present application has the following advantages:

[0042] 1. The present application converts the discontinuous angle regression problem into a continuous key point estimation problem, realizes ship target detection and heading discrimination. First, the ship target arranged in any direction is marked with a rotating frame with a bow point, and a direction invariant model and a SimAM non-parametric attention mechanism are introduced into the backbone network to enhance the network's attention to the ship target and improve the target's representation ability; Second, the center point of the ship target is predicted using a heat map, and the offset and width of the target are regressed; At the same time, the bow point of the ship target and the offset are further regressed to determine the accurate position of the bow point; Then, the predicted bow point and center point position information are used to determine the rotation angle of the ship, and the ship width and height are combined to realize the rotation frame detection of the target; Finally, the bow point and the center point are connected to determine the heading.

[0043] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples. DETAILED DESCRIPTION

[0044] Figure 1 A flowchart of a ship target detection and heading discrimination method based on a bow point auxiliary remote sensing image according to the present application is disclosed.

[0045] Figure 2 The ship target rotating frame with a bow point is marked.

[0046] Figure 3The figures show the experimental results of three ship target heading detection methods: (a) the method using ResNet18 as the feature extraction network; (b) the method using DLA34 as the feature extraction network; (c) the method using Hourglass as the feature extraction network; and (d) the method of this invention.

[0047] Figure 4 The detection results for the estimated bow point and center point are given using the method disclosed in Embodiment 1 of the present invention. Detailed Implementation

[0048] Example 1

[0049] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for ship target detection and course determination based on bow point-assisted remote sensing images. The method includes:

[0050] S1. Ship targets arranged in any direction will be marked using a rotating frame with a bow point; specifically including:

[0051] The ship's orientation is determined by using the coordinates of its bow and center point. Remotely sensed ship targets are marked with the coordinates of their bow point (h). x ,h y ) rotating frame (c x ,c y ,w,h,h x ,h y ), where (c x ,c y The coordinates of the center point of the rotating frame of the ship target are represented by , w represents the width of the ship target, and h represents the height of the ship target; for example... Figure 2 As shown, this representation method is different from that of a rotated frame (c x ,x y Labeling ship targets with (w,h,θ) can transform continuous angle regression into a keypoint estimation problem.

[0052] S2. Introduce an orientation-invariant model and the SimAM parameterless attention mechanism into the feature extraction network to extract rotation-invariant bow and center point feature maps; specifically including:

[0053] Step S201: Generate orientation-invariant feature maps using the orientation-invariant model.

[0054] The direction-invariant model is introduced into the DLA34 backbone network, so that the ship target obtains angle and position information, and has complete rotational invariance; the direction-invariant model mainly includes a rotation filter for generating a feature map with direction and coding information, and a response pooling for generating a rotation-invariant deep feature by discarding direction information through pooling. The new angle feature map is a filter corresponding to an angle for convolutional transformation on the last layer feature map, and the finally output rotation-invariant feature map is a combination of feature maps of all angles. The rotation filter generates a feature map with direction and coding information in the input The kth direction output of the image is:

[0055]

[0056] wherein M is a feature map, F is an active rotation filter, θ represents a clockwise rotation angle, and the response pooling outputs a feature map in the kth direction:

[0057]

[0058] Step S202, the SimAM parameter-free attention mechanism enhances the attention degree of the ship target:

[0059] The parameter-free attention module SimAM is added in the DLA34 backbone network, compared with the original attention channel and spatial attention module, the SimAM attention mechanism has no parameter burden on the network, and can infer the 3-D target attention weight of the feature map. Inspired by human vision, in order to obtain spatial and channel attention at the same time, the SimAM attention is designed so that each neuron has its own weight. In neurology, neurons are very sensitive to the richness of information. In order to linearly separate the information-rich neurons from other neurons, an energy function is designed to define the neurons:

[0060]

[0061] wherein t is a target neuron (information-rich and abnormally active) of an input feature, x i is another neuron, (w t x i +b t ) and (w t t+b t ) are linear transformations on different neurons, and M is the number of neurons. The SimAM attention is inserted into the feature extraction network to improve the target detection accuracy.

[0062] S3, the top point of the center point feature map is selected as the center point to form a ship center point estimation sub-network, and the center point position, center point regression offset, width and height of the ship target are predicted.

[0063] Step S301, the ship center point estimation sub-network predicts the center point position:

[0064] The remote sensing image of the target ship is subjected to a ship target feature extraction network to generate an image center point heat map, and a Gaussian kernel function is used to map the center point of the ship target rotation frame to the heat map. Since the positive samples in the heat map are only center points and the negative samples are excessive, focal loss is used for training:

[0065]

[0066] Where N is the number of detected targets, and a and β are constraint coefficients, Y xyc is the predicted center point heat map, and Y xyc is the true heat map.

[0067] Step S302, the ship center point estimation subnetwork predicts the regression offset at the center point:

[0068] The input remote sensing image of the target ship is subjected to 4 times of down-sampling by the feature network. When the feature is mapped to the original image again, an error will be generated. It is assumed that all classes share the same offset prediction. The training process uses the L1 loss function to optimize the position offset of the center point. Specifically:

[0069]

[0070] Where O p is the predicted offset at the center point, is the predicted value of the center point coordinate, is the offset of the center point coordinate.

[0071] Step S303, the ship center point estimation subnetwork predicts the target width and height:

[0072] The L1 distance loss function is used to predict the width and height of the ship at the ship target center point:

[0073]

[0074] Where S cwh is the predicted width and height of the ship target, and S wh is the width and height of the original target.

[0075] S4, extract the potential head point in the ship head point feature map to form a ship head point estimation subnetwork, predict the ship head point position, ship head point regression offset, and determine the final ship head point position of the ship target; Specifically includes:

[0076] Step S401, the ship head point estimation subnetwork predicts the target ship head point:

[0077] Assuming the coordinates of the ship head point are h k =(hx h y ), consistent with the idea of step S301, after 4 times down-sampling, the key point corresponding to the bow point of the input data is Further use the Gaussian kernel function to get the heat map of the bow point and the offset, and then continuously optimize the target function through training:

[0078]

[0079] Where Exyc is the predicted heat map of the bow point, E xyc is the real heat map.

[0080] Step S402, the bow point estimation sub-network regresses the bow point offset to obtain the final bow point position:

[0081] The offset at the bow point is optimized by L1 loss function:

[0082]

[0083] Where O OH is the offset of the predicted bow point, headk / R is the feature of the data after R times down-sampling, head is the predicted bow point position, and (head k / R-head) is the offset of the precise coordinate position of the bow point.

[0084] S5, connect the predicted bow point position and the center point position to determine the sailing direction of the ship target and the rotation angle of the ship target, and combine the predicted width and height of the ship target; specifically including:

[0085] In combination with steps S2, S3 and S4, the rotation frame coordinates of the ship target are (x+δ x ,y+δ y ,w,h, x +ζ x ,h y +ζ y ), where (x,y) is the estimated center point of the ship target, (δ x ,δ y ) is the offset at the center point, w and h are the target width and height respectively, (ζ x ,ζ y ) is the offset of the bow point, and further connecting the center point and the bow point can determine the heading of the ship target, realizing the discrimination of the ship heading.

[0086] 1. Simulation conditions

[0087] In order to test the effectiveness of the present application, the effectiveness of the ship target detection and identification method based on the ship head point auxiliary and the heading discrimination method is verified, and the following experiments are carried out. The remote sensing image ship target data set HRSC2016 and FGSD2021 data set are used, a total of 6134, of which 5498 and 636 are used for training and testing respectively, including aircraft carriers, Austin class, Whitby island class, Arlie burke class and submarines, a total of 20 kinds of ships. The present application is based on Ubuntu18.04 system and deep learning Pytorch framework for experiment, and the environment configuration is Intel(R)Core(TM)i7-10750HCPU@2.59GHz notebook computer.

[0088] 2. Simulation experiment

[0089] Table 1 shows the experimental results of the ship target heading detection method based on the ship head point.

[0090]

[0091]

[0092] Table 1 shows the experimental results of the ship target heading detection method based on the ship head point. The model based on DLA34 as the backbone network adds the direction invariant model and SimAM non-parametric attention, and the detection accuracy value reaches 82.3%, which is obviously improved. Figure 3 The experimental results of the ship target heading detection method based on the key point are shown in the figure, Figure 3 (a-d) are the feature extraction networks respectively Resnet18, DLA34, Hourglass and the method mentioned in the present application, and the red points in the figure are marked as the ship head point. Among them Figure 3 In (a), resnet18 is selected as the feature extraction network, and more false detections are generated, and the network feature learning and fitting ability is not strong; Figure 3 (d) is the improved DLA34 network used in the present application, which has good detection effect, less false detection and missed detection, and can estimate the coordinate position of the ship head point, and then effectively detect the ship target in the remote sensing image.

[0093] Figure 4 The detection results of the ship head point and the center point estimation are shown in the figure, the green color is the center point of the ship target, the blue color is the estimated correct ship head point, and the yellow color is the predicted wrong ship head point. Most of the ship head points can be correctly detected, and when there are dense small targets similar to the ship head, a small amount of ship head estimation may be opposite, and the heading of the ship head is further discriminated by connecting the ship head point and the center point.

[0094] The above merely describes preferred embodiments of the present application, and is not intended to limit the present application. Any simple modification, change and equivalent variation of the above embodiments according to the technical essence of the present application are still within the protection scope of the technical scheme of the present application.

Claims

1. A ship target detection and heading discrimination method based on bow point auxiliary remote sensing image, characterized in that, Comprise: S1, mark the ship target arranged in any direction with a rotating frame with a ship head point, and the ship target is marked with , , , w , h , S2, in the feature extraction network, a direction-invariant model and a SimAM parameter-free attention mechanism are introduced to extract the rotation-invariant ship head point and center point feature maps; S3, the top point of the center point feature map is selected as the center point to form a ship center point estimation subnetwork, which predicts the center point position, center point regression offset, width and height of the ship target; S4, the potential head point in the ship head point feature map is extracted to form a ship head point estimation subnetwork, which predicts the ship head point position, ship head point regression offset, and determines the final ship head point position of the ship target; S5, the predicted ship head point position and center point position are connected to determine the sailing direction of the ship target and the rotation angle of the ship target; In S2, a direction-invariant model and a SimAM parameter-free attention mechanism are introduced in the feature extraction network; specifically comprising: Step S201, the direction-invariant model generates a direction-invariant feature map: A direction-invariant model is introduced in the DLA34 backbone network, which enables the ship target to obtain angle and position information and has complete rotation invariance; the direction-invariant model includes a rotation filter to generate a feature map with direction and encoding information, and a response pooling to generate a rotation-invariant deep feature by discarding direction information through pooling; the new angle feature map is a convolutional transformation of the filter corresponding to the angle on the last layer feature map, and the finally output rotation-invariant feature map is a combination of feature maps of all angles; Step S202, SimAM parameter-free attention mechanism, enhances the attention of the ship target: The SimAM attention mechanism has no parameter burden on the network, and can infer the 3-D target attention weight of the feature map; inspired by human vision, in order to obtain spatial and channel attention at the same time, the SimAM attention is designed so that each neuron has its own weight.

2. The method of claim 1, wherein the method further comprises: In S3, the top point of the center point feature map is selected as the center point to form a ship center point estimation subnetwork, which predicts the center point position, center point regression offset, width and height of the ship target; specifically comprising: Step S301, the ship center point estimation subnetwork predicts the center point position: The remote sensing image of the target ship is processed by the ship target feature extraction network to generate an image center point heat map, and the center point of the ship target rotation frame is mapped to the heat map using a Gaussian kernel function; since the positive samples in the heat map are only the center points, and the negative samples are too many, focal loss is used for training: ； wherein is the number of targets detected, is the constraint coefficient, is the predicted center point heat map, is the real center point heat map; Step S302, the ship center point estimation subnetwork predicts the regression offset at the center point: The input remote sensing image of the target ship is processed by the feature network for 4 times of downsampling, and when the feature is mapped to the original image again, an error will be generated; assuming that all classes share the same offset prediction, the L1 loss function is used to optimize the position offset of the center point during the training process, specifically: ； wherein is a predicted offset at the center point, is a predicted value of the center point position coordinate, is an offset of the center point position coordinate; Step S303, the ship center point estimation subnetwork predicts the width and height of the ship target: The L1 distance loss function is used to predict the width and height of the ship at the ship target center point: ； wherein is the width of the ship target to be estimated, is the width of the original target.

3. The method of claim 1, wherein the method further comprises: In S4, a bow point estimation sub-network is formed by extracting potential bow points in the bow point feature map, and the bow point position of the ship target, the bow point regression offset, and the final bow point position of the ship target are predicted; specifically including: Step S401, the bow point estimation sub-network predicts the target bow point: Assume the coordinates of the bow point are , after 4 times of down-sampling of the input data, the key point corresponding to the bow point is , further use the Gaussian kernel function to obtain the heat map of the bow point and the offset, and then continuously optimize through the training objective function: ； wherein is a predicted bow point heat map, is a real bow point heat map; Step S402, the bow point estimation sub-network regresses the bow point offset to obtain the final bow point position: The offset at the bow point is optimized by using an L1 loss function: ； wherein is the offset of the predicted aft ship head point, is the feature of the data after R times down-sampling, is the predicted ship head point position, is the offset of the final coordinate position of the ship head point.

4. The method of claim 1, wherein the method further comprises: In S5, the predicted bow point position and the center point position are connected to determine the sailing direction of the ship target and determine the rotation angle of the ship target; specifically including: The obtained rotating frame coordinates of the ship target are wherein is a center point of the ship target, is an offset at the center point, are a width and a height of the ship target, respectively, is an offset of a bow point, and further connecting the center point and the bow point can determine a heading of the ship target, thereby realizing discrimination of the ship heading.

Images