A ship detection method integrating the temporal and spatial information of synchronous orbit sequence optical images

Abstract

The present invention provides a ship detection method that integrates the spatiotemporal information of synchronous orbit sequence optical images. For low-resolution synchronous orbit sequence optical images, the complementary and differential information of moving ships in different frame images is used to detect moving ships, thereby improving the detection accuracy. In model training and application, on the basis of multi-frame image superposition, the ship target sequence in the synthetic image is used as the detection object as a whole, abandoning the traditional practice of using tiny individuals as detection objects, avoiding the introduction of complex feature fusion networks, and only using the existing network model to extract and fuse the features of multi-frame images to obtain better small target detection performance. In addition, the present invention takes into account the difference in morphological stability between the target and the false alarm, excavates the change information contained in the multi-frame images, and determines whether the target morphology has changed by calculating the similarity of the target in the first and last frames, thereby eliminating the false alarms that have changed and reducing the false alarm rate.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image ship target detection, and in particular relates to a ship detection method that integrates the spatiotemporal information of synchronous orbit sequence optical images. Background Art

[0002] Remote sensing target detection is an important field of remote sensing data application today. Synchronous orbit optical satellites can achieve continuous observation of the same location at a high imaging frequency and obtain continuous frame images. However, the orbital altitude of synchronous orbit satellites is high and the imaging distance is far. Its imaging resolution is generally low, and ship target detection is difficult. The purpose of the present invention is to effectively utilize the complementary information and difference information contained in multiple frame images to improve the detection accuracy of moving ship targets in low-resolution sequence remote sensing images.

[0003] In terms of preprocessing, when using continuous frame images for moving target detection, the frame difference method is a commonly used background suppression algorithm. Geostationary orbit satellites are relatively stationary with the earth, and the areas except for moving targets and some interference in continuous frame images remain constant. By making a difference between adjacent frames, most of the static background can be removed. This method can improve the distinguishability of targets in the image and lay the foundation for subsequent target detection algorithms.

[0004] In terms of target feature extraction from multiple frames, in order to utilize the target information in multiple frames, there is a method to input multiple images into a neural network at the same time to extract features, and add a self-attention model to the feature extraction network. The self-attention model can measure the similarity of different features and weightedly fuse them to obtain enhanced and more discriminative features. By associating the same targets in different frames and fusing their features, the available information is further increased and the target detection accuracy is improved.

[0005] It can be seen that the existing methods all take independent individuals at different times as detection objects. Under this constraint, no matter what kind of preprocessing or target enhancement algorithm is used, the convolutional network only cares about local individual information and cannot aggregate information from different times without increasing the parameter scale. Although the self-attention-based mechanism can fuse the features of the same target at different times to a certain extent, this method brings huge computational burden and time consumption. In addition, whether it is a convolutional neural network or a fusion algorithm implemented by the self-attention mechanism, the result is to generate a union of multiple frames of information, so it cannot reflect the target's temporal change information, which is crucial for false alarm removal. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a ship detection method that integrates the spatiotemporal information of synchronous orbit sequence optical images, which can better realize the detection of ship targets in low-resolution sequence images.

[0007] A ship detection method integrating the temporal and spatial information of synchronous orbit sequence optical images comprises the following steps:

[0008] S1: Using the inter-frame difference algorithm to remove the background information in N frames of remote sensing images, obtain N-1 frames of frame difference images, where N is at least 3;

[0009] S2: Integrate N-1 frame difference images into a frame of synthetic image through a pixel-level fusion algorithm, wherein the synthetic image contains a suspected ship sequence consisting of the same target at different times;

[0010] S3: Detect the synthetic image using the trained ship sequence detection model to obtain location information of the suspected ship sequence;

[0011] S4: Based on the position information of the suspected ship sequence, the bounding boxes of the ships constituting the sequence in each frame of the remote sensing image are parsed through threshold segmentation and bounding rectangle algorithm;

[0012] S5: According to the bounding box determined in step S4, corresponding slices are extracted from the 1st frame and the Nth frame of the remote sensing image, and features of the two slices are extracted using a ResNet18 network to obtain corresponding feature vectors;

[0013] S6: The cosine similarity algorithm is used to calculate the cosine similarity between the two feature vectors, and it is determined whether the cosine similarity is greater than a set threshold. If so, the target contained in the suspected ship sequence is a ship, otherwise it is a false alarm target.

[0014] Furthermore, the training method of the ship sequence detection model is:

[0015] Obtain M groups of remote sensing images as training samples, and each group contains N consecutive frames of remote sensing images, where M is at least 100;

[0016] The background information in each group of remote sensing images is removed by using the inter-frame difference algorithm to obtain the frame difference images corresponding to each group of remote sensing images;

[0017] Then, each set of frame difference images is integrated into a frame of synthetic image through pixel-level fusion algorithm;

[0018] Use image annotation software to annotate the entire circumscribed rectangular frame of the ship sequence in each composite image;

[0019] With each synthetic image as input and the circumscribed rectangular box corresponding to the ship sequence in each synthetic image as output, the YOLO detection model, the Faster-RCNN detection model or the SSD detection model is trained to obtain a ship sequence detection model.

[0020] Furthermore, the cosine similarity (F1, F N ) is calculated as follows:

[0021]

[0022] Among them, F1 is the feature vector corresponding to the slice in the first frame of remote sensing image, F N is the feature vector corresponding to the slice in the Nth frame of remote sensing image.

[0023] Furthermore, the threshold value set in step S6 is 0.5.

[0024] Furthermore, the background information in the N frames of remote sensing images is removed by using an inter-frame difference algorithm. Specifically, the background information is suppressed by making a difference between adjacent frames, and the target information is retained.

[0025] Furthermore, the method for obtaining the composite image is:

[0026] Superimpose N-1 frame difference images to obtain the initial superimposed image s;

[0027] The initial superimposed image s is stretched to obtain a composite image as follows:

[0028]

[0029] Among them, Max s is the maximum value of pixels in the initial superposition image s, Min s is the minimum pixel value in the initial superposition image s.

[0030] Beneficial effects:

[0031] 1. The present invention provides a ship detection method that integrates the spatiotemporal information of synchronous orbit sequence optical images. In model training and application, the ship target sequence in the synthetic image is regarded as a whole, and the whole is set as the detection object. Based on this, the present invention can use a basic feature extraction network to directly extract target features from multiple frames of images without increasing the complexity of the neural network, and use it as a basis for determining the target. In other words, the present invention abandons the traditional practice of taking tiny individuals as detection objects, directly avoids designing a complex feature fusion network, and only uses the existing network model to directly extract feature information from multiple frames of images, thereby enabling the target detection algorithm for low-resolution sequence images to obtain better detection performance.

[0032] 2. The present invention provides a ship detection method that integrates the spatiotemporal information of synchronous orbit sequence optical images. The ship target is a rigid body whose shape does not change with time, which serves as an important basis for distinguishing a moving ship from broken clouds whose shape changes with time. Specifically, the present invention uses the morphological stability of the target as a target determination factor, and uses the appearance similarity of the target in the first frame and the last frame of N consecutive frames to quantitatively judge the trend of the target's change over time, thereby determining whether the target changes within the time window, thereby removing false targets with unstable shapes and reducing the false alarm rate of the detection algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flow chart of a ship detection method that integrates the temporal and spatial information of synchronous orbit sequence optical images provided by the present invention;

[0034] Figure 2 A comparison diagram of the isolated target and the target sequence provided by the present invention;

[0035] Figure 3 A flow chart of a single-frame target position calculation algorithm provided by the present invention;

[0036] Figure 4 This is a flow chart of the target morphology change discrimination algorithm based on feature similarity provided by the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0038] like Figure 1 As shown, the present invention provides a ship detection method that integrates the spatiotemporal information of synchronous orbit sequence optical images, comprising the following steps:

[0039] S1: Use an inter-frame difference algorithm to remove background information in N frames of remote sensing images to obtain N-1 frames of frame difference images, where N is at least 3.

[0040] It should be noted that the inter-frame difference suppresses background information by subtracting adjacent frames, retains target information, and avoids the annihilation of the target due to the upward saturation of pixel values ​​caused by the superposition of multiple frames (due to the accumulation of pixel values, the values ​​of all pixels in the image are greater than 255). i The calculation can be achieved according to formula (1):

[0041] diff i =max(I i+1 -I i ,0),i∈{1,2,3,…,N-1} (1)

[0042] S2: Integrate N-1 frame difference images into a composite image through a pixel-level fusion algorithm, wherein the composite image contains a sequence of suspected ships consisting of the same target at different times, such as Figure 2 shown.

[0043] The corresponding pixel superposition method is further used to superimpose multiple frame difference images into one image, so that it carries the information of the target at different times. The calculation method of the superimposed image s is:

[0044]

[0045] In order to avoid oversaturation caused by the addition operation resulting in pixel values ​​greater than the maximum value 255 of the 8-bit image, the initial superimposed image s is stretched to the maximum and minimum values ​​to obtain the composite image The stretching method is as follows:

[0046]

[0047] Among them, Max s is the maximum value of pixels in the initial superposition image s, Min s is the minimum pixel value in the initial superposition image s.

[0048] S3: Detect the synthetic image using the trained ship sequence detection model to obtain position information of the suspected ship sequence.

[0049] In the model training and application, the present invention regards the ship target sequence in the superimposed image as a whole, and sets the whole as the detection object. Based on this, the present invention can directly extract the target features from multiple frames of images using a basic feature extraction network without increasing the complexity of the neural network, and use it as the basis for determining the target.

[0050] During training, image annotation software can be used to annotate the bounding rectangles of the target sequences as a whole, rather than the bounding rectangles of each individual target. A training sample set is constructed in the above manner, and the training set is used to train the YOLO detection model, and finally a ship sequence detection model for low-resolution sequence remote sensing images is obtained, which specifically includes the following steps:

[0051] M groups of remote sensing images are obtained as training samples, and each group contains N consecutive frames of remote sensing images, where M is at least 100; the background information in each group of remote sensing images is removed by using an inter-frame difference algorithm to obtain a frame difference map corresponding to each group of remote sensing images; each group of frame difference maps is then integrated into a frame of synthetic image by using a pixel-level fusion algorithm; an image annotation software is used to annotate the entire circumscribed rectangular frame of the ship sequence in each synthetic image; with each synthetic image as input and the circumscribed rectangular frame corresponding to the ship sequence in each synthetic image as output, a YOLO detection model, a Faster-RCNN detection model or an SSD detection model is trained to obtain a ship sequence detection model.

[0052] It should be noted that in terms of detection algorithms, the single-stage detection algorithm YOLO (you only look once) has become the mainstream target detection method based on deep neural networks due to its characteristics of balancing efficiency and accuracy. YOLO solves object detection as a regression problem, and based on a separate end-to-end network, completes the output from the input of the original image to the position and category of the object. In the YOLO algorithm, the input image is first divided into S*S grids. If the center point of the target falls into a certain grid, then the grid is responsible for predicting the target. According to the characteristics of the convolutional neural network, the grid corresponds to the B*5+C parameters in the network output S*S*(B*5+C)-dimensional matrix. This group of parameters represents the category and location attributes of the target, where B represents the number of detection boxes in each grid, and C is the number of categories.

[0053] The training of the YOLO model depends on training samples. Given the location and category information of the target, the grid to which the target center point belongs is calculated, and then the intersection and union ratio of the 9 pre-set pre-selected boxes and the target annotation box is calculated one by one. The area in the prior box with the largest intersection and union ratio is taken as the positive sample. If the intersection and union ratio of a prior box and the circumscribed rectangular box of all targets in the image is less than the threshold of 0.5, it is considered that the pre-selected box does not contain the target and is recorded as a negative sample. After obtaining a large number of learning samples, the network is trained under the constraints of the loss function so that it can extract target features and distinguish targets from interference.

[0054] S4: Based on the position information of the suspected ship sequence, the bounding boxes of the ships constituting the sequence in each frame of the remote sensing image are parsed through threshold segmentation and bounding rectangle algorithm.

[0055] That is to say, after the training of the ship sequence detection model is completed, in the subsequent target detection application, the sequence images are firstly superimposed and processed according to the method of the present invention and then sent to the ship sequence detection model. The ship sequence detection model selects the ship targets that meet the sequence characteristics according to the learned target characteristics. At the same time, since the result obtained by each detection is the whole object rather than the individual, it is necessary to intercept the target slice in the single frame image according to the position information of the sequence, and use the threshold segmentation, connected domain extraction and bounding rectangle algorithm to determine the bounding box of the ship in each frame of remote sensing image. The algorithm flow is shown in the attached figure. Figure 3 shown.

[0056] It should be noted that in the continuous frame images obtained by the continuous imaging of the earth by the geostationary satellite, since the ship is a rigid body, its color and shape will not change significantly over time within a certain time window; while the broken clouds that are prone to cause false alarms are not rigid bodies, and their shapes are controlled by natural conditions such as wind, temperature and humidity. As time goes by, their shapes will change greatly. Therefore, the present invention uses it as an important basis for distinguishing between moving ships and broken clouds, and uses this feature to design a false alarm removal algorithm based on feature similarity, which uses the appearance similarity of the target in the first and last frames in N consecutive frames to determine whether the target has changed, so as to remove false targets with unstable shapes. For details, refer to steps S5 to S6.

[0057] S5: According to the bounding box determined in step S4, corresponding slices are extracted from the 1st frame and the Nth frame of the remote sensing image, and the ResNet18 network is used to extract features of the two slices to obtain corresponding feature vectors.

[0058] S6: The cosine similarity algorithm is used to calculate the cosine similarity between the two feature vectors, and it is determined whether the cosine similarity is greater than a set threshold. If so, the target contained in the suspected ship sequence is a ship, otherwise it is a false alarm target.

[0059] That is to say, the present invention utilizes the target morphology stability to remove the false alarms caused by unstable morphology, and the measurement method is the feature similarity of the target in the first frame and the Nth frame; specifically, Figure 4 As shown in the figure, according to the bounding box of the target in the single-frame image, the target slices are intercepted in the first frame and the Nth frame respectively, and the ResNet18 network is used to extract the features, and then the cosine similarity algorithm is used to judge the similarity of the target features, and the cosine similarity between the features is calculated. The calculation method is as follows:

[0060]

[0061] Among them, F1 is the feature vector corresponding to the slice in the first frame of remote sensing image, F Nis the feature vector corresponding to the slice in the Nth frame of remote sensing image. When the similarity is greater than the threshold t, it indicates that the target shape has not changed significantly over time, and the target can be confirmed to be a ship. Otherwise, it indicates that the target is not stable in time series and is judged as a false alarm.

[0062]

[0063] It can be seen that the present invention proposes a target detection and false alarm suppression algorithm based on the temporal information of a comprehensive sequence of low-resolution remote sensing images; first, the inter-frame difference algorithm is used to remove the main background information, and the information in N frames of images is integrated into one frame of image through a pixel-level fusion algorithm, and a ship sequence consisting of the same target at different times will appear in the synthetic image; in view of this feature, the present invention sets these sequences with larger size and richer information as a whole as the detection target, and on the basis of the overall detection results, extracts the position information of a single target through a morphological method to form a new training and detection paradigm; finally, the present invention further uses a feature similarity discrimination algorithm to quantitatively determine the target stability in the time dimension, removes false targets that undergo morphological changes within the time window, that is, removes unstable interference targets, and makes full use of the inter-frame change information to reduce the false alarm rate of detection of moving ship targets.

[0064] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A ship detection method integrating the temporal and spatial information of synchronous orbit sequence optical images, characterized in that: The following steps are involved: S1: Using the inter-frame difference algorithm to remove the background information in N frames of remote sensing images, obtain N-1 frames of frame difference images, where N is at least 3; S2: Integrate N-1 frame difference images into a frame of synthetic image through a pixel-level fusion algorithm, wherein the synthetic image contains a suspected ship sequence consisting of the same target at different times; S3: Detect the synthetic image using the trained ship sequence detection model to obtain location information of the suspected ship sequence; S4: Based on the position information of the suspected ship sequence, the bounding boxes of the ships constituting the sequence in each frame of the remote sensing image are parsed through threshold segmentation and bounding rectangle algorithm; S5: According to the bounding box determined in step S4, corresponding slices are extracted from the 1st frame and the Nth frame of the remote sensing image, and features of the two slices are extracted using a ResNet18 network to obtain corresponding feature vectors; S6: The cosine similarity algorithm is used to calculate the cosine similarity between the two feature vectors, and it is determined whether the cosine similarity is greater than a set threshold. If so, the target contained in the suspected ship sequence is a ship, otherwise it is a false alarm target.

2. A ship detection method based on the temporal and spatial information of synchronous orbit sequence optical images as claimed in claim 1, characterized in that: The training method of the ship sequence detection model is: Obtain M groups of remote sensing images as training samples, and each group contains N consecutive frames of remote sensing images, where M is at least 100; The background information in each group of remote sensing images is removed by using the inter-frame difference algorithm to obtain the frame difference images corresponding to each group of remote sensing images; Then, each set of frame difference images is integrated into a frame of synthetic image through pixel-level fusion algorithm; Use image annotation software to annotate the entire circumscribed rectangular frame of the ship sequence in each composite image; With each synthetic image as input and the circumscribed rectangular box corresponding to the ship sequence in each synthetic image as output, the YOLO detection model, the Faster-RCNN detection model or the SSD detection model is trained to obtain a ship sequence detection model.

3. The ship detection method of integrating the spatiotemporal information of synchronous orbit sequence optical images according to claim 1, characterized in that: The cosine similarity between two feature vectors (F1, F N ) is calculated as follows: Among them, F1 is the feature vector corresponding to the slice in the first frame of remote sensing image, F N is the feature vector corresponding to the slice in the Nth frame of remote sensing image.

4. The ship detection method of integrating the spatiotemporal information of synchronous orbit sequence optical images as claimed in claim 1, characterized in that: The threshold value set in step S6 is 0.

5.

5. The ship detection method of integrating the temporal and spatial information of synchronous orbit sequence optical images as claimed in claim 1, characterized in that: The inter-frame difference algorithm is used to remove the background information in N frames of remote sensing images. Specifically, the background information is suppressed by making a difference between adjacent frames and the target information is retained.

6. A ship detection method based on the temporal and spatial information of synchronous orbit sequence optical images as claimed in claim 1, characterized in that: The method for obtaining the composite image is: Superimpose N-1 frame difference images to obtain the initial superimposed image s; The initial superimposed image s is stretched to obtain a composite image as follows: Among them, Max s is the maximum value of pixels in the initial superposition image s, Min s is the minimum pixel value in the initial superposition image s.

Images