A method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images

Abstract

The present invention discloses a method for detecting and tracking targets at sea using visible light sequence images of synchronous orbit satellites. The present invention first groups and synthesizes multiple frames of images, so as to facilitate the detection of candidate targets using the motion information of the targets, thereby improving the detection rate and reducing the false alarm rate; then, after the input sequence images are divided into blocks, adaptive quantization preprocessing is performed to reduce the influence of brightness on target detection; then, a convolutional neural network is used to train and load a detection model for the target, which can adapt to target detection in complex scenes; the trained detection model is used to perform target detection on a single set of images, and target association is performed on the detection results on multiple sets of images based on the intersection-and-union ratio. The present method automatically learns good features that can describe the target in the data in a data-driven manner, and the computational complexity changes linearly with the number of detected targets, with high computational efficiency, and can meet timeliness requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image target detection and perception, and in particular to a method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images. Background Art

[0002] The detection and tracking of maritime targets is an important task in ocean surveillance. With the development of satellite technology, earth observation methods such as synthetic aperture radar and optical satellites are playing an increasingly important role in maritime target surveillance. Low-orbit satellite images usually have a high spatial resolution, and a lot of work has been done on the detection and recognition of maritime targets in low-orbit satellite images. However, low-orbit satellites have a small single observation range and a long revisit cycle, making it difficult to continuously observe the same area in a short period of time. Compared with low-orbit satellites, the spatial resolution of images acquired by geosynchronous orbit satellites is lower, but they have the advantages of a large observation range, a short revisit cycle, and a fast response. They can obtain sequential images of the same observation point, thereby achieving near-real-time continuous monitoring of a large area, and play an important role in the dynamic monitoring of maritime targets.

[0003] In recent years, there are two main difficulties in monitoring maritime targets based on geosynchronous orbit optical satellite images. First, the low spatial resolution of the images makes the target features unclear, which can easily lead to missed detections. Second, clouds will cause great interference to detection. The target and some broken clouds are very similar in shape, which can easily lead to false alarms.

[0004] "Zhang Zhixin, Xu Qingjun, Zhang Chuan, Zhao Dong. Ship motion feature detection based on single-view Gaofen-4 satellite multispectral image [J]. Remote Sensing Technology and Applications, 2019, 34(4): 892-900." The study applies the detection method based on single-frame images to each frame of the sequence image to improve the detection rate and calculate the target's heading and speed.

[0005] "Jiao Tengzhang, Hu Yuxin, Lv Peng, Zhang Kai, Tai Xianqing. A method for on-orbit marine multi-moving ship target detection and tracking [J]. Journal of University of Chinese Academy of Sciences, 2020, 37(3): 368-378." The implementation process of the method is as follows: First, the edge detection operator is used to extract candidate targets on a single frame image; then, the area, aspect ratio and other features of the candidate targets are extracted, and the classifier is constructed using the features to remove false alarms such as clouds and islands; finally, the detection results of multiple frames of images are associated using the joint probabilistic data association method to obtain the trajectory of the target.

[0006] The implementation process of the method in “L.Yao,Y.Liu,and Y.He.A Novel Ship-Tracking Method for GF-4SatelliteSequential Images[J].Sensors,2018,18:1-14.” is as follows: First, a saliency detection method based on statistical analysis is used to extract candidate targets, and false alarms are removed according to the size of the candidate targets; then, a rational polynomial coefficient model is used to calculate the geographic location of the target; finally, the detection results of multiple frames of images are associated through a multi-hypothesis tracking method to obtain the trajectory of the target.

[0007] The technical route of the existing methods can be summarized as follows: first, based on the grayscale difference between the target and the sea background in a single frame image, image processing methods such as threshold segmentation and edge extraction or statistical analysis-based methods such as constant false alarm detection and saliency detection are used to extract candidate targets. Then, false alarms are removed based on the geometric features of the candidate targets such as size, area, aspect ratio, etc., and then joint probabilistic data association or multi-hypothesis tracking methods are used to associate the detection results of multiple frames to obtain the target trajectory. The method flow chart is shown in the figure. Figure 1 shown.

[0008] The technical defects of the existing methods are mainly reflected in the following four aspects: 1) The existing methods do not perform preprocessing operations such as quantization of the brightness of the input image, which may easily lead to missed detection when the brightness of the target on the image is low; 2) The existing methods only detect candidate targets on a single frame image, and the detection effect on weak targets is poor, which may easily lead to missed detection or false detection; 3) The existing methods use conventional image processing or statistical analysis methods such as threshold segmentation, edge extraction, and statistical calculation, and use artificially designed features for target detection. It is difficult to accurately detect targets in complex scenes with more broken clouds; 4) The existing methods use target association based on probability models or multi-hypothesis tracking algorithms. The computational complexity varies exponentially with the number of detected targets, the computational complexity is high, and it is difficult to meet timeliness requirements. Summary of the invention

[0009] In view of this, the present invention provides a method for detecting and tracking marine targets in visible light sequence images of synchronous orbit satellites. The method performs adaptive quantization processing on the input images and utilizes the motion information of the targets. The method can improve the detection rate and reduce the false alarm rate. The method automatically learns good features that can describe the targets in the data in a data-driven manner. The computational complexity varies linearly with the number of detected targets. The method has high computational efficiency and can meet the timeliness requirements.

[0010] A method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images, comprising:

[0011] Step 1: Read the sequence images in sequence and group them; unify the length and width of the images in the group and splice them to form the image tensor Y i ;

[0012] Step 2: Divide the image into blocks and unify the brightness within the image blocks. i Split into image block tensor y by block i ;

[0013] Step 3: construct a neural network as a marine target detection model, where the input of the neural network is the image block tensor y of each group i ; Use the sample set to train the marine target detection model to obtain a trained marine target detection model;

[0014] Step 4, inputting the image sequences to be detected into the trained marine target detection model in groups to obtain target detection results for each group;

[0015] Step 5: Based on the intersection-and-union ratio, target association is performed on the detection results of each group of images in turn to obtain the target trajectory;

[0016] Step 6: Among the obtained target trajectories, the target trajectory containing the number of targets greater than or equal to N / 2 is the final target detection and tracking result, where N is the total number of frames of the image in the image sequence to be detected.

[0017] Furthermore, a linear stretching method, a histogram equalization method or pixel statistics are used to quantify the brightness of a unified image block.

[0018] Furthermore, the specific method of using pixel statistics to quantify the brightness of a unified image block is:

[0019] Calculate the minimum value z of the pixels in the image block min , maximum value z max , mean μ and standard deviation σ;

[0020] If σ<10, the image block is quantized according to formula (1):

[0021] x=min((zz min )×5,255) (1)

[0022] Among them, x represents the quantized pixel value, and z is the original image pixel;

[0023] Otherwise, the image block is quantized according to formula (2):

[0024]

[0025] Among them, T b =min(T a ,zmax ), T a =μ+rσ;

[0026] r is the standard deviation gain, and high values ​​of r up ∈[4.5,5.5], the low value of r down ∈[2.5,3.5]; if S a / S b >τ s , then r=r up , otherwise r=r down , where the pixel ratio threshold τ s ∈[0.01,0.03], S a is the number of pixels in the image block whose pixel value is greater than μ+3σ, S b is the total number of pixels in the image block.

[0027] Furthermore, in step 1, for each group of images, the minimum height h of the M frames contained therein is calculated. min and minimum width w min , the height and width of the M frame image are uniformly cut to h min and w min .

[0028] Furthermore, the specific method of target association based on intersection-over-union ratio in step 5 is:

[0029] For the image tensor Y i , when i=1, each detected target All of them are taken as a target trajectory to obtain the current target trajectory, and the number of current target trajectories T = K;

[0030] When i ≥ 2, assume that there are T current target trajectories, and calculate the last target R of the tth target trajectory. t =[x t ,y t ,h t ,w t ] T Newly obtained unassociated detected targets in this group Perform intersection-and-union ratios in sequence. If the maximum value of the intersection-and-union ratio is greater than the threshold τ, the detected target corresponding to the maximum intersection-and-union ratio is associated with the tth target track, and the associated detected targets are recorded. association state; sequentially associate the T current target trajectories obtained with the newly obtained unassociated detected targets in this group; all associated target trajectories and all unassociated targets are taken as the obtained current target trajectories.

[0031] Furthermore, record the detected target The method of associating the state is as follows: During the target association process, a vector c=[c1,...,c K ] T , detect the target Not yet associated, marked c k =0; If it is already associated, mark c k =1.

[0032] Furthermore, the neural network in step 3 is a deep convolutional neural network.

[0033] Furthermore, the deep convolutional neural network adopts Faster RCNN, SSD or R3Det.

[0034] Furthermore, in step 5, the detection results of multiple groups of images are associated using the target center point distance.

[0035] Beneficial effects:

[0036] a. To address the problem of missed detection when the brightness of the target on the image is low, the present invention performs adaptive quantization processing on the input image to brighten the too dark targets and dim the too bright targets, thereby reducing the influence of brightness on target detection;

[0037] b. In view of the problem that the detection effect of small and weak targets is poor and missed detection or wrong detection is easy to occur when detecting on a single frame image, the present invention detects candidate targets on a composite image composed of multiple frames of images. Since the size of the target is increased on the composite image, the problem that small and weak targets are difficult to detect can be solved. At the same time, since the motion information of the target is used, the detection rate can be improved and the false alarm rate can be reduced;

[0038] c. Aiming at the problem that it is difficult to accurately detect targets in complex scenes with more broken clouds based on artificially designed features, the present invention uses a convolutional neural network to automatically learn good features that can describe targets in the data in a data-driven manner, which can adapt to target detection in complex scenes and effectively reduce the false alarm rate;

[0039] d. In order to solve the problem that the existing target association model has high computational complexity and is difficult to meet timeliness requirements, the present invention utilizes the characteristic that the same target in adjacent synthetic images has a large overlap and performs target association based on the intersection-and-union ratio, so that the computational complexity changes linearly with the number of detected targets, the operation efficiency is high, and the timeliness requirements can be met. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of the existing method.

[0041] Figure 2 Flow chart of the method of the present invention.

[0042] Figure 3 This is the first group of image detection results of step 4 in the embodiment.

[0043] Figure 4 This is the second group of image detection results in step 4 in the embodiment.

[0044] Figure 5 This is the third group of image detection results of step 4 in the embodiment.

[0045] Figure 6 This is the fourth group of image detection results of step 4 in the embodiment.

[0046] Figure 7 This is the fifth group of image detection results in step 4 of the embodiment.

[0047] Figure 8 This is the detection result of the sixth group of images in step 4 of the embodiment.

[0048] Fig. 9 This is the final target detection and tracking result of step 5 in the embodiment.

[0049] The circular box represents the detected target in the current image, the rectangular box represents the detected target in the historical image, and the arrow represents the target trajectory. DETAILED DESCRIPTION

[0050] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0051] The present invention provides a method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images, comprising the following steps:

[0052] Step 1: read the sequence of N frames in sequence and group them; unify the length and width of the images in the group and splice them to form the image tensor Y i ; First, group N = 8 frames of images into groups of M = 3 adjacent frames, and obtain N-M+1 = 6 groups of images, among which the i-th group of images X i The present embodiment groups the 8 frames of images into groups of 3 adjacent frames to obtain 6 groups of composite images X. i , i = 1, ..., 6, where the i-th combination forms image X i It consists of the i-th to i+2-th frame images, that is:

[0053] The first group of composite images X1 consists of the first to third frames of images.

[0054] The second group of composite images X2 consists of the 2nd to 4th frame images.

[0055] And so on.

[0056] For each group of images, calculate the minimum height h among the M frames it contains min and width w min , M = 3, the height and width of the three frames are unified to h by cutting off the redundant pixels min and w min .

[0057] The three frames of images after the size is unified are spliced ​​along the channel direction to form a size h min × min ×M image tensor Y i . Transform the image tensor Y i Cut the image into blocks of size p×p×M along the row and column directions. i In an embodiment, h min 1600,w min is 3200. The three frames of images after the size is unified are spliced ​​along the channel direction to form the image tensor Y i , the dimensions are 1600×3200×3.

[0058] The image tensor Y i Cropping along the row and column directions, the resulting image block tensor y i , the size is 320×320×3.

[0059] Step 2: Divide the image into blocks and unify the brightness within the image blocks. i Split into image block tensor y by block i ;

[0060] The image is divided into blocks to form image blocks of size p×p. In this embodiment, N=8, and the image block size p is set to 320.

[0061] Perform adaptive quantization processing on each image block, use the statistics of the image block to calculate the threshold, and calculate the minimum value z of the pixels in the image block. min , maximum value z max , mean μ and standard deviation σ;

[0062] If σ<10, the image block is quantized according to formula (1):

[0063] x=min((zz min )×5,255) (1)

[0064] Among them, x represents the quantized pixel value;

[0065] Otherwise, the quantized pixel value is calculated according to formula (2) to quantize the image block:

[0066]

[0067] Among them, T b =min(T a ,z max ), T a =μ+rσ;

[0068] Calculate the number of pixels S in the image block whose pixel value is greater than μ+3σ a , the total number of pixels in the image block S b ;

[0069] If S a / S b >τ s When r=r up , otherwise r=r down . Where r is the standard deviation gain. In this embodiment, the relevant parameters of the adaptive quantization process are set as follows: Let the pixel ratio threshold τ s =0.02, standard deviation gain high value r up =5, standard deviation gain low value r down =3.

[0070] Each image block is adaptively processed according to the above method.

[0071] Step 3: Build a neural network as the target detection model. Set the size of the neural network input layer to 320×320×3, and the input of the neural network is y i ,y i It is a tensor formed by splicing three adjacent frames of images along the channel direction. Then, the constructed neural network is trained using the training sample set to obtain a trained detection model.

[0072] Step 4: Group the image sequence to be detected in the same way as step 1, and obtain the image tensor Y for each group i , the image block tensor y it contains i Input into the trained detection model; for each group of image tensors Y i Perform target detection and obtain the detection result R i .

[0073] Assume R i There are K detected targets in total, and the kth detected target is recorded as in Respectively represent the horizontal and vertical coordinates, height, and width of the center point of the target, k∈N. Process the image tensors Y1 to Y N-M+1 .

[0074] Step 5: Based on the intersection-over-union ratio, target association is performed on the detection results of multiple groups of images.

[0075] like Figure 3-9 As shown, the circular box represents the detected target in the current image, the rectangular box represents the detected target, and the arrow represents the target trajectory.

[0076] In an embodiment, the intersection-over-union ratio threshold τ is set to 0.1.

[0077] For the image tensor Y1, the image patch tensor y1 it contains is input into the detection model f θ (·), the detection result R1 is obtained. The image detection result R1 is as follows Figure 3 As shown, the detection results on the current group of images are: and The number of targets detected in the current image is K=3. These three targets correspond to a target track respectively, namely, target tracks 1, 2 and 3. Therefore, the current number of target tracks is T=3.

[0078] For the image tensor Y2, the image block tensor y2 it contains is input into the detection model f θ (·), the test result R2 is obtained. The test result R2 is as follows Figure 4 As shown in the figure, two targets are detected in the current group of images, namely: and Therefore, the number of targets detected in the current image is K = 2. Associated with target track 2; and the detected target Not associated with an existing target track, so the target will be detected As a new target track, namely, target track 4. After the target association is performed on the first two groups of images, there are 4 target tracks in total, namely, the current number of target tracks T=4.

[0079] For the image tensor Y3, the image block tensor y3 it contains is input into the detection model f θ (·), the test result R3 is obtained. The test result R3 is as follows Figure 5 As shown, a total of 3 targets are detected in the current group of images, namely: and Therefore, the number of targets detected in the current image is K = 3. and They are associated with target tracks 2 and 4 respectively, and the detected target Not associated with an existing target track, so the target will be detected As a new target track, namely target track 5. After target association is performed on the first three groups of images, there are 5 target tracks in total, namely T is 5.

[0080] For the image tensor Y4, the image block tensor y4 it contains is input into the detection model fθ (·), the test result R4 is obtained. The test result R4 is as follows Figure 6 As shown in the figure, two targets are detected in the current group of images, namely: and Therefore, the number of targets detected in the current image is K, which is 2. and They are associated with target tracks 4 and 5 respectively. After target association is performed on the first four groups of images, there are 5 target tracks in total, that is, T is 5.

[0081] For the image tensor Y5, the image block tensor y5 it contains is input into the detection model f θ (·), the test result R5 is obtained. The test result R5 is as follows Figure 7 As shown, a total of 3 targets are detected in the current group of images, namely: and Therefore, the number of targets detected in the current image is K, which is 3. and They are associated with target tracks 2 and 4 respectively, and the target track is detected Not associated with the existing target track, so the newly detected target As a new target track, namely, target track 6. After target association is performed on the first five groups of images, there are 6 target tracks in total, namely, T is 6.

[0082] For the image tensor Y6, the image block tensor y6 it contains is input into the detection model f θ (·), the detection result R6 is obtained. The detection result of the sixth group of images is as follows Figure 8 As shown in the figure, two targets are detected in the current group of images, namely: and Therefore, the number of targets detected in the current image is K, which is 2. and They are associated with target tracks 2 and 4 respectively. After target association of the six groups of images, there are 6 target tracks in total, that is, T is 6.

[0083] Step 5: Generate and output the target track.

[0084] After completing the target detection and association of all images in the sequence, a set of target tracks is obtained, and the target tracks containing the number of targets greater than or equal to N / 2 are taken as the final target detection and tracking results and output. In this embodiment, N is 8, so only the target tracks containing the number of targets greater than or equal to 4 are retained as the final target detection and tracking results. The number of targets contained in target tracks 1 to 6 is 1, 5, 1, 5, 2, and 1 respectively, so only target tracks 2 and 4 are retained. The final target detection and tracking results are shown in Figure 1. Fig. 9Target tracks 2 and 4 correspond to two ship targets, and the remaining target tracks are false alarms caused by clouds. The present invention correctly detects the ship targets and removes the false alarms.

[0085] The effect of the scheme of the present invention was verified using 10 groups of measured sequences, with an average image size of 10000×10000. Table 1 shows the detection rate, false alarm rate and running time of the scheme of the present invention on each test sequence. The average detection rate of the scheme of the present invention is 92%, the false alarm rate is 30%, and the running time is 8s. The average detection rate of the existing method on the same 10 groups of measured sequences is 70%, the false alarm rate is 40%, and the running time is 125s. The scheme of the present invention is significantly better than the existing methods in terms of detection rate and false alarm rate, which proves the effectiveness of the scheme of the present invention in generating synthetic images using multiple frames of images to detect targets in sequence images; in terms of timeliness, the scheme of the present invention is 15.6 times better than the existing method based on multi-hypothesis tracking, which shows the effectiveness of the target association based on intersection-over-union ratio in the scheme of the present invention.

[0086] Table 1 Detection results and running time of the scheme of the present invention on 10 groups of measured sequences

[0087]

[0088]

[0089] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images, characterized in that: include: Step 1: read N frames of sequential images in sequence, and group them into groups of M adjacent frames to obtain N-M+1 groups of images X1,…,X i …,X N-M+1 , where X i It consists of the i-th to i+M-1-th frame images; unify the length and width of the images in the group and splice them to form the image tensor Y i ; Step 2: Divide the image into blocks and unify the brightness within the image blocks. i Split into image block tensor y by block i ; Step 3: construct a neural network as a marine target detection model, where the input of the neural network is the image block tensor y of each group i ; Use the sample set to train the marine target detection model to obtain a trained marine target detection model; Step 4, inputting the image sequences to be detected into the trained marine target detection model in groups to obtain target detection results for each group; Step 5: Based on the intersection-and-union ratio, target association is performed on the detection results of each group of images in turn to obtain the target trajectory; Step 6: Among the obtained target trajectories, the target trajectory containing the number of targets greater than or equal to N / 2 is the final target detection and tracking result, where N is the total number of frames of the image in the image sequence to be detected.

2. The method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images as claimed in claim 1, characterized in that: The brightness of a unified image block is quantified by using linear stretching, histogram equalization or pixel statistics.

3. The method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images as claimed in claim 2, characterized in that: The specific method of using pixel statistics to quantify the brightness of a unified image block is: Calculate the minimum value z of the pixels in the image block min , maximum value z max , mean μ and standard deviation σ; If σ<10, the image block is quantized according to formula (1): x=min((zz min )×5,255) (1) Among them, x represents the quantized pixel value, and z is the original image pixel; Otherwise, the image block is quantized according to formula (2): Among them, T b =min(T a ,z max ),T a =μ+rσ; r is the standard deviation gain, and high values ​​of r up ∈[4.5,5.5], the low value of r down ∈[2.5,3.5]; if S a / S b >τ s , then r=r up , otherwise r=r down , where the pixel ratio threshold τ s ∈[0.01,0.03], S a is the number of pixels in the image block whose pixel value is greater than μ+3σ, S b is the total number of pixels in the image block.

4. The method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images as claimed in claim 1, characterized in that: In step 1, for each group of images, the minimum height h of the M frames contained therein is calculated. min and minimum width w min , the height and width of the M frame image are uniformly cut to h min and w min .

5. The method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images as claimed in claim 1, characterized in that: The specific method of target association based on intersection-over-union ratio in step 5 is: For the image tensor Y i , when i=1, each detected target All of them are taken as a target trajectory to obtain the current target trajectory, and the number of current target trajectories T = K; When i ≥ 2, assume that there are T current target trajectories, and calculate the last target R of the tth target trajectory. t =[x t ,y t ,h t ,w t ] T Newly obtained unassociated detected targets in this group Perform intersection-and-union ratios in sequence. If the maximum value of the intersection-and-union ratio is greater than the threshold τ, the detected target corresponding to the maximum intersection-and-union ratio is associated with the tth target track, and the associated detected targets are recorded. association state; sequentially associate the T current target trajectories obtained with the newly obtained unassociated detected targets in this group; all associated target trajectories and all unassociated targets are taken as the obtained current target trajectories.

6. The method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images as claimed in claim 5, characterized in that: Recording Checkout Targets The method of associating the state is as follows: During the target association process, a vector c=[c1,...,c K ] T , detect the target Not yet associated, marked c k =0; If it is already associated, mark c k =1.

7. The method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images as claimed in claim 1, characterized in that: The neural network in step 3 is a deep convolutional neural network.

8. The method for detecting and tracking marine targets using synchronous orbit satellite visible light sequence images as claimed in claim 7, characterized in that: The deep convolutional neural network adopts Faster RCNN, SSD or R3Det.

Images