A method for extracting small rigid targets in uniform linear motion

Abstract

The present invention discloses a method for extracting a rigid small target with uniform linear motion, which belongs to the field of motion target extraction. In view of the problem that the extraction results of the prior art are inaccurate and the temporal and spatial consistency is poor when extracting moving targets, the present method first performs sub-pixel alignment on the foreground image according to the motion parameters to solve a consistent shape; secondly, the motion estimation of the foreground on different frames is performed using a related method to obtain a more accurate motion estimation result; finally, the motion estimation based on uniform motion and the model based on rigid body are combined to realize the extraction task of a rigid small target with uniform motion. The present invention can be applied to fields such as video monitoring, and is used to extract specific rigid targets in monitoring videos.

Description

Technical Field

[0001] The invention relates to a target extraction method, in particular to a uniform linear motion rigid small target extraction method, belonging to the field of motion target extraction. Background Art

[0002] Moving object extraction refers to extracting the foreground moving object of interest from the background in a sequence of images. Generally, moving object extraction can be roughly divided into three categories: frame difference method, background subtraction-based method and optical flow-based method.

[0003] The frame difference method extracts foreground targets by measuring the pixel similarity between adjacent frames, and has the characteristics of fast speed and low computational complexity. However, the disadvantage of this type of method is that it is sensitive to illumination and extremely dependent on the color difference between the foreground and background, which is prone to artifact problems. When faced with related problems related to real scenes, such as background jitter and color camouflage, the extraction results cannot accurately reflect the shape of the object. At the same time, due to the dynamic changes of interference factors, the temporal and spatial consistency of the extraction results is not ideal. Based on the background subtraction method, various background models are used to model the background to obtain a robust background, and the current frame and the modeled background are subtracted to obtain a more stable result, and the disadvantage of "artifacts" can be overcome. Although this method reduces the influence of interference factors to a certain extent, it still cannot accurately extract targets, nor can it obtain more ideal results with temporal and spatial consistency. Based on the optical flow method, the optical flow method is used to obtain the optical flow information of the image and then obtain the motion characteristics of the objects in the image, and the different motion characteristics between the foreground object and the background object are used to distinguish the foreground and the background, which can handle backgrounds with complex motion conditions. However, this method mainly relies on the accuracy of optical flow solution. In actual application scenarios, due to the influence of lighting, color camouflage, etc., the solution of optical flow is not stable, so it is difficult to achieve robust extraction of small targets, and the spatiotemporal consistency of the shape is still not ideal. Summary of the invention

[0004] In view of the problems in the prior art of extracting moving targets, such as inaccurate extraction results and poor spatiotemporal consistency, the present invention discloses a method for extracting small rigid targets in uniform linear motion, aiming to improve the accuracy and spatiotemporal consistency of the extraction results of small rigid targets in uniform linear motion in complex scenes.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] First, the foreground image is sub-pixel aligned according to the motion parameters to solve a consistent shape, because the true shape of the rigid target always remains unchanged, but in each frame, due to the limited resolution of the imaging device and the sub-pixel motion of the moving target, the rigid target in each frame has a slight shape change. Secondly, the motion of the foreground on different frames is estimated using related methods to obtain more accurate motion estimation results; finally, the motion estimation based on uniform motion and the rigid body-based model are combined to achieve the task of extracting small rigid targets with uniform motion.

[0007] The specific implementation steps of the present invention are as follows:

[0008] Step 1: Obtain sequence images of the scene;

[0009] Step 2: Obtain background B;

[0010] Step 3: Obtain the initial motion parameters of the target;

[0011] Based on the input of step 1, extract small targets on each frame;

[0012] Based on the small target extraction results, the extraction results are processed by graphics;

[0013] Based on the small target extraction result after graphics processing, the center point position of the small target is obtained;

[0014] For the kth frame image I k , the extraction result after graphics processing A k ′, the center point of the small target is (x′ k ,y′ k );

[0015] x′ k ,y′ k Calculated by the following formula:

[0016]

[0017] Among them, the image size is M×N, i and j represent row and column coordinates respectively;

[0018] For all center point coordinates {(x′1,y′1),…,(x′ T ,y′ T )} to fit and obtain the parameter p that can describe the target motion;

[0019] For the k-th frame grayscale image I k , use the parameter p to calculate the coordinates of the center point on the kth frame (x k ,y k ), with this coordinate as the center, cut out the image block Yk (p), cut out image block B from modeled background B k (p), the coordinates of the image block are determined by the center point position (x k ,y k ) is determined by the radius, and its size is r×r, where r represents the radius.

[0020] Step 4: Foreground extraction;

[0021] For the image block, the unique corresponding foreground F and template value A are calculated by optimizing the following formula;

[0022]

[0023] Where T is the number of sequence images and D represents the total variation operator.

[0024] Step 5: motion estimation;

[0025] For the image block, its uniform motion parameter is calculated by optimizing the following formula:

[0026]

[0027] Step 6: Repeat the above steps 4 and 5 for multiple iterations until the specified conditions are met;

[0028] Step 7: Obtain extraction results based on the results of multiple iterations of steps 4 and 5;

[0029] According to the acquired motion parameter p and the frame number k of the image, the center coordinates (x k ,y k );

[0030] According to the coordinates of the center point, template A k Perform translation operation to obtain A k ′.

[0031] Beneficial effects:

[0032] 1. The present invention uses a continuous template to replace the binary template, which has a more detailed depiction of the edge part of the smaller target, better retains the edge details of the small target, and improves the accuracy of extraction;

[0033] 2. Based on high-precision estimation of moving targets, the present invention improves the temporal and spatial consistency of the extracted target shape by modeling the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a structural flow chart of a method for extracting a small rigid target in uniform linear motion according to the present invention;

[0035] Figure 2 is a grayscale image of the scene (the one in the black box is the aircraft target to be extracted);

[0036] Figure 3 The background B is reconstructed using a mixed Gaussian model;

[0037] Figure 4 It is the result of preliminary extraction of the target to obtain the initial motion parameters;

[0038] Among them: Figure a is the subtraction result of the first two frames of the three-frame difference method; b is the subtraction result of the last two frames of the three-frame difference method; c is the final result of the three-frame difference; d is the final result processed by the graphics method; e is the foreground cut out according to the motion; f is the background cut out according to the motion.

[0039] Figure 5 is the comparison of the results of our method with other methods in the field;

[0040] Among them: Figure a, Figure g, Figure m, Figure s are image blocks containing airplanes input from four adjacent frames, which are used to intuitively display the target; Figure b, Figure h, Figure n, Figure t are the extraction results of the PBAS method on four adjacent frames; Figure c, Figure i, Figure o, Figure u are the extraction results of the SuBSENSE method on four adjacent frames; Figure d, Figure j, Figure p, Figure v are the extraction results of the SGSM method on four adjacent frames; Figure e, Figure k, Figure q, Figure w are the extraction results of the FgSegNet-v2 method on four adjacent frames; Figure f, Figure l, Figure r, Figure x are the extraction results of the present invention on four adjacent frames. DETAILED DESCRIPTION

[0041] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. The technical problems solved by the technical solution of the present invention and the beneficial effects are also described. It should be noted that the described embodiments are only intended to facilitate the understanding of the present invention and do not have any limiting effect on it.

[0042] Here, we use the proposed method to extract flying aircraft targets in satellite remote sensing videos. Since an aircraft is a typical rigid object, satellite remote sensing videos have a high temporal resolution. At the same time, they are limited by the limited spatial resolution of remote sensing satellites. Therefore, the aircraft targets in remote sensing videos meet the limitations of the proposed method, that is, they have uniform motion and rigid body characteristics. Of course, this method also has good results for rigid targets that can be approximated as uniform motion, such as car targets in surveillance videos.

[0043] A method for extracting a small rigid target in uniform linear motion, the method comprising the following steps:

[0044] Step 1: Obtain sequence images of the scene;

[0045] The RGB image of the scene may introduce a high degree of noise, making the solution more difficult. In this case, the RGB sequence image of the scene is converted into a grayscale sequence image, such as Figure 2 As shown, the robustness of the processing process is improved (the black box is the aircraft target that needs to be extracted).

[0046] Step 2: Get background B

[0047] The background B is reconstructed from the grayscale sequence using a mixed Gaussian model, such as Figure 3 shown.

[0048] Step 3: Get the initial position of the target

[0049] Based on the input of step 1, the three-frame difference method is used to extract small targets. The results of the first two frames of the three-frame difference are as follows: Figure 4 As shown in (a), the results of the last two frames of the three-frame difference are as follows Figure 4 (b) shows the result of the three-frame difference method. Figure 4 (c)

[0050] Based on the small target extraction results, the area and continuity methods are used to perform graphical processing on the extraction results. The results are as follows: Figure 4 (d)

[0051] Based on the small target extraction result after graphics processing, the center point position of the small target is obtained, and the center point is the coordinate of the center of gravity;

[0052] For the kth frame image I k , the extraction result after graphics processing A k ′, the center point of the small target is (x′ k ,y′ k );

[0053] x′ k ,y′ k Calculated by the following formula:

[0054]

[0055] Among them, the image size is M×N, i and j represent row and column coordinates respectively;

[0056] For all center point coordinates {(x′1,y′1),…,(x′ T ,y′ T )}, motion fitting is performed based on the least squares method to obtain the parameter p that can describe the target motion;

[0057] For the k-th frame grayscale image I k , use the parameter p to calculate the coordinates of the center point on the kth frame (x k ,yk ).

[0058] x k ,y k Calculated by the following formula:

[0059]

[0060] With this coordinate as the center, cut out the image block Y k (p) Figure 4 As shown in (e), the image block B is cut out from the modeled background B. k (p) Figure 4 As shown in (f), the coordinates of the image block are determined by the center point position (x k ,y k ), the size is r×r, where r represents the radius; the r is determined by the maximum value of multiple radii that can surround the target.

[0061] Step 4: For each image block, calculate its unique corresponding foreground F and template value A by optimizing the following formula;

[0062]

[0063] Where T is the number of sequence images, and D represents the total variation operator;

[0064] Preferably, the optimization method is an alternating direction multiplier method;

[0065] Step 5: For each of the image blocks, calculate its uniform motion parameter by optimizing the following formula;

[0066]

[0067] Preferably, the optimization method is the Newton-Raphson iteration method, and its initial value is obtained by step 2; when

[0068] Step 6: Repeat the above steps 4 and 5 for multiple iterations until the specified conditions are met;

[0069] Preferably, the specified condition is that the distance between two iteration results is less than a specified value or the number of iterations reaches a maximum number. In this embodiment, the distance between two iterations changes less than 0.1%, and the maximum number of iterations is 4;

[0070] Step 7: Use the following method to obtain the foreground based on the results of multiple iterations of steps 4 and 5;

[0071] According to the acquired motion parameter p and the frame number k of the image, the center coordinates (x k ,y k );

[0072] x k ,y k Calculated by the following formula:

[0073]

[0074] According to the sub-pixel uniqueness of the center point, the template A is translated to obtain A k ′.

[0075] The results of this embodiment are compared with those obtained by other methods. Figure 5 As shown. Compare the shapes of the aircraft in Figures b, c, d, e, and f with the shape of the aircraft in Figure a. The shapes of the aircraft in Figures b, c, d, and e obtained by other methods are difficult to accurately represent the shape of the aircraft itself, while the shape in Figure f obtained by this method can more accurately represent the shape of the aircraft itself, so this method improves the accuracy of the extraction results. By comparison, it can be found that the results of the PBAS method (such as Figure 5 (b), Figure 5 (h), Figure 5 (n), Figure 5 (t)), the results of the SuBSENSE method (such as Figure 5 (c), Figure 5 (i), Figure 5 (o), Figure 5 (u)), the results of the SGSM method (such as Figure 5 (d), Figure 5 (j), Figure 5 (p), Figure 5 (v)), the results of the FgSegNet-v2 method (such as Figure 5 (e), Figure 5 (k), Figure 5 (q), Figure 5 (w)) show obvious inconsistency, which contradicts the common sense that the aircraft target is a rigid body and its shape should not change significantly. The results obtained by the method of the present invention have better consistency (such as Figure 5 (f), Figure 5 (l), Figure 5 (r), Figure 5 (x)).

[0076] At the same time, we use simulation data to evaluate the performance of the proposed method. The performance evaluation indicator is the F1-score used in this field (the larger the better).

[0077] In the simulation data, we tested the algorithm using 4 backgrounds, 4 aircraft types, 2 noises, and 2 spatial resolution levels. Each scene contains 21 frames of data, and the average F1-score of the algorithm on the 21 frames is shown. The test results are as follows:

[0078] Table 1: F1-score results at spatial resolution 1m and 1% noise

[0079]

[0080]

[0081] Table 2: F1-score results at spatial resolution 1m and 3% noise

[0082] A1S1 A2S1 A3S1 A4S1 A1S2 A2S2 A3S2 A4S2 A1S3 A2S3 A3S3 A4S3 A1S4 A2S4 A3S4 A4S4 average value Ours 0.89 0.91 0.94 0.92 0.92 0.92 0.94 0.93 0.92 0.91 0.95 0.93 0.94 0.93 0.96 0.96 0.93 PBAS 0.7 0.8 0.83 0.77 0.75 0.82 0.83 0.79 0.81 0.85 0.83 0.79 0.8 0.85 0.81 0.79 0.8 SuBSENSE 0.03 0.3 0.57 0.32 0.05 0.35 0.62 0.39 0.69 0.78 0.79 0.68 0.52 0.74 0.75 0.64 0.51 SGSM 0.14 0.38 0.73 0.24 0.33 0.59 0.81 0.49 0.59 0.73 0.92 0.64 0.63 0.77 0.94 0.73 0.6 FgSegnet-v2 0.43 0.6 0.84 0.64 0.56 0.75 0.87 0.71 0.86 0.82 0.89 0.85 0.87 0.8 0.9 0.83 0.76

[0083] Table 3: F1-score results at spatial resolution 2m and 1% noise

[0084] A1S1 A2S1 A3S1 A4S1 A1S2 A2S2 A3S2 A4S2 A1S3 A2S3 A3S3 A4S3 A1S4 A2S4 A3S4 A4S4 average value Ours 0.73 0.81 0.89 0.8 0.71 0.85 0.9 0.84 0.86 0.87 0.92 0.9 0.84 0.9 0.92 0.92 0.85 PBAS 0.59 0.67 0.78 0.58 0.48 0.7 0.8 0.57 0.71 0.76 0.67 0.72 0.79 0.81 0.75 0.76 0.7 SuBSENSE 0.02 0.08 0.26 0.06 0.07 0.09 0.09 0.1 0 0.25 0.46 0.33 0.14 0.47 0.61 0.49 0.22 SGSM 0.02 0.06 0.52 0.01 0.01 0.14 0.62 0.05 0.02 0.29 0.69 0.31 0.06 0.53 0.92 0.61 0.3 FgSegnet-v2 0.07 0.16 0.53 0.09 0.14 0.44 0.74 0.34 0.56 0.7 0.74 0.55 0.77 0.81 0.9 0.79 0.52

[0085] Table 4: F1-score results at spatial resolution 2m and 3% noise

[0086] A1S1 A2S1 A3S1 A4S1 A1S2 A2S2 A3S2 A4S2 A1S3 A2S3 A3S3 A4S3 A1S4 A2S4 A3S4 A4S4 average value Ours 0.73 0.8 0.89 0.8 0.71 0.85 0.9 0.84 0.86 0.87 0.92 0.9 0.84 0.9 0.92 0.92 0.85 PBAS 0.59 0.67 0.78 0.58 0.49 0.7 0.8 0.57 0.71 0.76 0.67 0.72 0.79 0.81 0.75 0.76 0.7 SuBSENSE 0.01 0.09 0.22 0.08 0.07 0.09 0.1 0.09 0 0.24 0.48 0.33 0.05 0.46 0.62 0.51 0.21 SGSM 0.02 0.06 0.52 0.01 0.01 0.14 0.62 0.05 0.02 0.29 0.69 0.31 0.06 0.53 0.92 0.61 0.3 FgSegnet-v2 0.07 0.16 0.54 0.09 0.15 0.44 0.74 0.34 0.56 0.7 0.74 0.55 0.76 0.81 0.9 0.79 0.52

[0087] The above simulation results show that the performance evaluation index of the method for extracting small rigid targets in uniform linear motion disclosed by the present invention is better than other methods under the same environment.

[0088] In order to illustrate the content and implementation method of the present invention, this specification provides a specific embodiment. The purpose of introducing details in the embodiment is not to limit the scope of the claims, but to help understand the method of the present invention. Those skilled in the art should understand that various modifications, changes or replacements of the steps of the best embodiment are possible without departing from the spirit and scope of the present invention and the claims attached thereto. Therefore, the present invention should not be limited to the content disclosed in the best embodiment and the drawings.

Claims

1. A method for extracting small rigid targets in uniform linear motion, characterized by: The following steps are involved: Step 1: Obtain sequence images of the scene; Step 2: Obtain background B; Step 3: Obtain the initial motion parameters of the target; The implementation method of step three is: Based on the input of step 1, extract small targets on each frame; Based on the small target extraction results, the extraction results are processed by graphics; Based on the small target extraction result after graphics processing, the center point position of the small target is obtained; For the kth frame image I k , the extraction result after graphics processing A k ', the center point of the small target is (x' k ,y' k ); x' k ,y' k Calculated by the following formula: Among them, the image size is M×N, i and j represent row and column coordinates respectively; For all center point coordinates {(x'1,y'1),…,(x' T ,y' T )} to fit and obtain the parameter p that can describe the target motion; For the kth frame image I k , use the parameter p to calculate the coordinates of the center point on the kth frame (x k ,y k ), with this coordinate as the center, cut out the image block Y k (p), cut out image block B from modeled background B k (p), the coordinates of the image block are determined by the center point position (x k ,y k ), the size is r×r, where r represents the radius; the r is determined by the maximum value of multiple radii that can surround the target; Step 4: Foreground extraction; For the image block, the unique corresponding foreground F and template value A are calculated by optimizing the following formula; Where T is the number of sequence images, and D represents the total variation operator; Step 5: motion estimation; For the image block, its uniform motion parameter is calculated by optimizing the following formula: Step 6: Repeat the above steps 4 and 5 for multiple iterations until the specified conditions are met; Step 7: Obtain the extraction result based on the results of multiple iterations of steps 4 and 5.

2. The method for extracting a small rigid target in uniform linear motion according to claim 1, characterized in that: The implementation method of step seven is: According to the obtained optimized motion parameter p and the frame number k of the image, the center coordinates (x k ,y k ); According to the coordinates of the center point, the template A is translated to obtain A k ”.

Images