Satellite video moving target detection method

Abstract

The invention discloses a satellite video moving target detection method, which adopts an enhanced three-dimensional total variation regular term and a Gaussian model to accurately model a background and a target respectively, and adopts a low-rank matrix decomposition model to separate the target as a special sparse noise in complex background noise. And then solving the low-rank matrix decomposition model by an ADMM method to obtain a noise component containing a real target. According to the method, the detection precision of a dark weak target and a low-speed moving target is improved in satellite video weak moving target detection, and false alarms can be inhibited at the same time.

Description

Technical Field

[0001] The invention relates to a moving target detection technology, in particular to a satellite video moving target detection method. Background Art

[0002] In recent years, significant breakthroughs have been made in the field of dynamic target detection in satellite videos. In order to meet this challenge, researchers have proposed many innovative solutions, including both traditional technical methods and cutting-edge algorithms based on deep learning. Currently, traditional methods are mainly divided into frame difference method, optical flow method and background modeling method.

[0003] The frame difference method mainly includes two-frame frame difference method and three-frame frame difference method. Ao et al. proposed a moving target detection and tracking algorithm based on noise modeling. The algorithm first uses local noise modeling to separate the background and the target. In order to better distinguish the moving target from the noise, the method uses the target discrimination based on region growth and multiple morphological information, and finally uses the target tracking technology based on the Kalman filter. Under the assumption of constant brightness, small motion and spatial consistency, the optical flow method uses the change of pixel intensity in the time domain of the sequence image and the correlation between adjacent frames to determine the movement of the pixel position, thereby determining the movement of the target. Satellite video moving target detection methods based on background modeling are mainly divided into three categories: basic background modeling method, statistical model modeling method and robust subspace learning based on matrix decomposition. The basic background modeling method includes median background modeling and mean background modeling. Kopsiaftis (Kopsiaftis G, Karantzalos K. Vehicle detection and traffic density monitoring from very high resolution satellite video data [C]. In 2015 IEEE International Geoscience and RemoteSensing Symposium. 2015: 1881–1884.) proposed an automatic vehicle detection and traffic density estimation algorithm. This method uses the median background modeling method to obtain the moving target area, then uses mathematical morphology and statistical analysis to obtain the target contour, and finally obtains the traffic density estimation value through the low-resolution grid. Background modeling methods based on statistical models include mixed Gaussian background models, codebook models, ViBe algorithms, etc. The robust subspace learning method based on matrix decomposition separates the time-varying foreground related to the moving object and the background in the low-dimensional subspace from the video, thereby realizing the detection of moving objects in the video.

[0004] At present, Roman (Pflugfelder R, Weissenfeld A, Wagner J. On learning vehicle detection in satellite video [J]. arXiv preprint arXiv:2001.10900.2020.) et al. proposed a moving target detection method for satellite videos based on FoveaNet4Sat. In order to improve the target detection performance, Chen (Chen R, Li X, Li S. A lightweight CNN model for refining moving vehicle detection from satellite videos [J]. IEEE Access. 2020, 8: 221897–221917.) et al. proposed a moving target detection method based on background modeling and lightweight convolutional neural network (CNN). First, the input video is preprocessed using an adaptive filter, and then the background modeling method is used to obtain a preliminary detection result with high recall rate but low precision. Finally, a lightweight convolutional neural network (CNN) is used to eliminate false alarms and obtain a detection result with higher precision. However, since it is difficult to annotate high-quality satellite video moving target detection datasets, the detection performance of weak moving targets in satellite videos is poor. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a satellite video moving target detection method to solve the problems of difficulty in detecting weak targets and inability to detect slow-moving targets submerged in complex background motion.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a satellite video moving target detection method, comprising the following steps: S1. Obtain a satellite video sequence tensor and expand the satellite video sequence tensor into a video matrix , set the regularization term coefficient ; h, w, t are the spatial width, height and temporal length of the satellite video sequence respectively; S2. Initialization ;in, Respectively represent the initial background component and the initial noise component, is the initial value of the Lagrange multiplier; is the initial coefficient matrix, is the initial transformation matrix, is the initial value of auxiliary variables; S3. Update the background component using the following formula: ; ; ; and They are Fourier transform and its inverse transform, is the element-wise multiplication, is a positive penalty scalar, , , , are the gradient map matrices along the spatial width, height, and time dimensions of the satellite video sequence tensor, respectively. is the background component of k+1 iterations; fold() represents the expansion operation, is the penalty parameter; S4. Update the noise component using the following formula: ; S5, update using the following formula : ; () is the soft threshold operator; S6, update using the following formula : ; is a left singular orthogonal matrix, F is a diagonal matrix, is the transpose of the diagonal matrix, C is a right singular orthogonal matrix; S7. Update Lagrange multipliers : ; S8, Update : ; is a constant; ; S9, determine whether the convergence condition is met or whether the maximum number of iterations is reached. If not, add 1 to the value of k and return to step S3; otherwise, end and obtain the final background component and noise component.

[0007] The present invention no longer uses TV terms to model the background, but directly treats target detection as an image denoising task, solving the problem that weak target detection is difficult and slow-moving targets submerged in complex background motion cannot be detected. The method of the present invention avoids singular value decomposition, has a small amount of calculation, is simple to calculate, has low complexity, and has high detection accuracy.

[0008] Gradient map matrix and satellite video sequence tensor The relationship is: , , r is the rank of the video matrix.

[0009] In the present invention .

[0010] In the present invention ;in, = , = .

[0011] The convergence condition is expressed as: ; Among them, B is the background component of the video sequence image, is a positive number set. is the 2-norm.

[0012] In the present invention, the maximum number of iterations is set to 400.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention takes into account the characteristics of complex dynamic background and slow-moving target motion during the detection process, so the detection results obtained are more accurate and have fewer false alarms; the method of the present invention avoids singular value decomposition, has a small amount of calculation, is simple to calculate, has low complexity, and has high detection accuracy. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] In this article, the terms "first", "second" and other similar words are not intended to imply any order, quantity and importance, but are only used to distinguish different elements. In this article, the terms "one", "an" and other similar words are not intended to indicate that there is only one of the things described, but rather indicate that the relevant description is only for one of the things described, and the things described may have one or more. In this article, the terms "comprise", "include" and other similar words are intended to indicate logical mutual relationships, and cannot be regarded as indicating spatial structural relationships. For example, "A includes B" is intended to indicate that B belongs to A logically, but does not indicate that B is located inside A spatially. In addition, the meanings of the terms "comprise", "include" and other similar words should be regarded as open, rather than closed. For example, "A includes B" is intended to indicate that B belongs to A, but B does not necessarily constitute the whole of A, and A may also include other elements such as C, D, and E.

[0016] The total variation regularization term (TV) was first proposed by Rudin et al. Since the TV regularization term can effectively preserve the edge and corner structure of the image, it is widely used in image denoising and image restoration. Since video is three-dimensional data, the above definition is further extended to the three-dimensional tensor space to describe the local smoothness of the spatial and temporal dimensions in the video, preserving the edge and corner structure of the image, thereby obtaining a cleaner image. For a given satellite video sequence tensor (h, w, t are the spatial width, height and temporal length of the satellite video sequence respectively) The variance formulas for different directions are as follows: (1-2) in Represents the t frame in The pixel value at the position. Then the resulting matrix is ​​vectorized to obtain Merge into a matrix, and you get D e=[ D h e T , D w e T , D t e T ] ,in Respectively represent the intensity of the horizontal, vertical and time axes. Intensity is Gradient map of ,in , , Along The gradient map matrix of the spatial width, height and time dimensions of . Therefore, the three-dimensional total variation regularization term is positioned as (3DTV) as follows: (1-3) However, the three-dimensional total variation regularization term does not fully utilize the data structure information of gradient maps in different directions, where the data structure information includes the differences in the sparsity of gradient maps in different dimensions of space and time domains, and the gradient maps of videos have obvious correlation in the time domain.

[0017] In order to improve the image denoising performance, the enhanced three-dimensional TV (E-3DTV) norm is used to constrain the background to obtain a more accurate background image. For a given video sequence, the definition of the enhanced three-dimensional TV (E-3DTV) is as follows: (1-4) .

[0018] The Gaussian Model (GM) is essentially a probability-based clustering model. It usually assumes that there are a certain number of Gaussian distributions, each of which represents a cluster. This method uses a soft clustering method and tends to group data belonging to the same type of distribution into one cluster. The mixed Gaussian model uses K single Gaussian models to project the data. Each Gaussian model is a component. When the fractions are large enough, the Gaussian mixture model can approximate any continuous distribution with high accuracy, which helps to better characterize the spatial distribution and characteristics of the data. The Gaussian mixture model has been widely used in image restoration, image segmentation, and object recognition due to its excellent mathematical properties. The general single Gaussian distribution probability density function is defined as: (1-5) represent the mean and standard deviation of the Gaussian distribution respectively.

[0019] Due to the existence of slow-moving targets and moving backgrounds in satellite videos, it is difficult to distinguish between targets and background (noise). By analyzing the characteristics of the background and target distribution between frames of a video sequence, it can be found that the value of the same pixel in the background should be the same in consecutive frames, while the pixel value of noise or moving objects should change. In the noise mode, the change of true noise pixels is irregular, and the change of moving target pixels is a regular, clustered change point. By accurately modeling the noise between consecutive frames of a video sequence, it is possible to accurately distinguish between targets and noise. Therefore, the embodiment of the present invention uses a Gaussian model to accurately model the noise between different frames.

[0020] Given a satellite video sequence of t frames, it is vectorized as ,in are the height and width of the video frame respectively. In theory, the video sequence images It can be expressed as the following additive model: (1-6) Among them, B and E represent the background component and the noise component respectively. The target can be regarded as a special "noise" with sparsity, which can be separated after obtaining the noise component E. Next, the background component and the noise component will be modeled and analyzed respectively.

[0021] Since the background matrix satisfies the low-independence property, the detection of moving targets in satellite videos can be achieved by solving the following optimization problem: (1-7) Where λ is the regularization coefficient, which is used to balance the low-inheritance component and the sparse component.

[0022] In order to better separate the background and the target and detect slow-moving targets, we introduced the corresponding enhanced three-dimensional total variation regularization term (E-3DTV) and Frobenius norm in the optimization problem, which are expressed as follows: (1-8) Where E-3DTV is the enhanced three-dimensional total score regularization term, and its specific definition is shown in equation (1-4). F refers to the Frobenius norm of the noise matrix, and λ is the regularization coefficient used to balance the low-inheritance component and the sparse component. The model expression is as follows: (1-9) Based on formula (1-4) and formula (1-9), the model can be written as: (1-10) in is the gradient operator symbol.

[0023] 1. Model solution The augmented Lagrangian form of problem (1-10) can be described as: (1-11) where Y and is the Lagrange multiplier, is a positive penalty scalar.

[0024] In the ADMM framework, each variable involved in the optimization (1-11) needs to be solved alternately , while keeping all other variables unchanged, the solution method for the variables involved in its sub-problems will be described below.

[0025] (1) Update of B: Extract all terms containing variable B from equation (1-12), fix other variables, and solve for B, which can be transformed into solving the following linear problem: (1-12) In order to solve equation (1-12), the partial derivative of the objective function with respect to variable B is obtained, that is, Then it can be transformed into solving the following linear problem: (1-13) in is a linear operator of B, , yes The transpose operation of , so the solution of B is: (1-14) Where 1 represents a tensor whose elements are all 1, is the element-wise multiplication, is the correlation difference filter, and They are Fourier transform and its inverse transform, is the element-wise square operation.

[0026] (2) Update Extract all the formulas (1-11) containing , fix other variables, and solve (1-15) Using soft threshold operator This subproblem can be solved effectively, and the soft threshold operator is defined as follows: (1-16) in is the threshold and is greater than 0.

[0027] The approximate solution of formula (1-16) can be obtained by the soft threshold operator as follows: (1-17) (3) Update Extract all the variables from formula (1-11) , fix other variables, and solve : (1-18) The global solution to this subproblem can be derived from the theorem (Q. Xie et al., “Multispectral imagesdenoising by intrinsic tensor sparsity regularization,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 1692–1700.), The closed-form solution of is as follows: [ A , C , F ]=svd( ( D n + G n k μ ) T U n k ) V n k+1 = OF T (1-19) (4) Update of E Extract all terms including variable E from equation (1-11), fix other variables and calculate E (1-20) In the ADMM framework, the update of the Lagrange multiplier is as follows: (1-21) In summary, the entire ADMM-based algorithm can be obtained as shown below, which is used to calculate models 1-9.

[0028] The proposed moving target detection algorithm based on enhanced three-dimensional total variation regularization term (E3DTV-MOD) is compared with existing algorithms. These include DECOLOR (Zhou Proceedings of the 28th International Conference on Machine Learning,. 2011.), ClusterNet (LaLonde R, Zhang D, ShahM. Clusternet: Detecting small objects in large scenes by exploiting spatio-temporal information [C]. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition. 2018: 4003–4012), DTTP (Ahmadi SA, GhorbanianA, Mohammadzadeh A. Moving vehicle detection, tracking and traffic parameter estimation from a satellite video: A perspective on a smarter city [J].International Journal of Remote Rensing. 2019, 40 (22): 8379–8394.), D&T (Ao W, Fu Y, Hou X, et al.Needles in a haystack: Tracking city-scale moving vehicles from continuously moving satellite [J]. IEEE Transactions on Image Processing. 2019, 29: 1944–1957.), E-LSD (Zhang J, Jia X, Hu J. Error bounded foreground and background modeling for moving object detection in satellite videos [J]. IEEE Transactions on Geoscience and Remote Sensing. 2019, 58 (4): 2659–2669.), and DSFNet (Xiao C, Yin Q, Ying X, et al. DSFNet: Dynamic and static fusion network for moving object detection in satellite videos [J]. IEEE Geoscience and Remote Sensing Letters. 2021, 19: 1–5.). Among them, DSFNet and ClusterNet (LaLonde R, Zhang D, Shah M. Clusternet: Detecting small objects in large scenes by exploiting spatio-temporal information [C]. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4003–4012.) is a method based on deep learning, D&T is an improved frame difference method (foreground detection method), and the rest are methods based on background modeling. First, the proposed moving target detection algorithm based on enhanced three-dimensional total variation regularization term is verified on the VISOT dataset, and the performance of all comparison methods is quantitatively analyzed and compared using the mentioned evaluation indicators. The quantitative results are shown in Table 1, where the optimal result is marked in bold. The E3DTV-MOD algorithm proposed in the embodiment of the present invention achieves the highest accuracy of 89% on the VISOT dataset, and also achieves the highest result value of 83% in the F1 indicator. The results show that the proposed E3DTV-MOD algorithm is superior to other comparison methods in false alarm suppression and target detection. From the results in Table 1, it can be seen that the E3DTV-MOD algorithm achieves the highest F1 result value on video 003 and video 005, and the highest accuracy result value on the 5 video sequences from video 002 to video 005. Although the recall rate of the E3DTV-MOD algorithm is slightly lower than that of the deep learning-based method DSFNet, the proposed E3DTV-MOD algorithm does not require the use of large-scale manually annotated datasets for training and has clear physical principles. Compared with the D&T method, the proposed algorithm has significantly improved the accuracy and F1 value on all videos. Compared with all background modeling methods, the proposed method has achieved considerable performance improvements in both accuracy and F1 indicators. This is because the E3DTV-MOD algorithm considers the characteristics of complex dynamic background and slow-moving target motion during the detection process, so the detection results obtained are more accurate and have fewer false alarms. .

[0029] Table 1 Detection performance evaluation of different methods in video 001 to video 006

[0030] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0031] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A satellite video moving target detection method, characterized in that: The following steps are involved: S1. Obtain a satellite video sequence tensor and expand the satellite video sequence tensor into a video matrix , set the regularization term coefficient ; h, w, t are the spatial width, height and temporal length of the satellite video sequence respectively; S2. Initialization ;in, Respectively represent the initial background component and the initial noise component, is the initial value of the Lagrange multiplier; is the initial coefficient matrix, is the initial transformation matrix, is the initial value of auxiliary variables; S3. Update the background component using the following formula: ; ; ; and They are Fourier transform and its inverse transform, is the element-wise multiplication, is a positive penalty scalar, , , , are the gradient map matrices along the spatial width, height, and time dimensions of the satellite video sequence tensor, respectively. is the background component of k+1 iterations; fold() represents the expansion operation, is the penalty parameter; S4. Update the noise component using the following formula: ; S5. Update using the following formula : ; () is the soft threshold operator; S6, update using the following formula : ; is a left singular orthogonal matrix, F is a diagonal matrix, is the transpose of the diagonal matrix, C is a right singular orthogonal matrix; S7. Update Lagrange multipliers : ; S8, Update : ; is a constant; ; S9, determine whether the convergence condition is met or whether the maximum number of iterations is reached. If not, add 1 to the value of k and return to step S3; otherwise, end and obtain the final background component and noise component.

2. The satellite video moving target detection method according to claim 1, characterized in that: Gradient map matrix and satellite video sequence tensor The relationship is: , , r is the rank of the video matrix.

3. The satellite video moving target detection method according to claim 1, characterized in that: 。 4. The satellite video moving target detection method according to claim 1, characterized in that: ;in, = , = .

5. The satellite video moving target detection method according to claim 1, characterized in that: The convergence condition is expressed as: ； Among them, B is the background component of the video sequence image, is a positive number set. is the 2-norm.

6. The satellite video moving target detection method according to claim 1, characterized in that: The maximum number of iterations is set to 400.

Images