Infrared image target and background separation method and system based on space-time tensor decomposition

Abstract

The application discloses an infrared image target and background separation method and system based on space-time tensor decomposition, relates to the field of image processing and target detection, and aims to solve the problem of incomplete separation of targets and backgrounds in infrared images and partial background remaining in target images. The application stacks infrared images along a time dimension on the basis of background registration, and divides the infrared images into a plurality of sub-tensor blocks. The application models the background as the sum of a low-rank background component and a sparse background component according to whether the background meets a low-rank assumption, and designs a sparse constraint term and a trajectory constraint term according to the distribution and motion characteristics of the target, so as to realize accurate model construction. The sub-tensor blocks obtained through division are input into the model in parallel, and separated target tensor blocks, low-rank background tensor blocks and sparse background tensor blocks are obtained through iterative solution. Finally, the separated tensor blocks are restored into target image sequences and background image sequences through reconstruction, so as to realize the separation of targets and backgrounds in infrared images.

Description

Technical Field

[0001] This invention relates to the field of image processing and target detection, specifically to a method and system for separating targets and backgrounds in infrared images based on spatiotemporal tensor decomposition. Background Technology

[0002] Infrared search and track systems based on infrared moving target detection have wide applications in various fields, including military and civilian sectors. Achieving moving target detection through target-background separation techniques is a promising area for further exploration in this field.

[0003] The problem of separating the target from the background is also known as "foreground-background separation." Classical foreground-background separation methods include background subtraction, frame subtraction, and optical flow methods. These methods often only utilize differences in the temporal domain, neglecting spatial features that can aid in separation. Pixel-level classification algorithms based on deep learning can also be used to solve the foreground-background separation problem. In contrast to classical algorithms, these algorithms often focus on a single frame and cannot effectively utilize temporal information. Due to the aforementioned problems and the limitations of infrared imaging quality, the target images obtained by existing methods often contain a large number of false alarms or missing targets.

[0004] To this end, this invention considers both the low-rank nature of the background and the sparsity of the target in the spatial and temporal domains of infrared images through tensor modeling, and designs constraint terms for cases where some infrared backgrounds do not conform to the low-rank assumption, thereby further improving the foreground-background separation performance and robustness of the model. Summary of the Invention

[0005] To address the issues of false alarms and missing targets in existing target-background separation methods, this invention aims to provide a method and system for separating infrared image targets and backgrounds based on spatiotemporal tensor decomposition.

[0006] A target-background separation method for infrared images based on spatiotemporal tensor decomposition includes the following steps:

[0007] Step 1. Read 2f+1 frames of infrared images of size w×h consecutively from the original infrared image sequence. Use the intermediate frame as a reference to perform background registration on the remaining 2f frames. Stack the registered images together with the intermediate frame along the time dimension to form a third-order tensor D∈R. w×h×(2f+1) f is 3 to 10;

[0008] Step 2. Model the third-order tensor D formed from the original infrared image sequence as the target component T and the low-rank background component B. L With sparse background component B S The sum of the three. T is constrained by the tensor L1 norm, and B is constrained by the improved tensor kernel norm based on the low-rank matrix kernel. L The joint constraint B is minimized by the tensor L1 norm carrying the relaxation factor and the frame difference accumulation.S Construct the objective function of the model;

[0009] Step 3. In the spatial domain, divide the third-order tensor obtained in Step 1 into n original sub-tensor blocks D, using pz as the unit. (i) ∈R pz×pz×(2f+1) The model described in step 2 is input in parallel, and the target tensor block T is obtained through iterative solution. (i) Low-rank background tensor block B L(i) and sparse background tensor block B S(i) The convergence value, i = 1, 2, ..., n, pz takes values ​​from 20 to 100;

[0010] Step 4. Reconstruct the target tensor block into a target image sequence, and fuse the low-rank background tensor block with the sparse background tensor block to reconstruct the background image sequence, ultimately achieving the separation of the target and the background in the infrared image.

[0011] In the above technical solution, step 2 includes the following steps:

[0012] Step 2.1 The third-order tensor D formed from the original image sequence can be modeled as the sum of the following three components:

[0013] D = T + B L +B S

[0014] Where T is the target component, B L For low-rank background components, B S For sparse background components;

[0015] Step 2.2 Construct constraint terms ||T||1 using tensor L1 norm constraints T, and constrain B using improved tensor kernel norm constraints based on low-rank matrix kernels. L Constraint term ||B L || ITNN The constraint B is minimized by combining the tensor L1 norm carrying the relaxation factor α with the frame difference accumulation. S Constituting the constraint term α||B S ||1 and||FD(B) S 1. α is taken as 0.1 to 0.9;

[0016] Step 2.3 Based on Step 2.2, introduce balance factors λ1 and λ2 to form the objective function of the model:

[0017]

[0018] st||DB L -TB S || F ≤δ 0<α<1

[0019] Where δ represents the error caused by noise. H1 is 1 to 3, and H2 is 10 to 100.

[0020] In the above technical solution, step 3, which involves iteratively solving to obtain the convergence values ​​of the target tensor block, the low-rank background tensor block, and the sparse background tensor block, includes the following steps, where subscripts (i) are omitted in the following formulas:

[0021] Step 3.1 Introduce auxiliary variables V1 and V2 into the objective function described in Step 2, and use variance σ to characterize the noise error, with σ ranging from 0.5 to 2. Then the objective function can be rewritten as:

[0022]

[0023] stV1=B S V2=FD(B S )

[0024] Step 3.2 Construct the augmented Lagrangian function of the objective function described in Step 3.1:

[0025]

[0026] Where Y1 and Y2 are Lagrange multipliers, μ is the penalty factor, and μ is 10. -4 Order of magnitude;

[0027] Step 3.3 Update B by alternating the following iterative formula based on the augmented Lagrangian function described in Step 3.1. L ,T,B S until convergence:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] Y1=Y1+μ(V1-B S )

[0034] Y2=Y2+μ(V2-FD(B S ))

[0035] Among them, prox ITNN (g) is the solution operator for improving robust tensor principal component analysis, and S(g) is the soft threshold shrinkage operator.

[0036] Step 3.4 Iterate using the formula described in Step 3.3 until convergence or the upper limit of the number of iterations is reached to obtain the separated target tensor block T. (i) Low-rank background tensor block B L(i) and sparse background tensor block B S(i) The preferred convergence condition is... The maximum number of iterations is set to 100.

[0037] The infrared image target and background separation system based on spatiotemporal tensor decomposition, employing the infrared image target and background separation method based on spatiotemporal tensor decomposition as described in any one of claims 1 to 5, includes:

[0038] Tensor block construction module, which uses the original infrared sequence to construct the original tensor block;

[0039] The model building module models the original tensor block as the sum of the target component, the low-rank background component, and the sparse background component, and constructs the objective function of the model through norm constraints.

[0040] The tensor block segmentation module, based on background registration and temporal stacking, segments the original tensor blocks to form a sub-tensor set input decomposition module that supports parallel computation.

[0041] The tensor decomposition module combines the sub-tensor set with the model's objective function and uses iterative formulas to solve for the convergence values ​​of the target tensor block, low-rank background tensor block, and sparse background tensor block.

[0042] The tensor block reconstruction module restores the convergence values ​​of the acquired target tensor block, low-rank background tensor block, and sparse background tensor block into the target image sequence and the background image sequence.

[0043] An electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the unmanned aerial vehicle (UAV) arrival navigation method as described in any one of claims 1 to 5.

[0044] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the unmanned aerial vehicle (UAV) arrival navigation method as described in any one of claims 1 to 5.

[0045] Due to the adoption of the above technical solutions, the beneficial effects of this invention are:

[0046] This invention models the infrared background as the sum of a low-rank background component and a sparse background component based on whether the infrared background conforms to the low-rank assumption. It employs a tensor L1 norm constraint T and an improved tensor kernel norm constraint B based on a low-rank matrix kernel.L The constraint B is the minimization of the combined frame difference accumulation of the tensor L1 norm carrying a relaxation factor. S This invention achieves accurate modeling of the spatiotemporal features of targets and backgrounds in infrared images. Building upon this, the invention avoids false alarms caused by infrared background components violating the low-rank assumption entering the T component through the design of relaxation factors, and further prevents the real infrared target from shifting from the T component to the B component through strict frame difference accumulation minimization constraints. S The component may cause missed detections; thanks to the accurate model construction, this method is robust and can achieve complete separation of the target and the background; thanks to the idea of ​​tensor block partitioning and parallel input, this method has low time overhead and can achieve real-time detection. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart of the present invention;

[0049] Figure 2 This is a frame and its three-dimensional distribution map from the image sequence to be processed in Embodiment 1 of the present invention;

[0050] Figure 3 It is the corresponding target image sequence obtained by separation in Embodiment 1 of the present invention. Figure 2 A frame and its three-dimensional distribution map;

[0051] Figure 4 It is the corresponding low-rank background image sequence obtained by separation in Embodiment 1 of the present invention. Figure 2 A frame and its three-dimensional distribution map;

[0052] Figure 5 It is the corresponding sparse background image sequence obtained by separation in Embodiment 1 of the present invention. Figure 2 A frame and its three-dimensional distribution map.

[0053] Figure 6 This is a schematic diagram of the infrared image target and background separation system based on spatiotemporal tensor decomposition according to Embodiment 2 of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0055] Example 1

[0056] See attached document Figure 1 - Appendix Figure 5 A target-background separation method for infrared images based on space-time tensor decomposition includes the following steps:

[0057] Step 1. Read 15 consecutive 256×256 infrared images from the sequence. The images in the sequence and their three-dimensional distribution map are shown below. Figure 2 As shown. Background registration is performed on the remaining 14 frames using the intermediate frame as a reference. The registered images, along with the intermediate frame, are stacked along the time dimension to form a third-order tensor D∈R. 256×256×15 ;

[0058] Step 2. Model the third-order tensor D formed from the original infrared image sequence as the target component T and the low-rank background component B. L With sparse background component B S The sum of the three. T is constrained by the tensor L1 norm, and B is constrained by the improved tensor kernel norm based on the low-rank matrix kernel. L The constraint B is minimized by combining the tensor L1 norm carrying a relaxation factor with the frame difference accumulation. S Construct the objective function of the model;

[0059] Step 2.1 The third-order tensor D formed from the original image sequence can be modeled as the sum of the following three components:

[0060] D = T + B L +B S

[0061] Where T is the target component, B L For low-rank background components, B S For sparse background components;

[0062] Step 2.2 Construct constraint terms ||T||1 using tensor L1 norm constraints T, and constrain B using improved tensor kernel norm constraints based on low-rank matrix kernels. LConstraint term ||B L || ITNN The joint constraint B is minimized by the tensor L1 norm carrying a relaxation factor α = 0.5 and the frame difference accumulation. S Constituting the constraint term α||B S ||1 and||FD(B) S )||1;

[0063] Step 2.3 Based on Step 2.2, introduce balance factors λ1 and λ2 to form the objective function of the model:

[0064]

[0065]

[0066] Where δ represents the error caused by noise.

[0067] Step 3. In the spatial domain, divide the third-order tensor obtained in Step 1 into 64 original sub-tensor blocks D, using 32-pixel units as the unit. (i) ∈R 32×32×15 The model described in step 2 is input in parallel, and the target tensor block T is obtained through iterative solution. (i) Low-rank background tensor block B L(i) and sparse background tensor block B S(i) The convergence value, i = 1, 2, ..., 64; subscripts (i) are omitted in the following formulas:

[0068] Step 3.1 Introduce auxiliary variables V1 and V2 into the objective function described in Step 2, and characterize the noise error with variance σ = 1.5. Then the objective function can be rewritten as:

[0069]

[0070] stV1=B S V2=FD(B S )

[0071] Step 3.2 Construct the augmented Lagrangian function of the objective function described in Step 3.1:

[0072]

[0073] Where Y1 and Y2 are Lagrange multipliers, μ = 10 -4 As a penalty factor;

[0074] Step 3.3 Update B by alternating the following iterative formula based on the augmented Lagrangian function described in Step 3.1. L ,T,B Suntil convergence:

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] Y1=Y1+μ(V1-B S )

[0081] Y2=Y2+μ(V2-FD(B S ))

[0082] Among them, prox ITNN (g) is the solution operator for improving robust tensor principal component analysis, and S(g) is the soft threshold shrinkage operator.

[0083] Step 3.4 Iterate using the formula described in Step 3.3 until convergence or the upper limit of the number of iterations is reached to obtain the separated target tensor block T. (i) Low-rank background tensor block B L(i) and sparse background tensor block B S(i) The convergence condition is set to... The maximum number of iterations is set to 100.

[0084] Step 4. Reconstruct the target tensor block into a target image sequence, the low-rank background tensor block into a low-rank background image sequence, and the sparse background tensor block into a sparse background image sequence, ultimately achieving target-background separation in the infrared image. The resulting target image and its 3D distribution map are shown below. Figure 3 As shown, the target is highlighted while the background area is almost unresponsive, achieving clean target-background separation. The resulting low-rank background image, sparse background image, and their respective 3D distribution maps are shown below. Figure 4 and Figure 5 As shown, the bright areas in the sparse background image are easily left in the target image, resulting in incomplete separation of the target and background. This method achieves effective separation of the two and greatly reduces false alarms.

[0085] Example 2

[0086] Reference Appendix Figure 6 An infrared image target and background separation system based on space-time tensor decomposition, employing the infrared image target and background separation method based on space-time tensor decomposition as described in Example 1, includes:

[0087] Tensor block construction module, which uses the original infrared sequence to construct the original tensor block;

[0088] The model building module models the original tensor block as the sum of the target component, the low-rank background component, and the sparse background component, and constructs the objective function of the model through norm constraints.

[0089] The tensor block segmentation module, based on background registration and temporal stacking, segments the original tensor blocks to form a sub-tensor set input decomposition module that supports parallel computation.

[0090] The tensor decomposition module combines the sub-tensor set with the model's objective function and uses iterative formulas to solve for the convergence values ​​of the target tensor block, low-rank background tensor block, and sparse background tensor block.

[0091] The tensor block reconstruction module restores the convergence values ​​of the acquired target tensor block, low-rank background tensor block, and sparse background tensor block into the target image sequence and the background image sequence.

[0092] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. A method for separating targets and backgrounds in infrared images based on spatiotemporal tensor decomposition, characterized in that, Includes the following steps: Step 1: Continuously read from the original infrared image sequence Frame size is Infrared images, using the middle frame as a reference for the remaining... Frames are registered to the background, and the registered images, along with intermediate frames, are stacked along the time dimension to form a third-order tensor. ,in Represents European-style space; Step 2: Convert the original infrared image sequence into a third-order tensor Model as target component Low-rank background components With sparse background component The sum of the three; by tensor Norm Constraints Improved tensor kernel norm constraint based on low-rank matrix kernel Tensors carrying relaxation factors Joint constraint of norm and frame difference accumulation minimization Construct the objective function of the model; Step 3: In the airspace Divide the third-order tensor obtained in step 1 into units of... A primitive sub-tensor block The model described in step 2 is input in parallel, and the target tensor block is obtained through iterative solution. Low-rank background tensor blocks and sparse background tensor blocks The convergence value, where ; Step 4: Transfer the target tensor block Reconstructed into target image sequence and low-rank background tensor blocks fused sparse background tensor blocks The image is reconstructed into a sequence of background images, ultimately achieving the separation of the target from the background in the infrared image; Step 2 includes the following steps: Step 2.1 The third-order tensor formed from the original image sequence It can be modeled as the sum of the following three components: ； in, For the target component, For low-rank background components, For sparse background components; Step 2.2: Using tensors Norm Constraints Constraints Improved tensor kernel norm constraint based on low-rank matrix kernel Constraints To carry relaxation factors tensor Norm joint frame difference accumulation minimization constraint Constraints and ; Step 2.3: Introduce a balance factor based on Step 2.

2. and The objective function that constitutes the model is: ； in, Errors caused by noise; Step 3, which involves iteratively solving for the convergence values ​​of the target tensor block, the low-rank background tensor block, and the sparse background tensor block, includes the following steps: Step 3.1: Introduce auxiliary variables and Meanwhile, with variance Characterize the noise error and then rewrite the objective function of the model described in step 2 to obtain the rewritten objective function; Step 3.2: Construct the augmented Lagrangian function of the rewritten objective function described in Step 3.1; Step 3.3: Update the augmented Lagrangian function described in Step 3.

2. , , , , , , Until convergence, among which and It is a Lagrange multiplier.

2. The infrared image target and background separation method based on spatiotemporal tensor decomposition according to claim 1, characterized in that, Subscripts are omitted in the following formulas. : The rewritten objective function in step 3.1 is: ； The augmented Lagrangian function in step 3.2 is: ； in, and For Lagrange multipliers, As a penalty factor; Update in step 3.3 , , , , , , During the convergence process, the following iterative formula is used alternately: ； ； ； ； ； ； ； in, To improve the solution operator for robust tensor principal component analysis, This is a soft threshold shrinkage operator; Step 3.4 Iterate using the formula described in Step 3.3 until convergence or the upper limit of the number of iterations is reached to obtain the separated target tensor block. Low-rank background tensor blocks and sparse background tensor blocks .

3. The infrared image target and background separation method based on spatiotemporal tensor decomposition according to claim 1, characterized in that, The balance factor in step 2.3 and for: ； in, , As the tuning factor, in this method Should be compared Larger by 1 to 2 orders of magnitude.

4. An infrared image target and background separation system based on spatiotemporal tensor decomposition, characterized in that, The infrared image target and background separation method based on spatiotemporal tensor decomposition as described in any one of claims 1 to 3 includes: Tensor block construction module, which uses the original infrared sequence to construct the original tensor block; The model building module models the original tensor block as the sum of the target component, the low-rank background component, and the sparse background component, and constructs the objective function of the model through norm constraints. The tensor block segmentation module, based on background registration and temporal stacking, segments the original tensor blocks to form a sub-tensor set input decomposition module that supports parallel computation. The tensor decomposition module combines the sub-tensor set with the model's objective function and uses iterative formulas to solve for the convergence values ​​of the target tensor block, low-rank background tensor block, and sparse background tensor block. The tensor block reconstruction module restores the convergence values ​​of the acquired target tensor block, low-rank background tensor block, and sparse background tensor block into the target image sequence and the background image sequence.

5. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the infrared image target and background separation method based on spatiotemporal tensor decomposition as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the infrared image target and background separation method based on space-time tensor decomposition as described in any one of claims 1 to 3.

Images