Foreground and background separation method based on weighted F norm robust principal component analysis

Abstract

The invention relates to the technical field of image processing, in particular to a foreground and background separation method based on weighted F norm robust principal component analysis, which comprises the following steps: acquiring a video to be processed, and reconstructing the video into a two-dimensional matrix; a weighted least square method and low-rank matrix decomposition are combined, and a video foreground and background separation model based on robust principal component analysis is constructed; the two-dimensional matrix is input into the video foreground and background separation model, an alternating minimization method is adopted for solving, two low-rank matrixes U and V and a sparse matrix S are obtained, the result obtained by multiplying the two low-rank matrixes U and V serves as the background of the video to be processed, and the sparse matrix S serves as the foreground of the video to be processed. The method is simple in solving process and has higher accuracy and robustness.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and more particularly to a method for separating foreground and background based on weighted F-norm robust principal component analysis. Background Art

[0002] Robust principal component analysis (RPCA) is a key technique for decomposing data into low-rank and sparse components, and plays an important role in applications such as image processing and anomaly detection. Traditional RPCA methods usually use l1 norm regularization to enforce sparsity, but this method may introduce bias, especially when dealing with large noise or outliers, which can lead to suboptimal estimates. To solve these problems, some scholars have proposed non-convex regularization methods, but these methods are complex to optimize and are easily affected by initial conditions, resulting in unstable solutions.

[0003] Therefore, how to simplify the solution process to improve the efficiency of separation of image foreground and background and avoid distortion and information loss has become a technical problem that technical personnel in this field need to solve urgently. Summary of the invention

[0004] In view of this, the present invention provides a foreground and background separation method based on weighted F-norm robust principal component analysis, which has a simpler solution process and is more accurate and robust.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] S1, obtain the video to be processed and reconstruct it into a two-dimensional matrix;

[0007] S2, combining weighted least squares method and low-rank matrix decomposition to construct a video foreground and background separation model based on robust principal component analysis;

[0008] S3. Input the two-dimensional matrix into the video foreground and background separation model, and use the alternating minimization method to solve it, so as to obtain two low-rank matrices U and V and a sparse matrix S, wherein the result obtained by multiplying the two low-rank matrices U and V is used as the background of the video to be processed, and the sparse matrix S is used as the foreground of the video to be processed.

[0009] Furthermore, the expression of the video foreground and background separation model is:

[0010]

[0011] Where Y represents the input two-dimensional matrix, || || F represents the Frobenius norm, · represents element-by-element multiplication, W represents the weight matrix, and λ is the regularization parameter.

[0012] Furthermore, the step of solving the video foreground-background separation model using an alternating minimization method includes:

[0013] S31, initialization parameter W 0 ,U 0 ,V 0 and proximal parameter t, iteratively update each variable;

[0014] S32. At the k+1th iteration, given W k ,U k ,V k , solve the quadratic optimization problem about the sparse matrix S, and update S k+1 ;

[0015] S33, given S k+1 , use the proximal block coordinate descent method to update the two low-rank matrices U and V, and update U k+1 、V k+1

[0016] S34, given S k+1 and W k , update the weight matrix W k+1 .

[0017] Further, we introduce the update S k+1 , U k+1 ,V k+1 and W k+1 steps.

[0018] At the k+1th iteration, S k+1 It is expressed as:

[0019]

[0020] Through S ij Taking the derivative and setting it to zero, we can deduce S ij The update formula is:

[0021]

[0022] in, Y ij ,(U k V k ) ij , Respectively represent the matrix S k+1 ,Y,(U k V k ),W k The element value at the i,j position of ; Representation Elements The square of .

[0023] At the k+1th iteration, U k+1 The update process includes: given U k ,S k+1 ,V k , use the proximal block coordinate descent method to update U k+1 :

[0024]

[0025] Among them, t is the proximal parameter, and the optimal condition is determined as:

[0026] (Y-UV k -S k+1 )(V k ) T +t(UU k )=0.

[0027] get:

[0028] U k+1 =[tU k +(YS k+1 )(V k ) T ][V k (V k ) T +tI] -1

[0029] At the k+1th iteration, given S k+1 ,U k+1 , use the proximal block coordinate descent method to update V k+1 :

[0030]

[0031] The optimal conditions are determined as:

[0032] -(U k+1 ) T ((YU k+1 VS k+1 ))+t(VV k )=0

[0033] get:

[0034] V k+1 =[tI+(U k+1 ) T U k+1 ] -1 [tV k +(U k+1 ) T (YS k+1 )].

[0035] At the k+1th iteration, given S k+1 Weight matrix W k+1 The update process includes:

[0036] Estimation of t k for:

[0037]

[0038] Calculate the weight of the current step

[0039]

[0040] Among them, p>0.

[0041] Update the weight matrix W k+1 for:

[0042]

[0043] It can be seen from the above technical solutions that, compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention proposes a novel video foreground and background separation model based on robust principal component analysis that combines weighted least squares and low-rank matrix decomposition. The weighted F-norm is used to represent sparse components, which simplifies the solution process and reduces the deviation compared with the traditional l1-norm method. At the same time, an alternating minimization algorithm is adopted, in which each subproblem has an explicit solution, thereby improving the computational efficiency. Its performance and stability are better than the existing non-convex regularization method, and it shows better accuracy and robustness in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A flowchart of the foreground and background separation method based on weighted F-norm robust principal component analysis provided by the present invention;

[0047] Figure 2 is the simulation result of the method of the present invention on synthetic data;

[0048] Figure 3 The foreground and background separation results of the monitoring video sequence by the method of the present invention;

[0049] Figure 4This is the result of removing facial shadows using the method of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, the embodiment of the present invention discloses a method for separating foreground and background based on weighted F-norm robust principal component analysis, comprising the following steps:

[0052] S1, obtain the video to be processed and reconstruct it into a two-dimensional matrix;

[0053] S2, combining weighted least squares method and low-rank matrix decomposition to construct a video foreground and background separation model based on robust principal component analysis;

[0054] S3. Input the two-dimensional matrix into the video foreground and background separation model, and use the alternating minimization method to solve it, and obtain two low-rank matrices U and V and a sparse matrix S, where the result of multiplying the two low-rank matrices U and V is used as the background of the video to be processed, and the sparse matrix S is used as the foreground of the video to be processed.

[0055] Below, the above steps of the present invention are further described.

[0056] S1. Obtain the video to be processed and reconstruct it into a two-dimensional matrix. In order to realize matrix decomposition, the three-dimensional video data is reconstructed into a two-dimensional matrix for subsequent analysis.

[0057] S2. Construct a video foreground and background separation model based on robust principal component analysis.

[0058] Principal Component Analysis (PCA) is a widely used dimensionality reduction technique that projects high-dimensional data into a low-dimensional subspace by maximizing the variance of the principal components. The core of PCA lies in performing a singular value decomposition (SVD) on the data matrix to capture its principal components. These components correspond to the directions of maximum variance. Although PCA is very useful in real-world applications, it assumes that the data is clean and follows a Gaussian distribution, which makes it highly sensitive to noise and outliers. This sensitivity poses a huge challenge in real-world applications, such as image processing and computer vision, where data is often contaminated by large errors or outliers.

[0059] To address these limitations, robust principal component analysis (RPCA) was proposed as a more flexible and powerful variant of PCA. RPCA takes a given data matrix It is decomposed into a low-rank matrix X, representing the underlying data structure, and a sparse matrix S, capturing noise or outliers, i.e.:

[0060] Y=X+S (1)

[0061] This approach effectively separates signal from noise, thereby enhancing the robustness of data analysis. RPCA is particularly useful in areas such as image processing, background subtraction, and anomaly detection, where data are often heavily contaminated or contain sparse high-amplitude outliers.

[0062] RPCA can be formulated as a problem of minimizing the rank of the low-rank component and the l0 norm of the sparse component, that is, the following optimization problem:

[0063]

[0064] Its purpose is to find a low-rank matrix that best represents the underlying structure of the data while minimizing the number of non-zero entries in the sparse matrix that capture outliers or noise. Where λ is the regularization parameter; rank represents the rank function, defined as the number of non-zero singular values; ||X||0 represents the number of non-zero elements in the matrix X.

[0065] However, both the rank function and the l0 norm are non-convex, which makes this optimization problem an NP-hard problem. To circumvent this problem, a common method is to replace the rank function with the nuclear norm and the l0 norm with the l1 norm, then model (2) can be transformed into:

[0066]

[0067] Where the nuclear norm ||X|| * Defined as the sum of the singular values ​​of the matrix X; The nuclear norm is the best convex substitute for the rank function, favoring low rank, while the l1 norm is a well-known convex relaxation of the l0 norm, favoring sparsity. This relaxation transforms the original non-convex problem into a convex optimization problem, which can be efficiently solved by existing convex optimization techniques with well-established convergence properties, such as the Accelerated Proximal Gradient Method (APG), the Alternating Direction Method (ALM), the Inexact Augmented Lagrangian Multiplier Method (IALM), etc.

[0068] The main limitation of model (3) is that SVD needs to be calculated at each iteration. As the data dimension increases, the computational burden of SVD increases dramatically, making these methods unable to efficiently process large matrices. Therefore, RPCA is defined as a matrix decomposition problem, whose goal is to approximate Y by the product of two low-rank matrices U and V plus a sparse matrix S:

[0069]

[0070] in,

[0071] The common point of the above methods is the use of l1 norm as a regularization term to enforce sparsity. Although l1 norm is favored for its convexity, which makes it easier to optimize and performs well in constructing sparse solutions, it tends to over-penalize large noise or outliers, resulting in biased solutions. This bias problem becomes particularly evident when dealing with real data containing significant outliers or noise. To solve this problem, some non-convex penalty functions are introduced into decomposition-based RPCA. Consider the following non-convex optimization problem:

[0072]

[0073] here, where φ is a non-convex sparse promotion function, such as l p norm, MCP, SCAD and other functions. With the introduction and development of non-convex regularization technology, RPCA has made significant progress in processing large-scale data and noise outliers, effectively alleviating the limitations of traditional l1 regularization, and achieving better results in image processing, video analysis and anomaly detection.

[0074] Although non-convex regularization methods have advantages in reducing over-penalty and bias, they also have some disadvantages. First, non-convex regularization problems are usually difficult to optimize because they involve non-convex functions and are prone to fall into local optimality, which increases the complexity of algorithm design and solution. In addition, these methods are sometimes too sensitive to initial values, resulting in unstable solutions, especially when dealing with high-noise data, which may hinder the acquisition of a stable global optimal solution. In order to meet the challenges, the present invention proposes a new method to design a simpler model, which not only avoids the biased estimation introduced by the l1 norm, but also solves the complexity and instability problems of non-convex optimization. To this end, an RPCA model based on weighted least squares and low-rank matrix decomposition is introduced, namely the video foreground background separation model, which is expressed as:

[0075]

[0076] Where Y represents the input two-dimensional matrix, |||| Frepresents the Frobenius norm, · represents element-by-element multiplication, W represents the weight matrix, and λ is the regularization parameter. The model uses the weighted Frobenius norm to describe the sparse components, which simplifies the solution process while maintaining the low-rank decomposition capability. Compared with the traditional l1 norm, this method effectively reduces the deviation and significantly simplifies the computational complexity. The model is solved using the alternating minimization method, in which all subproblems have explicit solutions, which improves the computational efficiency of the algorithm. Numerical experiments show that despite the simplicity of the model, it outperforms existing non-convex regularization methods in terms of experimental results.

[0077] S3. Solve by alternating minimization method.

[0078] This step uses the alternating minimization method to solve U, V, and S in the model. This method iteratively updates each variable by fixing other variables, and can decompose the original complex optimization problem into simpler sub-problems. The specific solution process includes:

[0079] Initialization parameter W 0 =I,Y=U y S y V y T ,U 0 =U y ,V 0 =V y , t=0.38, iteratively update each variable;

[0080] At the k+1th iteration, given U k ,V k ,W k , solve the quadratic optimization problem about the sparse matrix S, and update S k+1 :

[0081]

[0082] Through S ij Taking the derivative and setting it to zero gives;

[0083]

[0084] Therefore, it can be deduced that S ij The update formula is:

[0085]

[0086] in, Y ij ,(U k V k ) ij , Respectively represent the matrix S k+1 ,Y,(Uk V k ),W k The element value at the i,j position of ; Representation Elements The square of .

[0087] Given S k+1 , use the proximal block coordinate descent method to update U k+1 ,V k+1 ;

[0088]

[0089] Where t>0 is the proximal parameter. Given S k+1 ,V k , taking the derivative of (9) with respect to the variable U and setting it to 0, we can obtain the optimality condition:

[0090] (Y-UV k -S k+1 )(V k ) T +t(UU k )=0 (11)

[0091] After finishing, we can get:

[0092] U k+1 =[tU k +(YS k+1 )(V k ) T ][V k (V k ) T +tI] -1 (12)

[0093] Similarly, taking the derivative of (10) with respect to the variable U and setting it to 0 gives the optimality condition:

[0094] -(U k+1 ) T ((YU k+1 VS k+1 ))+t(VV k )=0 (13)

[0095] After finishing, we can get:

[0096] V k+1 =[tI+(U k+1 ) T U k+1 ] -1 [tV k +(U k+1 ) T (YS k+1)] (14)

[0097] Next, we introduce how to update the weight W. k . Ideally, if we know in advance which positions in the sparse component S are zero, we can simply set the weights of these positions to 1 and the weights of other positions to 0. For example, when an image is interfered by impulse noise, it can be modeled as an RPCA problem, where the weights of pixels damaged by noise are set to 0, while the weights of unaffected pixels are set to 1. This approach transforms the denoising problem into a low-rank matrix completion problem. However, in most cases, we do not have prior knowledge of the sparse components. Therefore, it is necessary to design a mechanism to estimate appropriate weights to identify which positions in the sparse components are zero. By setting different weights for different pixels, this method can effectively avoid excessive punishment of outliers, thereby enhancing the robustness of the model. Given S k and W k , determine the weight matrix W k+1 The process is as follows:

[0098] 1) Estimate the normalization factor t k for:

[0099]

[0100] 2) Calculate the weight of the current step

[0101]

[0102] Among them, p>0 is the penalty factor;

[0103] 3) Update the weight matrix W k+1 for:

[0104]

[0105] Next, the effect of the method of the present invention is verified by simulation.

[0106] (1)Synthetic data simulation experiment.

[0107] In this experiment, we will show the results of model (6) in numerical experiments. First, we use X = UV to generate a low-rank matrix X (see Figure 2 (a)), where U∈R 500×10 ,V∈R 10×500 is a randomly generated matrix following a normal distribution. Next, a Gaussian noise with a signal-to-noise ratio (SNR) of 10 is generated and multiplied by the sparse mask to obtain the sparse component S (see Figure 2(b) in the figure), where the sparsity of S is 26.4%. Finally, the sparse noise is added to the low-rank matrix to obtain the low-rank matrix Y with sparse Gaussian noise (see Figure 2 (c) in ). Then, consider the low-rank sparse decomposition of the observation matrix Y to recover the original low-rank matrix from the observations and sparse components The root mean square error (RMSE) was used to quantitatively evaluate the separation performance, which is defined as:

[0108]

[0109] in, is the separated low-rank matrix. Figure 2 The experimental results shown in detail demonstrate the effectiveness of the proposed low-rank sparse decomposition method applied to the observation matrix. Figure 2 As shown in (d), the low-rank component The recovery is very close to the true low-rank component X, with an RMSE of 7.2×10 -9 , which shows that the proposed method maintains high accuracy in recovering the underlying low-rank structure. In addition to the low-rank matrix, the sparse components is also accurately separated from the observed data, e.g. Figure 2 As shown in (e), the RMSE is 1.1×10 -8 .

[0110] (2) Simulation experiment of surveillance video foreground and background separation.

[0111] In this experiment, robust principal component analysis (RPCA) is used to model the surveillance video data, aiming to effectively separate the dynamic foreground (sparse component) and stable background (low-rank component) in the video. Four surveillance videos of 200 frames each are selected, so it can be assumed that the size of the original video data is m×n×200.

[0112] In order to achieve matrix decomposition, the 3D video data is first reconstructed into an mn×200 matrix for subsequent analysis. Considering the temporal invariance of the background image, the matrix Y can be decomposed into a low-rank matrix X with a rank of 1 and a sparse matrix S. This decomposition process can effectively distinguish the background and foreground in the video. Subsequently, the decomposed matrices X and S are recombined by column to form m×n×200 data, thus successfully achieving the separation of foreground and background.

[0113] In order to verify the effectiveness of the method of the present invention, a frame is selected as an example to show the effect of separating the low-rank matrix (background) and the sparse matrix (foreground). The experimental results are shown in Figure 3As shown, the present invention can accurately extract the dynamic foreground and maintain the stability of the background. The method provides a new solution for the application in the field of video surveillance, which helps to improve the analysis accuracy and practicality of surveillance videos.

[0114] (3) Simulation experiment of face shadow removal.

[0115] The present invention conducts systematic experiments on the task of facial shadow removal, aiming to evaluate the effectiveness of the method of the present invention in different tasks. Specifically, the Yale B dataset is used, which contains 64 168×192 face images under various lighting conditions and different angles. In this task, the present invention uses a decomposition strategy of low-rank and sparse components to more effectively deal with the shadow problem. Specifically, low-rank components usually represent relatively constant background information in the image, such as the contour, skin color, and overall shape of the face. This information does not change much in different frames, so it can be represented by a low-rank matrix. In contrast, sparse components correspond to dynamically changing shadows and lighting artifacts. For example, when strong light shines on the face, obvious shadow areas may be generated, and these shadows are usually local and sudden. Therefore, they can be represented by sparse matrices because these shadows do not always appear in most frames.

[0116] In order to perform matrix decomposition, these 64 images are pulled into vector form and then form a 32256×64 matrix Y. Similarly, we need to decompose the matrix Y into the sum of a low-rank matrix X with a rank of 1 and a sparse matrix S. Through this low-rank and sparse decomposition strategy, the present invention can effectively eliminate shadow artifacts caused by lighting conditions while retaining facial details and texture information, ensuring the naturalness and realism of the final image. The experimental results are shown in Figure 2. Figure 4 As shown, it can be shown that the method of the present invention has shown excellent shadow removal effects on all tested face images. In particular, under the conditions of complex lighting changes and different shooting angles, the algorithm can still maintain the integrity of the facial area and avoid common distortion and information loss problems. This shows that the present invention has good robustness and adaptability in practical applications and can effectively meet the needs of fields such as face recognition and image processing.

[0117] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0118] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A foreground and background separation method based on weighted F-norm robust principal component analysis, characterized in that: The following steps are involved: S1, obtain the video to be processed and reconstruct it into a two-dimensional matrix; S2, combining weighted least squares method and low-rank matrix decomposition to construct a video foreground and background separation model based on robust principal component analysis; S3. Input the two-dimensional matrix into the video foreground and background separation model, and use the alternating minimization method to solve it, so as to obtain two low-rank matrices U and V and a sparse matrix S, wherein the result obtained by multiplying the two low-rank matrices U and V is used as the background of the video to be processed, and the sparse matrix S is used as the foreground of the video to be processed.

2. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 1 is characterized in that: The expression of the video foreground background separation model is: Where Y represents the input two-dimensional matrix, || || F represents the Frobenius norm, · represents element-by-element multiplication, W represents the weight matrix, and λ is the regularization parameter.

3. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 2 is characterized in that: In S3, the step of solving the video foreground-background separation model using an alternating minimization method includes: S31, initialization parameter W 0 ,U 0 ,V 0 and proximal parameter t, iteratively update each variable; S32. At the k+1th iteration, given W k ,U k ,V k , solve the quadratic optimization problem about the sparse matrix S, and update S k+1 ; S33, given S k+1 , use the proximal block coordinate descent method to update the two low-rank matrices U and V, and update U k+1 、V k+1 ; S34, given S k+1 and W k , update the weight matrix W k+1 .

4. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 3 is characterized in that: At the k+1th iteration, S k+1 It is expressed as: Through S ij Taking the derivative and setting it to zero, we can deduce S ij The update formula is: in, X ij ,(U k V k ) ij , Represents the matrix S k+1 ,X,(U k V k ),W k The element value at the i,j position of ; Representation elements The square of .

5. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 3 is characterized in that: At the k+1th iteration, U k+1 The update process includes: Given S k+1 , update U using the proximal block coordinate descent method: Among them, t is the proximal parameter, and its optimal condition is: (Y-UV k -S k+1 )(V k ) T +t(U-U k )=0 You can get: OR k+1 =[tU k +(YS k+1 )(V k ) T ][V k (V k ) T +tI] -1 。 6. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 3 is characterized in that: At the k+1th iteration, V k+1 The update process includes: Given S k+1 ,U k+1 , update V using the proximal block coordinate descent method: The optimal conditions are determined as: -(U k+1 ) T ((Y-U k+1 V-S k+1 ))+t(V-V k )=0 get: V k+1 =[tI+(U k+1 ) T IN k+1 ] -1 [TV k +(In k+1 ) T (YS k+1 )]。 7. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 3 is characterized in that: At the k+1th iteration, the weight matrix W k+1 The updating process includes: estimating t k for: Calculate the weight of the current step Among them, p>0; Update the weight matrix W k+1 for:

Images