Foreground and background separation method based on weighted f-norm robust principal component analysis

Abstract

The application relates to the technical field of image processing, in particular to a foreground and background separation method based on weighted F norm robust principal component analysis, which comprises the following steps: acquiring a video to be processed and reconstructing the video into a two-dimensional matrix; combining a weighted least square method and low-rank matrix decomposition to construct a video foreground and background separation model based on robust principal component analysis; inputting the two-dimensional matrix into the video foreground and background separation model, solving by using an alternating minimization method, obtaining two low-rank matrices U and V and a sparse matrix S, wherein the product of the two low-rank matrices U and V is taken as the background of the video to be processed, and the sparse matrix S is taken as the foreground of the video to be processed. The application has a simple solving process and is more accurate and robust.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a foreground and background separation method based on weighted F-norm robust principal component analysis. Background Technology

[0002] Robust Principal Component Analysis (RPCA) is a key technique for decomposing data into low-rank and sparse components, playing a crucial role in applications such as image processing and anomaly detection. Traditional RPCA methods typically use... Norm regularization enforces sparsity, but this method can introduce bias, especially when dealing with large noise or outliers, leading to suboptimal estimates. To address these issues, researchers have proposed non-convex regularization methods, but these methods are complex to optimize, susceptible to initial conditions, and result in unstable solutions.

[0003] Therefore, how to simplify the solution process to improve the efficiency of separating the foreground and background of the image and avoid distortion and information loss has become a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a foreground and background separation method based on weighted F-norm robust principal component analysis, which has a simpler solution process and is more accurate and robust.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] S1. Obtain the video to be processed and reconstruct it into a two-dimensional matrix;

[0007] S2. Combining weighted least squares and low-rank matrix decomposition, a video foreground-background separation model based on robust principal component analysis is constructed.

[0008] S3. Input the two-dimensional matrix into the video foreground-background separation model and solve it using the alternating minimization method to obtain two low-rank matrices U and V and a sparse matrix S. The result of multiplying the two low-rank matrices U and V is used as the background of the video to be processed, and the sparse matrix S is used as the foreground of the video to be processed.

[0009] Furthermore, the expression for the video foreground-background separation model is:

[0010]

[0011] Where Y represents the input two-dimensional matrix, || || F denoted by Frobenius norm, · denotes element-wise multiplication, W represents the weight matrix, and λ is the regularization parameter.

[0012] Furthermore, the steps for solving the video foreground-background separation model using the alternating minimization method include:

[0013] S31, Initialize parameter W 0 U 0 V 0 Iteratively update each variable along with the near-end parameter t;

[0014] S32. In the (k+1)th iteration, given W k U k V k Solve the quadratic optimization problem with respect to the sparse matrix S, and update S. k+1 ;

[0015] S33, Given S k+1 The two low-rank matrices U and V are updated using the near-end block coordinate descent method. U is updated... k+1 V k+1

[0016] S34, Given S k+1 and W k Update the weight matrix W k+1 .

[0017] Furthermore, an update to S is introduced. k+1 U k+1 V k+1 and W k+1 The steps.

[0018] In the (k+1)th iteration, S k+1 Represented as:

[0019]

[0020] By analyzing S ij Taking the derivative and setting it to zero, we derive S. ij Update formula:

[0021]

[0022] in, Y ij ,(U k V k ) ij , Representing matrix S respectively k+1 ,Y,(U k V k ),W k The value of the element at position i,j; Represents element The square of.

[0023] In the (k+1)th iteration, U k+1 The update process includes: given U k ,S k+1 V k U is updated using the proximal block coordinate descent method. k+1 :

[0024]

[0025] Where t is the proximal parameter, and the optimal condition is:

[0026] (Y-UV k -S k+1 (V) k ) T +t(UU k ) = 0.

[0027] get:

[0028] U k+1 =[tU k +(YS k+1 (V) k ) T ][V k (V k ) T +tI] -1

[0029] In the (k+1)th iteration, given S k+1 U k+1 V is updated using the proximal block coordinate descent method. k+1 :

[0030]

[0031] The optimal conditions are determined as follows:

[0032] -(U k+1 ) T ((YU k+1 VS k+1 ))+t(VV k ) = 0

[0033] get:

[0034] V k+1 =[tI+(U k+1 ) T U k+1 ] -1 [tV k +(U k+1 ) T (YS k+1 )].

[0035] In the (k+1)th iteration, given S k+1 Weight matrix W k+1 The update process includes:

[0036] Estimate t k for:

[0037]

[0038] Calculate the weight of the current step

[0039]

[0040] Where p > 0.

[0041] Update weight matrix W k+1 for:

[0042]

[0043] As can be seen from the above technical solution, compared with the prior art, the present invention has the following beneficial effects:

[0044] This invention proposes a novel video foreground-background separation model based on robust principal component analysis, combining weighted least squares and low-rank matrix factorization. It uses the weighted F-norm to represent sparse components, simplifying the solution process compared to traditional methods. The norm method reduces bias. Simultaneously, the alternating minimization algorithm, where each subproblem has an explicit solution, improves computational efficiency. Its performance and stability outperform existing non-convex regularization methods, demonstrating better accuracy and robustness in practical applications. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 A flowchart illustrating the foreground-background separation method based on weighted F-norm robust principal component analysis provided by this invention;

[0047] Figure 2 The simulation results of the method of the present invention on synthetic data;

[0048] Figure 3 This is the foreground / background separation result of the method of the present invention on a surveillance video sequence;

[0049] Figure 4 This is the result of facial shadow removal using the method of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, this invention discloses a foreground-background separation method based on weighted F-norm robust principal component analysis, comprising the following steps:

[0052] S1. Obtain the video to be processed and reconstruct it into a two-dimensional matrix;

[0053] S2. Combining weighted least squares and low-rank matrix decomposition, a video foreground-background separation model based on robust principal component analysis is constructed.

[0054] S3. Input the two-dimensional matrix into the video foreground-background separation model and solve it using the alternating minimization method to obtain two low-rank matrices U and V and a sparse matrix S. The result of multiplying the two low-rank matrices U and V is used as the background of the video to be processed, and the sparse matrix S is used as the foreground of the video to be processed.

[0055] The steps described above in this invention will be further explained below.

[0056] S1. Acquire the video to be processed and reconstruct it into a two-dimensional matrix. To achieve matrix decomposition, the three-dimensional video data is reconstructed into a two-dimensional matrix for subsequent analysis.

[0057] S2. Construct a video foreground / background separation model based on robust principal component analysis.

[0058] Principal Component Analysis (PCA) is a widely used dimensionality reduction technique that projects high-dimensional data into a low-dimensional subspace by maximizing the variance of the principal components. At its core, PCA performs Singular Value Decomposition (SVD) on the data matrix to capture its principal components. These components correspond to the directions of maximum variance. While PCA is very useful in practical applications, it assumes that the data is clean and follows a Gaussian distribution, making it highly sensitive to noise and outliers. This sensitivity poses a significant challenge in real-world applications, such as image processing and computer vision, where data is often contaminated with large errors or outliers.

[0059] To address these limitations, Robust Principal Component Analysis (RPCA) has been proposed as a more flexible and powerful variant of PCA. RPCA will take a given data matrix... It is decomposed into a low-rank matrix X, representing the underlying data structure, and a sparse matrix S, which captures noise or outliers, i.e.:

[0060] Y = X + S (1)

[0061] This method effectively separates signal from noise, thereby enhancing the robustness of data analysis. RPCA is particularly useful in areas such as image processing, background subtraction, and anomaly detection, where data is often heavily contaminated or contains sparse, high-amplitude outliers.

[0062] RPCA can be expressed as minimizing the rank of the low-rank component and the sparse component. The norm problem is essentially an optimization problem as follows:

[0063] The goal is to find the low-rank matrix that best represents the underlying structure of the data, while minimizing the number of non-zero terms in the sparse matrix that capture outliers or noise. Here, λ is the regularization parameter; rank represents the rank function, defined as the number of non-zero singular values; and ||X||0 represents the number of non-zero elements in matrix X.

[0064] However, the rank function and Norms are all nonconvex, making this optimization problem NP-hard. To circumvent this problem, a common method is to replace the rank function with the nuclear norm, using... Norm substitution Norm, then model (2) can be transformed into:

[0065]

[0066] The nuclear norm ||X|| * Defined as the sum of the singular values ​​of matrix X; The nuclear norm is the best convex substitution for the rank function, which is beneficial for low-rank properties, while Norm is The well-known convex relaxation of the norm is beneficial for sparsity. This relaxation transforms the original non-convex problem into a convex optimization problem, which can be effectively solved by existing convex optimization techniques with perfect convergence, such as the Accelerated Proximal Gradient (APG) method, the Alternating Direction Method (ALM), and the Inaccurate Enhanced Lagrange Multiplier Method (IALM).

[0067] The main limitation of model (3) is that SVD needs to be computed in each iteration. As the data dimension increases, the computational burden of SVD increases dramatically, making these methods inefficient for handling large-scale matrices. Therefore, RPCA is defined as a matrix factorization problem, the goal of which is to approximate Y by the product of two low-rank matrices U and V plus a sparse matrix S:

[0068]

[0069] in,

[0070] The common point of the above methods is the use of Norms are used as regularization terms to enforce sparsity, although Norms are favored for their convexity, making them easier to optimize and performing well in constructing sparse solutions. However, they tend to over-penalize large noise or outliers, leading to biased solutions. This bias problem becomes particularly pronounced when dealing with real-world data containing significant outliers or noise. To address this issue, some non-convex penalty functions have been introduced into decomposition-based RPCA. Consider the following non-convex optimization problem:

[0071] here, Where φ is a non-convex sparsity promotion function, for example Functions such as norm, MCP, and SCAD. With the introduction and development of nonconvex regularization techniques, RPCA has made significant progress in processing large-scale data and noisy outliers, effectively alleviating the limitations of traditional methods. It overcomes the limitations of regularization and achieves better results in fields such as image processing, video analysis, and anomaly detection.

[0072] While nonconvex regularization methods have advantages in reducing over-penalty and bias, they also have some drawbacks. First, nonconvex regularization problems are often difficult to optimize because they involve nonconvex functions, making them prone to getting trapped in local optima, increasing the complexity of algorithm design and solution. Furthermore, these methods are sometimes overly sensitive to initial values, leading to unstable solutions, especially when dealing with highly noisy data, which may hinder the attainment of a stable global optimum. To address these challenges, this invention proposes a novel method for designing a simpler model that not only avoids the problems caused by nonconvex regularization but also... The biased estimation introduced by the norm can also solve the complexity and instability problems of non-convex optimization. To this end, an RPCA model based on weighted least squares and low-rank matrix factorization, namely a video foreground-background separation model, is introduced, with the following expression:

[0073]

[0074] Where Y represents the input two-dimensional matrix, || || F Let represent the Frobenius norm, · denote element-wise multiplication, W represent the weight matrix, and λ be the regularization parameter. This model uses a weighted Frobenius norm to describe sparse components, simplifying the solution process while maintaining low-rank decomposition capabilities. Compared with traditional... Compared to norm-based methods, this approach effectively reduces bias and significantly simplifies computational complexity. The model is solved using an alternating minimization method, where all subproblems have explicit solutions, improving the algorithm's computational efficiency. Numerical experiments demonstrate that, despite the model's simplicity, it outperforms existing non-convex regularization methods in terms of experimental results.

[0075] S3. Solve using the alternating minimization method.

[0076] This step employs an alternating minimization method to solve for U, V, and S in the model. This method iteratively updates each variable while keeping other variables constant, thus decomposing the original complex optimization problem into simpler subproblems. The specific solution process includes:

[0077] Initialization parameter W 0 =I,Y=U y S y V y T U 0 =U y V 0 =V y t = 0.38, iteratively update each variable;

[0078] In the (k+1)th iteration, given U k V k W k Solve the quadratic optimization problem with respect to the sparse matrix S, and update S. k+1 :

[0079]

[0080] By analyzing S ij Taking the derivative and setting it to zero yields the following result;

[0081]

[0082] Therefore, S can be derived. ij Update formula:

[0083]

[0084] in, Y ij ,(U k V k ) ij , Representing matrix S respectively k+1 ,Y,(U k V k ),W k The value of the element at position i,j; Represents element The square of.

[0085] Given S k+1 U is updated using the proximal block coordinate descent method. k+1 V k+1 ;

[0086]

[0087] Where t>0 are the proximal parameters. Given S k+1 V k For (9), taking the derivative with respect to variable U and setting it to 0, we can obtain the optimality condition as follows:

[0088] (Y-UV k -S k+1 (V) k ) T +t(UU k )=0 (11) Simplifying, we get:

[0089] U k+1 =[tU k +(YS k+1 (V) k ) T ][V k (V k ) T +tI] -1 (12)

[0090] Similarly, (10) by taking the derivative with respect to variable U and setting it to 0, we can obtain the optimality condition as follows:

[0091] -(U k+1 ) T ((YU k+1 VS k+1 ))+t(VV k )=0 (13) Simplifying, we get:

[0092] V k+1 =[tI+(U k+1 ) T U k+1 ] -1 [tV k +(U k+1 ) T (YS k+1 (14)

[0093] Below, we will explain how to update the weight W. k Ideally, if we knew beforehand which positions in the sparse component S were zero, we could simply set the weights of those positions to 1 and the weights of the others to 0. For example, when an image is affected by impulse noise, it can be modeled as an RPCA problem, where the pixels damaged by noise are weighted 0, and the unaffected pixels are weighted 1. This approach transforms the denoising problem into a low-rank matrix completion problem. However, in most cases, we lack prior knowledge of the sparse components. Therefore, a mechanism is needed to estimate appropriate weights to identify which positions in the sparse component are zero. By setting different weights for different pixels, this method can effectively avoid over-penalizing outliers, thereby enhancing the robustness of the model. Given S k and W k Determine the weight matrix W k+1 The process is as follows:

[0094] 1) Estimate the normalization factor t k for:

[0095]

[0096] 2) Calculate the weight of the current step.

[0097]

[0098] Where p>0 is the penalty factor;

[0099] 3) Update the weight matrix W k+1 for:

[0100]

[0101] The effectiveness of the method of the present invention will be verified by simulation below.

[0102] (1) Synthetic data simulation experiment.

[0103] In this experiment, the results of model (6) in numerical experiments will be presented. First, a low-rank matrix X is generated using X = UV (see...). Figure 2 (a) in which U∈R 500×10 ,V∈R 10×500 It is a randomly generated matrix that follows a normal distribution. Next, Gaussian noise with a signal-to-noise ratio (SNR) of 10 is generated, and this noise is multiplied by a sparse mask to obtain the sparse component S (see...). Figure 2 In (b)), the sparsity of S is 26.4%. Finally, sparse noise is added to the low-rank matrix to obtain a low-rank matrix Y with sparse Gaussian noise (see [reference]). Figure 2 (c)). Subsequently, a low-rank sparse decomposition of the observation matrix Y is considered to recover the original low-rank matrix from the observations. and sparse components The root mean square error (RMSE) is used to quantitatively evaluate separation performance, and it is defined as follows:

[0104]

[0105] in, This is the separated low-rank matrix. Figure 2 The experimental results shown demonstrate in detail the effectiveness of the proposed low-rank sparse decomposition method applied to the observation matrix. For example... Figure 2 As shown in (d) in the figure, the low-rank component The recovery is very close to the true low-rank component X, with an RMSE of 7.2 × 10⁻⁶. -9 This demonstrates that the proposed method maintains high accuracy in recovering the underlying low-rank structure. Besides the low-rank matrix, the sparse components... It also accurately separates from the observation data, such as Figure 2 As shown in (e), the RMSE is 1.1 × 10⁻⁶. -8 .

[0106] (2) Simulation experiment on separation of foreground and background in surveillance video.

[0107] In this experiment, Robust Principal Component Analysis (RPCA) was used to model the surveillance video data, aiming to effectively separate the dynamic foreground (sparse component) and the stable background (low-rank component) in the video. Four surveillance video segments, each with 200 frames, were selected, so the size of the original video data can be assumed to be m×n×200.

[0108] To achieve matrix factorization, the 3D video data is first reconstructed into an mn×200 matrix for subsequent analysis. Considering the time invariance of the background image, matrix Y can be decomposed into the sum of a low-rank matrix X with rank 1 and a sparse matrix S. This decomposition process effectively distinguishes the background and foreground in the video. Subsequently, the decomposed matrices X and S are recombine column-wise to form m×n×200 data, thus successfully separating the foreground and background.

[0109] To verify the effectiveness of the method of this invention, a frame was selected as an example to demonstrate the effect of separating the low-rank matrix (background) and the sparse matrix (foreground). The experimental results are as follows: Figure 3 As shown, this invention can accurately extract dynamic foregrounds while maintaining background stability. This method provides a new solution for video surveillance applications, helping to improve the analysis accuracy and practicality of surveillance videos.

[0110] (3) Simulation experiment on removal of face shadows.

[0111] This invention presents a systematic experimental study on facial shadow removal to evaluate the effectiveness of the proposed method across various tasks. Specifically, the Yale B dataset was used, which contains 64 168×192 face images under diverse lighting conditions and at different angles. In this task, the invention utilizes a decomposition strategy of low-rank and sparse components to more effectively address the shadow problem. Specifically, low-rank components typically represent relatively constant background information in the image, such as facial contours, skin tone, and overall shape. This information does not change significantly across different frames and can therefore be represented using a low-rank matrix. In contrast, sparse components correspond to dynamically changing shadows and lighting artifacts. For example, when strong light shines on a face, it may produce noticeable shadow areas, which are typically localized and sudden. Therefore, they can be represented using a sparse matrix because these shadows do not always appear in most frames.

[0112] To perform matrix decomposition, the 64 images are stretched into vector form and then combined into a 32256×64 matrix Y. Similarly, matrix Y is decomposed into the sum of a low-rank matrix X (rank 1) and a sparse matrix S. Through this low-rank and sparse decomposition strategy, this invention effectively eliminates shadow artifacts caused by lighting conditions while preserving facial details and texture information, ensuring the naturalness and realism of the final image. Experimental results are as follows... Figure 4 As shown, the method of this invention exhibits excellent shadow removal performance on all tested face images. Especially under complex lighting conditions and different shooting angles, the algorithm can still maintain the integrity of the facial region, avoiding common distortion and information loss problems. This demonstrates that the invention possesses good robustness and adaptability in practical applications, effectively meeting the needs of fields such as face recognition and image processing.

[0113] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0114] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A foreground-background separation method based on weighted F-norm robust principal component analysis, characterized in that, Includes the following steps: S1. Obtain the video to be processed and reconstruct it into a two-dimensional matrix; S2. Combining weighted least squares and low-rank matrix decomposition, a video foreground-background separation model based on robust principal component analysis is constructed. S3. Input the two-dimensional matrix into the video foreground-background separation model and solve it using the alternating minimization method to obtain two low-rank matrices U and V and a sparse matrix S. The result of multiplying the two low-rank matrices U and V is used as the background of the video to be processed, and the sparse matrix S is used as the foreground of the video to be processed. The expression for the video foreground-background separation model is: Where Y represents the input two-dimensional matrix, || || F denoted by Frobenius norm, · denotes element-wise multiplication, W represents the weight matrix, and λ is the regularization parameter; The steps for solving the video foreground-background separation model using the alternating minimization method include: S31, Initialize parameter W 0 U 0 V 0 Iteratively update each variable along with the near-end parameter t; S32. In the (k+1)th iteration, given W k U k V k Solve the quadratic optimization problem with respect to the sparse matrix S, and update S. k+1 ; S33, Given S k+1 The two low-rank matrices U and V are updated using the near-end block coordinate descent method. U is updated... k+1 V k+1 ; S34, Given S k+1 and W k Update the weight matrix W k+1 .

2. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 1, characterized in that, In the (k+1)th iteration, S k+1 Represented as: By analyzing S ij Taking the derivative and setting it to zero, we derive S. ij Update formula: in, X ij ,(U k V k ) ij , Representing matrix S respectively k+1 ,X,(U k V k ),W k The value of the element at position i,j; Represents element The square of.

3. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 1, characterized in that, In the (k+1)th iteration, U k+1 The update process includes: Given S k+1 U is updated using the proximal block coordinate descent method: Where t is the proximal parameter, and its optimal condition is: (Y-UV k -S k+1 )(V k ) T +t(U-U k )=0 We can obtain: OR k+1 =[tU k +(YS k+1 )(V k ) T ][V k (V k ) T +tI] -1 。 4. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 1, characterized in that, In the (k+1)th iteration, V k+1 The update process includes: Given S k+1 U k+1 V is updated using the proximal block coordinate descent method: The optimal conditions are determined as follows: -(U k+1 ) T ((Y-U k+1 V-S k+1 ))+t(V-V k )=0 get: V k+1 =[tI+(U k+1 ) T IN k+1 ] -1 [TV k +(In k+1 ) T (YS k+1 )]。 5. The foreground and background separation method based on weighted F-norm robust principal component analysis according to claim 1, characterized in that, In the (k+1)th iteration, the weight matrix W k+1 The update process includes: Estimate t k for: Calculate the weight of the current step Where p>0; Update weight matrix W k+1 for:

Images