Cutting method based on sample pair screening and local optimization

Abstract

The present invention discloses a cutout method based on sample pair screening and local optimization, the method comprising: identifying a foreground area, a background area and a to-be-cutout area in an image to be processed; performing sample pair initialization on the image to be processed to obtain an initial cutout result; performing sample pair refinement on the initial cutout result to obtain a refined cutout result; performing local optimization processing on the refined cutout result to obtain a final cutout result. The cutout method based on sample pair screening and local optimization of the present invention adopts a sample pair screening and local iterative optimization method, has a small amount of calculation, and improves the efficiency of cutout. At the same time, since the need for computing performance is low, it can be applied to some devices with weak computing power.

Description

Technical Field

[0001] The invention relates to a cutout method based on sample pair screening and local optimization. Background Art

[0002] Cutting is a classic computer processing method that aims to smoothly separate foreground and background elements from an image or video. It is widely used in image or video editing applications, including layering, foreground extraction, background replacement, etc. Blue screen cutout used in 3D movies is the simplest cutout method, which is completed by fixing the background color to blue or green.

[0003] The popular methods nowadays are divided into three categories: sampling-based methods, continuity-based methods, and combined methods. The combined method not only considers the correlation between the foreground, background, and current pixel values, but also ensures the neighborhood continuity of opacity, so it has the best effect. Wang first adopted the strategy of color value joint sample pair screening to solve opacity, and used the random walk algorithm to ensure the continuity of the neighborhood. Considering the defects of Euclidean distance, Rhemann used geodesic distance instead, and used the Markov random field model to construct the opacity image, and optimized the final result through matrix optimization. Gastal modified and improved the sample pair screening strategy, and improved the effect of the sampling method by considering the reference results of the neighborhood sample point pairs.

[0004] However, the above solutions all have the problem of high computational complexity and cannot be used in applications such as mobile phones and real-time image cutouts. On the one hand, selecting and screening suitable sample pairs requires a large amount of search resources; on the other hand, the global optimization of the opacity matrix does produce accurate results, but it usually requires sparse solutions of matrices with millions of dimensions multiplied by millions of dimensions, which has extremely high requirements on time and space complexity. Summary of the invention

[0005] The present invention provides a cutout method based on sample pair screening and local optimization to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:

[0006] A cutout method based on sample pair screening and local optimization includes the following steps:

[0007] Identify the foreground area, the background area and the area to be cut out in the image to be processed;

[0008] Initialize the sample pairs of the image to be processed to obtain the initial cutout result;

[0009] Perform sample pair refinement on the initial cutout result to obtain a refined cutout result;

[0010] The refined cutout result is locally optimized to obtain the final cutout result.

[0011] Furthermore, the specific method for initializing the sample pairs of the image to be processed to obtain the initial cutout result is:

[0012] For the pixel points in the area to be cut out, the corresponding sample pairs are extracted from the foreground area and the background area to complete the sample pair initialization.

[0013] Furthermore, the specific method of extracting corresponding sample pairs from the foreground area and the background area to complete sample pair initialization is:

[0014] The sample pairs of pixel points in the area to be cut out are obtained through hybrid search.

[0015] Furthermore, a hybrid search is performed according to the following formula to obtain sample pairs of pixel points in the area to be cut out:

[0016]

[0017] Where i represents any pixel in the image. represents the angle of gradient search, represents the angle of uniform search, Represents the magnitude of the gradient, T a is the threshold corresponding to this amplitude value.

[0018] Furthermore, the uniform search angle Randomly select one from {0°, 20°, 40°, ..., 160°}.

[0019] Further, we calculate θ i Then according to θ i and θ i The search is performed simultaneously in the two directions of +180°, and the two nearest foreground values ​​and two background values ​​are extracted to form four sample pairs;

[0020] Calculate the confidence of each sample pair, retain the one with the highest confidence as the final sample pair for the pixel, and update it to the current pixel.

[0021] Furthermore, the specific method of performing sample pair refinement on the initial cutout result to obtain the refined cutout result is:

[0022] For each pixel in the area to be cut out, select N sample pairs from its own neighborhood;

[0023] Disassemble these N sample pairs and then recombine them to obtain NxN new sample pairs;

[0024] Calculate the confidence of each sample pair, select the sample pair with the largest confidence from these sample pairs, and update it to the current pixel.

[0025] Furthermore, the sample pair with the highest confidence is selected according to the following formula:

[0026]

[0027] Among them, σ c represents the threshold range of balanced confidence, c i Indicates the weight corresponding to the selected sample pair with the highest confidence, cr i represents the residual of the linear model, cf i Indicates the dissimilarity of foreground samples, cb i represents the dissimilarity of background samples,

[0028] in,

[0029] cr i =|I i -α′ i F i -(1-α′ i )B i | / |F i -B i |

[0030]

[0031]

[0032] α′ i =(I i -B i )(F i -B i ) / ||F i -B i || 2

[0033] Among them, I i represents the pixel value of pixel i, F i represents the foreground value at pixel i, B i Represents the background value at pixel i, I max is the threshold for judging the similarity of two color values.

[0034] Furthermore, the specific method of performing local optimization processing on the refined cutout result to obtain the final cutout result is:

[0035] Build the following Markov random field model:

[0036]

[0037] Among them, λ i is the balance coefficient between the data term and the smoothing term, represents the surrounding neighborhood of pixel i, D(α i ) is a data item, is the smoothing term,

[0038] D(α i )=|α i -α i ′|

[0039]

[0040] Among them, w ij Indicates the intimacy between pixels i and j, α i Indicates the opacity of pixel i, α j Represents the opacity of pixel j, assuming I i and observation value I i The error of ′ is n i ,but,

[0041]

[0042] The smoothness term is converted into the state noise between the optimized pixel neighborhoods, that is,

[0043]

[0044] Among them, m i represents state noise, redefines the Kalman noise system,

[0045]

[0046] Kalman denoising is used to process the above Kalman noise system to complete local optimization and obtain the final cutout result.

[0047] The invention is beneficial in that the provided cutout method based on sample pair screening and local optimization adopts the sample pair screening and local iterative optimization method, which has less computational complexity and improves the efficiency of cutout. At the same time, since the requirement for computing performance is low, it can be applied to some devices with weak computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a schematic diagram of a cutout method based on sample pair screening and local optimization of the present invention;

[0049] Figure 2 It is a schematic diagram of dividing the original image into a foreground area, a background area and an area to be cut out;

[0050] Figure 3 It is a schematic diagram comparing the global optimization and the local iterative optimization of this application. DETAILED DESCRIPTION

[0051] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] like Figure 1 The present application discloses a method for cutting out images based on sample pair screening and local optimization, comprising the following steps: S1: Identify the foreground area, background area and area to be cut out in the image to be processed. S2: Perform sample pair initialization on the image to be processed to obtain an initial cutout result. S3: Perform sample pair refinement on the initial cutout result to obtain a refined cutout result. S4: Perform local optimization processing on the refined cutout result to obtain the final cutout result. Through the above steps, the method for cutting out images based on sample pair screening and local optimization of the present application has less computational effort and improves the efficiency of cutting out images. At the same time, due to the low demand for computing performance, it can be applied to some devices with weaker computing power, such as mobile phones, tablet computers, etc. The above steps are described in detail below.

[0053] For step S1: a foreground area, a background area and an area to be cut out are identified in the image to be processed.

[0054] In the cutout problem, the pixel value I of each pixel i is i It can be understood as the foreground value F i and background value B i The weighted result is

[0055] I i =α i ·F i +(1-α i )·B i

[0056] Among them, α i It is called the opacity of the foreground, and its value range is between [0,1]. The foreground value F is calculated by inverting the pixel value I. i , background value B i and opacity α i It is a problem of solving an underdetermined equation.

[0057] The first step is to identify the pure foreground, pure background and the area to be cut out. Figure 2 As shown, the left picture is the original picture, the white in the right picture is the pure foreground, the black is the pure background, and the gray area is the area to be cut out.

[0058] For step S2: initialize the sample pairs of the image to be processed to obtain the initial cutout result.

[0059] In order to solve the underdetermined equation, our solution is to find a sample pair, that is, the foreground value F i and background value B i , then the opacity can be obtained as:

[0060]

[0061] If the appropriate foreground-background sample pairs can be selected, a rough optimization solution to the cutout problem can be achieved.

[0062] Specifically, the specific method for initializing the sample pair of the image to be processed to obtain the initial cutout result is:

[0063] For the pixel points in the area to be cut out, the corresponding sample pairs are extracted from the foreground area and the background area to complete the sample pair initialization.

[0064] In the present application, the specific method for extracting the corresponding sample pairs from the foreground area and the background area to complete the sample pair initialization is: obtain the sample pairs of pixel points of the area to be cut out through a mixed search. Specifically, in areas with uniform color values, such as hair, blue sky, white walls, etc., a uniform search method in all directions is a reasonable means. That is, with the current point as the center of the circle, rays are emitted to search for the foreground and background values. In areas with uneven color values, such as where different layers meet, a gradient-based search method can help us get an effective solution faster.

[0065] Specifically, a hybrid search is performed according to the following formula to obtain sample pairs of pixel points in the area to be cut out:

[0066]

[0067] Where i represents any pixel in the image. represents the angle of gradient search, represents the angle of uniform search, Represents the magnitude of the gradient, T a is the threshold corresponding to this amplitude value.

[0068] Among them, the uniform search angle Randomly select one from {0°, 20°, 40°, ..., 160°}.

[0069] Considering the symmetry of gradient and uniform search, in practice, the calculation of θ is i After that, according to θ i and θ i Search in both directions of +180° at the same time, extract the nearest foreground value and background value as sample pairs, then you will get two foreground values ​​and two background values ​​respectively. Extract the two nearest foreground values ​​and two background values ​​to form four sample pairs. Calculate the confidence of each sample pair, retain the one with the highest confidence as the final sample pair of the pixel, and update it to the current pixel to complete the sample pair initialization.

[0070] Preferably, the sample pair with the highest confidence is selected according to the following formula:

[0071]

[0072] Among them, σ c represents the threshold range of balanced confidence, c i Indicates the weight corresponding to the selected sample pair with the highest confidence, cr i represents the residual of the linear model, cf i Indicates the dissimilarity of foreground samples, cb i represents the dissimilarity of background samples, where

[0073] cr i =|I i -α′ i F i -(1-α′ i )B i | / |F i -B i |

[0074]

[0075]

[0076] α′ i =(I i -B i )(F i -B i ) / ||F i -B i || 2

[0077] Among them, I i represents the pixel value of pixel i, F i represents the foreground value at pixel i, B i Represents the background value at pixel i, I max is the threshold for judging the similarity of two color values. A threshold range that is primarily used to balance color values.

[0078] For step S3: perform sample pair refinement on the initial cutout result to obtain a refined cutout result.

[0079] It is understandable that the aforementioned search method is sparse, so it cannot guarantee that every pixel to be cut out can be initialized. In addition, the initialized pixel points cannot guarantee the similarity and continuity of the sample pairs, that is, sufficient information sharing is not completed. Therefore, it is necessary to refine the sample pairs of the initial cutout results.

[0080] The specific method of performing sample pair refinement on the initial cutout result to obtain the refined cutout result is:

[0081] For each pixel in the area to be cut out, select N sample pairs from its own neighborhood.

[0082] Disassemble these N sample pairs and then recombine them to obtain NxN new sample pairs.

[0083] Calculate the confidence of each sample pair, select the sample pair with the highest confidence from these sample pairs, and update it to the current pixel. Considering the localization of the above process, that is, the modification of the current pixel will affect the solution of the neighborhood, therefore, in this application, the pixels are optimized in order from high to low confidence, with the purpose that meaningful pixels should have greater influence.

[0084] For step S4: perform local optimization processing on the refined cutout result to obtain the final cutout result.

[0085] The specific method of performing local optimization processing on the refined cutout result to obtain the final cutout result is:

[0086] First, the smoothing problem of the entire opacity image is transformed into a Markov random field model. The constructed Markov random field model is as follows:

[0087]

[0088] Among them, λ i is the balance coefficient between the data term and the smoothing term, represents the surrounding neighborhood of pixel i, D(α i ) is a data item, is the smoothing term,

[0089] D(α i )=|α i -α i ′|

[0090]

[0091] Among them, w ij Indicates the intimacy between pixels i and j, α i Indicates the opacity of pixel i, α j Indicates the opacity of pixel j.

[0092] Considering the high time and space complexity brought by global optimization, in this application, the solution of the Markov random field model is transformed into an image denoising problem and completed through local iterative optimization. i and observation value I i The error of ′ is ni , then, the data item D(α i ) is rewritten as,

[0093]

[0094] Since the foreground and background values ​​are fixed, minimizing the data term is equivalent to minimizing the observation noise. The smoothing term is converted to optimizing the state noise between pixel neighborhoods, that is,

[0095]

[0096] Among them, m i Representing state noise, redefining the Kalman noise system, we get:

[0097]

[0098] Kalman denoising is used to process the above Kalman noise system to complete local optimization and obtain the final cutout result. Figure 3 As shown, the effect comparison between global optimization and local iterative optimization of this application is shown. Figure 3 a shows that the errors between the two are close. Figure 3 b shows that local iterative optimization can bring about a significant reduction in time and space complexity.

[0099] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A cutout method based on sample pair screening and local optimization, characterized in that: The following steps are involved: Identify the foreground area, the background area and the area to be cut out in the image to be processed; Initializing the sample pairs of the image to be processed to obtain an initial cutout result; The initial cutout result is subjected to sample pair refinement to obtain a refined cutout result, specifically: For each pixel point in the area to be cut out, select N sample pairs from its own neighborhood; Disassemble these N sample pairs and then recombine them to obtain NxN new sample pairs; Calculate the confidence of each sample pair according to the formula Select the sample pair with the highest confidence and update it to the current pixel, where σ c represents the threshold range of the balance confidence, c i Indicates the weight corresponding to the selected sample pair with the highest confidence, cr i represents the residual of the linear model, cf i Indicates the dissimilarity of foreground samples, cb i Indicates the dissimilarity of background samples, cr i =|I i -α′ i F i -(1-α′ i )B i | / |F i -B i |, α′ i =(I i -B i )(F i -B i ) / ||F i -B i || 2 , where I i represents the pixel value of pixel i, F i represents the foreground value at pixel i, B i Represents the background value at pixel i, I max is the threshold for judging the similarity of two color values; The refined cutout result is locally optimized to obtain a final cutout result.

2. The cutout method based on sample pair screening and local optimization according to claim 1, characterized in that: The specific method of performing sample pair initialization on the image to be processed to obtain the initial cutout result is: For the pixel points of the area to be cut out, the corresponding sample pairs are extracted from the foreground area and the background area to complete sample pair initialization.

3. The cutout method based on sample pair screening and local optimization according to claim 2, characterized in that: The specific method of extracting the corresponding sample pairs from the foreground area and the background area to complete sample pair initialization is: The sample pairs of pixel points in the area to be cut out are obtained through hybrid search.

4. The method for cutting out images based on sample pair screening and local optimization according to claim 3, characterized in that: A mixed search is performed according to the following formula to obtain sample pairs of pixels in the area to be cut out: Where i represents any pixel in the image. represents the angle of gradient search, represents the angle of uniform search, Represents the magnitude of the gradient, T a is the threshold corresponding to this amplitude value.

5. The cutout method based on sample pair screening and local optimization according to claim 4, characterized in that: Uniform search angle Randomly select one from {0°, 20°, 40°, ..., 160°}.

6. The method for cutting out images based on sample pair screening and local optimization according to claim 5, characterized in that: Calculate θ i Then according to θ i and θ i The search is performed simultaneously in the two directions of +180°, and the two nearest foreground values ​​and two background values ​​are extracted to form four sample pairs; Calculate the confidence of each sample pair, retain the one with the highest confidence as the final sample pair for the pixel, and update it to the current pixel.

7. The method for cutting out images based on sample pair screening and local optimization according to claim 6, characterized in that: The specific method of performing local optimization processing on the refined cutout result to obtain the final cutout result is: Build the following Markov random field model: Among them, λ i is the balance coefficient between the data term and the smoothing term, represents the surrounding neighborhood of pixel i, D(α i ) is a data item, is the smoothing term, D(α i )＝|α i -α i ′| Among them, w ij Indicates the intimacy between pixels i and j, α i Indicates the opacity of pixel i, α j Represents the opacity of pixel j, assuming I i and observation value I i The error of ′ is n i , according to I i =α i F i +(1-α i )B i Knowable but, The smoothness term is converted into the state noise between the optimized pixel neighborhoods, that is, Among them, m i represents state noise, redefines the Kalman noise system, Kalman denoising is used to process the above Kalman noise system to complete local optimization and obtain the final cutout result.

Images