A self-iterative local green screen matting method

Abstract

The present invention provides a self-iterative local green screen matting method, including a pre-processing stage and a formal operation stage; the pre-processing stage uses an iterative method to obtain as accurate a green screen area as possible to obtain a better camera posture and accurate color key; the formal operation stage is to perform realistic virtual scene synthesis and matting; (1) During the whole process, the positions of the camera and the green screen are fixed; (2) In the pre-processing stage, there are no foregrounds such as people and objects in front of the green screen, while in the formal operation stage, foregrounds such as people and objects will be added in front of the green screen for matting; Compared with the prior art, the present invention breaks through the limitations of traditional color key matting, is not constrained by the shooting environment, has low cost, and can simultaneously perform accurate camera posture estimation and pixel-level matting in a natural environment. The algorithm has the advantages of being automatic, practical, fast, and highly accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of green screen matting methods, and in particular to a self-iterative local green screen matting method. Background Art

[0002] In today's film and television industry, Chroma-Key imagematting is often used. This technology requires performers to perform in a green box (a semi-enclosed space with a pure green background). The algorithm picks up the typical green color in the background and sets it as the Chroma-Key to distinguish the foreground area (performers or devices) from the background area (green wall). After accurately detecting all the pixel areas belonging to the background in the camera's field of view, the algorithm can replace the current green background with any specified scene to achieve the purpose of enriching visual expression at a controllable cost.

[0003] In recent years, with the promotion of virtual reality and mixed reality applications, the foreground after color keying often needs to be integrated with the 3D virtual scene. In this case, the system needs to estimate the camera pose. Traditional camera pose estimation is achieved through mechanical equipment and sensors. Camera pose estimation is a prerequisite for generating mixed reality images in 3D scenes. The more accurate the camera pose estimation is, the stronger the sense of immersion of the synthesized mixed reality will be.

[0004] The disadvantages of traditional chroma key matting are that camera pose estimation and matting must be completed separately. Camera pose estimation relies on mechanical equipment, which is costly and has high requirements for the shooting environment of matting. The green backplane must cover the entire field of view of the camera, otherwise the natural scene before synthesis will appear around the virtual scene after synthesis. Summary of the invention

[0005] In order to solve the above technical problems, the present invention adopts a self-iterative local green screen area matting method, which can rely on the algorithm to simultaneously complete the two tasks of camera posture estimation and matting, complete green screen matting in a natural environment without affecting the subsequent virtual scene synthesis, and obtain accurate matting results and realistic virtual scene synthesis after the matting is completed. The algorithm is not limited by the shooting environment, does not rely on mechanical equipment, is low-cost, and is suitable for low-cost live broadcast rooms. It has the advantages of being automatic, practical, and fast. A self-iterative local green screen matting method is provided to solve the problems described in the above background technology.

[0006] The invention discloses a self-iterative local green screen matting method, which includes a pre-processing stage and a formal operation stage; the pre-processing stage uses an iterative method to obtain a green screen area as accurate as possible to obtain a better camera posture and a precise color key; the formal operation stage is to perform realistic virtual scene synthesis and matting;

[0007] (1) The positions of the camera and green screen are fixed throughout the process;

[0008] (2) In the pre-processing stage, there are no foreground people or objects in front of the green screen, while in the formal operation stage, people or objects will be added in front of the green screen for keying.

[0009] The specific steps are as follows:

[0010] The first step is to use the reference frame F R The key point set on F t Perform point matching to find the current frame F t The corresponding key point set.

[0011] The second step is to obtain the homography matrix H based on the geometric positions of the two point sets. t , the estimated green screen area B and pose estimation are calculated.

[0012] In the third and fourth steps, first perform color key extraction on area B, then calculate the α value according to the formula to obtain the green screen area mask Α t .

[0013] Step 5: t Obtain edge processing and Hough transform to determine the boundary of the area, and obtain a new green screen quadrilateral area B" and a new homography matrix At the same time, a more accurate camera pose is calculated.

[0014] In the sixth step, repeat the above steps, continuously update the homography matrix, and dynamically adjust the hotspot area until the search range ε ​​of the point matching is less than the given preset value, or the number of iteration steps reaches the preset upper limit, and the accurate camera posture and color key can be obtained.

[0015] In the seventh step, the virtual scene is placed based on the camera pose obtained in the preprocessing stage.

[0016] In the eighth step, each frame is keyed at the pixel level based on the precise color key obtained in the preprocessing stage.

[0017] Beneficial effects:

[0018] 1. The key point of this patent is to perform accurate camera pose estimation and pixel-level matting at the same time, update the homography matrix in an iterative manner, dynamically adjust the hot spot area, and improve the accuracy of the color key. Compared with the traditional matting method, it can obviously obtain accurate camera pose estimation and good matting quality;

[0019] 2. Compared with the prior art, the present invention breaks through the limitations of traditional color key matting, is not restricted by the shooting environment, has low cost, and can simultaneously perform accurate camera posture estimation and pixel-level matting in a natural environment. The algorithm has the advantages of being automatic, practical, fast, and highly accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A green screen and camera view diagram are shown;

[0021] Figure 2 A point matching graph is shown;

[0022] Figure 3 The estimated green screen quadrilateral area is shown;

[0023] Figure 4 The matting effect diagram is shown (the black area represents the green screen area detected by the algorithm);

[0024] Figure 5 Shows Figure 4 Edge graph after processing by Canny operator;

[0025] Figure 6 Shows Figure 5 Hough transform graph (voting results in the straight line parameter space);

[0026] Figure 7 An overall flow chart is shown. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] Specific steps:

[0029] The present invention discloses a self-iterative local green screen matting method, which includes a pre-processing stage and a formal operation stage; the pre-processing stage uses an iterative method to obtain as accurate a green screen area as possible (such as Figure 1 ) to obtain better camera posture and accurate color key; the official operation stage is to perform realistic virtual scene synthesis and matte;

[0030] (1) The positions of the camera and green screen are fixed throughout the process;

[0031] (2) In the pre-processing stage, there are no foreground people or objects in front of the green screen, while in the formal operation stage, people or objects will be added in front of the green screen for keying.

[0032] The specific steps are as follows:

[0033] The first step is to use the reference frame F R The key point set on F t Perform point matching to find the current frame Ft The corresponding key point set;

[0034] Step 1 point matching: current frame F t , is a green screen image without foreground taken by a camera. Reference frame F R , is a standard green screen image obtained in advance. The edges, endpoints and key points of the green screen area in the image are all known.

[0035] Assume that, currently in the reference frame F R There are key point sets Contains N key points. Each key point The 2D pixel position is The image features are And the homography matrix estimated in the previous step is H t-1 (When t=1, H t-1 =I, i.e., the unit matrix), corresponding to the current frame F t There are search domains on:

[0036]

[0037] Among them, ||·|| is the bi-norm (Euclidean distance) function, and ε is the radius of the search range. Note that for the region In the matching algorithm, the key point is determined as:

[0038]

[0039] in, is the current frame F t The upper 2D position is The image features of the key points.

[0040] In the first point matching process, H t-1 is the unit matrix. To search for the key points corresponding to the key points of the reference frame in the current frame, the search range ε ​​needs to be set to infinity;

[0041] Among them, key points are visually significant points on the image, such as ORB or SIFT points, or manually pre-defined 2D projections of certain 3D points on real objects on the image, such as the center point of a car tire, the end point of a motorcycle handlebar, etc.;

[0042] Point matching is the process of matching the key points of the same object in two pictures with different perspectives, and point matching can use SIFT, SURF or deep learning algorithms.

[0043] The second step is to obtain the homography matrix H based on the geometric positions of the two point sets. t , calculate the estimated green screen area B and pose estimation;

[0044] Step 2 Calculate the homography matrix: Assume that after point matching, in the current frame F t Find the corresponding reference frame key point set Key point set We also get a matching graph with several key point pairs, such as Figure 2 shown.

[0045] F t The green screen area can be obtained by the coordinate transformation matrix H between the two frames. t Get. t Also called homography matrix, specifically, assuming F R Upper key point Corresponding to the current frame F t Point on And their 2D coordinates are and Then the homography matrix between the two frames is satisfy:

[0046]

[0047] According to the coordinate positions of the two key point sets, H is calculated by solving the multivariate equations. t , we can also use the RANSAC method to improve H t Robustness of the estimation. In the estimated homography matrix H t Based on , the algorithm projects the coordinates of the four vertices of the green screen in the reference frame onto the current frame to obtain an estimate B of the area occupied by the green screen in the current frame, such as Figure 3 As shown. The positional relationship between the camera and the green screen, i.e. the camera posture, is estimated through the three-dimensional object imaging formula.

[0048] Among them, the homography matrix is ​​a 3×3 matrix that describes the correspondence between the key points of the same planar object in two pictures with different perspectives.

[0049] In the third and fourth steps, first perform color key extraction on area B, then calculate the α value according to the formula to obtain the green screen area mask Α t ;

[0050] Step 3 Color key: The estimated green screen area B should be a quadrilateral (may be truncated by the edge of the image). It can be seen that the analytical equation of the four sides is:

[0051]

[0052] Due to the phenomenon of key point mismatch, the estimated H t It is not very accurate, resulting in a large difference between the obtained green screen area B and the actual green screen area, and the following steps are needed to correct it.

[0053] Take the color key in area B and perform keying. First, convert the RGB values ​​of the pixels in area B into YUV values ​​using the following formula:

[0054]

[0055] The UV values ​​of all pixels in the area can be obtained, and then the color key of the pixels in the area can be obtained. The color key can be obtained using the K-Means clustering method.

[0056] Step 4α value estimation: The basic idea of ​​obtaining the α value is to solve the equation using certain prior conditions:

[0057] I z =α z F+(1-α z )B'

[0058] Among them, α z =1 indicates pixel I z Belongs to the foreground F; α z =0 means pixel I z Background B B'; ​​if α z ∈(0, 1), then it means pixel I z It is a mixture of foreground and background. So after getting the color key, we need to estimate the alpha value to get the foreground and background. The alpha value is obtained by the following formula:

[0059]

[0060] Among them, I={(u1, v1), (u2, v2),...(u z , v z )...(u n , v n )} is the U and V value set of the current frame pixel, (u i , v i ) is the UV value of the color key obtained in the previous step, and m and n are hyperparameters. Since the α value range is [0,1], the result d of the above formula is input into the Sigmoid function and mapped on the [0,1] interval to obtain the α value of each pixel in the current frame, and the frame pixels are divided into three categories: foreground F, background B' and mixed pixel set M, which is the green screen area mask Α t ;

[0061] Among them, the α value is the specific task of matting, which is to determine the α value of each pixel. The α value of the pixel point is in [0,1]. α=0 represents the background area, α=1 represents the foreground area, and 0<α<1 represents the edge mixed area of ​​the foreground and background.

[0062] The alpha value of a color key pixel is determined by the relationship between the color of the pixel and the color of the color key. The alpha values ​​of pixels close to the color of the color key are set to 0.

[0063] Step 5: t Obtain edge processing and Hough transform to determine the boundary of the area, and obtain a new green screen quadrilateral area B" and a new homography matrix At the same time, a more accurate camera pose is calculated;

[0064] Step 5 Extract the straight line: After getting Α t Afterwards, t Perform Canny operator to obtain edge processing (such as Figure 4 As shown), perform Hough transform (as Figure 5 As shown in Figure 2), by voting on the line parameter space, the analytical equations of the main lines in the image space {l'1, l'2, l'3, ..., l' Q}. In this set, corresponding to each of the four estimated edge lines l above i , and propose a line that is closest to it, so the algorithm obtains 4 edge lines estimated by the Hough algorithm (such as Figure 6 As shown), these four edge lines also determine a quadrilateral, which is the new green screen quadrilateral area B". Using the four vertex positions determined by the quadrilateral B", the new homography matrix H' can be estimated. t , for H' t and H t Find the mean, the algorithm obtains the updated homography matrix (Note that there is a special calculation formula here, not a simple algebraic mean). Estimate more accurate camera pose through 3D object imaging formula;

[0065] Among them, Hough transform is a method for detecting straight lines in an image. By voting in the parameter space, the position and angle of the main straight lines in the image (i.e., the straight line parameters) are determined.

[0066] Step 6: Repeat the above steps, continuously update the homography matrix, and dynamically adjust the hotspot area until the search range ε ​​of the point matching is less than the given preset value, or the number of iteration steps reaches the preset upper limit, and then the accurate camera posture and color key can be obtained;

[0067] Step 6 Iteration: The next iteration is still for the current frame F t With reference frame F R Perform point matching and convert the homography matrix estimated in the previous step As the H of the current frame t-1 , and the key points of the reference frame are known. In the search process of point matching, due to H t-1The hotspot area obtained by calculating the homography matrix will be closer to the actual green screen area, and the camera posture estimation will be more accurate, which improves the accuracy of the color key. Then the color key is taken and the α value is calculated to optimize the green screen area mask Α t , after another Hough transform update H t Finally, a more accurate green screen area and posture estimation is obtained, thus achieving the purpose of iteratively optimizing the camera posture estimation and green screen estimation area.

[0068] Repeat steps 1-5 to continuously update H t , gradually narrow the point matching search range, correct the estimated camera pose, dynamically adjust the hotspot area, and improve the accuracy of the color key until the search range ε ​​is less than the preset value, or the number of iteration steps reaches the preset upper limit. After several rounds of iterations, the accurate camera pose and color key can be obtained;

[0069] Among them, the hot spot area refers to the estimated area of ​​the green screen;

[0070] Among them, YUV value conversion is compared to RGB value. YUV value separates color and brightness. Color is only related to UV value, which is more conducive to color-based keying, so conversion is required.

[0071] Step 7: Place the virtual scene based on the camera pose obtained in the preprocessing stage;

[0072] Based on the precise camera posture obtained in the preprocessing stage, the current image is cut out and placed in the virtual 3D space to achieve a virtual-real combination effect.

[0073] Step 8: Perform pixel-level keying on each frame based on the precise color key obtained in the preprocessing stage;

[0074] Use the precise color key (or keys) obtained in the pre-processing stage to key each frame to obtain high-quality keying results.

Claims

1. A self-iterative local green screen keying method, including a pre-processing stage and a formal operation stage; the pre-processing stage uses an iterative method to obtain the most accurate green screen area to obtain a better camera posture and accurate color key; The official operation stage is to carry out realistic virtual scene synthesis and keying; (1) The positions of the camera and green screen are fixed throughout the process; (2) In the pre-processing stage, there are no people or objects in front of the green screen, but in the formal operation stage, people and objects will be added in front of the green screen for keying; It is characterized in that the specific steps are as follows: The first step is to use the reference frame F R The key point set on F t Perform point matching to find the current frame F t The corresponding key point set; Step 1 point matching: current frame F t , is a green screen image without foreground taken by a camera, reference frame F R , is a standard green screen image obtained in advance, in which the edges, endpoints and comparison key points of the green screen area are all known; Assume that, currently in the reference frame F R There are key point sets Contains N key points, each key point The 2D pixel position is The image features are And the homography matrix estimated in the previous step is H t-1 , when t=1, H t-1 =I, which is the unit matrix, corresponding to the current frame F t There are search domains on: Among them, ||·|| is the bi-norm function, ε is the radius of the search range, and note that for the In the matching algorithm, the key point is determined as: in, is the current frame F t The upper 2D position is Image features of key points; In the first point matching process, H t-1 is the unit matrix. To search for the key points corresponding to the key points of the reference frame in the current frame, the search range ε ​​needs to be set to infinity; The second step is to obtain the homography matrix H based on the geometric positions of the two point sets. t , calculate the estimated green screen area B and pose estimation; Step 2 Calculate the homography matrix: Assume that after point matching, in the current frame F t Find the corresponding reference frame key point set Key point set A matching graph with several key point pairs is obtained; F t The green screen area is transformed by the coordinate transformation matrix H between the two frames. t Get, H t Also called homography matrix, specifically, assuming F R Upper key point Corresponding to the current frame F t Point on And their 2D coordinates are and Then the homography matrix between the two frames is satisfy: According to the coordinate positions of the two key point sets, H is calculated by solving the multivariate equations. t , or by using the RANSAC method to improve H t The robustness of the estimate, in the estimated homography matrix H t Based on the , the algorithm projects the coordinates of the four vertices of the green screen in the reference frame onto the current frame, obtains the estimated area B occupied by the green screen in the current frame, and estimates the positional relationship between the camera and the green screen, i.e., the camera posture, through the three-dimensional object imaging formula; In the third and fourth steps, first perform color key extraction on area B, then calculate the α value according to the formula to obtain the green screen area mask A t ; Step 3 Color key: The estimated green screen area B should be a quadrilateral. The analytical equations of the four sides are: Due to the phenomenon of key point mismatch, the estimated H t It is not very accurate, resulting in a large difference between the obtained green screen area B and the actual green screen area, and the following steps are needed to correct it; Take the color key in area B and perform keying. First, convert the RGB values ​​of the pixels in area B into YUV values ​​using the following formula: The UV values ​​of all pixels in the area can be obtained, and then the color keys of the pixels in the area are taken, and the K-Means clustering method is used to take the color keys; Step 4α value estimation: The basic idea of ​​obtaining the α value is to solve the equation using certain prior conditions: I z =a z F+(1-a z )B′ Among them, α z =1 indicates pixel I z Belongs to the foreground F; α z =0 means pixel I z Background B B′; if α z ∈(0, 1), then it means pixel I z It is a mixture of foreground and background, so after getting the color key, the alpha value needs to be estimated to get the foreground and background. The alpha value is obtained by the following formula: Among them, I={(u1, v1), (u2, v2),...(u z , v z )…(u n , v n )} is the U and V value set of the current frame pixel, (u i , v i ) is the UV value of the color key obtained in the previous step, m and n are hyperparameters. Since the α value range is [0, 1], the result d of the above formula is input into the Sigmoid function and mapped on the [0, 1] interval to obtain the α value of each pixel in the current frame. The frame pixels are divided into three categories: foreground F, background B′ and mixed pixel set M, which is the green screen area mask A. t , Step 5: A t Obtain edge processing and Hough transform to determine the boundary of the area, and obtain a new green screen quadrilateral area B″ and a new homography matrix At the same time, a more accurate camera pose is calculated; Step 5 Extract the straight line: After obtaining A t Afterwards, t Perform the Canny operator to obtain edge processing and perform the Hough transform: By voting on the line parameter space, the analytical equations of the main lines in the image space {l′1, l′2, l′3, ..., l′ Q }, in this set, corresponding to each of the four estimated edge lines l above i , and a line closest to it is proposed, so the algorithm obtains 4 edge lines estimated by the Hough algorithm. These four edge lines also determine a quadrilateral, which is the new green screen quadrilateral area B″. Using the four vertex positions determined by the quadrilateral area B″, the new homography matrix H′ can be estimated t , for H′ t and H t Find the mean, the algorithm obtains the updated homography matrix Known updated Estimate more accurate camera pose through 3D object imaging formula; Step 6: Repeat the above steps, continuously update the homography matrix, and dynamically adjust the hotspot area until the search range ε ​​of the point matching is less than the given preset value, or the number of iteration steps reaches the preset upper limit, and then the accurate camera posture and color key can be obtained; Step 6 Iteration: The next iteration is still for the current frame F t With reference frame F R Perform point matching and convert the homography matrix estimated in the previous step As the H of the current frame t-1 , and the key points of the reference frame are known, in the search process of point matching, due to H t-1 The corresponding feature points can be obtained by narrowing the search range ε. The hot spot area obtained by calculating the homography matrix will be closer to the position of the real green screen area, and the camera posture estimation will be more accurate, which improves the accuracy of the color key. Then the color key is taken and the α value is calculated to optimize the green screen area mask A. t , after another Hough transform update H t After that, a more accurate green screen area and posture estimation is obtained, thus achieving the purpose of iteratively optimizing the camera posture estimation and green screen estimation area. Repeat steps 1-5 to continuously update H t , gradually narrow the point matching search range, correct the estimated camera pose, dynamically adjust the hotspot area, and improve the accuracy of the color key until the search range ε ​​is less than the preset value, or the number of iteration steps reaches the preset upper limit. After several rounds of iterations, the accurate camera pose and color key can be obtained; Step 7: Place the virtual scene based on the camera pose obtained in the preprocessing stage; Based on the accurate camera posture obtained in the preprocessing stage, the current image is matted and placed in the virtual 3D space to achieve a virtual-real combination effect; Step 8: Perform pixel-level keying on each frame based on the precise color key obtained in the preprocessing stage; Use the precise color key obtained in the preprocessing stage to key each frame to obtain high-quality keying results.

2. The self-iterative local green screen matting method according to claim 1, characterized in that: The point matching uses SIFT, SURF or deep learning algorithms.

3. The self-iterative local green screen matting method according to claim 1, characterized in that: The number of the color keys is one or more.

Images