A light field cutout method and device based on empty angle consistency

Abstract

The present invention discloses a method and device for light field matting based on space angle consistency, which comprises: step 1, extracting a central sub-aperture image of light field data and calculating its parallax; step 2, calculating an alpha map of the central sub-aperture image; step 3, propagating the alpha map of the central sub-aperture image through a light field alpha map propagation model to obtain the alpha map of the light field. u,v (x, y). By adopting the method provided by the present invention, accurate and consistent light field cutout can be achieved and its spatial angle consistency can be evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of optical information processing, and in particular to a method and device for light field cutout based on void angle consistency. Background Art

[0002] The light field describes the distribution of light in free space. Light field data captures the radiation of light from all directions and records the spatial, angular and intensity information of the light. Light field imaging can capture richer visual information, make up for the angular information lost in traditional photography, provide stronger capabilities for scene understanding, and improve the performance of traditional computer vision problems.

[0003] A 4D light field data can be represented by a 2D sub-aperture image array, so the 2D image cutout algorithm performed on sub-aperture images can be regarded as realizing the cutout of 4D light field data to a certain extent. The sub-aperture image operation method does not consider the connection between different sub-aperture images, and cannot maintain the spatial angle consistency of the light field data. The results obtained are difficult to meet the requirements of downstream technologies and applications. In addition, the light field data size is large, and there is a lot of redundant information between sub-aperture images. Simple sub-aperture image processing has too many redundant calculations and is too inefficient. Summary of the invention

[0004] The object of the present invention is to provide a method for light field cutout with spatial angle consistency and its evaluation index, which can avoid the destruction of the spatial angle consistency of light field data during the cutout process and reduce calculation redundancy.

[0005] To achieve the above object, the present invention provides a light field cutout method based on empty angle consistency, which comprises:

[0006] Step 1, extracting the central sub-aperture image of the light field data and calculating its disparity;

[0007] Step 2, calculating the central sub-aperture image alpha map;

[0008] Step 3: Propagate the central sub-aperture image alpha map through the light field alpha map propagation model provided by formula (1) to obtain the light field alpha map alpha u,v (x,y);

[0009]

[0010] In the formula, is the central viewpoint (u 0 ,v 0 ) is the alpha map of the sub-aperture image under the central sub-aperture image. disp(x,y) is the disparity corresponding to the image point (x,y) in the central sub-aperture image. Δu and Δv are the disparity between any viewpoint (u,v) and the central viewpoint (u 0,v 0 ) in the u and v directions.

[0011] Furthermore, the calculation formula of the disparity map disp(x,y) in step 2 is formula (2):

[0012] disp(x,y)=F / Z(x,y)+F (2)

[0013] Where F is the focal length of the camera and Z(x,y) is the depth map of the scene.

[0014] Furthermore, the method for obtaining the central sub-aperture image alpha map in step 2 specifically includes:

[0015] Step 21: The confidence q of any pixel i in the disparity map disp(x,y) is i Set as formula (3):

[0016]

[0017] In the formula, p i is the disparity value of any pixel i in the disparity map disp(x,y), D=max(disp(x,y)) is the maximum value in the disparity map, and d=min(disp(x,y)) is the minimum value in the disparity map;

[0018] Step 22, set the confidence threshold t 1 and t 2 , when the confidence q i Greater than t 1 When , the area belongs to the foreground, and when the confidence q i Less than t 2 When , the area belongs to the background, and the other areas are uncertain areas in the trigraph, then the trigraph described by formula (4) is obtained:

[0019]

[0020] Step 23, convert the three-part image into I 1 and I 2 , I 1 The pixels marked in the foreground and background are 1, and the pixel values ​​in other places are 0. 1 Transformed into a sparse diagonal matrix, D s , I 2 The pixel value of the foreground mark is 1, and the other pixel values ​​are 0. Converting it into a column vector is b s :

[0021]

[0022]

[0023] Step 24, solve the alpha map of the central sub-aperture image by equation (7):

[0024] α=λ(L+λ) -1 b s (7)

[0025] Where λ is the weight coefficient of the label, and L is the Laplacian matrix of the cutout.

[0026] Furthermore, the light field cutout method based on empty angle consistency also includes:

[0027] Step 4: Evaluate the consistency of the empty corners of the light field cutout.

[0028] Furthermore, step 4 specifically evaluates the consistency of the empty angle in the light field alpha map through the empty angle consistency evaluation index S-cons obtained in the spatial domain provided by formula (8): the value range of the index S-cons is (0-1), the closer the index S-cons is to 1, the smaller the variance of the boundary area is, and the better the consistency of the light field data is; the closer the index S-cons is to 0, the worse the consistency of the light field data is;

[0029]

[0030] Where N EPI is the number of EPI (Epipolar Plane Image, EPI) of the light field alpha image, E j is the corresponding j-th EPI. There are l boundary areas in each EPI, denoted as Ω l , N l is the Ω in the jth EPI l The total number of pixels in the area, E ji is the i-th pixel in the j-th EPI of the light field alpha image, is the Ω in the jth EPI l The mean of the pixel values ​​at the boundary;

[0031] Alternatively, step 4 specifically uses the empty angle consistency evaluation index F-cons obtained in the frequency domain provided by formula (9) to evaluate the empty angle consistency in the light field alpha map: the index F-cons value range is (0-1), the larger the index F-cons is, the smaller the range occupied by the support set is, the better the empty angle consistency of the light field data is, and vice versa;

[0032]

[0033] Where:

[0034]

[0035] In the formula, L′(Ω u ,Ω v ,Ω x ,Ω y ) is the spectrum diagram L(Ω u ,Ω v ,Ω x ,Ω y ) After binarization, the spectrum graph is counted, ε is a small number, M is the number of pixels in the spectrum graph, Ω u ,Ω v ,Ω x ,Ω y They are the coordinate axes of the light field spectrum diagram respectively.

[0036] The present invention also provides a light field cutout device based on empty angle consistency, which comprises:

[0037] A central sub-aperture image extraction unit, which is used to extract a central sub-aperture image of the light field data;

[0038] A calculation unit, which is used to calculate the disparity of pixels in the central sub-aperture image of the light field data and the central sub-aperture image alpha map;

[0039] The light field cutout unit is used to propagate the central sub-aperture image alpha map through the light field alpha map propagation model provided by formula (1) to obtain the light field alpha map alpha u,v (x,y);

[0040]

[0041] In the formula, is the central viewpoint (u 0 ,v 0 ) is the alpha map of the sub-aperture image under the central sub-aperture image. disp(c,y) is the disparity corresponding to the image point (x,y) in the central sub-aperture image. Δu and Δv are the disparity between any viewpoint (u,v) and the central viewpoint (u 0 ,v 0 ) in the u and v directions.

[0042] Furthermore, the computing unit comprises:

[0043] The disparity calculation subunit is used to calculate the disparity map disp(x,y) according to the depth map of the light field data by formula (2):

[0044] disp(x,y)=F / Z(x,y)+F (2)

[0045] Where F is the focal length of the camera and Z(x,y) is the depth map of the scene.

[0046] Furthermore, the computing unit comprises:

[0047] The central sub-aperture image alpha map calculation subunit is used to obtain the central sub-aperture image alpha map according to the central sub-aperture image of the light field data through the following specific alpha closed solution method:

[0048] Step 21: The confidence q of any pixel i in the disparity map disp(x,y) is i Set as formula (3):

[0049]

[0050] In the formula, p i is the disparity value of any pixel i in the disparity map disp(x,y), D=max(disp(x,y)) is the maximum value in the disparity map, and d=min(disp(x,y)) is the minimum value in the disparity map;

[0051] Step 22, set the confidence threshold t 1 and t 2 , when the confidence q i Greater than t 1 When , the area belongs to the foreground, and when the confidence q i Less than t 2 When , the area belongs to the background, and the other areas are uncertain areas in the trigraph, then the trigraph described by formula (4) is obtained:

[0052]

[0053] Step 23, convert the three-part image into I 1 and I 2 , I 1 The pixels marked in the foreground and background are 1, and the pixel values ​​in other places are 0. 1 Transformed into a sparse diagonal matrix, D s , I 2 The pixel value of the foreground mark is 1, and the other pixel values ​​are 0. Converting it into a column vector is b s :

[0054]

[0055]

[0056] Step 24, solve the alpha map of the central sub-aperture image by equation (7):

[0057] α=λ(L+λ) -1 b s (7)

[0058] Where λ is the weight coefficient of the label, and L is the Laplacian matrix of the cutout.

[0059] Furthermore, the light field cutout device based on empty angle consistency also includes:

[0060] An evaluation unit is used to evaluate the consistency of empty corners in the light field cutout.

[0061] Furthermore, the evaluation unit specifically includes a spatial domain evaluation subunit or a frequency domain evaluation subunit:

[0062] The spatial domain evaluation subunit is used to evaluate the spatial angle consistency of the light field data in the light field alpha map by using the spatial angle consistency index S-cons obtained in the spatial domain provided by formula (8): the index S-cons value range is (0-1), the closer the index S-cons is to 1, the smaller the variance of the boundary area is, the better the spatial angle consistency of the light field data is, and the closer the index S-cons is to 0, the worse the consistency of the light field data is;

[0063]

[0064] Where N EPI is the number of EPI (Epipolar Plane Image, EPI) of the light field alpha image, E j is the corresponding j-th EPI. There are l boundary areas in each EPI, denoted as Ω l , N l is the Ω in the jth EPI l The total number of pixels in the area, E ji is the i-th pixel in the j-th EPI of the light field alpha image, is the Ω in the jth EPI l The mean of the pixel values ​​at the boundary;

[0065] The frequency domain evaluation subunit is used to provide a spatial angle consistency evaluation index F-cons obtained in the frequency domain through formula (9) to evaluate the spatial angle consistency of the light field data in the light field alpha map: the index F-cons value range is (0-1), the larger the index F-cons is, the smaller the range occupied by the representative support set is, the better the spatial angle consistency of the light field data is, and vice versa;

[0066]

[0067] Where:

[0068]

[0069] In the formula, L′(Ω u ,Ω v,Ω x ,Ω y ) is the spectrum diagram L(Ω u ,Ω v ,Ω x ,Ω y ) After binarization, the spectrum graph is counted, ε is a small number, M is the number of pixels in the spectrum graph, Ω u ,Ω v ,Ω x ,Ω y They are the coordinate axes of the light field spectrum diagram respectively.

[0070] The present invention adopts the above technical solution, which has the following advantages:

[0071] By adopting the method of light field cutout based on spatial angle consistency and its evaluation index provided by the present invention, accurate and consistent light field cutout is achieved, the spatial angle consistency of light field data is ensured and calculation redundancy is reduced, and the established evaluation index can effectively evaluate the spatial angle consistency of the obtained light field data. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic diagram of the process of a light field cutout method based on empty angle consistency provided in an embodiment of the present invention.

[0073] Figure 2 A schematic diagram of the flow of a method for evaluating light field cutouts based on empty angle consistency provided in an embodiment of the present invention.

[0074] Figure 3 It is the alpha map disparity map of the central sub-aperture image.

[0075] Figure 4 Schematic diagram of the propagation model.

[0076] Figure 5 Schematic diagram of spatial domain evaluation indicators.

[0077] Figure 6 Schematic diagram of frequency domain evaluation indicators. DETAILED DESCRIPTION

[0078] In the drawings, the same or similar reference numerals are used to represent the same or similar elements or elements with the same or similar functions. The embodiments of the present invention are described in detail below in conjunction with the drawings.

[0079] like Figure 1 As shown, the method for light field cutout based on empty angle consistency provided by the embodiment of the present invention includes:

[0080] Step 1: extract the central sub-aperture image of the light field data and calculate its disparity.

[0081] Step 2: Calculate the central sub-aperture image alpha map.

[0082] Step 3: Propagate the central sub-aperture image alpha map through the light field alpha map propagation model provided by formula (1) to obtain the light field alpha map alpha u,v (x,y);

[0083]

[0084] In the formula, is the central viewpoint (u 0 ,v 0 ) is the alpha map of the sub-aperture image under the central sub-aperture image. disp(x,y) is the disparity corresponding to the image point (x,y) in the central sub-aperture image. Δu and Δv are the disparity between any viewpoint (u,v) and the central viewpoint (u 0 ,v 0 ) in the u and v directions.

[0085] According to the propagation model constructed in step 3, the alpha map of step 2 is propagated to obtain the light field alpha map under 9*9 viewpoints, as shown in Figure 4 (a) shows the central subaperture image alpha Figure 2 and parallax Figure 3 , Figure 4 (b) shows the alpha of the sub-aperture image obtained by propagation at 9*9 viewpoints. Figure 4 .

[0086] The present invention establishes a light field matting propagation model that satisfies spatial angle consistency. The establishment of the model is based on the spatial angle coupling property of four-dimensional light field data, and can quickly obtain a high-quality light field alpha map with better spatial angle consistency. To achieve consistent light field matting, the Cho et al. light field matting method can be used to enhance the spatial angle consistency of the estimated alpha map in the sub-aperture image by introducing the smoothness constraint of the epipolar plane image (EPI).

[0087] In one embodiment, in step 2, Figure 3 As shown in a in FIG, according to the central sub-aperture image of the light field data, the disparity map disp(x, y) of the light field data is calculated using equation (2). Since the depth and disparity satisfy such an inverse relationship:

[0088] disp(x,y)=F / Z(x,y)+F (2)

[0089] Where F is the focal length of the camera and Z(x,y) is the depth map of the scene.

[0090] The depth can be estimated by using a fast light field image depth estimation method based on multi-view stereo matching proposed by Mishiba K et al. The depth can also be estimated by using an occlusion-aware depth estimation algorithm proposed by Wang TC et al. Once the depth map is obtained, the disparity map can be obtained by converting it using formula (2).

[0091] In one embodiment, Figure 3 As shown in b, the method for obtaining the central sub-aperture image alpha map in step 2 specifically includes:

[0092] Step 21, in order to automatically obtain the three-part image of the central sub-aperture image and solve the alpha image, it is assumed that the light field data disparity map is accurate, and the confidence is introduced into the disparity map and a threshold is selected. The accurate foreground, background and uncertain area are set according to the confidence threshold. The confidence of the disparity map is set as follows:

[0093]

[0094] In the formula, p i is the disparity value of any pixel i in the disparity map disp(x,y), and the confidence q at the pixel i Represents the probability that the point belongs to the foreground, D = max (disp (x, y)) is the maximum value in the disparity map, and d = min (disp (x, y)) is the minimum value in the disparity map. Set the threshold t 1 and t 2 , when the confidence level is greater than t 1 When the confidence is less than t 2 When , this part of the area is considered as the background, and the other areas are the uncertain areas in the three-part image, and the three-part image can be obtained:

[0095]

[0096] Where, t 1 and t 2 is a point on the straight line formed by the minimum and maximum confidence values, and t 1 and t 2 The selection of t is related to the proportion of the depth range of the foreground and background in the overall depth of the scene. When the foreground depth range accounts for a large proportion, t 1 The larger the t 2 The smaller the background depth range, the larger the t 1 The smaller the t 2 The bigger.

[0097] After obtaining the three-part image, convert the three-part image into I 1 and I 2 :I 1The pixels marked in the foreground and background are 1, and the pixel values ​​in other places are 0. 1 Transformed into a sparse diagonal matrix, D s ;I 2 The pixel value of the foreground mark is 1, and the other pixel values ​​are 0. Converting it into a column vector is b s Of course, you can use manual marking 1 and I 2 , and existing automatic methods can also be used for labeling.

[0098]

[0099]

[0100] Step 22, based on the obtained D s and b s The alpha map of the central sub-aperture image can be solved by formula (7):

[0101] α=λ(L+λ) -1 b s (7)

[0102] In the formula, λ is the weight coefficient of the mark, which is generally a larger number, such as 100, and L is the Laplace matrix of the cutout.

[0103] In addition, the central sub-aperture image can be matted by Random Walk matting, Poisson matting and other methods.

[0104] In one embodiment, Figure 5 and Figure 6 As shown, the light field cutout method based on empty angle consistency also includes:

[0105] Step 4, such as Figure 5 As shown in the figure, the object points at the boundary of the scene form a bunch of straight lines in the EPI of the alpha map. The smaller the overall difference of the straight lines, the better the consistency at the boundary. Therefore, the area at the boundary is recorded as Ω. By calculating the variance of the pixel values ​​in the Ω area, the consistency of the obtained light field data is measured. If the variance is smaller, the consistency is better. u,v The null angle consistency of (x,y) is evaluated.

[0106] Step 4 specifically obtains the spatial angle consistency evaluation index S-cons in the spatial domain provided by formula (8) to evaluate the spatial angle consistency of the light field data in the light field alpha map: the index S-cons value range is (0-1), the closer the index S-cons is to 1, the smaller the variance of the boundary area is, the better the spatial angle consistency of the light field data is, and the closer the index S-cons is to 0, the worse the spatial angle consistency of the light field data is;

[0107]

[0108] Where N EPI is the number of EPI (Epipolar Plane Image, EPI) of the light field alpha image, E j is the corresponding j-th EPI. There are l boundary areas in each EPI, denoted as Ω l , N l is the Ω in the jth EPI l The total number of pixels in the area, E ji is the i-th pixel in the j-th EPI of the light field alpha image, is the Ω in the jth EPI l The mean of the pixel values ​​at the boundary;

[0109] Or, if Figure 6 As shown in a in , the spectrum of light field data satisfies the shape of hypercone, so the spatial angle consistency of light field data can be reflected by the range of the support in the spectrum. If the spatial angle consistency of light field data is good, the linear shape can be better maintained in the frequency domain, the smaller the range of the spectrum support is, and the smaller the proportion of the overall spectrum graph is; on the contrary, if the consistency is destroyed, the structure of the hypercone is destroyed, the linear structure cannot be maintained in the frequency domain, and the larger the support range is, the larger the proportion of the overall spectrum graph is.

[0110] Based on this, step 4 specifically provides the binarization of the spectrum graph through equation (9): Figure 6 As shown in b in the figure, the empty angle consistency evaluation index F-cons obtained in the frequency domain by formula (10) is used to evaluate the empty angle consistency of the light field data in the light field alpha map: the index F-cons value range is (0-1), the larger the index F-cons is, the smaller the range occupied by the support set is, the better the empty angle consistency of the light field data is, and vice versa;

[0111]

[0112] Where:

[0113]

[0114] In the formula, L′(Ωu ,Ω v ,Ω x ,Ω y ) is the spectrum diagram L(Ω u ,Ω v ,Ω x ,Ω y ) After binarization, the spectrum graph is counted, ε is a small number, M is the number of pixels in the spectrum graph, Ω u ,Ω v ,Ω x ,Ω y They are the coordinate axes of the light field spectrum diagram respectively.

[0115] In the present invention, a quantitative evaluation index of the light field alpha map is proposed. The establishment of this index is based on the essential properties of the light field data and can objectively evaluate the spatial angle consistency of the light field alpha map after propagation. Cho et al. also proposed an index for evaluating the spatial angle consistency of light field data, which can also be evaluated using MSE, SAD, etc.

[0116] The light field cutout device based on spatial angle consistency provided by an embodiment of the present invention comprises a central sub-aperture image extraction unit, a calculation unit and a light field cutout unit, wherein:

[0117] The central sub-aperture image extraction unit is used to extract the central sub-aperture image of the light field data.

[0118] The calculation unit is used to calculate the disparity of pixels in the central sub-aperture image of the light field data and the central sub-aperture image alpha map.

[0119] The light field cutout unit is used to propagate the central sub-aperture image alpha map through the light field alpha map propagation model provided by formula (1) to obtain the light field alpha map alpha u,v (x,y);

[0120]

[0121] In the formula, is the central viewpoint (u 0 ,v 0 ) is the alpha map of the sub-aperture image under the central sub-aperture image. disp(c,y) is the disparity corresponding to the image point (x,y) in the central sub-aperture image. Δu and Δv are the disparity between any viewpoint (u,v) and the central viewpoint (u 0 ,v 0 ) in the u and v directions.

[0122] In one embodiment, the calculation unit includes a disparity calculation subunit, which is used to calculate the disparity map disp(x, y) according to the depth map of the light field data by formula (2):

[0123] disp(x,y)=F / Z(x,y)+F (2)

[0124] Where F is the focal length of the camera and Z(x,y) is the depth map of the scene.

[0125] In one embodiment, the calculation unit includes a central sub-aperture image alpha map calculation sub-unit, which is used to obtain the central sub-aperture image alpha map according to the central sub-aperture image of the light field data by the following specific alpha closed solution method:

[0126] Step 21: The confidence q of any pixel i in the disparity map disp(x,y) is i Set as formula (3):

[0127]

[0128] In the formula, p i is the disparity value of any pixel i in the disparity map disp(x,y), D=max(disp(x,y)) is the maximum value in the disparity map, and d=min(disp(x,y)) is the minimum value in the disparity map;

[0129] Step 22, set the confidence threshold t 1 and t 2 , when the confidence q i Greater than t 1 When , the area belongs to the foreground, and when the confidence q i Less than t 2 When , the area belongs to the background, and the other areas are uncertain areas in the trigraph, then the trigraph described by formula (4) is obtained:

[0130]

[0131] Step 23, convert the three-part image into I 1 and I 2 , I 1 The pixels marked in the foreground and background are 1, and the pixel values ​​in other places are 0. 1 Transformed into a sparse diagonal matrix, D s , I 2 The pixel value of the foreground mark is 1, and the other pixel values ​​are 0. Converting it into a column vector is b s :

[0132]

[0133]

[0134] Step 24, solve the alpha map of the central sub-aperture image by equation (7):

[0135] α=λ(L+λ) -1 b s (7)

[0136] Where λ is the weight coefficient of the label, and L is the Laplacian matrix of the cutout.

[0137] In one embodiment, the light field cutout device based on space angle consistency further includes an evaluation unit, which is used to evaluate the light field alpha map alpha. u,v The null angle consistency of (x,y) is evaluated.

[0138] In one embodiment, the evaluation unit specifically includes a spatial domain evaluation subunit or a frequency domain evaluation subunit:

[0139] The spatial domain evaluation subunit is used to evaluate the spatial angle consistency of the light field data in the light field alpha map by using the spatial angle consistency evaluation index S-cons obtained in the spatial domain provided by formula (8): the index S-cons value range is (0-1), the closer the index S-cons is to 1, the smaller the variance of the boundary area is, the better the spatial angle consistency of the light field data is, and the closer the index S-cons is to 0, the worse the spatial angle consistency of the light field data is;

[0140]

[0141] Where N EPI is the number of EPI (Epipolar Plane Image) of the light field alpha image, E j is the corresponding j-th EPI. There are l boundary areas in each EPI, denoted as Ω l , N l is the Ω in the jth EPI l The total number of pixels in the area, E ji is the i-th pixel in the j-th EPI of the light field alpha image, is the Ω in the jth EPI l The mean of the pixel values ​​at the boundary;

[0142] The frequency domain evaluation subunit is used to evaluate the light field data empty angle consistency in the light field alpha map by using the empty angle consistency evaluation index F-cons obtained in the frequency domain provided by formula (9): the index F-cons value range is (0-1), the larger the index F-cons is, the smaller the range occupied by the representative support set is, the better the light field data empty angle consistency is, and vice versa;

[0143]

[0144] Where:

[0145]

[0146] In the formula, L′(Ω u ,Ω v ,Ω x ,Ω y ) is the spectrum diagram L(Ω u ,Ω v ,Ω x ,Ω y ) After binarization, the spectrum graph is counted, ε is a small number, M is the number of pixels in the spectrum graph, Ω u ,Ω v ,Ω x ,Ω y They are the coordinate axes of the light field spectrum diagram respectively.

[0147] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Those skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents; these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A light field cutout method based on empty angle consistency, characterized in that: include: Step 1, extracting the central sub-aperture image of the light field data and calculating its disparity; Step 2, calculating the central sub-aperture image alpha map; The method for obtaining the central sub-aperture image alpha map in step 2 specifically includes: Step 21: The confidence q of any pixel i in the disparity map disp(x,y) is i Set as formula (3): In the formula, p i is the disparity value of any pixel i in the disparity map disp(x,y), D=max(disp(x,y)) is the maximum value in the disparity map, and d=min(disp(x,y)) is the minimum value in the disparity map; Step 22, set confidence thresholds t1 and t2, when confidence q i When it is greater than t1, the area belongs to the foreground. i When it is less than t2, the area belongs to the background, and the other areas are uncertain areas in the trigraph. Then the trigraph described by formula (4) is obtained: Step 23, convert the three-part image into I1 and I2, where the pixels at the foreground and background marks in I1 are 1, and the pixel values ​​at other places are 0, and I1 is converted into a sparse diagonal matrix, which is D s , the pixel value of the foreground mark in I2 is 1, and the other pixel values ​​are 0. Converting it into a column vector is b s : Step 24, solve the alpha map of the central sub-aperture image by equation (7): α=λ(L+λ) -1 b s (7) Where λ is the weight coefficient of the mark, and L is the Laplace matrix of the cutout; Step 3: Propagate the central sub-aperture image alpha map through the light field alpha map propagation model provided by formula (1) to obtain the light field alpha map alpha u,v (x,y); In the formula, is the alpha map of the sub-aperture image under the central viewpoint (u0, v0), disp(x, y) is the disparity corresponding to the image point (x, y) in the central sub-aperture image, Δu and Δv are the distances between any viewpoint (u, v) and the central viewpoint (u0, v0) in the u and v directions respectively; Step 4, evaluating the consistency of the empty corners of the light field cutout; Step 4 specifically uses the empty angle consistency evaluation index S-cons obtained in the spatial domain provided by formula (8) to evaluate the empty angle consistency in the light field alpha map: the index S-cons value range is (0, 1), the closer the index S-cons is to 1, the smaller the variance of the boundary area is, and the better the consistency of the light field data is; the closer the index S-cons is to 0, the worse the consistency of the light field data is; Where N EPI is the number of EPI (Epipolar Plane Image, EPI) of the light field alpha map, E j is the corresponding j-th EPI. There are l boundary areas in each EPI, denoted as Ω l , N l is the Ω in the jth EPI l The total number of pixels in the area, E ji is the i-th pixel in the j-th EPI of the light field alpha image, is the Ω in the jth EPI l The mean of the pixel values ​​at the boundary; Alternatively, step 4 specifically uses the empty angle consistency evaluation index F-cons obtained in the frequency domain provided by formula (9) to evaluate the empty angle consistency in the light field alpha map: the index F-cons value range is (0-1), the larger the index F-cons is, the smaller the range occupied by the support set is, the better the empty angle consistency of the light field data is, and vice versa; Where: In the formula, L'(Ω u ,Ω v ,Ω x ,Ω y ) is the spectrum diagram L(Ω u ,Ω v ,Ω x ,Ω y ) After binarization, the spectrum graph is counted, ε is a small number, M is the number of pixels in the spectrum graph, Ω u ,Ω v ,Ω x ,Ω y They are the coordinate axes of the light field spectrum diagram respectively.

2. The method for light field cutout based on empty angle consistency according to claim 1, characterized in that: The calculation formula of the disparity map disp(x,y) in step 2 is formula (2): disp(x,y)=F / Z(x,y)+F (2) where F is the focal length of the camera and Z(x,y) is the depth map of the scene.

3. A light field cutout device based on empty angle consistency, characterized in that: include: A central sub-aperture image extraction unit, which is used to extract a central sub-aperture image of the light field data; A calculation unit, which is used to calculate the disparity of pixels in the central sub-aperture image of the light field data and the central sub-aperture image alpha map; the calculation unit includes: The central sub-aperture image alpha map calculation subunit is used to obtain the central sub-aperture image alpha map according to the central sub-aperture image of the light field data through the following specific alpha closed solution method: Step 21: The confidence q of any pixel i in the disparity map disp(x,y) is i Set as formula (3): In the formula, p i is the disparity value of any pixel i in the disparity map disp(x,y), D=max(disp(x,y)) is the maximum value in the disparity map, and d=min(disp(x,y)) is the minimum value in the disparity map; Step 22, set confidence thresholds t1 and t2, when confidence q i When it is greater than t1, the area belongs to the foreground. i When it is less than t2, the area belongs to the background, and the other areas are uncertain areas in the trigraph. Then the trigraph described by formula (4) is obtained: Step 23, convert the three-part image into I1 and I2, where the pixels at the foreground and background marks in I1 are 1, and the pixel values ​​at other places are 0, and I1 is converted into a sparse diagonal matrix, which is D s , the pixel value of the foreground mark in I2 is 1, and the other pixel values ​​are 0. Converting it into a column vector is b s : Step 24, solve the alpha map of the central sub-aperture image by equation (7): α=λ(L+λ) -1 b s (7) Where λ is the weight coefficient of the mark, and L is the Laplace matrix of the cutout; The light field cutout unit is used to propagate the central sub-aperture image alpha map through the light field alpha map propagation model provided by formula (1) to obtain the light field alpha map alpha u,v (x,y); In the formula, is the alpha map of the sub-aperture image under the central viewpoint (u0, v0), disp(x, y) is the disparity corresponding to the image point (x, y) in the central sub-aperture image, Δu and Δv are the distances between any viewpoint (u, v) and the central viewpoint (u0, v0) in the u and v directions respectively; An evaluation unit is used to evaluate the spatial angle consistency of the light field cutout. The evaluation unit specifically includes a spatial domain evaluation subunit or a frequency domain evaluation subunit: The spatial domain evaluation subunit is used to evaluate the spatial angle consistency of the light field data in the light field alpha map by using the spatial angle consistency index S-cons obtained in the spatial domain provided by formula (8): the index S-cons value range is (0-1), the closer the index S-cons is to 1, the smaller the variance of the boundary area is, the better the spatial angle consistency of the light field data is, and the closer the index S-cons is to 0, the worse the consistency of the light field data is; Where N EPI is the number of EPI (Epipolar Plane Image, EPI) of the light field alpha map, E j is the corresponding j-th EPI. There are l boundary areas in each EPI, denoted as Ω l , N l is the Ω in the jth EPI l The total number of pixels in the area, E ji is the i-th pixel in the j-th EPI of the light field alpha image, is the Ω in the jth EPI l The mean of the pixel values ​​at the boundary; The frequency domain evaluation subunit is used to provide a spatial angle consistency evaluation index F-cons obtained in the frequency domain through formula (9) to evaluate the spatial angle consistency of the light field data in the light field alpha map: the index F-cons value range is (0, 1), the larger the index F-cons is, the smaller the range occupied by the representative support set is, the better the spatial angle consistency of the light field data is, and vice versa; Where: In the formula, L'(Ω u ,Ω v ,Ω x ,Ω y ) is the spectrum diagram L(Ω u ,Ω v ,Ω x ,Ω y ) After binarization, the spectrum graph is counted, ε is a small number, M is the number of pixels in the spectrum graph, Ω u ,Ω v ,Ω x ,Ω y They are the coordinate axes of the light field spectrum diagram respectively.

4. The light field cutout device based on empty angle consistency according to claim 3, characterized in that: The computing unit includes: The disparity calculation subunit is used to calculate the disparity map disp(x,y) according to the depth map of the light field data by formula (2): disp(x,y)=F / Z(x,y)+F (2) Where F is the focal length of the camera and Z(x,y) is the depth map of the scene.

Images