Light field angle super-resolution method based on conditional VAE model

Abstract

The invention relates to a light field angle super-resolution method based on a conditional VAE model, which belongs to the technical field of computational imaging, machine vision and digital image processing, and comprises the following steps of: respectively inputting a light field sub-aperture image array LF gamma and a central sub-aperture image LF0 which are sparse at equal intervals in a viewpoint plane into a parallax estimation module (P) and a feature extraction module (F); obtaining a scene initial disparity map and center sub-aperture features of different hierarchies; the method does not depend on a large-scale high-resolution training data set, angle domain super-resolution reconstruction can be synchronously carried out on all light field sub-aperture images, and an air angle consistency mixed loss function based on light field data is designed. When a new model is used for reconstructing a sub-aperture image with high angle resolution, parallax information coupled to a sub-aperture image variety can be utilized, a numerical experiment result on a synthetic and real data set is close to a supervised method, and meanwhile, the super-resolution of a method based on deep learning to a light field angle domain can be met.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computational imaging, machine vision and digital image processing, and specifically relates to a light field angle super-resolution method based on a conditional VAE model. Background Art

[0002] Currently, light field angle super-resolution based on the conditional VAE model adopts an unsupervised learning method, and the super-resolution performance of the model depends on training on a data set consisting of a large number of high-resolution light field data pairs.

[0003] In practical applications, it is difficult to collect high-resolution light field data for a large number of scenes. In addition, when the imaging parameters and scenes of the light field camera are limited, the generalization performance of the deep learning model trained in new scenes and new imaging parameters is weak. Based on this, a light field angle super-resolution method based on the conditional VAE model is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a light field angle super-resolution method based on a conditional VAE model in order to solve the above-mentioned problem.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] A light field angle super-resolution method based on a conditional VAE model comprises the following steps:

[0007] S1: By collecting sparse light field sub-aperture image arrays LF at equal intervals in the viewpoint plane γ The central sub-aperture image LF0 and the central sub-aperture image LF0 are respectively input into the disparity estimation module (P) and the feature extraction module (F) to obtain the scene initial disparity map and central sub-aperture features of different levels;

[0008] S2: LF γ The initial disparity map obtained with S1 and the central sub-aperture features at different levels are used as the conditional VAE light field angle super-resolution model encoder Input, output latent variable z in the latent space;

[0009] S3: The latent variable z obtained in S2 and the initial disparity map obtained in S1 and the central sub-aperture features at different levels are used as the conditional VAE light field angle super-resolution model decoder The input is densely spaced light field sub-aperture images in the viewpoint plane.

[0010] S4: The loss function is designed by constraining the spatial angle consistency of light field data to optimize the parameters of the conditional VAE model;

[0011] S5: The sparse light field sub-aperture images at equal intervals in the viewpoint plane are subjected to light field angle super-resolution through a conditional VAE model constrained by spatial angle consistency to obtain dense light field sub-aperture images at equal intervals in the viewpoint plane.

[0012] As a further optimization scheme of the present invention, the calculation formula of the central sub-aperture characteristic is:

[0013] d=P(LF γ ), m=F1(LF0) and x6=F2(m);

[0014] Where F1 represents the feature extraction module of the first three layers of convolution, F2 represents the feature extraction module of the last three layers of convolution, d represents the initial disparity map, m represents the feature variable extracted by LF0 through F1, and x6 represents the feature variable extracted by LF0 through F1 and F2.

[0015] As a further optimization scheme of the present invention, the calculation formula of the latent variable z in the latent space is:

[0016]

[0017] Where θ1 is , z is a hidden variable in the latent space, where the elements in z obey the standard normal distribution N(0,1).

[0018] As a further optimization solution of the present invention, the light field sub-aperture image The calculation formula is:

[0019]

[0020] In the formula, θ2 is A collection of parameters.

[0021] As a further optimization scheme of the present invention, the design method of the loss function is as follows:

[0022] The loss function is designed using the image constraint of the viewpoint position of the subaperture array and the straight line constraint of the EPI;

[0023] The overall loss function L of the model during the training phase Total The definition is as follows:

[0024] L Total =L MSE +λ1·L MMD +λ2·L EPI

[0025] Where, L MSE Represents minimizing the mean square error loss between the network model predicted image and the real image, L MMDRepresents the difference loss between the distribution of the network model fitting and the real data distribution. This loss uses the Wasserstein distance to measure the network model fitting and reconstruction of dense light field data LF n distributed and real dense light field data LF n Distribution q Φ (z|LF n ), that is, L EPI It represents minimizing the mean square error loss between the pixel values ​​of the corresponding positions of all object points in the central view of the image predicted by the network model at different viewpoints;

[0026] L MSE It represents minimizing the mean square error loss between the network model predicted image and the real image. Its calculation formula is as follows:

[0027]

[0028] In the formula, represents the kth reconstructed view in the N×N dense light field, Indicates the corresponding real perspective;

[0029] L EPI It represents the mean square error loss between the pixel values ​​of the corresponding positions of all object points at different viewpoints in the central view of the image predicted by the network model, and its calculation formula is as follows:

[0030]

[0031] In the formula, is the pixel value at the point (x0, y0) in the central sub-aperture image of the light field, (x0, y0) is the position of the object point in the central sub-aperture image of the light field, I u,v (x, y) is the pixel value at each object point in the light field sub-aperture image, x=x0+(u-u0)disp, and disp is the disparity value corresponding to the object point at that position in the light field sub-aperture image.

[0032] As a further optimization solution of the present invention, the encoder is mainly composed of an encoding module 1, an encoding module 2 (E2), a feature extraction module and a disparity estimation module (P);

[0033] The encoding module 1 is divided into horizontal encoding modules and vertical encoding module Used to process 4D light field data and save computing resources;

[0034] The feature extraction module is divided into a feature extraction module (F1) and a feature extraction module (F2).

[0035] As a further optimization scheme of the present invention, the encoder uses the space angle consistency constraint relationship of the 4D light field data to convert the input 4D light field data LF n Mapping to the latent space to obtain the latent variable z, the specific process is as follows:

[0036] Step 1: Input light field data LF n pass and Get the features respectively and Denoted as x1 and x2;

[0037] Input LF n Through P, we get the feature P(LF n ), denoted as d;

[0038] The input central sub-aperture image (LF0) is passed through F1 to obtain the feature F1(LF0), denoted as m;

[0039] Step 2: Concatenate channels x1 and x2 to obtain x3 = concate[x1,x2];

[0040] Concatenate x3, d and m channels to get feature x4 = concate[x3, d, m], and use it as the input of E2 to get feature E2(x4), denoted as x5;

[0041] Step 3: Concatenate x5 and x6 channels to get x7 = concate[x5,x6], which is used as a fully connected layer (f c ) input to obtain the latent variable z = f c (x7), where m is passed through F2 to obtain the feature F2(m), denoted as x6.

[0042] As a further optimization solution of the present invention, the decoder is mainly composed of a decoding module 1, a decoding module 2 (D2), a feature extraction module and a disparity estimation module (P); wherein the decoding module 1 is divided into a horizontal decoding module and vertical decoding module

[0043] As a further optimization scheme of the present invention, the decoder uses the spatial angle consistency constraint relationship of the 4D light field to reversely map the latent variable z in the latent space to generate a dense light field The specific process is as follows:

[0044] Step 1: Input LF0 and obtain F2(m) through F1 and F2, denoted as x6;

[0045] Concatenate x6 and latent variable z to obtain feature y1=concate[x6,z], and use y1 as f c The input gets the feature fc (y1), denoted as y2;

[0046] Step 2: Input y2 into D2 to obtain feature D2(y2), denoted as y3;

[0047] Step 3: LF0 obtains feature F1(LF0) through F1, denoted as m;

[0048] Input LF n Through P, we get the feature P(LF n ), denoted as d;

[0049] Concatenate m, d and y3 to get y4 = concate[y3, d, m], and input y4 into and Get Features and Denoted as y5 and y6;

[0050] Step 4: Concatenate y5 and y6 to obtain feature y7 = concate[y5,y6], and input y7 into the four-dimensional convolution layer (f) to generate a dense light field

[0051] A light field angle super-resolution imaging device based on unsupervised learning, comprising a collection unit and an angle domain super-resolution reconstruction unit;

[0052] The acquisition unit is used to acquire sparse light field sub-aperture images LF at equal intervals on the viewpoint plane. γ As the input of the disparity estimation module (P), the light field data LF γ The central sub-aperture image LF0 is used as the input of the feature extraction module (F), and the initial disparity map of the scene and the central sub-aperture features of different levels are output;

[0053] The angle domain super-resolution reconstruction unit uses a light field angle super-resolution model based on a conditional VAE model to reconstruct the LF γ All the light field sub-aperture images in the angular domain are reconstructed synchronously to obtain light field data with t times the angular domain resolution.

[0054] Also includes an encoder unit and a decoder unit;

[0055] The encoder unit is used to convert the light field sub-aperture image LF into a sparse light field image LF with equal intervals in the viewpoint plane. γ The obtained initial disparity map and the central sub-aperture features at different levels are used as the conditional VAE light field angle super-resolution model encoder E θ1 Input, output latent variable z in the latent space;

[0056] The decoder unit is used to combine the obtained latent variable z with the obtained initial disparity map and the central sub-aperture features of different levels as the conditional VAE light field angle super-resolution model decoder D θ2 The input is densely spaced light field sub-aperture images in the viewpoint plane.

[0057] The beneficial effects of the present invention are as follows: the present invention can be independent of large-scale high-resolution training data sets, and can simultaneously perform super-resolution reconstruction in the angle domain for all light field sub-aperture images. A space-angle consistency hybrid loss function based on light field data is designed, so that the new model can utilize the disparity information coupled in the sub-aperture images when reconstructing sub-aperture images with high angle resolution. The numerical experimental results on synthetic and real data sets are close to those of supervised methods, and can also meet the requirements of deep learning-based methods for super-resolution in the angle domain of light fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a schematic flow diagram of the overall method of the present invention;

[0059] Figure 2 It is a schematic diagram of a summary model of the light field angle super-resolution method based on the conditional VAE model of the present invention;

[0060] Figure 3 It is a schematic diagram of the encoder structure of the present invention;

[0061] Figure 4 It is a schematic diagram of the decoder structure of the present invention. DETAILED DESCRIPTION

[0062] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.

[0063] Example

[0064] like Figure 1-Figure 4 As shown, a light field angle super-resolution method based on a conditional VAE model includes the following steps:

[0065] S1: By collecting sparse light field sub-aperture image arrays LF at equal intervals in the viewpoint plane γ The central sub-aperture image LF0 is input into the disparity estimation module (P) and the feature extraction module (F) respectively to obtain the initial disparity map d∈R H×W and central subaperture characteristics at different levels;

[0066] The calculation formula of the central subaperture characteristic is:

[0067] d=P(LF γ ), m=F1(LF0) and x6=F2(m);

[0068] Where, F1 represents the feature extraction module of the first three layers of convolution, F2 represents the feature extraction module of the last three layers of convolution, d represents the initial disparity map, m represents the feature variable extracted by LF0 through F1, and x6 represents the feature variable extracted by LF0 through F1 and F2;

[0069] It should be noted that obtaining the initial disparity map and feature variables at different levels specifically includes:

[0070] For example, we select the public light field dataset HCI-4D and Stanford light field dataset, and collect sparse light field sub-aperture images LF at equal intervals in the viewpoint plane. γ ∈R 3×U×V×H×W , R represents a real number, 3 represents the number of channels, U×V represents the angular resolution, and H×W represents the spatial resolution; the spatial resolution currently used is 512×512; taking the acquisition angle dimension of 3×3 as an example, that is, U×V=3×3, the dimension of each sub-aperture image in the spatial domain is 3×H×W=3×512×512, and the light field data LF collected from the viewpoint plane is equally spaced and sparse γ ∈R 3×3×3×512×512 , light field data LF γ The central sub-aperture image LF0∈R 3×1×1×512×512 .

[0071] S2: LF γ The initial disparity map obtained by S1 and the central sub-aperture features at different levels are used as the conditional VAE light field angle super-resolution model encoder E θ1 Input, output latent variable z in the latent space;

[0072] The calculation formula of the latent variable z in the latent space is:

[0073]

[0074] Where θ1 is , z is a latent variable in the latent space, where the elements in z follow the standard normal distribution N(0,1);

[0075] The encoder is mainly composed of encoding module 1, encoding module 2 (E2), feature extraction module and disparity estimation module (P); encoding module 1 is divided into horizontal encoding module and vertical encoding module Used to process 4D light field data and save computing resources; the feature extraction module is divided into a feature extraction module (F1) and a feature extraction module (F2);

[0076] The encoder uses the spatial angle consistency constraint relationship of the 4D light field data to convert the input 4D light field data LF n Mapping to the latent space obtains the latent variable z; the specific process is as follows:

[0077] Step 1: Input light field data LF n pass and Get the features respectively and Denoted as x1 and x2;

[0078] Input LF n Through P, we get the feature P(LF n ), denoted as d;

[0079] The input central sub-aperture image (LF0) is passed through F1 to obtain the feature F1(LF0), denoted as m;

[0080] Step 2: Concatenate channels x1 and x2 to obtain x3 = concate[x1,x2];

[0081] Concatenate x3, d and m channels to get feature x4 = concate[x3, d, m], and use it as the input of E2 to get feature E2(x4), denoted as x5;

[0082] Step 3: Concatenate x5 and x6 channels to get x7 = concate[x5,x6], which is used as a fully connected layer (f c ) input to obtain the latent variable z = f c (x7), where m is obtained by F2 to obtain the feature F2(m), denoted as x6;

[0083] It should be noted that obtaining the hidden variable z in the latent space specifically includes:

[0084] The light field sub-aperture image LF is sparsely spaced in the viewpoint plane. γ ∈R 3×U×V×H×W The initial disparity map d∈R obtained from S1 H×W and the central sub-aperture features at different levels as conditional VAE light field angle super-resolution model encoder E θ1 The input and output latent variables z∈R in the latent space C′×tU×tV×H×W :

[0085]

[0086] Where θ1 is Parameter set, LF γ is an array of equally spaced and sparse light field sub-aperture images in the viewpoint plane, z is a latent variable in the latent space, where the elements in z obey the standard normal distribution N(0,1), and t is a multiple of the angular super-resolution.

[0087] For example, for reconstructing 3x angular super-resolution, t is 3, and the conditional VAE model encoder network Input LF γ ∈R 3×3×3×512×512 and LF0∈R 3×1×1×512×512 , output hidden variable z∈R C′×9×9×512×512 , where C′=160.

[0088] S3: The latent variable z obtained in S2 and the initial disparity map obtained in S1 and the central sub-aperture features at different levels are used as the conditional VAE light field angle super-resolution model decoder The input is densely spaced light field sub-aperture images in the viewpoint plane.

[0089] Light field subaperture images The calculation formula is:

[0090]

[0091] In the formula, θ2 is The parameter set of

[0092] The decoder is mainly composed of decoding module 1, decoding module 2 (D2), feature extraction module and disparity estimation module (P); the decoding module 1 is divided into horizontal decoding module and vertical decoding module

[0093] The decoder uses the spatial angle consistency constraint of the 4D light field to reversely map the latent variable z in the latent space to generate a dense light field The specific process is as follows:

[0094] Step 1: Input LF0 and obtain F2(m) through F1 and F2, denoted as x6;

[0095] Concatenate x6 and latent variable z to obtain feature y1=concate[x6,z], and use y1 as f c The input gets the feature f c (y1), denoted as y2;

[0096] Step 2: Input y2 into D2 to obtain feature D2(y2), denoted as y3;

[0097] Step 3: LF0 obtains feature F1(LF0) through F1, denoted as m;

[0098] Input LF n Through P, we get the feature P(LF n ), denoted as d;

[0099] Concatenate m, d and y3 to get y4 = concate[y3, d, m], and input y4 into and Get Features and Denoted as y5 and y6;

[0100] Step 4: Concatenate y5 and y6 to obtain feature y7 = concate[y5,y6], and input y7 into the four-dimensional convolution layer (f) to generate a dense light field

[0101] It should be noted that obtaining densely spaced light field sub-aperture images in the viewpoint plane Specifically include:

[0102] The hidden variable z∈R obtained by S2 C′×tU×tV×H×W The initial disparity map d∈R obtained from S1 H×W And the central sub-aperture features at different levels as the conditional VAE light field angle super-resolution model decoder The input is densely spaced light field sub-aperture images in the viewpoint plane.

[0103]

[0104] In the formula, θ2 is A collection of parameters.

[0105] For example, for reconstructing 3x angular super-resolution, t is 3, and the conditional VAE model decoder network The input z∈R C′×9×9×512×512 and LF0∈R 3×1×1×512×512 , output densely spaced light field sub-aperture images in the viewpoint plane

[0106] S4: The loss function is designed by constraining the spatial angle consistency of light field data to optimize the parameters of the conditional VAE model;

[0107] The loss function is designed as follows:

[0108] The loss function is designed using the image constraint of the viewpoint position of the subaperture array and the straight line constraint of the EPI;

[0109] The overall loss function L of the model during the training phase Total The definition is as follows:

[0110] L Total =L MSE +λ1·L MMD +λ2·L EPI

[0111] Where, L MSE Represents minimizing the mean square error loss between the network model predicted image and the real image, L MMD Represents the difference loss between the distribution of the network model fitting and the real data distribution. This loss uses the Wasserstein distance to measure the network model fitting and reconstruction of dense light field data LF n distributed and real dense light field data LF n Distribution q Φ (z|LF n ), that is, L EPI It represents minimizing the mean square error loss between the pixel values ​​of the corresponding positions of all object points in the central view of the network model predicted on the image at different viewpoints. In the numerical experiment, the weights are λ1=100, λ2=1;

[0112] L MSE It represents minimizing the mean square error loss between the network model predicted image and the real image. Its calculation formula is as follows:

[0113]

[0114] In the formula, represents the kth reconstructed view in the N×N dense light field, Indicates the corresponding real perspective;

[0115] L EPI It represents the mean square error loss between the pixel values ​​of the corresponding positions of all object points at different viewpoints in the central view of the image predicted by the network model, and its calculation formula is as follows:

[0116]

[0117] In the formula, is the pixel value at the point (x0, y0) in the central sub-aperture image of the light field, (x0, y0) is the position of the object point in the central sub-aperture image of the light field, I u,v (x, y) is the pixel value at each object point in the light field sub-aperture image, x = x0 + (u-u0) disp, disp is the disparity value corresponding to the object point at that position in the light field sub-aperture image;

[0118] The spatial angle consistency of light field data is reflected differently in different light field parameterization representations. The light field data based on subaperture array parameterization has global spatial angle consistency, which is reflected in the parallax constraint in viewpoint propagation. The spatial angle consistency of EPI-based parameterized light field data is reflected in the straight line constraint of pixel points. The loss function is designed by using the image constraint of the viewpoint position of the subaperture array and the straight line constraint of EPI, which can effectively ensure the spatial angle consistency of light field data. This strategy can effectively improve the performance of the conditional VAE model.

[0119] It should be noted that the unsupervised learning-based method is introduced into the angular super-resolution reconstruction task of light field data, and the generative network is introduced in the neural network model architecture design to generate light field sub-aperture images with dense planes with equal viewpoint intervals. θ (·) The conditional VAE model is used as the backbone network. The architecture of the conditional VAE model is improved to make it suitable for the angle super-resolution task of the light field. Its structure is as follows Figure 2 As shown;

[0120] Specifically, the neural network N θ (·) It includes an encoder, a decoder, a disparity estimation model, and a feature extraction model, and inputs a sparse light field sub-aperture image LF with equal spacing in the viewpoint plane. γ ∈R 3×3×3×512×512 , and simultaneously reconstruct the light field sub-aperture image in the angle domain to obtain light field data with 3 times the angle domain resolution. Specifically include:

[0121] The encoder part is mainly composed of encoding module 1, encoding module 2 (E2), feature extraction module and disparity estimation module (P), such as Figure 3 As shown;

[0122] In order to better process 4D light field data and save computing resources, the encoding module 1 is divided into horizontal encoding modules and vertical encoding module The feature extraction module is divided into feature extraction module (F1) and feature extraction module (F2) according to different levels of convolution. The encoder uses the spatial angle consistency constraint relationship of the 4D light field data to convert the input 4D light field data LF n Map to the latent space to obtain the latent variable z;

[0123] The decoder part is mainly composed of decoding module 1, decoding module 2 (D2), feature extraction module and disparity estimation module (P), such as Figure 4 As shown; wherein the decoding module 1 is divided into a horizontal decoding module and vertical decoding module The decoder uses the spatial angle consistency constraint of the 4D light field to reversely map the latent variable z in the latent space to generate a dense light field

[0124] In one embodiment, the encoder receives the input viewpoint plane equally spaced sparse light field sub-aperture images LF γ The method of sequentially performing convolution and feature extraction specifically includes:

[0125] LF light field sub-aperture image with equal spacing and sparseness in the viewpoint plane γ First, sequentially pass through a first convolution kernel of a first preset size, a first normalization layer BN, and a first activation function layer Relu;

[0126] Then, the first convolution kernel of the same first preset size, the first normalization layer BN, and the first activation function layer Relu are sequentially passed through for 5 times;

[0127] Finally, 5 feature maps of different scales are output;

[0128] Among them, the normalization layer improves the performance of the network by normalizing the input data, reducing the gradient disappearance and gradient explosion problems, accelerating network training, improving network generalization performance, and reducing dependence on hyperparameters. The activation function layer removes negative values ​​in the convolution results and keeps the positive values ​​unchanged.

[0129] The decoder obtains densely spaced light field sub-aperture images in the viewpoint plane. The methods specifically include:

[0130] The decoding layer first passes through the first normalization layer BN, the first convolution kernel of a preset size, the first normalization layer BN, and the first activation function layer Relu in sequence;

[0131] Then, it passes through the second convolution kernel of preset size, the first normalization layer BN, the first activation function layer Relu, and the last layer is the final network output (Output), obtaining a dense light field sub-aperture image with equal intervals in the viewpoint plane.

[0132] It should be noted that the obtained light field sub-aperture image with dense and equal spacing in the viewpoint plane It may need to be adjusted according to the parameters determined above.

[0133] It should be noted that, the encoder disclosed in the above embodiment can also be used to input the equally spaced and sparse light field sub-aperture image LF of the viewpoint plane. γ Method for sequentially performing convolution and feature extraction, decoder obtains light field sub-aperture images with dense equal spacing in the viewpoint plane The method can be appropriately modified to obtain a similar method with the same function.

[0134] S5: The light field sub-aperture image with equal intervals and sparseness in the viewpoint plane is subjected to light field angle super-resolution through a conditional VAE model constrained by spatial angle consistency to obtain a light field sub-aperture image with equal intervals and denseness in the viewpoint plane;

[0135] Complete angular domain super-resolution imaging of four-dimensional light field data.

[0136] By using the light field angle super-resolution method based on the conditional VAE model provided by the present invention, it is possible to achieve angle super-resolution of the light field based on an unsupervised deep learning method, providing important technical support for computational imaging and three-dimensional scene reconstruction.

[0137] A light field angle super-resolution imaging device based on unsupervised learning, comprising a collection unit and an angle domain super-resolution reconstruction unit;

[0138] The acquisition unit is used to acquire sparse light field sub-aperture images LF at equal intervals in the viewpoint plane. γ As the input of the disparity estimation module (P), the light field data LF γ The central sub-aperture image LF0 is used as the input of the feature extraction module (F), and the initial disparity map of the scene and the central sub-aperture features of different levels are output;

[0139] The angle domain super-resolution reconstruction unit uses the light field angle super-resolution model based on the conditional VAE model to reconstruct the LF γ All the light field sub-aperture images in the angular domain are reconstructed synchronously to obtain light field data with t times the angular domain resolution.

[0140] Also includes an encoder unit and a decoder unit;

[0141] The encoder unit is used to convert the sparse light field sub-aperture images LF into equally spaced sparse light field images in the viewpoint plane. γ The obtained initial disparity map and the central sub-aperture features at different levels are used as the conditional VAE light field angle super-resolution model encoder Input, output latent variable z in the latent space;

[0142] The decoder unit is used to combine the obtained latent variable z with the obtained initial disparity map and the central sub-aperture features at different levels as the conditional VAE light field angle super-resolution model decoder The input is densely spaced light field sub-aperture images in the viewpoint plane.

[0143] The light field angle domain super-resolution imaging method based on the conditional VAE model combined with the spatial angle consistency loss provided by the present invention is adopted, so that the light field super-resolution based on deep learning does not rely on large-scale high-resolution training data sets.

[0144] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A light field angle super-resolution method based on conditional VAE model, It is characterized in that The steps include: S1: By collecting sparse light field sub-aperture image arrays LF at equal intervals in the viewpoint plane γ and the central subaperture image LF 0 The disparity estimation module (P) and feature extraction module (F) are input respectively to obtain the initial disparity map of the scene and the central sub-aperture features of different levels; S2: LF γ The initial disparity map obtained by S1 and the central sub-aperture features at different levels are used as the conditional VAE light field angle super-resolution model encoder E θ1 Input, output latent variable z in the latent space; S3: The latent variable z obtained in S2 and the initial disparity map obtained in S1 and the central sub-aperture features at different levels are used as the conditional VAE light field angle super-resolution model decoder The input is densely spaced light field sub-aperture images in the viewpoint plane. S4: The loss function is designed by constraining the spatial angle consistency of light field data to optimize the parameters of the conditional VAE model; S5: The sparse light field sub-aperture images at equal intervals in the viewpoint plane are subjected to light field angle super-resolution through a conditional VAE model constrained by spatial angle consistency to obtain dense light field sub-aperture images at equal intervals in the viewpoint plane.

2. According to the light field angle super-resolution method based on the conditional VAE model of claim 1, Features: The calculation formula of the central sub-aperture characteristic is: d = P(LF γ )、m = F 1 (LF 0 ) and x 6 = F 2 (m); In the formula, F 1 represents the feature extraction module of the first three convolution layers, F 2 represents the feature extraction module of the last three layers of convolution, d represents the initial disparity map, and m represents LF 0 By F 1 The extracted feature variables, x 6 Represents LF 0 By F 1 、F 2 Extracted feature variables.

3. According to the light field angle super-resolution method based on the conditional VAE model of claim 2, Features: The calculation formula of the latent variable z in the latent space is: In the formula, θ 1 for , z is a hidden variable in the latent space, where the elements in z obey the standard normal distribution N(0,1).

4. According to the light field angle super-resolution method based on the conditional VAE model of claim 3, Features: The light field subaperture image The calculation formula is: In the formula, θ 2 for A collection of parameters.

5. According to the light field angle super-resolution method based on the conditional VAE model of claim 4, Features: The loss function is designed as follows: The loss function is designed using the image constraint of the viewpoint position of the subaperture array and the straight line constraint of the EPI; The overall loss function L of the model during the training phase Total The definition is as follows: L Total =L MSE +λ 1 ·L MMD +λ 2 ·L EPI Where, L MSE Represents minimizing the mean square error loss between the network model predicted image and the real image, L MMD Represents the difference loss between the distribution of the network model fitting and the real data distribution. This loss uses the Wasserstein distance to measure the network model fitting and reconstruction of dense light field data LF n distributed and real dense light field data LF n Distribution q Φ (z|LF n ), that is, L EPI It represents minimizing the mean square error loss between the pixel values ​​of the corresponding positions of all object points in the central view of the image predicted by the network model at different viewpoints; L MSE It represents minimizing the mean square error loss between the network model predicted image and the real image. Its calculation formula is as follows: In the formula, represents the kth reconstructed view in the N×N dense light field, Indicates the corresponding real perspective; L EPI It represents minimizing the mean square error loss between the pixel values ​​of the corresponding positions of all object points in the central view of the network model predicted on the image at different viewpoints. The calculation formula is as follows: In the formula, is the sub-aperture image at the center of the light field (x 0 ,y 0 ) pixel value at the point position, (x 0 ,y 0 ) is the position of the object point in the sub-aperture image at the center of the light field, I u,v (x, y) is the pixel value at each object point in the light field subaperture image, x = x 0 +(uu 0 )disp, disp is the disparity value corresponding to the object point at that position in the light field sub-aperture image.

6. According to the light field angle super-resolution method based on the conditional VAE model of claim 5, Features: The encoder is mainly composed of encoding module 1, encoding module 2 (E 2 ), feature extraction module and disparity estimation module (P); The encoding module 1 is divided into horizontal encoding modules and vertical encoding module Used to process 4D light field data and save computing resources; The feature extraction module is divided into a feature extraction module (F 1 ) and feature extraction module (F 2 ).

7. The light field angle super-resolution method based on the conditional VAE model according to claim 6, Features: The encoder uses the spatial angle consistency constraint relationship of the 4D light field data to convert the input 4D light field data LF n Mapping to the latent space to obtain the latent variable z, the specific process is as follows: Step 1: Input light field data LF n pass and Get the features respectively and Note that x 1 and x 2 ; Input LF n Through P, we get the feature P(LF n ), denoted as d; Input central subaperture image (LF 0 )Through F 1 Get feature F 1 (LF 0 ), denoted as m; Step 2: x 1 and x 2 Channel splicing to get x 3 =concate[x 1 ,x 2 ]; x 3 , d and m channels are concatenated to obtain feature x 4 =concate[x 3 ,d,m], and take it as E 2 The input gets the feature E 2 (x 4 ), denoted as x 5 ; Step 3: x 5 and x 6 Channel splicing to get x 7 =concate[x 5 ,x 6 ], and use it as a fully connected layer (f c ) input to obtain the latent variable z = f c (x 7 ), where m is obtained by F 2 Get feature F 2 (m), denoted by x 6 .

8. According to the light field angle super-resolution method based on the conditional VAE model of claim 7, Features: The decoder is mainly composed of a decoding module 1 and a decoding module 2 (D 2 ), feature extraction module and disparity estimation module (P); the decoding module 1 is divided into horizontal decoding module and vertical decoding module 9. The light field angle super-resolution method based on the conditional VAE model according to claim 8, Features: The decoder uses the spatial angle consistency constraint of the 4D light field to reversely map the latent variable z in the latent space to generate a dense light field The specific process is as follows: Step 1: Enter LF 0 By F 1 and F 2 Get F 2 (m), denoted by x 6 ; x 6 Concatenate the channel with the latent variable z to get the feature y 1 =concate[x 6 ,z], and y 1 As f c The input gets the feature f c (y 1 ), denoted as y 2 ; Step 2: y 2 Input D 2 Get feature D 2 (y 2 ), denoted as y 3 ; Step 3: LF 0 By F 1 Get feature F 1 (LF 0 ), denoted as m; Input LF n Through P, we get the feature P(LF n ), denoted as d; m, d and y 3 Perform channel splicing to get y 4 =concate[y 3 ,d,m], and y 4 Enter separately and Get Features and Note: y 5 and 6 ; Step 4: y 5 and 6 Perform channel splicing to obtain feature y 7 =concate[y 5 ,y 6 ], and change y 7 Input to the 4D convolutional layer (f) to generate a dense light field 10. A light field angle super-resolution imaging device based on unsupervised learning, applied to the light field angle super-resolution method based on conditional VAE model as claimed in claim 9, Features: It includes an acquisition unit and an angle-domain super-resolution reconstruction unit; The acquisition unit is used to acquire sparse light field sub-aperture images LF at equal intervals on the viewpoint plane. γ As the input of the disparity estimation module (P), the light field data LF γ The central subaperture image LF 0 As the input of the feature extraction module (F), the initial disparity map of the scene and the central sub-aperture features at different levels are output; The angle domain super-resolution reconstruction unit uses a light field angle super-resolution model based on a conditional VAE model to reconstruct the LF γ All the light field sub-aperture images in the angular domain are reconstructed synchronously to obtain light field data with t times the angular domain resolution. Also includes an encoder unit and a decoder unit; The encoder unit is used to convert the light field sub-aperture image LF into a sparse light field image LF with equal intervals in the viewpoint plane. γ The obtained initial disparity map and the central sub-aperture features at different levels are used as the conditional VAE light field angle super-resolution model encoder Input, output latent variable z in the latent space; The decoder unit is used to combine the obtained latent variable z with the obtained initial disparity map and the central sub-aperture features of different levels as a conditional VAE light field angle super-resolution model decoder The input is densely spaced light field sub-aperture images in the viewpoint plane.

Images