Multi-defocus scene-oriented light field image super-resolution reconstruction method and device

Abstract

The invention provides a light field image super-resolution reconstruction method and device for multiple out-of-focus scenes. The method comprises the following steps: by taking a depth mask as guidance, carrying out image partitioning on a low-resolution multi-defocus light field image by utilizing an image partitioning algorithm to obtain a low-resolution local light field image; inputting the low-resolution local light field image into a pre-trained super-resolution reconstructed image network model to obtain a local light field super-resolution reconstructed image; the network model comprises a fuzzy feature extraction module, a light field super-resolution module and a degradation generation module, the fuzzy feature extraction module is used for providing multi-scale fuzzy information for the light field super-resolution module, the light field super-resolution module is used for extracting light field features and performing super-resolution on the light field features, and the degradation generation module embeds degradation features into the light field super-resolution module; the degradation guiding module is used for performing degradation guiding on the light field super-resolution module; and adding the plurality of local light field super-resolution reconstructed images to obtain a full-focusing light field super-resolution reconstructed image. According to the scheme, the super-resolution of the out-of-focus fuzzy light field image in a multi-depth complex scene is realized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of light field image super-resolution reconstruction, and in particular to a light field image super-resolution reconstruction method and device for multi-defocus scenes. Background Art

[0002] As a new type of snapshot imaging technology, light field imaging technology has attracted widespread attention due to its special four-dimensional imaging structure. The uniqueness of light field imaging technology is mainly due to the advanced equipment it relies on - the light field camera. Thanks to its innovative design, this camera can record the intensity and direction information of light in the form of a light field, where the viewing angle dimension records the direction of light and the spatial dimension records the intensity of light. In a single shot, it can capture multiple images of a scene from different perspectives, providing more comprehensive visual information.

[0003] Although light field imaging technology has brought many innovations and advantages, it also faces some limitations and challenges. Due to the need to save both spatial and angular information, and the size of the detector is limited, the spatial resolution of light field images is lower than that of traditional imaging technologies that only need to save spatial information. At the same time, in complex scenes, especially when there are multiple target objects and the distances between them differ greatly, targets at different distances will show different degrees of defocus blur in the light field image during the imaging process because the light fails to converge on a common plane before reaching the imaging sensor. However, existing light field super-resolution networks are generally not optimized specifically for this specific problem in complex scenes.

[0004] Therefore, there is an urgent need to provide a light field image super-resolution reconstruction method and device for multi-defocus scenes. Summary of the invention

[0005] In order to solve the problem that traditional light field super-resolution networks cannot achieve super-resolution of defocused light field images in multi-depth complex scenes, an embodiment of the present invention provides a light field image super-resolution reconstruction method and device for multi-defocus scenes.

[0006] In a first aspect, an embodiment of the present invention provides a light field image super-resolution reconstruction method for multi-defocus scenes, the method comprising:

[0007] Using the pre-generated depth mask as a guide, the obtained low-resolution multi-defocus light field image is segmented using an image segmentation algorithm to obtain a number of low-resolution local light field images at different depths;

[0008] Input each low-resolution local light field image of different depths into a pre-trained super-resolution reconstructed image network model to obtain a corresponding number of local light field super-resolution reconstructed images of different depths;

[0009] The super-resolution image reconstruction network model is trained with a light field image data set as a training sample set. The super-resolution reconstruction network model includes a fuzzy feature extraction module, a light field super-resolution module and a generation degradation module. The fuzzy feature extraction module is used to provide multi-scale fuzzy information for the light field super-resolution module. The light field super-resolution module is used to extract light field features and perform super-resolution on them. The generation degradation module embeds degradation features into the light field super-resolution module to guide the degradation of the light field super-resolution module.

[0010] The local light field super-resolution reconstructed images at each depth are added together to obtain the full-focus light field super-resolution reconstructed image.

[0011] In a second aspect, an embodiment of the present invention further provides a light field image super-resolution reconstruction device for multi-defocus scenes, the device comprising:

[0012] An image blocking unit is used to block the obtained low-resolution multi-defocus light field image using an image blocking algorithm with a pre-generated depth mask as a guide, so as to obtain a plurality of low-resolution local light field images at different depths;

[0013] An image super-resolution reconstruction unit is used to input each low-resolution local light field image of different depths into a pre-trained super-resolution reconstruction image network model to obtain a corresponding number of local light field super-resolution reconstruction images of different depths;

[0014] The super-resolution image reconstruction network model is trained with a light field image data set as a training sample set. The super-resolution reconstruction network model includes a fuzzy feature extraction module, a light field super-resolution module and a generation degradation module. The fuzzy feature extraction module is used to provide multi-scale fuzzy information for the light field super-resolution module. The light field super-resolution module is used to extract light field features and perform super-resolution on them. The generation degradation module embeds degradation features into the light field super-resolution module to guide the degradation of the light field super-resolution module.

[0015] The combining unit is used to add the local light field super-resolution reconstructed images at each depth to obtain the full-focus super-resolution reconstructed image.

[0016] In a third aspect, an embodiment of the present invention further provides a computing device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in any embodiment of this specification is implemented.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method described in any embodiment of this specification.

[0018] On the other hand, an embodiment of the present application further provides a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes any of the methods described in the above embodiments.

[0019] The embodiment of the present invention provides a light field image super-resolution reconstruction method and device for multi-defocus scenes. First, a multi-defocus light field degradation process is designed according to a light field imaging model, thereby constructing a multi-defocus light field training data set; then, guided by a depth mask, an image segmentation algorithm is used to perform image segmentation to obtain low-resolution local light field images of different depths, and a super-resolution image network model is built to design it as a pyramid-type multi-scale feature extraction structure. At the same time, on the basis of introducing degradation prior information, the network model is used to perform image super-resolution on the extracted low-resolution local light field images of different depths to obtain a local light field super-resolution reconstructed image; finally, by synthesizing the local light field super-resolution reconstructed images of all depth layers, the generation of a complete all-focus super-resolution image is achieved. It can be seen that this scheme can fully consider the degradation characteristics of different depth layers in the image reconstruction process, thereby achieving the super-resolution effect of defocused blurred light field images in multi-depth complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of a light field image super-resolution reconstruction method for multiple defocus scenes provided by one embodiment of the present invention;

[0022] Figure 2 It is a framework diagram of a light field image super-resolution reconstruction method for multiple defocus scenes provided by an embodiment of the present invention;

[0023] Figure 3 It is an overall structural diagram of a super-resolution image reconstruction network model provided by an embodiment of the present invention;

[0024] Figure 4 It is a specific structural diagram of a deep convolution submodule and a multi-feature extraction layer in a super-resolution image reconstruction network model provided by an embodiment of the present invention;

[0025] Figure 5It is a specific structural diagram of the degenerate integration layer in the deep convolution submodule provided by an embodiment of the present invention;

[0026] Figure 6 It is a schematic diagram of a process of generating a degradation fuzzy matrix by using a generation degradation module provided by an embodiment of the present invention;

[0027] Figure 7 is a specific structural diagram of three feature extractors provided in one embodiment of the present invention;

[0028] Figure 8 Schematic diagram of the process of constructing a polar plane three-dimensional feature matrix provided by an embodiment of the present invention;

[0029] Fig. 9 It is a qualitative result diagram of double super-resolution at different depths provided by an embodiment of the present invention;

[0030] Fig.10 is a hardware architecture diagram of a computing device provided by an embodiment of the present invention;

[0031] Fig.11 It is a structural diagram of a light field image super-resolution reconstruction device for multiple defocus scenes provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] At present, light field image super-resolution reconstruction technology based on deep learning has become the most widely adopted method for improving spatial resolution due to its lower cost, easier implementation and better effect. Most light field image super-resolution algorithms based on deep learning are designed to improve the spatial resolution of light field images. However, these algorithms do not consider that targets with different pixel depths have different defocus blurs after actual imaging. Moreover, most of them can only complete the super-resolution task of one depth and restore the detailed information of a certain depth, but cannot handle image blurs at other depths.

[0034] To address this problem, the inventors explored how to make full use of the depth information in light field images, and on this basis carried out full-focus light field super-resolution research based on deep learning, considering the degradation characteristics of light field images with different defocus levels, thereby achieving full-focus light field image reconstruction of multi-target depth scenes, which has important practical value for subsequent actual light field applications.

[0035] The specific implementation of the above concept is described below.

[0036] Please refer to Figure 1 The embodiment of the present invention provides a light field image super-resolution reconstruction method for multi-defocus scenes, the method comprising:

[0037] Step 100, using a pre-generated depth mask as a guide, using an image segmentation algorithm to segment the obtained low-resolution multi-defocus light field image to obtain a plurality of low-resolution local light field images at different depths;

[0038] Step 102, inputting each low-resolution local light field image of different depths into a pre-trained super-resolution reconstruction image network model to obtain a corresponding number of local light field super-resolution reconstruction images of different depths; wherein the super-resolution image reconstruction network model is trained using a light field image data set as a training sample set, and the super-resolution reconstruction network model includes a fuzzy feature extraction module, a light field super-resolution module, and a generation degradation module, the fuzzy feature extraction module is used to provide multi-scale fuzzy information for the light field super-resolution module, the light field super-resolution module is used to extract light field features and perform super-resolution on them, and the generation degradation module embeds degradation features into the light field super-resolution module to guide the degradation of the light field super-resolution module;

[0039] Step 104 , adding the local light field super-resolution reconstructed images at each depth to obtain a full-focus super-resolution reconstructed image.

[0040] In the embodiment of the present invention, Figure 2 As shown in the figure, firstly, the multi-defocus light field degradation process is designed according to the light field imaging model, so as to construct a multi-defocus light field training data set; then, guided by the depth mask, the image segmentation algorithm is used to segment the image to obtain low-resolution local light field images of different depths, and the super-resolution image network model is built, which is designed as a pyramid-type multi-scale feature extraction structure. At the same time, on the basis of introducing degradation prior information, the network model is used to perform image super-resolution on the extracted low-resolution local light field images of different depths to obtain the local light field super-resolution reconstructed image; finally, by synthesizing the local light field super-resolution reconstructed images of all depths, the generation of a complete full-focus super-resolution image is achieved. It can be seen that this scheme can fully consider the degradation characteristics of different depth layers in the image reconstruction process, thereby achieving the super-resolution effect of defocused blurred light field images in multi-depth complex scenes.

[0041] For step 100:

[0042] In some embodiments, step 100 includes:

[0043] Based on the light field standard disparity map in the light field image dataset, N depth mask matrices are generated; wherein the pixel value of the i-th depth in each mask matrix is ​​1, and the pixel values ​​of the remaining depths are 0;

[0044] Expanding the pixels of each depth mask matrix according to the resolution of the low-resolution multi-defocus light field image to obtain an expanded mask matrix with N dimensions matching the low-resolution multi-defocus light field image;

[0045] Each of the extended mask matrices is multiplied by the low-resolution multi-defocus light field image to obtain a plurality of low-resolution local light field images at different depths.

[0046] In the embodiment of the present invention, the light field standard disparity map The resolution is consistent with the spatial resolution of the high-resolution light field image. By traversing the pixels in the light field standard disparity map, N mask matrices are generated. After obtaining the mask matrix, the resolution of the mask matrix can be made consistent with the resolution of the light field sub-aperture image, which is convenient for subsequent operations with the light field sub-aperture image. By assigning each pixel point in the mask matrix to a pixel of the light field sub-aperture image, the mask matrix is ​​expanded to a matrix of size U×V to obtain an extended mask matrix. Since the pixel value of the i-th depth in the extended mask matrix is ​​1 and the other pixel values ​​are 0, multiplying it with the input low-resolution multi-defocus light field image can retain the target pixel value at that depth and set other values ​​to 0, thereby obtaining several low-resolution local light field images at different depths.

[0047] In the embodiment of the present invention, the image block operation guided by the depth mask can extract all pixels at the same depth in the four-dimensional light field image, and set the pixel values ​​of other depths to zero, so as to obtain local light field images of different depths. At the same time, considering that the pixel depths under the same macro pixel are similar, in order to reduce the amount of calculation, this step can be carried out at the macro pixel level. Through the above-mentioned image block processing, the network model can focus more accurately on processing the target object at the same depth layer, laying the foundation for the subsequent super-resolution process.

[0048] Regarding step 102:

[0049] In some embodiments, the light field image dataset includes a high-resolution light field image and a light field standard disparity map; the high-resolution light field image standard form is a light field sub-aperture image array;

[0050] The super-resolution image reconstruction network model is trained using low-resolution multi-defocus light field images as an input training sample set and the high-resolution light field images as an output training sample set; wherein the low-resolution multi-defocus light field images are obtained by blurring and degrading the high-resolution light field images based on the light field standard disparity map.

[0051] In the embodiment of the present invention, in order to realize the supervised all-focus convolutional neural network training guided by depth information to obtain a super-resolution reconstructed image network model, it is necessary to use low-resolution multi-defocus light field images as input training sample data sets, and high-resolution all-focus images as high-resolution true values ​​(output training sample data sets), and it is also necessary to provide corresponding light field standard disparity maps as guidance. In this embodiment, the public light field depth data sets HCInew, HCIold, and DUTLF are selected as training sample sets and test sample sets.

[0052] However, the above-mentioned disclosed light field depth dataset only provides high-resolution images and their standard disparity maps. In order to obtain input-output training data pairs for training models, in one embodiment of the present invention, after obtaining the light field depth dataset, the following steps 1020-1026 are required to perform blur degradation processing on the high-resolution light field image based on the light field standard disparity map:

[0053] Step 1020, generating N depth mask matrices using the light field standard disparity map;

[0054] Step 1022, multiplying each high-resolution light field sub-aperture image by the corresponding depth mask matrix pixel by pixel to obtain N high-resolution light field sub-aperture images of different depths;

[0055] Step 1024, using N degenerate blur kernels to sequentially convolve the high-resolution light field sub-aperture images at each depth, downsample them by a factor of α, and add them to additive noise to obtain a plurality of degenerate light field sub-aperture images; wherein the degenerate blur kernels are simulated by using N two-dimensional Gaussian functions with different blurring degrees;

[0056] Step 1026, respectively add the degraded light field sub-aperture images at N depths pixel by pixel to obtain a low-resolution multi-defocus sub-aperture image, and then combine each low-resolution multi-defocus sub-aperture image into an image array to obtain a low-resolution multi-defocus light field image for training.

[0057] In a specific implementation, each degraded light field sub-aperture image is obtained using the following formula:

[0058]

[0059] In the formula, represents the high-resolution light field subaperture image at the ith depth, Indicates the resolution of the image. represents the degraded light field sub-aperture image at the i-th depth, α is the downsampling factor, and the resolution is reduced to α times of the true value image. It means that the standard deviation at the i-th depth is σ i Gaussian blur kernel, k is the size of the degenerate blur kernel, N i represents additive noise.

[0060] Furthermore, the low-resolution multi-defocus sub-aperture image SAI lr The following formula is used to obtain:

[0061]

[0062] The image array is composed of the corresponding low-resolution multi-defocus sub-aperture images obtained by adding all (U×V) degraded light field sub-aperture images And use this image array as the input low-resolution multi-defocus light field image in the training dataset.

[0063] like Figure 3 As shown, the super-resolution reconstruction network model in the embodiment of the present invention adopts a pyramid multi-scale feature extraction structure, including a three-layer structure, which is executed in sequence from bottom to top. Figure 3 It can be seen that the bottom layer (the third layer) and the middle layer (the second layer) are fuzzy feature extraction modules, which are used to process low-resolution images in the input pyramid and provide richer fuzzy information for super-resolution. The top layer (the first layer) is the light field super-resolution module, which is used to reconstruct low-resolution images through super-resolution.

[0064] In some embodiments, the fuzzy feature extraction module includes a first fuzzy feature extraction layer and a second fuzzy feature extraction layer arranged sequentially from top to bottom;

[0065] After the low-resolution local light field image obtained in step 100 is input into the super-resolution reconstruction network model, for each depth of the low-resolution local light field image, the following operations are performed:

[0066] Downsampling the low-resolution local light field image twice to obtain a first-level light field image and a second-level light field image respectively;

[0067] The second-level light field image is input into the second fuzzy feature extraction layer for fuzzy feature extraction to obtain a first reconstructed light field image; wherein the resolution of the first reconstructed light field image is the same as the resolution of the first-level light field image;

[0068] After connecting the first hierarchical light field image and the first reconstructed light field image, the first hierarchical light field image is input into the first fuzzy feature extraction layer for fuzzy feature extraction to obtain a second reconstructed light field image; wherein the resolution of the second reconstructed light field image is the same as the resolution of the low-resolution local light field image;

[0069] After the second reconstructed light field image is connected with the low-resolution local light field image, the image is input into the light field super-resolution module and super-resolved under the guidance of the generation degradation module to obtain the local light field super-resolution reconstructed image.

[0070] Continue to refer Figure 3 , the low-resolution local light field image (Level 0) input into the super-resolution reconstruction network model is first down-sampled twice to obtain the first-level light field image (Level 1) and the second-level light field image (Level 2), respectively. After that, the second blurred feature extraction layer of the blurred feature extraction module (Blurred Feature Extraction) starts from processing the second-level light field image with the lowest resolution, and after two 3×3 convolutional layers, a residual block and an upsampling module, generates a first reconstructed light field image consistent with the next level resolution; next, the first reconstructed light field image is connected with the second-level light field image in the first blurred feature extraction layer so as to be transmitted in the light field super-resolution module in the network model. Finally, the second reconstructed light field image output from the second blurred feature extraction layer is connected with the original input low-resolution local light field image, and input together into the light field super-resolution module (LF Super-resolution) for super-resolution.

[0071] In some embodiments, the light field super-resolution module includes four deep convolution submodules; wherein each deep convolution submodule includes a degraded integration layer, four multi-feature extraction layers and a convolution layer connected in sequence; the multi-feature extraction layer is used to extract spatial features, angle features and disparity information of the light field image; the degraded integration layer is used to integrate the generated degraded information into the network to enhance the network's super-resolution reconstruction capability for images at different depth layers;

[0072] Continue to refer Figure 3 The main structure of the light field super-resolution module in the embodiment of the present invention includes four deep convolution submodules (Deep Convolution, DeepConv), two 3×3 convolution layers and an upsampling submodule; wherein the upsampling submodule consists of Figure 3The last four layers in the network model. To avoid the problem of network performance degradation, each deep convolution submodule adopts a residual structure. Upsampling module: Located at the end of the light field super-resolution module, it reconstructs the image through multi-layer convolution and residual connection, from low resolution to high resolution. The residual structure is used to prevent the gradient vanishing and gradient exploding problems, ensuring the stability of the network and the training effect.

[0073] Further, such as Figure 4 As shown in Figure 1, each deep convolution submodule includes a degradation integration layer (Degeneration-Integration Block, DI-Block), four multiple feature extraction layers (Multiple Feature Extraction, MFE) and a 3×3 convolution layer, in which the degradation blur matrix generated by the generation degradation module is embedded in the degradation integration layer for feature extraction, as shown in Figure 1. Figure 4 Indicated by the red arrow in (a).

[0074] The following is an introduction to the structure of each layer in each deep convolution submodule:

[0075] (1) Degeneration-Integration Block (DI-Block):

[0076] In some embodiments, the degraded integration layer includes a degraded convolution block, a residual block and a degraded connection block; wherein the degraded convolution block and the residual block are connected in parallel to perform degraded convolution on the input image features respectively, and the degraded connection block is used to connect the output results of the degraded convolution of the degraded convolution block and the residual block and the degraded blur matrix generated by the generation degradation module to integrate the degraded features into the light field super-resolution module.

[0077] In the embodiment of the present invention, in order to utilize degradation information to improve the image super-resolution effect, the corresponding degradation fuzzy matrix is ​​first generated by using the generated degradation module, and then further embedded into the degradation integration layer (DI-Block), so as to integrate the given degradation information into the network model. The structure of the degradation integration layer is as follows: Figure 5 The degenerate integration layer adopts a two-branch structure (composed of Figure 5 As can be seen in the figure); one of the branches uses a degraded convolution block to perform a degraded convolution operation on the input image features, thereby integrating the degradation information into the network model; the other branch uses a residual block to perform a degraded convolution on the input image features, so that the integrated degradation features match the network features. Finally, after the output results of the two branches are added, the degradation features are further integrated into the network through the degraded connection layer. In this way, the degradation information can be effectively introduced into the network to improve the adaptability of the network to specific degradation conditions.

[0078] In some specific implementations, the degenerate fuzzy matrix is ​​generated in the following manner:

[0079] Select different Gaussian blur kernels according to low-resolution local light field images at different depths;

[0080] Expanding the dimension of each Gaussian blur kernel based on the angular resolution of the light field image to obtain a four-dimensional Gaussian blur kernel;

[0081] The four-dimensional Gaussian blur kernel is subjected to dimensionality reduction, convolution and linear mapping in sequence, and is superimposed with a noise matrix of the same dimension to obtain a degraded blur matrix.

[0082] In the embodiment of the present invention, the Gaussian blur kernel is selected in the following manner: in the construction part of the aforementioned low-resolution multi-defocus light field image, the defocus blur of the degraded light field sub-aperture image at the i-th depth is simulated by the convolution of the i-th Gaussian blur kernel. Therefore, in order to match the super-resolution network with the input low-resolution image, the Gaussian blur kernel K of the corresponding depth used in constructing the data set is selected. i The Gaussian blur kernel used as input in the generation degradation module is expanded to match the dimension of the light field image. The dimension of the expanded four-dimensional Gaussian blur kernel is U×V×k×k.

[0083] Afterwards, if Figure 6 As shown in the figure, the expanded four-dimensional Gaussian blur kernel is first reduced in dimensionality by stretching the two spatial dimensions to one dimension, thereby reducing it from a four-dimensional cube to a three-dimensional cube. The degraded cube is then input into multiple 1×1 convolutional layers and Leaky ReLU activation layers, and further superimposed with noise information of the same dimension to generate the final degraded blur matrix.

[0084] In the embodiment of the present invention, a known blur kernel is selected for the local low-resolution image of the light field that has been segmented, and the degradation kernel is generated by using a generative degradation module (GDM) and then embedded in the top-level light field super-resolution module in the network, so that the network model has both the ability of de-degradation and anti-noise.

[0085] By analyzing the properties of light field images, it is found that targets at different depths have images with different blur levels, that is, they have different degradation prior information. Therefore, in order to make full use of this degradation prior information, in the embodiment of the present invention, a known degradation blur kernel is selected for the local low-resolution image of the light field that has been divided into blocks, so that the degradation blur kernels with different blur levels can correctly simulate the corresponding four-dimensional degradation characteristics of the light field, and the degradation blur kernel is generated by the Generative Degradation Module (GDM) and then embedded in the top-level light field super-resolution module in the network, so that the network model has both the ability to degrade and resist noise, so as to more accurately remove multi-defocus blur.

[0086] (2) Multiple Feature Extraction (MFE):

[0087] In order to make full use of the special structure of the light field and extract various types of information from the light field data, the network model in this embodiment adopts multiple feature extraction layers. Figure 4 (b) Each multi-feature extraction layer uses a parallel structure to connect four different feature extractors, connect the different features extracted by them together, and process them through convolutional layers and activation layers to output the final result.

[0088] In some embodiments, Figure 7 As shown, the feature extractor includes a spatial feature extractor, an angle feature extractor and two disparity feature extractors in different directions, each feature extractor being used to extract spatial features, angle features and disparity features of the input low-resolution local light field image.

[0089] In the embodiment of the present invention, by designing the structure of each feature extractor and designing different convolution kernel types in each feature extractor, the feature extractor can extract the spatial, angular and parallax features of the input light field image according to different input feature structures.

[0090] Generally speaking, different parameter designs in the convolution layer, such as convolution kernel size, step size, dilation coefficient, etc., can extract different image features. Therefore, each feature extractor consists of a specially designed convolution layer, a regular convolution layer, and two corresponding Leaky ReLU activation layers. Special convolution layers designed for different input features are as follows: Figure 7 This design can extract more representative and discriminative features according to the characteristics of the input data, thereby providing richer information for subsequent super-resolution tasks.

[0091] The following describes the structure of each feature extractor:

[0092] like Figure 7 (a) is the specific structure of the spatial feature extractor (Spatial Feature Extractor, SFE), which is designed as a dilated convolution layer with a convolution kernel size of 3×3, a step size of 1, and a dilation coefficient of U×V. This feature extractor is used to extract the input macro-pixel light field features. The convolution operation is performed. This design ensures that the pixels involved in the convolution operation have different spatial coordinates but the same angular coordinates.

[0093] like Figure 7 (b) is the specific structure of the Angular Feature Extractor (AFE), which is designed to consist of a convolution layer (Angconv) with a convolution kernel size of U×V, a step size of 1, and a dilation coefficient of 1. The three-dimensional spatial feature is input to the feature extractor, which reduces the viewing angle feature to a one-dimensional In contrast to the spatial feature extractor, the angular feature extractor only operates on pixels with different angular coordinates but the same spatial coordinates during the convolution process. Therefore, it can focus on extracting the angular information in the light field image, thereby better capturing the directional characteristics of the scene.

[0094] like Figure 7 (c) shows the specific structure of two disparity feature extractors (EPI Feature Extractor, EFE), whose input is the three-dimensional feature matrix of the polar plane. The construction process of the feature matrix is ​​as follows Figure 8 As shown in the figure, first, the input light field feature image is stacked in the horizontal (U dimension) and vertical (V dimension) directions to form two four-dimensional stacks, namely EPI-Vs and EPI-Hs stacks. Then, for these two stacks, dimensionality reduction is performed in the horizontal and vertical directions to form four polar plane three-dimensional feature matrices. Since there are polar plane three-dimensional feature matrices in two directions, each direction includes two polar plane three-dimensional feature matrices, namely and Therefore, four independent disparity feature extractors are designed in the network. Each feature extractor has a different EPI convolution layer, and the convolution kernel size is 1×V, 1×U, U×1, and V×1. The above design can make full use of the perspective information contained in the light field image, thereby achieving a better super-resolution effect.

[0095] For example, the training method of the super-resolution image reconstruction network model can be described as follows:

[0096] The angular resolution is 5×5 in the center. In order to save GPU memory, the sub-aperture images in the true value are cropped into slices of 64×64 size, and the slice step size is 16×16, that is, in the training samples, the dimensions of the high-resolution light field image are 5×5×3×64×64, where 3 represents the number of channels, which means that the training data is an RGB three-channel image. The multi-defocus light field image training data is also an RGB light field image with an angular resolution of 5×5 and 3 channels. If we take double super-resolution as an example, the spatial resolution of the multi-defocus light field image is 32×32 pixels. In addition, each pair of data is randomly enhanced by flipping, transposing, and shuffling the color channels to enhance robustness.

[0097] During the training process, the loss function can be taken as the L1 mean absolute error:

[0098]

[0099] In the formula, P is the number of training data, is the jth pixel value in the super-resolution image, is the jth pixel value in the high-resolution light field image, I sr Reconstruct the image for full focus super resolution, I hr High-resolution light field image.

[0100] Regarding step 104:

[0101] In the embodiment of the present invention, based on the local light field super-resolution reconstructed image of each depth obtained in the above step 102, the local light field super-resolution reconstructed images of all depth layers are further synthesized through step 106, thereby realizing the generation of a complete all-focus super-resolution image.

[0102] The following is a detailed description of the method performance combined with quantitative and qualitative results:

[0103] Table 1 shows the results of image quality evaluation indicators using three test light field datasets. The average results of PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index) of the three datasets are shown in columns 3-5 and 6-8, respectively.

[0104] Under the same magnification super-resolution case, the first and second rows show the double super-resolution results of 2-depth scenes and 3-depth scenes respectively, and compare the light field image super-resolution results of the same magnification but different depths. It can be seen from Table 1 that both effects are ideal. The average PSNR of the 2-depth scene is 4.98% higher than that of the 3-depth scene, and the average SSIM is 4.15% higher. The first and third rows select 2-depth scenes to compare the light field image super-resolution results of different magnifications. The gap between the double and quadruple reconstruction results is within the ideal range, proving that the method has the ability to handle multi-depth scene targets of different magnifications.

[0105] Table 1 Quantitative results of different depths and magnifications of the method

[0106]

[0107] Furthermore, Fig. 9 The upper middle part shows the super-resolution reconstruction results of the high-resolution reference image, 2-depth, and 3-depth scenes from left to right. The lower part shows the image reconstruction details at different depths. The quantitative evaluation results of the image are marked under each image, with PSNR on the left and SSIM on the right. In the images of the 2-depth scene, the images marked with red and blue boxes represent the image reconstruction at the same depth layer. For the 3-depth scene, the three different colored boxes each represent three different depth layers. Fig. 9 It can be seen that by observing the super-resolution results at each depth layer, it is found that their results are similar to each other, which verifies the ability of DPM to process targets at multiple depth levels and further proves the excellent performance of this method in solving the full-focus light field super-resolution problem in multi-target scenes.

[0108] like Fig.10 , Fig.11 As shown, an embodiment of the present invention provides a light field image super-resolution reconstruction device for multi-defocus scenes. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From the hardware level, Fig. 9 FIG. 1 is a hardware architecture diagram of a computing device where a light field image super-resolution reconstruction device for multiple defocus scenes provided by an embodiment of the present invention is located, except Fig.10 In addition to the processor, memory, network interface, and non-volatile memory shown in the figure, the computing device in which the device is located in the embodiment may also generally include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, Fig.11 As shown, as a device in a logical sense, the CPU of the computing device in which it is located reads the corresponding computer program in the non-volatile memory into the memory and runs it. This embodiment provides a light field image super-resolution reconstruction device for multi-defocus scenes, the device comprising:

[0109] An image blocking unit 901 is used to block the obtained low-resolution multi-defocus light field image using an image blocking algorithm with a pre-generated depth mask as a guide to obtain a plurality of low-resolution local light field images at different depths;

[0110] The image super-resolution reconstruction unit 902 is used to input each low-resolution local light field image of different depths into a pre-trained super-resolution reconstruction image network model to obtain a plurality of local light field super-resolution reconstruction images of different depths;

[0111] The super-resolution image reconstruction network model is trained with a light field image data set as a training sample set. The super-resolution reconstruction network model includes a fuzzy feature extraction module, a light field super-resolution module and a generation degradation module. The fuzzy feature extraction module is used to provide multi-scale fuzzy information for the light field super-resolution module. The light field super-resolution module is used to extract light field features and perform super-resolution on them. The generation degradation module embeds degradation features into the light field super-resolution module to guide the degradation of the light field super-resolution module.

[0112] The combining unit 903 is used to add the local light field super-resolution reconstructed images at each depth to obtain the full-focus super-resolution reconstructed image.

[0113] In the embodiment of the present invention, the image segmentation unit 901 may be used to execute step 100 in the above method embodiment, the image super-resolution reconstruction unit 902 may be used to execute step 102 in the above method embodiment, and the combination unit 903 may be used to execute step 104 in the above method embodiment.

[0114] In one embodiment of the present invention, in the image blocking unit 901, the image blocking algorithm is used to block the obtained low-resolution multi-defocus light field image with the pre-generated depth mask as a guide, including:

[0115] Based on the light field standard disparity map in the light field image dataset, N depth mask matrices are generated; wherein the pixel value of the i-th depth in each mask matrix is ​​1, and the pixel values ​​of the remaining depths are 0;

[0116] Expanding the pixels of each depth mask matrix according to the resolution of the low-resolution multi-defocus light field image to obtain an expanded mask matrix with N dimensions matching the low-resolution multi-defocus light field image;

[0117] Each of the extended mask matrices is multiplied by the low-resolution multi-defocus light field image to obtain a plurality of low-resolution local light field images at different depths.

[0118] In one embodiment of the present invention, in the image super-resolution reconstruction unit 902, the light field image data set includes a high-resolution light field image and a light field standard disparity map; the high-resolution light field image standard form is a light field sub-aperture image array;

[0119] The super-resolution image reconstruction network model is trained using low-resolution multi-defocus light field images as an input training sample set and the high-resolution light field images as an output training sample set; wherein the low-resolution multi-defocus light field images are obtained by blurring and degrading the high-resolution light field images based on the light field standard disparity map.

[0120] In one embodiment of the present invention, in the image super-resolution reconstruction unit 902, the blurring and degradation processing of the high-resolution light field image based on the light field standard disparity map includes:

[0121] Generate N depth mask matrices using the light field standard disparity map;

[0122] Multiply each high-resolution light field sub-aperture image by the corresponding depth mask matrix pixel by pixel to obtain N high-resolution light field sub-aperture images of different depths;

[0123] The high-resolution light field sub-aperture images at N depths are convolved in sequence using N degenerate blur kernels, downsampled by α times and then added with additive noise to obtain a number of degraded light field sub-aperture images; wherein the degenerate blur kernel is simulated by N two-dimensional Gaussian functions with different blurring degrees;

[0124] The degraded light field sub-aperture images at N depths are added pixel by pixel to obtain a low-resolution multi-defocus sub-aperture image, and then each low-resolution multi-defocus sub-aperture image is combined into an image array to obtain a low-resolution multi-defocus light field image for training.

[0125] In one embodiment of the present invention, in the image super-resolution reconstruction unit 902, the fuzzy feature extraction module includes a first fuzzy feature extraction layer and a second fuzzy feature extraction layer arranged sequentially from top to bottom;

[0126] For each depth of low-resolution local light field image, perform:

[0127] Downsampling the low-resolution local light field image twice to obtain a first-level light field image and a second-level light field image respectively;

[0128] The second-level light field image is input into the second fuzzy feature extraction layer for fuzzy feature extraction to obtain a first reconstructed light field image; wherein the resolution of the first reconstructed light field image is the same as the resolution of the first-level light field image;

[0129] After connecting the first hierarchical light field image and the first reconstructed light field image, the first hierarchical light field image is input into the first fuzzy feature extraction layer for fuzzy feature extraction to obtain a second reconstructed light field image; wherein the resolution of the second reconstructed light field image is the same as the resolution of the low-resolution local light field image;

[0130] After the second reconstructed light field image is connected with the low-resolution local light field image, the image is input into the light field super-resolution module for super-resolution under the guidance of the degenerate fusion layer to obtain the local light field super-resolution reconstructed image.

[0131] In one embodiment of the present invention, the light field super-resolution module includes four deep convolution sub-modules; wherein each deep convolution sub-module includes a degenerate fusion layer, four multi-feature extraction layers and a convolution layer connected in sequence.

[0132] In one embodiment of the present invention, each of the multi-feature extraction submodules includes four feature extractors connected in parallel; wherein the feature extractors include a spatial feature extractor, an angular feature extractor and two disparity feature extractors in different directions, and each feature extractor is used to extract the spatial features, angular features and disparity features of the input low-resolution local light field image.

[0133] In one embodiment of the present invention, the degenerate integration layer includes a degenerate convolution block, a residual block and a degenerate connection block; wherein the degenerate convolution block and the residual block are connected in parallel to perform degenerate convolution on the input image features respectively, and the degenerate connection block is used to connect the output results of the degenerate convolution of the degenerate convolution block and the residual block and the degenerate blur matrix generated by the generation degradation module to integrate the degenerate features into the light field super-resolution module;

[0134] In one embodiment of the present invention, the degenerate fuzzy matrix is ​​generated in the following manner:

[0135] Select different Gaussian blur kernels according to low-resolution local light field images at different depths;

[0136] Expanding the dimension of each Gaussian blur kernel based on the angular resolution of the light field image to obtain a four-dimensional Gaussian blur kernel;

[0137] The four-dimensional Gaussian blur kernel is subjected to dimensionality reduction, convolution and linear mapping in sequence, and is superimposed with a noise matrix of the same dimension to obtain a degraded blur matrix.

[0138] It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a light field image super-resolution reconstruction device for multiple defocus scenes. In other embodiments of the present invention, a light field image super-resolution reconstruction device for multiple defocus scenes may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0139] The information interaction, execution process and other contents between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0140] An embodiment of the present invention further provides a computing device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a light field image super-resolution reconstruction method for multi-defocus scenes in any embodiment of the present invention is implemented.

[0141] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes a light field image super-resolution reconstruction method for multiple defocus scenes in any embodiment of the present invention.

[0142] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0143] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0144] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0145] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0146] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0147] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement a light field image super-resolution reconstruction method for multi-defocus scenes provided by the above-mentioned method embodiments.

[0148] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0149] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A light field image super-resolution reconstruction method for multi-defocus scenes, characterized in that: include: Using the pre-generated depth mask as a guide, the obtained low-resolution multi-defocus light field image is segmented using an image segmentation algorithm to obtain a number of low-resolution local light field images at different depths; Input each low-resolution local light field image of different depths into a pre-trained super-resolution reconstructed image network model to obtain a corresponding number of local light field super-resolution reconstructed images of different depths; The super-resolution image reconstruction network model is trained with a light field image data set as a training sample set. The super-resolution reconstruction network model includes a fuzzy feature extraction module, a light field super-resolution module and a generation degradation module. The fuzzy feature extraction module is used to provide multi-scale fuzzy information for the light field super-resolution module. The light field super-resolution module is used to extract light field features and perform super-resolution on them. The generation degradation module embeds degradation features into the light field super-resolution module to guide the degradation of the light field super-resolution module. The local light field super-resolution reconstructed images at each depth are added together to obtain the full-focus light field super-resolution reconstructed image.

2. The method according to claim 1, characterized in that The light field image data set includes a high-resolution light field image and a light field standard disparity map; the standard form of the high-resolution light field image is a light field sub-aperture image array; The super-resolution image reconstruction network model is trained using low-resolution multi-defocus light field images as an input training sample set and the high-resolution light field images as an output training sample set; wherein the low-resolution multi-defocus light field images are obtained by blurring and degrading the high-resolution light field images based on the light field standard disparity map.

3. The method according to claim 2, characterized in that The performing blur degradation processing on the high-resolution light field image based on the light field standard disparity map comprises: Generate N depth mask matrices using the light field standard disparity map; Multiply each high-resolution light field sub-aperture image by the corresponding depth mask matrix pixel by pixel to obtain N high-resolution light field sub-aperture images of different depths; Using N degenerate blur kernels, convolving the high-resolution light field sub-aperture images at N depths in sequence, downsampling by a factor of α and adding additive noise, to obtain a number of degenerate light field sub-aperture images; wherein the degenerate blur kernels are simulated by N two-dimensional Gaussian functions with different blurring degrees; The degraded light field sub-aperture images at N depths are added pixel by pixel to obtain a low-resolution multi-defocus sub-aperture image, and then each low-resolution multi-defocus sub-aperture image is combined into an image array to obtain a low-resolution multi-defocus light field image for training.

4. The method according to claim 2, characterized in that: The method uses the pre-generated depth mask as a guide and uses an image segmentation algorithm to segment the obtained low-resolution multi-defocus light field image, including: Based on the light field standard disparity map in the light field image dataset, N depth mask matrices are generated; wherein the pixel value of the i-th depth in each mask matrix is ​​1, and the pixel values ​​of the remaining depths are 0; Expanding the pixels of each depth mask matrix according to the resolution of the low-resolution multi-defocus light field image to obtain an expanded mask matrix with N dimensions matching the low-resolution multi-defocus light field image; Each of the extended mask matrices is multiplied by the low-resolution multi-defocus light field image to obtain a plurality of low-resolution local light field images at different depths.

5. The method according to claim 1, characterized in that The fuzzy feature extraction module includes a first fuzzy feature extraction layer and a second fuzzy feature extraction layer arranged sequentially from top to bottom; For each depth of low-resolution local light field image, perform: Downsampling the low-resolution local light field image twice to obtain a first-level light field image and a second-level light field image respectively; The second-level light field image is input into the second fuzzy feature extraction layer for fuzzy feature extraction to obtain a first reconstructed light field image; wherein the resolution of the first reconstructed light field image is the same as the resolution of the first-level light field image; After connecting the first hierarchical light field image and the first reconstructed light field image, the first hierarchical light field image is input into the first fuzzy feature extraction layer for fuzzy feature extraction to obtain a second reconstructed light field image; wherein the resolution of the second reconstructed light field image is the same as the resolution of the low-resolution local light field image; After the second reconstructed light field image is connected with the low-resolution local light field image, the image is input into the light field super-resolution module and super-resolved under the guidance of the generation degradation module to obtain the local light field super-resolution reconstructed image.

6. The method according to claim 1, characterized in that The light field super-resolution module includes four deep convolution submodules; wherein each deep convolution submodule includes a degenerate fusion layer, four multi-feature extraction layers and a convolution layer connected in sequence; and / or Each of the multi-feature extraction submodules includes four feature extractors connected in parallel; wherein the feature extractors include a spatial feature extractor, an angular feature extractor and two disparity feature extractors in different directions, and each feature extractor is used to extract the spatial features, angular features and disparity features of the input low-resolution local light field image.

7. The method according to any one of claims 1 to 6, characterized in that The degenerate integration layer includes a degenerate convolution block, a residual block and a degenerate connection block; wherein the degenerate convolution block and the residual block are connected in parallel to perform degenerate convolution on the input image features respectively, and the degenerate connection block is used to connect the output results of the degenerate convolution of the degenerate convolution block and the residual block and the degenerate blur matrix generated by the generation degradation module, so as to integrate the degenerate features into the light field super-resolution module; and / or The degenerate fuzzy matrix is ​​generated in the following way: Select different Gaussian blur kernels according to low-resolution local light field images at different depths; Expand the dimension of each Gaussian blur kernel based on the angular resolution of the light field image to obtain a four-dimensional Gaussian blur kernel; The four-dimensional Gaussian blur kernel is subjected to dimensionality reduction, convolution and linear mapping in sequence, and is superimposed with a noise matrix of the same dimension to obtain a degraded blur matrix.

8. A light field image super-resolution reconstruction device for multi-defocus scenes, characterized in that: include: An image blocking unit is used to block the obtained low-resolution multi-defocus light field image using an image blocking algorithm with a pre-generated depth mask as a guide, so as to obtain a plurality of low-resolution local light field images at different depths; An image super-resolution reconstruction unit is used to input each low-resolution local light field image of different depths into a pre-trained super-resolution reconstruction image network model to obtain a corresponding number of local light field super-resolution reconstruction images of different depths; The super-resolution image reconstruction network model is trained with a light field image data set as a training sample set. The super-resolution reconstruction network model includes a fuzzy feature extraction module, a light field super-resolution module and a generation degradation module. The fuzzy feature extraction module is used to provide multi-scale fuzzy information for the light field super-resolution module. The light field super-resolution module is used to extract light field features and perform super-resolution on them. The generation degradation module embeds degradation features into the light field super-resolution module to guide the degradation of the light field super-resolution module. The combining unit is used to add the local light field super-resolution reconstructed images at each depth to obtain the full-focus super-resolution reconstructed image.

9. A computing device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images