Light field image spatial super-resolution reconstruction method and device, equipment and storage medium

Abstract

The invention discloses a light field image spatial super-resolution reconstruction method, device and equipment and a storage medium, and relates to the technical field of image super-resolution reconstruction. According to the specific implementation scheme, the method comprises the steps of obtaining a sample set, constructing a reconstruction model, performing model training on the reconstruction model through the sample set until a target loss function converges, and obtaining a target reconstruction model; and performing spatial super-resolution reconstruction on the input light field image through the target reconstruction model to obtain a spatial high-resolution reconstruction image corresponding to the input light field image. According to the invention, the problem of poor precision of spatial super-resolution reconstruction of the light field image in the prior art can be solved, and the precision of spatial super-resolution reconstruction of the light field image is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image super-resolution reconstruction, and in particular to a method, apparatus, device and storage medium for spatial super-resolution reconstruction of light field images. Background Art

[0002] Compared to traditional cameras, light field cameras record not only the spatial information of light but also its angular information, providing more complete scene data and depth extraction capabilities, improving image quality and data utilization. However, light field camera sensors have limited resolution, and during the capture process, light field cameras must strike a balance between spatial and angular resolution. Attempting to capture more angular information (i.e., increasing angular resolution) results in insufficient spatial resolution in the resulting light field image.

[0003] At present, deep learning methods are generally used to perform super-resolution reconstruction on light field images to obtain spatially high-resolution reconstructed images.

[0004] However, existing deep learning methods cannot fully extract the deep spatial features of light field images, resulting in poor accuracy of spatial super-resolution reconstruction of light field images. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for spatial super-resolution reconstruction of light field images, thereby solving the problem of poor accuracy in spatial super-resolution reconstruction of light field images in the prior art and improving the accuracy of spatial super-resolution reconstruction of light field images.

[0006] In a first aspect, an embodiment of the present application provides a method for spatial super-resolution reconstruction of a light field image, comprising: A sample set is obtained, which includes multiple samples, each sample includes a corresponding light field image and a spatial high-resolution label image; a reconstruction model is constructed, and the reconstruction model is trained through the sample set until the target loss function converges to obtain a target reconstruction model; the input light field image is spatially super-resolution reconstructed through the target reconstruction model to obtain a spatial high-resolution reconstructed image corresponding to the input light field image; wherein the reconstruction model includes a first processing module, a feature extraction module and an upsampling module; the light field image is input into the first processing module for reshaping processing to obtain a first sub-aperture image and a first macro-pixel image corresponding to the input light field image; the first sub-aperture image and the first macro-pixel image are convolved and dimensionally increased to obtain a first feature and a second feature; the first feature and the second feature are input into the feature extraction module for feature extraction and fusion of the subspace to obtain a third feature, so as to learn different feature representations in different subspaces and improve model performance; the third feature is input into the upsampling module for upsampling, and the upsampled image is added to the image obtained by bilinear upsampling of the first sub-aperture image to obtain a spatial high-resolution reconstructed image corresponding to the input light field image.

[0007] Furthermore, the feature extraction module includes multiple extraction and fusion modules connected in sequence; among the multiple extraction and fusion modules, the input of the first extraction and fusion module is the first feature and the second feature, and the output of the last extraction and fusion module is the third feature; among every two adjacent extraction and fusion modules, the input of the latter extraction and fusion module includes the fourth feature output by the previous extraction and fusion module, and the fifth feature obtained by sub-aperture transformation of the fourth feature output by the previous extraction and fusion module.

[0008] The extraction and fusion module includes an extraction unit and a fusion unit; the extraction unit is used to extract the first feature or the fifth feature of the input through a spatial convolution filter, and perform macro-pixel transformation processing on the obtained feature to obtain the sixth feature; it is also used to extract the second feature or the fourth feature of the input through an angular convolution filter to obtain the seventh feature; it is also used to extract the second feature or the fourth feature of the input through a horizontal EPI (Epipolar Plane Image) convolution filter, and perform convolution and one-dimensional pixel scrambling on the obtained feature to obtain the eighth feature; it is also used to extract the second feature or the fourth feature of the input through a vertical EPI convolution filter, and perform convolution and one-dimensional pixel scrambling on the obtained feature to obtain the ninth feature; the fusion unit is used to fuse the sixth feature, the seventh feature, the eighth feature, the ninth feature, and the second feature or the fourth feature of the input extraction unit to obtain the output feature of the fusion unit.

[0009] Furthermore, the fusion unit is used to splice the sixth feature, the seventh feature, the eighth feature, and the ninth feature obtained by the extraction unit and reshape them to obtain a first feature matrix; determine the covariance matrix according to the first feature matrix through the covariance feature module; perform feature normalization on the covariance matrix through the normalization module to obtain a normalized covariance matrix; determine the adaptive weight matrix according to the normalized covariance matrix through the self-attention module, weight the first feature matrix using the adaptive weight matrix to obtain a weighted feature matrix, input the weighted feature matrix into the gated linear unit to obtain a gating signal, update the first feature matrix according to the gating signal to obtain a second feature matrix; reshape the second feature matrix and then convolve it to obtain a tenth feature; add the tenth feature to the second feature or the fourth feature of the input extraction unit to obtain the output feature of the fusion unit.

[0010] Furthermore, the number of extracted fusion modules is 16.

[0011] Furthermore, the upsampling module is used to perform sub-aperture transformation on the third feature, and then perform convolution, two-dimensional pixel shuffling, and convolution in sequence to obtain an output image of the upsampling module.

[0012] Furthermore, the objective loss function is as follows:

[0013]

[0014]

[0015]

[0016] In the above formula, represents the target loss function, represents the weight coefficient, represents the first loss function, represents the second loss function, represents the height of the light field image, represents the width of the light field image, Represents the spatial high-resolution label pixels in the sample image The corresponding pixel value, Represents pixels in a spatially high-resolution reconstructed image The corresponding pixel value, Represents the binary edge detection map, Represents pixels The corresponding gradient amplitude is represents the first threshold, represents the second threshold; In pixels When there is a pixel with a gradient amplitude greater than the first threshold among the adjacent pixels of When there is no pixel with a gradient amplitude greater than the first threshold among the surrounding adjacent pixels, the value is 0.

[0017] Furthermore, after obtaining the sample set, the method further includes: performing data preprocessing and data enhancement on the samples in the sample set.

[0018] In a second aspect, an embodiment of the present application provides a light field image spatial super-resolution reconstruction device, comprising: The acquisition module is used to acquire a sample set, where the sample set includes multiple samples, and each sample includes a light field image and a spatial high-resolution label image that correspond to each other.

[0019] The module construction and training module is used to construct a reconstruction model, train the reconstruction model through a sample set until the target loss function converges, and obtain the target reconstruction model.

[0020] The image reconstruction module is used to perform spatial super-resolution reconstruction on the input light field image through the target reconstruction model to obtain a spatial high-resolution reconstructed image corresponding to the input light field image.

[0021] Among them, the reconstruction model includes a first processing module, a feature extraction module and an upsampling module; the light field image is input into the first processing module for reshaping processing to obtain a first sub-aperture image and a first macro-pixel image corresponding to the input light field image; the first sub-aperture image and the first macro-pixel image are convolved and dimensionally upgraded respectively to obtain a first feature and a second feature; the first feature and the second feature are input into the feature extraction module for feature extraction and fusion of the subspace to obtain a third feature, so as to learn different feature representations in different subspaces and improve model performance; the third feature is input into the upsampling module for upsampling, and the upsampled image is added to the image obtained by bilinear upsampling of the first sub-aperture image to obtain a spatial high-resolution reconstructed image corresponding to the input light field image.

[0022] In a third aspect, an embodiment of the present application provides a device comprising: a processor; a memory for storing processor-executable instructions; and a method for implementing the first aspect or any possible implementation of the first aspect when the processor executes the executable instructions.

[0023] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, which includes a device for storing a computer program or instruction, and when the computer program or instruction is executed, the method of the first aspect or any possible implementation method of the first aspect is implemented.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: In an embodiment of the present application, a sample set is acquired to construct a reconstruction model that learns different feature representations in different subspaces to improve model performance. The reconstruction model is trained to obtain a target reconstruction model. The target reconstruction model is used to perform spatial super-resolution reconstruction on an input light field image to obtain a spatial high-resolution reconstructed image. This solves the problem of poor accuracy in spatial super-resolution reconstruction of light field images in the prior art and can improve the accuracy of spatial super-resolution reconstruction of light field images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A schematic diagram of the flow of a light field image spatial super-resolution reconstruction method provided in an embodiment of the present application; Figure 2 A schematic diagram of a reconstruction model provided in an embodiment of the present application; Figure 3 A schematic diagram of the extraction and fusion module provided in an embodiment of the present application; Figure 4 A schematic diagram of an upsampling module provided in an embodiment of the present application; Figure 5 A schematic diagram of the composition of a light field image spatial super-resolution reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0028] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0029] Traditional cameras can only capture limited light position information. When processing high-speed moving objects, multi-subject target scenes, and images with complex depth, problems such as defocus, out-of-focus, and loss of details are prone to occur. In low-light environments, in order to reduce imaging errors, it is usually necessary to narrow the aperture, resulting in underexposure.

[0030] Compared to traditional cameras, light field cameras record not only the spatial information of light but also its angular information, providing more complete scene data and depth extraction capabilities, improving image quality and data utilization. However, light field camera sensors have limited resolution, and during the capture process, light field cameras must strike a balance between spatial and angular resolution. Attempting to capture more angular information (i.e., increasing angular resolution) results in insufficient spatial resolution in the resulting light field image.

[0031] At present, deep learning methods are generally used to perform super-resolution reconstruction on light field images to obtain spatially high-resolution reconstructed images.

[0032] However, existing deep learning methods cannot fully extract the deep spatial features of light field images, resulting in poor accuracy of spatial super-resolution reconstruction of light field images.

[0033] Against this background, the present disclosure provides a method for spatial super-resolution reconstruction of light field images, which can improve the accuracy of spatial super-resolution reconstruction of light field images.

[0034] The method for spatial super-resolution reconstruction of light field images provided in the embodiments of the present disclosure may be performed by a computer or server, or other electronic devices with data processing capabilities; alternatively, the method may be performed by a processor (e.g., a central processing unit (CPU)) in the aforementioned electronic devices; alternatively, the method may be performed by an application (APP) installed in the aforementioned electronic devices that is capable of implementing the functions of the method; alternatively, the method may be performed by a functional module or unit in the aforementioned electronic devices that performs the functions of the method. The method is not limited to the method's performer herein.

[0035] The light field image spatial super-resolution reconstruction method is exemplarily described below with reference to the accompanying drawings.

[0036] Figure 1 : is a flow chart of the light field image spatial super-resolution reconstruction method provided by the embodiment of the present application. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of the light field image spatial super-resolution reconstruction method. If the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse. Figure 1 As shown, the method may include: S101: Obtain a sample set.

[0037] The sample set includes multiple samples, and each sample includes a corresponding light field image and a spatial high-resolution label image.

[0038] For example, a publicly available light field image dataset (such as STFgantry) may be obtained to construct a sample set, or a sample set may be constructed independently, without limitation.

[0039] In some possible implementations, after obtaining the sample set, the method further includes: performing data preprocessing and data enhancement on the samples in the sample set.

[0040] For example, data preprocessing includes image cropping (preserving a fixed center region of the image), image color space conversion (converting the image from RGB to YCbCr), and low-resolution image generation (performing bicubic downsampling of the Y-pass of the image). Data augmentation can include at least one of image rotation, horizontal flipping, and vertical flipping. This can accelerate model training, enhance model robustness, and prevent overfitting.

[0041] S102: Construct a reconstruction model, and train the reconstruction model using the sample set until the target loss function converges to obtain a target reconstruction model.

[0042] Among them, the reconstruction model includes a first processing module, a feature extraction module and an upsampling module; the light field image is input into the first processing module for reshaping processing to obtain a first sub-aperture image and a first macro-pixel image corresponding to the input light field image; then the first sub-aperture image and the first macro-pixel image are convolved and dimensionally upgraded to obtain a first feature and a second feature; the first feature and the second feature are input into the feature extraction module for feature extraction and fusion of the subspace to obtain a third feature, so as to learn different feature representations in different subspaces and improve model performance; the third feature is input into the upsampling module for upsampling, and the upsampled image is added to the image obtained by bilinear upsampling of the first sub-aperture image to obtain a spatial high-resolution reconstructed image corresponding to the input light field image.

[0043] Specifically, the feature extraction module includes multiple extraction and fusion modules connected in sequence; among the multiple extraction and fusion modules, the input of the first extraction and fusion module is the first feature and the second feature, and the output of the last extraction and fusion module is the third feature.

[0044] In every two adjacent extraction and fusion modules, the input of the latter extraction and fusion module includes the fourth feature output by the previous extraction and fusion module and the fifth feature obtained by performing sub-aperture transformation on the fourth feature output by the previous extraction and fusion module.

[0045] It can be understood that after the first feature and the second feature are input into the first extraction and fusion module, the first extraction and fusion module outputs a fourth feature. The feature extraction module then performs sub-aperture transformation on the fourth feature to obtain a fifth feature. These two features are then input into the second extraction and fusion module. The second extraction and fusion module outputs another fourth feature. The feature extraction module then performs sub-aperture transformation on the fourth feature to obtain another fifth feature. These two features are then input into the third extraction and fusion module. The input and output of the subsequent extraction and fusion modules are similar to those of the second extraction and fusion module described above and will not be repeated here.

[0046] It can be understood that the feature output by the last extraction fusion module is also the output feature of the feature extraction module (i.e., the third feature).

[0047] Preferably, the number of extraction fusion modules is 16.

[0048] The extraction and fusion module includes an extraction unit and a fusion unit; the extraction unit is used to perform feature extraction on the first feature or the fifth feature of the input through a spatial convolution filter, and perform macro-pixel transformation processing on the obtained feature to obtain the sixth feature; it is also used to perform feature extraction on the second feature or the fourth feature of the input through the spatial convolution filter to obtain the seventh feature; it is also used to perform feature extraction on the second feature or the fourth feature of the input through a horizontal EPI convolution filter, and perform convolution and one-dimensional pixel scrambling on the obtained feature to obtain the eighth feature; it is also used to perform feature extraction on the second feature or the fourth feature of the input through a vertical EPI convolution filter, and perform convolution and one-dimensional pixel scrambling on the obtained feature to obtain the ninth feature.

[0049] The Spatial Convolutional Filter (SCF) is used to efficiently extract spatial feature information from sub-aperture images of a light field. In the SCF, the input first sub-aperture image is first split into multiple separate sub-aperture images, and then a convolution operation is performed on each sub-aperture image to capture the spatial feature information from the same perspective.

[0050] The Angular Convolutional Filter (ACF) is used to efficiently extract angular feature information from macropixel images of light fields. In the ACF, the input macropixel image is first split into multiple separate macropixel images based on angular resolution. Each macropixel image represents the light field data at a specific spatial coordinate and contains pixel information from different viewpoints at that spatial coordinate. A convolution operation is then performed on each macropixel image to capture the angular feature information at the same spatial coordinate.

[0051] In this way, the sixth feature representing the spatial subspace, the seventh feature representing the angle subspace, the eighth feature representing the horizontal EPI subspace, and the ninth feature representing the vertical EPI subspace can be obtained through the extraction unit.

[0052] The fusion unit is used to fuse the sixth feature, the seventh feature, the eighth feature, the ninth feature, and the second feature or the fourth feature of the input extraction unit to obtain an output feature of the fusion unit.

[0053] For example, the fusion module may perform feature concatenation on the sixth feature, the seventh feature, the eighth feature, the ninth feature, and the second feature or the fourth feature of the input extraction unit to obtain the output feature of the fusion unit.

[0054] In this way, the feature extraction module can obtain a third feature that can contain feature information of different subspaces, so that the model can learn different feature representations in different subspaces and improve performance.

[0055] Furthermore, the fusion unit is used to splice the sixth feature, the seventh feature, the eighth feature, and the ninth feature obtained by the extraction unit and reshape them to obtain a first feature matrix; determine the covariance matrix according to the first feature matrix through the covariance feature module; perform feature normalization on the covariance matrix through the normalization module to obtain a normalized covariance matrix; determine the adaptive weight matrix according to the normalized covariance matrix through the self-attention module, weight the first feature matrix using the adaptive weight matrix to obtain a weighted feature matrix, input the weighted feature matrix into the gated linear unit to obtain a gating signal, update the first feature matrix according to the gating signal to obtain a second feature matrix; reshape the second feature matrix and then convolve it to obtain a tenth feature; add the tenth feature to the second feature or the fourth feature of the input extraction unit to obtain the output feature of the fusion unit.

[0056] It can be understood that the sixth feature characterizes the feature of the spatial subspace, the seventh feature characterizes the feature of the angle subspace, the eighth feature characterizes the feature of the horizontal EPI subspace, and the ninth feature characterizes the feature of the vertical EPI subspace.

[0057] The fusion unit can first analyze the sixth feature , the seventh characteristic , the eighth characteristic 、Ninth characteristic Splice and get the feature tensor ,in Represents a splicing operation, and represent the height and width of the light field image respectively, The number of channels representing the features. It should be noted that Right now , Right now , represents the angular resolution, Indicates the unit height, Represents the unit width. Then the eigenvector Reshape into a two-dimensional matrix (i.e. the first characteristic matrix) ,in , represents the total dimension of each feature. This reshaping process flattens the original feature tensor into a 2D matrix, which is convenient for further statistical calculations.

[0058] The covariance feature module is based on the first feature matrix , the covariance matrix is ​​calculated using the following formula :

[0059] In the above formula, , represents the centered feature matrix, which represents the difference between each feature and its mean, Represents the first characteristic matrix The mean of .

[0060] The covariance matrix obtained above can capture the correlation and dependency between different features. As a second-order statistic, it can effectively integrate the interaction information between features and improve the reconstruction quality of image details and textures.

[0061] In order to maintain numerical stability, the L1 normalization method is used to normalize each row of the covariance matrix through the normalization module to obtain the normalized covariance matrix .

[0062] Through the self-attention module, according to the normalized covariance matrix Determine the adaptive weight matrix ,in Represents the Sigmoid activation function, which is used to normalize the weights to the [0,1] interval to ensure numerical stability. is a learnable scaling parameter that controls the normalized covariance matrix influence, Is a numerical constant with a small value, used to avoid the problem of zero division in numerical calculations. Normalized covariance matrix It plays a key role in this process. It not only models the interdependence between different features, but also guides the attention mechanism to reasonably allocate feature weights and achieve precise adjustment of the contribution of different subspaces.

[0063] Using adaptive weight matrix For the first characteristic matrix Perform weighting to obtain the weighted feature matrix .

[0064] The weighted feature matrix Input the gated linear unit to obtain the corresponding gating signal.

[0065] To further enhance feature selection capabilities, the Gated Linear Unit (GLU) is introduced. The GLU dynamically adjusts the contribution of feature channels through a set of gating signals, highlighting key information while suppressing irrelevant features, thereby improving the expressiveness of information. The calculation formula corresponding to the gating signal is as follows:

[0066] In the above formula, The input of the gated linear unit is The corresponding gating signal is and Represent two sets of learnable parameters, represents the Sigmoid activation function, Represents an element-by-element multiplication operation. GLU enhances the model's ability to focus on key subspace features by dynamically adjusting the contribution of feature channels. This complements the self-attention modulation mechanism, making the model more adaptable in different scenarios.

[0067] The first characteristic matrix is ​​updated according to the gating signal to obtain a second characteristic matrix.

[0068] The weighted feature matrix is ​​dynamically modulated by the gate signal calculated by GLU to obtain the second feature matrix ,in, The input of the gated linear unit is The corresponding gating signal.

[0069] The second feature matrix is ​​reshaped and then convolved to obtain the tenth feature; the tenth feature is added to the second feature or the fourth feature of the input extraction unit to obtain the output feature of the fusion unit.

[0070] It can be understood that for the fusion unit of the first extraction and fusion module, the second feature and the tenth feature input to the first extraction and fusion module are added together to obtain the output feature of the fusion unit, which is also the output feature of the first extraction and fusion module; for the fusion unit of the subsequent extraction and fusion module, the fourth feature and the tenth feature input to the previous extraction and fusion module are added together to obtain the output feature of the fusion unit, which is also the output feature of the extraction and fusion module. The output feature of the last extraction and fusion module is the output feature of the feature extraction module, which is also the third feature.

[0071] Specifically, the upsampling module is used to perform sub-aperture transformation on the third feature, and then perform convolution, two-dimensional pixel shuffling, and convolution in sequence to obtain the output image of the upsampling module.

[0072] By improving spatial resolution through a lightweight upsampling module, high-quality images can be restored while minimizing parameters, thereby improving model efficiency.

[0073] For example, the target loss function can be an L1 loss function, which is used to measure the pixel-level difference between the spatially high-resolution reconstructed image output by the reconstruction model and the spatially high-resolution labeled image in the sample. The target loss function can also be other functions, which are not limited to this.

[0074] In some possible implementations, the objective loss function can be expressed as follows:

[0075]

[0076]

[0077]

[0078] In the above formula, represents the target loss function, represents the weight coefficient, represents the first loss function, represents the second loss function, represents the height of the light field image, represents the width of the light field image, Represents the spatial high-resolution label pixels in the sample image The corresponding pixel value, Represents pixels in a spatially high-resolution reconstructed image The corresponding pixel value, Represents the binary edge detection map, Represents pixels The corresponding gradient amplitude is represents the first threshold, represents the second threshold; In pixels When there is a pixel with a gradient amplitude greater than the first threshold among the adjacent pixels of When there is no pixel with a gradient amplitude greater than the first threshold among the surrounding adjacent pixels, the value is 0.

[0079] Understandably, pixels The surrounding adjacent pixels refer to the corresponding 8 neighborhood pixels, including pixels , pixels , pixels , pixels , pixels , pixels , pixels , pixels , representing the adjacent pixels at the upper left, above, upper right, left, right, lower left, below, and lower right respectively.

[0080] By introducing the edge weighting mechanism (i.e. ), can effectively prevent the edge blur or distortion in the image, enhance the edge details, and make the final reconstructed image clearer and sharper. ) ensures the overall perceived quality of the image by maintaining the brightness and contrast of the image, helping the model reduce changes in image brightness and color, and ensuring smooth transitions and detail retention in non-edge areas of the image. The model not only focuses on the pixel restoration ability of the overall image, but also enhances the restoration effect of the edge area, thereby improving the image quality at the overall level.

[0081] During the training process of the reconstruction model, the model parameters are continuously updated using the target loss function through the back-propagation algorithm, gradually reducing the difference between the predicted image (i.e., the spatially high-resolution reconstructed image output by the reconstruction model) and the real image (i.e., the spatially high-resolution label image in the sample), optimizing the reconstruction quality of key detail areas such as image edges and textures, and obtaining the target reconstruction model.

[0082] The reconstruction model in the light field image spatial super-resolution reconstruction method is further illustrated below using a specific embodiment.

[0083] Figure 2 This is a schematic diagram of the reconstruction model provided in the embodiment of the present application. Figure 2 As shown in Figure 1, a light field image of size 64×64×5×5×1 (size H×W×A×A×C) is used as the input of the reconstruction model, and the first sub-aperture image and the first macro-pixel image are obtained after reshaping. The first sub-aperture image is input into a convolution layer with a convolution kernel of 3×3 (i.e. Figure 2The first macro pixel image is input into another convolution layer with a convolution kernel of 3×3 (i.e. Figure 2 The convolutional layer performs convolution on the right side of the image (3×3Conv), with a stride of 1, a dilation rate of 5, and an expansion of 5. The convolutional features are then dimensionally upgraded to obtain the first and second features of size 64×64×5×5×64, respectively. The feature extraction module can include 16 sequentially connected extraction and fusion modules.

[0084] The first and second features are input into the first extraction and fusion module in the feature extraction module. The output feature of the first extraction and fusion module (i.e., the fourth feature) and the feature obtained by sub-aperture transformation of the output feature of the first extraction and fusion module (i.e., the fifth feature) are used as inputs to the second extraction and fusion module. Similar to the second extraction and fusion module, the third to sixteenth extraction and fusion modules have the output feature of the previous extraction and fusion module (i.e., the fourth feature) and the feature obtained by sub-aperture transformation of the output feature of the previous extraction and fusion module (i.e., the fifth feature).

[0085] It should be noted that Figure 2 The 16 sequentially connected extraction and fusion modules are not directly reflected in the figure, but only the first extraction and fusion module, the second extraction and fusion module and the sixteenth extraction and fusion module are reflected, and the middle part between the second extraction and fusion module and the sixteenth extraction and fusion module is indicated by a dotted line.

[0086] Figure 3 This is a schematic diagram of the extraction and fusion module provided in the embodiment of this application. Figure 3 As shown, the extraction and fusion module includes an extraction unit and a fusion unit. The extraction unit includes a spatial convolution filter (i.e. Figure 3 SCF in), angular convolution filter (i.e. Figure 3 ACF in ), horizontal EPI convolution filter (i.e. Figure 3 HEPICF in), vertical EPI convolution filter (i.e. Figure 3 VEPICF in ); Among them, the convolution kernel of the spatial convolution filter is 3×3, the stride is 1, and the dilation rate is 1; the convolution kernel of the angular convolution filter is 3×3, the stride is 1, and the dilation rate is 1; the convolution kernel of the horizontal EPI convolution filter is 1×3A, the stride is [1,A], and the dilation rate is 1; the convolution kernel of the vertical EPI convolution filter is 3A×1, the stride is [A,1], and the dilation rate is 1. The horizontal EPI convolution filter and the vertical EPI convolution filter are both connected to a convolution layer with a convolution kernel of 1×1 (i.e. Figure 3 1×1Conv connected to HEPICF and VEPICF respectively).

[0087] The feature obtained by the spatial convolution filter is converted into a macro pixel and used as one of the output features of the extraction unit. The feature obtained by the angular convolution filter is used as one of the output features of the extraction unit. The output of the convolution layer connected to the horizontal EPI convolution filter is shuffled in one dimension (i.e. Figure 3 The feature obtained by 1D Pixel Shuffling in the extraction unit is used as one of the output features of the extraction unit, and the output of the convolution layer connected by the vertical EPI convolution filter is shuffled in one dimension (i.e. Figure 3 The features obtained by 1D Pixel Shuffling in the extraction unit are used as one of the output features of the extraction unit, namely the sixth feature, the seventh feature, the eighth feature, and the ninth feature. The size is the same as the feature size of the input extraction unit, which is 64×64×5×5×64.

[0088] The fusion unit includes a covariance feature module, a normalization module, a self-attention module, and a convolution layer with a convolution kernel of 1×1. The fusion unit first concatenates the sixth feature, the seventh feature, the eighth feature, and the ninth feature to obtain features of size 4×64×64×5×5×64, and then reshapes them to obtain the corresponding feature matrix (i.e., the first feature matrix). The first feature matrix is ​​then processed in sequence through the covariance feature module, the normalization module, and the self-attention module (the functions of the covariance feature module, the normalization module, and the self-attention module are referred to above and will not be repeated here). The processed feature matrix (i.e., the second feature matrix) is obtained. The second feature matrix is ​​reshaped again to obtain features of size 4×64×64×5×5×64, and then passed through a convolution layer with a convolution kernel of 1×1 (i.e., Figure 3 The 1×1Conv in the lower middle part) obtains a feature of size 64×64×5×5×64, and concatenates it with the feature of the angular convolution filter in the current extraction unit to obtain the output feature of the fusion unit with a size of 64×64×5×5×64.

[0089] It should be noted that the output features of the fusion units of the first to fifteenth extraction and fusion modules are all the fourth features; the output features of the fusion unit of the last extraction and fusion module are the output features of the feature extraction module, that is, the third feature.

[0090] Figure 4 Schematic diagram of the upsampling module provided in the embodiment of the present application. Figure 4 As shown in the figure, the upsampling module performs sub-aperture transformation on the input features and then passes them through a 1×1 convolution layer (i.e. Figure 4 The first 1×1Conv from top to bottom is convolved, and then the two-dimensional pixels are shuffled (i.e. Figure 4 2D Pixel Shuffling in ), and then through a 1×1 convolution layer (i.e. Figure 4The second 1×1Conv from top to bottom is convolved to obtain an image of size 128×128×5×5×1, which is used as the output image of the upsampling module.

[0091] Continue to refer Figure 2 The output image of the upsampling module is added to the image obtained by bilinear upsampling of the first sub-aperture image to obtain a spatial high-resolution reconstructed image, which is the output of the reconstruction model.

[0092] S103 , performing spatial super-resolution reconstruction on the input light field image using the target reconstruction model to obtain a spatial high-resolution reconstructed image corresponding to the input light field image.

[0093] For example, after obtaining the target reconstruction model, the light field image that needs to be spatially super-resolution reconstructed can be input into the target reconstruction model, and the target reconstruction model performs spatial super-resolution reconstruction on the light field image to obtain a spatial high-resolution reconstructed image corresponding to the light field image.

[0094] In an embodiment of the present application, a sample set is acquired to construct a reconstruction model that learns different feature representations in different subspaces to improve model performance. The reconstruction model is trained to obtain a target reconstruction model. The target reconstruction model is used to perform spatial super-resolution reconstruction on the input light field image to obtain a spatial high-resolution reconstructed image, which can improve the accuracy of spatial super-resolution reconstruction of the light field image.

[0095] Although this application provides method operation steps such as embodiments or flowcharts, more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in this embodiment is only one way of executing the steps among many, and does not represent the only execution order. When an actual device or client product executes, the method can be executed sequentially according to the embodiment or the accompanying drawings, or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0096] like Figure 5 As shown, the embodiment of the present application also provides a light field image spatial super-resolution reconstruction device. The device includes: The acquisition module 501 is configured to acquire a sample set, where the sample set includes a plurality of samples, and each sample includes a light field image and a spatial high-resolution label image that correspond to each other.

[0097] The module construction and training module 502 is used to construct a reconstruction model, and train the reconstruction model through a sample set until the target loss function converges to obtain a target reconstruction model.

[0098] The image reconstruction module 503 is configured to perform spatial super-resolution reconstruction on the input light field image using a target reconstruction model to obtain a spatial high-resolution reconstructed image corresponding to the input light field image.

[0099] Among them, the reconstruction model includes a first processing module, a feature extraction module and an upsampling module; the light field image is input into the first processing module for reshaping processing to obtain a first sub-aperture image and a first macro-pixel image corresponding to the input light field image; the first sub-aperture image and the first macro-pixel image are convolved and dimensionally upgraded respectively to obtain a first feature and a second feature; the first feature and the second feature are input into the feature extraction module for feature extraction and fusion of the subspace to obtain a third feature, so as to learn different feature representations in different subspaces and improve model performance; the third feature is input into the upsampling module for upsampling, and the upsampled image is added to the image obtained by bilinear upsampling of the first sub-aperture image to obtain a spatial high-resolution reconstructed image corresponding to the input light field image.

[0100] The beneficial effects and specific implementation methods of the present device embodiment can be referred to the aforementioned method embodiment, and will not be described in detail here.

[0101] Some modules in the apparatus described herein may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0102] The devices or modules described in the above application embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function in various modules. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0103] The methods, devices, or modules described herein can be implemented in the form of computer-readable program code. The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller in pure computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions may be considered to be both a software module for implementing the method and a structure within a hardware component.

[0104] An embodiment of the present application further provides a device comprising: a processor; a memory for storing processor-executable instructions; and when the processor executes the executable instructions, the method described in the embodiment of the present application is implemented.

[0105] The embodiments of the present application also provide a non-volatile computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the method described in the embodiments of the present application is implemented.

[0106] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist independently, or two or more modules may be integrated into one module.

[0107] The above-mentioned storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. Such memory can be used to store computer program instructions.

[0108] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, or can be embodied through the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments.

[0109] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0110] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A light field image spatial super-resolution reconstruction method, characterized in that: include: Acquire a sample set, the sample set comprising a plurality of samples, each sample comprising a light field image and a spatial high-resolution label image corresponding to each other; Constructing a reconstruction model, and performing model training on the reconstruction model using the sample set until a target loss function converges, thereby obtaining a target reconstruction model; Performing spatial super-resolution reconstruction on the input light field image using the target reconstruction model to obtain a spatial high-resolution reconstructed image corresponding to the input light field image; Wherein, the reconstruction model includes a first processing module, a feature extraction module and an upsampling module; Inputting the light field image into the first processing module for reshaping processing to obtain a first sub-aperture image and a first macro-pixel image corresponding to the input light field image; performing convolution on the first sub-aperture image and the first macro-pixel image to obtain a first feature and a second feature respectively; Inputting the first feature and the second feature into a feature extraction module to extract and fuse features in the subspace to obtain a third feature, so as to learn different feature representations in different subspaces and improve model performance; The third feature is input into an upsampling module for upsampling, and the upsampled image is added to the image obtained by bilinear upsampling of the first sub-aperture image to obtain a spatial high-resolution reconstructed image corresponding to the input light field image.

2. The method according to claim 1, characterized in that The feature extraction module includes a plurality of extraction and fusion modules connected in sequence; Among the multiple extraction and fusion modules, the input of the first extraction and fusion module is the first feature and the second feature, and the output of the last extraction and fusion module is the third feature; In every two adjacent extraction and fusion modules, the input of the latter extraction and fusion module includes the fourth feature output by the previous extraction and fusion module and the fifth feature obtained by performing sub-aperture transformation on the fourth feature output by the previous extraction and fusion module; The extraction and fusion module includes an extraction unit and a fusion unit; The extraction unit is used to perform feature extraction on the first feature or the fifth feature of the input through a spatial convolution filter, and perform macro-pixel transformation processing on the obtained feature to obtain a sixth feature; it is also used to perform feature extraction on the second feature or the fourth feature of the input through an angular convolution filter to obtain a seventh feature; it is also used to perform feature extraction on the second feature or the fourth feature of the input through a horizontal EPI convolution filter, and perform convolution and one-dimensional pixel scrambling on the obtained feature to obtain an eighth feature; it is also used to perform feature extraction on the second feature or the fourth feature of the input through a vertical EPI convolution filter, and perform convolution and one-dimensional pixel scrambling on the obtained feature to obtain a ninth feature; The fusion unit is used to fuse the sixth feature, the seventh feature, the eighth feature, the ninth feature, and the second feature or the fourth feature input to the extraction unit to obtain the output feature of the fusion unit.

3. The method according to claim 2, characterized in that The fusion unit is used to splice the sixth feature, the seventh feature, the eighth feature, and the ninth feature obtained by the extraction unit and reshape them to obtain a first feature matrix; determine a covariance matrix based on the first feature matrix through a covariance feature module; and perform feature normalization on the covariance matrix through a normalization module to obtain a normalized covariance matrix; Determining an adaptive weight matrix according to the normalized covariance matrix through a self-attention module, weighting the first feature matrix using the adaptive weight matrix to obtain a weighted feature matrix, inputting the weighted feature matrix into a gated linear unit to obtain a gating signal, updating the first feature matrix according to the gating signal to obtain a second feature matrix; reshaping the second feature matrix and then performing convolution to obtain a tenth feature; The tenth feature is added to the second feature or the fourth feature of the input extraction unit to obtain the output feature of the fusion unit.

4. The method according to claim 2, characterized in that The number of extraction fusion modules is 16.

5. The method according to claim 1, characterized in that The upsampling module is used to perform sub-aperture transformation on the third feature, and then perform convolution, two-dimensional pixel shuffling, and convolution in sequence to obtain the output image of the upsampling module.

6. The method according to claim 1, characterized in that The objective loss function is shown as follows: In the above formula, represents the target loss function, represents the weight coefficient, represents the first loss function, represents the second loss function, represents the height of the light field image, represents the width of the light field image, Represents the spatial high-resolution label pixels in the sample image The corresponding pixel value, Represents pixels in a spatially high-resolution reconstructed image The corresponding pixel value, Represents the binary edge detection map, Represents pixels The corresponding gradient amplitude is represents the first threshold, represents the second threshold; In pixels When there is a pixel with a gradient amplitude greater than the first threshold among the surrounding adjacent pixels, the value is 1. When there is no pixel with a gradient amplitude greater than the first threshold among the surrounding adjacent pixels, the value is 0.

7. The method according to claim 1, characterized in that After obtaining the sample set, the method further includes: Perform data preprocessing and data enhancement on the samples in the sample set.

8. A light field image spatial super-resolution reconstruction device, characterized in that: include: An acquisition module, configured to acquire a sample set, wherein the sample set includes a plurality of samples, each sample including a light field image and a spatial high-resolution label image corresponding to each other; A module construction and training module is used to construct a reconstruction model, and to train the reconstruction model using the sample set until the target loss function converges to obtain a target reconstruction model; An image reconstruction module is used to perform spatial super-resolution reconstruction on the input light field image using the target reconstruction model to obtain a spatial high-resolution reconstructed image corresponding to the input light field image; Among them, the reconstruction model includes a first processing module, a feature extraction module and an upsampling module; the light field image is input into the first processing module for reshaping processing to obtain a first sub-aperture image and a first macro-pixel image corresponding to the input light field image; the first sub-aperture image and the first macro-pixel image are convolved and dimensionally increased to obtain a first feature and a second feature; the first feature and the second feature are input into the feature extraction module for feature extraction and fusion of the subspace to obtain a third feature, so as to learn different feature representations in different subspaces and improve model performance; the third feature is input into the upsampling module for upsampling, and the upsampled image is added to the image obtained by bilinear upsampling of the first sub-aperture image to obtain a spatial high-resolution reconstructed image corresponding to the input light field image.

9. A device for performing a light field image spatial super-resolution reconstruction method, characterized in that: include: processor; a memory for storing processor-executable instructions; When the processor executes the executable instructions, the method according to any one of claims 1 to 7 is implemented.

10. A non-volatile computer-readable storage medium, characterized in that: The device comprises a computer program or an instruction for storing the computer program or the instruction, which, when executed, enables the method according to any one of claims 1 to 7 to be implemented.

Images