Light field image real-time super-resolution reconstruction method and device based on self-supervised learning

Abstract

The invention provides a light field image real-time super-resolution reconstruction method and device based on self-supervised learning, and the method comprises the steps: obtaining a high-resolution light field image, carrying out the down-sampling of the high-resolution light field image, obtaining a low-resolution light field image, and converting the low-resolution light field image into a macro image; inputting the central viewpoint image and the macro image of the high-resolution light field image into an angle super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image; the angle super-resolution reconstruction model comprises a super-resolution texture extractor layer, a bidirectional parallax estimation network layer, a fusion up-sampling layer and a reverse mapping layer. According to the process, the performance of the angle super-resolution reconstruction model and the angle super-resolution reconstruction speed are effectively improved, possibility expansion of real-time three-dimensional reconstruction is facilitated, and meanwhile a good display effect on light field equipment can be supported through the number of virtual viewpoints generated by the angle super-resolution reconstruction model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for real-time super-resolution reconstruction of light field images based on self-supervised learning. Background Art

[0002] In recent years, the quality of 3D display has been gradually improved with the advancement of 3D light field display technology. 3D light field display is currently a mainstream implementation method that can achieve large-scale dynamic true 3D display. It can provide viewers with a stereoscopic visual experience close to the real scene, which is in line with people's real feelings of observing the objective world, fully satisfying the physiological and psychological depth clues of the human visual system, and can provide more complete physiological and psychological 3D perception information. Demonstration applications have been achieved in the fields of medical care, national defense, communications, exhibitions, etc. One of the important factors to improve the viewing effect of the human eye is the viewing smoothness when switching viewing angles. This depends on the size of the angular resolution, that is, the number of viewpoints. In theory, within a certain viewing angle, the more viewpoints, the smoother the viewing.

[0003] However, the dense viewpoints currently captured by camera arrays are limited by the transportation and cost of hardware equipment and cannot be obtained conveniently and quickly. At the same time, the quality of light field images collected by light field cameras will be greatly impaired, and the total baseline is too short, resulting in a small viewing angle.

[0004] Therefore, angular super-resolution reconstruction by inputting sparse viewpoints, that is, generating dense virtual viewpoints, is one of the effective methods to increase the number of viewpoints within a certain viewing angle. At the same time, people's requirements for latency have increased, and high-quality real-time display of display content can better meet people's current psychological expectations for technological development. Summary of the invention

[0005] The present invention provides a method and device for real-time super-resolution reconstruction of light field images based on self-supervised learning, so as to solve the defect of high time delay of angular super-resolution reconstruction in the prior art and realize high-quality real-time display of display content.

[0006] The present invention provides a method for real-time super-resolution reconstruction of light field images based on self-supervised learning, comprising the following steps: Acquire a high-resolution light field image, downsample the high-resolution light field image to obtain a low-resolution light field image, and convert the low-resolution light field image into a macro image; Inputting the central viewpoint image of the high-resolution light field image and the macro image into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model; The angular super-resolution reconstruction model includes a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, a fusion upsampling layer and a reverse mapping layer; The bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; The super-resolution texture extractor layer is used to input the central viewpoint images of the continuous high-resolution light field images into the super-resolution texture extractor to obtain a feature map containing high-resolution image texture information output by the super-resolution texture extractor; The fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; The reverse mapping layer is used to reversely map the high-resolution light field image to a corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain the high-resolution virtual viewpoint image.

[0007] According to a method for real-time super-resolution reconstruction of light field images based on self-supervised learning provided by the present invention, the training step of the angular super-resolution reconstruction model includes: Acquire a sample high-resolution light field image, downsample the sample high-resolution light field image to obtain a sample low-resolution light field image, convert the sample low-resolution light field image into a sample macro image, and determine an initial angular super-resolution reconstruction model; Inputting the central viewpoint image of the sample high-resolution light field image and the sample macro image into an initial angular super-resolution reconstruction model to obtain a predicted virtual viewpoint image output by the initial angular super-resolution reconstruction model; Based on the difference between the sample high-resolution light field image and the predicted virtual viewpoint image, a target loss is determined, and based on the target loss, parameter iteration is performed on the initial angular super-resolution reconstruction model to obtain the angular super-resolution reconstruction model.

[0008] According to a light field image real-time super-resolution reconstruction method based on self-supervised learning provided by the present invention, the predicted virtual viewpoint image includes a plurality of first predicted virtual viewpoint images and a plurality of second predicted virtual viewpoint images; The plurality of first predicted virtual viewpoint images and the plurality of second predicted virtual viewpoint images are obtained one by one through reverse mapping based on the predicted bidirectional high-resolution disparity map and the sample high-resolution light field image; the first predicted virtual viewpoint image is a forward predicted virtual viewpoint image, and the second predicted virtual viewpoint image is a reverse predicted virtual viewpoint image; the forward direction is mapped from left to right, and the reverse direction is mapped from right to left; The predicted bidirectional high-resolution disparity map is output by a fusion upsampling layer in the initial angular super-resolution reconstruction model; The determining of the target loss based on the difference between the sample high-resolution light field image and the predicted virtual viewpoint image comprises: determining a first loss based on a difference between the sample high-resolution light field image and the first predicted virtual viewpoint image; determining a second loss based on a difference between the sample high-resolution light field image and the second predicted virtual viewpoint image; The target loss is determined based on the first loss and the second loss.

[0009] According to a light field image real-time super-resolution reconstruction method based on self-supervised learning provided by the present invention, the target loss is determined based on the first loss and the second loss, including: stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image in a height dimension to obtain a first stacked image and a second stacked image; Stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image corresponding to the sample high-resolution light field image at the same viewpoint position in the height dimension to obtain a third stacked image and a fourth stacked image; determining a stack loss based on a difference between the first stacked image and the third stacked image, and a difference between the second stacked image and the fourth stacked image; The target loss is determined based on the first loss, the second loss, and the stacking loss.

[0010] According to a real-time super-resolution reconstruction method for light field images based on self-supervised learning provided by the present invention, the super-resolution texture extractor includes a first convolutional layer and a multi-layer residual block, and the residual block sequentially includes a second convolutional layer, a batch normalization layer and an activation layer.

[0011] According to a light field image real-time super-resolution reconstruction method based on self-supervised learning provided by the present invention, the bidirectional disparity estimation network layer includes a horizontal space feature extraction module, a cost volume construction module, an aggregation module and a regression module; The horizontal spatial feature extraction module is used to extract spatial feature information of the macro image in the horizontal direction to obtain a first feature map; The cost volume construction module is used to obtain a void angle high-dimensional feature volume based on the disparity level, the void rate, the filling value, and the first feature map; The aggregation module is used to globally aggregate the empty corner high-dimensional feature bodies to obtain a target aggregate; The regression module is used to regress the target aggregate to obtain the bidirectional low-resolution disparity map.

[0012] The present invention also provides a light field image real-time super-resolution reconstruction device based on self-supervised learning, comprising the following units: An acquisition unit, configured to acquire a high-resolution light field image, downsample the high-resolution light field image to obtain a low-resolution light field image, and convert the low-resolution light field image into a macro image; An input unit, configured to input the central viewpoint image of the high-resolution light field image and the macro image into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model; The angular super-resolution reconstruction model includes a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, a fusion upsampling layer and a reverse mapping layer; The bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; The super-resolution texture extractor layer is used to input the central viewpoint images of the continuous high-resolution light field images into the super-resolution texture extractor to obtain a feature map containing high-resolution image texture information output by the super-resolution texture extractor; The fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; The reverse mapping layer is used to reversely map the high-resolution light field image to a corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain the high-resolution virtual viewpoint image.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for real-time super-resolution reconstruction of light field images based on self-supervised learning as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for real-time super-resolution reconstruction of light field images based on self-supervised learning.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for real-time super-resolution reconstruction of light field images based on self-supervised learning.

[0016] The present invention provides a method and device for real-time super-resolution reconstruction of light field images based on self-supervised learning. The method and device downsample the input continuous high-resolution light field images to low resolution, convert them into macro images, and input them into a bidirectional disparity estimation network to roughly estimate the bidirectional low-resolution disparity map between each two adjacent downsampled low-resolution input images. The central viewpoint image of the continuous high-resolution image is selected and input into a super-resolution texture extractor to obtain a feature map containing texture information of the high-resolution image, which is then fused and up-sampled with the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map. The bidirectional high-resolution disparity map can be used in combination with the reverse mapping method to map out two groups of high-resolution virtual viewpoint images with the same virtual viewpoint position but different mapping directions, and the virtual viewpoint image is obtained by cropping and fusion. This process effectively improves the performance of the angular super-resolution reconstruction model and the speed of angular super-resolution reconstruction, which helps to expand the possibility of real-time three-dimensional reconstruction. At the same time, the number of virtual viewpoints generated by the angular super-resolution reconstruction model can support good display effects on light field devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the method for real-time super-resolution reconstruction of light field images based on self-supervised learning provided by the present invention.

[0019] Figure 2 It is a structural schematic diagram of the super-resolution texture extractor provided by the present invention.

[0020] Figure 3 Schematic diagram of the structure of the residual block provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the bidirectional disparity estimation network layer provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the structure of the horizontal space residual block provided by the present invention.

[0023] Figure 6 This is the second flow chart of the method for real-time super-resolution reconstruction of light field images based on self-supervised learning provided by the present invention.

[0024] Figure 7 It is a schematic diagram of the bidirectional parallax reverse mapping provided by the present invention.

[0025] Figure 8It is a structural schematic diagram of a light field image real-time super-resolution reconstruction device based on self-supervised learning provided by the present invention.

[0026] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0029] Figure 1 FIG. 1 is one of the flow charts of the method for real-time super-resolution reconstruction of light field images based on self-supervised learning provided by the present invention. Figure 1 As shown, the method includes step 110 and step 120.

[0030] Step 110, acquiring a high-resolution light field image, downsampling the high-resolution light field image to obtain a low-resolution light field image, and converting the low-resolution light field image into a macro image; Specifically, a high-resolution light field image can be obtained. The high-resolution light field image can be represented by HRs. Here, a light field image is an image that records the direction and intensity of light propagation in space, and can capture the depth information and rich visual details in the scene.

[0031] Then, the high-resolution light field image is downsampled to obtain a low-resolution light field image, and the low-resolution light field image is converted into a macro image. The low-resolution light field image can be represented by LRs, and the macro image can be represented by LRM.

[0032] For example, a high-resolution light field image with dimensions of 7×3×H×W is converted to the Y channel and then downsampled by a factor of s to convert it into a macro image in the horizontal direction only, with dimensions of 1×sH×7sW (7 represents 7 consecutive viewpoints on the same baseline, 3 represents the number of RGB channels, H and W represent the height and width of the high-resolution image, and s is the downsampling factor).

[0033] Step 120, inputting the central viewpoint image of the high-resolution light field image and the macro image into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model; The angular super-resolution reconstruction model includes a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, a fusion upsampling layer and a reverse mapping layer; The bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; The super-resolution texture extractor layer is used to input the central viewpoint images of the continuous high-resolution light field images into the super-resolution texture extractor to obtain a feature map containing high-resolution image texture information output by the super-resolution texture extractor; The fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; The reverse mapping layer is used to reversely map the high-resolution light field image to a corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain the high-resolution virtual viewpoint image.

[0034] Specifically, after obtaining the macro image, the central viewpoint image HRc of the high-resolution light field image and the macro image LRM may be input into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model.

[0035] Among them, the angular super-resolution reconstruction model includes a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, a fusion upsampling layer and a reverse mapping layer.

[0036] Here, the bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network. The 2D convolution module with a convolution kernel size of 3, a step size of 1, and a hole rate and a filling rate of (1, 7) is used to accurately extract the spatial feature information B×C×sH×7sW (C is the number of channels) in the horizontal direction, and the D-layer disparity level is set. The 2D convolution module with a gradually increasing receptive field is used to extract the angular feature information and construct a high-dimensional feature volume B×160×D×sH×sW (B is the batch number, 160 is the feature volume dimension, and D is the number of disparity layers) containing spatial and angular feature information. The 3D convolution module is used to aggregate and reduce the feature volume containing spatial and angular feature information to B×V×D×sH×sW (V is the number of bidirectional low-resolution disparity maps). Softmax is used to multiply the weights of each layer of the D dimension and add them together to obtain a bidirectional low-resolution disparity map B×V×sH×sW.

[0037] The super-resolution texture extractor layer is used to input the central viewpoint image of the continuous high-resolution light field image into the super-resolution texture extractor to obtain the feature map containing the texture information of the high-resolution image output by the super-resolution texture extractor. Specifically, the initial convolution block, the four-layer residual block, and the convolution block with a convolution kernel of 1 are used as the super-resolution texture extractor to extract the 1×H×W texture information of the Y channel of the central viewpoint high-resolution image.

[0038] The fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map. That is, the extracted texture information is fused with the bidirectional low-resolution disparity map using the unfold function, which indirectly guides the bidirectional low-resolution disparity map to be upsampled to the bidirectional high-resolution disparity map, thereby realizing super-resolution reconstruction of the low-resolution disparity map.

[0039] The reverse mapping layer is used to reversely map the high-resolution light field image to the corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain a high-resolution virtual viewpoint image. That is, by using the bidirectional high-resolution disparity map obtained in the above steps and combining it with the reverse mapping method, two sets of high-resolution virtual viewpoint images with the same virtual viewpoint position but different mapping directions between each two adjacent images in HRs can be mapped through the bidirectional high-resolution disparity map. By cropping and fusing these two sets of images, 30 dense high-quality high-resolution virtual viewpoint images are finally obtained to achieve angular super-resolution reconstruction of sparse viewpoints.

[0040] The method provided by the embodiment of the present invention downsamples the input continuous high-resolution light field images to low resolution, converts them into macro images and inputs them into a bidirectional disparity estimation network, roughly estimates the bidirectional low-resolution disparity map between each two adjacent downsampled low-resolution input images; selects the central viewpoint image of the continuous high-resolution image and inputs it into a super-resolution texture extractor to obtain a feature map containing the texture information of the high-resolution image, and then fuses and upsamples it with the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; combined with the reverse mapping method, the bidirectional high-resolution disparity map can be used to map two groups of high-resolution virtual viewpoint images with the same virtual viewpoint position but different mapping directions, and the virtual viewpoint image is obtained by cropping and fusion. This process effectively improves the performance of the angular super-resolution reconstruction model and the speed of angular super-resolution reconstruction, helps to expand the possibility of real-time three-dimensional reconstruction, and at the same time, the number of virtual viewpoints generated by the angular super-resolution reconstruction model can support good display effects on light field devices.

[0041] Based on the above embodiment, the training step of the angular super-resolution reconstruction model includes: Step 210, acquiring a sample high-resolution light field image, downsampling the sample high-resolution light field image to obtain a sample low-resolution light field image, converting the sample low-resolution light field image into a sample macro image, and determining an initial angular super-resolution reconstruction model; Step 220, inputting the central viewpoint image of the sample high-resolution light field image and the sample macro image into an initial angular super-resolution reconstruction model to obtain a predicted virtual viewpoint image output by the initial angular super-resolution reconstruction model; Step 230, determining a target loss based on the difference between the sample high-resolution light field image and the predicted virtual viewpoint image, and performing parameter iteration on the initial angular super-resolution reconstruction model based on the target loss to obtain the angular super-resolution reconstruction model.

[0042] Specifically, in order to better obtain the angular super-resolution reconstruction model and improve the model performance and the speed of angular super-resolution reconstruction, the angular super-resolution reconstruction model can be trained based on the following steps: Firstly, a sample high-resolution light field image is obtained, the sample high-resolution light field image is downsampled to obtain a sample low-resolution light field image, the sample low-resolution light field image is converted into a sample macro image, and an initial angular super-resolution reconstruction model is determined.

[0043] Here, the parameters of the initial angular super-resolution reconstruction model may be randomly generated or preset, and this is not specifically limited in the embodiment of the present invention.

[0044] Then, the central viewpoint image and the sample macro image of the sample high-resolution light field image can be input into the initial angular super-resolution reconstruction model to obtain the predicted virtual viewpoint image output by the initial angular super-resolution reconstruction model.

[0045] After obtaining the predicted virtual viewpoint image based on the initial angular super-resolution reconstruction model, the target loss can be determined based on the difference between the sample high-resolution light field image and the predicted virtual viewpoint image, and based on the target loss, the parameters of the initial angular super-resolution reconstruction model can be iterated, and the initial angular super-resolution reconstruction model after the parameter iteration can be used as the angular super-resolution reconstruction model.

[0046] It can be understood that the greater the difference between the sample high-resolution light field image and the predicted virtual viewpoint image, the greater the target loss; and the smaller the difference between the sample high-resolution light field image and the predicted virtual viewpoint image, the smaller the target loss.

[0047] Based on the above embodiment, the predicted virtual viewpoint image includes a plurality of first predicted virtual viewpoint images and a plurality of second predicted virtual viewpoint images; The plurality of first predicted virtual viewpoint images and the plurality of second predicted virtual viewpoint images are obtained one by one through reverse mapping based on the predicted bidirectional high-resolution disparity map and the sample high-resolution light field image; the first predicted virtual viewpoint image is a forward predicted virtual viewpoint image, and the second predicted virtual viewpoint image is a reverse predicted virtual viewpoint image; the forward direction is mapped from left to right, and the reverse direction is mapped from right to left; The predicted bidirectional high-resolution disparity map is output by a fusion upsampling layer in the initial angular super-resolution reconstruction model; Determining the target loss based on the difference between the sample high-resolution light field image and the predicted virtual viewpoint image in step 230 includes: Step 231, determining a first loss based on a difference between the sample high-resolution light field image and the first predicted virtual viewpoint image; Step 232, determining a second loss based on a difference between the sample high-resolution light field image and the second predicted virtual viewpoint image; Step 233: determine the target loss based on the first loss and the second loss.

[0048] Specifically, the predicted virtual viewpoint images include a plurality of first predicted virtual viewpoint images and a plurality of second predicted virtual viewpoint images.

[0049] Here, the plurality of first predicted virtual viewpoint images are obtained one by one by reverse mapping based on the predicted bidirectional high-resolution disparity map and the sample high-resolution light field image. For example, the high-resolution bidirectional disparity map predicted by the network is input into 7 consecutive sample high-resolution light field images. The first 6 viewpoints (i=0,1,2,3,4,5,6) are mapped to the positions of the last 6 viewpoints in sequence to obtain the corresponding first predicted virtual viewpoint image (i=1,2,3,4,5,6).

[0050] The plurality of second predicted virtual viewpoint images are obtained by reverse mapping one by one based on the predicted bidirectional high-resolution disparity map and the sample high-resolution light field image. For another example, the high-resolution bidirectional disparity map predicted by the network maps the last 6 viewpoints of the input 7 consecutive sample high-resolution light field images HRs to the positions of the first 6 viewpoints in sequence to obtain the corresponding second predicted virtual viewpoint image. (i=0,1,2,3,4,5).

[0051] That is, the first predicted virtual viewpoint image is a predicted virtual viewpoint image in the forward direction (mapped from left to right), and the second predicted virtual viewpoint image is a predicted virtual viewpoint image in the reverse direction (mapped from right to left).

[0052] Accordingly, a first loss is determined based on a difference between the sample high-resolution light field image and the first predicted virtual viewpoint image, and a second loss is determined based on a difference between the sample high-resolution light field image and the second predicted virtual viewpoint image.

[0053] Finally, based on the first loss and the second loss, the target loss is determined. The formula of the target loss L1Loss is as follows: (1)

[0054] in, represents the first predicted virtual viewpoint image, represents a sample high-resolution light field image, represents the second predicted virtual viewpoint image, where n is 6 and m is 5.

[0055] Based on the above embodiment, step 233 includes: Step 2331, stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image in a height dimension to obtain a first stacked image and a second stacked image; Step 2332, stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image corresponding to the sample high-resolution light field image at the same viewpoint position in the height dimension to obtain a third stacked image and a fourth stacked image; Step 2333, determining a stacking loss based on a difference between the first stacked image and the third stacked image, and a difference between the second stacked image and the fourth stacked image; Step 2334, determining the target loss based on the first loss, the second loss, and the stacking loss.

[0056] Specifically, the first predicted virtual viewpoint image and the second predicted virtual viewpoint image are stacked in the height dimension to obtain a first stacked image and a second stacked image. , and then stack the sample high-resolution light field images corresponding to the same viewpoint position of the first predicted virtual viewpoint image and the second predicted virtual viewpoint image in the height dimension to obtain a third stacked image and a fourth stacked image denoted as , u and v represent stacking the u-th to v-1-th input high-resolution images in the H dimension.

[0057] Then, the stacking loss can be determined based on the difference between the first stacked image and the third stacked image, and the difference between the second stacked image and the fourth stacked image. The stacking loss is calculated as follows: (2)

[0058] Among them, stackLoss represents stack loss, (i=1,2) represents the third and fourth stacked images, (i=1,2) represents the first and second stacked images, and u and v represent stacking the u-th to v-1-th input high-resolution images in the H dimension.

[0059] Finally, the target loss can be determined based on the first loss, the second loss, and the stacking loss. For example, by giving a larger weight to formula (1) at the beginning of training and gradually increasing the weight of formula (2) at the end of training, the quality of the stacked virtual viewpoint images can be gradually improved until the overall loss function converges. Finally, in the inference stage, the high-quality generation of virtual novel viewpoint images is guaranteed, while the overall mapping speed is increased by about five times.

[0060] The method provided by the embodiment of the present invention proposes a stacked mapping loss based on the L1Loss loss function, and maps the new stacked image after stacking the continuous images of the front part of HRs on the vertical axis, and the image corresponding to the continuous images of the back part of HRs stacked on the vertical axis, and constrains them using the mean absolute error (MAE), and vice versa. By reducing the errors in each of the two sets of constraints, the bidirectional high-resolution disparity map required for mapping can be continuously optimized, which not only ensures the quality of the virtual viewpoint image, but also improves the overall mapping speed.

[0061] Based on the above embodiment, the super-resolution texture extractor includes a multi-layer residual block and a first convolution layer, and the residual block sequentially includes a second convolution layer, a batch normalization layer and an activation layer.

[0062] Specifically, considering that the texture features of high-resolution images are needed as guiding information for super-resolution reconstruction, and because the central viewpoint image of a set of continuous high-resolution light field images contains the richest texture information and position information of the main content of the image and nearby objects, the Y channel data of the high-resolution central viewpoint image is input into the super-resolution texture extractor.

[0063] Figure 2 is a schematic diagram of the structure of the super-resolution texture extractor provided by the present invention, Figure 3 is a schematic diagram of the structure of the residual block provided by the present invention, such as Figure 2 , Figure 3 As shown, the super-resolution texture extractor includes a first convolution layer, a multi-layer residual block and a convolution layer with a convolution kernel of 1, wherein the residual block sequentially includes a second convolution layer, a batch normalization layer (BatchNormalization, BN) and an activation layer.

[0064] Among them, the first convolutional layer and the second convolutional layer can be a multi-layer convolutional neural network (CNN) with a cascade structure, or a deep neural network (DNN), or a combination structure of CNN and DNN, etc., which is not specifically limited in the embodiment of the present invention.

[0065] The activation layer here may use an LReLU (Leaky ReLU) activation function, a Softmax activation function, or a GELU (Gaussian Error Linear Unit) activation function, which is not specifically limited in the embodiment of the present invention.

[0066] After the entire network, the output feature map has a dimension of B×128×sH×sW. Then, through the upsampling module, which includes extracting the mask to highlight the salient features while adjusting the dimension of the feature map to the required B×36×sH×sW, and using the view function to adjust the dimension to (B, 1, 9, 2, 2, sH, sW), and through the unfold function, the mask information is fused with the bidirectional low-resolution disparity map, indirectly guiding the upsampling to a high-resolution bidirectional disparity map with a dimension of B×V×H×W. This module realizes the task of indirectly guiding the upsampling of the bidirectional low-resolution disparity map to the bidirectional high-resolution disparity map.

[0067] Based on the above embodiments, Figure 4 is a schematic diagram of the structure of the bidirectional disparity estimation network layer provided by the present invention, such as Figure 4 As shown, the bidirectional disparity estimation network layer includes a horizontal space feature extraction module, a cost volume construction module, an aggregation module and a regression module; The horizontal spatial feature extraction module is used to extract spatial feature information of the macro image in the horizontal direction to obtain a first feature map; The cost volume construction module is used to obtain a void angle high-dimensional feature volume based on the disparity level, the void rate, the filling value, and the first feature map; The aggregation module is used to globally aggregate the empty corner high-dimensional feature bodies to obtain a target aggregate; The regression module is used to regress the target aggregate to obtain the bidirectional low-resolution disparity map.

[0068] Specifically, the bidirectional disparity estimation network layer includes a horizontal space feature extraction module, a cost volume construction module, an aggregation module and a regression module.

[0069] Among them, the horizontal spatial feature extraction module is used to extract the spatial feature information in the horizontal direction of the macro image to obtain the first feature map. For example, after converting the B×7×3×H×W high-resolution light field image HRc to the YCRCB channel and performing 2x downsampling, the data of the Y channel is converted into a macro image of B×sH×7sW in the horizontal direction only (B is the batch number, 7 represents 7 consecutive viewpoints on the same baseline, 3 represents the number of RGB channels, s represents the downsampling multiple, and H and W represent the height and width of the high-resolution image). When processing macro images with only horizontal pixel offsets, the horizontal spatial feature extraction module is used to accurately extract the spatial feature information in the horizontal direction of the low-resolution macro image.

[0070] The horizontal spatial feature extraction module consists of eight layers of horizontal spatial residual blocks HSR. Figure 5 is a schematic diagram of the structure of the horizontal spatial residual block provided by the present invention, such as Figure 5 As shown in the figure, HSEF represents the convolution layer, where the convolution kernel parameters are as follows: the convolution kernel is 3, the step size is 1, the void ratio and the padding are both (1, 7), the purpose is to expand the receptive field in the horizontal direction and increase the global spatial feature information extraction.

[0071] The cost volume construction module is used to obtain the high-dimensional feature volume of the empty corner based on the disparity level, the hole rate, the filling value, and the first feature map. For example, the dimension of the first feature map after extracting the spatial feature information is B×C×sH×7sW. Combined with the disparity layering method, 16 layers of disparity levels are set. Each layer uses a convolution with a gradually increasing hole rate to improve the receptive field to extract the feature information of the angular space. At the same time, in order to maintain the size of the feature map, the hole rate is determined according to the formula dila And fill value pad Satisfies the following formula: (3)

[0072] (4)

[0073] In formulas (3) and (4), d is the preset disparity value, which is a non-negative integer, and A is the number of input viewpoints. Figure 4 In the construction of the mid-cost volume, the convolution parameters are specifically set to a convolution kernel of 7, a step size, a void ratio, and a padding of 1×7, 1×dila, and 3×pad, respectively. A high-dimensional feature volume B×160×16×sH×sW (160 is the number of channels, and 16 is the number of disparity level layers) containing angular feature information and spatial feature information is constructed through 2D convolution layers with gradually increasing receptive fields.

[0074] Here, the aggregation module is used to globally aggregate the empty corner high-dimensional feature body to obtain the target aggregate. For example, the empty corner high-dimensional feature body is globally aggregated to the target aggregate B×V×16×sH×sW, where the global aggregation module is composed of multiple layers of 3D convolution and 3D convolution residual blocks.

[0075] The regression module is used to regress the target aggregate to obtain a bidirectional low-resolution disparity map. For example, the target aggregate is regressed, and the weights of the 16 layers of disparity are multiplied and added using the Softmax function to obtain the output low-resolution bidirectional disparity map B×V×sH×sW (V represents the number of required bidirectional disparity maps). In summary, the above four parts realize the fast estimation of bidirectional low-resolution disparity maps.

[0076] Based on any of the above embodiments, Figure 6 FIG. 2 is a flow chart of the method for real-time super-resolution reconstruction of light field images based on self-supervised learning provided by the present invention. Figure 6 As shown, the method includes: The embodiment of the present invention uses the disparity super-resolution network and the bidirectional disparity reverse mapping method to complete the task of sparse viewpoint angle super-resolution reconstruction, while generating dense viewpoints and shortening the model execution time. The specific implementation process includes data preprocessing and network model processing, which includes Figure 5 A bidirectional disparity estimation network that achieves millisecond-level fast estimation of bidirectional low-resolution disparity maps, and Figure 3 The super-resolution texture extractor is used to guide the task of super-resolution reconstruction of bidirectional low-resolution disparity maps. The specific scheme is as follows: Data preprocessing: Because the human eye is more sensitive to changes in brightness, training only the Y channel data in YCrCb can meet the human eye's visual requirements for object changes, and adopting a single-channel training method can also reduce the amount of network calculations. In order to achieve better training results, the data is enhanced, mainly using a combination of cropping and inverting the images in the data set, in order to enhance the learning ability and robustness of the network.

[0077] The first step is to obtain a high-resolution light field image, downsample the high-resolution light field image to obtain a low-resolution light field image, and convert the low-resolution light field image into a macro image.

[0078] In the second step, the central viewpoint image and the macro image of the high-resolution light field image are input into the angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model.

[0079] Here, the angular super-resolution reconstruction model includes a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, a fusion upsampling layer, and a reverse mapping layer.

[0080] The bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; the bidirectional disparity estimation network layer includes a horizontal spatial feature extraction module, a cost volume construction module, an aggregation module and a regression module; the horizontal spatial feature extraction module is used to extract the spatial feature information in the horizontal direction of the macro image to obtain a first feature map; the cost volume construction module is used to obtain a high-dimensional feature volume of an empty angle based on the disparity level, the hole rate, the filling value, and the first feature map; the aggregation module is used to globally aggregate the high-dimensional feature volume of the empty angle to obtain a target aggregate; the regression module is used to regress the target aggregate to obtain a bidirectional low-resolution disparity map.

[0081] The super-resolution texture extractor layer is used to input the central viewpoint image of the continuous high-resolution light field image into the super-resolution texture extractor to obtain the feature map containing the texture information of the high-resolution image output by the super-resolution texture extractor. The super-resolution texture extractor includes a multi-layer residual block and a first convolution layer, and the residual block sequentially includes a second convolution layer, a batch normalization layer, and an activation layer.

[0082] The fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; the reverse mapping layer is used to obtain a high-resolution virtual viewpoint image based on the bidirectional high-resolution disparity map.

[0083] The training steps of the angular super-resolution reconstruction model include: S1. Acquire a sample high-resolution light field image, downsample the sample high-resolution light field image to obtain a sample low-resolution light field image, convert the sample low-resolution light field image into a sample macro image, and determine an initial angular super-resolution reconstruction model; S2, inputting the central viewpoint image and the sample macro image of the sample high-resolution light field image into the initial angle super-resolution reconstruction model to obtain a predicted virtual viewpoint image output by the initial angle super-resolution reconstruction model; S3, determining a first loss based on a difference between the sample high-resolution light field image and the first predicted virtual viewpoint image; S4, determining a second loss based on a difference between the sample high-resolution light field image and the second predicted virtual viewpoint image; S5, stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image in the height dimension to obtain a first stacked image and a second stacked image; stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image corresponding to the sample high-resolution light field image at the same viewpoint position in the height dimension to obtain a third stacked image and a fourth stacked image; determining a stacking loss based on a difference between the first stacked image and the third stacked image, and a difference between the second stacked image and the fourth stacked image; S6. Determine a target loss based on the first loss, the second loss, and the stacking loss.

[0084] S7. Based on the target loss, iterate the parameters of the initial angular super-resolution reconstruction model to obtain an angular super-resolution reconstruction model.

[0085] The method provided by an embodiment of the present invention, the above process can be briefly described as follows: on a light field image dataset, a bidirectional low-resolution disparity map between each two adjacent low-resolution input images is quickly estimated using an architecture based on a self-supervised learning algorithm, and the bidirectional low-resolution disparity map is upsampled to high resolution, and a dense high-resolution virtual viewpoint image is generated using these bidirectional high-resolution disparity maps by a reverse mapping method, comprising the following steps: 1) 7 consecutive high-resolution light field images HRs are downsampled to obtain low-resolution light field images LRs and then converted into a macro image LRM, the LRM spatial features and angular features are extracted to construct an empty angular high-dimensional feature body, and the high-dimensional feature body is globally aggregated through a 3D convolution block, and finally the aggregate is regressed using a Softmax function to obtain a bidirectional low-resolution disparity Figure 2 ) Select the central viewpoint image HRc of the continuous high-resolution image and input it into the super-resolution texture extractor to obtain a feature map containing the texture information of the high-resolution image, and then fuse it with the bidirectional low-resolution disparity map and upsample it to obtain a bidirectional high-resolution disparity map. 3) Combined with the reverse mapping method, the bidirectional high-resolution disparity map can be used to map out two groups of high-resolution virtual viewpoint images with the same virtual viewpoint position but different mapping directions between each two adjacent images in HRs. By cropping and fusing these two groups of images, we can finally get a dense high-resolution virtual viewpoint image. 4) Based on the stacking mapping loss proposed by the L1Loss loss function, some continuous input high-resolution images HRs are stacked and mapped as a whole to get a new stacked image, and the image stacked with another part of the continuous images in HRs is constrained using the mean absolute error.

[0086] Figure 7 is a schematic diagram of the bidirectional parallax reverse mapping provided by the present invention, such as Figure 7 As shown in FIG, the high-resolution bidirectional disparity map obtained in the disparity super-resolution network is used, that is, two sets of high-resolution disparity maps with opposite mapping directions, that is, bidirectional mapping from the left viewpoint to the middle viewpoint and from the right viewpoint to the middle viewpoint. Figure 7Using the reverse mapping method, when the left and right viewpoints Vl and Vr are mapped to the middle virtual viewpoint Vi, when the left viewpoint is mapped to Vi, the right side of the virtual viewpoint image at the position of Vi will have a "stretching" problem. Similarly, when the right viewpoint is mapped to Vi, the left side of the virtual viewpoint image at the same position Vi will have a "stretching" problem. This is an unavoidable problem due to reverse mapping. In order to solve this problem, the embodiment of the present invention selects a simple and effective method, that is, two virtual viewpoint images with opposite mappings between every two input high-resolution images are generated by the reverse mapping method. The virtual viewpoint positions of the two images are the same, but they are obtained from different mapping directions, that is, the directions in which the two virtual viewpoint images produce the "stretching" problem are opposite. Therefore, it is only necessary to align the two images and perform cropping and fusion to output a complete and high-quality high-resolution novel viewpoint image. Similarly, when multiple virtual viewpoints are mapped between two high-resolution images, two groups of novel viewpoint images with the same virtual viewpoint positions but different mapping directions can be obtained. After cropping and fusion, 30 high-quality novel viewpoint images are output, achieving the task of real-time reconstruction of angular super-resolution by inputting sparse viewpoints.

[0087] The method provided in the embodiment of the present invention utilizes the obtained bidirectional high-resolution disparity map in combination with the reverse mapping method, and can map out two groups of high-resolution virtual viewpoint images with the same virtual viewpoint position but different mapping directions between each two adjacent images in HRs through the bidirectional high-resolution disparity map. By cropping and fusing the two groups of images, 30 dense, high-quality, high-resolution virtual viewpoint images are finally obtained, thereby realizing angular super-resolution reconstruction of sparse viewpoints.

[0088] The following is a description of the real-time super-resolution reconstruction device for light field images based on self-supervised learning provided by the present invention. The real-time super-resolution reconstruction device for light field images based on self-supervised learning described below and the real-time super-resolution reconstruction method for light field images based on self-supervised learning described above can correspond to each other.

[0089] Based on any of the above embodiments, the present invention provides a light field image real-time super-resolution reconstruction device based on self-supervised learning, Figure 8 is a schematic diagram of the structure of a light field image real-time super-resolution reconstruction device based on self-supervised learning provided by the present invention. Figure 8 As shown, the device comprises: An acquisition unit 810 is used to acquire a high-resolution light field image, downsample the high-resolution light field image to obtain a low-resolution light field image, and convert the low-resolution light field image into a macro image; An input unit 820, configured to input the central viewpoint image of the high-resolution light field image and the macro image into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model; The angular super-resolution reconstruction model includes a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, a fusion upsampling layer and a reverse mapping layer; The bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; The super-resolution texture extractor layer is used to input the central viewpoint images of the continuous high-resolution light field images into the super-resolution texture extractor to obtain a feature map containing high-resolution image texture information output by the super-resolution texture extractor; The fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; The reverse mapping layer is used to reversely map the high-resolution light field image to a corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain the high-resolution virtual viewpoint image.

[0090] The device provided by the embodiment of the present invention downsamples the input continuous high-resolution light field images to low resolution, converts them into macro images, and inputs them into a bidirectional disparity estimation network, roughly estimates the bidirectional low-resolution disparity map between each two adjacent downsampled low-resolution input images; selects the central viewpoint image of the continuous high-resolution image and inputs it into a super-resolution texture extractor to obtain a feature map containing the texture information of the high-resolution image, and then fuses and upsamples it with the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; combined with the reverse mapping method, the bidirectional high-resolution disparity map can be used to map two groups of high-resolution virtual viewpoint images with the same virtual viewpoint position but different mapping directions, and the virtual viewpoint image is obtained by cropping and fusion. This process effectively improves the performance of the angular super-resolution reconstruction model and the speed of angular super-resolution reconstruction, which helps to expand the possibility of real-time three-dimensional reconstruction. At the same time, the number of virtual viewpoints generated by the angular super-resolution reconstruction model can support good display effects on light field devices.

[0091] Based on any of the above embodiments, it further includes a training unit, wherein the training unit is specifically used to: A sample acquisition unit is used to acquire a sample high-resolution light field image, downsample the sample high-resolution light field image to obtain a sample low-resolution light field image, convert the sample low-resolution light field image into a sample macro image, and determine an initial angular super-resolution reconstruction model; A prediction unit, configured to input the central viewpoint image of the sample high-resolution light field image and the sample macro image into an initial angular super-resolution reconstruction model to obtain a predicted virtual viewpoint image output by the initial angular super-resolution reconstruction model; A parameter iteration unit is used to determine a target loss based on a difference between the sample high-resolution light field image and the predicted virtual viewpoint image, and to perform parameter iteration on the initial angular super-resolution reconstruction model based on the target loss to obtain the angular super-resolution reconstruction model.

[0092] Based on any of the above embodiments, the predicted virtual viewpoint image includes a plurality of first predicted virtual viewpoint images and a plurality of second predicted virtual viewpoint images; The plurality of first predicted virtual viewpoint images and the plurality of second predicted virtual viewpoint images are obtained one by one through reverse mapping based on the predicted bidirectional high-resolution disparity map and the sample high-resolution light field image; the first predicted virtual viewpoint image is a forward predicted virtual viewpoint image, and the second predicted virtual viewpoint image is a reverse predicted virtual viewpoint image; the forward direction is mapped from left to right, and the reverse direction is mapped from right to left; The predicted bidirectional high-resolution disparity map is output by a fusion upsampling layer in the initial angular super-resolution reconstruction model; The parameter iteration unit is specifically used for: a first loss determining unit, configured to determine a first loss based on a difference between the sample high-resolution light field image and the first predicted virtual viewpoint image; a second loss determining unit, configured to determine a second loss based on a difference between the sample high-resolution light field image and the second predicted virtual viewpoint image; The target loss determination unit is configured to determine the target loss based on the first loss and the second loss.

[0093] Based on any of the above embodiments, the target loss determination unit is specifically used to: stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image in a height dimension to obtain a first stacked image and a second stacked image; Stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image corresponding to the sample high-resolution light field image at the same viewpoint position in the height dimension to obtain a third stacked image and a fourth stacked image; determining a stack loss based on a difference between the first stacked image and the third stacked image, and a difference between the second stacked image and the fourth stacked image; The target loss is determined based on the first loss, the second loss, and the stacking loss.

[0094] Based on any of the above embodiments, the super-resolution texture extractor includes a first convolutional layer and a multi-layer residual block, and the residual block sequentially includes a second convolutional layer, a batch normalization layer and an activation layer.

[0095] Based on any of the above embodiments, the bidirectional disparity estimation network layer includes a horizontal space feature extraction module, a cost volume construction module, an aggregation module and a regression module; The horizontal spatial feature extraction module is used to extract spatial feature information of the macro image in the horizontal direction to obtain a first feature map; The cost volume construction module is used to obtain a void angle high-dimensional feature volume based on the disparity level, the void rate, the filling value, and the first feature map; The aggregation module is used to globally aggregate the empty corner high-dimensional feature bodies to obtain a target aggregate; The regression module is used to regress the target aggregate to obtain the bidirectional low-resolution disparity map.

[0096] Fig. 9 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Fig. 9 As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communications interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute a real-time super-resolution reconstruction method for light field images based on self-supervised learning, the method comprising: acquiring a high-resolution light field image, downsampling the high-resolution light field image to obtain a low-resolution light field image, and converting the low-resolution light field image into a macro image; inputting the central viewpoint image of the high-resolution light field image and the macro image into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model; the angular super-resolution reconstruction model comprises a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, a fusion upsampling layer and a reverse mapping layer; the bidirectional disparity estimation network The layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; the super-resolution texture extractor layer is used to input the central viewpoint image of the continuous high-resolution light field image into the super-resolution texture extractor to obtain a feature map containing high-resolution image texture information output by the super-resolution texture extractor; the fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; the reverse mapping layer is used to reversely map the high-resolution light field image to the corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain the high-resolution virtual viewpoint image.

[0097] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0098] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the real-time super-resolution reconstruction method of light field images based on self-supervised learning provided by the above methods, and the method includes: acquiring a high-resolution light field image, downsampling the high-resolution light field image to obtain a low-resolution light field image, and converting the low-resolution light field image into a macro image; inputting the central viewpoint image of the high-resolution light field image and the macro image into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model; the angular super-resolution reconstruction model includes a super-resolution texture extractor layer, A bidirectional disparity estimation network layer, a fusion upsampling layer and a reverse mapping layer; the bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; the super-resolution texture extractor layer is used to input the central viewpoint image of the continuous high-resolution light field image into the super-resolution texture extractor to obtain a feature map containing high-resolution image texture information output by the super-resolution texture extractor; the fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; the reverse mapping layer is used to reversely map the high-resolution light field image to the corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain the high-resolution virtual viewpoint image.

[0099] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the real-time super-resolution reconstruction method of light field images based on self-supervised learning provided by the above-mentioned methods, the method comprising: acquiring a high-resolution light field image, downsampling the high-resolution light field image to obtain a low-resolution light field image, and converting the low-resolution light field image into a macro image; inputting a central viewpoint image of the high-resolution light field image and the macro image into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model; the angular super-resolution reconstruction model comprises a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, and a fusion upsampling layer. layer and a reverse mapping layer; the bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; the super-resolution texture extractor layer is used to input the central viewpoint image of the continuous high-resolution light field image into the super-resolution texture extractor to obtain a feature map containing high-resolution image texture information output by the super-resolution texture extractor; the fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; the reverse mapping layer is used to reversely map the high-resolution light field image to the corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain the high-resolution virtual viewpoint image.

[0100] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time super-resolution reconstruction method for light field images based on self-supervised learning, characterized in that: include: Acquire a high-resolution light field image, downsample the high-resolution light field image to obtain a low-resolution light field image, and convert the low-resolution light field image into a macro image; Inputting the central viewpoint image of the high-resolution light field image and the macro image into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model; The angular super-resolution reconstruction model includes a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, a fusion upsampling layer and a reverse mapping layer; The bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; The super-resolution texture extractor layer is used to input the central viewpoint images of the continuous high-resolution light field images into the super-resolution texture extractor to obtain a feature map containing high-resolution image texture information output by the super-resolution texture extractor; The fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; The reverse mapping layer is used to reversely map the high-resolution light field image to a corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain the high-resolution virtual viewpoint image.

2. The method for real-time super-resolution reconstruction of light field images based on self-supervised learning according to claim 1, characterized in that: The training steps of the angular super-resolution reconstruction model include: Acquire a sample high-resolution light field image, downsample the sample high-resolution light field image to obtain a sample low-resolution light field image, convert the sample low-resolution light field image into a sample macro image, and determine an initial angular super-resolution reconstruction model; Inputting the central viewpoint image of the sample high-resolution light field image and the sample macro image into an initial angular super-resolution reconstruction model to obtain a predicted virtual viewpoint image output by the initial angular super-resolution reconstruction model; Based on the difference between the sample high-resolution light field image and the predicted virtual viewpoint image, a target loss is determined, and based on the target loss, parameter iteration is performed on the initial angular super-resolution reconstruction model to obtain the angular super-resolution reconstruction model.

3. The method for real-time super-resolution reconstruction of light field images based on self-supervised learning according to claim 2, characterized in that: The predicted virtual viewpoint images include a plurality of first predicted virtual viewpoint images and a plurality of second predicted virtual viewpoint images; The plurality of first predicted virtual viewpoint images and the plurality of second predicted virtual viewpoint images are obtained one by one through reverse mapping based on the predicted bidirectional high-resolution disparity map and the sample high-resolution light field image; the first predicted virtual viewpoint image is a forward predicted virtual viewpoint image, and the second predicted virtual viewpoint image is a reverse predicted virtual viewpoint image; the forward direction is mapped from left to right, and the reverse direction is mapped from right to left; The predicted bidirectional high-resolution disparity map is output by a fusion upsampling layer in the initial angular super-resolution reconstruction model; The determining of the target loss based on the difference between the sample high-resolution light field image and the predicted virtual viewpoint image comprises: determining a first loss based on a difference between the sample high-resolution light field image and the first predicted virtual viewpoint image; determining a second loss based on a difference between the sample high-resolution light field image and the second predicted virtual viewpoint image; The target loss is determined based on the first loss and the second loss.

4. The method for real-time super-resolution reconstruction of light field images based on self-supervised learning according to claim 3, characterized in that: The determining the target loss based on the first loss and the second loss includes: stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image in a height dimension to obtain a first stacked image and a second stacked image; Stacking the first predicted virtual viewpoint image and the second predicted virtual viewpoint image corresponding to the sample high-resolution light field image at the same viewpoint position in the height dimension to obtain a third stacked image and a fourth stacked image; Based on the difference between the first stack image and the third stack image, and the difference between the second stack image and the fourth stack image, a stack loss is determined; based on the first loss, the second loss, and the stack loss, the target loss is determined.

5. The method for real-time super-resolution reconstruction of light field images based on self-supervised learning according to any one of claims 1 to 4, characterized in that: The super-resolution texture extractor includes a first convolutional layer and a multi-layer residual block, and the residual block sequentially includes a second convolutional layer, a batch normalization layer and an activation layer.

6. The method for real-time super-resolution reconstruction of light field images based on self-supervised learning according to any one of claims 1 to 4, characterized in that: The bidirectional disparity estimation network layer includes a horizontal space feature extraction module, a cost volume construction module, an aggregation module and a regression module; The horizontal spatial feature extraction module is used to extract spatial feature information of the macro image in the horizontal direction to obtain a first feature map; The cost volume construction module is used to obtain a void angle high-dimensional feature volume based on the disparity level, the void rate, the filling value, and the first feature map; The aggregation module is used to globally aggregate the empty corner high-dimensional feature bodies to obtain a target aggregate; The regression module is used to regress the target aggregate to obtain the bidirectional low-resolution disparity map.

7. A light field image real-time super-resolution reconstruction device based on self-supervised learning, characterized in that: include: An acquisition unit, configured to acquire a high-resolution light field image, downsample the high-resolution light field image to obtain a low-resolution light field image, and convert the low-resolution light field image into a macro image; An input unit, configured to input the central viewpoint image of the high-resolution light field image and the macro image into an angular super-resolution reconstruction model to obtain a high-resolution virtual viewpoint image output by the angular super-resolution reconstruction model; The angular super-resolution reconstruction model includes a super-resolution texture extractor layer, a bidirectional disparity estimation network layer, a fusion upsampling layer and a reverse mapping layer; The bidirectional disparity estimation network layer is used to input the macro image into the bidirectional disparity estimation network to obtain a bidirectional low-resolution disparity map output by the bidirectional disparity estimation network; The super-resolution texture extractor layer is used to input the central viewpoint images of the continuous high-resolution light field images into the super-resolution texture extractor to obtain a feature map containing high-resolution image texture information output by the super-resolution texture extractor; The fusion upsampling layer is used to fuse and upsample the feature map and the bidirectional low-resolution disparity map to obtain a bidirectional high-resolution disparity map; The reverse mapping layer is used to reversely map the high-resolution light field image to a corresponding viewpoint position through the bidirectional high-resolution disparity map to obtain the high-resolution virtual viewpoint image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for real-time super-resolution reconstruction of light field images based on self-supervised learning is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for real-time super-resolution reconstruction of light field images based on self-supervised learning is implemented as claimed in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for real-time super-resolution reconstruction of light field images based on self-supervised learning is implemented as claimed in any one of claims 1 to 6.

Images