Light field spatial-angular continuum super-resolution method based on local neural geometric learning

Abstract

The application discloses a light field space-angle continuous domain super-resolution method based on local neural geometric learning, belongs to the field of deep learning and computer vision, and comprises the following steps: S1, acquiring a light field polar geometric image and constructing a neural network model; S2, sending a sub-aperture image array of the light field image into a space-angle perception geometric coding module to obtain latent geometric coding of space-angle perception; S3, sending the latent geometric coding of space-angle perception into a local neural geometric learning module to obtain space-angle continuous domain latent geometric coding; S4, sending the space-angle continuous domain latent geometric coding into an extension rendering module to obtain a light field image; S5, setting a loss function; and S6, testing a light field-space angle continuous domain super-resolution task by using the neural network model. The light field space-angle continuous domain super-resolution method based on local neural geometric learning can realize super-resolution of the light field image space dimension and angle dimension simultaneously and at any scale.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and computer vision, and particularly relates to a light field space-angle continuous domain super-resolution method based on local neural geometric learning. BACKGROUND

[0002] A light field camera based on a microlens array records the angle and light radiation information of incident light rays by inserting a microlens array (MLA) between an image sensor and a main lens, thereby recording the three-dimensional geometric information of a scene in terms of light ray space and angle. However, due to the limitation of the imaging resolution of the image sensor, there is a trade-off between the spatial resolution and the angular resolution in the light field imaging process, resulting in that the spatial and angular resolutions of the light field image are difficult to meet the actual application requirements. Therefore, using the sparse and low-resolution sub-aperture image array in the light field image, a dense and high-resolution sub-aperture image array is reconstructed, and more high-resolution images under different viewing angles are provided for actual light field applications, that is, the spatial and angular super-resolution reconstruction of the light field image has become an important research task in the field of light field imaging. The existing light field image super-resolution reconstruction method has two main limitations: (1) The traditional light field image super-resolution reconstruction method is based on a light field imaging geometric model, and its performance depends on the accurate estimation of the camera internal parameters and the scene depth information. In actual applications, the focal length and other camera internal parameters will change constantly, and the scene depth is difficult to accurately obtain; (2) The existing light field image super-resolution reconstruction method can only perform super-resolution reconstruction in a single dimension of space or angle, and cannot realize simultaneous super-resolution reconstruction in space and angle. Moreover, the light field image can only be super-resolved to a fixed scale, such as obtaining a two-fold or four-fold resolution image in the spatial dimension, or obtaining a 7x7 or 9x9 sub-aperture image array in the angular dimension, and cannot realize reconstruction of any resolution in the space and angle continuous domain. SUMMARY

[0003] The purpose of the present application is to provide a light field space-angle continuous domain super-resolution method based on local neural geometric learning to solve the problems in the background.

[0004] To achieve the above purpose, the present application provides a light field space-angle continuous domain super-resolution method based on local neural geometric learning, which takes a sparse and low-resolution sub-aperture image array as input, feeds it into a neural network model, and draws an arbitrary spatial and angular resolution sub-aperture image array; comprising the following steps:

[0005] S1, acquiring a light field polar geometric image, and constructing a light field space-angle continuous domain super-resolution neural network model, including a space-angle perception geometric coding module, a local neural geometric learning module and an extended rendering module;

[0006] S2, input the sparse and low-resolution light field image sub-aperture image array into the space-angle perception geometry coding module to obtain the space-angle perception potential geometry coding;

[0007] S3, input the space-angle perception potential geometry coding into the local neural geometry learning module to obtain the space-angle continuous domain potential geometry coding;

[0008] S4, input the space-angle continuous domain potential geometry coding into the extended rendering module to obtain the dense and high-resolution light field image;

[0009] S5, set the loss function for the neural network model;

[0010] S6, use the trained neural network model to test the light field-space angle continuous domain super-resolution task on the test data set.

[0011] Preferably, S2 is to input the sparse and low-resolution light field image into a convolutional layer with a 3x3 convolution kernel first to obtain an initial feature map array F with a dimension of (U, V, X, Yv, C) init , and then input into the space-angle perception geometry coding module to obtain the space-angle perception potential geometry coding, the space-angle perception geometry coding module is composed of an EPIConv module, an SAConv module and a space-angle perception Transformer module; for the light field image L(u, v, x, y), the EPIConv module is used to extract the EPI geometric features in the horizontal EPI image and the vertical EPI image, the SAConv module is used to extract the spatial and angular features on the (x, y) and (u, v) planes, and the space-angle perception Transformer module is used to obtain the global dependency relationship of the features obtained by the EPIConv module and the SAConv module.

[0012] Preferably, the specific steps of the EPIConv module are:

[0013] According to the horizontal EPI image extraction method, VxY horizontal EPI feature images are extracted from F init , and the horizontal polar geometric features with a dimension of (VY, U, X, C) are obtained by splicing, denoted as F init_h ; F init_h is input into a convolutional layer with a 3xU kernel, and then a point convolutional layer with a 1x1 kernel is used to obtain the horizontal EPI feature F epi_h ; similarly, according to the vertical EPI image extraction method, UxX vertical EPI feature images are extracted from F init , and the vertical polar geometric features with a dimension of (UX, V, Y, C) are obtained by splicing, denoted as F init_v ; F init_vInput to a convolution layer with a kernel of 3×V and a point convolution layer with a kernel of 1×1 to extract the vertical EPI feature F epi_v , F epi_h and F epi_v After concatenation in the channel dimension, it is input into a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 to generate the EPI feature F epi Finally, F epi is reorganized into a feature vector T of dimension (VY,UX,C / 2) epi .

[0014] Preferably, the SAConv module consists of two feature extraction branches and a feature fusion layer. The two feature extraction branches include an upper branch and a lower branch. The upper branch is used to extract spatial features. init Input into two convolutional layers with a kernel of 3×3 to obtain the spatial features F of the light field image spa ; The lower branch is used to extract angle features. First, F init The angle dimension is stacked into the channel dimension to obtain C×U×V feature maps of size (X,Y), denoted as F init_ang ; Then F init_ang Input to two convolutional layers with a kernel of 1×1 to generate the angle feature F of the light field image ang ; Then F ang The reorganized feature array is of dimension U×V×X×Y×C and compared with F spa Splicing is performed on the channel dimension to obtain the composite feature F spa_ang , and then generate the space-corner feature F through a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 sa ;Finally, similar to the EPIConv module, F sa is reorganized into a null-angle eigenvector T of dimension (VY,UX,C / 2) sa .

[0015] Preferably, the space-angle-aware Transformer module consists of an encoder E s and encoder E c Composition, encoder E s is a standard Transformer encoder with a self-attention mechanism to obtain the global dependencies of the input feature vectors; the encoder E c is a cross-attention encoder that retains extremely geometrically relevant null-corner features while ignoring irrelevant detail features. Specifically:

[0016] First, T is transformed into epi and T sa Splicing to get the composite vector T epi_sa As Es Input; then E s The output is re-decomposed into the potential EPI code Z epi and enhanced null-angle marker T' sa ; In encoder E c In the middle, Z epi As the query vector of the cross attention mechanism, T' sa As the key vector and value vector of the cross attention mechanism, the output is the latent space-angle code Z with geometric meaning sa ; Z epi and Z sa Stacked in the channel dimension to form the final latent geometric code Z g , the dimensions are (VY,UX,C).

[0017] Preferably, S3 specifically includes:

[0018] The local neural geometry learning module consists of a LIGF _h module and a LIGF _V The cascade structure consists of modules; according to the divide-and-conquer learning strategy, the latent geometric code Z g The light field implicit function learning is transformed into the horizontal and vertical two-dimensional light field geometric implicit function cascade learning, which is:

[0019] According to the horizontal EPI image extraction method, first Z g Decomposed into V×Y horizontal latent geometric codes Then the local implicit image function (LIIF) method is used to convert Z h Interpolation to latent feature map Finally, Z l and Z h The implicit code Z is fed into an MLP with 256 hidden layers and a ReLU activation layer. l The local feature map is mapped to the horizontal feature map Z h The corresponding coordinates of the light field are obtained to obtain the complete potential geometric coding in the horizontal direction

[0020] According to the vertical EPI image extraction method, Z' is first decomposed into U'×X' horizontal latent geometric codes. Then the local implicit image function method is used to transform Z V Interpolation to latent feature map Finally, Z l and Z v The implicit code Z is fed into an MLP with 256 hidden layers and a ReLU activation layer. l The local feature map is mapped to the vertical feature map Z vcorresponding coordinates of the light field to obtain potential geometry coding in the horizontal direction of the light field

[0021] Preferably, S4 specifically comprises:

[0022] Angularly continuous domain potential geometry coding Z C is input into a rendering module composed of three three-dimensional convolution layers in cascade, and the convolution kernel size of each layer is 1*1, so as to gradually compress the angularly continuous domain potential geometry coding Z of the light field C to the target output channel number c, and reshape it into a light field macro-pixel image Finally, it is converted into a light field sub-aperture array image with high angular resolution, which is

[0023] Preferably, the network model in S5 adopts the absolute value error (L1) between the output view and the GT view as the loss function, and specifically comprises:

[0024] The loss function Loss between the output high-angular resolution sub-aperture array image and the real high-angular resolution sub-aperture image The calculation formula of the loss function Loss is:

[0025]

[0026] Preferably, S6 specifically comprises:

[0027] The light field angularly continuous domain super-resolution method based on local neural geometry learning can reconstruct a light field with M times higher spatial resolution and N*N high angular resolution, which can be regarded as an N*N M times sub-aperture array. Therefore, as long as the down-sampling ratio of the input is controlled, the output can be aligned with the real label in the spatial domain after super-resolution, and from this array, a sub-aperture array with the same angular resolution as the original light field image can be obtained through a certain step interval, denoted as The original light field image and the reconstructed image are evaluated in terms of structural similarity index (SSIM) and Peak Signal to Noise Ratio (PSNR), and the evaluation results of the reconstructed image are taken as the evaluation results of the light field angularly continuous domain super-resolution task test.

[0028] Therefore, the light field angularly continuous domain super-resolution method based on local neural geometry learning has the following beneficial effects:

[0029] ​(1) Provide a light field space-angle continuous domain super-resolution method based on local neural geometric learning, which can realize simultaneous and arbitrary scale super-resolution of light field image space dimension and angle dimension;

[0030] (2) By mapping the light field polar geometric image to an interpolable latent space, learning the space and angle information, a space-angle consistent local neural geometric learning framework for simultaneous super-resolution in space-angle continuous domain is proposed;

[0031] (3) A space-angle perception geometric encoder is proposed to extract light field polar geometric latent geometry code, integrate the local and global dependencies of light field polar geometry, and embed the light field space-angle correlation into the latent geometry code through the cross-attention mechanism of space-angle perception;

[0032] (4) A divide-and-conquer local neural geometric learning strategy is used to effectively reduce memory occupation by converting four-dimensional light field implicit function learning into two-dimensional light field polar geometry implicit function learning with shared weights.

[0033] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The neural network structure diagram of the light field space-angle continuous domain super-resolution based on local neural geometric learning of the embodiment of the present application is shown in the figure;

[0035] Figure 2 The structure diagram of the EPIConv module of the present application is shown in the figure;

[0036] Figure 3 The structure diagram of the SAConv module of the present application is shown in the figure;

[0037] Figure 4 The structure diagram of the space-angle perception Transformer module of the present application is shown in the figure;

[0038] Figure 5 The effect diagram of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0039] The following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0040] The bi-plane representation method of the light field image is denoted as L(u, v, x, y), wherein (u, v) is the angular coordinate of the light field image, (x, y) is the spatial coordinate of the light field image, wherein y ∈ [1, U], v ∈ [1, V], x ∈ [0, X], y ∈ [0, Y]. L(u, v)(x, y) represents a sub-aperture image (SAI) under a given (u, v) angular coordinate. The light field image can be regarded as a set of sub-aperture image arrays.

[0041] The light field EPI image (epipolar plane image) is obtained by stacking a row (or a column) of pixels in the same row (or the same column) of sub-aperture images in the light field sub-aperture image array: given the v and y coordinates in the light field image, a horizontal EPI image L(v, y)(u, x) can be obtained. Given the u and x coordinates in the light field image, a vertical EPI image L(u, x)(v, y) can be obtained. A light field image with an angular resolution of U × V and a spatial resolution of X × Y can obtain V × Y horizontal EPI images and U × X vertical EPI images.

[0042] Please refer to Figure 1 , the light field space-angle continuous domain super-resolution method based on local neural geometric learning, comprising the following steps:

[0043] S1, a horizontal EPI image (or a vertical EPI image) of the light field polar geometric image is obtained by stacking a row (or a column) of pixels in a row (or a column) of sub-aperture images in the light field sub-aperture image array; then a light field space-angle continuous domain super-resolution neural network model is constructed, which mainly includes a space-angle perception geometric coding module, a local neural geometric learning module and an extended rendering module;

[0044] S2, the sparse and low-resolution light field image is first input into a convolution layer with a convolution kernel of 3 × 3 to obtain an initial feature map array F with a dimension of (U, V, X, Y, C) init , and then input into the space-angle perception geometric coding module to obtain the space-angle perception latent geometric code, the space-angle perception geometric coding module is composed of three modules: an EPIConv module, an SAConv module and a space-angle perception Transformer module; for the light field image L(u, v, x, y), the EPIConv module is used to extract the EPI geometric features in the horizontal EPI image and the vertical EPI image, the SAConv module is used to extract the spatial and angular features on the (x, y) and (u, v) planes, and the space-angle perception Transformer module is used to obtain the global dependency relationship of the features obtained by the EPIConv module and the SAConv module.

[0045] The EPIConv module, for example,Figure 2 As shown, according to the horizontal EPI image extraction method, from F init Extract V×Y horizontal EPI feature images and stitch them together to obtain horizontal polar geometric features with dimensions (VY, U, X, C), denoted as F init_h ; F init_h Input to a convolution layer with a kernel of 3×U, and then pass through a point convolution layer with a kernel of 1×1 to obtain the horizontal EPI feature F epi_h Similarly, according to the vertical EPI image extraction method, from F init Extract U×X vertical EPI feature images and stitch them together to obtain vertical polar geometric features with dimensions (UX, V, Y, C), denoted as F init_v ; F init_v Input to a convolution layer with a kernel of 3×V and a point convolution layer with a kernel of 1×1 to extract the vertical EPI feature F epi_v , F epi_h and F epi_v After concatenation in the channel dimension, it is input into a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 to generate the EPI feature F epi Finally, F epi is reorganized into a feature vector F of dimension (VY,UX,C / 2) epi .

[0046] SAConv module, such as Figure 3 As shown, it is used to extract and fuse the spatial and angular features of the light field. It consists of two feature extraction branches and a feature fusion layer. The two feature extraction branches include an upper branch and a lower branch. The upper branch is used to extract spatial features. init Input into two convolutional layers with a kernel of 3×3 to obtain the spatial features F of the light field image spa ; The lower branch is used to extract angle features. First, F init The angle dimension is stacked into the channel dimension to obtain C×U×V feature maps of size (X,Y), denoted as F init_ang ; Then F init_ang Input to two convolutional layers with a kernel of 1×1 to generate the angle feature F of the light field image ang ; Then F ang The reorganized feature array is of dimension U×V×X×Y×C and compared with F spa Splicing is performed on the channel dimension to obtain the composite feature F spa_ang , and then generate the space-corner feature F through a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 sa ;Finally, similar to the EPIConv module, F sa is reorganized into a null-angle eigenvector T of dimension (VY,UX,C / 2) sa.

[0047] The air-angle aware Transformer module is used to obtain the global dependency of light field space, angle and polar geometry features, as shown in Figure 4 The air-angle aware Transformer module is used to obtain the global dependency of light field space, angle and polar geometry features, as shown in s c The air-angle aware Transformer module is used to obtain the global dependency of light field space, angle and polar geometry features, as shown in s The air-angle aware Transformer module is used to obtain the global dependency of light field space, angle and polar geometry features, as shown in c The air-angle aware Transformer module is used to obtain the global dependency of light field space, angle and polar geometry features, as shown in

[0048] First, T epi and T sa are spliced in the channel dimension to obtain a composite vector T epi_sa as the input of E s ; then the output of E s is re-decomposed into latent EPI encoding Z epi and enhanced air-angle label T' sa ; in the encoder E c , Z epi is taken as the query vector of the cross-attention mechanism, T' sa is taken as the key vector and value vector of the cross-attention mechanism, and the output is the latent air-angle encoding Z sa with geometric meaning; Z epi and Z sa are stacked in the channel dimension to form the final latent geometric encoding Z g , with the dimension (VY, UX, C).

[0049] S3, the air-angle aware latent geometric encoding is sent into the local neural geometry learning module to obtain the air-angle continuous domain latent geometric encoding; specifically:

[0050] The local neural geometry learning module is composed of a cascade structure of an LIGF _h module and an LIGF _V module; according to the learning strategy of divide and conquer, the light field implicit function learning of the latent geometric encoding Z g is converted into horizontal and vertical two-dimensional light field polar geometry implicit function cascade learning, which is:

[0051] According to the horizontal EPI image extraction method, Z g is first decomposed into VxY horizontal latent geometric encodings Then the local implicit image function (LIIF) method is used to interpolate Z h to the latent feature map Finally, Z​l and Z h The implicit code Z is fed into an MLP with 256 hidden layers and a ReLU activation layer. l The local feature map is mapped to the horizontal feature map Z h The corresponding coordinates of the light field are obtained to obtain the complete potential geometric coding in the horizontal direction

[0052] According to the vertical EPI image extraction method, Z' is first decomposed into U'×X' horizontal latent geometric codes. Then the local implicit image function method is used to transform Z V Interpolation to latent feature map Finally, Z l and Z v The implicit code Z is fed into an MLP with 256 hidden layers and a ReLU activation layer. l The local feature map is mapped to the vertical feature map Z v The corresponding coordinates of the light field are obtained to obtain the potential geometric coding in the horizontal direction

[0053] S4. Send the spatial-angular continuous domain latent geometry code to the extended rendering module to obtain a dense and high-resolution light field image; specifically:

[0054] Space-angle continuous field latent geometry coding Z C It is fed into a rendering module consisting of three cascaded three-dimensional convolution layers, with a convolution kernel size of 1×1 in each layer, which gradually compresses the light field space-angle continuous domain potential geometric code Z C The number of feature channels C is converted to the target output channel number c, and reshaped into a light field macro pixel image. Finally, it is converted into a light field subaperture array image with high spatial-angular resolution, which is

[0055] S5. Build a network model and set the loss function; specifically:

[0056] In this embodiment, the network model uses the absolute value error (L1) between the output view and the ground-truth view as the loss function;

[0057] Output high-altitude angular resolution sub-aperture array image and true high-angle resolution sub-aperture images The calculation formula of the loss function Loss between is:

[0058]

[0059] S6, using the trained network model, performing light field-angular continuous domain super-resolution task test on the test data set, specifically:

[0060] The light field-angular continuous domain super-resolution method based on local neural geometric learning can reconstruct a light field with M times higher spatial resolution and N*N high angular resolution, which can be regarded as an N*N M times sub-aperture array. Therefore, as long as the down-sampling ratio of the input is controlled, the output can be aligned with the real label in the spatial domain after super-resolution. At the same time, from this array, a sub-aperture array with the same angular resolution as the original light field can be obtained through a certain step interval, denoted as The evaluation results of the original light field image and the reconstructed light field image are evaluated in terms of structural similarity index (SSIM) and Peak Signal to Noise Ratio (PSNR), and the evaluation results of the original light field image and the reconstructed light field image are evaluated in terms of structural similarity index (SSIM) and Peak Signal to Noise Ratio (PSNR). The evaluation results of the original light field image and the reconstructed light field image are evaluated in terms of structural similarity index (SSIM) and Peak Signal to Noise Ratio (PSNR), and the evaluation results of the original light field image and the reconstructed light field image are evaluated in terms of structural similarity index (SSIM) and Peak Signal to Noise Ratio (PSNR).

[0061] Under the light field-angular continuous domain super-resolution task of angle domain 2*2 to 5*5 and spatial domain 2*2, the index comparison of the method of the embodiment and other methods is shown in Table 1:

[0062] Table 1 Comparison of indexes of different methods

[0063]

[0064] Since other methods cannot achieve simultaneous super-resolution in angular and spatial domains, the angle super-resolution module is first sent, then the spatial super-resolution module is sent, and then the method is compared with the method. It can be seen that the method has good performance in multiple data sets, and the actual effect is shown in Figure 5

[0065] ​​Therefore, the application adopts the light field space-angle continuous domain super-resolution method based on local neural geometric learning, and first obtains a horizontal EPI image (or a vertical EPI image) of light field polar geometry through the stacking of a row (or a column) of pixels in a row (or a column) of sub-aperture images in a light field sub-aperture image array; then a light field space-angle continuous domain super-resolution neural network model is constructed, which mainly includes a space-angle perception geometry coding module, a local neural geometry learning module and an extended rendering module. The space-angle perception geometry coding module obtains the local and global dependence of light field geometry from the horizontal EPI image (or the vertical EPI image) through a convolutional neural network and a self-attention mechanism, and integrates the correlation of light field space and angle through a cross-attention mechanism; the local neural geometry learning module converts four-dimensional implicit function learning into cascade learning of two two-dimensional local implicit functions through the idea of divide and conquer; and the extended rendering module is used to generate a light field macro-pixel image with high space-angle resolution.

[0066] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A light field spatial-angular continuous domain super-resolution method based on local neural geometric learning takes a sparse and low-resolution sub-aperture image array as input and feeds it into a neural network model to draw a sub-aperture image array with arbitrary spatial and angular resolution. The method is characterized by: The following steps are involved: S1. Obtain the light field polar geometry image and construct a light field spatial-angular continuous domain super-resolution neural network model, including a spatial-angular perception geometric encoding module, a local neural geometry learning module, and an extended rendering module; S2, sending the sub-aperture image array of the sparse and low-resolution light field image to the space-angle-aware geometric coding module to obtain the space-angle-aware latent geometric code; S3, feeding the space-angle perception latent geometry code into the local neural geometry learning module to obtain the space-angle continuous domain latent geometry code; S4, sending the spatial-angular continuous domain latent geometry code to the extended rendering module to obtain a dense and high-resolution light field image; S5. Set the loss function for the neural network model; S6. Use the trained neural network model to perform light field-space angle continuous domain super-resolution task test on the test dataset; S2 is to input the sparse and low-resolution light field image into a convolution layer with a convolution kernel of 3×3, and obtain a dimension of The initial feature map array , and then input it into the space-angle-aware geometric coding module to obtain the space-angle-aware potential geometric coding. The space-angle-aware geometric coding module consists of the EPIConv module, the SAConv module, and the space-angle-aware Transformer module. For light field images , the EPIConv module is used to extract EPI geometric features in horizontal EPI images and vertical EPI images, and the SAConv module is used to extract and For spatial and angular features on a plane, the spatial-angular-aware Transformer module is used to obtain the global dependency of features obtained by the EPIConv module and the SAConv module; S3 specifically includes: The local neural geometry learning module consists of a LIGF _h module and a LIGF _V Cascade structure composed of modules; according to the divide-and-conquer learning strategy, the latent geometry is encoded The light field implicit function learning is transformed into the horizontal and vertical two-dimensional light field geometric implicit function cascade learning, which is: According to the horizontal EPI image extraction method, first Decompose into Horizontal Latent Geometry Code , and then use the local implicit image function method to Interpolation to latent feature map ; Finally, and The implicit code is fed into an MLP with 256 hidden layers and ReLU as the activation layer. The local feature map is mapped to the horizontal feature map The corresponding coordinates of the light field are obtained to obtain the complete potential geometric coding in the horizontal direction ; According to the vertical EPI image extraction method, first Decompose into Horizontal Latent Geometry Code , and then use the local implicit image function method to Interpolation to latent feature map ; Finally, and The implicit code is fed into an MLP with 256 hidden layers and ReLU as the activation layer. The local feature map is mapped to the vertical feature map The corresponding coordinates of the light field are obtained to obtain the potential geometric coding in the horizontal direction ; S4 specifically includes: Latent geometry coding in space-angle continuous domain The image is fed into a rendering module consisting of three cascaded 3D convolutional layers, each with a convolution kernel size of 1×1, which gradually compresses the potential geometric code of the light field space-angle continuous domain. The number of feature channels To the target output channel number , and reshape into a light field macro-pixel image ; finally converted into a light field sub-aperture array image, ; The network model in S5 uses the absolute value error between the output view and the ground-truth view as the loss function, specifically including: Output high-altitude angular resolution sub-aperture array image and true high-angle resolution sub-aperture images The loss function between The calculation formula is: ； S6 specifically includes: Consider the light field as a of A subaperture array is obtained by stepping the interval to obtain a subaperture array aligned with the original light field angular resolution, denoted as ;Will The original light field image is evaluated based on the structural similarity index and peak signal-to-noise ratio. The evaluation results are used as the evaluation results of the light field spatial-angular continuous domain super-resolution task test.

2. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 1 is characterized in that The specific steps of the EPIConv module are: According to the horizontal EPI image extraction method, Extract V×Y horizontal EPI feature images and stitch them together to get a dimension of The horizontal pole geometric characteristics of ;Will Input to a convolution layer with a kernel of 3×U, and then pass through a point convolution layer with a kernel of 1×1 to obtain the horizontal EPI feature ; Similarly, according to the vertical EPI image extraction method, from Extract U×X vertical EPI feature images and stitch them together to get a dimension of The vertical polar geometric characteristics are denoted as ;Will Input to a convolution layer with a kernel of 3×V and a point convolution layer with a kernel of 1×1 to extract vertical EPI features ,Will and After splicing in the channel dimension, it is input into a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 to generate EPI features ;at last, is reorganized into dimensions The eigenvector of .

3. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 2 is characterized in that The SAConv module consists of two feature extraction branches and a feature fusion layer. The two feature extraction branches include an upper branch and a lower branch. The upper branch is used to extract spatial features. Input into two convolutional layers with a kernel of 3×3 to obtain the spatial features of the light field image ; The lower branch is used to extract angle features. First, The angle dimension is stacked onto the channel dimension to obtain The size is The feature map is recorded as ; then Input to two convolutional layers with a kernel of 1×1 to generate the angular features of the light field image ; then The reorganized dimension is The feature array, and with Splicing is performed on the channel dimension to obtain composite features , and then generate space-corner features through a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 ;at last, is reorganized into dimensions The null-angle eigenvector of .

4. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 3 is characterized in that The space-angle-aware Transformer module consists of an encoder and encoder Composition, encoder It is a Transformer encoder with self-attention mechanism, which is used to obtain the global dependency of the input feature vector; encoder is a crisscross attention encoder that preserves extremely geometrically relevant null-angle features, specifically: First, in the channel dimension and Splice to get a composite vector As Then, The output of the latent EPI code is re-decomposed into and enhanced null-corner marking ; In the encoder In As the query vector of the cross attention mechanism, As the key vector and value vector of the cross attention mechanism, output potential null-angle encoding ;Will and Stacking in the channel dimension forms the final latent geometric code , the dimension is .

Images