Light field space-angle continuous domain super-resolution method based on local neural geometric learning

Abstract

The invention discloses a light field space-angle continuous domain super-resolution method based on local neural geometric learning, and belongs to the field of deep learning and computer vision, and the method comprises the steps: S1, obtaining a light field polar geometric image, and constructing a neural network model; s2, sending the sub-aperture image array of the light field image into a space-angle sensing geometric coding module to obtain a space-angle sensing potential geometric code; s3, sending the space-angle perceived potential geometric code into a local neural geometry learning module to obtain a space-angle continuous domain potential geometric code; s4, sending the space-angle continuous domain potential geometric code into an expansion rendering module to obtain a light field image; s5, setting a loss function; s6, performing light field-space angle continuous domain super-resolution task testing by using the neural network model; according to the light field space-angle continuous domain super-resolution method based on local neural geometric learning provided by the invention, the super-resolution of any scale at the same time of the space dimension and the angle dimension of the light field image can be realized.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and computer vision technology, and in particular to a light field space-angle continuous domain super-resolution method based on local neural geometry learning. Background Art

[0002] Light field cameras based on microlens arrays record the angle and light radiation information of incident light by inserting a microlens array (MLA) between the image sensor and the main lens, thereby recording the three-dimensional geometric information of the scene in terms of light space and angle. However, due to the limitation of the imaging resolution of the image sensor, there is a trade-off between spatial resolution and angular resolution in the light field imaging process, resulting in the spatial and angular resolutions of light field images being difficult to meet the requirements of practical applications. Therefore, using the sparse and low-resolution sub-aperture image array in the light field image to reconstruct a dense and high-resolution sub-aperture image array, providing high-resolution images from more perspectives for practical light field applications, that is, realizing spatial and angular super-resolution reconstruction of light field images has become an important research task in the field of light field imaging. There are two main limitations of existing light field image super-resolution reconstruction methods: (1) Traditional light field image super-resolution reconstruction methods are based on the light field imaging geometric model, and their performance depends on the accurate estimation of camera internal parameters and scene depth information. In practical applications, camera internal parameters such as focal length will change continuously, and scene depth is difficult to obtain accurately. (2) Existing light field image super-resolution reconstruction methods can only perform super-resolution reconstruction in a single dimension of space or angle, and cannot achieve simultaneous super-resolution reconstruction of space and angle. Moreover, they can only super-resolve light field images to a certain fixed scale, such as obtaining an image with double or quadruple resolution in the spatial dimension, or obtaining a 7×7 or 9×9 sub-aperture image array in the angular dimension. Reconstruction of arbitrary resolution cannot be achieved in the spatial and angular continuous domains. Summary of the invention

[0003] The purpose of the present invention is to provide a light field spatial-angular continuous domain super-resolution method based on local neural geometric learning to solve the problems existing in the above-mentioned background technology.

[0004] To achieve the above object, the present invention provides a light field spatial-angular continuous domain super-resolution method based on local neural geometry learning, which takes a sparse and low-resolution sub-aperture image array as input, feeds it into a neural network model, and draws a sub-aperture image array with arbitrary spatial and angular resolution; the method comprises the following steps:

[0005] S1, obtain the light field extreme geometry image, and construct the light field space-angle continuous domain super-resolution neural network model, including space-angle perception geometry encoding module, local neural geometry learning module and extended rendering module;

[0006] S2, sending the sub-aperture image array of the sparse and low-resolution light field image to the space-angle-aware geometric coding module to obtain the space-angle-aware latent geometric coding;

[0007] S3, sending the space-angle perceived latent geometry code into the local neural geometry learning module to obtain the space-angle continuous domain latent geometry code;

[0008] S4, sending the spatial-angular continuous domain latent geometry code to the extended rendering module to obtain a dense and high-resolution light field image;

[0009] S5. Set the loss function for the neural network model;

[0010] S6. Use the trained neural network model to perform light field-space angle continuous domain super-resolution task test on the test dataset.

[0011] Preferably, S2 is to first input the sparse and low-resolution light field image into a convolution layer with a convolution kernel of 3×3 to obtain an initial feature map array F with dimensions (U, V, X, Yv, C) init , and then input it into the space-angle-aware geometric coding module to obtain the space-angle-aware latent geometric coding. The space-angle-aware geometric coding module consists of an EPIConv module, a SAConv module and a space-angle-aware Transformer module. For the light field image L(u,v,x,y), the EPIConv module is used to extract the EPI geometric features in the horizontal EPI image and the vertical EPI image, the SAConv module is used to extract the spatial and angular features on the (x,y) and (u,v) planes, and the space-angle-aware Transformer module is used to obtain the global dependency of the features obtained by the EPIConv module and the SAConv module.

[0012] Preferably, the specific steps of the EPIConv module are:

[0013] According to the horizontal EPI image extraction method, from F init Extract V×Y horizontal EPI feature images from the image and concatenate them to obtain horizontal polar geometric features with dimensions (VY,U,X,C), denoted as F init_h ; F init_h Input to a convolution layer with a kernel of 3×U, and then pass through a point convolution layer with a kernel of 1×1 to obtain the horizontal EPI feature F epi_h Similarly, according to the vertical EPI image extraction method, from F init Extract U×X vertical EPI feature images from the image and concatenate them to obtain vertical polar geometric features with dimensions (UX, V, Y, C), denoted as F init_v ; F init_vInput to a convolution layer with a kernel of 3×V and a point convolution layer with a kernel of 1×1 to extract the vertical EPI feature F epi_v , F epi_h and F epi_v After concatenation in the channel dimension, it is input into a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 to generate the EPI feature F epi ; Finally, F epi is reorganized into a feature vector T of dimension (VY,UX,C / 2) epi .

[0014] Preferably, the SAConv module consists of two feature extraction branches and a feature fusion layer. The two feature extraction branches include an upper branch and a lower branch. The upper branch is used to extract spatial features. init Input to two convolutional layers with a kernel of 3×3 to obtain the spatial features F of the light field image spa ; The lower branch is used to extract angle features. First, F init The angle dimension is stacked into the channel dimension to obtain C×U×V feature maps of size (X,Y), denoted as F init_ang ; then F init_ang Input to two convolutional layers with a kernel of 1×1 to generate the angle feature F of the light field image ang ; then F ang The reorganized feature array is U×V×X×Y×C and combined with F spa Concatenate in the channel dimension to obtain the composite feature F spa_ang , and then generate the space-corner feature F through a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 sa ;Finally, similar to the EPIConv module, F sa is reorganized into a null-angle eigenvector T of dimension (VY,UX,C / 2) sa .

[0015] Preferably, the space-angle-aware Transformer module consists of an encoder E s and encoder E c Composition, encoder E s is a standard Transformer encoder with a self-attention mechanism, which is used to obtain the global dependencies of the input feature vectors; encoder E c is a cross-attention encoder that retains extremely geometrically relevant null-corner features while ignoring irrelevant detail features. Specifically:

[0016] First, T is transformed into epi and T sa Splice to get the composite vector T epi_sa As Es Input; then E s The output is re-decomposed into the potential EPI code Z epi and enhanced null-corner marker T' sa ; In encoder E c In the epi As the query vector of the cross attention mechanism, T' sa As the key vector and value vector of the cross attention mechanism, the output is the latent space-angle code Z with geometric meaning sa ; Z epi and Z sa Stacking in the channel dimension forms the final latent geometric code Z g , the dimensions are (VY,UX,C).

[0017] Preferably, S3 specifically includes:

[0018] The local neural geometry learning module consists of a LIGF _h Module and a LIGF _V The cascade structure consists of modules; according to the divide-and-conquer learning strategy, the latent geometric code Z g The light field implicit function learning is transformed into the horizontal and vertical two-dimensional light field geometric implicit function cascade learning, which is:

[0019] According to the horizontal EPI image extraction method, firstly Z g Decomposed into V×Y horizontal latent geometric codes Then the local implicit image function (LIIF) method is used to convert Z h Interpolation to latent feature map Finally, Z l and Z h The implicit code Z is fed into an MLP with 256 hidden layers and ReLU as the activation layer. l The local feature map is mapped to the horizontal feature map Z h The corresponding coordinates of the light field are obtained to obtain the complete potential geometric encoding in the horizontal direction.

[0020] According to the vertical EPI image extraction method, Z' is first decomposed into U'×X' horizontal latent geometric codes. Then, the local implicit image function method is used to convert Z V Interpolation to latent feature map Finally, Z l and Z v The implicit code Z is fed into an MLP with 256 hidden layers and ReLU as the activation layer. l The local feature map is mapped to the vertical feature map Z vThe corresponding coordinates of the light field are obtained to obtain the potential geometric coding in the horizontal direction.

[0021] Preferably, S4 specifically includes:

[0022] Latent geometry coding Z in space-angle continuous domain C It is sent to the rendering module composed of three cascaded three-dimensional convolutional layers. The convolution kernel size of each layer is 1×1, which gradually compresses the potential geometric code Z of the light field space-angle continuous domain C The number of feature channels C is converted to the number of target output channels c, and reshaped into a light field macro-pixel image. Finally, it is converted into a light field subaperture array image with high spatial-angular resolution, which is

[0023] Preferably, the network model in S5 uses the absolute value error (L1) between the output view and the GT view as the loss function, specifically including:

[0024] Output high-altitude angular resolution sub-aperture array image and true high-angle resolution sub-aperture images The calculation formula of the loss function Loss between is:

[0025]

[0026] Preferably, S6 specifically includes:

[0027] The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning can reconstruct a light field with M times higher spatial resolution and N×N higher angular resolution, which can be regarded as an N×N M times sub-aperture array. Therefore, as long as the downsampling rate of the input is controlled, the output can be aligned with the true label in the spatial domain after super-resolution. At the same time, from this array, a sub-aperture array aligned with the angular resolution of the original light field can be obtained through a certain step interval, which is recorded as Will The original light field image is evaluated based on the structural similarity index (SSIM) and peak signal to noise ratio (PSNR). The evaluation results are used as the evaluation results of the light field spatial-angular continuous domain super-resolution task test.

[0028] Therefore, the present invention adopts the above-mentioned light field space-angle continuous domain super-resolution method based on local neural geometry learning, which has the following beneficial effects:

[0029] (1) Provide a light field spatial-angular continuous domain super-resolution method based on local neural geometric learning, which can achieve super-resolution of light field images in both spatial and angular dimensions at any scale;

[0030] (2) We propose a spatial-angle-consistent local neural geometry learning framework for simultaneous super-resolution in the spatial-angular continuum domain by mapping the light field polar geometry image into an interpolable latent space to learn spatial and angular information.

[0031] (3) A space-angle-aware geometry encoder is proposed to extract the latent geometry code of the light field polar geometry, integrate the local and global dependencies of the light field polar geometry, and embed the light field space-angle correlation into the latent geometry code through a space-angle-aware cross-attention mechanism;

[0032] (4) A divide-and-conquer local neural geometric learning strategy is used to effectively reduce memory usage by converting the learning of four-dimensional light field implicit functions into the cascade learning of two two-dimensional light field extreme geometric implicit functions with shared weights.

[0033] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a diagram of a light field spatial-angular continuous domain super-resolution neural network structure based on local neural geometry learning in an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the EPIConv module structure of the present invention;

[0036] Figure 3 Schematic diagram of the SAConv module structure of the present invention;

[0037] Figure 4 This is a schematic diagram of the structure of the space-angle sensing Transformer module of the present invention;

[0038] Figure 5 Schematic diagram of the effect of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The dual-plane representation of the light field image is denoted as L(u,v,x,y), where (u,v) is the angular coordinate of the light field image, (x,y) is the spatial coordinate of the light field image, where y∈[1,U], v∈[1,V], x∈[0,X], y∈[0,Y]. L(u,v)(x,y) represents the sub-aperture image (SAI) at a given (u,v) angular coordinate. The light field image can be viewed as a set of sub-aperture image arrays.

[0041] The light field EPI image (epipolar plane image) is obtained by stacking pixels in a row (or column) of the sub-aperture image in the same row (or column) of the light field sub-aperture image array: given the v and y coordinates in the light field image, a horizontal EPI image L(v,y)(u,x) can be obtained. Given the u and x coordinates in the light field image, a vertical EPI image L(u,x)(v,y) can be obtained. A light field image with an angular resolution of U×V and a spatial resolution of X×Y can obtain V×Y horizontal EPI images and U×X vertical EPI images.

[0042] See also Figure 1 , a light field spatial-angular continuous domain super-resolution method based on local neural geometric learning, comprising the following steps:

[0043] S1. Obtain a horizontal EPI image (or vertical EPI image) of a light field polar geometry image by stacking pixels of a row (or column) of a sub-aperture image in a row (or column) of a light field sub-aperture image array; then construct a light field space-angle continuous domain super-resolution neural network model, which mainly includes a space-angle perception geometry encoding module, a local neural geometry learning module, and an extended rendering module;

[0044] S2, the sparse and low-resolution light field image is first input into a convolution layer with a convolution kernel of 3×3 to obtain an initial feature map array F with dimensions (U, V, X, Y, C) init , and then input it into the space-angle-aware geometric coding module to obtain the space-angle-aware latent geometric coding. The space-angle-aware geometric coding module consists of three modules: an EPIConv module, a SAConv module and a space-angle-aware Transformer module; for the light field image L(u,v,x,y), the EPIConv module is used to extract EPI geometric features in horizontal EPI images and vertical EPI images, the SAConv module is used to extract spatial and angular features on the (x,y) and (u,v) planes, and the space-angle-aware Transformer module is used to obtain the global dependency of the features obtained by the EPIConv module and the SAConv module.

[0045] EPIConv modules, such as Figure 2 As shown, according to the horizontal EPI image extraction method, from F init Extract V×Y horizontal EPI feature images from the image and concatenate them to obtain horizontal polar geometric features with dimensions (VY,U,X,C), denoted as F init_h ; F init_h Input to a convolution layer with a kernel of 3×U, and then pass through a point convolution layer with a kernel of 1×1 to obtain the horizontal EPI feature F epi_h Similarly, according to the vertical EPI image extraction method, from F init Extract U×X vertical EPI feature images from the image and concatenate them to obtain vertical polar geometric features with dimensions (UX, V, Y, C), denoted as F init_v ; F init_v Input to a convolution layer with a kernel of 3×V and a point convolution layer with a kernel of 1×1 to extract the vertical EPI feature F epi_v , F epi_h and F epi_v After concatenation in the channel dimension, it is input into a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 to generate the EPI feature F epi ; Finally, F epi is reorganized into a feature vector F of dimension (VY,UX,C / 2) epi .

[0046] SAConv module, such as Figure 3 As shown, it is used to extract and fuse the spatial and angular features of the light field. It consists of two feature extraction branches and a feature fusion layer. The two feature extraction branches include an upper branch and a lower branch. The upper branch is used to extract spatial features. init Input to two convolutional layers with a kernel of 3×3 to obtain the spatial features F of the light field image spa ; The lower branch is used to extract angle features. First, F init The angle dimension is stacked into the channel dimension to obtain C×U×V feature maps of size (X,Y), denoted as F init_ang ; then F init_ang Input to two convolutional layers with a kernel of 1×1 to generate the angle feature F of the light field image ang ; then F ang The reorganized feature array is U×V×X×Y×C and combined with F spa Concatenate in the channel dimension to obtain the composite feature F spa_ang , and then generate the space-corner feature F through a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 sa ;Finally, similar to the EPIConv module, F sa is reorganized into a null-angle eigenvector T of dimension (VY,UX,C / 2) sa.

[0047] The spatial-angular-aware Transformer module is used to capture the global dependencies of light field spatial, angular, and polar geometric features, such as Figure 4 As shown, by the encoder E s and encoder E c Composition, encoder E s is a standard Transformer encoder with a self-attention mechanism, which is used to obtain the global dependencies of the input feature vectors; encoder E c is a cross-attention encoder that retains extremely geometrically relevant null-corner features while ignoring irrelevant detail features. Specifically:

[0048] First, T is transformed into epi and T sa Splice to get the composite vector T epi_sa As E s Input; then E s The output is re-decomposed into the potential EPI code Z epi and enhanced null-corner marker T' sa ; In encoder E c In the epi As the query vector of the cross attention mechanism, T' sa As the key vector and value vector of the cross attention mechanism, the output is the latent space-angle code Z with geometric meaning sa ; Z epi and Z sa Stacking in the channel dimension forms the final latent geometric code Z g , the dimensions are (VY,UX,C).

[0049] S3, sending the space-angle perceived latent geometry code to the local neural geometry learning module to obtain the space-angle continuous domain latent geometry code; specifically:

[0050] The local neural geometry learning module consists of a LIGF _h Module and a LIGF _V The cascade structure consists of modules; according to the divide-and-conquer learning strategy, the latent geometric code Z g The light field implicit function learning is transformed into the horizontal and vertical two-dimensional light field geometric implicit function cascade learning, which is:

[0051] According to the horizontal EPI image extraction method, firstly Z g Decomposed into V×Y horizontal latent geometric codes Then the local implicit image function (LIIF) method is used to convert Z h Interpolation to latent feature map Finally, Zl and Z h The implicit code Z is fed into an MLP with 256 hidden layers and ReLU as the activation layer. l The local feature map is mapped to the horizontal feature map Z h The corresponding coordinates of the light field are obtained to obtain the complete potential geometric encoding in the horizontal direction.

[0052] According to the vertical EPI image extraction method, Z' is first decomposed into U'×X' horizontal latent geometric codes. Then, the local implicit image function method is used to convert Z V Interpolation to latent feature map Finally, Z l and Z v The implicit code Z is fed into an MLP with 256 hidden layers and ReLU as the activation layer. l The local feature map is mapped to the vertical feature map Z v The corresponding coordinates of the light field are obtained to obtain the potential geometric coding in the horizontal direction.

[0053] S4, sending the spatial-angular continuous domain latent geometry code to the extended rendering module to obtain a dense and high-resolution light field image; specifically:

[0054] Latent geometry coding Z in space-angle continuous domain C It is sent to the rendering module composed of three cascaded three-dimensional convolutional layers. The convolution kernel size of each layer is 1×1, which gradually compresses the potential geometric code Z of the light field space-angle continuous domain C The number of feature channels C is converted to the number of target output channels c, and reshaped into a light field macro-pixel image. Finally, it is converted into a light field subaperture array image with high spatial-angular resolution, which is

[0055] S5. Build a network model and set the loss function; specifically:

[0056] In this embodiment, the network model uses the absolute value error (L1) between the output view and the GT view as the loss function;

[0057] Output high-altitude angular resolution sub-aperture array image and true high-angle resolution sub-aperture images The calculation formula of the loss function Loss between is:

[0058]

[0059] S6. Using the trained network model, perform the light field-space angle continuous domain super-resolution task test on the test dataset, specifically:

[0060] The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning can reconstruct a light field with M times higher spatial resolution and N×N higher angular resolution, which can be regarded as an N×N M times sub-aperture array. Therefore, as long as the downsampling rate of the input is controlled, the output can be aligned with the true label in the spatial domain after super-resolution. At the same time, from this array, a sub-aperture array aligned with the angular resolution of the original light field can be obtained through a certain step interval, which is recorded as Will The original light field image is evaluated based on the structural similarity index (SSIM) and peak signal to noise ratio (PSNR). The evaluation results are used as the evaluation results of the light field spatial-angular continuous domain super-resolution task test.

[0061] In the light field space-angle continuous domain super-resolution task with an angle domain of 2×2 to 5×5 and a spatial domain of 2×, the comparison of the indicators of the method of this embodiment with other methods is shown in Table 1:

[0062] Table 1 Comparison of indicators of different methods

[0063]

[0064] Since other methods cannot achieve simultaneous spatial and angular super-resolution, they are first sent to the angular super-resolution module and then to the spatial super-resolution module for comparison with this method. It can be seen that this method has good performance in multiple data sets, and the actual results are as follows: Figure 5 shown.

[0065] Therefore, the present invention adopts the above-mentioned light field space-angle continuous domain super-resolution method based on local neural geometry learning. First, by stacking a row (or column) of pixels in a sub-aperture image of a row (or column) in the light field sub-aperture image array, a horizontal EPI image (or vertical EPI image) of the light field extreme geometry image is obtained; then a light field space-angle continuous domain super-resolution neural network model is constructed, which mainly includes a space-angle perception geometry encoding module, a local neural geometry learning module, and an extended rendering module. Among them, the space-angle perception geometry encoding module obtains the local and global dependencies of the light field geometry from the horizontal EPI image (or vertical EPI image) through a convolutional neural network and a self-attention mechanism, and integrates the correlation between the light field space and angle through a cross-attention mechanism; the local neural geometry learning module transforms the four-dimensional implicit function learning into the cascade learning of two two-dimensional local implicit functions through the idea of ​​divide and conquer; the extended rendering module is used to generate a light field macro-pixel image with space-angle high resolution.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A light field spatial-angular continuous domain super-resolution method based on local neural geometric learning, characterized in that: The following steps are involved: S1, obtain the light field extreme geometry image, and construct the light field space-angle continuous domain super-resolution neural network model, including space-angle perception geometry encoding module, local neural geometry learning module and extended rendering module; S2, sending the sub-aperture image array of the sparse and low-resolution light field image to the space-angle-aware geometric coding module to obtain the space-angle-aware latent geometric coding; S3, sending the space-angle perceived latent geometry code into the local neural geometry learning module to obtain the space-angle continuous domain latent geometry code; S4, sending the spatial-angular continuous domain latent geometry code to the extended rendering module to obtain a dense and high-resolution light field image; S5. Set the loss function for the neural network model; S6. Use the trained neural network model to perform light field-space angle continuous domain super-resolution task test on the test dataset.

2. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 1 is characterized in that: S2 is to input the sparse and low-resolution light field image into a convolution layer with a convolution kernel of 3×3 to obtain an initial feature map array F with dimensions (U, V, X, Y, C) init , and then input it into the space-angle-aware geometric coding module to obtain the space-angle-aware latent geometric coding. The space-angle-aware geometric coding module consists of an EPIConv module, a SAConv module and a space-angle-aware Transformer module. For the light field image L(u,v,x,y), the EPIConv module is used to extract the EPI geometric features in the horizontal EPI image and the vertical EPI image, the SAConv module is used to extract the spatial and angular features on the (x,y) and (u,v) planes, and the space-angle-aware Transformer module is used to obtain the global dependency of the features obtained by the EPIConv module and the SAConv module.

3. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 2 is characterized in that: The specific steps of the EPIConv module are: According to the horizontal EPI image extraction method, from F init Extract V×Y horizontal EPI feature images from the image and concatenate them to obtain horizontal polar geometric features with dimensions (VY,U,X,C), denoted as F init_h ; F init_h Input to a convolution layer with a kernel of 3×U, and then pass through a point convolution layer with a kernel of 1×1 to obtain the horizontal EPI feature F epi_h Similarly, according to the vertical EPI image extraction method, from F init Extract U×X vertical EPI feature images from the image and concatenate them to obtain vertical polar geometric features with dimensions (UX, V, Y, C), denoted as F init_v ; F init_v Input to a convolution layer with a kernel of 3×V and a point convolution layer with a kernel of 1×1 to extract the vertical EPI feature F epi_v , F epi_h and F epi_v After concatenation in the channel dimension, it is input into a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 to generate the EPI feature F epi ; Finally, F epi is reorganized into a feature vector T of dimension (VY,UX,C / 2) epi .

4. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 3 is characterized in that: The SAConv module consists of two feature extraction branches and a feature fusion layer. The two feature extraction branches include an upper branch and a lower branch. The upper branch is used to extract spatial features. init Input to two convolutional layers with a kernel of 3×3 to obtain the spatial features F of the light field image spa ; The lower branch is used to extract angle features. First, F init The angle dimension is stacked into the channel dimension to obtain C×U×V feature maps of size (X,Y), denoted as F init_ang ; then F init_ang Input to two convolutional layers with a kernel of 1×1 to generate the angle feature F of the light field image ang ; then F ang The reorganized feature array is U×B×X×Y×C and combined with F spa Concatenate in the channel dimension to obtain the composite feature F spa_ang , and then generate the space-corner feature F through a point convolution layer with a kernel of 1×1 and a convolution layer with a kernel of 3×3 sa ; Finally, F sa is reorganized into a null-angle eigenvector T of dimension (VY,UX,C / 2) sa .

5. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 4 is characterized in that: The space-angle-aware Transformer module consists of an encoder E s and encoder E c Composition, encoder E s is a Transformer encoder with a self-attention mechanism, which is used to obtain the global dependency of the input feature vector; encoder E c is a cross-attention encoder that preserves extremely geometrically related null-corner features, specifically: First, T is transformed into epi and T sa Splice to get the composite vector T epi_sa As E s Then E s The output is re-decomposed into the potential EPI code Z epi and enhanced null-corner marker T' sa ; In encoder E c In the epi As the query vector of the cross attention mechanism, T' sa As the key vector and value vector of the cross attention mechanism, the output potential space-angle code Z sa ; Z epi and Z sa Stacking in the channel dimension forms the final latent geometric code Z g , the dimensions are (VY,UX,C).

6. The light field spatial-angular continuous domain super-resolution method based on local neural geometry learning according to claim 5, characterized in that: S3 specifically includes: The local neural geometry learning module consists of a LIGF _h Module and a LIGF _V The cascade structure consists of modules; according to the divide-and-conquer learning strategy, the latent geometric code Z g The light field implicit function learning is transformed into the horizontal and vertical two-dimensional light field geometric implicit function cascade learning, which is: According to the horizontal EPI image extraction method, firstly Z g Decomposed into B×Y horizontal latent geometric codes Then, the local implicit image function method is used to convert Z h Interpolation to latent feature map Finally, Z l and Z h The implicit code Z is fed into an MLP with 256 hidden layers and ReLU as the activation layer. l The local feature map is mapped to the horizontal feature map Z h The corresponding coordinates of the light field are obtained to obtain the complete potential geometric encoding in the horizontal direction. According to the vertical EPI image extraction method, Z' is first decomposed into U'×X' horizontal latent geometric codes. Then, the local implicit image function method is used to convert Z V Interpolation to latent feature map Finally, Z l and Z v The implicit code Z is fed into an MLP with 256 hidden layers and ReLU as the activation layer. l The local feature map is mapped to the vertical feature map Z v The corresponding coordinates of the light field are obtained to obtain the potential geometric coding in the horizontal direction.

7. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 6, characterized in that: S4 specifically includes: Latent geometry coding Z in space-angle continuous domain C It is sent to the rendering module composed of three cascaded three-dimensional convolutional layers. The convolution kernel size of each layer is 1×1, which gradually compresses the potential geometric code Z of the light field space-angle continuous domain C The number of feature channels C is converted to the number of target output channels c, and reshaped into a light field macro-pixel image. Finally, it is converted into a light field sub-aperture array image, which is 8. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 1, characterized in that: The network model in S5 uses the absolute value error between the output view and the GT view as the loss function, which includes: Output high-altitude angular resolution sub-aperture array image and true high-angle resolution sub-aperture images The calculation formula of the loss function Loss between is:

9. The light field spatial-angular continuous domain super-resolution method based on local neural geometric learning according to claim 1, characterized in that: S6 specifically includes: The light field is regarded as an N×N array of M times sub-apertures. In this array, a sub-aperture array aligned with the angular resolution of the original light field is obtained through the step interval, which is denoted as Will The original light field image is evaluated on the basis of the structural similarity index and peak signal-to-noise ratio. The evaluation results are used as the evaluation results of the light field spatial-angular continuous domain super-resolution task test.

Images