Light field image enhancement method and device, electronic equipment and readable storage medium

Through the fusion of global and spatial features of the image enhancement neural network model, the problem of low-illumination light field image quality is solved, and more comprehensive information representation and quality improvement is achieved.

CN118941460APending Publication Date: 2024-11-12CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410986272.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The light field image quality is low under low illumination conditions, and the existing enhancement methods have problems such as color distortion and missing image information.

Method used

Using the image enhancement neural network model, the first feature of global and local features is extracted through the first module, and the second module extracts the second feature of spatial features, and fuses and reconstructions to obtain the enhanced light field image.

Benefits of technology

The quality of the light field image is improved, making its information representation more comprehensive, and the enhanced image quality is lower in the light field image under the conditions of illumination.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941460A_ABST
    Figure CN118941460A_ABST
Patent Text Reader

Abstract

The invention discloses a light field image enhancement method and device, electronic equipment and a readable storage medium, and belongs to the technical field of computer vision. The light field image enhancement method comprises the steps that a first module is controlled to perform feature extraction on to-be-processed light field image data to obtain a first feature, and the first feature is used for indicating a global feature and a local feature of a to-be-processed light field image; controlling a second module to perform feature extraction on the to-be-processed light field image data to obtain a second feature which is used for indicating the spatial feature of the to-be-processed light field image; and inputting the first feature and the second feature into a reconstruction module, and controlling the reconstruction module to perform feature fusion and light field reconstruction on the first feature and the second feature to obtain an enhanced light field image. According to the light field image enhancement method provided by the invention, the feature fused by the first feature and the second feature can more comprehensively represent the information of the light field image, and the quality of the light field image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and specifically relates to a light field image enhancement method, device, electronic device and readable storage medium. Background Art

[0002] Light field images can record rich structural and texture information of scenes and are widely used in a variety of computer vision tasks. Low illumination conditions will affect the quality of light field images, thereby reducing the performance of computer vision systems.

[0003] For low-light light field images, related enhancement methods include, for example, using a three-level recurrent neural network to enhance each light field sub-viewpoint image; designing three inputs consisting of stacked views, using a multi-stream interaction module to aggregate features from different restoration streams, and entering a multi-stage restoration network to reconstruct the light field after the fusion. Although the above enhancement methods can achieve certain denoising and brightness restoration effects, the enhanced light field images still have problems such as color distortion and missing image information, and the quality of the enhanced light field images is low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a light field image enhancement method, device, electronic device and readable storage medium, which can solve the problem of low light field image quality under low illumination conditions.

[0005] In a first aspect, an embodiment of the present application provides a light field image enhancement method applied to an image enhancement neural network model, wherein the image enhancement neural network model includes a first module, a second module, and a reconstruction module, wherein the first module and the second module are respectively connected to the reconstruction module, and the method includes:

[0006] Controlling the first module to perform feature extraction on the light field image data to be processed to obtain a first feature, wherein the first feature is used to indicate a global feature and a local feature of the light field image to be processed;

[0007] Controlling the second module to perform feature extraction on the light field image data to be processed to obtain a second feature, wherein the second feature is used to indicate a spatial feature of the light field image to be processed;

[0008] The first feature and the second feature are input into the reconstruction module, and the reconstruction module is controlled to perform feature fusion and light field reconstruction on the first feature and the second feature to obtain an enhanced light field image.

[0009] In a second aspect, an embodiment of the present application provides a light field image enhancement device, which is applied to an image enhancement neural network model, wherein the image enhancement neural network model includes a first module, a second module and a reconstruction module, wherein the first module and the second module are respectively connected to the reconstruction module, and the device includes:

[0010] A first control module, used for controlling the first module to perform feature extraction on the light field image data to be processed to obtain a first feature, wherein the first feature is used to indicate a global feature and a local feature of the light field image to be processed;

[0011] A second control module, used for controlling the second module to perform feature extraction on the light field image data to be processed to obtain a second feature, wherein the second feature is used to indicate a spatial feature of the light field image to be processed;

[0012] The third control module is used to input the first feature and the second feature into the reconstruction module, and control the reconstruction module to perform feature fusion and light field reconstruction on the first feature and the second feature to obtain an enhanced light field image.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the light field image enhancement method as described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the light field image enhancement method as described in the first aspect are implemented.

[0015] According to a fifth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the light field image enhancement method according to the first aspect.

[0016] In the embodiment of the present application, the first module extracts the first feature used to represent the global features and local features of the light field image data to be processed, and the second module extracts the second feature used to represent the spatial features of the light field image data to be processed. After the first feature and the second feature are fused, the light field is reconstructed based on the fused features to obtain an enhanced light field image. Compared with the prior art, the features after the fusion of the first feature and the second feature can more comprehensively represent the information of the light field image and improve the quality of the light field image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A schematic diagram of a light field image enhancement method according to an embodiment of the present application;

[0019] Figure 2 A schematic diagram of the process of performing light field image enhancement using an image enhancement neural network model;

[0020] Figure 3 A schematic diagram of the structure of a light field image enhancement device provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0023] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] The light field image enhancement method provided by the embodiment of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0025] like Figure 1 As shown, Figure 1A flow chart of a light field image enhancement method provided in an embodiment of the present application is provided. The light field image enhancement method of this embodiment is applied to an image enhancement neural network model. The image enhancement neural network model includes a first module, a second module and a reconstruction module. The first module and the second module are respectively connected to the reconstruction module. Specifically, the following steps are included:

[0026] Step 101: Control the first module to perform feature extraction on the light field image data to be processed to obtain a first feature, wherein the first feature is used to indicate the global feature and local feature of the light field image to be processed.

[0027] A light field image (LFI) is an image that can capture and display a collection of light rays in a scene. It not only contains the planar information of traditional 2D images, but also records the depth and perspective information of the scene. Light field images are generally captured by a light field camera, which can capture the propagation path of light in space, including the direction and position information of the light.

[0028] The light field image data to be processed in this embodiment is in the form of four-dimensional (4D) data of (C, U*V, H, W). Among them, C represents the number of channels, C is generally 3, representing RGB channels; U*V represents angular resolution, and H and W represent spatial resolution. It can be understood that a light field camera is used to shoot the target object to obtain a light field image to be processed of the target object. The light field image to be processed includes multiple sub-viewpoint images, and each sub-viewpoint image corresponds to a different shooting angle. For example, the light field camera shoots the target object from the left and right sides of the target object respectively, and two sub-viewpoint images can be obtained. The contents of the two sub-viewpoint images are different, but they both belong to the image of the target object and are part of the light field image to be processed. In the light field image data to be processed, U*V represents the angular resolution, such as (5*5), which means that the number of sub-viewpoint images of the image to be processed represented by the image data to be processed is 25, and each sub-viewpoint image corresponds to a shooting angle. H and W represent the length and width of each sub-viewpoint image. For example, (H, W) is (128, 128), which means that the length and width of the image are both 128 pixels.

[0029] In this step, the first module is used to extract global features and local features of the light field image data to be processed. Global features include color features, texture features, shape features, etc. Local features are features extracted from local areas of the image, which can be edges, corners, lines, curves, and regions of feature attributes.

[0030] Step 102: Control the second module to perform feature extraction on the light field image data to be processed to obtain a second feature, wherein the second feature is used to indicate a spatial feature of the light field image to be processed.

[0031] Please see further Figure 2 , Figure 2 The figure is a flow chart of light field image enhancement using an image enhancement neural network model. Figure 2 In the example shown, the first module is a global-local feature extraction module (GLFEM), and the second module is an attention module. The first module and the second module are connected in parallel. The light field image data to be processed is Figure 2 Light field image shown.

[0032] In an optional embodiment, partial image data located in the central area is selected from the light field image data to be processed, and then the second module is controlled to perform feature extraction on the partial image data located in the central area to obtain a second feature representing the spatial feature, such as Figure 2 As shown, the light field image data to be processed (light field image) includes 9 sub-viewpoint images, from which partial image data located in the central area (including the light field sub-views of 5 sub-viewpoint images) are selected and input into the second module (attention module) to extract the second feature. It should be noted that the second feature includes at least an angle feature.

[0033] Step 103: input the first feature and the second feature into the reconstruction module, and control the reconstruction module to perform feature fusion and light field reconstruction on the first feature and the second feature to obtain an enhanced light field image.

[0034] In the light field image enhancement method provided in the embodiment of the present application, a first module is used to extract a first feature for representing the global features and local features of the light field image data to be processed, and a second module is used to extract a second feature for representing the spatial features of the light field image data to be processed. After the first feature and the second feature are fused, the light field is reconstructed based on the fused feature to obtain an enhanced light field image. Compared with the prior art, the feature after the fusion of the first feature and the second feature can more comprehensively represent the information of the light field image and improve the quality of the light field image.

[0035] Optionally, the first module includes a residual block, the residual block includes a quantum convolution layer and a 2D convolution layer, and the controlling the first module to perform feature extraction on the processed light field image data to obtain the first feature includes:

[0036] Acquire the light field image data to be processed, and obtain a light field image matrix to be processed according to the light field image data to be processed;

[0037] Controlling the quantum convolution layer to perform quantum convolution calculation on the light field image data to be processed to obtain a quantum convolution matrix;

[0038] Controlling the 2D convolution layer to perform 2D convolution calculation on the quantum convolution matrix to obtain a 2D convolution matrix;

[0039] A sum of the light field image matrix to be processed and the 2D convolution matrix is ​​calculated, and the sum of the light field image matrix to be processed and the 2D convolution matrix is ​​determined as the first feature.

[0040] In this embodiment, the first module includes a residual block, which includes a quantum convolution layer (i.e. Figure 2 Quantum Conv shown) and 2D convolutional layer (i.e. Figure 2 The convolution kernel size of the quantum convolution layer and the 2D convolution layer can be 3×3.

[0041] The quantum convolution layer encodes the light field image data to be processed into the quantum state of qubit, and then performs gate operations through quantum gates with trainable weights. The evolved quantum state is measured to derive the classical output. The above process can be called quantum convolution calculation, and the obtained data can be called quantum convolution matrix, which is expressed by the following formula:

[0042] F qcnn (I)

[0043] Wherein, I represents the light field image data to be processed.

[0044] The 2D convolution layer is used to perform 2D convolution calculation on the quantum convolution matrix obtained after the above quantum convolution calculation to obtain a 2D convolution matrix, which is expressed by the following formula:

[0045] F conv2d (F qcnn (I))

[0046] Then the sum of the light field image matrix to be processed and the 2D convolution matrix is ​​determined as the first feature:

[0047] I gl =F conv2d (F qcnn (I))+I

[0048] In this embodiment, the residual block combines classical convolution and quantum convolution. Since the quantum convolution layer can better extract global features and significantly improve the computing efficiency when processing large-scale, high-dimensional data, while retaining the advantages of the 2D convolution layer such as nonlinearity, locality, and scalability to deep structures, the quantum computing characteristics of quantum superposition and entanglement, high parallelism, etc. are utilized to enhance the feature extraction capability and computing efficiency of the first module, which can improve the quality and generation speed of light field images.

[0049] Optionally, the first module includes N residual blocks, where N is an integer greater than 1;

[0050] When N is 2, the input data of the first residual block is the light field image data to be processed, the input data of the second residual block is the sum of the output data of the first residual block and the light field image data to be processed, and the first feature is the sum of the output data of the second residual block and the output data of the first residual block;

[0051] When N is greater than 2, the input data of the first residual block is the light field image data to be processed, the input data of the second residual block is the sum of the output data of the first residual block and the light field image data to be processed, the input data of the i-th residual block is the sum of the output data of the i-1-th residual block and the output data of the i-2-th residual block, and the first feature is the sum of the output data of the N-th residual block and the output data of the N-1-th residual block, where i ranges from 2 to N.

[0052] In this embodiment, if Figure 2 As shown, the first module (global-local feature extraction module) includes multiple residual blocks, and the multiple residual blocks use residual connection, that is, the input data of a residual block is added to the output data of the residual block as the input data of the next residual block. In this way, the gradient vanishing problem caused by the increase in the number of residual blocks can be solved, and the accuracy and efficiency of information transmission can be improved.

[0053] In an optional embodiment, the first module further includes a first 2D convolutional layer, such as Figure 2 As shown, in the first module (global-local feature extraction module), after the square image data to be processed enters the first module, the light field image data to be processed is transformed into (B*U*V, C, H, W) through a reshape operation by the first 2D convolution layer for 2D convolution calculation, where B is the batch size. After being processed by the first 2D convolution layer, multiple residual blocks are used for feature extraction. In this embodiment, the first 2D convolution layer is used to adjust the shape of the light field image data to be processed, so as to facilitate feature extraction by the residual block and improve the accuracy and efficiency of feature extraction.

[0054] Optionally, the second module includes at least two three-dimensional (3D) dilated convolution layers, each of the 3D dilated convolution layers corresponds to a different dilation factor, and the controlling the second module to perform feature extraction on the light field image data to be processed to obtain a second feature includes:

[0055] Acquire first central viewpoint image data of the light field image data to be processed, wherein the first central viewpoint image data is sub-viewpoint image data located at a central position in the light field image data to be processed;

[0056] Based on the first central viewpoint image data, determining target viewpoint image data from the light field image data to be processed, wherein the target viewpoint image data includes: image data corresponding to sub-viewpoints adjacent to the first central viewpoint image data and located in upper, lower, left, and right directions;

[0057] Controlling the at least two 3D dilated convolution layers to perform 3D dilated convolution calculations on the first central viewpoint image data and the target viewpoint image data, respectively, to obtain at least two second sub-features, wherein the at least two second sub-features correspond one-to-one to the at least two 3D dilated convolution layers;

[0058] The at least two second sub-features are subjected to feature fusion to obtain the second feature.

[0059] In this embodiment, if Figure 2 As shown in FIG. 1 , the second module (attention module) includes three 3D dilated convolutional layers and one 3D convolutional layer. The three 3D dilated convolutional layers are connected in parallel. The convolution kernel size of the 3D dilated convolutional layer can be 3×3×3, and the corresponding dilation factors are 1, 2, and 3. Different dilation factors can extract the spatial and angular features of the light field image to be processed under different receptive fields, namely, the three second sub-features: F dconv1 (I) F dconv2 (I) F dconv3 (I), where F dconv () indicates 3D dilated convolution calculation. Afterwards, the three second sub-features are connected in the channel dimension and enter the 3D convolution layer for feature fusion to obtain the second feature:

[0060] F conv3d (F dconv1 (I), F dconv1 (I), F dconv1 (I))

[0061] Among them, F conv3d () indicates 3D convolution calculation.

[0062] It should be noted that, in the process of feature extraction in the second module, it is necessary to first extract some data from the light field image data to be processed, such as Figure 2 As shown, the light field image data (light field image) to be processed includes 9 sub-viewpoint images, the first central viewpoint image is the image located at the center, and the adjacent sub-viewpoint images located in the four directions of up, down, left and right are selected with the first central viewpoint image as the center, that is, the target viewpoint image, and the target viewpoint image data and the first central viewpoint image data are the image data (light field sub-view) processed by the second module when performing feature extraction.

[0063] In this embodiment, it is not necessary to process all the light field image data to be processed. Only the first central viewpoint image data and part of the image data adjacent thereto need to be extracted to realize the extraction of spatial angle features, thereby reducing the data processing amount of the second module and improving the processing efficiency.

[0064] Optionally, before controlling the first module to perform feature extraction on the light field image data to be processed to obtain the first feature, the method further includes:

[0065] Acquire initial light field image data;

[0066] Decomposing the initial light field image data into a plurality of sub-viewpoint image data, wherein the plurality of sub-viewpoint image data are arranged in an array, and each of the sub-viewpoint image data has a consistent spatial resolution;

[0067] Acquire second central viewpoint image data, wherein the second central viewpoint image data is sub-viewpoint image data located at a central position among the plurality of sub-viewpoint image data;

[0068] Taking the second central viewpoint image data as the center, selecting adjacent viewpoint image data of adjacent areas from the plurality of sub-viewpoint image data;

[0069] The central viewpoint image data and the adjacent viewpoint image data are determined as the light field image data to be processed.

[0070] The solution of this embodiment is described with the following example:

[0071] Initial light field image data captured by a light field camera is obtained, and the initial light field image data is decomposed into 15×15 sub-viewpoint image data with a spatial resolution of 434×625, and the second central viewpoint image data located at the center and 5×5 sub-viewpoint image data in its adjacent area are selected, and the 5×5 sub-viewpoint image data in its adjacent area are adjacent viewpoint image data, and the second central viewpoint image data and the adjacent viewpoint image data are randomly cropped into image data with a spatial resolution of 128×128 to obtain light field image data to be processed.

[0072] In this embodiment, preprocessing the data in the above manner can reduce the amount of data to be processed, reduce computing costs, and improve efficiency.

[0073] Optionally, the reconstruction module includes multiple 3D convolutional layers, and the multiple 3D convolutional layers are residually connected.

[0074] like Figure 2 As shown, the reconstruction module includes multiple 3D convolutional layers, and cross-layer residual connections can be performed between 3D convolutional layers. The first feature and the second feature are connected in the channel dimension and input into the reconstruction module. The reconstruction module can model the spatial and angular information of the light field, and at the same time use shared weights and feature representations to reduce the number of parameters in the module network. Through cross-layer residual links, the extracted features can be transmitted and reused in the module network, helping the module network to better fuse features and reconstruct light field information. Among them, cross-layer residual links can span multiple layers, and residual connections across 1 layer retain information between adjacent layers. Residual connections across multiple layers can transmit information at a deeper level to improve the quality of the final light field image.

[0075] Optionally, the method further comprises:

[0076] Get sample data;

[0077] Performing an N-th training on the initial network model according to the sample data;

[0078] When the initial network model after the Nth training meets the preset convergence condition, the initial network model after the Nth training is determined as the image enhancement neural network model;

[0079] Among them, the loss function of the initial network model includes: a first function and a second function, the image structure similarity (Structural Similarity, SSIM) between the sample data and the output data of the initial network model is associated with the first function, and the spatial consistency loss between the sample data and the output data of the initial network model is associated with the second function.

[0080] In this embodiment, before applying the image enhancement neural network model for image enhancement, it is also necessary to train the image enhancement neural network model. During the training process, the loss function is used to adjust the parameters of the first module, the second module, and the reconstruction module in the initial network model. The loss function adopts a combination of supervised and unsupervised methods, wherein the supervised first function includes:

[0081]

[0082] Among them, I1 represents the light field image output by the initial network model, I 2 Represents the corresponding true value. φ() represents the network calculation process of the pre-trained classic feature extraction model VGG-16, SSIM(I 1 , I 2 ) is the structural similarity between the output light field image and the true value.

[0083] The unsupervised second function includes:

[0084]

[0085] The number of domains is K, Ω(k) is the domain in the four directions of up, down, left, and right with the region as the center, and I ′ k and I k Represent the features of the kth region of the initial network model output image and input image, I j ′ and I j They represent the j-th domain features of the k-th region of the initial network model output image and the input image respectively.

[0086] Combining the first function and the second function, the loss function is:

[0087]

[0088] Among them, the hyperparameter λ is used to adjust the weights of supervised loss and unsupervised loss.

[0089] In this embodiment, the parameters of the image enhancement neural network model training process are adjusted according to the image structure similarity and spatial consistency loss to improve the accuracy of the trained image enhancement neural network model, thereby improving the quality of the enhanced light field image obtained by using the image enhancement neural network model.

[0090] like Figure 3 As shown, the embodiment of the present application also provides a light field image enhancement device, which is applied to an image enhancement neural network model. The image enhancement neural network model includes a first module, a second module and a reconstruction module. The first module and the second module are respectively connected to the reconstruction module. The light field image enhancement device 300 includes:

[0091] A first control module 301 is used to control the first module to perform feature extraction on the light field image data to be processed to obtain a first feature, wherein the first feature is used to indicate the global feature and local feature of the light field image to be processed;

[0092] A second control module 302 is used to control the second module to extract features from the light field image data to be processed to obtain a second feature, wherein the second feature is used to indicate a spatial feature of the light field image to be processed;

[0093] The third control module 303 is used to input the first feature and the second feature into the reconstruction module, and control the reconstruction module to perform feature fusion and light field reconstruction on the first feature and the second feature to obtain an enhanced light field image.

[0094] Optionally, the first module includes a residual block, the residual block includes a quantum convolution layer and a 2D convolution layer, and the first control module 301 includes:

[0095] A first acquisition submodule is used to acquire the light field image data to be processed, and obtain a light field image matrix to be processed according to the light field image data to be processed;

[0096] A first control submodule, used for controlling the quantum convolution layer to perform quantum convolution calculation on the light field image data to be processed to obtain a quantum convolution matrix;

[0097] A second control submodule is used to control the 2D convolution layer to perform 2D convolution calculation on the quantum convolution matrix to obtain a 2D convolution matrix;

[0098] The first calculation submodule is used to calculate the sum of the light field image matrix to be processed and the 2D convolution matrix, and determine the sum of the light field image matrix to be processed and the 2D convolution matrix as the first feature.

[0099] Optionally, the first module includes N residual blocks, where N is an integer greater than 1;

[0100] When N is 2, the input data of the first residual block is the light field image data to be processed, the input data of the second residual block is the sum of the output data of the first residual block and the light field image data to be processed, and the first feature is the sum of the output data of the second residual block and the output data of the first residual block;

[0101] When N is greater than 2, the input data of the first residual block is the light field image data to be processed, the input data of the second residual block is the sum of the output data of the first residual block and the light field image data to be processed, the input data of the i-th residual block is the sum of the output data of the i-1-th residual block and the output data of the i-2-th residual block, and the first feature is the sum of the output data of the N-th residual block and the output data of the N-1-th residual block, where i ranges from 2 to N.

[0102] Optionally, the second module includes at least two 3D dilated convolutional layers, each of the 3D dilated convolutional layers corresponds to a different dilation factor, and the second control module 302 includes:

[0103] A second acquisition submodule, configured to acquire first central viewpoint image data of the light field image data to be processed, wherein the first central viewpoint image data is sub-viewpoint image data located at a central position in the light field image data to be processed;

[0104] A first determination submodule is configured to determine target viewpoint image data from the light field image data to be processed based on the first central viewpoint image data, wherein the target viewpoint image data includes: image data corresponding to sub-viewpoints adjacent to the first central viewpoint image data and located in upper, lower, left, and right directions;

[0105] A third control submodule is used to control the at least two 3D dilated convolution layers to perform 3D dilated convolution calculations on the first central viewpoint image data and the target viewpoint image data, respectively, to obtain at least two second sub-features, wherein the at least two second sub-features correspond one-to-one to the at least two 3D dilated convolution layers;

[0106] The feature fusion submodule is used to perform feature fusion on the at least two second sub-features to obtain the second feature.

[0107] Optionally, the light field image enhancement device 300 is further used for:

[0108] Acquire initial light field image data;

[0109] Decomposing the initial light field image data into a plurality of sub-viewpoint image data, wherein the plurality of sub-viewpoint image data are arranged in an array, and each of the sub-viewpoint image data has a consistent spatial resolution;

[0110] Acquire second central viewpoint image data, wherein the second central viewpoint image data is sub-viewpoint image data located at a central position among the plurality of sub-viewpoint image data;

[0111] Taking the second central viewpoint image data as the center, selecting adjacent viewpoint image data of adjacent areas from the plurality of sub-viewpoint image data;

[0112] The central viewpoint image data and the adjacent viewpoint image data are determined as the light field image data to be processed.

[0113] Optionally, the reconstruction module includes multiple 3D convolutional layers, and the multiple 3D convolutional layers are residually connected.

[0114] Optionally, the light field image enhancement device 300 is further used for:

[0115] Get sample data;

[0116] Performing an N-th training on the initial network model according to the sample data;

[0117] When the initial network model after the Nth training meets the preset convergence condition, the initial network model after the Nth training is determined as the image enhancement neural network model;

[0118] Among them, the loss function of the initial network model includes: a first function and a second function, the image structure similarity SSIM between the sample data and the output data of the initial network model is associated with the first function, and the spatial consistency loss between the sample data and the output data of the initial network model is associated with the second function.

[0119] It should be noted that the light field image enhancement device 300 provided in the embodiment of the present application can achieve the following Figure 1 The entire technical process of the light field image enhancement method shown in the embodiment achieves the same technical effect, and will not be described again here to avoid repetition.

[0120] The embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program is executed by the processor to implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0121] For details, see Figure 4 As shown, an embodiment of the present application further provides an electronic device, including a bus 401 , a transceiver 402 , an antenna 403 , a bus interface 404 , a processor 405 and a memory 406 .

[0122] In this embodiment, the electronic device further includes: a computer program stored in the memory 406 and executable on the processor 405. When the computer program is executed by the processor 405, the following can be achieved: Figure 1 Each process of the light field image enhancement method shown in the embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.

[0123] exist Figure 4In the embodiment, a bus architecture (represented by bus 401) is shown, bus 401 may include any number of interconnected buses and bridges, bus 401 links various circuits including one or more processors represented by processor 405 and memory represented by memory 406. Bus 401 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 404 provides an interface between bus 401 and transceiver 402. Transceiver 402 may be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by processor 405 is transmitted on a wireless medium via antenna 403, and further, antenna 403 also receives data and transmits the data to processor 405.

[0124] Processor 405 is responsible for managing bus 401 and general processing, and may also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 406 may be used to store data used by processor 405 when performing operations.

[0125] Optionally, the processor 405 may be a CPU, an ASIC, an FPGA or a CPLD.

[0126] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned light field image enhancement method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. The computer-readable storage medium is, for example, a ROM, RAM, a magnetic disk or an optical disk.

[0127] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the light field image enhancement method embodiment shown in the figure can achieve the same technical effect, and will not be described again here to avoid repetition.

[0128] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0130] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A light field image enhancement method, applied to an image enhancement neural network model, wherein the image enhancement neural network model comprises a first module, a second module and a reconstruction module, wherein the first module and the second module are respectively connected to the reconstruction module, It is characterized in that The method comprises: Controlling the first module to perform feature extraction on the light field image data to be processed to obtain a first feature, wherein the first feature is used to indicate a global feature and a local feature of the light field image to be processed; Controlling the second module to perform feature extraction on the light field image data to be processed to obtain a second feature, wherein the second feature is used to indicate a spatial feature of the light field image to be processed; The first feature and the second feature are input into the reconstruction module, and the reconstruction module is controlled to perform feature fusion and light field reconstruction on the first feature and the second feature to obtain an enhanced light field image.

2. The method according to claim 1, It is characterized in that The first module includes a residual block, the residual block includes a quantum convolution layer and a 2D convolution layer, and the controlling the first module to extract features from the processed light field image data to obtain a first feature includes: Acquire the light field image data to be processed, and obtain a light field image matrix to be processed according to the light field image data to be processed; Controlling the quantum convolution layer to perform quantum convolution calculation on the light field image data to be processed to obtain a quantum convolution matrix; Controlling the 2D convolution layer to perform 2D convolution calculation on the quantum convolution matrix to obtain a 2D convolution matrix; A sum of the light field image matrix to be processed and the 2D convolution matrix is ​​calculated, and the sum of the light field image matrix to be processed and the 2D convolution matrix is ​​determined as the first feature.

3. The method according to claim 2, It is characterized in that The first module includes N residual blocks, where N is an integer greater than 1; When N is 2, the input data of the first residual block is the light field image data to be processed, the input data of the second residual block is the sum of the output data of the first residual block and the light field image data to be processed, and the first feature is the sum of the output data of the second residual block and the output data of the first residual block; When N is greater than 2, the input data of the first residual block is the light field image data to be processed, the input data of the second residual block is the sum of the output data of the first residual block and the light field image data to be processed, the input data of the i-th residual block is the sum of the output data of the i-1-th residual block and the output data of the i-2-th residual block, and the first feature is the sum of the output data of the N-th residual block and the output data of the N-1-th residual block, where i ranges from 2 to N.

4. The method according to claim 1, It is characterized in that The second module includes at least two 3D dilated convolution layers, each of the 3D dilated convolution layers corresponds to a different dilation factor, and the controlling the second module to extract features from the light field image data to be processed to obtain a second feature includes: Acquire first central viewpoint image data of the light field image data to be processed, wherein the first central viewpoint image data is sub-viewpoint image data located at a central position in the light field image data to be processed; Based on the first central viewpoint image data, determining target viewpoint image data from the light field image data to be processed, wherein the target viewpoint image data includes: image data corresponding to sub-viewpoints adjacent to the first central viewpoint image data and located in upper, lower, left, and right directions; Controlling the at least two 3D dilated convolution layers to perform 3D dilated convolution calculations on the first central viewpoint image data and the target viewpoint image data, respectively, to obtain at least two second sub-features, wherein the at least two second sub-features correspond one-to-one to the at least two 3D dilated convolution layers; The at least two second sub-features are subjected to feature fusion to obtain the second feature.

5. The method according to claim 1, It is characterized in that Before controlling the first module to extract features from the light field image data to be processed to obtain the first features, the method further includes: Acquire initial light field image data; Decomposing the initial light field image data into a plurality of sub-viewpoint image data, wherein the plurality of sub-viewpoint image data are arranged in an array, and each of the sub-viewpoint image data has a consistent spatial resolution; Acquire second central viewpoint image data, wherein the second central viewpoint image data is sub-viewpoint image data located at a central position among the plurality of sub-viewpoint image data; Taking the second central viewpoint image data as the center, selecting adjacent viewpoint image data of adjacent areas from the plurality of sub-viewpoint image data; The central viewpoint image data and the adjacent viewpoint image data are determined as the light field image data to be processed.

6. The method according to any one of claims 1 to 5, It is characterized in that The reconstruction module includes multiple 3D convolutional layers, and the multiple 3D convolutional layers are residually connected.

7. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: Get sample data; Performing an N-th training on the initial network model according to the sample data; When the initial network model after the Nth training meets the preset convergence condition, the initial network model after the Nth training is determined as the image enhancement neural network model; Among them, the loss function of the initial network model includes: a first function and a second function, the image structure similarity SSIM between the sample data and the output data of the initial network model is associated with the first function, and the spatial consistency loss between the sample data and the output data of the initial network model is associated with the second function.

8. A light field image enhancement device, applied to an image enhancement neural network model, wherein the image enhancement neural network model comprises a first module, a second module and a reconstruction module, wherein the first module and the second module are respectively connected to the reconstruction module, It is characterized in that The device comprises: A first control module, used for controlling the first module to perform feature extraction on the light field image data to be processed to obtain a first feature, wherein the first feature is used to indicate a global feature and a local feature of the light field image to be processed; A second control module, used for controlling the second module to perform feature extraction on the light field image data to be processed to obtain a second feature, wherein the second feature is used to indicate a spatial feature of the light field image to be processed; The third control module is used to input the first feature and the second feature into the reconstruction module, and control the reconstruction module to perform feature fusion and light field reconstruction on the first feature and the second feature to obtain an enhanced light field image.

9. An electronic device, It is characterized in that include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the light field image enhancement method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the light field image enhancement method according to any one of claims 1 to 7 are implemented.

11. A computer program product, It is characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the light field image enhancement method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • View consistency-oriented light field image super-resolution reconstruction method

    CN114463172A

  • Infrared image super-resolution method based on light field

    CN116523740A

  • Compressed light field quality enhancement method based on space angle deformable convolutional network

    CN116934647A

  • Light field image super-resolution reconstruction method fusing space and angle features

    CN116977168A

  • Light field angle reconstruction method based on geometric perception multi-feature fusion network

    CN117455787A