A super-resolution image enhancement method based on focal stacking

By adopting a super-resolution image enhancement method based on focus stack in the light field camera, separating the focus and defocusing areas and using CNN network processing, the problem of insufficient in-depth exploration of focus stacks in the prior art is solved, and more efficient Image enhancement effect.

CN118570070BActive Publication Date: 2024-12-03深圳市斯贝达电子有限公司
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410724470.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-05
Publication Date
2024-12-03
Estimated Expiration
2044-06-05

AI Technical Summary

Technical Problem

Existing light field cameras ignore in-depth exploration of the focus stack and pixel enhancement in image super-resolution enhancement, resulting in unsatisfactory super-resolution enhancement.

Method used

Using a super-resolution image enhancement method based on the focus stack, the light field area image is acquired through the microlens array in the light field image acquisition device, pre-processing and super-resolution processing, separation of the focus and defocus areas, and using CNN convolution Network and cost aggregation technology improves image resolution.

Benefits of technology

A clearer and more accurate image enhancement effect is achieved, highlighting the main part of the image and improving the overall quality of the light field image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570070B_ABST
    Figure CN118570070B_ABST
Patent Text Reader

Abstract

The present invention provides a super-resolution image enhancement method based on focal stack, which relates to the field of light field image enhancement. A microlens array arranged inside a light field image acquisition device is used to capture and obtain an image of a desired light field area to obtain an original light field image. A focal stack of the image is generated by a light field image preprocessing unit and then transmitted to a focal stack super-resolution processing unit. The focal stack is then subjected to super-resolution processing to obtain a final image. When the focal stack is processed by the focal stack super-resolution processing unit, a depth-based focal stack super-resolution method is adopted. The final image is subjected to integration processing by a light field image post-processing unit to obtain an integrated image, which is displayed by a light field display unit. The present invention uses focal stack depth cues to perform super-resolution processing on light field images to improve the quality of images captured by a light field camera. The focus area will be clearer and the main part of the light field image can be better displayed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a light field image enhancement method, and in particular to a super-resolution image enhancement method based on focal stacking. Background Art

[0002] Light field images are an image form that captures information about light propagating in space. Unlike traditional 2D images, light field images record not only the intensity and color information of light, but also the direction and position of light. This technology can achieve functions that traditional images cannot achieve, such as depth of field adjustment, by capturing the 4D information of light. As an important information carrier, the quality of images directly affects the ability to express information. Therefore, image super-resolution technology has been widely used in many fields due to its powerful ability to restore high-quality images. Compared with traditional two-dimensional images, light field images meet people's needs for high-dimensional information. However, light field images captured by mainstream light field cameras always face the problem of insufficient image resolution. Depth cues are information used by the visual system to perceive the distance and depth of objects. In light field images, depth cues are obtained by capturing the direction and position of light. Depth cues play an important role in light field super-resolution enhancement, but most existing works focus on how to use focal stacks, while ignoring the in-depth exploration and pixel enhancement of focal stacks, which leads to unsatisfactory super-resolution enhancement effects. Summary of the invention

[0003] The present invention provides a super-resolution image enhancement method based on focus stacking, which solves the problem of in-depth exploration and pixel enhancement of focus stacking. The technical solution is as follows:

[0004] A super-resolution image enhancement method based on focal stacking comprises the following steps:

[0005] S1: Use a microlens array provided inside a light field image acquisition device to capture an image of a desired light field area to obtain an original light field image, the original light field image is processed by a light field image preprocessing unit to obtain a focal stack, and the focal stack is transmitted to a focal stack super-resolution processing unit;

[0006] S2: using a focal stack super-resolution processing unit, processing the focal stack using a depth-based focal stack super-resolution method to obtain a final image;

[0007] S3: The final image is integrated and processed by the light field image post-processing unit to obtain an integrated image, and the integrated image is displayed by the light field display unit. The light field display unit includes a computer, an image transmitter, and an LED screen connected in sequence.

[0008] In step S1, when obtaining the desired light field area image, a microlens array and a two-dimensional sensor are used to record the propagation direction of light, and the depth calculation formula is obtained by using the microlens imaging principle:

[0009] ;

[0010] Where: S is the parallax, X is the distance between the two viewpoints, D' represents the distance from the main lens to the focusing plane, and F' is the distance from the main lens to the microlens array. The depth D can be calculated according to the above formula.

[0011] In step S1, the specific steps of the light field image preprocessing unit obtaining the focal stack are:

[0012] S11: First, adjust and process the original light field image and perform gamma correction on it. The formula is as follows:

[0013] ;

[0014] in, is the brightness value of the original image, is the corrected brightness value, is the Gamma value;

[0015] S12: Then traverse all viewing angles of the adjusted image, perform a translation accumulation operation according to the depth D, record each tilt parameter, and generate a focus stack image, where the tilt parameter controls the focal plane position of the image in the focus stack.

[0016] In step S2, the focal stack super-resolution processing unit uses a super-resolution network based on different regions of the focal stack when processing the focal stack, specifically including:

[0017] S21: Obtaining a depth map corresponding to the focus stack;

[0018] S22: Corresponding the depth information to the focus stack according to the depth map;

[0019] S23: dividing the focus stack into a focused area and an out-of-focus area according to the depth value;

[0020] S24: the focused area in the focus stack is fed into the super-resolution network with different noise levels;

[0021] S25: Evaluate multiple focus stacks after processing at different noise levels;

[0022] S26: after evaluation, the best noise level of the focus stack focus area is obtained;

[0023] S27: The out-of-focus area in the focus stack is also fed into the super-resolution network;

[0024] S28: Integrate the best areas of each focus stack and output it as the final image result.

[0025] In step S21, obtaining a depth map corresponding to the focus stack includes the following steps:

[0026] Step S201: first load the focal stack obtained through the original light field image;

[0027] Step S202: applying the RDF filter to generate a focus map for each image in the focus stack;

[0028] Step S203: using cost aggregation technology and guided filtering technology to improve the accuracy of focus mapping;

[0029] Step S204: comparing and analyzing the depth value of each pixel with the depth values ​​of its surrounding pixels;

[0030] Step S205: Calculate the final depth value of each pixel according to the focus map to obtain a depth map.

[0031] In step S24, the focus area in the focus stack enters the super-resolution network with different noise levels, including the following steps:

[0032] Step S301: setting different noise levels for the focus stack image of the focus area for matching, with the range set to [0-30] and the interval set to 1;

[0033] Step S302: adding an isotropic Gaussian blur kernel before entering the network, and using the LDA dimension reduction method after vectorization;

[0034] Step S303: Mapping the processed blur kernel and noise level and connecting them to the image; the image is one of the focus stacks entering the network;

[0035] Step S304: taking the integrated image as input and entering a 3×3 CNN convolutional network;

[0036] Step S305: converting the multiple HR sub-images into a single HR image through a sub-pixel convolution layer, wherein the multiple HR sub-images are generated by the sub-pixel convolution layer.

[0037] In step S25, the plurality of focus stacks processed with different noise levels are evaluated, including the following steps:

[0038] Step S401: The same focus area of ​​each focus stack is calculated using the natural statistical characteristic no-reference index NIQE, and the MVG model density function is first used, and the formula is as follows:

[0039] ;

[0040] is the NSS feature, x and ∑ represent the mean and covariance matrix of the MVG model, respectively, obtained by standard maximum likelihood estimation, and k represents a natural number;

[0041] Step S402: The distance between the NSS feature and the MVG extracted from the distorted image feature is used to represent the image quality, and the focus areas with different noise levels in the same area of ​​the focus stack are evaluated for quality. The formula is as follows:

[0042] ;

[0043] Represent the mean vectors of the MVG natural and distorted models MVG models, respectively, Represent the covariance matrices of the MVG natural and distorted models MVG models respectively.

[0044] In step S27, when the defocused area in the focus stack enters the super-resolution network, the noise level of the defocused area is selected to be twice the optimal noise level of the focused area, and the network processing method is the same as step S24.

[0045] In step S3, the light field image post-processing unit integrates the focal stack, including the following steps:

[0046] Step S501: using cost aggregation technology and guided filtering technology to improve the accuracy of focus mapping;

[0047] Step S502: determining the image where the best focus of each pixel is located after processing by calculating the focus map;

[0048] Step S503: selecting a pixel with the best focus at each pixel position in the focus area;

[0049] Step S504: The pixel values ​​of the focused area with the best focus and the pixel values ​​of the processed defocused area are combined together to form an integrated image.

[0050] The super-resolution image enhancement method based on focus stack has a better image enhancement effect through in-depth exploration and pixel enhancement of focus stack. The present invention has the following advantages: (1) the characteristics of light field image recording the direction and position of light are used to process the focus stack based on the depth clue of light field image; (2) the focus stack is divided into separation and focus areas, and the processing area is more accurate; (3) different noise levels can be used to process the focus stack to select the optimal image; (4) the focus stack processed by the present invention is more prominent and has a better effect than the ordinary super-resolution focus stack; (5) the super-resolution network uses a CNN convolutional network, and the network structure is simple. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic flow chart of the super-resolution image enhancement method based on focal stack;

[0052] Figure 2 is a structural principle diagram of the light field image acquisition device;

[0053] Figure 3 is an operation flow chart of the focal stack super-resolution processing unit;

[0054] Figure 4 is a flow chart of the operation of obtaining a depth map by stacking the focus;

[0055] Figure 5 The focus area enters the super-resolution network operation flow chart with different noise levels;

[0056] Figure 6 is an operation flow chart of the light field image post-processing unit.

[0057] like Figure 1 As shown, the present invention provides a super-resolution image enhancement method based on focal stacking, which can more clearly and accurately display the light field image on the screen, comprising the following steps:

[0058] S1: Use a microlens array provided inside a light field image acquisition device to capture an image of a desired light field area to obtain an original light field image, the original light field image is processed by a light field image preprocessing unit to obtain a focal stack, and the focal stack is transmitted to a focal stack super-resolution processing unit;

[0059] S2: using a focal stack super-resolution processing unit, processing the focal stack using a depth-based focal stack super-resolution method to obtain a final image;

[0060] S3: The final image is integrated and processed by the light field image post-processing unit to obtain an integrated image, and the integrated image is displayed by the light field display unit. The light field display unit serves as a display device and includes a computer, an image transmitter, and an LED screen connected in sequence.

[0061] To achieve a clear display of light field images, the resolution needs to be high enough, so an LED screen is used, and the LED module is included in the LED screen. The integrated image is first transmitted to the computer, and the image transmitter is connected to the DP port of the computer through the DP cable. The image transmitter is then connected to the 3D light field display device through the HDMI port.

[0062] In step S1, when obtaining the desired light field area image, a microlens array and a two-dimensional sensor are used to record the propagation direction of light, and the depth is calculated using the microlens imaging principle to obtain the calculation formula:

[0063] ;

[0064] Where: S is the parallax, X is the distance between the two viewpoints, D' represents the distance from the main lens to the focusing plane, and F' is the distance from the main lens to the microlens array. The depth D can be calculated according to the above formula.

[0065] Combination Figure 2 As shown, the light field image acquisition device is provided with a microlens array. The image of the object plane passes through the lens principal plane and the microlens array and then reaches the sensor and is stored to form an original light field image.

[0066] The specific steps of the light field image preprocessing unit to obtain the focal stack are:

[0067] S11: First, adjust and process the original light field image and perform gamma correction on it. The formula is as follows:

[0068] ;

[0069] is the brightness value of the original image, is the corrected brightness value, is the Gamma value.

[0070] S12: Then, all viewing angles of the adjusted image are traversed, a Shift-and-Sum operation is performed according to the depth parameter D, each slope parameter is recorded, and a focus stack image is generated. The slope parameter controls the focal plane position of the image in the focus stack.

[0071] The Shift-and-Sum refers to a translation and accumulation operation. By properly translating and accumulating the light, the light is focused on a specific focal plane, thereby achieving image refocusing. The slope parameter is a parameter used to describe the offset of the light in space, called a tilt parameter or a slope parameter.

[0072] like Figure 3 As shown, when processing the focus stack, the focus stack super-resolution processing unit adopts a super-resolution network based on different regions of the focus stack, specifically including:

[0073] S21: Obtaining a depth map corresponding to the focus stack;

[0074] S22: Corresponding the depth information to the focus stack according to the depth map;

[0075] S23: dividing the focus stack into a focused area and an out-of-focus area according to the depth value;

[0076] S24: the focused area in the focus stack is fed into the super-resolution network with different noise levels;

[0077] S25: Evaluate multiple focus stacks after processing at different noise levels;

[0078] S26: after evaluation, the best noise level of the focus stack focus area is obtained;

[0079] S27: The out-of-focus area in the focus stack is also fed into the super-resolution network;

[0080] S28: Integrate the best areas of each focus stack and output it as the final image result.

[0081] like Figure 4 As shown, in step S21, obtaining a depth map corresponding to the focus stack includes the following steps:

[0082] Step S201: first load the focal stack obtained through the original light field image;

[0083] Step S202: applying the RDF filter to generate a focus map for each image; each image represents an image in a focus stack, and the focus stack is generated from the original light field image;

[0084] Step S203: Use cost aggregation technology and guided filtering technology to improve the accuracy of focus mapping; the cost aggregation technology is widely used in fields such as stereo matching and image segmentation in computer vision. After improvement, it can effectively remove noise while retaining image details and edges.

[0085] Step S204: comparing and analyzing the depth value of each pixel with the depth values ​​of its surrounding pixels;

[0086] Step S205: Calculate the final depth value of each pixel according to the focus map, and then obtain a depth map. The depth value can be obtained by focusing on the depth information in the focus map.

[0087] In summary, it can be seen that the process of generating a depth map includes focus mapping acquisition, cost aggregation, guided filtering, depth relationship analysis, and converting the calculated depth value into a depth map.

[0088] In step S24, the focus area in the focus stack enters the super-resolution network with different noise levels, including the following steps:

[0089] Step S301: setting different noise levels for the focus stack image of the focus area for matching, with the range set to [0-30] and the interval set to 1;

[0090] Step S302: before entering the network, an isotropic Gaussian blur kernel is added, and after vectorization, the LDA dimension reduction method is used; LDA is a linear transformation method for data dimension reduction and classification.

[0091] Step S303: Map the processed blur kernel and noise level and connect them to the image; wherein the image is one of the focus stacks entering the network, the focus stack has many images, but each image in the focus stack must enter the network.

[0092] Step S304: taking the integrated image as input and entering a 3×3 CNN convolutional network;

[0093] Step S305: converting the multiple HR sub-images into a single HR image through a sub-pixel convolution layer. The multiple HR sub-images are generated by the sub-pixel convolution layer, and the working principle of the sub-pixel convolution layer is to expand the number of channels through pixel rearrangement operation, thereby achieving upsampling.

[0094] like Figure 5 As shown, in step S25, the plurality of focus stacks processed with different noise levels are evaluated, including the following steps:

[0095] Step S401: The same focus area of ​​each focus stack is calculated using the natural statistical characteristic no-reference index NIQE (Naturalness Image Quality Evaluator), and the MVG model density function (multivariate Gaussian distribution model density function) is first used. The formula is as follows:

[0096] ;

[0097] is the NSS feature, x and ∑ represent the mean and covariance matrix of the MVG model respectively, which are obtained by standard maximum likelihood estimation; k represents a natural number.

[0098] Step S402: The distance between the NSS feature and the MVG extracted from the distorted image feature is used to represent the image quality, and the focus areas with different noise levels in the same area of ​​the focus stack are evaluated for quality. The formula is as follows:

[0099] ;

[0100] Represent the mean vectors of the MVG natural and distorted models MVG models, respectively, Represent the covariance matrices of the MVG natural and distorted models MVG models respectively.

[0101] The NSS feature is a feature representation method for eye movement data analysis, and is mainly used to study the behavior of human visual attention when viewing images or videos. The Chinese name of the NSS feature is the Normalized Saccade Activation Feature.

[0102] In step S27, the defocused area in the focus stack enters the super-resolution network, which includes the following steps: the noise level of the defocused area is selected to be twice the optimal noise level of the focused area, and the network processing method is the same as the operation steps of the focused area in step S24.

[0103] like Figure 6 As shown, in the step S3, the light field image post-processing unit integrates the focal stack, including the following steps:

[0104] Step S501: using cost aggregation technology and guided filtering technology to improve the accuracy of focus mapping;

[0105] Cost aggregation technology can fuse information from multiple sources together when processing focus mapping to reduce noise and improve image quality. At the same time, guided filtering technology can use the structural information of the image to smooth the focus mapping to ensure that the output mapping results are more coherent and reliable. The combined use of these two technologies can effectively improve the accuracy of focus mapping.

[0106] Step S502: determining the image where the best focus of each pixel is located after processing by calculating the focus map;

[0107] Step S503: selecting a pixel with the best focus at each pixel position in the focus area;

[0108] Step S504: The pixel values ​​of the focused area with the best focus and the pixel values ​​of the processed defocused area are combined together to form an integrated image.

[0109] As a display system, the light field display unit is composed of a computer, an image transmitter, and an LED screen, and the image display is finally displayed on the LED screen.

[0110] The present invention uses the focus stack depth clue to perform super-resolution processing on the light field image to improve the quality of the image taken by the light field camera. Through this method, the focus area will be clearer and the main part in the light field image can be better displayed.

Claims

1. A super-resolution image enhancement method based on focal stacking, comprising the following steps: S1: Use a microlens array provided inside a light field image acquisition device to capture an image of a desired light field area to obtain an original light field image, the original light field image is processed by a light field image preprocessing unit to obtain a focal stack, and the focal stack is transmitted to a focal stack super-resolution processing unit; S2: using a focal stack super-resolution processing unit, processing the focal stack using a depth-based focal stack super-resolution method to obtain a final image; Wherein, when processing the focal stack, the focal stack super-resolution processing unit adopts a super-resolution network based on different regions of the focal stack, specifically including: S21: Obtaining a depth map corresponding to the focus stack; S22: Corresponding the depth information to the focus stack according to the depth map; S23: dividing the focus stack into a focused area and an out-of-focus area according to the depth value; S24: the focused area in the focus stack is fed into the super-resolution network with different noise levels; S25: Evaluate multiple focus stacks after processing at different noise levels; S26: after evaluation, the best noise level of the focus stack focus area is obtained; S27: The out-of-focus area in the focus stack is also fed into the super-resolution network; S28: Integrate each best area in the focus stack together to output as the final image result; In step S21, a depth map corresponding to the focus stack is obtained. The process of generating the depth map includes focus map acquisition, cost aggregation, guided filtering, depth relationship analysis, and converting the calculated depth value into a depth map. Specifically, the following steps are included: Step S201: first load the focal stack obtained through the original light field image; Step S202: applying the RDF filter to generate a focus map for each image in the focus stack; Step S203: using cost aggregation technology and guided filtering technology to improve the accuracy of focus mapping; Step S204: comparing and analyzing the depth value of each pixel with the depth values ​​of its surrounding pixels; Step S205: Calculate the final depth value of each pixel according to the focus map to obtain a depth map; In step S24, the focus area in the focus stack enters the super-resolution network with different noise levels, including the following steps: Step S301: setting different noise levels for the focus stack image of the focus area for matching, with the range set to [0-30] and the interval set to 1; Step S302: adding an isotropic Gaussian blur kernel before entering the network, and using the LDA dimension reduction method after vectorization; Step S303: Mapping the processed blur kernel and noise level and connecting them to the image; the image is one of the focus stacks entering the network; Step S304: taking the integrated image as input and entering a 3×3 CNN convolutional network; Step S305: converting the multiple HR sub-images into a single HR image through a sub-pixel convolution layer, wherein the multiple HR sub-images are generated by the sub-pixel convolution layer; S3: The final image is integrated and processed by the light field image post-processing unit to obtain an integrated image, and the integrated image is displayed by the light field display unit. The light field display unit includes a computer, an image transmitter, and an LED screen connected in sequence.

2. The method for super-resolution image enhancement based on focal stacking according to claim 1, characterized in that: In step S1, when obtaining the desired light field area image, a microlens array and a two-dimensional sensor are used to record the propagation direction of light, and the depth calculation formula is obtained by using the microlens imaging principle: ; Where: S is the parallax, X is the distance between the two viewpoints, D' represents the distance from the main lens to the focusing plane, and F' is the distance from the main lens to the microlens array. The depth D can be calculated according to the above formula.

3. The method for super-resolution image enhancement based on focal stacking according to claim 1, characterized in that: In step S1, the specific steps of the light field image preprocessing unit obtaining the focal stack are: S11: First, adjust and process the original light field image and perform gamma correction on it. The formula is as follows: ; in, is the brightness value of the original image, is the corrected brightness value, is the Gamma value; S12: Then traverse all viewing angles of the adjusted image, perform a translation accumulation operation according to the depth D, record each tilt parameter, and generate a focus stack image, where the tilt parameter controls the focal plane position of the image in the focus stack.

4. The method for super-resolution image enhancement based on focal stacking according to claim 1, characterized in that: In step S25, the plurality of focus stacks processed with different noise levels are evaluated, including the following steps: Step S401: The same focus area of ​​each focus stack is calculated using the natural statistical characteristic no-reference index NIQE, and the MVG model density function is first used, and the formula is as follows: ; is the NSS feature, x and ∑ represent the mean and covariance matrix of the MVG model, respectively, obtained by standard maximum likelihood estimation, and k represents a natural number; Step S402: The distance between the NSS feature and the MVG extracted from the distorted image feature is used to represent the image quality, and the focus areas with different noise levels in the same area of ​​the focus stack are evaluated for quality. The formula is as follows: ; Represent the mean vectors of the MVG natural and distorted models MVG models, respectively, Represent the covariance matrices of the MVG natural and distorted models MVG models respectively.

5. The method for super-resolution image enhancement based on focal stacking according to claim 1, characterized in that: In step S27, when the defocused area in the focus stack enters the super-resolution network, the noise level of the defocused area is selected to be twice the optimal noise level of the focused area, and the network processing method is the same as step S24.

6. The method for super-resolution image enhancement based on focal stacking according to claim 1, characterized in that: In step S3, the light field image post-processing unit integrates the focal stack, including the following steps: Step S501: using cost aggregation technology and guided filtering technology to improve the accuracy of focus mapping; Step S502: determining the image where the best focus of each pixel is located after processing by calculating the focus map; Step S503: selecting a pixel with the best focus at each pixel position in the focus area; Step S504: The pixel values ​​of the focused area with the best focus and the pixel values ​​of the processed defocused area are combined together to form an integrated image.

Citation Information

Patent Citations

  • Light field feature extraction method fusing focus stack and full-focus image

    CN112465796A

  • Image processing method based on refocusing light field detection

    CN116684732A