Self-adaptive light field image enhancement pipeline system and method

Abstract

The invention provides a self-adaptive light field image enhancement pipeline system and method, and belongs to the field of image processing. The problem of low efficiency caused by the fact that one or more image processing algorithms are often needed in the light field image processing process is solved. According to a processing method required by a light field image, image classification is carried out to establish a training set, image features of the training set are extracted through a feature extractor, the image features and classification labels are used as input to train a neural network model, and the trained neural network model is obtained to complete a classification task of the image processing method. And the image is input into a light field image processing assembly line to be processed by some or all subsequent technical methods. The light field image processing assembly line carries out processing according to the four steps of image noise reduction, contrast enhancement, color enhancement and super-resolution in sequence. Through neural network image classification and whole pipeline processing, efficient and rapid classification and processing of light field images can be realized, and accuracy and adaptive capacity are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an adaptive light field image enhancement pipeline system and method. Background Art

[0002] A light field image is an image representation that captures information about the propagation of light in a scene. It not only records the brightness of each pixel in the scene, but also the direction of the light. Unlike traditional two-dimensional images, light field images can capture light from different angles and positions, allowing for later manipulations such as changing the perspective and adjusting the focal length. However, due to the limitations of light field image acquisition equipment, problems such as uneven light field color and low resolution often occur. Furthermore, due to environmental influences, light field images can also exhibit high levels of noise and color distortion, requiring subsequent processing.

[0003] Due to the varying quality of images, one or more image processing algorithms are often required during light field image processing. This application aims to establish an adaptive image processing pipeline to automate batch processing of light field images, improving image processing efficiency and effectiveness. Summary of the Invention

[0004] In order to solve the above technical problems, this application proposes an adaptive light field image enhancement pipeline system and method, which can adapt to various types of images and use a neural network-based image classification method to improve image processing efficiency.

[0005] The technical solution adopted in this application is: an adaptive light field image enhancement pipeline system, including a light field acquisition device, a light field image processing device, a light field display device and a DDR3 storage unit, the light field acquisition device is used to obtain the original light field image and transmit it to the light field image processing device;

[0006] The light field image processing device includes an image receiving module, an image preprocessing unit, an image classification unit, a denoising unit, a contrast enhancement unit, a color enhancement unit, a super-resolution unit and an image transmission unit. The image receiving module receives the original light field image transmitted by the light field acquisition device and stores it in the DDR3 storage unit. The image preprocessing unit converts the original light field image into different types of light field images as needed. The image classification unit completes the classification task of the preprocessed light field image according to the quality of the light field image and determines the processing method required for the image. The denoising unit, contrast enhancement unit, color enhancement unit and super-resolution unit respectively complete image denoising, contrast enhancement, color enhancement and super-resolution processing.

[0007] The light field display device is used to display the light field image processed by the light field image processing device.

[0008] The light field acquisition device includes an optical system module, an image sensor module and an image information transmission module. The optical system module consists of a main lens, a microlens array, a color filter array and a sensor array, and is used to obtain the position, direction and color information of the light field. The image sensor module is used to record the obtained light field position, direction and color information and generate an original light field image. The image information transmission module is used to save the original light field image generated by the image sensor module and transmit it to the light field image processing device.

[0009] It also includes an AI acceleration module, which is connected to the light field image processing device to accelerate the analysis and inference calculations of the image processing process.

[0010] An adaptive light field image enhancement pipeline method, using the adaptive light field image enhancement pipeline system, includes the following steps:

[0011] S1: Use light field acquisition equipment to complete the acquisition of original light field images;

[0012] S2: Perform image classification based on the processing method required for light field images, establish training sets and test sets, train a neural network model based on the training sets, and complete the image classification task based on the model;

[0013] S3: Complete image denoising in the light field image processing pipeline through the denoising unit;

[0014] S4: completing image contrast enhancement in the light field image processing pipeline through the contrast enhancement unit;

[0015] S5: Complete image color enhancement in the light field image processing pipeline through the color enhancement unit;

[0016] S6: Complete the image super-resolution processing in the light field image processing pipeline through the super-resolution unit.

[0017] Step S2 specifically includes:

[0018] S201: Use a microlens array provided in a light field image acquisition device to capture images of a desired light field area, obtain original light field images, and establish light field image training sets and test sets for various scenes;

[0019] S202: Build an image feature extractor to extract relevant indicators for evaluating image quality;

[0020] S203: Using the image features and classification labels extracted by the feature extractor from the training set as input, the neural network model is trained, and parameters in the model are continuously optimized through training;

[0021] S204: Using the trained neural network model to classify light field images that require different processing methods.

[0022] Step S3 specifically includes:

[0023] S301: extracting sub-aperture images and macro pixels from the original light field image through an image preprocessing unit;

[0024] S302: Using a convolutional autoencoder to extract texture structure information from the sub-aperture image and macro pixels;

[0025] S303: Perform two-dimensional discrete inverse cosine transform on the sub-aperture image and the macro pixel to obtain a frequency map;

[0026] S304: Sort the frequencies in order by frequency size and combine them into four categories of three-dimensional frequency maps. The first category contains the frequency map that retains the basic information of the image, which is the majority of the area with continuous changes, that is, the DC component in the frequency map. The second, third, and fourth categories are the high-frequency parts of the frequency map, that is, the parts with faster changes such as edge lines. We can evenly divide the frequency map into three parts based on the frequency map, or we can unequally divide it into three parts based on the frequency distribution. That is, the second category contains the frequency map with lower frequencies in the high-frequency part, the third category contains the frequency map with medium frequencies in the high-frequency part, and the fourth category contains the frequency map with higher frequencies in the high-frequency part;

[0027] S305: Input the above four types of frequency maps into the three-dimensional residual network respectively to perform frequency domain denoising;

[0028] S306: Convert the denoised frequency map into a pixel domain image through a two-dimensional inverse cosine transform, and fuse the pixel domain image with the light field image with texture structure information extracted in advance to obtain a denoised image.

[0029] Step S5 specifically includes:

[0030] S501: Decomposing the light field image into a structure component and a texture component using a structure-texture decomposition model;

[0031] S502: Convert the structural component from RGB space to Lab space, separate the luminance and chrominance channels, calculate color similarity using the color difference formula, select the first P main colors as the initial cluster centers, remove isolated pixels after iterative optimization, and generate a sub-block {B1, B2, B3} with uniform illumination;

[0032] S503: Extract the texture components through LBP features to generate a texture weight map W T ;

[0033] S504: The obtained sub-block {B1, B2, B3} is input into a network consisting of 16 RCAB residual modules connected in series for processing. The RCAB module consists of a Conv layer, a ReLU activation layer, a channel attention layer, and finally a residual connection;

[0034] S505: Perform multi-scale feature fusion on the enhanced structural components {B1', B2', B3'} and the texture weight map, convert them back to RGB space, and obtain the enhanced image after gamma correction.

[0035] Step S6 specifically includes:

[0036] S601: converting the original light field image into a sub-aperture image through an image preprocessing unit;

[0037] S602: Using a bicubic interpolation algorithm to preliminarily enlarge the sub-aperture image to generate a pre-super-resolution image. Using Gaussian convolution kernels of different scales, multi-scale convolution is performed on the pre-super-resolution image to generate a series of feature maps of different scales, capturing multi-level information of the image.

[0038] S603: Construct parallel 3×3 and 5×5 convolutional kernel networks to extract local details and larger-scale context features, respectively. Feature maps extracted by convolution kernels of different sizes are jointly processed to enhance the diversity of feature expression.

[0039] S604: Calculate the number of jumps, the total number of edges, and the number of center pixels, where the number of jumps is obtained by counting the number of grayscale value mutations within a pixel neighborhood, the total number of edges is obtained by detecting the number and distribution of edge structures in the image, and the number of center pixels is obtained by dynamically adjusting the classification threshold based on the difference between the center pixel and the neighborhood; based on the above indicators, divide the light field image into three types of regions: smooth area, edge area, and complex texture area;

[0040] S605: Pre-train multiple sets of filter vectors for different texture types, and dynamically select the optimal filter kernel for each region based on the real-time classification results;

[0041] S606: Capture high-frequency detail features through a parallel structure of 3×3 and 5×5 convolution kernels, input the multi-scale feature map extracted in step S602, and generate a high-frequency detail map containing sharp edges and textures;

[0042] S607: The filtered main image outputted in step S605 and the high-frequency detail image outputted in step S606 are superimposed according to the weights. The formula is as follows:

[0043] I fused =a·I filtered +b·I detail ;

[0044] Where a and b are weights, I filtered is the main image after filtering, I detail It is a high-frequency detail map;

[0045] S608: Using the differential results of Gaussian kernels of different scales to perform differential Gaussian filtering on the fused image, so as to suppress noise while sharpening edges, and finally generate a high-resolution image.

[0046] The convolutional autoencoder in step S302 consists of an encoder part, a latent space layer, and a decoder part;

[0047] The encoder part compresses the input image into a low-dimensional feature representation, retaining the most expressive texture and structural features. The first layer of the encoder is a convolution layer, which is used to extract the primary texture features of the image; the second layer is an activation and pooling layer, which is used to retain edge changes and main texture structures; the third and fourth layers are convolution and pooling layers, which are used to extract more complex texture combinations and structural features.

[0048] The latent space layer is the connection layer between the encoder and the decoder, which is a compact representation of the image spatial texture structure;

[0049] The decoder part is used to restore the structure and reconstruct the image. The decoder part consists of multiple layers of upsampling and deconvolution layers, which simulates the reverse process of encoding, gradually restores the original image, and maps the abstract structural information back to the original texture level.

[0050] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0051] The beneficial effects of the present application compared to the prior art are as follows: Based on light field images, the present application establishes a light field image processing pipeline taking into account the low image quality problem caused by light field equipment acquisition. Based on the processing method required for light field images, image classification is performed to establish a training set, and the image features of the training set are extracted through a feature extractor. These image features and classification labels are used as input to train the neural network model, and the trained neural network model completes the classification task of the image processing method. On this basis, the image is input into the light field image processing pipeline for processing through some or all of the subsequent technical methods. The light field image processing pipeline performs processing in four steps: image denoising, contrast enhancement, color enhancement, and super-resolution. Through neural network image classification and the entire pipeline processing, efficient and rapid classification and processing of light field images can be achieved, improving accuracy and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present application will be further described below with reference to the accompanying drawings:

[0053] Figure 1 A schematic diagram of the principle of a light field camera provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the structure of an adaptive image enhancement pipeline system provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of an adaptive image enhancement pipeline device provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the image classification training process provided in an embodiment of the present application;

[0057] Figure 5 Schematic diagram of an image feature extractor provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of a multi-layer perceptron network for image classification provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the denoising process provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of the structure of a convolutional autoencoder provided in an embodiment of the present application;

[0061] Figure 9 A schematic diagram of the contrast enhancement process provided in an embodiment of the present application;

[0062] Figure 10 Schematic diagram of the interpolation calculation method for different regions in the contrast enhancement processing provided in the embodiment of the present application;

[0063] Figure 11 A schematic diagram of the color enhancement process provided in an embodiment of the present application;

[0064] Figure 12 A schematic diagram of the super-resolution process flow provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] like Figures 1 to 12As shown, the present application provides an adaptive light field image enhancement pipeline system, including a light field acquisition device, a light field image processing device, a light field display device, a DDR3 storage unit and an AI acceleration module. The light field acquisition device includes an optical system module, an image sensor module, and an image information transmission module. The optical system module consists of a main lens, a microlens array, a color filter array and a sensor array, and is used to obtain the position, direction and color information of the light field. The image sensor module is used to record the acquired light field position, direction and color information, and generate an original light field image. The image information transmission module is used to save the original light field image generated by the image sensor module and transmit it to the light field image processing device.

[0066] The light field image processing device includes an image receiving module, an image preprocessing unit, an image classification unit, a denoising unit, a contrast enhancement unit, a color enhancement unit, a super-resolution unit, and an image transmission unit. The image receiving module receives the original light field image transmitted by the previous device. The image preprocessing unit converts the original light field image into a sub-aperture image, an extreme plane image, a focal stack image, etc. as needed. The image classification unit completes the classification task of the preprocessed light field image based on the quality of the light field image and determines the processing method required for the image (i.e., whether denoising, contrast enhancement, color enhancement, super-resolution processing, or a combination of two, three, or four processing methods is required). The denoising unit, contrast enhancement unit, color enhancement unit, and super-resolution unit respectively complete image denoising, contrast enhancement, color enhancement, and super-resolution processing.

[0067] The light field display device includes an image receiving unit and a display unit. The light field image processed by the light field image processing device is received and displayed on the light field image display unit.

[0068] The DDR3 storage unit receives the original light field image from the image receiving module of the light field image processing device, and the image pre-processing unit of the light field image processing device reads the original light field image from the DDR3 storage unit.

[0069] The AI acceleration module is connected to the light field image processing device to accelerate the analysis and inference calculations of the image processing process. Optional models include NVIDIA A100 and Huawei Ascend 910 / 310.

[0070] Take the NVIDIA A100, for example. The NVIDIA A100 is a high-performance, data center-class GPU based on the Ampere architecture. It boasts exceptional acceleration capabilities and is designed for high-performance computing (HPC), deep learning training and inference, and large-scale data processing tasks. For image processing algorithms, the A100, through its unique hardware architecture and software ecosystem, can achieve significant acceleration at multiple levels. The NVIDIA A100 accelerates image enhancement algorithms as follows: 1. The NVIDIA A100 uses the CUDA platform to run a large number of threads in parallel to process pixel-level image operations. For example, each pixel or region is assigned to a thread, allowing numerous threads to execute simultaneously, improving overall throughput. Shared memory is optimized to store small regions of image data (such as convolution kernels), reducing global memory accesses. 2. Tensor Cores, the A100's dedicated hardware for matrix multiplication and addition (MatMul+Add), are ideally suited for accelerating image enhancement models such as CNNs and Transformers. Convolutional and linear layers can be converted into matrix multiplications, allowing Tensor Cores to complete multiple matrix multiplication and addition operations per clock cycle (up to 20x speedup compared to traditional CUDA cores). 3. Multi-stream processing supports batch image processing. The A100 supports a large number of CUDA streams, enabling asynchronous concurrent operations. Image pre-processing, model inference, and post-processing tasks can be performed in parallel across multiple streams, reducing GPU idle time and improving computing utilization. 4. Large-capacity, high-speed video memory accelerates high-resolution image processing. The A100 has a maximum of 80GB of HBM2e high-bandwidth video memory with a bandwidth of up to 2TB / s, which has certain advantages in high-resolution image processing and image batch processing. High video memory can avoid frequent data transfer between the GPU and the host, reducing bottlenecks.

[0071] The embodiment of the present application also proposes an adaptive light field image enhancement pipeline algorithm, based on the above system, including the following steps:

[0072] S1: Use Figure 3 The light field acquisition device shown completes the acquisition of the original light field image;

[0073] S2: If Figure 4 As shown, the image classification is performed based on the processing method required for light field images, and a training set and a test set are established. Figure 6 The neural network model is used to complete the image classification task;

[0074] S3: According to Figure 7 The steps shown complete the image denoising in the light field image processing pipeline;

[0075] S4: According to Figure 9The steps shown complete the image contrast enhancement in the light field image processing pipeline;

[0076] S5: According to Figure 11 The steps shown complete the image color enhancement in the light field image processing pipeline;

[0077] S6: According to Figure 12 The steps shown complete the image super-resolution processing in the light field image processing pipeline.

[0078] Based on the processing method required for light field images, a training set is established for image classification. A neural network model is obtained based on the training set, and the image classification task is completed based on the model. The steps include:

[0079] S201: Use Figure 3 The microlens array provided inside the light field image acquisition device shown captures images of the desired light field area, obtains original light field images, and establishes light field image training sets and test sets for various scenes;

[0080] S202: Establish Figure 5 The image feature extractor shown extracts relevant indicators for evaluating image quality and determines which one or more image processing methods are required based on the above relevant indicators;

[0081] S203: Take the image features and classification labels extracted by the feature extractor of the training set as input Figure 6 The multi-layer perceptron network is trained and the parameters in the network are continuously optimized through training;

[0082] S204: Using the trained multi-layer perceptron network to classify light field images that require different processing methods.

[0083] In step S201, a portion of the captured light field images is selected and annotated based on the desired processing algorithm (e.g., denoising, contrast enhancement, color enhancement, super-resolution, or a combination of two, three, or four). The images are then classified into 15 categories. Additional samples are generated through methods such as rotation, translation, scaling, and color change, resulting in a training set of 600 images per category. Another portion of the light field images is then rotated, translated, scaled, and color-changed to create a training set of 1,000 images.

[0084] The 15 categories are: denoising, contrast enhancement, color enhancement, super-resolution, denoising-contrast enhancement, denoising-color enhancement, denoising-super-resolution, contrast enhancement-color enhancement, contrast enhancement-super-resolution, color enhancement-super-resolution, denoising-contrast enhancement-color enhancement, denoising-contrast enhancement-super-resolution, denoising-color enhancement-super-resolution, contrast enhancement-color enhancement-super-resolution, denoising-contrast enhancement-color enhancement, denoising-contrast enhancement-super-resolution, denoising-color enhancement-super-resolution, contrast enhancement-color enhancement-super-resolution, denoising-contrast enhancement-color enhancement-super-resolution.

[0085] like Figure 5 As shown, in step S202, parameters related to image quality are input into the feature extractor to determine whether the image needs denoising. The features that can be extracted include but are not limited to local entropy calculation, noise intensity extraction, etc. To determine whether the image needs contrast enhancement processing, the features that can be extracted include but are not limited to grayscale histograms. Grayscale information can be extracted from the grayscale histogram at equal or unequal intervals. To determine whether the image needs color enhancement processing, the features that can be extracted include but are not limited to color histograms. Color information can be extracted from the color histogram at equal or unequal intervals. To determine whether the image needs super-resolution processing, Sobel edge calculation or other methods can be used to extract the edge information of the image. The sharper the edge, the richer the image details, which usually indicates a higher resolution. The above only lists a few methods for evaluating image quality. Users can choose other feature extraction indicators according to their needs. After the feature extraction is completed, multi-feature fusion and K-Means clustering are performed.

[0086] In step S203, the input of the multi-layer perceptron network is the extracted feature information and image classification label, and the output is one of the 15 classification methods required for the image.

[0087] The steps for implementing denoising in the denoising unit are as follows:

[0088] S301: extracting sub-aperture images and macro-pixels from the original light field image through an image pre-processing unit, wherein the sub-aperture image contains spatial information and the macro-pixels contain angular information;

[0089] S302: Using a convolutional autoencoder to extract texture structure information from the sub-aperture image and macro pixels;

[0090] S303: Perform two-dimensional discrete inverse cosine transform on the sub-aperture image and the macro pixel to obtain a frequency map;

[0091] S304: Frequencies are sorted and combined into four categories of three-dimensional frequency maps. The first category contains the frequency map that retains the basic information of the image, which is the majority of the continuously changing area, i.e., the DC component in the frequency map. The second, third, and fourth categories are the high-frequency portion of the frequency map, i.e., the rapidly changing portion such as edge lines. The frequency map can be divided into three parts evenly or unevenly based on the frequency distribution. That is, the second category contains the frequency map with lower frequencies in the high-frequency portion, the third category contains the frequency map with medium frequencies in the high-frequency portion, and the fourth category contains the frequency map with higher frequencies in the high-frequency portion.

[0092] S305: Input the above four types of frequency maps into the three-dimensional residual network respectively to perform frequency domain denoising;

[0093] S306: Convert the denoised frequency map into a pixel domain image through a two-dimensional inverse cosine transform, and fuse the pixel domain image with the light field image with texture structure information extracted in advance to obtain a denoised image.

[0094] like Figure 8 As shown in Figure 1, the convolutional autoencoder consists of an encoder part, a latent space layer, and a decoder part.

[0095] The encoder compresses the input image into a low-dimensional feature representation, preserving the most expressive texture and structural features. The first layer of the encoder is a convolutional layer, which extracts primary texture features such as local edges, lines, and corners. The second layer is an activation and pooling layer, using the ReLU function as the activation function to increase the network's nonlinearity and preserve edge variations. The pooling layer compresses the spatial size, enhances local translation invariance, and preserves the main texture structure. The third and fourth layers are convolutional and pooling layers, which extract more complex texture combinations and structural features. The feature map size is reduced, but the number of channels is increased, and the dimensionality is enriched. The number of convolutional and pooling layers can be increased as needed.

[0096] The latent space layer is the connection layer between the encoder and decoder, and is a compressed representation of the image. It represents the core texture structure of the entire image and is a compact representation of the image space texture structure.

[0097] The decoder is responsible for restoring the structure and reconstructing the image. Composed of multiple layers of upsampling and deconvolution, the decoder simulates the reverse encoding process, gradually restoring the original image and mapping the abstracted structural information back to the original texture level.

[0098] The steps of contrast enhancement unit to achieve contrast enhancement are as follows:

[0099] S401: converting the original light field image into a sub-aperture image through an image preprocessing unit;

[0100] S402: Divide the sub-aperture image into m×m small blocks. The entire image is divided into three regions: corner region (CR), border region (BR), and interior region (IR).

[0101] S403: Counting the number of pixels of each gray level in the image, with the gray level as the horizontal axis and the frequency of each gray level as the vertical axis, to obtain a histogram of each small block;

[0102] S404: Calculate the shear limit value β and redistribute the histogram based on this value so that its height does not exceed the shear limit value β. The calculation formula of β is as follows:

[0103]

[0104] Where M is the number of pixels in each region, N is the number of gray levels (a fixed value), α is the clipping factor, and S max is the maximum allowable slope, which can be obtained through experiments.

[0105] S405: We define the cumulative distribution function CDF of the histogram as follows:

[0106]

[0107] Where n = 0, 1, 2, ..., N-1, M is the number of pixels in each area, N is the number of gray levels (a fixed value), h i,j (k) is the histogram of the (i, j) region;

[0108] S406: Interpolate the grayscale value of each pixel and perform weighted calculations to obtain a final grayscale value.

[0109] In step S406, different calculation methods are used for different regions. Figure 10 The grayscale value calculation formula for areas 1, 2, and 3 in is as follows:

[0110] Grayscale value of area 1 (corner area):

[0111] p new =f i,j (p old );

[0112] where p new is the new grayscale value of pixel p, f i,j (p old ) is the cumulative distribution function of the original grayscale histogram of the (i, j) region, p old is the original grayscale value of the pixel.

[0113] Grayscale value of region 2 (inner region):

[0114]

[0115] Where x, y, s, and r are Figure 10 The distance from the pixel to each boundary in the region, f i-1,j-1 (p old ) is the cumulative distribution function of the original grayscale histogram of the (i-1, j-1) region, f i,j-1 (p old ) is the cumulative distribution function of the original grayscale histogram of the (i, j-1) region, f i-1,j (p old ) is the cumulative distribution function of the original grayscale histogram of the (i-1, j) region, f i,j (p old ) is the cumulative distribution function of the original grayscale histogram of the (i, j) region;

[0116] Grayscale value of region 3 (border):

[0117]

[0118] Where s and r are Figure 10 The distance from the pixel to the boundary represented by .

[0119] The steps for the color enhancement unit to achieve color enhancement are as follows:

[0120] S501: Decomposing the original light field image into a structure component and a texture component using a structure-texture decomposition model;

[0121] S502: Convert the structural component from RGB space to Lab space, separating the luminance and chrominance channels. Calculate color similarity using the color difference formula, select the first P main colors as the initial cluster centers, and remove isolated pixels after iterative optimization to generate uniformly illuminated sub-blocks {B1, B2, B3}.

[0122] S503: Extract the texture components through LBP features to generate a texture weight map W T ;

[0123] S504: The obtained sub-block {B1, B2, B3} is input into a network consisting of 16 RCAB residual modules connected in series for processing. The RCAB module consists of a Conv layer, a ReLU activation layer, a channel attention layer, and finally a residual connection;

[0124] S505: Perform multi-scale feature fusion on the enhanced structural components {B1', B2', B3'} and the texture weight map, convert them back to RGB space, and obtain the enhanced image after gamma correction.

[0125] The structure-texture decomposition model can adopt the total variation (TV) decomposition model, the goal of which is to decompose the image I into the structure component u and the texture component v, I = u + v, as follows:

[0126]

[0127] in is the total variation of the image structure, ‖v‖ X is a measure of texture, and λ is a tuning parameter used to control the weights of the structure component and the texture component.

[0128] Select TV-L1 model, select Right now:

[0129]

[0130] Introducing auxiliary variables but:

[0131]

[0132] By alternating between u and d, we can obtain the structural component u, and the texture component v through the formula v=Iu.

[0133] The steps for super-resolution processing by the super-resolution unit are as follows:

[0134] S601: The original light field image is converted into a sub-aperture image (i.e., a low-resolution image, LR image) through an image preprocessing unit;

[0135] S602: Use the bicubic interpolation algorithm to perform preliminary magnification on the sub-aperture image to generate a pre-super-resolution image. Multi-scale convolution is performed on the pre-super-resolution image using Gaussian convolution kernels of different scales (e.g., σ = 1, 2, 3) to generate a series of feature maps of different scales, capturing the multi-level information of the image.

[0136] S603: Construct parallel 3×3 and 5×5 convolutional kernel networks to extract local details and larger-scale context features, respectively. Feature maps extracted by convolution kernels of different sizes are jointly processed to enhance the diversity of feature expression.

[0137] S604: Calculate the number of transitions, the total number of edges, and the number of center pixels. The number of transitions is obtained by counting the number of grayscale value mutations within the pixel neighborhood. The total number of edges is obtained by detecting the number and distribution of edge structures in the image. The number of center pixels is obtained by dynamically adjusting the classification threshold based on the difference between the center pixel and the neighborhood. Based on the above indicators, the image is divided into three types of areas: smooth area (low transitions, few edges), edge area (significant edge structure), and complex texture area (high transitions, dense details).

[0138] S605: Pre-train multiple sets of filter vectors for different texture types (smooth, edge, complex). Based on the real-time classification results, dynamically select the optimal filter kernel for each area, such as using a denoising kernel for smooth areas and an edge-preserving kernel for edge areas.

[0139] S606: Capture high-frequency detail features through a parallel structure of 3×3 and 5×5 convolution kernels. Input the multi-scale feature map extracted in step S602 to generate a high-frequency detail map containing sharp edges and textures.

[0140] S607: The filtered main image outputted in step S605 and the high-frequency detail image outputted in step S606 are superimposed according to the weights. The formula is as follows:

[0141] I fused =a·I filtered +b·I detail ;

[0142] The weights a and b are adaptively adjusted according to regional characteristics. For example, b is increased in complex texture areas to enhance details. filtered is the main image after filtering, I detail It is a high-frequency detail map;

[0143] S608: Using the differential results of Gaussian kernels of different scales to perform differential Gaussian filtering on the fused image, so as to suppress noise while sharpening edges, and finally generate a high-resolution image.

[0144] An embodiment of the present application further provides a computer device / equipment / system, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0145] This application uses light field images as a basis, taking into account the low image quality problem caused by light field equipment acquisition, and establishes a light field image processing pipeline. It also uses neural network-based image classification to achieve adaptive image processing.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An adaptive light field image enhancement pipeline system, characterized by: It includes a light field acquisition device, a light field image processing device, a light field display device and a DDR3 storage unit. The light field acquisition device is used to obtain the original light field image and transmit it to the light field image processing device; The light field image processing device includes an image receiving module, an image preprocessing unit, an image classification unit, a denoising unit, a contrast enhancement unit, a color enhancement unit, a super-resolution unit and an image transmission unit. The image receiving module receives the original light field image transmitted by the light field acquisition device and stores it in the DDR3 storage unit. The image preprocessing unit converts the original light field image into different types of light field images as needed. The image classification unit completes the classification task of the preprocessed light field image according to the quality of the light field image and determines the processing method required for the image. The denoising unit, contrast enhancement unit, color enhancement unit and super-resolution unit respectively complete image denoising, contrast enhancement, color enhancement and super-resolution processing. The light field display device is used to display the light field image processed by the light field image processing device.

2. The adaptive light field image enhancement pipeline system according to claim 1, characterized in that: The light field acquisition device includes an optical system module, an image sensor module and an image information transmission module. The optical system module consists of a main lens, a microlens array, a color filter array and a sensor array, and is used to obtain the position, direction and color information of the light field. The image sensor module is used to record the obtained light field position, direction and color information and generate an original light field image. The image information transmission module is used to save the original light field image generated by the image sensor module and transmit it to the light field image processing device.

3. The adaptive light field image enhancement pipeline system according to claim 1 or 2, characterized in that: It also includes an AI acceleration module, which is connected to the light field image processing device to accelerate the analysis and inference calculations of the image processing process.

4. An adaptive light field image enhancement pipeline method, characterized by: The adaptive light field image enhancement pipeline system according to any one of claims 1 to 3 comprises the following steps: S1: Use light field acquisition equipment to complete the acquisition of original light field images; S2: Perform image classification based on the processing method required for light field images, establish training sets and test sets, train a neural network model based on the training sets, and complete the image classification task based on the model; S3: Complete image denoising in the light field image processing pipeline through the denoising unit; S4: completing image contrast enhancement in the light field image processing pipeline through the contrast enhancement unit; S5: Complete image color enhancement in the light field image processing pipeline through the color enhancement unit; S6: Complete the image super-resolution processing in the light field image processing pipeline through the super-resolution unit.

5. The adaptive light field image enhancement pipeline method according to claim 4, characterized in that: Step S2 specifically includes: S201: Use a microlens array provided in a light field image acquisition device to capture images of a desired light field area, obtain original light field images, and establish light field image training sets and test sets for various scenes; S202: Build an image feature extractor to extract relevant indicators for evaluating image quality; S203: Using the image features and classification labels extracted by the feature extractor from the training set as input, the neural network model is trained, and parameters in the model are continuously optimized through training; S204: Using the trained neural network model to classify light field images that require different processing methods.

6. The adaptive light field image enhancement pipeline method according to claim 4, characterized in that: Step S3 specifically includes: S301: extracting sub-aperture images and macro pixels from the original light field image through an image preprocessing unit; S302: Using a convolutional autoencoder to extract texture structure information from the sub-aperture image and macro pixels; S303: Perform two-dimensional discrete inverse cosine transform on the sub-aperture image and the macro pixel to obtain a frequency map; S304: sorting the frequency map into four categories according to frequency magnitude, wherein the first category contains the frequency map that retains basic information of the image, which is the majority of the area with continuous changes; the second, third, and fourth categories contain the high-frequency part of the frequency map; S305: Input the above four types of frequency maps into the three-dimensional residual network respectively to perform frequency domain denoising; S306: Convert the denoised frequency map into a pixel domain image through a two-dimensional inverse cosine transform, and fuse the pixel domain image with the light field image with texture structure information extracted in advance to obtain a denoised image.

7. The adaptive light field image enhancement pipeline method according to claim 4, characterized in that: Step S5 specifically includes: S501: Decomposing the light field image into a structure component and a texture component using a structure-texture decomposition model; S502: Convert the structural component from RGB space to Lab space, separate the luminance and chrominance channels, calculate color similarity using the color difference formula, select the first P main colors as the initial cluster centers, remove isolated pixels after iterative optimization, and generate a sub-block {B1, B2, B3} with uniform illumination; S503: Extract the texture components through LBP features to generate a texture weight map W T ; S504: The obtained sub-block {B1, B2, B3} is input into a network consisting of 16 RCAB residual modules connected in series for processing. The RCAB module consists of a Conv layer, a ReLU activation layer, a channel attention layer, and finally a residual connection; S505: Perform multi-scale feature fusion on the enhanced structural components {B1', B2', B3'} and the texture weight map, convert them back to RGB space, and obtain the enhanced image after gamma correction.

8. The adaptive light field image enhancement pipeline method according to claim 4, characterized in that: Step S6 specifically includes: S601: converting the original light field image into a sub-aperture image through an image preprocessing unit; S602: Using a bicubic interpolation algorithm to preliminarily enlarge the sub-aperture image to generate a pre-super-resolution image. Using Gaussian convolution kernels of different scales, multi-scale convolution is performed on the pre-super-resolution image to generate a series of feature maps of different scales, capturing multi-level information of the image. S603: Construct parallel 3×3 and 5×5 convolutional kernel networks to extract local details and larger-scale context features, respectively. Feature maps extracted by convolution kernels of different sizes are jointly processed to enhance the diversity of feature expression. S604: Calculate the number of jumps, the total number of edges, and the number of center pixels, where the number of jumps is obtained by counting the number of grayscale value mutations within a pixel neighborhood, the total number of edges is obtained by detecting the number and distribution of edge structures in the image, and the number of center pixels is obtained by dynamically adjusting the classification threshold based on the difference between the center pixel and the neighborhood; based on the above indicators, divide the light field image into three types of regions: smooth area, edge area, and complex texture area; S605: Pre-train multiple sets of filter vectors for different texture types, and dynamically select the optimal filter kernel for each region based on the real-time classification results; S606: Capture high-frequency detail features through a parallel structure of 3×3 and 5×5 convolution kernels, input the multi-scale feature map extracted in step S602, and generate a high-frequency detail map containing sharp edges and textures; S607: The filtered main image outputted in step S605 and the high-frequency detail image outputted in step S606 are superimposed according to the weights. The formula is as follows: I fused =a·I filtered +b·I detail ; Where a and b are weights, I filtered is the main image after filtering, I detail It is a high-frequency detail map; S608: Using the differential results of Gaussian kernels of different scales to perform differential Gaussian filtering on the fused image, so as to suppress noise while sharpening edges, and finally generate a high-resolution image.

9. The adaptive light field image enhancement pipeline method according to claim 6, characterized in that: The convolutional autoencoder in step S302 consists of an encoder part, a latent space layer, and a decoder part; The encoder part compresses the input image into a low-dimensional feature representation, retaining the most expressive texture and structural features. The first layer of the encoder part is a convolutional layer, which is used to extract the primary texture features of the image; The second layer is an activation and pooling layer, which is used to preserve edge changes and main texture structures; the third and fourth layers are convolution and pooling layers, which are used to extract more complex texture combinations and structural features; The latent space layer is the connection layer between the encoder and the decoder, which is a compact representation of the image spatial texture structure; The decoder part is used to restore the structure and reconstruct the image. The decoder part consists of multiple layers of upsampling and deconvolution layers, which simulates the reverse process of encoding, gradually restores the original image, and maps the abstract structural information back to the original texture level.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 4 to 9.

Images