Under-screen shot image restoration method and system based on lightweight dual-domain network

Abstract

The invention discloses an under-screen shot image recovery method and system based on a lightweight dual-domain network, and belongs to the technical field of image processing, and the method comprises the steps: obtaining an original degraded RGB image, and inputting the original degraded RGB image into a decomposition network; constructing an amplitude-frequency-phase-frequency transconductance module, performing weighted correction on a phase component by using a guide graph generated by amplitude residual, and inversely transforming the corrected amplitude and phase back to a spatial domain to obtain a frequency domain enhancement feature; constructing a multi-scale mixed cavity convolution module to obtain spatial domain enhancement features; and splicing the frequency domain enhancement feature and the spatial domain enhancement feature in a channel dimension, and adding the residual image and the original degraded image to obtain a restored clear image. According to the invention, the existing degradation problems of image blurring, noise interference, color shift and the like caused by screen pixel diffraction and low light transmittance are solved. According to the method, the texture detail recovery capability and the color fidelity of the under-screen shot image are remarkably improved while the light weight of the parameter quantity is ensured.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for under-display camera image restoration based on a lightweight dual-domain network. Background Technology

[0002] As consumer demand for full-screen display devices increases, smartphone manufacturers have begun exploring placing the front-facing camera under the screen (under-display camera technology, UDC) to achieve a larger screen-to-body ratio. Currently, the mainstream UDC display solutions include T-OLED and P-OLED. However, this imaging system faces significant optical challenges: light must pass through the display panel with its complex pixel arrangement and circuit structure before reaching the sensor, leading to various image degradation phenomena, primarily including:

[0003] 1) Severe diffraction effect: The screen pixel structure acts like a grating, causing complex diffraction diffusion of the point spread function (PSF), resulting in blurred images, loss of a large amount of high-frequency details, and glare around the light source.

[0004] 2) Low transmittance and noise: The screen blocks most of the light (transmittance is usually less than 20%), which significantly reduces the number of photons received by the sensor, resulting in severe shot noise and readout noise under low light conditions.

[0005] 3) Color shift: Different wavelengths of light have different attenuation rates when passing through the screen material, which leads to color shift problems caused by uneven spectral transmittance (such as the common yellow or green tint in P-OLED).

[0006] Existing UDC image restoration methods primarily rely on deep learning techniques. Early methods based on convolutional neural networks (CNNs) typically employed end-to-end image mapping strategies, lacking modeling of the imaging physics process, resulting in poor model interpretability. Furthermore, existing methods have the following limitations:

[0007] First, many methods process directly in the RGB domain, ignoring the physical differences in reflectivity (texture) and illumination (brightness) degradation in under-display camera imaging, making it difficult to achieve accurate decoupled restoration.

[0008] Second, although some studies have introduced frequency domain information, they are often just simple spectrum superpositions, failing to make full use of the structural semantics contained in the phase information, and failing to solve the coupling relationship between frequency domain amplitude and phase.

[0009] Third, in pursuit of high performance, existing models (such as Transformer-based Restormer) often have a huge number of parameters and extremely high computational complexity, making them difficult to deploy on mobile devices with extremely high requirements for real-time performance and power consumption.

[0010] Therefore, developing a UDC image restoration scheme that is physically interpretable, can fully utilize complementary information in the frequency and spatial domains, and is lightweight and efficient is a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0011] The purpose of this invention is to provide a method and system for under-display camera image restoration based on a lightweight dual-domain network. While ensuring a lightweight parameter count, it significantly improves the ability to restore texture details and color fidelity of under-display camera images, thus solving the problems mentioned in the background art.

[0012] To achieve the above objectives, the present invention provides the following technical solution:

[0013] A method for image restoration from an under-display camera based on a lightweight dual-domain network includes:

[0014] Image decomposition: The original degraded RGB image captured by the under-display camera system is obtained and input into a decomposition network based on Retinex theory to decouple and output the reflection map and illumination map;

[0015] Frequency domain restoration: Construct an amplitude-phase frequency cross-conduction module to transform the reflection map to the frequency domain, separate the amplitude component and the phase component, extract the amplitude residual and phase residual through a convolutional network, and use the guide map generated by the phase residual to perform weighted correction on the amplitude component and the guide map generated by the amplitude residual to perform weighted correction on the phase component. Finally, inverse transform the corrected amplitude and phase back to the spatial domain to obtain the frequency domain enhancement features.

[0016] Spatial domain restoration: Construct a multi-scale hybrid dilated convolution module to downsample the reflection map and illumination map at multiple scales. At each scale, spatial features are extracted using a hybrid dilated convolution block. The hybrid dilated convolution block is composed of dilated convolution layers with different dilation rates connected in series. Finally, the features at each scale are upsampled and fused to obtain spatial domain enhancement features.

[0017] Fusion reconstruction: Frequency domain enhancement features and spatial domain enhancement features are concatenated along the channel dimension, a residual image is generated through a fusion convolutional layer, and the residual image is added to the original degraded image to obtain the final restored clear image.

[0018] Preferably, in the image decomposition, the decomposition network includes consecutively stacked convolutional layers. The first few convolutional layers are followed by a ReLU activation function, and the last convolutional layer is followed by a Sigmoid activation function, outputting a feature map with 4 channels. The first 3 channels represent the reflection map R, and the 4th channel represents the illumination map I. This decomposition process satisfies physical constraints. , where S is the degraded image.

[0019] Preferably, in the frequency domain recovery, the processing procedure of the amplitude-phase frequency cross-conduction module is as follows:

[0020] The reflection map R is transformed into a frequency domain representation R_f = A + jP using the Fast Fourier Transform, where A is the amplitude spectrum and P is the phase spectrum;

[0021] A and P are processed separately by convolutional layers to obtain the refined amplitude A' and phase P', and the residuals are calculated:

[0022] A_res = A' - A

[0023] P_res = P' - P

[0024] Generate cross-differential weights using global average pooling and the Softmax function:

[0025] Pool_P = Softmax(Pooling(P_res))

[0026] Pool_A = Softmax(Pooling(A_res))

[0027] Perform mutual conduction correction, use phase information to guide amplitude recovery, and use amplitude information to guide phase recovery:

[0028] A_out = A'⊙Pool_P

[0029] P_out = P'⊙Pool_A

[0030] Where ⊙ represents element-wise multiplication;

[0031] Inverse Fast Fourier Transform is performed using A_out and P_out to output frequency domain enhanced features.

[0032] Preferably, in the spatial domain restoration, the multi-scale hybrid dilated convolution module contains at least two parallel branches of different scales;

[0033] For the medium-scale branch, the input feature map size is downsampled to 1 / 2 of the original image. The hybrid dilated convolutional block used contains three cascaded convolutional layers, each with a kernel size of 5×5 and dilation rates of 5, 3, and 1 respectively.

[0034] For the small-scale branch, the input feature map size is downsampled to 1 / 4 of the original image. The hybrid dilated convolutional block used contains three cascaded convolutional layers, each with a kernel size of 3×3, and the dilation rates are set to 3, 2, and 1 respectively.

[0035] The mixed dilatation rate setting is used to eliminate mesh artifacts caused by a single dilatation rate and to cover the continuous receptive field.

[0036] Preferably, during the training phase, a composite loss function L_total is used for optimization, and its formula is as follows:

[0037] L_total = L_rec +α·L_fre +β·L_ret +γ·L_gper

[0038] Wherein, L_rec is the reconstruction loss, which uses the L1 norm to calculate the difference between the restored image and the real image; L_fre is the frequency domain loss, which calculates the L1 distance between the amplitude spectrum and phase spectrum of the restored image and the real image in the frequency domain; L_ret is the Retinex consistency loss, which constrains the consistency between the decomposed reflection map and illumination map and the smoothness of the illumination map; L_gper is the global perception loss, which uses a pre-trained Transformer network to extract feature maps of the restored image and the real image and calculates the Euclidean distance between the feature maps.

[0039] Preferably, the formula for calculating the global perception loss L_gper is:

[0040] L_gper = || Trans(Y) - Trans(Y_hat) ||_2

[0041] Where Trans(·) represents the features extracted by the Vision Transformer model pre-trained on ImageNet, Y is the real clear image, and Y_hat is the restored image output by the network.

[0042] According to another aspect of the present invention, a lightweight dual-domain network-based under-display camera image restoration system is provided for implementing the above-described lightweight dual-domain network-based under-display camera image restoration method, comprising:

[0043] The image acquisition module is used to acquire the raw RGB image captured by the under-display camera module;

[0044] The decomposition module is equipped with a Retinex decomposition network to decouple the original image into reflection and illumination components.

[0045] The frequency domain enhancement module is equipped with an amplitude-phase frequency cross-conduction unit, which is used to recover the high-frequency details of the reflection component in the frequency domain using a cross residual guidance mechanism;

[0046] The spatial enhancement module is equipped with multi-scale hybrid dilated convolutional units, which are used to extract large receptive field features in the spatial domain using hybrid dilation rate convolution.

[0047] The fusion reconstruction module is used to fuse frequency domain and spatial domain features to generate a final clear image.

[0048] Preferably, the image acquisition module includes a CMOS image sensor installed below the display panel. The CMOS image sensor is connected to the system motherboard via MIPI CSI, and the light signal acquired by the CMOS image sensor is converted into digital raw data or RGB image data.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] This invention, based on Retinex theory decomposition and a dual-domain parallel processing architecture, enables the model to specifically repair degradation from both physical properties (reflection / illumination) and signal properties (frequency domain / spatial domain). Its amplitude-phase frequency cross-conduction module effectively solves the problem of phase and amplitude reconstruction imbalance through a unique cross-conduction mechanism, significantly improving the clarity of high-frequency textures in the image. The multi-scale hybrid dilated convolution module, through a hybrid dilation rate design, achieves a large effective receptive field while maintaining low computational cost, effectively removing large areas of hazy glare. Compared with large models such as Restormer, through carefully designed convolution combinations, it still achieves competitive performance with a significantly reduced number of parameters (e.g., less than 1M parameters), making it suitable for mobile applications. Attached Figure Description

[0051] Figure 1 This is a flowchart of the under-display camera image restoration method based on a lightweight dual-domain network according to the present invention;

[0052] Figure 2 This is a detailed structural diagram of the image decomposition network and multi-scale hybrid dilated convolution module of the present invention;

[0053] Figure 3 This is a detailed structural diagram of the amplitude-frequency-phase-frequency cross-conduction module of the present invention;

[0054] Figure 4 This is a schematic diagram of the frequency domain feature analysis of the clear image of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] To address the image blurring, noise interference, and color shift issues caused by screen pixel diffraction and low transmittance in existing under-display camera systems, please refer to [link to relevant documentation]. Figures 1-4 This embodiment provides the following technical solution:

[0057] An under-display camera image restoration system based on a lightweight dual-domain network includes: an image acquisition module, a decomposition module, a frequency domain enhancement module, a spatial domain enhancement module, and a fusion reconstruction module.

[0058] The system includes: an image acquisition module for acquiring the original RGB image captured by the under-display camera module; a decomposition module equipped with a Retinex decomposition network for decoupling the original image into reflection and illumination components; a frequency domain enhancement module equipped with an amplitude-phase frequency cross-conduction unit for recovering high-frequency details of the reflection component in the frequency domain using a cross-residual guidance mechanism; a spatial domain enhancement module equipped with a multi-scale hybrid dilated convolution unit for extracting large receptive field features in the spatial domain using hybrid dilation rate convolution; and a fusion and reconstruction module for fusing frequency and spatial domain features to generate the final clear image.

[0059] The image acquisition module includes a CMOS image sensor installed below the display panel. The CMOS image sensor is connected to the system motherboard via MIPI CSI (Mobile Industry Processor Interface - Camera Serial Interface). The light signal acquired by the CMOS image sensor is converted into digital raw data or RGB image data. Due to diffraction and attenuation when light passes through the display screen (such as a transparent OLED), the acquired data is the "degraded image" described in this embodiment. This interface is responsible for transmitting this data to the system's buffer storage area at high speed.

[0060] Before entering the deep neural network, the raw data usually needs to go through the basic ISP (Image Signal Processing) pipeline. The preprocessing unit in this embodiment can be implemented by a dedicated ISP chip or the ISP module in the SoC. Its main functions include: 1) Demosaicing, converting the raw data of the Bayer pattern into an RGB full-color image; 2) Automatic White Balance (AWB), correcting the basic color temperature deviation caused by the screen material; 3) Black level correction: subtracting dark current noise. The preprocessed RGB image is normalized to the [0,1] range and adjusted to the size required for the network input (such as 1024×1024 or 512×512).

[0061] This embodiment is also equipped with high-speed random access memory (RAM, such as LPDDR5) and non-volatile memory (Flash, such as UFS).

[0062] Flash: Used to persistently store the weight parameters of the LD²Net network model in this embodiment. Since this embodiment adopts a lightweight design, the number of model parameters is small (e.g., less than 1MB), so it occupies very little storage space, making it very suitable for mobile devices.

[0063] RAM: Used as an input / output buffer to temporarily store degraded image frames to be processed, feature maps of intermediate layers of the network, and the finally restored clear image.

[0064] This embodiment also includes a neural network acceleration unit (NPU / GPU), which is the core computing module of this embodiment. It is typically integrated into the SoC (System-on-a-Chip) of a mobile terminal. It can be a dedicated neural processing unit (NPU) or a general-purpose graphics processing unit (GPU). This unit is equipped with a parallel computing core for loading the LD²Net model in memory and performing the inference calculations described in Embodiment 1. Specifically, it performs the following operations: decomposition: using convolution operations to separate the RGB image into a reflection map and an illumination map; frequency domain transformation: using a built-in FFT hardware accelerator or software library to quickly calculate the amplitude and phase of the reflection map; cross-correlation and convolution: performing cross-correlation calculations in the APMG module and multi-scale dilated convolutions in the MHDC module in parallel; and fusion: calculating the final reconstruction result.

[0065] Thanks to the network's lightweight design, this unit can complete inference with low power consumption and latency, supporting real-time preview or fast photo processing for the under-display camera.

[0066] This embodiment also includes a display unit, comprising a display driver IC (DDIC) and a display panel (i.e., the T-OLED or P-OLED screen mentioned above). The restored high-quality and clear image is written into the framebuffer and transmitted to the display driver IC through the MIPI DSI interface, and finally presented to the user on the screen.

[0067] Specifically, by combining the Retinex physical model with deep learning, and through the design of a unique amplitude-phase frequency cross-conduction module and a multi-scale hybrid dilated convolution module, it achieves efficient and high-quality restoration of degraded images from under-display cameras, especially in texture detail reconstruction and color correction.

[0068] To better demonstrate the under-display camera image restoration method based on a lightweight dual-domain network, this embodiment provides an under-display camera image restoration method based on a lightweight dual-domain network, implemented based on the aforementioned under-display camera image restoration system based on a lightweight dual-domain network, including:

[0069] Image decomposition: The original degraded RGB image captured by the under-display camera system is obtained and input into a decomposition network based on Retinex theory to decouple and output the reflection map and the illumination map.

[0070] In this embodiment, the image decomposition network comprises consecutively stacked convolutional layers. The first few convolutional layers are followed by a ReLU activation function, and the last convolutional layer is followed by a Sigmoid activation function. The output is a feature map with 4 channels, where the first 3 channels represent the reflection map R and the 4th channel represents the illumination map I. This decomposition process satisfies physical constraints. , where S is the degraded image.

[0071] Specifically, a lightweight decomposition network is used to decompose the input degraded image S into a reflection map R and an illumination map I. This network consists of multiple convolutional layers and aims to decouple the texture details (reflection component) of the image from the brightness distribution (illumination component). This conforms to the physical imaging law, allowing subsequent modules to specifically handle the blur in the reflection map and the noise in the illumination map.

[0072] Frequency domain restoration: Construct an amplitude-phase-frequency cross-conduction module (APMG) to transform the reflection map to the frequency domain, separate the amplitude component and the phase component, extract the amplitude residual and phase residual through a convolutional network, and use the guide map generated by the phase residual to perform weighted correction on the amplitude component, and use the guide map generated by the amplitude residual to perform weighted correction on the phase component. Finally, inverse transform the corrected amplitude and phase back to the spatial domain to obtain the frequency domain enhancement feature.

[0073] In this embodiment, the processing procedure of the amplitude-phase frequency cross-conduction module in the frequency domain recovery is as follows:

[0074] The reflection map R is transformed into a frequency domain representation R_f = A + jP using the Fast Fourier Transform (FFT), where A is the amplitude spectrum and P is the phase spectrum.

[0075] A and P are processed separately by convolutional layers to obtain the refined amplitude A' and phase P', and the residuals are calculated:

[0076] A_res = A' - A

[0077] P_res = P' - P

[0078] Generate cross-differential weights using global average pooling and the Softmax function:

[0079] Pool_P = Softmax(Pooling(P_res))

[0080] Pool_A = Softmax(Pooling(A_res))

[0081] Perform mutual conduction correction, use phase information to guide amplitude recovery, and use amplitude information to guide phase recovery:

[0082] A_out = A'⊙Pool_P

[0083] P_out = P'⊙Pool_A

[0084] Where ⊙ represents element-wise multiplication;

[0085] Inverse Fast Fourier Transform (IFFT) is performed using A_out and P_out to enhance the frequency domain features.

[0086] Specifically, to address the severe spectral attenuation problem exhibited by under-display camera images in the frequency domain, an amplitude-phase frequency cross-conduction module was designed to process the reflection map R. The specific process is as follows:

[0087] Transformation and Separation: The reflection image is transformed to the frequency domain using Fast Fourier Transform (FFT) to obtain the amplitude spectrum A and the phase spectrum P. The amplitude spectrum mainly contains the brightness and color information of the image, while the phase spectrum contains the key structural information of the image.

[0088] Mutual guidance mechanism: This embodiment believes that amplitude and phase should promote each other during the recovery process. First, the amplitude residual A_res and the phase residual P_res are extracted by convolutional layers respectively. Then, global average pooling and Softmax operation are used to transform the phase residual into an amplitude guiding map Pool_P and the amplitude residual into a phase guiding map Pool_A.

[0089] Cross-correction: The amplitude is weighted using Pool_P and the phase is weighted using Pool_A. This cross-guided mechanism uses noise-resistant phase information to constrain the amplitude reconstruction, while using amplitude information to assist in the phase structure correction, thereby accurately recovering high-frequency details.

[0090] Spatial recovery: A multi-scale hybrid dilated convolutional module (MHDC) is constructed to downsample the reflection map and illumination map at multiple scales. At each scale, spatial features are extracted using a hybrid dilated convolutional block (HDCB). The HDCB is composed of dilated convolutional layers with different dilation rates connected in series. Finally, the features at each scale are upsampled and fused to obtain spatial enhancement features.

[0091] In this embodiment, the spatial domain recovery process includes a multi-scale hybrid dilated convolution module containing at least two parallel branches of different scales.

[0092] For the medium-scale branch, the input feature map size is downsampled to 1 / 2 of the original image. The hybrid dilated convolutional block used contains three cascaded convolutional layers, each with a kernel size of 5×5 and dilation rates of 5, 3, and 1 respectively.

[0093] For the small-scale branch, the input feature map size is downsampled to 1 / 4 of the original image. The hybrid dilated convolutional block used contains three cascaded convolutional layers, each with a kernel size of 3×3, and the dilation rates are set to 3, 2, and 1 respectively.

[0094] The mixed dilatation rate setting is used to eliminate mesh artifacts caused by a single dilatation rate and to cover the continuous receptive field.

[0095] Specifically, in order to jointly recover the reflection map and illumination map in the spatial domain and capture a wide range of contextual information (which is crucial for eliminating large-area glare and fogging), this embodiment designs a multi-scale hybrid dilated convolution module, the specific process of which is as follows:

[0096] Multi-scale architecture: Input features are downsampled to different scales (e.g., original resolution, 1 / 2, 1 / 4) to handle spatial degradation at different frequencies;

[0097] Hybrid dilated convolution: A hybrid dilation rate strategy is adopted on each scale branch. For example, in the medium-scale branch, convolutional layers with dilation rates of 5, 3, and 1 are stacked consecutively; in the small-scale branch, convolutional layers with dilation rates of 3, 2, and 1 are stacked. This zigzag dilation rate setting not only expands the receptive field but also fills the sampling holes caused by traditional dilated convolution, effectively eliminating the grid effect and ensuring the continuity of features.

[0098] Fusion reconstruction: Frequency domain enhancement features and spatial domain enhancement features are concatenated along the channel dimension, a residual image is generated through a fusion convolutional layer, and the residual image is added to the original degraded image to obtain the final restored clear image.

[0099] In this embodiment, during the training phase, a composite loss function L_total is used for optimization, and its formula is as follows:

[0100] L_total = L_rec +α·L_fre +β·L_ret +γ·L_gper

[0101] Wherein, L_rec is the reconstruction loss, which uses the L1 norm to calculate the difference between the restored image and the real image; L_fre is the frequency domain loss, which calculates the L1 distance between the amplitude spectrum and phase spectrum of the restored image and the real image in the frequency domain; L_ret is the Retinex consistency loss, which constrains the consistency between the decomposed reflection map and illumination map and the smoothness of the illumination map; L_gper is the global perception loss, which uses a pre-trained Transformer network to extract feature maps of the restored image and the real image and calculates the Euclidean distance between the feature maps.

[0102] In this embodiment, the global perception loss L_gper is calculated as follows:

[0103] L_gper = || Trans(Y) - Trans(Y_hat) ||_2

[0104] Where Trans(·) represents the features extracted by the Vision Transformer model pre-trained on ImageNet, Y is the real clear image, and Y_hat is the restored image output by the network.

[0105] Specifically, the frequency domain enhanced features are fused with the spatial domain enhanced features, and the final restored image is generated through residual connections. During training, a Transformer-based global perceptual loss function (L_gper) is introduced. Unlike the traditional VGG-based perceptual loss, L_gper uses a VisionTransformer network pre-trained on ImageNet to extract features. Since the Transformer has a global receptive field, this loss function can more effectively constrain the overall structure and semantic consistency of the restored image, significantly improving visual quality.

[0106] like Figure 1 As shown, it mainly includes four core steps: image decomposition, frequency domain processing, spatial domain processing, and feature fusion.

[0107] Step S1: Image decomposition based on Retinex:

[0108] Acquire the RGB degraded image captured by the under-display camera module, with dimensions denoted as H×W×3, and input the image into the decomposition network;

[0109] The specific structure of the decomposition network is as follows: Figure 2 As shown, it contains 4 convolutional layers:

[0110] Layer 1: 9×9 convolutional kernel, 64 output channels, ReLU activation;

[0111] Layer 2: 3×3 convolutional kernels, 64 output channels, ReLU activation;

[0112] Layer 3: 3×3 convolutional kernels, 64 output channels, ReLU activation;

[0113] Layer 4: 3×3 convolutional kernels, 4 output channels, Sigmoid activation.

[0114] Of the four output channels, the first three are defined as reflection maps R (corresponding to RGB three channels), and the fourth channel is defined as illumination maps I (corresponding to single-channel brightness). The Sigmoid function ensures that the output value is within the range of [0, 1].

[0115] Step S2, Amplitude-Phase Frequency Interconductance Processing (APMG):

[0116] This step aims to recover the details of the reflectance map R in the frequency domain, and the process is as follows: Figure 3 As shown, its design principle is based on Figure 4 The frequency domain characteristics are shown.

[0117] Theoretical basis: such as Figure 4 As shown, for a clear image (a), its amplitude information (b) mainly presents a smooth brightness and color distribution, while the phase information (c) retains the edges and texture structure of objects (such as animal fur and eye outlines) very clearly. The spectrum (d) shows that the clear image has rich high-frequency components. Due to the diffraction effect, the under-display camera system severely attenuates the high-frequency components in (d) and destroys the phase structure in (c). Therefore, the core of this module is to use the cross-conduction mechanism to restore the phase structure and amplitude color.

[0118] Frequency domain transformation: Perform a Fast Fourier Transform (FFT) on the reflection map R to obtain the frequency domain feature R_f. According to Euler's formula, R_f can be expressed as the real part Re and the imaginary part Im, and then the amplitude and phase can be calculated.

[0119] A(u,v) = sqrt(Re² + Im²);

[0120] P(u,v) = arctan(Im / Re).

[0121] Feature refinement: Input the magnitude A into a subnetwork containing convolutional layers to obtain the refined magnitude A';

[0122] The phase P is input into another symmetric subnetwork to obtain the refined phase P';

[0123] Calculate the amplitude residual and phase residual:

[0124] A_res = A' - A;

[0125] P_res = P' - P;

[0126] Cross-guided weight generation: Global average pooling is performed on the phase residual P_res, and then passed through the Softmax layer to generate the amplitude guiding weight Pool_P;

[0127] The amplitude residual A_res is subjected to global average pooling, and then passed through a Softmax layer to generate the phase guiding weight Pool_A;

[0128] Mutual conduction correction: Amplitude correction using phase-guided weights.

[0129] A_out = A' ⊙ Pool_P;

[0130] Phase correction using amplitude-guided weights:

[0131] P_out = P' ⊙ Pool_A;

[0132] Inverse Transform: Reconstruct the frequency domain features based on the corrected amplitude A_out and phase P_out, and use the inverse fast Fourier transform (IFFT) to transform them back to the spatial domain, outputting the frequency domain enhanced features F_freq.

[0133] Step S3: Multi-scale Hybrid Dilated Convolution Processing (MHDC):

[0134] This step involves the joint processing of the reflection map R and the illumination map I in the spatial domain.

[0135] Multi-scale input: Concatenate R and I and downsample to generate two scale inputs: medium scale (1 / 2 of the original image) and small scale (1 / 4 of the original image).

[0136] Hybrid Hollow Convolutional Blocks (HDCB):

[0137] Medium-scale branch: contains 3 convolutional layers;

[0138] Layer 1: Convolution kernel 5×5, dilation rate 5, LeakyReLU activation;

[0139] Layer 2: Convolution kernel 5×5, dilation rate 3, LeakyReLU activation;

[0140] Layer 3: Convolution kernel 5×5, dilation rate 1, LeakyReLU activation.

[0141] This 5-3-1 dilatation ratio combination can cover a large receptive field without holes.

[0142] Small-scale branch: contains 3 convolutional layers;

[0143] Layer 1: 3×3 convolution kernels, dilation rate of 3, LeakyReLU activation;

[0144] Layer 2: 3×3 convolution kernels, dilation rate of 2, LeakyReLU activation;

[0145] Layer 3: Convolution kernel 3×3, dilation rate 1, LeakyReLU activation.

[0146] Feature fusion: Upsample the output of each branch to the original size and fuse it with the features at the original scale to output the spatially enhanced feature F_spatial.

[0147] Step S4, Fusion and Reconstruction:

[0148] The frequency domain feature F_freq and the spatial domain feature F_spatial are concatenated along the channel dimension and input into a fusion module (containing 3 layers of 3×3 convolutional layers), and the output is the residual image R_final;

[0149] Final image recovery: Y_hat = S + R_final;

[0150] Where S is the original degraded image.

[0151] Step S5, Model Training:

[0152] The network is trained in an end-to-end manner, and the loss function L_total is defined as follows:

[0153] L_total = L_rec + 0.1·L_fre + 0.01·L_ret + 0.05·L_gper;

[0154] Where L_rec: reconstruction loss, || Y - Y_hat ||_1;

[0155] L_fre: Frequency domain loss, || A(Y) - A(Y_hat) ||_1 + || P(Y) - P(Y_hat) ||_1;

[0156] L_ret: Retinex loss, constraining the rationality of the decomposition of R and I and the smoothness of I (using a total variation model);

[0157] L_gper: Global Perception Loss. Y and Y_hat are input into a pre-trained Transformer network to extract deep feature maps and calculate their L2 distance. This helps the network focus on the global semantics and texture structure of the image.

[0158] In summary, the decomposition and dual-domain parallel processing architecture based on Retinex theory enables the model to specifically repair degradation from both physical properties (reflection / illumination) and signal properties (frequency domain / spatial domain). Its amplitude-phase frequency conversion module effectively solves the problem of phase and amplitude reconstruction imbalance through a unique conversion mechanism, significantly improving the clarity of high-frequency textures in the image. The multi-scale hybrid dilated convolution module, through a hybrid dilation rate design, achieves a large effective receptive field while maintaining low computational cost, effectively removing large areas of hazy glare. Compared with large models such as Restormer, through carefully designed convolution combinations, it still achieves competitive performance with a significantly reduced number of parameters (e.g., less than 1M parameters), making it suitable for mobile applications.

[0159] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0160] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for image restoration from an under-display camera based on a lightweight dual-domain network, characterized in that, include: Image decomposition: The original degraded RGB image captured by the under-display camera system is obtained and input into a decomposition network based on Retinex theory to decouple and output the reflection map and illumination map; Frequency domain restoration: Construct an amplitude-phase frequency cross-conduction module to transform the reflection map to the frequency domain, separate the amplitude component and the phase component, extract the amplitude residual and phase residual through a convolutional network, and use the guide map generated by the phase residual to perform weighted correction on the amplitude component and the guide map generated by the amplitude residual to perform weighted correction on the phase component. Finally, inverse transform the corrected amplitude and phase back to the spatial domain to obtain the frequency domain enhancement features. Spatial domain restoration: Construct a multi-scale hybrid dilated convolutional module to downsample the reflection map and illumination map at multiple scales. At each scale, spatial features are extracted using a hybrid dilated convolutional block. The hybrid dilated convolutional block is composed of dilated convolutional layers with different dilation rates. Finally, the features at each scale are upsampled and fused to obtain spatial domain enhancement features. Fusion reconstruction: Frequency domain enhancement features and spatial domain enhancement features are concatenated along the channel dimension, a residual image is generated through a fusion convolutional layer, and the residual image is added to the original degraded image to obtain the final restored clear image.

2. The under-display camera image restoration method based on a lightweight dual-domain network according to claim 1, characterized in that, In the image decomposition, the decomposition network comprises consecutively stacked convolutional layers. The first few convolutional layers are followed by a ReLU activation function, and the last convolutional layer is followed by a Sigmoid activation function. The output is a feature map with 4 channels, where the first 3 channels represent the reflection map R, and the 4th channel represents the illumination map I. This decomposition process satisfies physical constraints. , where S is the degraded image.

3. The under-display camera image restoration method based on a lightweight dual-domain network according to claim 2, characterized in that, In the frequency domain recovery, the processing procedure of the amplitude-phase frequency cross-conduction module is as follows: The reflection map R is transformed into a frequency domain representation R_f = A + jP using the Fast Fourier Transform, where A is the amplitude spectrum and P is the phase spectrum; A and P are processed separately by convolutional layers to obtain the refined amplitude A' and phase P', and the residuals are calculated: A_res = A' - A; P_res = P' - P; Generate cross-differential weights using global average pooling and the Softmax function: Pool_P = Softmax(Pooling(P_res)); Pool_A = Softmax(Pooling(A_res)); Perform mutual conduction correction, use phase information to guide amplitude recovery, and use amplitude information to guide phase recovery: A_out = A'⊙Pool_P; P_out = P'⊙Pool_A; Where ⊙ represents element-wise multiplication; Inverse Fast Fourier Transform is performed using A_out and P_out to output frequency domain enhanced features.

4. The under-display camera image restoration method based on a lightweight dual-domain network according to claim 3, characterized in that, In the spatial domain restoration, the multi-scale hybrid dilated convolution module contains at least two parallel branches of different scales; For the medium-scale branch, the input feature map size is downsampled to 1 / 2 of the original image. The hybrid dilated convolutional block used contains three cascaded convolutional layers, each with a kernel size of 5×5 and dilation rates of 5, 3, and 1 respectively. For the small-scale branch, the input feature map size is downsampled to 1 / 4 of the original image. The hybrid dilated convolutional block used contains three cascaded convolutional layers, each with a kernel size of 3×3 and dilation rates of 3, 2, and 1 respectively. The mixed dilatation rate setting is used to eliminate mesh artifacts caused by a single dilatation rate and to cover the continuous receptive field.

5. The under-display camera image restoration method based on a lightweight dual-domain network according to claim 4, characterized in that, During the training phase, a composite loss function L_total is used for optimization, and its formula is as follows: L_total = L_rec +α·L_fre +β·L_ret +γ·L_gper; Wherein, L_rec is the reconstruction loss, which uses the L1 norm to calculate the difference between the restored image and the real image; L_fre is the frequency domain loss, which calculates the L1 distance between the amplitude spectrum and phase spectrum of the restored image and the real image in the frequency domain; L_ret is the Retinex consistency loss, which constrains the consistency between the decomposed reflection map and illumination map and the smoothness of the illumination map; L_gper is the global perception loss, which uses a pre-trained Transformer network to extract feature maps of the restored image and the real image and calculates the Euclidean distance between the feature maps.

6. The under-display camera image restoration method based on a lightweight dual-domain network according to claim 5, characterized in that, The formula for calculating the global perception loss L_gper is: L_gper = || Trans(Y) - Trans(Y_hat) ||_2; Where Trans(·) represents the features extracted by the Vision Transformer model pre-trained on ImageNet, Y is the real clear image, and Y_hat is the restored image output by the network.

7. A lightweight dual-domain network-based under-display camera image restoration system, used to implement the lightweight dual-domain network-based under-display camera image restoration method as described in claim 6, characterized in that, include: The image acquisition module is used to acquire the raw RGB image captured by the under-display camera module; The decomposition module is equipped with a Retinex decomposition network to decouple the original image into reflection and illumination components. The frequency domain enhancement module is equipped with an amplitude-phase frequency cross-conduction unit, which is used to recover the high-frequency details of the reflection component in the frequency domain using a cross residual guidance mechanism; The spatial enhancement module is equipped with multi-scale hybrid dilated convolutional units, which are used to extract large receptive field features in the spatial domain using hybrid dilation rate convolution. The fusion reconstruction module is used to fuse frequency domain and spatial domain features to generate a final clear image.

8. The under-display camera image restoration system based on a lightweight dual-domain network according to claim 7, characterized in that, The image acquisition module includes a CMOS image sensor installed below the display panel. The CMOS image sensor is connected to the system motherboard via MIPI CSI. The light signal acquired by the CMOS image sensor is converted into digital raw data or RGB image data.

Images