A method for under-screen camera image restoration based on a U-shaped dynamic network

Abstract

The application discloses a kind of under-screen camera image restoration methods based on U type dynamic network.The method comprises the following steps: collecting target image;Target image is input to trained image restoration model, and reconstruction image is obtained.The image restoration model includes basic network, conditional branch and kernel branch, the basic network is used to extract the multiscale information of input image;The conditional branch is used to adaptively modulate the intermediate feature extracted by basic network, to generate different spatial resolution condition characteristics for input image;The kernel branch is based on the feature that input image and point spread function feature are merged in channel dimension, generates different spatial resolution dynamic convolution kernel;During the forward propagation process of the basic network to input image, the conditional feature and the dynamic convolution kernel are integrated into the set position.The application improves the quantitative performance and visual quality of image restoration.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method for under-display camera image restoration based on a U-shaped dynamic network. Background Technology

[0002] Under-display camera (UDC) is a novel imaging system that mounts a display screen onto a traditional digital camera lens. It enables full-screen display without the need for punch holes or cutouts, providing a better user experience and thus attracting widespread attention in the industry. UDC utilizes the high light transmittance of OLED screens, allowing the phone's content to be displayed normally on the OLED when not taking photos. When taking photos, external light passes through the OLED screen to create the image.

[0003] However, retaining the full functionality of an imaging sensor under a display is relatively difficult, as the display inevitably affects the light propagation process. During imaging, light must pass through the screen covering the camera, resulting in various forms of optical diffraction and interference. Therefore, images captured by UDC systems often contain flares, haze, blur, and noise. Furthermore, in real-world scenarios, UDC images are often captured in high dynamic range (HDR) scenes, where severe oversaturation occurs in highlight areas. Halo and blur phenomena in UDC images significantly impact the user experience.

[0004] Image restoration tasks aim to recover clean, high-quality images from degraded image offsets, such as denoising, deblurring, super-resolution, and HDR reconstruction. Similar to these tasks, UDC image restoration aims to reconstruct degraded images generated by UDC systems. To model the complex degradation process of UDC systems, existing techniques suggest utilizing a special diffraction blur kernel for image recovery, namely the point spread function (PSF). For example, the UDC image restoration task can be viewed as an inversion problem given a precise measurement of the PSF. Another approach is to model the image generation process of UDC systems using deconvolution-based pipelines (DeP) and data-driven learning methods to solve the UDC image restoration problem. These variants of UNet lack consideration for HDR scenes in data generation and PSF measurement, resulting in images captured by UDCs often containing noise, flares, haze, and blur artifacts. Although existing methods have made some progress in image restoration, the quality of restored images still needs improvement. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for under-display camera image restoration based on a U-shaped dynamic network. This method includes the following steps:

[0006] Acquire target image;

[0007] The target image is input into a trained image restoration model to obtain a reconstructed image;

[0008] The image restoration model includes a base network, a conditional branch, and a kernel branch. The base network is used to extract multi-scale information from the input image. The conditional branch is used to adaptively modulate the intermediate features extracted by the base network to generate conditional features with different spatial resolutions for the input image. The kernel branch generates dynamic convolutional kernels with different spatial resolutions based on the features of the input image and the point spread function features merged in the channel dimension. During the forward propagation of the input image by the base network, the conditional features and the dynamic convolutional kernels are integrated into a set position.

[0009] Compared with existing technologies, the advantages of this invention lie in proposing a novel deep network model that can be used to solve the image restoration problem of under-display camera systems with known point spread function (PSF) in HDR scenes. The provided network model includes a base network utilizing multi-scale information, a conditional branch for spatial deformation modulation, and a kernel branch providing prior knowledge of the given PSF. Furthermore, based on the characteristics of HDR data, a tone mapping loss is further designed for the network model to stabilize its optimization and improve the visual quality of the restored image.

[0010] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0012] Figure 1 This is a flowchart of an under-display camera image restoration method based on a U-shaped dynamic network according to an embodiment of the present invention;

[0013] Figure 2 This is a network structure diagram of an image restoration model according to an embodiment of the present invention;

[0014] Figure 3 This is an image restoration effect diagram according to an embodiment of the present invention;

[0015] In the attached diagram, Conv represents convolution; Down-sampling represents downsampling; Up-sampling represents upsampling; ResidualBlock represents residual block; Residual SFT Block represents residual spatial feature transformation block; Dynamic Conv represents dynamic convolution; Element-wise Sum represents element-wise addition; Concatenate represents concatenation; and Element-wise Multiply represents element-wise multiplication. Detailed Implementation

[0016] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0017] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0018] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0019] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0021] See Figure 1 As shown, the provided under-display camera image restoration method based on U-shaped dynamic networks includes the following steps:

[0022] Step S110: Obtain the dataset and the point spread function (PSF).

[0023] The point spread function (PSF) is the irradiance distribution of a point source in object space after passing through an optical system. It describes the response of an imaging system to a point source or point object. Traditional image restoration methods typically estimate the PSF first, and then estimate the original sharp image. Prior knowledge of the PSF has been proven effective for image restoration. In one embodiment of the present invention, to reduce model training costs, a public dataset and a public PSF are used.

[0024] Step S120: Construct an image restoration model, which includes a base network for extracting multi-scale information, a conditional branch for modulating spatial transformations of different regions, and a kernel branch for generating dynamic convolutional kernels.

[0025] Figure 2 This is an example of the proposed novel image restoration model, or UDC-UNet model, which can be used to restore images captured by UDC in HDR scenes. From top to bottom, the model framework generally includes a conditional branch, a base network, and a kernel branch, where C represents the number of channels and K represents the dynamic convolutional kernel size. First, a base network is constructed to extract multi-scale information hierarchically. Then, to achieve spatial variation modulation of different regions under different exposures, multiple Spatial Feature Transform (SFT) layers are used to construct the conditional branch. In addition, a kernel branch is added, which uses PSF to refine the intermediate features of the base network. Preferably, considering the data characteristics of HDR images, a novel tone mapping loss is designed to normalize image values ​​to [0,1]. This approach not only balances the influence of pixels with different intensities but also stabilizes the model training process.

[0026] In one embodiment, a U-shaped structure is used to construct the base network for extracting features at different scales, which can be divided into shallow features and deep features. Shallow features are extracted progressively by the shallow network through an ever-growing receptive field, and the shallow network maps the input image into a high-dimensional representation. Deep features are learned during the decoding process. Furthermore, the skip connections of the U-shaped network can effectively combine shallow and deep features. This U-shaped structure helps the network fully utilize the hierarchical, multi-scale information of the input image.

[0027] For example, for the base network, the feature channel dimension C can be set to 32 (setting C to 20 yields UDC-UNet). S The version, or simplified version of UDC-UNet), has the number of Residual SFT Blocks in the base network set to [2, 2, 2, 8, 2, 2, 2].

[0028] In HDR scenes, the key to UDC image restoration lies in handling blurring in unsaturated areas (0 ≤ pixel value ≤ 1) and resolving flare artifacts in oversaturated areas (1 < pixel value ≤ 500). Traditional convolutional kernels apply the same convolutional weights across the entire image, limiting convolutional operations when dealing with spatially different characteristics. In one embodiment of this invention, a Spatial Feature Transform (SFT) layer is used to achieve HDR reconstruction with denoising and dequantization capabilities. A transformable spatial feature module is used to construct conditional branches to adaptively modulate the intermediate features extracted by the base network.

[0029] For example, the input image is first fed into the conditional branch, then passes through a 3×3 convolution to convert the number of channels to C, and then enters the residual block layer (e.g., with a quantity of 2). Next, through a branch consisting of four 1×1 convolutional kernels and a downsampling layer, spatial conditional features with different resolutions are generated, with feature channel dimensions of C, 2C, 4C, and 8C respectively. Finally, these features are organically integrated into the forward propagation of the base network through the SFT layer. The operation of the SFT layer can be represented as:

[0030] SFT(x)=α⊙x+β (1)

[0031] Here, "⊙" represents pointwise multiplication, x∈R^(C×H×W) are intermediate features in the base network, and α∈R^(C×H×W) and β∈R^(C×H×W) are modulation coefficient feature maps learned by the SFT layer from the output features of the conditional branches. H represents the height of the feature map, and W represents the width of the feature map. Using this spatial feature transformation mechanism, the proposed network model can easily perform spatially diverse mappings to different regions. Therefore, by introducing conditional branches, the network can perform spatial feature transformations on regions with different characteristics, thereby selectively extracting information from different regions.

[0032] Furthermore, considering that the point spread function (PSF) in the UDC imaging system can serve as prior knowledge for UDC image restoration to improve the image restoration effect, a kernel branch is introduced to utilize the PSF to refine or improve the intermediate features extracted by the base network.

[0033] In one embodiment, the utilization of the PSF includes the following steps. First, Principal Component Analysis (PCA) is used to extract the most important information from the PSF, which is then enlarged to the same size as the input image as the PSF feature. Next, this PSF feature is merged with the input image in the feature dimension as the input to the kernel branch, and after a 3×3 convolution transformation of the number of channels, it enters the Residual Block (e.g., the number is set to 2). Finally, dynamic convolution kernels with different spatial resolutions are output through four branches. For example, for an intermediate feature of dimension c×h×w, the corresponding dynamic convolution kernel has a dimension of ck^2×h×w, where k is the size of the dynamic convolution kernel (e.g., set to 3). Then, dynamic convolution is performed on each pixel, specifically as follows:

[0034] F(i,j,c)=K(i,j,c)·M(i,j,c) (2)

[0035] Where K(i,j,c) represents the k×k convolutional kernel learned from the feature at position (i,j,c). M(i,j,c) represents the image patch extracted from the feature centered at (i,j,c), F(i,j,c) represents the element value at position (i,j,c) after dynamic convolution, and "." represents the inner product operation. The dynamic convolutional kernel generated by the kernel branch makes it more flexible to jump to deeper layers when shallow features are connected. The jump connections are no longer direct additions, thus refining the intermediate features extracted by the base network.

[0036] It should be noted that the kernel branch input can select several types of input, including: no input; image information only; point spread function (PSF) information only; and both image and PSF input. Experiments have shown that the image restoration effect is best when both image and PSF are input simultaneously.

[0037] Step S130: Train the image restoration model based on the set loss function.

[0038] In one embodiment, a new loss function is designed for the characteristics of UDC images, expressed as:

[0039] Mapping_L1(Y,X)=|Mapping(Y)-Mapping(X)| (3)

[0040] Here, Y represents the restored image generated by the network model, X represents the corresponding real image, and Mapping is a tone mapping function used to convert the image into a standard image and normalize the image values ​​to [0,1]. Preferably, Mapping(I) = I / (I+0.25) is set to convert the HDR image into a standard image. The L1 norm of the difference between the tone-mapped Y and X is used as the loss function of the network. Training is performed under the constraint of this loss function to obtain the parameters to be learned in the network model, such as weights and biases. It should be understood that the loss function can also be replaced with other types, such as L2 loss. However, experimental results show that the model trained based on Mapping_L1 loss performs best.

[0041] In actual model training, based on the collected training dataset, the gradient descent algorithm can be used to iteratively optimize the network model until convergence, for example, when the designed loss function Mapping_L1 loss no longer decreases or reaches the set loss criterion. For example, the initial learning rate is set to 2×10. -4 The learning rate variation strategy uses cosine annealing, with a minimum learning rate of η. min =1×10 -7 The maximum learning rate is η max =2×10 -4 And in [5×104 1.5×10 5 3×10 5 4.5×10 5 Restart after [number] iterations.

[0042] Specifically, the training process of the image restoration model includes the following steps:

[0043] Step S1: The UDC image is used as input. The PSF is processed by principal component analysis (PCA) to obtain a 5-dimensional vector. Then, the vector is copied and its spatial dimensions are expanded to match those of the input image to obtain the PSF features.

[0044] Step S2: The input image enters the conditional branch to generate conditional features with different spatial resolutions;

[0045] Step S3: The input image and PSF features are concatenated along the channel dimension. After concatenation, the image enters the kernel branch to generate dynamic convolutional kernels with different spatial resolutions.

[0046] In step S4, the input image is fed into the basic network for forward propagation, and the obtained different conditional features are integrated with dynamic convolutional kernels into specific locations. The network model is continuously optimized through iterative optimization.

[0047] In summary, through multifaceted analysis and observation of UDC data, the image restoration model provided in this invention incorporates conditional branches and kernel branches into the UNet base network, enhancing the network's representational capabilities. Furthermore, a Mapping_L1 loss tailored to the training data is designed to stabilize network optimization and improve the visual quality of the restored images. In contrast, the performance of the conventional UNet network differs significantly from the network structure provided in this invention. Moreover, using only conventional training data while ignoring the kernel branches designed in this invention fails to achieve the desired results. In addition, the design of the Mapping_L1 loss achieves optimal performance and avoids the risk of training convergence failure.

[0048] Step S140: For the acquired target image, perform image reconstruction using a trained image restoration model.

[0049] In practical applications, the target image can be directly input into the trained image restoration model to obtain the reconstructed image.

[0050] To further verify the effectiveness of the present invention, simulation experiments were conducted. Figure 3 The results are visual comparisons, and Tables 1 to 4 show the quantitative experimental results.

[0051] In Table 1, the number of parameters is expressed in M, where M stands for millions; computational complexity is expressed in G, an abbreviation for gigabit floating-point operations; PSNR and SSIM represent peak signal-to-noise ratio and structural similarity, respectively, reflecting the image reconstruction effect; the higher the value, the better the restoration effect; LPIPS represents image similarity, and the lower the value, the better the restoration effect. Uformer, HDRUNet, and DISCNet are existing representative neural network algorithm models that can be used for UDC image restoration. UDC-UNet S UDC-UNet and UDC-UNet respectively represent the UDC-UNet method of this invention and the simplified version of the UDC-UNet method.

[0052] Table 1: Comparative Experiment Results of Existing Image Restoration Models and the Invention

[0053] method PSNR SSIM LPIPS Params Uformer 37.97 0.9784 0.0285 20.0M HDRUNet 40.23 0.9832 0.0240 1.7M DISCNet 39.89 0.9864 0.0152 3.8M UDC-UNet 47.18 0.9927 0.0100 14.0M <![CDATA[UDC-UNet S ]]> 45.98 0.9913 0.0128 5.7M

[0054] As shown in Table 1, the present invention exhibits superior performance in terms of restoration effectiveness (indicators PSNR, SSIM, and LPIPS). Furthermore, by comparing the number of parameters and computational cost (GMACs), it can be found that even after compression, the model of the present invention still surpasses other models in performance while further saving computational resources.

[0055] Table 2 shows the structural ablation comparison experiments, where √ and × represent the use and non-use of the corresponding structures, respectively. PSNR and SSIM represent peak signal-to-noise ratio and structural similarity, respectively, reflecting the reconstruction effect of the algorithm. The higher the value, the better the restoration effect. LPIPS represents image similarity. The lower the value, the better the restoration effect.

[0056] Table 2: Comparative Test Structures of Structural Ablation

[0057] Model (a) (b) (c) (d) (e) U-shaped basic network × √ √ √ √ Conditional branches × × × √ √ kernel branch × × √ × √ PSNR 42.19 44.50 44.58 45.23 45.37 SSIM 0.9884 0.9897 0.9893 0.9897 0.9898 LPIPS 0.0164 0.0155 0.0157 0.0166 0.0162

[0058] As shown in Table 2, the U-shaped base network, conditional branch, and kernel branch used in this invention can significantly improve the network reconstruction performance. The network performs best when using the U-shaped base network and adding conditional and kernel branches. The experimental results in Tables 2 and 3 also demonstrate the effectiveness of this conditional branch.

[0059] Table 3 reflects the changes in quantitative results resulting from using different inputs in the kernel branch. The inputs for the kernel branch include: (a) no input; (b) only image information; (c) only point spread function (PSF) information; and (d) both image and PSF information.

[0060] Table 3: Changes in quantitative results caused by input usage in kernel branches

[0061] method enter PSNR SSIM LPIPS (a) None 45.23 0.9897 0.0166 (b) Image 45.17 0.9896 0.0162 (c) PSF 45.26 0.9895 0.0166 (d) Image+PSF 45.37 0.9898 0.0162

[0062] As can be seen from Table 3, this invention achieves a strong image restoration effect by adding a separate kernel branch to the network and comprehensively using information from the input image itself and the point spread function (PSF).

[0063] Table 4 shows the comparison experiments of loss functions. As can be observed from Table 4, this invention, by replacing the commonly used L1 loss in the prior art with Mapping_L1 loss, further improves the image restoration effect. Furthermore, experiments demonstrate that the Mapping_L1 loss used is more suitable for UDC image restoration than Mapping_L2 loss, achieving clearer visual quality.

[0064] Table 4: Results of the comparative experiment on loss functions

[0065] loss function PSNR SSIM LPIPS <![CDATA[L1]]> 41.30 0.9812 0.0301 <![CDATA[Mapping_L2]]> 40.19 0.9838 0.0238 <![CDATA[Mapping_L1]]> 45.37 0.9898 0.0162

[0066] Experimental results show that the present invention surpasses the most advanced methods in both quantitative performance and visual quality, and can generate visually satisfactory results without obvious artifacts even in oversaturated regions.

[0067] In summary, compared with the prior art, the technical effects of the present invention are mainly reflected in the following aspects:

[0068] 1) In existing technologies, the same filter weights are applied to all regions. This invention uses a spatial feature transformation layer, enabling the network to give different levels of attention to regions with different dynamic ranges, thereby improving model performance.

[0069] 2) Existing technologies ignore fuzzy kernel information and directly train the network. This invention adds a separate kernel branch and incorporates the measured point spread function (PSF) related to the UDC imaging system as prior knowledge into the network training, thereby improving model performance.

[0070] 3) Existing technologies use traditional L1 or L2 loss functions. This invention designs a tone mapping loss function, Mapping_L1, based on the characteristics of UDC images, to further improve model performance, stabilize network optimization, and thus achieve better visual quality.

[0071] 4) This invention provides an end-to-end network to alleviate problems such as halo, fog, blur and noise in UDC images under HDR scenes, which can bring users a better sensory experience and is of certain significance for the further promotion and application of UDC systems.

[0072] 5) This invention can be applied not only to the restoration of UDC images, but also to other low-level vision tasks, especially image restoration in HDR scenes. Therefore, it also has certain guiding significance for image restoration work in other HDR scenes. In addition, the basic network of this invention can also adopt other existing network structures.

[0073] This invention can be applied to electronic devices, servers, or the cloud. It reconstructs a clear restored image from a captured target image using a trained image restoration model. The electronic device can be a terminal device or a server. Terminal devices include any device such as mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, and smart wearable devices (smartwatches, virtual reality glasses, virtual reality headsets, etc.). Servers include, but are not limited to, application servers or web servers, and can be standalone servers, cluster servers, or cloud servers.

[0074] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0075] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0076] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0077] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0078] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0079] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0080] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0081] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0082] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for under-display camera image restoration based on a U-shaped dynamic network, comprising the following steps: Acquire target image; The target image is input into a trained image restoration model to obtain a reconstructed image; The image restoration model includes a base network, a conditional branch, and a kernel branch. The base network is used to extract multi-scale information from the input image. The conditional branch is used to adaptively modulate the intermediate features extracted by the base network to generate conditional features with different spatial resolutions for the input image. The conditional branch includes a first convolutional layer, a first residual block layer, multiple downsampling branches, and a spatial feature transformation layer. The input image is convolved by the first convolutional layer and then enters the first residual block layer. The multiple downsampling branches generate spatial conditional features with different resolutions. The obtained spatial conditional features are then integrated into the forward propagation of the base network through the spatial feature transformation layer. The kernel branch includes a second convolutional layer, a second residual block layer, and multiple sampling branches. The second convolutional layer takes the point spread function features and the merged features of the input image in the feature dimension as input, and enters the second residual block layer after changing the number of channels. The multiple sampling branches are respectively connected to the second residual block layer to generate dynamic convolutional kernels with different spatial resolutions, and then perform dynamic convolution on each pixel of the dynamic convolutional kernel. During the forward propagation of the input image by the base network, the conditional features and the dynamic convolutional kernel are integrated into a set position.

2. The method according to claim 1, characterized in that, The basic network is a U-shaped network, which is divided into shallow and deep networks based on the depth of feature extraction. The shallow network gradually extracts shallow features by continuously increasing the receptive field and maps the input image into a high-dimensional representation. The deep network learns deep features from the decoding process, and the U-shaped network combines the shallow and deep features through skip connections.

3. The method according to claim 1, characterized in that, The operation of the Spatial Feature Transformation (SFT) layer is represented as follows: in" " " represents point-by-point multiplication, and x is an intermediate feature in the basic network. and It is the modulation coefficient feature map learned by the spatial feature transformation layer from the output features of the conditional branch.

4. The method according to claim 1, characterized in that, The loss function for training the image restoration model is set as follows: Where Y represents the restored image generated by the image restoration model, X represents the corresponding real image, and Mapping is a tone mapping function that converts the image into a standard image, used to normalize the image values ​​to [0,1]. This indicates that the L1 norm of the difference between the tone-mapped Y and X is used as the loss function.

5. The method according to claim 4, characterized in that, The tone mapping function is set as follows: Here, I represents the image, which is a general representation of the input to the tone mapping function Mapping.

6. The method according to claim 1, characterized in that, The point spread function characteristics are obtained according to the following steps: A multidimensional vector is obtained by performing principal component analysis on the point spread function; The multidimensional vector is copied and its spatial dimensions are expanded to match those of the input image to obtain the point spread function features.

7. A computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

8. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images