Methods, devices, and electronic components for optimizing the light-transmitting area structure of under-display cameras

Abstract

This application relates to a method, apparatus, electronic device, and storage medium for optimizing the structure of the light-transmitting region of an under-display camera. The method includes: determining the point spread function of the optical system of the under-display camera based on preset first structural information of the light-transmitting region; convolving a preset sample image with the point spread function to obtain a degraded image corresponding to the sample image; using the degraded image and the sample image as training data to input into a preset image restoration model, optimizing the parameter information of the image restoration model to obtain compensated structural parameters output by the image restoration model; and using the compensated structural parameters to optimize the first structural information to obtain second structural information. This application solves the problems of blurring, fogging, and glare in the images output by under-display cameras in related technologies, achieving the technical effect of improving the imaging performance of under-display cameras.

Description

Technical Field

[0001] This application relates to the field of under-display camera technology, and in particular to a method, apparatus, electronic device, and storage medium for optimizing the light-transmitting area structure of an under-display camera. Background Technology

[0002] In smartphones with full-screen displays, the front-facing camera needs to be hidden under the screen. Therefore, it is necessary to ensure that the area on the screen where the front-facing camera is hidden can both transmit light and display information.

[0003] Currently, the main approach is to reduce the pixel size of the light-transmitting area to leave gaps for light transmission, and to adjust the position of the light-transmitting area. This achieves both good display functionality and allows light to penetrate the screen to reach the camera for imaging, thus balancing camera imaging and screen display functions.

[0004] However, the size of the light-transmitting gap can reach 100 times the wavelength of light, which conforms to the Fraunhofer diffraction theory. Therefore, light diffraction will occur. After the light is transmitted to the camera for imaging, the image obtained by the camera will become blurry, especially in strong light areas, where multi-order diffraction glare will occur. Moreover, due to the influence of the multi-layer material of the screen and the limitation of the size of the light-transmitting area, the energy of the light passing through the screen will be drastically reduced, with the energy of the light decreasing by more than 70%. This results in insufficient light energy received by the camera. At the same time, the multi-layer material of the screen will cause light to be reflected multiple times. Therefore, the signal-to-noise ratio of the image signal output by the camera will not only be severely reduced, but the image will also produce a fogging phenomenon.

[0005] Currently, no effective solution has been proposed for the problems of blurriness, fogging, and glare in the images output by under-display cameras in related technologies. Summary of the Invention

[0006] This application provides a method, apparatus, electronic device, and storage medium for optimizing the light-transmitting area structure of an under-display camera, so as to at least solve the problems of blurring, fogging, and glare in the images output by under-display cameras in related technologies.

[0007] In a first aspect, embodiments of this application provide a method for optimizing the structure of the light-transmitting region of an under-display camera. The method includes: determining the point spread function of the optical system of the under-display camera based on preset first structural information of the light-transmitting region; convolving a preset sample image with the point spread function to obtain a degraded image corresponding to the sample image; inputting the degraded image and the sample image as training data into a preset image restoration model, optimizing the parameter information of the image restoration model to obtain compensation structure parameters output by the image restoration model, and using the compensation structure parameters to optimize the first structural information to obtain second structural information.

[0008] In some embodiments, the degraded image and the sample image are used as training data pairs and input into a preset image restoration model. Optimizing the parameter information of the image restoration model to obtain the compensation structure parameters output by the image restoration model includes: dividing the training data into blocks to obtain multiple image blocks, and performing data augmentation on each image block before inputting it into the image restoration model; constructing a loss function using preset image pixel value constraints and structural information constraints, training the image restoration model with the goal of minimizing the loss function, and optimizing the parameter information of the image restoration model to obtain the compensation structure parameters output by the image restoration model.

[0009] In some embodiments, determining the point spread function of the optical system of the under-display camera based on preset first structural information of the light-transmitting area includes: obtaining the preset point spread function of the imaging device of the optical system and the spectral response curve of the imaging sensor of the optical system; and determining the point spread function of the optical system using a preset optical path propagation model based on the first structural information, the preset point spread function and the spectral response curve.

[0010] In some embodiments, convolving a preset sample image with the point spread function to obtain a degraded image corresponding to the sample image includes: convolving the sample image with the point spread function, using ring segmentation and / or block stitching to combine different field angles, and adding preset Gaussian white noise to the generated blurred image to obtain a degraded image corresponding to the sample image.

[0011] In some embodiments, the method further includes: using the under-display camera to capture a preset point light source, determining the actual point spread function of the optical system of the under-display camera, and convolving the preset Raw image with the actual point spread function to obtain a Raw degraded image corresponding to the preset Raw image; using the preset Raw image and the Raw degraded image as a first training data pair; using the under-display camera to capture a preset display to obtain an actual Raw image and a projected RGB image corresponding to the display, performing degradation processing on the projected RGB image, and matching the degraded projected RGB image with the actual Raw image in multiple dimensions to generate a Ground Truth (GT) image corresponding to the actual Raw image; using the GT image and the actual Raw image as a second training data pair; inputting both the first training data pair and the second training data pair into the image restoration model to train the image restoration model and optimize the parameter information of the image restoration model.

[0012] In some embodiments, the image restoration model is built on a convolutional neural network based on the Res-Unet architecture.

[0013] In some embodiments, the structural information includes: position information of the light-transmitting area, shape information of the display pixels in the light-transmitting area, and arrangement information of the display pixels.

[0014] Secondly, embodiments of this application provide a device for optimizing the structure of the light-transmitting region of an under-display camera. The device includes: a determining module, configured to determine the point spread function of the optical system of the under-display camera based on preset first structural information of the light-transmitting region; a convolution module, configured to convolve a preset sample image with the point spread function to obtain a degraded image corresponding to the sample image; and an optimization module, configured to input the degraded image and the sample image as training data into a preset image restoration model, optimize the parameter information of the image restoration model to obtain compensation structure parameters output by the image restoration model, and optimize the first structural information using the compensation structure parameters to obtain second structural information.

[0015] Thirdly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method for optimizing the light-transmitting area structure of an under-display camera as described in the first aspect above.

[0016] Fourthly, embodiments of this application also provide a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method for optimizing the light-transmitting area structure of an under-display camera as described in the first aspect above.

[0017] Compared to related technologies, the method, apparatus, electronic device, and storage medium for optimizing the light-transmitting area structure of an under-display camera provided in this application determine the point spread function of the optical system of the under-display camera based on preset first structural information of the light-transmitting area; convolve a preset sample image with the point spread function to obtain a degraded image corresponding to the sample image; use the degraded image and the sample image as training data to input into a preset image restoration model, optimize the parameter information of the image restoration model, obtain the compensation structure parameters output by the image restoration model, and use the compensation structure parameters to optimize the first structural information to obtain the second structural information. This solves the problems of blurring, fogging, and glare in the images output by under-display cameras in related technologies, and achieves the technical effect of improving the imaging performance of under-display cameras.

[0018] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart of a method for optimizing the light-transmitting area structure of an under-display camera according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the network structure of a convolutional neural network according to an embodiment of this application;

[0022] Figure 3 This is a structural block diagram of the light-transmitting area structure optimization device for an under-display camera according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0025] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0026] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0027] This embodiment provides a method for optimizing the structure of the light-transmitting area of ​​an under-display camera. Figure 1 This is a flowchart of a method for optimizing the light-transmitting area structure of an under-display camera according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0028] Step S101: Determine the point spread function of the optical system of the under-display camera based on the preset first structural information of the light-transmitting area.

[0029] In this embodiment, the point spread function (PSF) is the light field distribution of the output image when the input object is a point light source.

[0030] In this embodiment, the first structural information is the initial structural information of the preset light-transmitting area, which may include the position information of the light-transmitting area, the arrangement information of the display pixels in the light-transmitting area, and the shape information of the display pixels, used to indicate the specific position of the display pixels on the light-transmitting area and the specific position of the light-transmitting point.

[0031] Step S102: Convolve the preset sample image with the point spread function to obtain the degraded image corresponding to the sample image.

[0032] In this embodiment, the degradation imaging process can be represented by convolution operation: y = x * a, where * represents convolution operation, y represents the degraded image, x is the sample image, the preset high-definition image, and a represents the point spread function (PSF).

[0033] In this embodiment, the point spread function may include the PSF of the RGB three channels at different object distances and different field of view angles.

[0034] In this embodiment, multiple sample images can be used, such as a high-definition RGB dataset with more than 2,000 images at 4K resolution. Each sample image in the high-definition RGB dataset is convolved with each PSF to simulate and generate multiple degraded images corresponding to each sample image at different object distances.

[0035] Step S103: The degraded image and the sample image are used as training data and input into the preset image restoration model. The parameter information of the image restoration model is optimized to obtain the compensation structure parameters output by the image restoration model. The first structure information is optimized using the compensation structure parameters to obtain the second structure information.

[0036] In this embodiment, the image restoration model can be a convolutional neural network model, and the training data pair can include a sample image and a degraded image corresponding to the sample image at a certain object distance, or a sample image and multiple degraded images corresponding to the sample image at multiple object distances.

[0037] Convolutional Neural Networks (CNNs) are a type of deep feedforward neural network that incorporates convolutional computations and has a deep structure. They are one of the representative algorithms in deep learning. CNNs possess connectivity and representation learning capabilities, thus enabling them to effectively learn relevant features from a large number of samples.

[0038] In this embodiment, a deep learning model can be used to learn the relationship between sample images and degraded images corresponding to the sample images, thereby achieving the image restoration effect of the image restoration model and improving the imaging quality of the under-display camera.

[0039] In one embodiment, a convolutional neural network with a Res-Unet structure can be used to construct the image restoration model.

[0040] Res-Unet is a U-net network with a residual structure. The U-net network has an encoder and a decoder, forming a U-shaped symmetrical structure, which is a widely used network structure.

[0041] In related technologies, some solutions achieve both under-display camera imaging and screen display functions by directly improving the screen structure. For example, this can be done by altering the pixel arrangement structure, pixel shape, and material of each layer of the screen through optical and screen design. Other solutions reduce the diffraction effect generated by the screen itself by placing diffraction elements behind the screen area corresponding to the under-display camera. Still other solutions optimize the imaging performance of the under-display camera based on calibration methods. For example, by obtaining transmittance and color conversion matrices through calibration, the calibrated matrices are multiplied by the pixel values ​​of the image output by the under-display camera to obtain the final corrected image.

[0042] However, these technical solutions often cannot simultaneously achieve the high imaging performance of the under-display camera and the high display performance of the screen. They also have certain requirements for installation accuracy and applicable devices, resulting in difficulties in application and incompatibility with various screen types.

[0043] In this embodiment, the goal of structural optimization of the light-transmitting area of ​​the screen is to increase the transmittance of the light-transmitting area and suppress the diffraction effect after light passes through the light-transmitting area, as well as to ensure that the imaging quality of the under-display camera, the resolution of the display area of ​​the screen, and the display quality are not reduced.

[0044] The method for optimizing the structure of the light-transmitting area of ​​the under-display camera provided in this application allows the structural design of the light-transmitting area of ​​the screen to be not an independent optimal design, but rather an iterative optimization after meeting the PSF and diffraction order sub-energy requirements proposed by the image restoration model. Together with the image restoration model, it achieves the optimal balance between global image quality and screen display effect, ensuring high display quality of the light-transmitting area of ​​the screen while improving the imaging quality of the under-display camera.

[0045] Through steps S101 to S103, the point spread function of the optical system of the under-display camera is determined based on the preset first structural information of the light-transmitting area. The preset sample image is convolved with the point spread function to obtain a degraded image corresponding to the sample image. The degraded image and the sample image are used as training data to input into a preset image restoration model, optimizing the parameter information of the image restoration model to obtain the compensation structure parameters output by the image restoration model. The compensation structure parameters are then used to optimize the first structural information to obtain the second structural information. This application solves the problems of blurring, fogging, and glare in the images output by under-display cameras in related technologies, achieving the technical effect of improving the imaging performance of under-display cameras.

[0046] In some embodiments, the degraded image and sample image are used as training data and input into a preset image restoration model to optimize the parameter information of the image restoration model. The compensation structure parameters output by the image restoration model are obtained through the following steps:

[0047] Step 1: Divide the training data into blocks to obtain multiple image blocks, and then perform data augmentation on each image block before inputting it into the image restoration model.

[0048] Step 2: Construct a loss function using preset image pixel value constraints and structural information constraints. With the goal of minimizing the loss function, train the image restoration model and optimize the parameter information of the image restoration model to obtain the compensation structural parameters output by the image restoration model.

[0049] In this embodiment, each image patch can be augmented using methods such as horizontal / vertical flipping, rotation, scaling, cropping, shearing, translation, contrast adjustment, color dithering, and adding noise. This increases the training dataset of the image restoration model, making the training dataset as diverse as possible and improving the generalization ability of the image restoration model.

[0050] In this embodiment, the loss function includes constraints on image pixel values, the shape and arrangement of the light-transmitting area and its displayed pixels. The training dataset is input into the image restoration model, and the loss function is globally minimized. Based on the loss function, the backpropagation function of the image restoration model is used to optimize the parameter information of the image restoration model. The constructed image restoration model can directly output the corresponding restored image after inputting a degraded image. At the same time, it can also output compensation structure parameters and use the compensation structure parameters to optimize the first structure information of the light-transmitting area. Thus, a higher quality restored image is obtained in the final forward inference process, while improving the display quality of the light-transmitting area of ​​the screen. This achieves "end-to-end" image restoration, optimizing the structure of the light-transmitting area and the imaging quality of the under-display camera from both software and hardware system levels. It is suitable for devices such as mobile phones that have high requirements for image quality, and improves problems such as blurry images and severe glare in under-display camera imaging.

[0051] In some embodiments, the dot spread function of the under-display camera's optical system is determined based on preset first structural information of the light-transmitting area through the following steps:

[0052] Step 1: Obtain the preset point spread function of the camera device of the optical system and the spectral response curve of the imaging sensor of the optical system.

[0053] Step 2: Based on the first structural information, the preset point spread function, and the spectral response curve, determine the point spread function of the optical system using the preset optical path propagation model.

[0054] In this embodiment, based on the initial structure of the light-transmitting area (i.e., the first structure information), the preset point spread function of the imaging device of the optical system, and the spectral response curve of the imaging sensor of the optical system, the optical system can be established using optical path propagation models such as diffraction models and geometric optics imaging models. This allows for the acquisition of the process of the imaging sensor forward calculating the point spread function of the optical system, and further, the point spread function of the optical system at different object distances within all field of view ranges can be calculated.

[0055] In some embodiments, convolving a preset sample image with a point spread function to obtain a degraded image corresponding to the sample image includes: convolving the sample image with a point spread function, using ring segmentation and / or block stitching to combine different field angles, and adding preset Gaussian white noise to the generated blurred image to obtain a degraded image corresponding to the sample image.

[0056] In this embodiment, in order to simulate the blurring caused by incomplete correction in the optical system, the sample image can be convolved with the point spread function, and then a preset Gaussian white noise can be added to the generated blurred image to simulate optical path noise. The standard deviation of the Gaussian white noise can take any value between 0 and 3. Furthermore, after obtaining the degraded images corresponding to the sample image at different object distances, the degraded images can be standardized and normalized to facilitate the subsequent training of the model.

[0057] In this embodiment, point spread functions at different field of view angles corresponding to the object distance can be obtained at near, medium, and far distances. The point spread functions at different object distances and different field of view angles are convolved with the sample images to obtain degraded images at near, medium, and far distances. Multiple degraded images at different object distances and the sample images are then input into the image restoration model for fine-tuning training to improve the performance of the image restoration model, further enhance the image restoration quality of the image restoration model, and improve the imaging quality of the under-display camera.

[0058] In this embodiment, the image restoration model mainly has two functions: first, to improve the clarity of the image output by the imaging sensor, suppress glare, and defogging; second, based on the image restoration performance of the image restoration model in the face of different point spread functions and diffraction orders of different shapes and energies, to adjust the structural information of the light-transmitting area of ​​the screen, adjust the shape of the light-transmitting area, the shape of the display pixels in the light-transmitting area, and the arrangement structure of the display pixels, and complete the iterative optimization of the light-transmitting area, so as to ensure high image restoration performance while taking into account the high display quality of the light-transmitting area during the forward inference process of the image restoration model.

[0059] Figure 2This is a schematic diagram of the network structure of a convolutional neural network according to an embodiment of this application, such as... Figure 2 As shown, the image restoration model is constructed using a convolutional neural network with a Res-Unet structure, which includes multiple convolutional layers, deconvolutional layers, and residual blocks.

[0060] Convolutional layers are used to extract different features from the image input to the image restoration model. Low-level convolutional layers may only extract some low-level features such as edges, lines and corners, while more layers of the network can iteratively extract more complex features from low-level features.

[0061] The residual structure provides greater adjustability to the image restoration model. The image restoration model can control the stacking ratio of the last layer by adjusting the weight factor K, thereby controlling the sharpness of the output image. The specific formula is as follows:

[0062] Output = Input + K * Layer out .

[0063] Among them, Layer out This is the output of the image restoration model. Output is the restored image, and Input is the degraded image.

[0064] In some embodiments, the method further performs the following steps:

[0065] Step 1: Use the under-display camera to capture a preset point light source, determine the actual point spread function of the optical system of the under-display camera, and convolve the preset Raw image with the actual point spread function to obtain the Raw degraded image corresponding to the preset Raw image. Use the preset Raw image and the Raw degraded image as the first training data pair.

[0066] Step 2: Use the under-display camera to capture the preset display to obtain the actual Raw image and the projected RGB image of the corresponding display. Perform degradation processing on the projected RGB image and match the degraded projected RGB image with the actual Raw image in multiple dimensions to generate the GT image corresponding to the actual Raw image. Use the GT image and the actual Raw image as the second training data pair.

[0067] Step 3: Input both the first training data pair and the second training data pair into the image restoration model to train the image restoration model and optimize the parameter information of the image restoration model.

[0068] In this embodiment, an actual under-display camera can be used to capture point light sources at different field of view angles, thereby obtaining the point spread function of the under-display camera at different field of view angles. The point spread function at different field of view angles corresponding to the object distance is obtained at near, medium and far distances. The point spread function at different object distances and different field of view angles is convolved by the preset raw image in blocks to obtain the corresponding first training data pair. The first training data pair includes the preset raw image and the raw degraded image formed after convolution.

[0069] An actual under-display camera can be used to capture images of a high-definition display at close, medium, and long distances to obtain the corresponding actual Raw images and projected RGB images. The projected RGB images are then subjected to Raw image degradation, and the positions, brightness, and colors are matched with the actual Raw images to generate Ground Truth (GT) images. The GT images and the actual Raw images are then used as the second training data pair.

[0070] The under-display camera can be an under-display camera made using second structural information, that is, an under-display camera that optimizes the light-transmitting area.

[0071] The first and second training data pairs are input into the image restoration model. The image restoration model is fine-tuned to improve its performance, further enhance the image restoration quality, and improve the imaging quality of the under-display camera.

[0072] This embodiment provides a device for optimizing the structure of the light-transmitting area of ​​an under-display camera. Figure 3 This is a structural block diagram of the light-transmitting area structure optimization device for an under-display camera according to an embodiment of this application, as shown below. Figure 3 As shown, the device includes: a determining module 31, used to determine the point spread function of the optical system of the under-display camera based on the preset first structural information of the light-transmitting area; a convolution module 32, used to convolve the preset sample image with the point spread function to obtain a degraded image corresponding to the sample image; and an optimization module 33, used to input the degraded image and the sample image as training data into a preset image restoration model, optimize the parameter information of the image restoration model, obtain the compensation structure parameters output by the image restoration model, and use the compensation structure parameters to optimize the first structural information to obtain the second structural information.

[0073] In some embodiments, the optimization module 33 is further configured to slice the training data into multiple image blocks, perform data augmentation on each image block, and input it into the image restoration model; construct a loss function using preset image pixel value constraints and structural information constraints, train the image restoration model with the goal of minimizing the loss function, and optimize the parameter information of the image restoration model to obtain the compensation structural parameters output by the image restoration model.

[0074] In some embodiments, the determining module 31 is further configured to acquire the preset point spread function of the imaging device of the optical system and the spectral response curve of the imaging sensor of the optical system; and to determine the point spread function of the optical system using a preset optical path propagation model based on the first structural information, the preset point spread function and the spectral response curve.

[0075] In some embodiments, the convolution module 32 is further configured to convolve the sample image with a point spread function, combine different field angles using ring segmentation and / or block stitching, and add preset Gaussian white noise to the generated blurred image to obtain a degraded image corresponding to the sample image.

[0076] In some embodiments, the device further includes a training module for capturing images of a preset point light source using an under-display camera, determining the actual point spread function of the under-display camera's optical system, and convolving the preset Raw image with the actual point spread function to obtain a Raw degraded image corresponding to the preset Raw image. The preset Raw image and the Raw degraded image are used as a first training data pair. The device also captures images of a preset display using the under-display camera to obtain the corresponding actual Raw image and projected RGB image. The projected RGB image is degraded, and the degraded projected RGB image is matched with the actual Raw image in multiple dimensions to generate a Ground Truth (GT) image corresponding to the actual Raw image. The GT image and the actual Raw image are used as a second training data pair. Both the first and second training data pairs are input into an image restoration model to train the image restoration model and optimize its parameter information.

[0077] In some embodiments, the image restoration model is built on a convolutional neural network based on the Res-Unet architecture.

[0078] In some embodiments, the structural information includes: the position information of the light-transmitting area, the shape information of the display pixels in the light-transmitting area, and the arrangement information of the display pixels.

[0079] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0080] This embodiment also provides an electronic device. Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, the electronic device includes a memory 404 and a processor 402. The memory 404 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0081] Specifically, the processor 402 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0082] The memory 404 may include a large-capacity memory for data or instructions. For example, and not limitingly, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 404 may include removable or non-removable (or fixed) media. Where appropriate, the memory 404 may be internal or external to the light-transmitting area structure optimization device of the under-display camera. In a particular embodiment, the memory 404 is non-volatile memory. In a particular embodiment, the memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0083] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0084] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the methods for optimizing the light-transmitting area structure of the under-display camera in the above embodiments.

[0085] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0086] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program:

[0087] S1, determine the point spread function of the optical system of the under-display camera based on the preset first structural information of the light-transmitting area.

[0088] S2, convolve the preset sample image with the point spread function to obtain the degraded image corresponding to the sample image.

[0089] S3. The degraded image and the sample image are used as training data and input into the preset image restoration model to optimize the parameter information of the image restoration model, obtain the compensation structure parameters output by the image restoration model, and use the compensation structure parameters to optimize the first structure information to obtain the second structure information.

[0090] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0091] Furthermore, in conjunction with the under-display camera light-transmitting area structure optimization method in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the under-display camera light-transmitting area structure optimization methods in the above embodiments.

[0092] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for optimizing the structure of the light-transmitting area of ​​an under-display camera, characterized in that, The method includes: The dot spread function of the optical system of the under-display camera is determined based on the preset first structural information of the light-transmitting area. The preset sample image is convolved with the point spread function to obtain a degraded image corresponding to the sample image; The degraded image and the sample image are used as training data and input into a preset image restoration model. The parameter information of the image restoration model is optimized to obtain the compensation structure parameters output by the image restoration model. The first structure information is optimized using the compensation structure parameters to obtain the second structure information. The method further includes: Using the under-display camera to capture a preset point light source, the actual point spread function of the optical system of the under-display camera is determined, and the preset Raw image is convolved with the actual point spread function to obtain a Raw degraded image corresponding to the preset Raw image. The preset Raw image and the Raw degraded image are used as the first training data pair. The under-display camera is used to capture images of a preset display to obtain the actual raw image and the projected RGB image corresponding to the display. The projected RGB image is degraded, and the degraded projected RGB image is matched with the actual raw image in multiple dimensions to generate a ground truth (GT) image corresponding to the actual raw image. The GT image and the actual raw image are used as a second training data pair. The first training data pair and the second training data pair are both input into the image restoration model to train the image restoration model and optimize the parameter information of the image restoration model.

2. The method for optimizing the light-transmitting area structure of an under-display camera according to claim 1, characterized in that, The degraded image and the sample image are used as training data and input into a preset image restoration model to optimize the parameter information of the image restoration model. The compensation structure parameters output by the image restoration model include: The training data is divided into blocks to obtain multiple image blocks, and each image block is augmented before being input into the image restoration model. A loss function is constructed using preset image pixel value constraints and structural information constraints. The image restoration model is trained with the goal of minimizing the loss function, and the parameter information of the image restoration model is optimized to obtain the compensation structural parameters output by the image restoration model.

3. The method for optimizing the light-transmitting area structure of an under-display camera according to claim 1, characterized in that, Based on the preset first structural information of the light-transmitting area, the point spread function of the optical system of the under-display camera is determined as follows: Obtain the preset point spread function of the imaging device of the optical system, and the spectral response curve of the imaging sensor of the optical system; Based on the first structural information, the preset point spread function, and the spectral response curve, the point spread function of the optical system is determined using a preset optical path propagation model.

4. The method for optimizing the light-transmitting area structure of an under-display camera according to claim 3, characterized in that, Convolving a preset sample image with the point spread function to obtain a degraded image corresponding to the sample image includes: The sample image is convolved with the point spread function, and different field of view angles are combined by using ring segmentation and / or block stitching. Preset Gaussian white noise is added to the generated blurred image to obtain a degraded image corresponding to the sample image.

5. The method for optimizing the light-transmitting area structure of an under-display camera according to any one of claims 1 to 4, characterized in that, The image restoration model is constructed based on a convolutional neural network with a Res-Unet structure.

6. The method for optimizing the light-transmitting area structure of an under-display camera according to any one of claims 1 to 4, characterized in that, The structural information includes: the position information of the light-transmitting area, the shape information of the display pixels in the light-transmitting area, and the arrangement information of the display pixels.

7. A device for optimizing the light-transmitting area structure of an under-display camera, characterized in that, The device includes: The determining module is used to determine the dot spread function of the optical system of the under-display camera based on the preset first structural information of the light-transmitting area; The convolution module is used to convolve a preset sample image with the point spread function to obtain a degraded image corresponding to the sample image; An optimization module is used to input the degraded image and the sample image as training data into a preset image restoration model, optimize the parameter information of the image restoration model, obtain the compensation structure parameters output by the image restoration model, and use the compensation structure parameters to optimize the first structure information to obtain the second structure information. The training module is used to capture images of a preset point light source using the under-display camera, determine the actual point spread function of the under-display camera's optical system, and convolve the preset Raw image with the actual point spread function to obtain a Raw degraded image corresponding to the preset Raw image. The preset Raw image and the Raw degraded image are used as a first training data pair. The module also captures images of a preset display using the under-display camera to obtain an actual Raw image and a projected RGB image corresponding to the display. The projected RGB image is degraded, and the degraded projected RGB image is matched with the actual Raw image in multiple dimensions to generate a Ground Truth (GT) image corresponding to the actual Raw image. The GT image and the actual Raw image are used as a second training data pair. Both the first and second training data pairs are input into the image restoration model to train the image restoration model and optimize its parameter information.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method for optimizing the light-transmitting area structure of the under-display camera according to any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the method for optimizing the light-transmitting area structure of the under-display camera according to any one of claims 1 to 6.

Images