An under-screen image processing method, device, equipment and storage medium

Abstract

This application discloses an under-display image processing method, apparatus, device, and storage medium. The method includes: acquiring a first under-display image captured by an under-display camera; performing 3A calculation on the first under-display image to intercept target 3A parameters; performing diffraction repair on the first under-display image to obtain a first on-screen image; and compensating the first on-screen image based on the target 3A parameters to obtain a second on-screen image. In this way, by intercepting target 3A parameters (e.g., white balance gain and digital gain of auto-exposure) in the image preprocessing flow, the under-display image is not compensated for gain using the intercepted target 3A parameters. The under-display image without gain compensation retains a clear light source shape, resulting in better diffraction repair. After repair, the intercepted target 3A parameters are applied to the image for gain compensation, thus improving the overall quality of the under-display captured image.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology, and particularly relates to a screen-under image processing method and device, equipment and storage medium. BACKGROUND

[0002] The screen-under camera, that is, the ordinary camera is hidden under the screen, the camera can take pictures through the screen above, the light is blocked by the anode of the screen display area, and the light reaching the camera is very scarce, so that the traditional light propagation characteristics become more complex. The light transmittance of the ordinary front hole camera is balanced in different channels of RGB, and the transmittance of different wave bands is very close, and the average transmittance is higher than 90%. The screen-under transmittance is quite different due to different screen designs, and the transmittance of different wave bands is different from 10% to 50%. Therefore, the light obtained by the screen-under camera itself has a lot of loss.

[0003] The screen-under camera imaging can be regarded as light transmission through a slit array imaging. Light transmission through a slit array imaging will cause light diffraction. The diffraction spot shape, diffusion degree and energy peak value of different forms of slit array (different size, slit width and combination) are different. Therefore, the screen-under camera captures pictures with different degrees of loss. Commonly, the pictures are seriously fogged, the light sources are generally overexposed, the picture details are lost, and the pictures are generally color cast. Therefore, how to optimize the imaging quality of the screen-under camera is a problem to be solved in the development of screen-under shooting technology. SUMMARY

[0004] To solve the above technical problems, the embodiment of the present application expects to provide a screen-under image processing method, device, equipment and storage medium.

[0005] The technical solution of the present application is realized as follows:

[0006] In a first aspect, a screen-under image processing method is provided, comprising:

[0007] obtaining a first screen-under image collected by a screen-under camera;

[0008] performing 3A calculation on the first screen-under image, and intercepting a target 3A parameter;

[0009] performing diffraction repair on the first screen-under image to obtain a first on-screen image;

[0010] compensating the first on-screen image based on the target 3A parameter to obtain a second on-screen image.

[0011] In a second aspect, a screen-under image processing device is provided, comprising:

[0012] The obtaining module is configured to obtain a first screen-under image collected by a screen-under camera;

[0013] The front-end processing module is configured to perform 3A calculation on the first under-screen image, and intercept target 3A parameters.

[0014] The image restoration module is configured to compensate the first on-screen image based on the target 3A parameters to obtain a second on-screen image.

[0015] In a third aspect, a terminal device is provided, comprising a processor and a memory configured to store a computer program capable of running on the processor,

[0016] The processor is configured to execute the steps of the foregoing method when the computer program is running.

[0017] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the foregoing method.

[0018] In the embodiments of the present application, an under-screen image processing method, device, equipment and storage medium are provided, and the method comprises: acquiring a first under-screen image collected by an under-screen camera; performing 3A calculation on the first under-screen image, and intercepting target 3A parameters; performing diffraction restoration on the first under-screen image to obtain a first on-screen image; and compensating the first on-screen image based on the target 3A parameters to obtain a second on-screen image. In this way, the target 3A parameters (such as white balance gain and automatic exposure digital gain) are intercepted in the image pre-processing process, that is, the target 3A parameters are not used for gain compensation on the under-screen image, the under-screen image without gain compensation retains clear light source patterns, the diffraction restoration effect is better, and the target 3A parameters are applied to the image after restoration for gain compensation, thereby improving the quality of the under-screen captured image as a whole. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 It is an anode distribution diagram of a screen;

[0020] Figure 2 It is a light diffraction diagram;

[0021] Figure 3 It is a diagram showing the influence of slit width on diffraction fringes;

[0022] Figure 4 It is a diagram showing an image captured by an on-screen camera;

[0023] Figure 5 It is a diagram showing an image captured by an under-screen camera;

[0024] Figure 6 It is a first flow diagram of the under-screen image processing method in the embodiments of the present application;

[0025] Figure 7 This is a schematic diagram of the second process of the under-display image processing method in the embodiments of this application;

[0026] Figure 8 This is a schematic diagram illustrating the implementation process of pipeline 1 in the embodiments of this application;

[0027] Figure 9 This is a schematic diagram illustrating the implementation process of pipeline2 in the embodiments of this application;

[0028] Figure 10 This is a schematic diagram of the first component structure of the under-display image processing device in an embodiment of this application;

[0029] Figure 11 This is a schematic diagram of the second component structure of the under-display image processing device in an embodiment of this application;

[0030] Figure 12 This is a schematic diagram of the third component structure of the under-display image processing device in the embodiments of this application.

[0031] Figure 13 This is a schematic diagram of the composition structure of the terminal device in the embodiments of this application. Detailed Implementation

[0032] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0033] An under-display camera (UPC) hides a regular camera beneath the screen, allowing it to capture images through the screen above. However, the light reaching the camera is severely limited due to the screen's anode blockage, complicating traditional light propagation characteristics. A typical punch-hole camera (often called an "on-screen camera") exhibits relatively uniform light transmittance across different RGB channels, with very similar transmittance across different wavelengths, averaging over 90%. However, transmittance varies significantly depending on screen design, ranging from 10% to 50% across different wavelengths. Therefore, under-display cameras inherently suffer considerable light loss. For example... Figure 1 This is a schematic diagram of the anode distribution on the screen, such as... Figure 1 As shown, the lower left corner is the light-transmitting area, and the other areas are the normal display areas. It can be seen that in order to increase the transmittance of the under-display camera, the anode density in the light-transmitting area is less than that in the normal display area.

[0034] Figure 2 This is a schematic diagram of light diffraction, such as... Figure 2As shown, the light source S forms a ring-shaped diffraction fringe on the imaging plane P when passing through the round hole H. The light source S forms a parallel diffraction fringe on the imaging plane P when passing through the single slit G.

[0035] Figure 3 A schematic diagram of the influence of the slit width on the diffraction fringe is shown as Figure 3 As shown, when the slit is very wide, the slit width is much larger than the wavelength of light, and the diffraction phenomenon is not obvious at all. The smaller the slit width, the more obvious the diffraction phenomenon.

[0036] The imaging of the under-screen camera can be regarded as the imaging of light passing through a slit array. The imaging of light passing through a slit array will cause diffraction of light. The diffraction spot shape, diffusion degree, and energy peak value of different forms of slit arrays (different sizes, slit widths, and combinations) are different. Therefore, the screen under the camera captures pictures with different degrees of loss. Commonly, the pictures are seriously fogged, the light sources are generally overexposed, the picture details are lost, and the pictures are generally color cast.

[0037] An exemplary Figure 4 A schematic diagram of an image captured by the on-screen camera is shown as Figure 5 A schematic diagram of an image captured by the under-screen camera is shown. The photos and videos captured by the under-screen camera through the screen have obvious light source diffraction phenomenon compared with the normal hole camera. In order to solve this problem, the embodiment of the present application provides an under-screen image processing method, Figure 6 A first flowchart of the under-screen image processing method in the embodiment of the present application is shown as Figure 6 The method can specifically include the following steps.

[0038] Step 601: acquiring a first under-screen image collected by an under-screen camera.

[0039] Here, the first under-screen image can be image data in a raw domain output by an image sensor such as a complementary metal-oxide-semiconductor (CMOS) or a charge coupled device (CCD).

[0040] The first under-screen image can also be image data in a YUV domain. The raw image data in the raw domain output by the image sensor in the under-screen camera is converted to the YUV domain by the ISP to obtain the image data of the first under-screen image. Image data refers to a set of numerical values representing the gray values of each pixel in an image. In the embodiment of the present application, an image can be understood as image data containing image information, that is, “image” and “image data” can be understood as the same concept.

[0041] Step 602: performing 3A calculation on the first under-screen image, and intercepting a target 3A parameter.

[0042] According to the current ambient light brightness and color temperature information, the 3A algorithm is used to calculate the auto exposure (AE), auto white balance (AWB) and auto focus (AF). The AF auto focus algorithm, the AE auto exposure algorithm and the AWB auto white balance algorithm are used to realize the maximum image contrast, improve the overexposure or underexposure of the main shooting object, compensate the color difference of the picture under different light irradiation, and present the image information with high quality.

[0043] Intercepting can be understood as, when performing the image pre-processing process on the first under-screen image, the target 3A parameter that needs to be compensated is intercepted, and the intercepted target 3A parameter is applied to the compensation after the diffraction repair, that is, no gain compensation is performed on the under-screen image in the image pre-processing process. The image without gain compensation retains the clear light source form, and the diffraction repair can improve the repair effect.

[0044] Exemplarily, the target 3A parameter includes a white balance gain (WB Gain) and / or an auto exposure digital gain (AEDigital Gain).

[0045] Exemplarily, in some embodiments, the 3A calculation on the first under-screen image and the interception of the target 3A parameter include: performing the 3A calculation on the first under-screen image and intercepting the under-screen white balance gain of the first under-screen image; wherein the under-screen white balance gain includes R gain and B gain; and no white balance adjustment is performed on the first under-screen image.

[0046] The white balance gain of the first under-screen image is set to 1 in the image pre-processing process, and the actually calculated white balance gain is intercepted.

[0047] Exemplarily, in some embodiments, the 3A calculation on the first under-screen image and the interception of the target 3A parameter include: performing the 3A calculation on the first under-screen image to obtain a digital gain of the first under-screen image; when the digital gain is greater than a gain threshold, determining a digital gain coefficient based on the digital gain and the gain threshold; taking the digital gain coefficient as the target 3A parameter; and performing brightness adjustment on the original RAW domain image based on the gain threshold.

[0048] It can be understood that, if the digital gain is greater than the gain threshold, the gain threshold is used for brightness compensation in the image pre-processing process, that is, the image brightness is reduced to retain the clear light source form.

[0049] Exemplarily, the interception condition includes:

[0050]

[0051] Red Gain = 1.00 video on

[0052] Blue Gain = 1.00 video on

[0053] In the formula, T is a gain threshold. Red Gain = 1.00 and Blue Gain = 1.00 indicate that R gain and B gain take 1 when processing the under-screen image, that is, no white balance adjustment is performed on the first under-screen image.

[0054] For example, T can take 4, 8, 16, 32, etc. Optionally, T takes 16. Interception is performed on a digital gain of 16 times or more, and if it exceeds 16 times, only 16 times takes effect. Red Gain and Blue Gain related to white balance are completely intercepted, and no white balance adjustment is performed on the image in the complete data stream of the sensor output Raw domain image before diffraction repair.

[0055] For example, in some embodiments, the method further includes: when the digital gain is less than or equal to the gain threshold, performing brightness adjustment on the first under-screen image based on the digital gain.

[0056] It can be understood that for automatic exposure gain interception, the interception condition is to intercept a digital gain greater than a preset gain threshold. If the digital gain is less than or equal to the gain threshold, normal brightness compensation is not intercepted.

[0057] Step 603: performing diffraction repair on the first under-screen image to obtain a first on-screen image;

[0058] For example, in some embodiments, performing diffraction repair on the first under-screen image to obtain a first on-screen image includes: converting the first under-screen image in the RAW domain to a YUV domain to obtain a second under-screen image in the YUV domain; and performing diffraction repair on the second under-screen image in the YUV domain to obtain the first on-screen image.

[0059] For example, the diffraction repair on the first under-screen image is performed based on a neural network to obtain the first on-screen image.

[0060] Correspondingly, the method further includes: obtaining a training data set; wherein the training data set includes: first on-screen image data and first under-screen image data, the first on-screen image data and the first under-screen image data correspond to each other; training a neural network based on the training data set to obtain a trained neural network; wherein the trained neural network is used to perform diffraction repair on the first under-screen image.

[0061] Specifically, the screen-under image data is processed by using the neural network to output screen-on image data; a loss value of the screen-on image data output by the neural network relative to screen-on image data in the training data set is calculated to adjust network parameters of the neural network.

[0062] Specifically, the screen-on image data and the screen-under image data obtained through 3A parameter interception and RAW domain to YUV domain format conversion are used to train the neural network, and the trained neural network can perform diffraction repair on the screen-under image in the YUV domain according to the diffraction characteristics of the screen to obtain the repaired screen-on image.

[0063] In some embodiments, if the neural network performs diffraction repair on the first screen-under image in the RAW domain, the screen-on and screen-under image data in the RAW domain are included in the training data set.

[0064] Step 604: Compensate the first screen-on image based on the target 3A parameter to obtain a second screen-on image.

[0065] It should be noted that the order of compensation based on the target 3A parameter in the embodiments of the present application is not strictly limited.

[0066] For example, the intercepted red gain and blue gain are first applied to perform white balance compensation, that is, the red gain and the blue gain are multiplied to the corresponding R channel and B channel respectively to complete the white balance compensation. Then, the intercepted digital gain is used to perform brightness compensation to achieve the ideal brightness.

[0067] For example, in some embodiments, when the target 3A parameter is a white balance gain, the compensation of the first screen-on image based on the target 3A parameter includes: determining a screen-on white balance gain corresponding to the screen-under white balance gain based on a preset mapping relationship, and performing white balance adjustment on the first screen-on image based on the screen-on white balance gain; wherein the mapping relationship includes a mapping relationship between the screen-on white balance gain and the screen-under white balance gain.

[0068] In the same color temperature environment, the screen-on white balance gain is the white balance gain obtained by 3A measurement on the screen-on image, and the screen-under white balance gain is the white balance gain obtained by 3A measurement on the screen-under image. By establishing a mapping relationship between the two, the actual measured white balance gain is adjusted when processing the screen-under image. In actual application, the presence of the screen will affect the white balance of the image, so adjusting the actual measured screen-under white balance gain can make the white balance adjustment more accurate.

[0069] In this way, the target 3A parameter (such as a white balance gain and a digital gain of automatic exposure) is intercepted in an image pre-processing flow, that is, gain compensation is not performed on the data under the screen, the image without gain compensation retains a clear light source form, diffraction repair is performed on the image, the repair effect is improved, gain compensation is performed on the repaired image by using the intercepted target 3A parameter, and a special processing manner of the 3A parameter is designed, so that diffraction elimination and color restoration of the video are achieved, and the overall screen under image quality is improved.

[0070] Exemplarily, in some embodiments, the method further includes: performing image post-processing on the second on-screen image to obtain a target image; and the image post-processing includes color correction and Gamma correction. In other embodiments, the image post-processing further includes noise reduction processing.

[0071] That is, after diffraction repair and compensation are performed on the image, a series of post-processing operations are performed on the image to optimize the image, and a target image finally displayed to a user is obtained. For example, the target image is a picture or a video in a jpg format.

[0072] Exemplarily, in a standard image processing flow, a sensor senses a light signal of an image to output original RAW domain image data to an image signal processor (ISP) for image processing. The ISP includes an image pre-processing flow and an image post-processing flow. The traditional image pre-processing flow can include 3A measurement and compensation on the original RAW domain image. The traditional image post-processing flow can include noise reduction processing, color correction, Gamma correction, and nonlinear brightening.

[0073] It should be noted that the under-screen image processing method provided in the embodiments of the present application can be applied to processing of a picture or a video collected by the under-screen camera, and in particular, for video processing. The RAW domain image is converted to a YUV domain, diffraction repair is performed based on the cropped YUV data, high repair efficiency is achieved, the video repair efficiency is improved, and real-time processing capability of the video is supported.

[0074] Figure 7 FIG. 2 shows a second flowchart of the under-screen image processing method in the embodiments of the present application. Figure 7 As shown in FIG. 2, the method can specifically include the following steps.

[0075] Step 701: acquiring a first under-screen image collected by an under-screen camera;

[0076] Exemplarily, the first under-screen image can be original RAW domain image data output by a sensor. Correspondingly, 3A measurement and interception can also be performed in the RAW domain.

[0077] Step 702: performing 3A measurement on the first under-screen image, and intercepting a target 3A parameter;

[0078] Exemplarily, the target 3A parameter includes a white balance gain (WB Gain) and / or an automatic exposure digital gain (AE Digital Gain).

[0079] Exemplarily, in some embodiments, the performing 3A measurement on the first under-screen image and intercepting the target 3A parameter includes: performing 3A measurement on the first under-screen image, and intercepting an under-screen white balance gain of the first under-screen image; wherein the under-screen white balance gain includes an R gain and a B gain; and no white balance adjustment is performed on the first under-screen image.

[0080] Exemplarily, in some embodiments, the performing 3A measurement on the first under-screen image and intercepting the target 3A parameter includes: performing 3A measurement on the first under-screen image, and obtaining a digital gain of the first under-screen image; when the digital gain is greater than a gain threshold, determining a digital gain coefficient based on the digital gain and the gain threshold; taking the digital gain coefficient as the target 3A parameter; and performing brightness adjustment on the original RAW domain image based on the gain threshold. Here, the digital gain coefficient can represent a multiple relationship between the digital gain and the gain threshold, and correspondingly, the compensating the first on-screen image based on the target 3A parameter includes: multiplying each pixel value in the first on-screen image by the digital gain coefficient.

[0081] Exemplarily, in some embodiments, the method further includes: when the digital gain is less than or equal to the gain threshold, performing brightness adjustment on the first under-screen image based on the digital gain.

[0082] Step 703: converting the first under-screen image in the RAW domain to a YUV domain, and performing noise reduction processing on the image in the YUV domain to obtain the second under-screen image in the YUV domain;

[0083] Step 704: performing diffraction repair on the second under-screen image in the YUV domain to obtain the first on-screen image;

[0084] Exemplarily, the diffraction repair on the first under-screen image based on a neural network to obtain the first on-screen image.

[0085] Correspondingly, the method further comprises: obtaining a training data set; wherein the training data set comprises: first on-screen image data and first off-screen image data, the first on-screen image data and the first off-screen image data corresponding to each other; training a neural network based on the training data set to obtain a trained neural network; wherein the trained neural network is used for diffraction repair of the first off-screen image.

[0086] Specifically, the neural network is used to process the off-screen image data to output on-screen image data; a loss value of the on-screen image data output by the neural network relative to the on-screen image data in the training data set is calculated to adjust the network parameters of the neural network.

[0087] Specifically, the on-screen image data and the off-screen image data obtained by intercepting the 3A parameters and converting the RAW domain to the YUV domain are used to train the neural network, and the trained neural network can perform diffraction repair on the off-screen image in the YUV domain according to the diffraction characteristics of the screen to obtain the repaired on-screen image.

[0088] Step 705: compensating the first on-screen image based on the target 3A parameter to obtain a second on-screen image;

[0089] For example, in some embodiments, when the target 3A parameter is a white balance gain, the compensation of the first on-screen image based on the target 3A parameter comprises: determining the on-screen white balance gain corresponding to the off-screen white balance gain based on a preset mapping relationship, and adjusting the first on-screen image based on the on-screen white balance gain; wherein the mapping relationship comprises the mapping relationship between the on-screen white balance gain and the off-screen white balance gain.

[0090] In this way, the target 3A parameter (such as the white balance gain and the digital gain of the automatic exposure) is intercepted in the image pre-processing process, that is, the off-screen data is not compensated by the gain, and the image without gain compensation retains a clear light source form. Diffraction repair can improve the repair effect, and the repaired image is compensated by the intercepted target 3A parameter, which is a special processing method for the design of 3A parameters, realizes diffraction elimination and color restoration of the video, and improves the quality of the off-screen image as a whole.

[0091] Step 706: performing image post-processing on the second on-screen image to obtain a target image.

[0092] The image post-processing comprises color correction and Gamma correction.

[0093] Exemplarily, in some embodiments, the image post-processing includes color correction, Gamma correction and noise reduction processing. After the original RAW domain image is converted to the YUV domain, noise reduction processing is performed on the image in the YUV domain, and noise reduction is performed again on the compensated second on-screen image during post-processing, which can improve the signal-to-noise ratio of the image.

[0094] That is, after the YUV is diffracted and compensated, a series of post-processing operations are performed on the image to optimize the image, and the target image finally displayed to the user is obtained.

[0095] Based on the above embodiments, the under-screen image processing method provided in the application can be implemented by two pipelines;

[0096] Figure 8 The implementation flowchart of pipeline 1 in the embodiments of the application is shown in FIG. 1, as shown in the figure, the processing flow of pipeline 1 is mainly realized by three parts of an image pre-processing module (IFE) + a neural network + an image post-processing module (IPE); Figure 8

[0097] 1. The image sensor collects an original RAW domain image;

[0098] 2. The IFE performs image pre-processing in the RAW domain, which includes image 3A calculation and interception;

[0099] According to the current ambient light brightness and color temperature information, the 3A algorithm is calculated to calculate the automatic exposure (AE), the automatic white balance (AWB) and the automatic focus (AF). The AF automatic focusing algorithm, the AE automatic exposure algorithm and the AWB automatic white balance algorithm are used to realize the maximum image contrast, improve the overexposure or underexposure of the main subject, compensate the color difference of the picture under different light irradiation, and thus present high-quality image information. The AE and AWB values calculated by the system need to be intercepted first, that is, the IFE does not use the intercepted 3A parameters for brightness and white balance adjustment.

[0100] 3. The IPE converts the RGB domain to the YUV domain to obtain a first under-screen image in the YUV domain;

[0101] The converted data has a 10-bit width (10-bit data range 0-1024, compared with 8-bit data range 0-255, the color expression is more delicate and the transition is more rich), which is used for network training and diffractive repair.

[0102] ​Specifically, in training the neural network, 10-bit YUV image data on-screen and off-screen is collected to form a data pair, and the neural network is trained offline. Different from using 8-bit YUV image data for network training, the network model is trained based on 10-bit original YUV image data, the model structure is the same, but the color performance is more rich.

[0103] 4. diffractive repair of the first off-screen image in the YUV domain by using the neural network to obtain a repaired first on-screen image;

[0104] 5. applying the real white balance gain and the digital gain intercepted in item 2 to the repaired first on-screen image output by the neural network to obtain a second on-screen image;

[0105] Specifically, the data repaired by the network is converted to the rgb domain, the red gain and blue gain values are adjusted accordingly, and then multiplied to the corresponding R and B channels to complete the compensation of the white balance gain. The digital gain value is also compensated to achieve the ideal brightness.

[0106] The adjustment of the white balance gain refers to: according to the mapping relationship between the white balance gain measured on the screen and the white balance gain measured off the screen in the same color temperature environment, the image is compensated for white balance by white balance gain mapping.

[0107] 6. image post-processing is completed by IPE.

[0108] The second on-screen image after network repair and compensation is color corrected, gamma corrected, and denoised, and other on-screen image ISP post-processing procedures.

[0109] Figure 9 The implementation process of pipeline2 in the embodiments of the present application is shown in the schematic diagram of the implementation process of pipeline2 as shown in Figure 9 The processing flow of pipeline2 is mainly realized by IFE+IPE1+neural network+IPE2 four parts.

[0110] 1. The image sensor collects original RAW domain images;

[0111] 2. IFE realizes image pre-processing in the RAW domain, which includes image 3A measurement and interception.

[0112] According to current ambient light brightness and color temperature information, through 3A algorithm measurement, Auto Exposure (AE), Auto White Balance (AWB) and Auto Focus (AF) are measured. AF auto focusing algorithm, AE auto exposure algorithm and AWB auto white balance algorithm are used to realize maximum image contrast, improve overexposure or underexposure of the main shooting object, compensate for color difference of the picture under different light irradiation, and present high-quality image information. The embodiment of the application first needs to intercept the AE and AWB values measured by the system, that is, IFE does not use the intercepted 3A parameters for brightness and white balance adjustment.

[0113] 3. IPE1 converts the RGB domain to the YUV domain and performs the first noise reduction processing to obtain the first under-screen image in the YUV domain;

[0114] The data after format conversion and noise reduction has 8-bit width, and the data is used for network training and diffraction repair. Specifically, when training the neural network, because IPE1 does not process the data brightness and dynamic range, the linearity of the YUV image data is ensured, and the YUV domain on-screen image data and under-screen image data are collected to form a data pair to train the neural network.

[0115] IPE 1 converts the RGB domain to the YUV domain, and other functions are closed except noise reduction. In IPE 2, the dynamic, color, noise reduction and other related modules are opened, and the repaired and compensated YUV data stream is post-processed.

[0116] 4. The first on-screen image after repair is obtained by using a neural network to repair the first under-screen image in the YUV domain;

[0117] 5. The real white balance gain and digital gain intercepted in 2 are applied to the first on-screen image after repair output by the neural network to obtain a second on-screen image;

[0118] Specifically, the data after network repair is converted to the rgb domain, the red gain and blue gain values are adjusted accordingly, and then multiplied to the corresponding R and B channels to complete the compensation of the white balance gain. Then, the digital gain value is compensated to achieve the ideal brightness.

[0119] The adjustment of the white balance gain refers to: according to the mapping relationship between the white balance gain measured on the screen and the white balance gain measured under the screen in the same color temperature environment, the image is compensated by white balance gain mapping.

[0120] 6. Image post-processing is completed by IPE.

[0121] IPE2 can be used to perform color correction and gamma correction on the image on the second screen. IPE2 can also perform a second noise reduction on YUV data, resulting in a better signal-to-noise ratio after two noise reduction processes.

[0122] Through the above-mentioned pipeline2 implementation method, a special processing method for the 3A parameter design is used to achieve diffraction elimination and color restoration of video, thereby improving the overall image quality under the screen.

[0123] The above-mentioned under-display image processing method can be applied to terminal devices equipped with under-display cameras, such as mobile phones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), wearable devices, cameras, etc.

[0124] To implement the method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide an under-display image processing device, such as... Figure 10 As shown, the device 100 includes:

[0125] The acquisition module 1001 is configured to acquire the first under-screen image captured by the under-screen camera;

[0126] The front-end processing module 1002 is configured to: perform 3A calculation on the first under-screen image and intercept the target 3A parameters;

[0127] The image restoration module 1003 is configured to compensate the image on the first screen based on the target 3A parameters to obtain the image on the second screen.

[0128] For example, in some embodiments, the front-end processing module 1002 is configured to:

[0129] Perform 3A measurement on the first under-screen image and intercept the under-screen white balance gain of the first under-screen image; wherein, the under-screen white balance gain includes R gain and B gain;

[0130] No white balance adjustment is performed on the first screen image.

[0131] For example, in some embodiments, the image restoration module 1003 is configured to:

[0132] Based on a preset mapping relationship, determine the on-screen white balance gain corresponding to the under-screen white balance gain, and adjust the white balance of the first on-screen image based on the on-screen white balance gain;

[0133] The mapping relationship includes the mapping relationship between on-screen white balance gain and off-screen white balance gain.

[0134] For example, in some embodiments, the front-end processing module 1002 is configured to:

[0135] Perform 3A measurement on the first under-screen image to obtain the digital gain of the first under-screen image;

[0136] When the digital gain is greater than the gain threshold, the digital gain coefficient is determined based on the digital gain and the gain threshold.

[0137] The digital gain coefficient is used as the target 3A parameter;

[0138] The brightness of the original RAW domain image is adjusted based on the gain threshold.

[0139] For example, in some embodiments, the front-end processing module 1002 is configured to:

[0140] When the digital gain is less than or equal to the gain threshold, the brightness of the first under-screen image is adjusted based on the digital gain.

[0141] For example, in some embodiments, the front-end processing module 1002 is configured to perform the 3A calculation and interception in the RAW domain.

[0142] For example, such as Figure 11 As shown, the device 100 further includes: a back-end processing module 1004;

[0143] The back-end processing module 1004 is configured to convert the first under-screen image in the RAW domain to the YUV domain to obtain a second under-screen image in the YUV domain.

[0144] The image restoration module 1002 is configured to perform diffraction restoration on the second under-screen image in the YUV domain to obtain the first on-screen image.

[0145] For example, the back-end processing module 1004 is configured to perform image post-processing on the image on the second screen to obtain a target image; wherein the image post-processing includes color correction and Gamma correction.

[0146] For example, such as Figure 12 As shown, the backend processing module 1004 includes a first processing submodule 1004a and a second processing submodule 1004b;

[0147] The first processing submodule 1004a is configured to: convert the first under-screen image in the RAW domain to the YUV domain, perform noise reduction processing on the image in the YUV domain, and obtain the second under-screen image in the YUV domain;

[0148] The second processing submodule 1004b is configured to perform image post-processing on the image on the second screen to obtain the target image;

[0149] The image post-processing includes color correction and gamma correction.

[0150] For example, in some embodiments, the device 100 further includes a training module;

[0151] The training module is configured to: acquire a training dataset; wherein the training dataset includes: image data on the first screen and image data below the first screen, the image data on the first screen and the image data below the first screen corresponding to each other; train a neural network based on the training dataset to obtain a trained neural network;

[0152] The trained neural network is used to perform diffraction repair on the first under-screen image.

[0153] It should be noted that the aforementioned under-display image processing device can be an image processing chip or a terminal device used in terminal devices.

[0154] By using the aforementioned under-display image processing device, target 3A parameters (such as white balance gain and digital gain of auto exposure) are intercepted in the image preprocessing process. That is, the under-display image data is not compensated for without the intercepted target 3A parameters. The under-display image without gain compensation retains a clear light source shape, and the diffraction repair effect is better. After repair, the intercepted target 3A parameters are applied to the image for gain compensation, which improves the overall quality of the under-display captured image.

[0155] Based on the hardware implementation of each unit in the above-described under-display image processing device, this application embodiment also provides a terminal device, such as... Figure 13 As shown, the terminal device 130 includes: a processor 1301 and a memory 1302 configured to store computer programs capable of running on the processor;

[0156] When the processor 1301 is configured to run a computer program, it executes the method steps described in the foregoing embodiments.

[0157] Of course, in practical applications, such as Figure 13 As shown, the various components in this terminal device are coupled together via bus system 1303. It can be understood that bus system 1303 is used to enable communication between these components. In addition to a data bus, bus system 1303 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 1303 in the figure.

[0158] In practical applications, the aforementioned processor can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the functions of the aforementioned processor can also be other types, and the embodiments of this application do not specifically limit them.

[0159] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0160] In practical applications, the aforementioned device can be a terminal device or a chip applied to a terminal device. In this application, the device can implement the functions of multiple units through software, hardware, or a combination of both, enabling the device to execute the under-display image processing method provided in any of the above embodiments. Furthermore, the technical effects of each technical solution of this device can be referenced to the technical effects of the corresponding technical solutions in the under-display image processing method, and will not be elaborated upon further in this application.

[0161] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by an under-display image processing processor to perform the steps of the aforementioned method.

[0162] This application also provides a computer program product, including computer program instructions.

[0163] Optionally, the computer program product can be applied to the terminal device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the terminal device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0164] This application also provides a computer program.

[0165] Optionally, the computer program can be applied to the terminal device in the embodiments of this application. When the computer program is run on the computer, it causes the computer to execute the corresponding processes implemented by the terminal device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0166] It should be understood that the terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. The expressions “having,” “may have,” “comprising,” and “including,” or “may include” and “may contain” used herein may be used to indicate the presence of a corresponding feature (e.g., an element such as a number, function, operation, or component), but do not exclude the presence of additional features.

[0167] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.

[0168] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and devices can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0170] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0172] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An under-display image processing method, characterized in that, The method includes: Acquire the first under-display image captured by the under-display camera; Perform 3A measurement on the first under-screen image and intercept target 3A parameters; wherein, intercepting target 3A parameters includes not performing gain compensation on the first under-screen image without using the target 3A parameters; Diffraction repair is performed on the first under-screen image to obtain the first on-screen image; The image on the first screen is compensated based on the target 3A parameters to obtain the image on the second screen.

2. The method according to claim 1, characterized in that, The step of performing 3A calculation on the first under-screen image and intercepting the target's 3A parameters includes: Perform 3A measurement on the first under-screen image and intercept the under-screen white balance gain of the first under-screen image; wherein, the under-screen white balance gain includes R gain and B gain; No white balance adjustment is performed on the first screen image.

3. The method according to claim 2, characterized in that, The compensation of the image on the first screen based on the target 3A parameters includes: Based on a preset mapping relationship, determine the on-screen white balance gain corresponding to the under-screen white balance gain, and adjust the white balance of the first on-screen image based on the on-screen white balance gain; The mapping relationship includes the mapping relationship between on-screen white balance gain and off-screen white balance gain.

4. The method according to claim 1 or 2, characterized in that, The step of performing 3A calculation on the first under-screen image and intercepting the target's 3A parameters includes: Perform 3A measurement on the first under-screen image to obtain the digital gain of the first under-screen image; When the digital gain is greater than the gain threshold, the digital gain coefficient is determined based on the digital gain and the gain threshold. The digital gain coefficient is used as the target 3A parameter; The brightness of the first under-screen image is adjusted based on the gain threshold.

5. The method according to claim 4, characterized in that, The method further includes: When the digital gain is less than or equal to the gain threshold, the brightness of the first under-screen image is adjusted based on the digital gain.

6. The method according to claim 1, characterized in that, The 3A calculation and interception are implemented in the RAW domain.

7. The method according to claim 1, characterized in that, The process of performing diffraction repair on the first under-screen image to obtain the first on-screen image includes: The first under-screen image in the RAW domain is converted to the YUV domain to obtain the second under-screen image in the YUV domain. The second under-screen image in the YUV domain is diffractively repaired to obtain the first on-screen image.

8. The method according to claim 7, characterized in that, The step of converting the first under-screen image in the RAW domain to the YUV domain to obtain a second under-screen image in the YUV domain includes: The first under-screen image in the RAW domain is converted to the YUV domain, and noise reduction processing is performed on the YUV domain image to obtain the second under-screen image in the YUV domain.

9. The method according to claim 1, characterized in that, The method further includes: Perform image post-processing on the image on the second screen to obtain the target image; The image post-processing includes color correction and gamma correction.

10. The method according to claim 1, characterized in that, The method further includes: Obtain a training dataset; wherein the training dataset includes: image data on the first screen and image data below the first screen, the image data on the first screen and the image data below the first screen corresponding to each other; The neural network is trained based on the training dataset to obtain the trained neural network; The trained neural network is used to perform diffraction repair on the first under-screen image.

11. An under-display image processing device, characterized in that, The device includes: The module is configured to acquire the first under-display image captured by the under-display camera. The front-end processing module is configured to: perform 3A measurement on the first under-screen image and intercept target 3A parameters; wherein, the interception of target 3A parameters includes not performing gain compensation on the first under-screen image before diffraction repair processing; The image restoration module is configured to: perform diffraction restoration on the first under-screen image to obtain the first on-screen image; and compensate the first on-screen image based on the target 3A parameters to obtain the second on-screen image.

12. The apparatus according to claim 11, characterized in that, The front-end processing module is configured as follows: Perform 3A measurement on the first under-screen image and intercept the under-screen white balance gain of the first under-screen image; wherein, the under-screen white balance gain includes R gain and B gain; No white balance adjustment is performed on the first screen image.

13. The apparatus according to claim 12, characterized in that, The image restoration module is configured to: Based on a preset mapping relationship, determine the on-screen white balance gain corresponding to the under-screen white balance gain, and adjust the white balance of the first on-screen image based on the on-screen white balance gain; The mapping relationship includes the mapping relationship between on-screen white balance gain and off-screen white balance gain.

14. The apparatus according to claim 11 or 12, characterized in that, The front-end processing module is configured as follows: Perform 3A measurement on the first under-screen image to obtain the digital gain of the first under-screen image; When the digital gain is greater than the gain threshold, the digital gain coefficient is determined based on the digital gain and the gain threshold. The digital gain coefficient is used as the target 3A parameter; The brightness of the first under-screen image is adjusted based on the gain threshold.

15. The apparatus according to claim 14, characterized in that, The front-end processing module is configured as follows: When the digital gain is less than or equal to the gain threshold, the brightness of the first under-screen image is adjusted based on the digital gain.

16. The apparatus according to claim 11, characterized in that, The front-end processing module is configured to perform the 3A calculation and interception in the RAW domain.

17. The apparatus according to claim 11, characterized in that, The device further includes: a back-end processing module; The backend processing module is configured to convert the first under-screen image in the RAW domain to the YUV domain to obtain a second under-screen image in the YUV domain. The image restoration module is configured to perform diffraction restoration on the second under-screen image in the YUV domain to obtain the first on-screen image.

18. The apparatus according to claim 17, characterized in that, The backend processing module is configured to perform image post-processing on the image on the second screen to obtain the target image; The image post-processing includes color correction and gamma correction.

19. The apparatus according to claim 17, characterized in that, The backend processing module includes a first processing submodule and a second processing submodule; The first processing submodule is configured to: convert the first under-screen image in the RAW domain to the YUV domain, perform noise reduction processing on the YUV domain image, and obtain the second under-screen image in the YUV domain; The second processing submodule is configured to perform image post-processing on the image on the second screen to obtain the target image; The image post-processing includes color correction and gamma correction.

20. The apparatus according to claim 11, characterized in that, The device also includes a training module; The training module is configured to: acquire a training dataset; wherein the training dataset includes: image data on the first screen and image data below the first screen, the image data on the first screen and the image data below the first screen corresponding to each other; The neural network is trained based on the training dataset to obtain the trained neural network; The trained neural network is used to perform diffraction repair on the first under-screen image.

21. A terminal device, characterized in that, The terminal device includes: a processor and a memory configured to store computer programs capable of running on the processor. Wherein, when the processor is configured to run the computer program, it performs the steps of the method according to any one of claims 1 to 10.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images