A display device and an under-screen photographing processing method

Abstract

An embodiment of the present invention discloses a display device and an under-screen photographing processing method. The under-screen photographing processing method is applied to a display device, and the display device includes a first camera disposed under the screen and a second camera disposed outside the screen. The under-screen photographing processing method includes: obtaining a first image captured by the first camera; processing the first image using a first image restoration model to obtain a first restored image; wherein the first image restoration model is determined according to the difference between the images captured by the first camera and the second camera under a first preset condition. The present invention improves the image quality of the photograph taken by the camera disposed under the screen.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of display technology, and in particular, to a display device and an under-screen photographing processing method. Background Art

[0002] With the development of display technology and the improvement of people's living standards, the requirement for the screen-to-body ratio of display devices is getting higher and higher, and full-screen display devices emerge as the times require. The front camera of a full-screen display device is disposed in the transparent display area of the display device. When the front camera takes a picture, light enters the front camera after passing through the transparent display area, resulting in poor image quality of the picture and affecting the user experience. Summary of the Invention

[0003] The present invention provides a display device and an under-screen photographing processing method to improve the photographing image quality of a first camera disposed under the screen.

[0004] In a first aspect, an embodiment of the present invention provides an under-screen photographing processing method applied to a display device. The display device includes a first camera disposed under the screen and a second camera disposed outside the screen.

[0005] The under-screen photographing processing method includes:

[0006] Obtaining a first image captured by the user using the first camera;

[0007] Processing the first image using a first image restoration model to obtain a first restored image; wherein, the first image restoration model is determined according to the difference between the images captured by the first camera and the second camera under a first preset condition.

[0008] Optionally, the display device further includes a third camera for simulating under-screen photographing of the first camera. When the third camera captures an image, light passes through a transparent shielding member and then enters the third camera;

[0009] The method further includes:

[0010] Displaying the first restored image to the user;

[0011] After receiving an image optimization instruction from the user, processing the first image using a first image conversion model to obtain a first converted image, and processing the first converted image using a second image restoration model to obtain a second restored image; wherein, the first image conversion model is determined according to the difference between the images captured by the first camera and the third camera under a second preset condition, and the second image restoration model is determined according to the difference between the images captured by the second camera and the third camera under a third preset condition.

[0012] Optionally, the method further includes:

[0013] Presenting the second restored image to the user;

[0014] After receiving the first image calibration instruction from the user, controlling the second camera to start and obtaining a first calibration image of the first calibration object captured by the user using the second camera;

[0015] Controlling the third camera to start and obtaining a second calibration image of the first calibration object captured by the user using the third camera;

[0016] Determining a first rear-screen image restoration coefficient calibration matrix based on the difference between the first calibration image and the second calibration image;

[0017] Processing the first converted image using the first rear-screen image restoration coefficient calibration matrix to obtain a first calibrated restored image;

[0018] Presenting the first calibrated restored image to the user, and after receiving the first image continue calibration instruction clicked and input by the user, iteratively optimizing the first rear-screen image restoration coefficient calibration matrix, and re-processing the first converted image using the iteratively optimized first rear-screen image restoration coefficient calibration matrix to obtain a first calibrated restored image until the user no longer clicks and inputs the first image continue calibration instruction;

[0019] Updating the second image restoration model according to the finally determined first rear-screen image restoration coefficient calibration matrix.

[0020] Optionally, the method further includes:

[0021] After receiving the second image calibration instruction clicked and input by the user, starting the first camera and obtaining a third calibration image of the second calibration object captured by the user using the first camera;

[0022] Starting the second camera and obtaining a fourth calibration image of the second calibration object captured by the user using the second camera;

[0023] Determining a front-screen image restoration coefficient calibration matrix based on the difference between the third calibration image and the fourth calibration image;

[0024] Processing the first image using the front-screen image restoration coefficient calibration matrix to obtain a second calibrated restored image;

[0025] Show the second calibrated restored image to the user, and iteratively optimize the front screen lower image restoration coefficient calibration matrix after receiving the second image continue calibration instruction clicked and input by the user. Use the iteratively optimized front screen lower image restoration coefficient calibration matrix to reprocess the first image to obtain a second calibrated restored image until the user no longer clicks and inputs the second image continue calibration instruction;

[0026] Update the first image restoration model according to the front screen lower image restoration coefficient calibration matrix.

[0027] Optionally, the method further includes:

[0028] After receiving the second image calibration instruction from the user, start the third camera and obtain a fifth calibrated image obtained by the user taking a second calibrated object with the third camera;

[0029] Determine the front screen lower image conversion coefficient calibration matrix according to the difference between the third calibrated image and the fifth calibrated image;

[0030] Determine the second rear screen lower image restoration coefficient calibration matrix according to the difference between the fourth calibrated image and the fifth calibrated image;

[0031] Process the first image with the front screen lower image conversion coefficient calibration matrix to obtain a second converted image, and process the second converted image with the second rear screen lower image restoration coefficient calibration matrix to obtain a third calibrated restored image;

[0032] Show the third calibrated restored image to the user, and iteratively optimize the front screen lower image conversion coefficient calibration matrix and the second rear screen lower image restoration coefficient calibration matrix after receiving the second image continue calibration instruction clicked and input by the user. Use the iteratively optimized front screen lower image conversion coefficient calibration matrix to reprocess the first image to obtain a second converted image, and use the iteratively optimized second rear screen lower image restoration coefficient calibration matrix to process the reprocessed second converted image to obtain a third calibrated restored image until the user no longer clicks and inputs the second image continue calibration instruction;

[0033] Update the first image conversion model according to the front screen lower image conversion coefficient calibration matrix;

[0034] Update the second image restoration model according to the second rear screen lower image restoration coefficient calibration matrix.

[0035] Optionally, processing the first image with the first image restoration model to obtain a first restored image includes:

[0036] Select a front - screen - under image restoration coefficient matrix corresponding to the current shooting environment conditions through a first image restoration model, where the first image restoration model includes front - screen - under image restoration coefficient matrices corresponding to multiple different environment conditions;

[0037] Correspondingly, processing the first image with a first image conversion model to obtain a first converted image, and processing the first converted image with a second image restoration model to obtain a second restored image, including:

[0038] Select a front - screen - under image conversion coefficient matrix corresponding to the current shooting environment conditions through the first image conversion model, and select a rear - screen - under image conversion coefficient matrix corresponding to the current shooting environment conditions through the second image restoration model; where the first image conversion model includes front - screen - under image conversion coefficient matrices corresponding to multiple different environment conditions, and the second image restoration model includes rear - screen - under image restoration coefficient matrices corresponding to multiple different environment conditions.

[0039] Optionally, the front - screen - under image restoration coefficient matrix is determined according to the differences between the images of the same preset object captured by the first camera and the second camera at the same position and under the same environment conditions, and the differences between the images of the same environmental scene captured at the same position and under the same environment conditions;

[0040] The front - screen - under image conversion coefficient matrix is determined according to the differences between the images of the same preset object captured by the first camera and the third camera at the same position and under the same environment conditions, and the differences between the images of the same environmental scene captured at the same position and under the same environment conditions, and the rear - screen - under image restoration coefficient matrix is determined according to the differences between the images of the same preset object captured by the second camera and the third camera at the same position and under the same environment conditions and the differences between the images of the same environmental scene captured at the same position and under the same environment conditions.

[0041] In a second aspect, an embodiment of the present invention further provides a display device, including:

[0042] A first camera disposed under the screen, a second camera disposed non - under the screen, and an under - screen photographing processing device;

[0043] The under - screen photographing processing device includes:

[0044] A first image acquisition module for acquiring a first image captured by the user using the first camera;

[0045] The first image restoration determination module is configured to process the first image by using a first image restoration model to obtain a first restored image; wherein, the first image restoration model is determined according to the differences between the images captured by the first camera and the second camera under a first preset condition.

[0046] Optionally, the display device further includes a display panel and a third camera, and a transparent shielding member is disposed on the light incident surface of the third camera;

[0047] The under-screen photographing processing device further includes:

[0048] The first image display module is configured to display the first restored image to the user;

[0049] The image optimization module is configured to, after receiving an image optimization instruction from the user, process the first image by using a first image conversion model to obtain a first converted image, and process the first converted image by using a second image restoration model to obtain a second restored image; wherein, the first image conversion model is determined according to the differences between the images captured by the first camera and the third camera under a second preset condition, the second image restoration model is determined according to the differences between the images captured by the second camera and the third camera under a third preset condition, and when the third camera captures an image, light passes through the transparent shielding member and then enters the third camera.

[0050] Optionally, the display device further includes:

[0051] A display panel, the display panel includes a transparent display area, the first camera is disposed on a non-light-emitting side of the transparent display area, and the light incident surface of the first camera is adjacent to the transparent display area.

[0052] In an embodiment of the present invention, by providing a second camera located non-under the screen, and obtaining a first image restoration model that restores the image quality of the image captured by the first camera to be close to the image quality level of the image captured by the second camera according to the differences between the images of the first camera and the second camera under a first preset condition, after the user uses the first camera to capture a first image, the display device system processes the first image by using the first image restoration model to obtain a first restored image. Since the image quality of the image captured by the second camera is at a normal level, and the image quality of the first restored image is close to the image quality level of the image captured by the second camera, the image quality of the first restored image is better, thereby improving the photographing quality of the first camera and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a schematic diagram of a light-emitting side of a display device provided by an embodiment of the present invention;

[0054] Figure 2It is a schematic diagram of the non-light-emitting side of a display device provided by an embodiment of the present invention;

[0055] Figure 3 It is a schematic diagram of a method for processing under-screen photography provided by an embodiment of the present invention;

[0056] Figure 4 It is a schematic diagram of the non-light-emitting side of another display device provided by an embodiment of the present invention;

[0057] Figure 5 It is a schematic diagram of another method for processing under-screen photography provided by an embodiment of the present invention;

[0058] Figure 6 It is a schematic diagram of the fabrication of an organic light-emitting display panel provided by an embodiment of the present invention;

[0059] Figure 7 It is a schematic diagram of another method for processing under-screen photography provided by an embodiment of the present invention;

[0060] Figure 8 It is a schematic diagram of a process for calibrating an under-screen image provided by an embodiment of the present invention;

[0061] Figure 9 It is a process diagram for generating a coefficient matrix for restoring a front under-screen image provided by an embodiment of the present invention;

[0062] Figure 10 It is a process diagram for generating a conversion coefficient matrix for a front under-screen image and a coefficient matrix for restoring a rear under-screen image provided by an embodiment of the present invention. Detailed implementation manners

[0063] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention are shown in the drawings, rather than all the structures.

[0064] Currently, the under-screen front camera of existing full-screen display devices still cannot obtain an external image well. Since there are still opaque parts in the transparent display area corresponding to the under-screen front camera of the display device, problems such as a dark, blurred image and colored stripes exist in the image captured by the under-screen front camera. Therefore, improving the photographing image quality of the under-screen front camera has become an urgent task for the development of full screens.

[0065] The present embodiment provides an under-screen photo processing method for a display device. The method is applicable to display devices such as LCD display devices, OLED display devices, Micro-LED display devices, and QLED display devices. In this application, only the AMOLED display device is used as an example to illustrate the specific structure of the display device, which is not a limitation of the present invention.

[0066] Figure 1 is a schematic diagram of a light-emitting side of a display device provided by an embodiment of the present invention, Figure 2 is a schematic diagram of a non-light-emitting side of a display device provided by an embodiment of the present invention, Figure 3 is a schematic diagram of an under-screen photo processing method provided by an embodiment of the present invention, with reference to Figure 1 and Figure 2 An embodiment of the present invention provides an under-screen photo processing method, which is applied to a display device, wherein the display device includes a first camera 21 arranged under the screen and a second camera 22 arranged outside the screen.

[0067] refer to Figure 2 , the under-screen photo processing method includes:

[0068] S110: Acquire a first image captured by a user using a first camera.

[0069] S120: Use a first image restoration model to process the first image to obtain a first restored image.

[0070] The first image restoration model is determined according to the difference between images captured by the first camera and the second camera under the first preset condition.

[0071] Specifically, refer to Figure 1 and Figure 2 The display panel 10 includes a transparent display area 11 and a main display area 12. The light transmittance of the transparent display area 11 is greater than that of the main display area 12. The first camera 21 can be arranged on the non-light-exiting side of the transparent display area 11, and the light-incident surface of the first camera 21 is adjacent to the transparent display area 11. When taking pictures, light passes through the transparent display area 11 and enters the first camera 21. The second camera 22 can be arranged on the non-light-exiting side of the display panel 10, or on the non-light-exiting side of the display panel 10. The second camera 22 is arranged not under the screen, that is, the light-incident surface of the second camera 22 is away from the display panel 10, and the light does not pass through the display panel 10, but directly enters the second camera 22. Figure 2 It is only shown by way of example that the second camera 22 is located on the non-light-emitting side of the display panel.

[0072] In addition, the first camera 21 and the second camera 22 may be the same type of camera or different cameras. An artificial intelligence algorithm, such as a neural network method, may be used to identify the difference between the images captured by the first camera and the second camera under the first preset condition, thereby obtaining a first image restoration model. The image captured under the first preset condition may be an image of the same object captured at the same position and in the same environment, and the first preset condition may also be its condition. This embodiment is not specifically limited, as long as a first image restoration model that processes the first image captured by the first camera 21 into a first restored image that is close in level to the image of the second camera 22 can be obtained.

[0073] Exemplarily, when the first camera 21 and the second camera 22 are the same type of cameras, the difference in images of the same object taken by the first camera 21 and the second camera 22 at the same position and in the same environment is mainly caused by the obstruction of the camera's light incident surface by the transparent display area 11. An artificial intelligence algorithm is used to identify the difference in images of the same object taken by the first camera 21 and the second camera 22 at the same position and in the same environment, and a first image restoration model can be learned and trained to restore the image quality of the image taken by the first camera 21 to a level close to the quality level of the image taken by the second camera 22.

[0074] Since the light incident surface of the second camera 22 is not blocked by the transparent display area 11, the quality of the image captured by the second camera 22 is at a normal level. By storing the first image restoration model in the display device system, when the user uses the first camera 21 to capture a first image, the first image restoration model can be used to process the first image to obtain a first restored image with an image quality close to the quality level of the image captured by the second camera 22, thereby improving the photo quality of the first camera 21 and enhancing the user experience.

[0075] In order to ensure that the transparent display area 11 has a high light transmittance, the internal structure of the transparent display area 11 meets the following requirements: ① Non-metallic film layers are all made of high-transmittance materials, and conductive traces are made of transparent materials (such as ITO); ② When the anode uses non-transparent materials, its opaque area is reduced; ③ When the cathode is a low-transmittance metal material, optionally, the planar cathode is changed to a patterned cathode with higher transmittance through precision metal mask evaporation or laser etching. At this time, the patterned cathode opening is preferably an irregular shape such as a circle or an ellipse, and is arranged in an irregular manner, so as to minimize the problem of screen light diffraction.

[0076] In order to improve the light transmission and diffraction problems in the transparent display area 11, the internal structure of the transparent display area 11 meets the following requirements: ① The openings of the pixel definition layer and the anodes are in irregular shapes such as circular or elliptical, and are arranged in a non-regular manner; ② At least part of the conductive traces are arranged in a curved form; ③ The positive projections of each TFT device and the corresponding sub-pixel anode on the substrate at least partially overlap, and the outer contour of the positive projection of each TFT device on the substrate is at least partially curved, so as to reduce the influence of light transmission diffraction generated by the TFT device; ④ Further, the pixel circuit in the transparent display area 11 can adopt a working mode of driving multiple sub-pixels by one TFT device, that is, reducing the number of TFT devices in the transparent display area 11 and reducing the overall light-blocking area of the TFT devices. At this time, the TFT devices are arranged below the sub-pixel anodes with a larger area; ⑤ When there is sufficient layout space for conductive traces in the transparent display area 11, a transition area can be set between the transparent display area 11 and the main display area 12, and all the TFT devices in the pixel circuit of the transparent display area 11 are placed in the transition area, so as to eliminate the influence of the TFT devices on the light transmittance and light transmission diffraction of the transparent display area 11.

[0077] In an embodiment of the present invention, a second camera 22 located under the screen is provided, and a first image restoration model for restoring the image quality of the image captured by the first camera 21 to be close to the image quality level of the image captured by the second camera 22 is obtained according to the difference between the images of the first camera 21 and the second camera 22 under the first preset condition. After the user uses the first camera 21 to capture the first image, the display device system processes the first image using the first image restoration model to obtain a first restored image. Since the image quality of the image captured by the second camera 22 is at a normal level, and the image quality of the first restored image is close to the image quality level of the image captured by the second camera 22, the image quality of the first restored image is better, thereby improving the photographing quality of the first camera 21 and enhancing the user experience.

[0078] Figure 4 is a schematic diagram of the non-light-emitting side of another display device provided by an embodiment of the present invention. Optionally, refer to Figure 4 , the display device further includes a third camera 23 for simulating the under-screen photographing of the first camera 21. When the third camera 23 captures an image, light passes through the transparent shielding member 30 and then enters the third camera 23.

[0079] Figure 5 is a schematic diagram of another under-screen photographing processing method provided by an embodiment of the present invention. Refer to Figure 1 、 Figure 4 and Figure 5 , the method includes:

[0080] S110. Obtain a first image captured by the user using the first camera.

[0081] S120. Process the first image using a first image restoration model to obtain a first restored image.

[0082] Wherein, the first image restoration model is determined according to the differences between the images captured by the first camera and the second camera under a first preset condition.

[0083] S130. Display the first restored image to the user.

[0084] S140. After receiving an image optimization instruction from the user, process the first image using a first image conversion model to obtain a first converted image, and process the first converted image using a second image restoration model to obtain a second restored image.

[0085] Wherein, the first image conversion model is determined according to the differences between the images captured by the first camera and the third camera under a second preset condition, and the second image restoration model is determined according to the differences between the images captured by the second camera and the third camera under a third preset condition.

[0086] Specifically, when the user is satisfied with the image quality of the first restored image, exit the shooting interface of the first camera. When the user is not satisfied with the image quality of the first restored image, the user can click on the image optimization option in the shooting interface, and at this time, the display device system receives the image optimization instruction.

[0087] The differences between the images captured by the first camera 21 and the third camera 23 can be identified using an artificial intelligence algorithm, and the first image conversion model can be learned and trained. The first image conversion model is used to process the image captured by the first camera 21 into an image with a quality level close to that of the image captured by the third camera 23. At the same time, the differences between the images captured by the third camera 23 and the second camera 22 can be identified using an artificial intelligence algorithm, and the second image restoration model can be learned and trained. The second image restoration model is used to process the image captured by the third camera 23 into an image with a quality level close to that of the image captured by the second camera 22. Finally, the first image conversion model and the second image restoration model are stored in the display device system.

[0088] Exemplarily, images of the same preset object can be taken by the first camera 21, the second camera 22, and the third camera 23 respectively under the same environmental conditions and at the same position in advance. An artificial intelligence algorithm is used to identify the image differences to determine the first image conversion model and the second image restoration model. When the first camera 21 is used by a user to take a first image, the display device system processes the first image by using the first image conversion model to obtain a first converted image, and processes the first converted image by using the second image restoration model to obtain a second restored image. The image quality of the obtained second restored image is close to the image quality level of the image taken by the second camera 22. Therefore, the quality of the second restored image is good, thereby improving the photographing quality of the first camera 21 and enhancing the user experience.

[0089] In addition, by providing a transparent shielding member on the light incident surface of the third camera, the film layer structure on the side of the light incident surface of the third camera is made close to that of the first camera, so that the image differences between the third camera and the first camera are small. The differences are mainly caused by the image displayed in the transparent display area at the light incident surface of the first camera. The first image conversion model can specifically process this difference, so that the photographing image quality of the first camera is restored to be the same as or close to that of the third camera. The structural difference between the third camera and the second camera lies in that a transparent shielding member is provided on the light incident surface of the third camera. The image differences between the third camera and the second camera are mainly caused by the transparent shielding member. The second image restoration model can specifically process this difference, so that the photographing image quality of the third camera is restored to be the same as or close to that of the second camera.

[0090] In this embodiment, the first image is subjected to image quality restoration processing through the dual effects of the first image conversion model and the second image restoration model. Compared with a single first image restoration model, the establishment of the method for jointly restoring the image quality by the two models of the first image conversion model and the second image restoration model has more constraint conditions, and the first image conversion model and the second image restoration model respectively restore different image differences, and their effects on restoring the image quality of the first image are better. Therefore, in the embodiment of the present invention, it is preferably to set the second camera 22 and the third camera 23 at the same time, and use the first image conversion model and the second image restoration model to process the first image.

[0091] Optionally, the first camera 21 is disposed on the non-light emitting side of the transparent display area 11 of the display panel 10, and the light incident surface of the first camera 21 is adjacent to the transparent display area 11. The transparent shielding member 30 has the same structure as the transparent display area 11, and the third camera 23 is the same type of camera as the first camera 21.

[0092] Specifically, the display panel 10 of this embodiment does not include a cover plate. The third camera 23 can be the first rear camera of the display device. The transparent shielding member 30 has the same structure as the transparent display area 11, except that it does not display an image. Exemplarily, both the transparent shielding member 30 and the transparent display area 11 can include film layers such as a pixel circuit, a light-emitting layer, a packaging layer, a touch layer, and a polarizer.

[0093] Figure 6 is a schematic diagram of manufacturing an organic light-emitting display panel provided by an embodiment of the present invention. As Figure 6 shown, when designing the layout of the screen body, by using the parts that originally need to be cut and removed from the substrate, without affecting the product layout rate, a plurality of transparent sample areas 40 are set. The middle part of the transparent sample area 40 is the transparent shielding member 30, and its shape and structure are the same as those of the transparent display area 11. At the same time, a single transparent sample area 40 is adjacent to the transparent display area 11 of the corresponding display panel 10 to reduce the differential influence of process film formation non-uniformity on the light transmittance and light diffraction between the transparent shielding member 30 and the transparent display area 11. In addition, in addition to the transparent shielding member 30 in the middle of the transparent sample area 40, it also includes an installation area surrounding the middle part, and its shape is not limited, as long as it is convenient to install the transparent sample area 40 into the rear lens module of the display device. After the screen body is cut, the transparent sample area 40 is not separated from the corresponding display panel 10, and the overall part of the module process is carried out. Before the module cover plate is attached, the transparent sample area 40 and the display panel 10 are separated by re-cutting, and only the display panel 10 is attached with the cover plate. Finally, the transparent sample area 40 without the cover plate attached and the display panel 10 with the cover plate attached are jointly used for the subsequent installation operation of the same display device.

[0094] It should be noted that referring to Figure 4 , the transparent shielding member 30 and the transparent display area 11 have the same structure, and the third camera 23 and the first camera 21 are the same type of camera, so that the film layer structures on the light incident surface sides of the third camera 23 and the first camera 21 are the same, further reducing the small difference in the images captured by the third camera 23 and the first camera 21, so that the first image conversion model can better restore the captured image quality of the first camera 21 to be the same as or close to the captured image quality of the third camera 23. In addition, the second camera 22 and the third camera 23 can be the same type of camera, and the third camera 23 and the second camera 22 can be both arranged on the same side of the display panel. Exemplarily, they can both be arranged on the non-light-emitting side of the display panel.

[0095] Figure 7 is a schematic diagram of another under-screen photographing processing method provided by an embodiment of the present invention. Referring to Figure 7 , this method includes:

[0096] S110. Obtain a first image captured by the user using the first camera.

[0097] S120. Process the first image using the first image restoration model to obtain a first restored image.

[0098] S130. Show the first restored image to the user.

[0099] S140. After receiving an image optimization instruction from the user, process the first image using the first image conversion model to obtain a first converted image, and process the first converted image using the second image restoration model to obtain a second restored image.

[0100] S150. Show the second restored image to the user.

[0101] S160. After receiving a first image calibration instruction from the user, control the second camera to start, and obtain a first calibration image obtained by the user shooting a first calibration object using the second camera.

[0102] Specifically, when the user is satisfied with the image quality of the second restored image, exit the shooting interface of the first camera 21. When the user is not satisfied with the image quality of the second restored image, the user can select the image calibration function. At this time, the user can click on the first image calibration option on the shooting interface to make the display device system receive the first image calibration instruction.

[0103] S170. Control the third camera to start, and obtain a second calibration image obtained by the user shooting the first calibration object using the third camera.

[0104] S180. Determine a first rear-screen image restoration coefficient calibration matrix based on the difference between the first calibration image and the second calibration image.

[0105] S190. Process the first converted image using the first rear-screen image restoration coefficient calibration matrix to obtain a first calibrated restored image.

[0106] S200. Show the first calibrated restored image to the user, and perform iterative optimization on the first rear-screen image restoration coefficient calibration matrix after receiving a first image continue calibration instruction clicked and input by the user; re-process the first converted image using the iteratively optimized first rear-screen image restoration coefficient calibration matrix to obtain a first calibrated restored image until the user no longer clicks and inputs a first image continue calibration instruction.

[0107] S210. Update the second image restoration model according to the finally determined first rear-screen image restoration coefficient calibration matrix.

[0108] Specifically, refer to Figure 1 and Figure 5, during the image calibration process, the display device first activates the second camera 22. Immediately, a fine grid coordinate system with horizontal and vertical intersections and a central red dot appear in the shooting interface. This fine grid coordinate system and the central red dot can help the user accurately locate the center and outer contour of the first calibration object in the shooting interface, and the first calibration image Normal_Cal_1 is obtained. Immediately afterwards, the display device activates the third camera 23. The user continues to use the fine grid coordinate system and the central red dot in the shooting interface to accurately locate the center and outer contour of the first calibration object in the shooting interface, so that the overall position of the first calibration object in the shooting interface is the same as when taking a picture with the second camera 22, thereby obtaining the second calibration image Rear_Cal_1. Subsequently, the display device system uses an artificial intelligence algorithm to identify the image differences between the first calibration image Normal_Cal_1 and the second calibration image Rear_Cal_1, learns and trains the first rear-screen image restoration coefficient calibration matrix Rear_Re_N+1, and processes the first converted image using the first rear-screen image restoration coefficient calibration matrix Rear_Re_N+1 to generate a first calibrated restoration image close to the image quality level of the image taken by the second camera 22. If the user is satisfied with the image quality of the first calibrated restoration image generated by the visual experience, the user can choose to exit the shooting interface, and the display device system will update the first rear-screen image restoration coefficient calibration matrix Rear_Re_N+1 to the second image restoration model; if not satisfied, the user can choose to continue the image calibration option. After receiving the first continue image calibration instruction, the display device system uses an artificial intelligence algorithm to iteratively identify the image differences between the first calibration image Normal_Cal_1 and the second calibration image Rear_Cal_1, learns and trains an optimized first rear-screen image restoration coefficient calibration matrix Rear_Re_N+1, reprocesses the first converted image, and generates a first calibrated restoration image close to the image quality level of the image taken by the second camera 22 until the user is satisfied.

[0109] Since the overall system structures of the first camera 21 and the third camera 23 are basically the same, for the same object under the same image acquisition environmental conditions, the differences between the images taken by the first camera 21 and the third camera 23 are relatively small. Therefore, in this embodiment during the image calibration process, the first image conversion model between the first camera 21 and the third camera 23 is first kept unchanged, and only the second image restoration model between the third camera 23 and the second camera 22 is calibrated and optimized, which can reduce the workload of the display device system for image calibration, thereby improving its image calibration efficiency.

[0110] Optionally, the under-screen photographing processing method of the display device further includes:

[0111] After receiving the second image calibration instruction clicked and input by the user, activate the first camera 21, and obtain the third calibration image obtained by the user taking a picture of the second calibration object with the first camera 21;

[0112] Activate the second camera 22 and obtain a fourth calibration image captured by the user using the second camera 22 for a second calibration object;

[0113] Determine a front under-screen image restoration coefficient calibration matrix based on the difference between the third calibration image and the fourth calibration image;

[0114] Process the first image using the front under-screen image restoration coefficient calibration matrix to obtain a second calibrated restoration image;

[0115] The display device system presents the second calibrated restoration image to the user, and after receiving a second image continue calibration instruction clicked by the user, iteratively optimizes the front under-screen image restoration coefficient calibration matrix, and re-processes the first image using the iteratively optimized front under-screen image restoration coefficient calibration matrix to obtain a second calibrated restoration image until the user no longer clicks to input a second image continue calibration instruction;

[0116] Update the first image restoration model according to the front under-screen image restoration coefficient calibration matrix.

[0117] Specifically, when the user is still not satisfied with the image quality of the first calibrated restoration image obtained after multiple iterations, the user can select a second image calibration option to perform a higher-level image calibration. At this time, the display device system receives the user's second image calibration instruction. In addition, the user can also directly select the second image calibration option when not satisfied with the image quality of the first restored image or the second restored image.

[0118] Figure 8 It is a schematic diagram of an under-screen image calibration process provided by an embodiment of the present invention. Refer to Figure 1 、 Figure 2 and Figure 8, after the display device system receives the second image calibration instruction, it activates the second camera 22. Along with the appearance of a fine grid coordinate system and a central red dot in the shooting interface, the user locates the center and outer contour of the second calibration object in the shooting interface using the fine grid coordinate system and the central red dot, and captures the fourth calibration image Normal_Cal_2. Then, the first camera 21 is activated, and its selfie mirror function is turned off, so that the second calibration object in the shooting screen with the fine grid coordinate system and the central red dot is kept consistent with the actual situation in terms of left and right. When using the first camera 21 to shoot, the user needs to flip the display device and observe the second calibration object in the shooting interface from the side or by means of a front reflector (such as a mirror) to make its overall position the same as that when the second camera 22 took pictures before, so as to obtain the third calibration image Front_Cal_2. Subsequently, the display device system uses an artificial intelligence algorithm to identify the image differences between the third calibration image Front_Cal_2 and the fourth calibration image Normal_Cal_2, and learns and trains the front screen image restoration coefficient calibration matrix Front_Re_N+1.

[0119] The display device system processes the first image using the front screen image restoration coefficient calibration matrix Front_Re_N+1, generates and displays a second calibration restoration image close to the image quality level of the image captured by the second camera 22. The user visually experiences the image quality of the second calibration restoration image. If satisfied, the user selects to exit the shooting interface, and the display device system will update the front screen image restoration coefficient calibration matrix Front_Re_N+1 to the first image restoration model. If not satisfied, the user can repeatedly select the second image to continue the calibration option, so that the display device system uses an artificial intelligence algorithm to iteratively identify the image differences between the third calibration image Front_Cal_2 and the fourth calibration image Normal_Cal_2, learns and trains an optimized front screen image restoration coefficient calibration matrix Front_Re_N+1, and re-processes the first image to generate a second calibration restoration image close to the image quality level of the image captured by the second camera 22 until the user is satisfied with the image quality of the second calibration restoration image. Finally, the first image restoration model can be updated according to the optimized front screen image restoration coefficient calibration matrix Front_Re_N+1.

[0120] Optionally, the screen image calibration process further includes:

[0121] After the display device system receives the user's second image calibration instruction, it activates the third camera 23 and obtains the fifth calibration image captured by the user using the third camera 23 for the second calibration object;

[0122] Determine the front screen image conversion coefficient calibration matrix according to the differences between the third calibration image and the fifth calibration image;

[0123] Determine the second rear screen image restoration coefficient calibration matrix based on the difference between the fourth calibration image and the fifth calibration image;

[0124] Process the first image using the front screen image conversion coefficient calibration matrix to obtain a second converted image, and process the second converted image using the second rear screen image restoration coefficient calibration matrix to obtain a third calibrated restoration image;

[0125] The display device system presents the third calibrated restoration image to the user, and after receiving the user's second image continue calibration instruction, iteratively optimize the front screen image conversion coefficient calibration matrix and the second rear screen image restoration coefficient calibration matrix. Use the iteratively optimized front screen image conversion coefficient calibration matrix to reprocess the first image to obtain a second converted image, and use the iteratively optimized second rear screen image restoration coefficient calibration matrix to process the reprocessed second converted image to obtain a third calibrated restoration image until no more second image continue calibration instructions are received from the user's click input;

[0126] Update the first image conversion model according to the front screen image conversion coefficient calibration matrix;

[0127] Update the second image restoration model according to the second rear screen image restoration coefficient calibration matrix.

[0128] Specifically, referring to Figure 1 、 Figure 2 、 Figure 4 and Figure 8 ,when the display device system receives the second image calibration instruction, it can also control the third camera 23 to start. Along with the appearance of a fine grid coordinate system and a center red dot in the shooting interface, the user uses the fine grid coordinate system and the center red dot to locate the center and its outer contour of the second calibration object in the shooting interface, making its position in the shooting interface the same as when the second camera 22 takes a picture, to obtain the fifth calibration image Rear_Cal_2. Subsequently, the display device system uses an artificial intelligence algorithm to identify the image difference between the fifth calibration image Rear_Cal_2 and the fourth calibration image Normal_Cal_2, learns and trains the second rear screen image restoration coefficient calibration matrix Rear_Re_N+2, and learns and trains the front screen image conversion coefficient calibration matrix Front_Con_N+1 according to the image difference between the third calibration image Front_Cal_2 and the fifth calibration image Rear_Cal_2.

[0129] Subsequently, the display device system processes the first image using the front screen image conversion coefficient calibration matrix Front_Con_N+1 to obtain a second converted image, and processes the second converted image using the second rear screen image restoration coefficient calibration matrix Rear_Re_N+2 to generate a third calibrated and restored image close to the image quality level captured by the second camera 22; the user visually experiences the image quality of the third calibrated and restored image. If satisfied, the user selects to exit the shooting interface, and the display device system will update the front screen image conversion coefficient calibration matrix Front_Con_N+1 and the second rear screen image restoration coefficient calibration matrix Rear_Re_N+2 to the first image conversion model and the second image restoration model respectively; if not satisfied, the user can select the second image continue calibration option multiple times. When the display device system receives the second image continue calibration instruction, it uses the artificial intelligence algorithm to iteratively identify the image differences between the third calibrated image Front_Cal_2 and the fifth calibrated image Rear_Cal_2, and the image differences between the fifth calibrated image Rear_Cal_2 and the fourth calibrated image Normal_Cal_2 respectively, and learns and trains an optimized front screen image conversion coefficient calibration matrix Front_Con_N+1 and a second rear screen image restoration coefficient calibration matrix Rear_Re_N+2, and then re-processes the first image to generate a third calibrated and restored image close to the image quality level captured by the second camera 22 until the user is satisfied with the image quality of the third calibrated and restored image.

[0130] Finally, update the optimized front screen image conversion coefficient calibration matrix Front_Con_N+1 to the first image conversion model; update the optimized second rear screen image restoration coefficient calibration matrix Rear_Re_N+2 to the second image restoration model.

[0131] In this embodiment, the second image calibration instruction clicked and input by the user can be received in real time, the images of the second calibration object under the same environmental conditions are captured by the first camera 21, the second camera 22, and the third camera 23, and the artificial intelligence algorithm is used to update and optimize the first image conversion model and the second image restoration model according to the differences between the images, so that the image quality of the images captured by the first camera 21 can be better restored and processed, and the image quality of the images captured by the first camera 21 can be improved.

[0132] Optionally, processing the first image using the first image restoration model to obtain the first restored image includes:

[0133] Select the front screen image restoration coefficient matrix corresponding to the current shooting environmental conditions through the first image restoration model, where the first image restoration model includes front screen image restoration coefficient matrices corresponding to multiple different environmental conditions;

[0134] Correspondingly, processing the first image using the first image conversion model to obtain a first converted image, and processing the first converted image using the second image restoration model to obtain a second restored image, includes:

[0135] Selecting a front-under-screen image conversion coefficient matrix corresponding to the current shooting environment conditions through the first image conversion model, and selecting a rear-under-screen image restoration coefficient matrix corresponding to the current shooting environment conditions through the second image restoration model; wherein, the first image conversion model includes front-under-screen image conversion coefficient matrices corresponding to multiple different environment conditions respectively, and the second image restoration model includes rear-under-screen image restoration coefficient matrices corresponding to multiple different environment conditions respectively.

[0136] Among them, the environment conditions include environmental parameters such as environmental brightness and environmental color temperature that affect the photographing effect. A first image restoration model including front-under-screen image restoration coefficient matrices corresponding to multiple different environment conditions respectively can be determined in advance. When the user uses the first camera 21 to capture an image, the display device system uses the first image restoration model to select the corresponding front-under-screen image restoration coefficient matrix according to the current shooting environment conditions, so as to better process the first image and improve its image quality.

[0137] Optionally, the front-under-screen image restoration coefficient matrix is determined according to the differences between the images of the same preset object captured by the first camera 21 and the second camera 22 at the same position and under the same environment conditions, and the differences between the images of the same environment scene captured at the same position and under the same environment conditions;

[0138] The front-under-screen image conversion coefficient matrix is determined according to the differences between the images of the same preset object captured by the first camera 21 and the third camera 23 at the same position and under the same environment conditions, and the differences between the images of the same environment scene captured at the same position and under the same environment conditions. The rear-under-screen image restoration coefficient matrix is determined according to the differences between the images of the same preset object captured by the second camera 22 and the third camera 23 at the same position and under the same environment conditions, and the differences between the images of the same environment scene captured at the same position and under the same environment conditions.

[0139] The specific determination process of the first image restoration model, the first image conversion model, and the second image restoration model includes:

[0140] 1. In an indoor environment with adjustable environmental conditions, for example, environmental parameters such as brightness and color temperature, fix the display device with the first camera 21, the second camera 22, and the third camera 23 vertically on the fixture stage, and simulate the selfie distance at 20 - 60 cm directly in front of the stage. Place a monitor to display any content image of the simulated photographed person.

[0141] 2. After selecting a certain environmental condition and the content of the display image, that is, the image acquisition environmental condition 1, simulating the shooting environment, the display device first obtains the display image through the first camera 21 to get the first object image Front_Obj_1. At this time, the self-portrait mirror function of the first camera 21 will be turned off. Then the second camera 22 directly shoots the environmental scene on the side of the display device facing away from the display to obtain the first environmental image Normal_Env_1. Immediately afterwards, the fixture stage controls the third camera 23 to move to the position where the second camera 22 shoots the environmental image, and shoots the environmental scene to obtain the second environmental image Rear_Env_1.

[0142] 3. The fixture stage first rotates 180°, controls the third camera 23 to move to the position where the first camera 21 shoots the object image, and shoots the display image to obtain the second object image Rear_Obj_1. Immediately afterwards, it controls the second camera 22 to move to the position where the first camera 21 shoots the object image, and shoots the display image to obtain the third object image Normal_Obj_1.

[0143] 4. The fixture stage controls the first camera 21 to move to the position where the second camera 22 shoots the environmental image, and shoots the environmental scene to obtain the third environmental image Front_Env_1; at this time, the self-portrait mirror function of the first camera 21 will be turned off.

[0144] 5. The fixture stage rotates 180° again, and controls the first camera 21, the second camera 22 and the third camera 23 to return to their respective initial positions before image acquisition, thus completing the entire process of one-time image acquisition of the display device, and obtaining 3 kinds of object images and 3 kinds of environmental images under a certain environmental condition, that is, the first object image Front_Obj_1, the second object image Rear_Obj_1, the third object image Normal_Obj_1, the first environmental image Normal_Env_1, the second environmental image Rear_Env_1 and the third environmental image Front_Env_1.

[0145] 6. Adjust different environmental conditions, and repeat the above display device image acquisition process to obtain 3 kinds of object images and 3 kinds of environmental images under each environmental condition.

[0146] 7. Process the 3 kinds of object images and 3 kinds of environmental images under each environmental condition to obtain the pre-display screen image restoration coefficient matrix, the pre-display screen image conversion coefficient matrix and the post-display screen image restoration coefficient matrix under each environmental condition.

[0147] 8. Pool the pre - display - screen image restoration coefficient matrices under each environmental condition to obtain a first image restoration model. Pool the pre - display - screen image conversion coefficient matrices under each environmental condition to obtain a first image conversion model. Pool the post - display - screen image restoration coefficient matrices under each environmental condition to obtain a second image restoration model.

[0148] The processing procedure for the relevant images obtained for image acquisition environmental condition 1 includes:

[0149] Figure 9 It is the generation process diagram of the pre - display - screen image restoration coefficient matrix provided by the embodiment of the present invention. Refer to Figure 9, the process includes: identifying the image difference between the first object image Front_Obj_1 and the third object image Normal_Obj_1 through an artificial intelligence algorithm, such as a neural network method, etc., and learning and training a pre-display image restoration coefficient matrix. Using the pre-display image restoration coefficient matrix to process the first object image Front_Obj_1 to generate a first object restored image, and then through an image quality evaluation function, such as the Vollaths function, etc., to determine whether the image quality of the first object restored image is close to the normal image quality level of the third object image Normal_Obj_1. If not qualified, continuously iterate using the artificial intelligence algorithm to identify the image difference between the first object image Front_Obj_1 and the third object image Normal_Obj_1, learn and train an optimized pre-display image restoration coefficient matrix, and re-process the first object image Front_Obj_1 to generate a first object restored image until the image quality evaluation function determines it to be qualified; if qualified, use the pre-display image restoration coefficient matrix that can make the image quality of the processed first object image Front_Obj_1 be determined to be qualified by the image quality evaluation function to continue processing the third environment image Front_Env_1 to generate a first environment restored image, and determine whether the image quality of the first environment restored image is close to the normal image quality level of the first environment image Normal_Env_1 through the image quality evaluation function; if qualified, output the pre-display image restoration coefficient matrix that can make the image quality of the processed third environment image Front_Env_1 be determined to be qualified by the image quality evaluation function as the coefficient matrix for processing the photographing image quality of the first camera 21 under environmental condition 1; if not qualified, re-use the artificial intelligence algorithm to continuously iterate to identify the image difference between the first object image Front_Obj_1 and the third object image Normal_Obj_1, learn and train an optimized pre-display image restoration coefficient matrix until the image quality evaluation function determines that the image qualities of the first object restored image and the first environment restored image generated by using the optimized pre-display image restoration coefficient matrix are both qualified, and output the finally iteratively optimized pre-display image restoration coefficient matrix as the coefficient matrix for processing the photographing image quality of the first camera 21 under environmental condition 1.

[0150] Figure 10 is the generation process diagram of the pre-display image conversion coefficient matrix and the post-display image restoration coefficient matrix provided by the embodiment of the present invention. Refer to Figure 10, the artificial intelligence algorithm is used to identify the image differences between the first object image Front_Obj_1 and the second object image Rear_Obj_1, and between the second object image Rear_Obj_1 and the third object image Normal_Obj_1 respectively, so as to learn and train the image conversion coefficient matrix under the front screen, enabling the image quality captured by the first camera 21 to be converted to be close to the image quality level captured by the third camera 23, and learning and training the image restoration coefficient matrix under the rear screen, enabling the image quality captured by the third camera 23 to be restored to be close to the image quality level captured by the second camera 22.

[0151] Similar to the output determination method of the image restoration coefficient matrix under the front screen above, first use the image conversion coefficient matrix under the front screen to process the first object image Front_Obj_1 to generate an object conversion image, and use the image restoration coefficient matrix under the rear screen to process the second object image Rear_Obj_1 to generate a second object restoration image; through the image quality evaluation function, respectively judge whether the image quality of the object conversion image is close to the image quality level of the second object image Rear_Obj_1 and whether the image quality of the second object restoration image is close to the normal image quality level of the third object image Normal_Obj_1, so as to directly or iteratively optimize and then give the qualified image conversion coefficient matrix under the front screen and the image restoration coefficient matrix under the rear screen.

[0152] Use the qualified pre - display - screen image conversion coefficient matrix obtained by the image quality evaluation function to process the third environmental image Front_Env_1 to obtain an environmental conversion image, and use the qualified post - display - screen image restoration coefficient matrix obtained by the image quality evaluation function to process the second environmental image Rear_Env_1 to generate a second environmental restoration image; again, through the image quality evaluation function, respectively determine whether the image quality of the environmental conversion image is close to the image quality level of the second environmental image Rear_Env_1 and whether the image quality of the second environmental restoration image is close to the normal image quality level of the first environmental image Normal_Env_1; if not qualified, repeatedly use the artificial intelligence algorithm to iteratively identify the image differences between the first object image Front_Obj_1 and the second object image Rear_Obj_1, and between the second object image Rear_Obj_1 and the third object image Normal_Obj_1, and learn and train an optimized pre - display - screen image conversion coefficient matrix and a post - display - screen image restoration coefficient matrix until the image quality evaluation function respectively determines that the image qualities of the environmental conversion image generated by using the optimized pre - display - screen image conversion coefficient matrix and the second environmental restoration image generated by using the optimized post - display - screen image restoration coefficient matrix are qualified; if qualified, use the qualified post - display - screen image restoration coefficient matrix obtained by the again image quality evaluation function to process the environmental conversion image to generate a third environmental restoration image. Once again, through the image quality evaluation function, determine whether the image quality of the third environmental restoration image is close to the normal image quality level of the first environmental image Normal_Env_1; if qualified, output the pre - display - screen image conversion coefficient matrix that can make the image quality of the processed third environmental image Front_Env_1 be determined as qualified by the again image quality evaluation function and the post - display - screen image restoration coefficient matrix that can make the image quality of the processed environmental conversion image be determined as qualified by the once - again image quality evaluation function as the coefficient matrix for processing the photographing image quality of the first camera 21 under environmental condition 1; if not qualified, repeatedly use the artificial intelligence algorithm to iteratively identify the image differences between the first object image Front_Obj_1 and the second object image Rear_Obj_1, and between the second object image Rear_Obj_1 and the third object image Normal_Obj_1, and learn and train an optimized pre - display - screen image conversion coefficient matrix and a post - display - screen image restoration coefficient matrix until the once - again image quality evaluation function determines that the image quality of the third environmental restoration image generated by using the optimized post - display - screen image restoration coefficient matrix is qualified, and output the finally iteratively optimized pre - display - screen image conversion coefficient matrix and post - display - screen image restoration coefficient matrix.

[0153] In addition, it should be noted that the specific positional relationship and distance between the second camera 22 and the third camera 23 can be set as required, and this embodiment does not make any limitations. Optionally, referring to Figure 1 along the direction parallel to the non-light-emitting side plane of the display panel 10, the distance between the second camera 22 and the third camera 23 is greater than or equal to 10 mm, so as to reduce the installation process difficulty of the two cameras.

[0154] This embodiment also provides a display device, including a first camera disposed under the screen, a second camera disposed outside the screen, and an under-screen photographing processing device;

[0155] The under-screen photographing processing device includes:

[0156] A first image acquisition module, configured to acquire a first image captured by the user using the first camera;

[0157] A first restored image determination module, configured to process the first image using a first image restoration model to obtain a first restored image; wherein, the first image restoration model is determined according to the difference between the images captured by the first camera and the second camera under a first preset condition.

[0158] Optionally, the display device further includes a display panel and a third camera, and a transparent shielding member is disposed on the light incident surface of the third camera;

[0159] The under-screen photographing processing device further includes:

[0160] An image optimization module, configured to, after receiving an image optimization instruction from the user, process the first image using a first image conversion model to obtain a first converted image, and process the first converted image using a second image restoration model to obtain a second restored image; wherein, the first image conversion model is determined according to the difference between the images captured by the first camera and the third camera under a second preset condition, the second image restoration model is determined according to the difference between the images captured by the second camera and the third camera under a third preset condition, and when the third camera captures an image, light passes through the transparent shielding member and then enters the third camera.

[0161] Optionally, the display device further includes:

[0162] A display panel, the display panel includes a transparent display area, the first camera is disposed on the non-light-emitting side of the transparent display area, and the light incident surface of the first camera is adjacent to the transparent display area.

[0163] Optionally, the under-screen photographing processing device further includes:

[0164] A second image display module for displaying the second restored image to the user;

[0165] A first calibration image acquisition module for, after receiving a first image calibration instruction from the user, controlling the second camera to start and acquiring a first calibration image obtained by the user taking a picture of a first calibration object with the second camera;

[0166] A second calibration image acquisition module for controlling the third camera to start and acquiring a second calibration image obtained by the user taking a picture of a first calibration object with the third camera;

[0167] A first calibration matrix determination module for determining a first rear - screen image restoration coefficient calibration matrix based on the difference between the first calibration image and the second calibration image;

[0168] A first calibration restored image determination module for processing the first converted image with the first rear - screen image restoration coefficient calibration matrix to obtain a first calibration restored image;

[0169] A first iterative optimization module for displaying the first calibration restored image to the user and, after receiving a first image continue calibration instruction clicked by the user, iteratively optimizing the first rear - screen image restoration coefficient calibration matrix, re - processing the first converted image with the iteratively optimized first rear - screen image restoration coefficient calibration matrix to obtain a first calibration restored image until the user no longer clicks to input a first image continue calibration instruction;

[0170] A first model update module for updating the second image restoration model according to the finally determined first rear - screen image restoration coefficient calibration matrix.

[0171] Optionally, the under - screen photographing processing device further includes:

[0172] A third calibration image acquisition module for, after receiving a second image calibration instruction clicked by the user, starting the first camera and acquiring a third calibration image obtained by the user taking a picture of a second calibration object with the first camera;

[0173] A fourth calibration image acquisition module for starting the second camera and acquiring a fourth calibration image obtained by the user taking a picture of a second calibration object with the second camera;

[0174] A second calibration matrix determination module for determining a front - screen image restoration coefficient calibration matrix based on the difference between the third calibration image and the fourth calibration image;

[0175] A second calibration restored image determination module for processing the first image with the front - screen image restoration coefficient calibration matrix to obtain a second calibration restored image;

[0176] A second iterative optimization module, configured to display the second calibrated recovery image to the user, and iteratively optimize the pre-display screen image recovery coefficient calibration matrix after receiving the second image continuous calibration instruction clicked and input by the user, and re-process the first image with the iteratively optimized pre-display screen image recovery coefficient calibration matrix to obtain a second calibrated recovery image until the user no longer clicks and inputs the second image continuous calibration instruction;

[0177] A second model update module, configured to update the first image recovery model according to the pre-display screen image recovery coefficient calibration matrix.

[0178] Optionally, the under-display photographing processing device further includes:

[0179] A fifth calibration image acquisition module, configured to start the third camera after receiving the second image calibration instruction of the user, and acquire a fifth calibration image obtained by the user photographing a second calibration object with the third camera;

[0180] A third calibration matrix determination module, configured to determine a pre-display screen image conversion coefficient calibration matrix according to the difference between the third calibration image and the fifth calibration image;

[0181] A fourth calibration matrix determination module, configured to determine a second post-display screen image recovery coefficient calibration matrix according to the difference between the fourth calibration image and the fifth calibration image;

[0182] A third calibrated recovery image determination module, configured to process the first image with the pre-display screen image conversion coefficient calibration matrix to obtain a second converted image, and process the second converted image with the second post-display screen image recovery coefficient calibration matrix to obtain a third calibrated recovery image;

[0183] A third iterative optimization module, configured to display the third calibrated recovery image to the user, and iteratively optimize the pre-display screen image conversion coefficient calibration matrix and the second post-display screen image recovery coefficient calibration matrix after receiving the second image continuous calibration instruction clicked and input by the user, re-process the first image with the iteratively optimized pre-display screen image conversion coefficient calibration matrix to obtain a second converted image, and process the re-processed second converted image with the iteratively optimized second post-display screen image recovery coefficient calibration matrix to obtain a third calibrated recovery image until the user no longer clicks and inputs the second image continuous calibration instruction;

[0184] A third model update module, configured to update the first image conversion model according to the pre-display screen image conversion coefficient calibration matrix, and update the second image recovery model according to the second post-display screen image recovery coefficient calibration matrix.

[0185] Optionally, the first restored image determination module is specifically configured to:

[0186] Select a front - screen image restoration coefficient matrix corresponding to the current shooting environment condition through a first image restoration model, where the first image restoration model includes front - screen image restoration coefficient matrices corresponding to multiple different environment conditions;

[0187] Correspondingly, the image optimization module is specifically configured to:

[0188] Select a front - screen image conversion coefficient matrix corresponding to the current shooting environment condition through a first image conversion model, and select a rear - screen image conversion coefficient matrix corresponding to the current shooting environment condition through the second image restoration model; where the first image conversion model includes front - screen image conversion coefficient matrices corresponding to multiple different environment conditions, and the second image restoration model includes rear - screen image restoration coefficient matrices corresponding to multiple different environment conditions.

[0189] Optionally, the front - screen image restoration coefficient matrix is determined according to the differences between the images of the same preset object captured by the first camera and the second camera at the same position and under the same environment condition, and the differences between the images of the same environment scene captured at the same position and under the same environment condition;

[0190] The front - screen image conversion coefficient matrix is determined according to the differences between the images of the same preset object captured by the first camera and the third camera at the same position and under the same environment condition, and the differences between the images of the same environment scene captured at the same position and under the same environment condition. The rear - screen image restoration coefficient matrix is determined according to the differences between the images of the same preset object captured by the second camera and the third camera at the same position and under the same environment condition, and the differences between the images of the same environment scene captured at the same position and under the same environment condition.

[0191] Specifically, the display device provided in the embodiments of the present invention may be a mobile phone, a wearable device with a display function, a computer, or other display devices.

[0192] Note that the above are only the preferred embodiments of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described here. Various obvious changes, re - adjustments, combinations with each other, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, it may include more other equivalent embodiments, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for processing under-screen photography, which is applied to a display device. The display device includes a first camera disposed under the screen and a second camera disposed outside the screen. Characterized in that, The method for processing under-screen photography includes: Obtaining a first image captured by the first camera; Processing the first image with a first image restoration model to obtain a first restored image. Wherein, the first image restoration model is determined according to the differences between the images captured by the first camera and the second camera under a first preset condition; The display device further includes a third camera for simulating under-screen photography of the first camera. When the third camera captures an image, light passes through a transparent shielding member and then enters the third camera; The method further includes: Displaying the first restored image to the user; After receiving an image optimization instruction from the user, processing the first image with a first image conversion model to obtain a first converted image, and processing the first converted image with a second image restoration model to obtain a second restored image. Wherein, the first image conversion model is determined according to the differences between the images captured by the first camera and the third camera under a second preset condition, and the second image restoration model is determined according to the differences between the images captured by the second camera and the third camera under a third preset condition.

2. The method according to claim 1, Characterized in that, It further includes: Displaying the second restored image to the user; After receiving a first image calibration instruction from the user, controlling the second camera to start, and obtaining a first calibration image captured by the user using the second camera for a first calibration object; Controlling the third camera to start, and obtaining a second calibration image captured by the user using the third camera for the first calibration object; Determining a first post-under-screen image restoration coefficient calibration matrix according to the differences between the first calibration image and the second calibration image; Processing the first converted image with the first post-under-screen image restoration coefficient calibration matrix to obtain a first calibrated restored image; Displaying the first calibrated restored image to the user, and iteratively optimizing the first post-under-screen image restoration coefficient calibration matrix after receiving a first image continue calibration instruction clicked and input by the user, and re-processing the first converted image with the iteratively optimized first post-under-screen image restoration coefficient calibration matrix to obtain a first calibrated restored image until the user no longer clicks and inputs a first image continue calibration instruction; Updating the second image restoration model according to the finally determined first post-under-screen image restoration coefficient calibration matrix.

3. The method according to any one of claims 1-2, Characterized in that, It further includes: After receiving a second image calibration instruction clicked and input by the user, starting the first camera, and obtaining a third calibration image of a second calibration object captured by the user using the first camera; Starting the second camera, and obtaining a fourth calibration image captured by the user using the second camera for the second calibration object; Determining a pre-under-screen image restoration coefficient calibration matrix according to the differences between the third calibration image and the fourth calibration image; Process the first image using the pre-display-under image restoration coefficient calibration matrix to obtain a second calibrated and restored image; Display the second calibrated and restored image to the user, and after receiving the user's click input of the second image continue calibration instruction, iteratively optimize the pre-display-under image restoration coefficient calibration matrix. Use the iteratively optimized pre-display-under image restoration coefficient calibration matrix to re-process the first image to obtain a second calibrated and restored image until the user no longer clicks to input the second image continue calibration instruction; Update the first image restoration model according to the pre-display-under image restoration coefficient calibration matrix.

4. The method according to claim 3, wherein, further comprising: After receiving the user's second image calibration instruction, activate the third camera and obtain a fifth calibrated image obtained by the user using the third camera to photograph a second calibrated object; Determine the pre-display-under image conversion coefficient calibration matrix according to the difference between the third calibrated image and the fifth calibrated image; Determine the second post-display-under image restoration coefficient calibration matrix according to the difference between the fourth calibrated image and the fifth calibrated image; Process the first image using the pre-display-under image conversion coefficient calibration matrix to obtain a second converted image, and process the second converted image using the second post-display-under image restoration coefficient calibration matrix to obtain a third calibrated and restored image; Display the third calibrated and restored image to the user, and after receiving the user's second image continue calibration instruction, iteratively optimize the pre-display-under image conversion coefficient calibration matrix and the second post-display-under image restoration coefficient calibration matrix. Use the iteratively optimized pre-display-under image conversion coefficient calibration matrix to re-process the first image to obtain a second converted image, and use the iteratively optimized second post-display-under image restoration coefficient calibration matrix to process the re-processed second converted image to obtain a third calibrated and restored image until the user no longer clicks to input the second image continue calibration instruction; Update the first image conversion model according to the pre-display-under image conversion coefficient calibration matrix; Update the second image restoration model according to the second post-display-under image restoration coefficient calibration matrix.

5. The method according to claim 1, wherein, Processing the first image using the first image restoration model to obtain a first restored image includes: Select a pre-display-under image restoration coefficient matrix corresponding to the current shooting environment condition through the first image restoration model, wherein the first image restoration model includes pre-display-under image restoration coefficient matrices corresponding to multiple different environment conditions; Correspondingly, processing the first image using the first image conversion model to obtain a first converted image, and processing the first converted image using the second image restoration model to obtain a second restored image, includes: Select a front - screen - under image conversion coefficient matrix corresponding to the current shooting environmental conditions through the first image conversion model, and select a rear - screen - under image conversion coefficient matrix corresponding to the current shooting environmental conditions through the second image restoration model; wherein, the first image conversion model includes front - screen - under image conversion coefficient matrices corresponding to multiple different environmental conditions respectively, and the second image restoration model includes rear - screen - under image restoration coefficient matrices corresponding to multiple different environmental conditions respectively.

6. The method according to claim 5, wherein: The front - screen - under image restoration coefficient matrix is determined according to the differences between the images of the same preset object taken by the first camera and the second camera at the same position and under the same environmental conditions, and the differences between the images of the same environmental scene taken at the same position and under the same environmental conditions; The front - screen - under image conversion coefficient matrix is determined according to the differences between the images of the same preset object taken by the first camera and the third camera at the same position and under the same environmental conditions, and the differences between the images of the same environmental scene taken at the same position and under the same environmental conditions. The rear - screen - under image restoration coefficient matrix is determined according to the differences between the images of the same preset object taken by the second camera and the third camera at the same position and under the same environmental conditions, and the differences between the images of the same environmental scene taken at the same position and under the same environmental conditions.

7. A display device, wherein, it includes: A first camera disposed under the screen, a second camera disposed outside the screen, and an under - screen photographing processing device; The under - screen photographing processing device includes: A first image acquisition module for acquiring a first image taken by the user using the first camera; A first image restoration determination module for processing the first image using a first image restoration model to obtain a first restored image; wherein, the first image restoration model is determined according to the differences between the images taken by the first camera and the second camera under a first preset condition; The display device further includes a display panel and a third camera, and a transparent shielding member is disposed on the light - incident surface of the third camera; The under - screen photographing processing device further includes: An image optimization module for, after receiving an image optimization instruction from the user, processing the first image using a first image conversion model to obtain a first converted image, and processing the first converted image using a second image restoration model to obtain a second restored image; wherein, the first image conversion model is determined according to the differences between the images taken by the first camera and the third camera under a second preset condition, the second image restoration model is determined according to the differences between the images taken by the second camera and the third camera under a third preset condition, and when the third camera takes an image, light passes through the transparent shielding member and then enters the third camera.

8. The display device according to claim 7, wherein, it further includes: A display panel, the display panel includes a transparent display area, the first camera is disposed on a non-light-emitting side of the transparent display area, and a light-incident surface of the first camera is adjacent to the transparent display area.

Images