An image processing method, device, mobile terminal and storage medium
By performing multiple rounds of deep learning processing on images captured by the under-display camera module, and by combining pre-trained and retrained models with different sample sets, the problem of poor image quality of the under-display camera module was solved, and the imaging effect of the under-display camera module was improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING XIAOMI MOBILE SOFTWARE CO LTD
- Filing Date
- 2021-09-02
- Publication Date
- 2026-06-05
Smart Images
Figure CN115767289B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of terminal technology, and in particular to an image processing method, apparatus, mobile terminal and storage medium. Background Technology
[0002] With the development of full-screen technology, its application in mobile terminals is becoming increasingly widespread. In related technologies, mobile terminals are equipped with a front-facing camera, and the portion of the display screen where the front-facing camera is mounted typically has a notch or hole to allow it to capture external images. However, this notch or hole reduces the screen-to-body ratio. To improve the screen-to-body ratio and achieve a true full-screen display, the front-facing camera can be hidden under the display without any openings. In this case, the camera can capture external images through the light-transmitting area of the display. However, due to the screen's obstruction, the image quality of images captured by this under-display camera is poor, affecting the user experience. Summary of the Invention
[0003] The purpose of this disclosure is to provide an image processing method, apparatus, mobile terminal, and storage medium, which aim to improve the image quality of images captured by an under-display camera module.
[0004] To achieve the above objectives, the present disclosure adopts the following technical solution:
[0005] According to a first aspect of the present disclosure, an image processing method is provided, applied to a mobile terminal, comprising:
[0006] Acquire a first image to be processed, wherein the first image is captured by a first camera module, and the first camera module is installed below the display screen of the mobile terminal;
[0007] The first image is processed using an image processing model to obtain a second image. The image processing model is trained using a preset image sample set. The sample images in the preset image sample set include images captured by a second camera module and a third camera model for the same subject, respectively. The second camera module is covered by the display screen, while the third camera module is not covered by the display screen.
[0008] In the above scheme, the image processing model includes: a pre-trained model and a re-trained model, wherein the model parameters of the pre-trained model and the re-trained model are different;
[0009] The step of processing the first image using an image processing model to obtain the second image includes:
[0010] The first image is processed using the pre-trained model to obtain the third image;
[0011] The retrained model is used to process the third image to obtain the second image.
[0012] In the above scheme, the preset image sample set includes: a first image sample set and a second image sample set, wherein the sample images included in the first image sample set and the second image sample set are different;
[0013] The method includes:
[0014] The pre-trained model is obtained by training the first image sample set, and the re-trained model is obtained by training the second image sample set.
[0015] In the above scheme, the method includes:
[0016] The second image sample set is obtained by processing the first sample image captured by the second camera module using the pre-trained model, and by collecting the third sample image from the third camera module for the same subject as the first sample image.
[0017] In the above scheme, the preset image sample set includes at least: a first training sample set, a first verification sample set, and a first test sample set;
[0018] The method includes:
[0019] A first model is obtained by training a preset model using the first training sample set and the first verification sample set. The first training sample set is used to train the preset model to obtain training results. The first verification sample set is used to verify the training results to indicate whether to end the training of the preset model.
[0020] The first model is tested using the first test sample set to obtain the first test result;
[0021] In response to the first test result indicating that the image quality of the image processed by the first model meets the image quality conditions, the first model is used as the image processing model.
[0022] In the above scheme, the first training sample set, the first verification sample set, and the first test sample set include different sample images.
[0023] In the above scheme, the method includes at least one of the following:
[0024] In response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, the preset model is trained using the Nth training sample set and the Nth validation sample set to obtain the Nth model, and the Nth model is tested using the Nth test sample set to obtain the Nth test result, until the Nth test result indicates that the image quality of the image processed by the Nth model meets the image quality condition, and the Nth model is used as the image processing model, wherein N is a positive integer greater than or equal to 2;
[0025] In response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, the model parameters of the first model are adjusted to obtain the Mth model. The Mth model is then tested using the first test sample set to obtain the Mth test result. This process continues until the Mth test result indicates that the image quality of the image processed by the Mth model meets the image quality condition. The Mth model is then used as the image processing model, where M is a positive integer greater than or equal to 2.
[0026] In the above scheme, processing the first image using the image processing model includes at least one of the following:
[0027] The image processing model is used to perform dehazing on the first image;
[0028] The image processing model is used to denoise the first image;
[0029] The first image is de-diffraction processed using the image processing model described above;
[0030] The image processing model is used to perform color correction on the first image.
[0031] According to a second aspect of the present disclosure, an image processing apparatus is provided, applied to a mobile terminal, comprising:
[0032] A model is acquired to acquire a first image to be processed, wherein the first image is captured by a first camera module, and the first camera module is installed below the display screen of the mobile terminal.
[0033] The first processing module is used to process the first image using an image processing model to obtain a second image. The image processing model is trained using a preset image sample set. The sample images in the preset image sample set include images captured by the second camera module and the third camera module for the same subject, respectively. The second camera module is covered by the display screen, and the third camera module is not covered by the display screen.
[0034] In the above scheme, the image processing model includes: a pre-trained model and a re-trained model, wherein the model parameters of the pre-trained model and the re-trained model are different;
[0035] The first processing module is used for:
[0036] The first image is processed using the pre-trained model to obtain the third image;
[0037] The retrained model is used to process the third image to obtain the second image.
[0038] In the above scheme, the preset image sample set includes: a first image sample set and a second image sample set, wherein the sample images included in the first image sample set and the second image sample set are different;
[0039] The device includes:
[0040] The second processing model is used to train the pre-trained model using the first image sample set and to train the re-trained model using the second image sample set.
[0041] In the above scheme, the device includes:
[0042] The third processing model is used to process the first sample image captured by the second camera module to obtain the second sample image, and the third sample image captured by the third camera module for the same subject as the first sample image, to obtain the second image sample set.
[0043] In the above scheme, the preset image sample set includes at least: a first training sample set, a first verification sample set, and a first test sample set;
[0044] The device includes:
[0045] The fourth processing module is used to train a preset model using the first training sample set and the first verification sample set to obtain a first model, wherein the first training sample set is used to train the preset model to obtain a training result; and the first verification sample set is used to verify the training result to indicate whether the training of the preset model has been successful.
[0046] The fifth processing module is used to test the first model using the first test sample set to obtain the first test result;
[0047] The sixth processing module is used to use the first model as the image processing model in response to the first test result indicating that the image quality of the image processed by the first model meets the image quality conditions.
[0048] In the above scheme, the first training sample set, the first verification sample set, and the first test sample set include different sample images.
[0049] In the above scheme, the device includes at least one of the following:
[0050] The seventh processing module is configured to, in response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, train the preset model using the Nth training sample set and the Nth validation sample set to obtain the Nth model, test the Nth model using the Nth test sample set to obtain the Nth test result, until the Nth test result indicates that the image quality of the image processed by the Nth model meets the image quality condition, and use the Nth model as the image processing model, wherein N is a positive integer greater than or equal to 2;
[0051] The eighth processing model is used to adjust the model parameters of the first model in response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, to obtain the Mth model, to test the Mth model using the first test sample set, to obtain the Mth test result, until the Mth test result indicates that the image quality of the image processed by the Mth model meets the image quality condition, and to use the Mth model as the image processing model, wherein M is a positive integer greater than or equal to 2.
[0052] In the above scheme, the first processing module is used for at least one of the following:
[0053] The image processing model is used to perform dehazing on the first image;
[0054] The image processing model is used to denoise the first image;
[0055] The first image is de-diffraction processed using the image processing model described above;
[0056] The image processing model is used to perform color correction on the first image.
[0057] According to a third aspect of the present disclosure, a mobile terminal is provided, comprising:
[0058] A processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, implements any of the image processing method steps described above.
[0059] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, the program being executed by a processor to implement any of the image processing method steps described above.
[0060] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:
[0061] In embodiments of this disclosure, a first image to be processed is acquired, wherein the first image is captured by a first camera module installed below the display screen of the mobile terminal; an image processing model is used to process the first image to obtain a second image, wherein the image processing model is trained using a preset image sample set, and the sample images in the preset image sample set include images captured by a second camera module and a third camera module respectively for the same subject, wherein the second camera module is covered by the display screen, and the third camera module is not covered by the display screen. In other words, the embodiments of this disclosure use images captured by the third camera module not covered by the display screen and images captured by the second camera module covered by the display screen as sample images of a preset image sample set to train an image processing model. Therefore, the first image captured by the first camera module under the display screen of the mobile terminal is processed using this trained image processing model to obtain a second image. This second image is at least close to the image captured by the camera module not covered by the display screen. In this way, the image captured by the first camera module is optimized, the image quality of the image captured by the first camera module is improved, and the imaging effect of the under-display camera module is improved.
[0062] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0063] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0064] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment;
[0065] Figure 2 This is another flowchart illustrating an image processing method according to an exemplary embodiment;
[0066] Figure 3 This is yet another flowchart illustrating an image processing method according to an exemplary embodiment;
[0067] Figure 4This is a technical block diagram illustrating an image processing model according to an exemplary embodiment;
[0068] Figure 5 This is a block diagram illustrating an image processing technique according to an example embodiment;
[0069] Figure 6 This is a technical block diagram illustrating an image processing model according to an exemplary embodiment;
[0070] Figure 7 This is a functional block diagram illustrating an image processing method according to an exemplary embodiment;
[0071] Figure 8 This is a block diagram illustrating a mobile terminal according to an exemplary embodiment. Detailed Implementation
[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0073] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.
[0074] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. In the image processing method of this disclosure, an image recognition model is trained using machine learning methods, thereby enabling better recognition of the image to be processed for accurate classification.
[0075] Deep learning (DL) is a new research direction in the field of machine learning, bringing it closer to its original goal—artificial intelligence (AI). Deep learning learns the inherent patterns and hierarchical representations of sample data; the information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, capable of recognizing data such as text, images, and sound. It is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies. Deep learning has yielded significant achievements in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as sight, hearing, and thought, solving many complex pattern recognition problems and leading to significant advancements in AI-related technologies.
[0076] Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that incorporate convolutional computations and have a deep structure. They are one of the representative algorithms of deep learning. CNNs possess representation learning capabilities, enabling them to perform shift-invariant classification of input information according to their hierarchical structure; therefore, they are also known as Shift-Invariant Artificial Neural Networks (SIANNs).
[0077] Generative Adversarial Networks (GANs) are deep learning models that generate remarkably good outputs through a game-like learning process between at least two modules: a generative model and a discriminative model. The generative model generates data out of thin air, while the discriminative model determines whether the data is real. Through iterative training of the generative and discriminative models, a deep learning model that can predict near-real data can be obtained.
[0078] Variational autoencoders (VAEs) are an important class of generative models that combine two of the best approaches: neural networks and Bayesian inference. They have become one of the most popular methods in unsupervised learning. The advantage of VAEs is that they can directly compare the differences between the reconstructed image and the original image through the encoding and decoding steps.
[0079] It's important to note that under-display camera modules emerged in response to the advent of full-screen displays and the need to reduce screen-to-body ratio. However, because the under-display camera module is covered by the display screen, it needs to receive external light through the screen. This can lead to various issues affecting image quality during the imaging process, such as image spot diffraction, image noise, image sharpness, and color saturation. Consequently, images captured by under-display camera modules often exhibit poor quality, hindering their widespread adoption.
[0080] The embodiments disclosed herein aim to improve the image quality of images captured by under-display camera modules.
[0081] The embodiments disclosed herein are applied to scenarios where mobile terminals take pictures using an under-display camera module.
[0082] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment, such as... Figure 1 As shown, the method is applied to a mobile terminal and includes the following steps:
[0083] Step 101: Acquire a first image to be processed, wherein the first image is captured by a first camera module, and the first camera module is installed below the display screen of the mobile terminal;
[0084] Step 102: Process the first image using an image processing model to obtain a second image. The image processing model is trained using a preset image sample set. The sample images in the preset image sample set include images captured by the second camera module and the third camera module for the same subject, respectively. The second camera module is covered by the display screen, while the third camera module is not covered by the display screen.
[0085] Here, the mobile terminal can be a mobile phone, tablet, laptop, or wearable device; among which, wearable devices can be smartwatches or smart bracelets, etc. It is understood that the mobile terminal can include any mobile terminal with a display screen and camera functionality.
[0086] Here, the first camera module is a camera module installed under the display screen of the mobile terminal, which can be understood as an under-display camera module.
[0087] For example, the first camera module and the second camera module can be the same camera module or different camera modules. For example, the first camera module and the second camera module can be camera modules on different mobile terminals.
[0088] It is understandable that if the first camera module and the second camera module are camera modules on the same mobile terminal, that is, they cover the same display screen, then the image processing model trained by using the image captured by the second camera module as a sample image can better process the first image, resulting in better image quality.
[0089] For example, the second camera module and the third camera module can be two different camera modules of the same specification, or they can be the same camera module.
[0090] The camera module here can be a webcam. The specifications of the camera module can include: the external dimensions of the webcam within the camera module, the webcam focal length, the field of view, or the aperture, etc.
[0091] Here, the second and third camera modules can be targeting the same subject—either a person or a landscape. In other words, the second and third camera modules targeting the same subject can be considered to be capturing the same content.
[0092] In some embodiments, in order to improve the sample accuracy of the preset image samples, the sample images in the preset image sample set may further include: images acquired by the second camera module and the third camera module respectively for the same shooting object, at the same position, using the same shooting angle, the same shooting environment, and the same shooting parameters.
[0093] Therefore, since the third camera module is not covered by the display screen, images captured by the third camera module rarely exhibit problems such as image spot diffraction, image noise, image sharpness, and image color saturation. Thus, images captured by the third camera module can be used as the target sample set in a preset image sample set, effectively mitigating the aforementioned problems such as image spot diffraction, image noise, image sharpness, and image color saturation caused by the display screen's obstruction.
[0094] In some embodiments, processing the first image using the image processing model includes at least one of the following:
[0095] The image processing model is used to perform dehazing on the first image;
[0096] The image processing model is used to denoise the first image;
[0097] The first image is de-diffraction processed using the image processing model described above;
[0098] The image processing model is used to perform color correction on the first image.
[0099] In this embodiment, the image processing model is trained using images captured by the third camera module not covered by the display screen and images captured by the second camera module covered by the display screen as sample images in a preset image sample set. This allows the trained image processing model to perform at least one of the following processing on the first image: dehazing, noise reduction, de-diffraction, and color correction. In other words, by processing the first image captured by the first camera module under the mobile terminal's display screen using this trained image processing model, the resulting second image is at least close to the image captured by the camera module not covered by the display screen. This reduces issues such as image spot diffraction, image noise, image sharpness, and image color saturation in the final image captured by the first camera module. Thus, the image captured by the first camera module is optimized, improving its image quality and thereby enhancing the imaging effect of the under-display camera module.
[0100] In some exemplary embodiments, the image processing model includes a pre-trained model and a re-trained model, wherein the model parameters of the pre-trained model and the re-trained model are different; see also Figure 2 , Figure 2 This is another flowchart illustrating an image processing method according to an exemplary embodiment, such as... Figure 2 As shown, step 102, namely, processing the first image using an image processing model to obtain the second image, includes:
[0101] Step 1021: Process the first image using the pre-trained model to obtain the third image;
[0102] Step 1022: Process the third image using the retrained model to obtain the second image.
[0103] Here, the pre-trained model can be one or more of the following: Convolutional Neural Network (CNN), Generative Adversarial Network (GAN), and Variational Autoencoder (VAE). Of course, the pre-trained model can also be other deep learning models.
[0104] Here, the retrained model can be one or more of the following: Convolutional Neural Network (CNN), Generative Adversarial Network (GAN), and Variational Autoencoder (VAE). Of course, the retrained model can also be other deep learning models.
[0105] In this implementation, the first image is processed by a pre-trained model to obtain a third image. At this point, the third image has already reduced various image problems compared to the first image. Then, the third image is processed by a retrained model to obtain a second image. This allows the final second image to further reduce the aforementioned image problems and improve the image quality of the second image, thereby improving the imaging quality of the under-display camera module.
[0106] It should be added that, in order to achieve better image processing results, both the pre-trained and retrained models can be optimized using various loss functions during training.
[0107] In some exemplary embodiments, the preset image sample set includes: a first image sample set and a second image sample set, wherein the sample images included in the first image sample set and the second image sample set are different;
[0108] The method includes:
[0109] The pre-trained model is obtained by training the first image sample set, and the re-trained model is obtained by training the second image sample set.
[0110] It is understandable that the sample images in the preset sample set appear in pairs; that is, for the same subject, images captured by the second camera module and images captured by the third camera module can be labeled with the same sample label. It is also understandable that sample images obtained for different subjects will have different sample labels. Alternatively, for the same subject, different shooting times, shooting angles, or shooting parameters will also result in different labels for the sample images. In short, different sample images correspond to different sample labels.
[0111] Here, the sample images in the first image sample set and the second image sample set are different, which can be understood as the sample labels of the sample images in the first image sample set and the second image sample set being partially or completely different.
[0112] Thus, in this embodiment of the present disclosure, compared to training a pre-trained model and a re-trained model using the same set of image samples, the pre-trained model and the re-trained model are trained using different sets of image samples. This makes the image restoration capabilities of the pre-trained model and the re-trained model non-repeatable, resulting in better image restoration effects when the pre-trained model and the re-trained model trained using different sets of image samples perform image processing sequentially. This can further improve the image quality of the under-display camera module.
[0113] In some other exemplary embodiments, the method includes:
[0114] The second image sample set is obtained by processing the first sample image captured by the second camera module using the pre-trained model to obtain the second sample image, and by collecting the third sample image from the third camera module for the same subject as the first sample image.
[0115] It is understood that the sample images in the preset sample set include: the input image to be learned and the output target image. Here, the input image to be learned refers to the image captured by the second camera module, while the output target image refers to the image captured by the third camera module. In this embodiment, the input image to be learned, i.e., the image captured by the second camera module, is optimized. Specifically, the pre-trained model is used to process the first sample image captured by the second camera module to obtain the second sample image, thereby reducing the amount of data processing required during the retraining of the model and improving the processing efficiency of the retraining model.
[0116] In this embodiment of the disclosure, by optimizing the second image sample set, the amount of data processing required for the retraining model is reduced, and the processing efficiency of the retraining model is improved.
[0117] Of course, in other embodiments, the number of samples in the second image sample set is greater than the number of samples in the first image sample set. This makes the image restoration capability of the retrained model stronger than that of the pre-trained model. Thus, the first image can be initially repaired by the pre-trained model, and then further refined by the retrained model, resulting in a second image of higher quality.
[0118] In other embodiments, the loss functions used by the pre-trained model and the retrained model are different. In this way, the first image can be processed from different loss perspectives to obtain a second image with better restoration effect, thereby improving the image quality of the images captured by the under-display camera module.
[0119] It should be added that, in some embodiments, the first image may be an image that has undergone preprocessing such as normalization.
[0120] This allows the image processing model to process less data from the first image, thus improving the processing efficiency of the image processing model.
[0121] Based on this, in some embodiments, the method includes:
[0122] The second image is then subjected to inverse normalization.
[0123] In other embodiments, the first image may be an image that has undergone preprocessing such as data augmentation.
[0124] In this way, the image processing model can process the first image with richer data, reduce the omission of problematic features in the image, and thus improve the image quality of the second image.
[0125] In other embodiments, the method includes:
[0126] The second image is then subjected to conventional image processing, such as color enhancement or background blurring.
[0127] Thus, in this embodiment of the disclosure, by further performing conventional image processing on the second image, the image quality of the second image is improved.
[0128] In some exemplary embodiments, the preset image sample set includes at least: a first training sample set, a first verification sample set, and a first test sample set; see also Figure 3 , Figure 3 This is another flowchart illustrating an image processing method according to an exemplary embodiment, such as... Figure 3 As shown, the method includes:
[0129] Step 301: Train the preset model using the first training sample set and the first verification sample set to obtain a first model, wherein the first training sample set is used to train the preset model to obtain training results; the first verification sample set is used to verify the training results to indicate whether to end the training of the preset model.
[0130] Step 302: Test the first model using the first test sample set to obtain the first test result;
[0131] Step 303: In response to the first test result indicating that the image quality of the image processed by the first model meets the image quality conditions, the first model is used as the image processing model.
[0132] For example, taking the image processing model as a Generative Adversarial Network (GAN), the first training sample set can be the sample set used to train the generative model, and the first validation sample set can be the sample set used to discriminate the model. Here, the first validation sample set is used to validate the training results to indicate whether the validation result indicates the end of training the preset model. It can be understood that if the first validation sample set verifies that the network function of the generative adversarial network has converged, it indicates the end of training the preset model.
[0133] Here, meeting the image quality conditions can include:
[0134] The overlap between the image obtained by the first model and the image captured by the third camera module for the same subject in the preset image sample set is greater than or equal to the overlap threshold, for example, it can be 90%.
[0135] Of course, meeting image quality requirements can also include:
[0136] The image quality score of the image processed by the first model is higher than the preset score. Here, the image quality score refers to the comprehensive score obtained by assigning corresponding weights to factors such as color, haze, diffraction, and noise in the image. It can be understood that a higher score corresponds to higher image quality.
[0137] In this embodiment, a first test sample set is introduced to test the first model. Based on the test results, it is determined whether the image quality of the image processed by the first model meets the image quality requirements. The first model that meets the image quality requirements is then used as the image processing model. This ensures the image processing quality of the image processing model, thereby guaranteeing the image quality of the images generated by the under-display camera module. Thus, the introduction of the first test sample set guarantees the image quality of the images generated by the under-display camera module.
[0138] In some exemplary embodiments, the first training sample set, the first verification sample set, and the first test sample set include different sample images.
[0139] Here, the first training sample set, the first verification sample set, and the first test sample set include different sample images. This can also be understood as the labels of the sample images in the first training sample set, the first verification sample set, and the first test sample set being partially or completely different.
[0140] Thus, in this embodiment, by using three non-overlapping image sample sets to train the image processing model, the image processing model can process a wider range of image data. Therefore, it can further ensure the image quality of the second image processed by the image processing model, thereby ensuring the image quality of the image captured by the under-display camera module.
[0141] It should be added that, in some embodiments, the pre-trained model and / or retrained model can be trained using different first training sample sets, first validation sample sets, and first test sample sets, respectively.
[0142] In some other exemplary embodiments, the method includes at least one of the following:
[0143] In response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, the preset model is trained using the Nth training sample set and the Nth validation sample set to obtain the Nth model, and the Nth model is tested using the Nth test sample set to obtain the Nth test result, until the Nth test result indicates that the image quality of the image processed by the Nth model meets the image quality condition, and the Nth model is used as the image processing model, wherein N is a positive integer greater than or equal to 2;
[0144] In response to a first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, the model parameters of the first model are adjusted to obtain the Mth model. The Mth model is then tested using the first test sample set to obtain the Mth test result. The Mth test result indicates that the image quality of the image processed by the Mth model meets the image quality condition. The Mth model is then used as the image processing model, where M is a positive integer greater than or equal to 2.
[0145] Here, if the first test result indicates that the image quality of the image processed by the first model does not meet the image quality conditions, the preset model can be trained again by changing the training samples and validation samples, and the trained model can be tested by changing the test samples, until the image quality of the image processed by the trained model meets the image quality conditions.
[0146] And / or, if the first test result indicates that the image quality of the image processed by the first model does not meet the image quality conditions, the model parameters can be adjusted, and the preset model with adjusted model parameters can be tested using the first test sample until the image quality of the image processed by the model meets the image quality conditions.
[0147] Thus, in this embodiment, the preset model can ultimately achieve the ideal model effect through different methods, that is, the image quality of the processed image meets the image quality conditions, and this model that can ultimately achieve the ideal model effect is used as the image processing model to process the first image obtained by the mobile terminal, thereby ensuring the image quality of the second image obtained after processing by the image processing model, and thus ensuring the image quality of the under-screen camera module of the mobile terminal.
[0148] In some embodiments, the image processing model may include at least one of the following: a first image processing model for denoising, a second image processing model for dehazing, a third image processing model for de-diffraction, and a fourth image processing model for color correction.
[0149] It should be noted that the first preset image sample set used by the first image processing model for dehazing can be a sample image set in which the proportion of sample images with a hazy appearance is greater than a first proportion threshold. The second preset image sample set used by the second image processing model for denoising can be a sample image set in which the proportion of sample images with noise is greater than a second proportion threshold. The third preset image sample set used by the third image processing model for dediffraction can be a sample image set in which the proportion of image samples with light spot diffraction is greater than a third proportion threshold. The fourth preset image sample set used by the fourth image processing model for color correction can be a sample image set in which the proportion of image samples with problems such as color saturation or sharpness is greater than a fourth proportion threshold.
[0150] In this way, by training a first image processing model for dehazing, a second image processing model for noise reduction, a third image processing model for de-diffraction, and a fourth image processing model for color correction on targeted sample images, the image processing models can process images more specifically, resulting in a second image with better processing effect and higher image quality.
[0151] In some embodiments, the processing of the first image using an image processing model, the second image may include:
[0152] The first image is denoised using the first image processing model to obtain the fourth image.
[0153] The fourth image is dehazed using the second image processing model to obtain the fifth image;
[0154] The fifth image is de-diffraction processed using the third image processing model to obtain the sixth image;
[0155] The sixth image is color-corrected using the fourth image processing model to obtain the second image.
[0156] For example, the method may further include: sequentially performing denoising and dehazing processing on the first image using conventional image processing algorithms. For example, denoising processing may be performed using at least one of the temporal domain, spatial domain, and transform domain, and then image enhancement or restoration dehazing processing may be applied to the denoised image.
[0157] For example, the de-diffraction processing of the fifth image using the third image processing model may include:
[0158] Extract the diffraction region from the third image and perform diffraction reconstruction on the diffraction region.
[0159] In other embodiments, the image processing model can also achieve conversion between different image domains. These image domains include at least one of the following: RAW, YUW, RGB, etc. For example, by incorporating image signal processing modules such as demosic and gamma correction into the image processing model, the model can achieve conversion between different image domains.
[0160] In other embodiments, the image processing model may also incorporate a network compression module. This module performs lightweight network processing such as knowledge distillation, network pruning, and parameter quantization on the preset training sample set of the image processing model. This reduces the number of parameters and computational load of the image processing model, improves the network computation and processing speed, and enhances the efficiency of image processing, thereby improving the user experience of the mobile terminal.
[0161] Furthermore, this disclosure also provides a specific implementation to further understand the image processing method provided by the embodiments of this disclosure.
[0162] Due to the rapid development of mobile terminals with integrated camera functions, such as smartphones, and the increasing demands from users for image quality and larger screens, under-display cameras, or the aforementioned under-display camera modules, have emerged as a technological advancement. However, existing under-display cameras suffer from several problems: in well-lit environments, they produce light spot diffraction; the images are noisy; they appear hazy and lack clarity; and the color saturation differs significantly from that of photos taken with a normal camera module not covered by the display. These issues negatively impact image quality and significantly reduce user experience.
[0163] Therefore, improving the image quality of images captured by under-display cameras or under-display camera modules has become an urgent technical problem to be solved.
[0164] This disclosure provides an image processing method that introduces an image processing model to process a first image captured by an under-display camera module to obtain a processed second image, thereby improving the poor image quality of images captured by the under-display camera module.
[0165] For example, please refer to Figure 4 ,like Figure 4As shown, the image processing model training process is provided. First, a preset image sample set is created and input into a preset model, so that the trained preset model, i.e., the image processing model, can automatically output a target image based on images captured by an under-display camera. The sample images in the preset image sample set include images captured by a second camera module and a third camera module respectively, targeting the same object. The second camera module is covered by the display screen, while the third camera module is not. The target image can also be understood as the test result output by the trained preset model using the first test sample set as described in the above embodiment.
[0166] Thus, since the preset model is trained using a preset image sample set, the trained preset model, i.e., the image processing model, can process the images captured by the under-display camera into images that are at least close to those captured by a normal camera, thereby improving the shooting effect of the under-display camera and enhancing the image quality of the images produced by the under-display camera.
[0167] To further improve image quality, for example, please refer to [link / reference]. Figure 5 ,like Figure 5 As shown, the method includes:
[0168] The input image to be processed is first preprocessed to obtain the first image; the image to be processed here is the image captured by the under-display camera of the mobile phone, and the preprocessing here can be understood as the normalization or data augmentation processing described in the above embodiments.
[0169] The first image is input into the pre-trained model in the image processing model to obtain the third image;
[0170] The third image is then input into the retrained model of the image processing model to obtain the second image;
[0171] The second image is then subjected to traditional image processing techniques, such as inverse normalization, color enhancement, background blurring, etc.
[0172] Finally, output the processed image.
[0173] In this embodiment, the images captured by the under-display camera of the mobile phone can undergo at least two deep learning model image processing steps, as well as other traditional image processing methods, to obtain a better final image quality. This further improves the imaging quality of the under-display camera and enhances the mobile phone's shooting performance.
[0174] For example, please refer to Figure 6 ,like Figure 6As shown, during the pre-trained model processing, the input image can be processed for dehazing, denoising, de-diffraction, and diffraction correction; during the retrained model processing, the image output by the pre-trained model can be further processed for denoising, de-diffraction, and color correction, and finally the output image is obtained.
[0175] In this embodiment, the image processing model processes the images captured by the under-display camera to produce images with color saturation, clarity, low noise, and no image spot diffraction that are close to those of images from a normal camera, thereby significantly improving the under-display camera's photography experience.
[0176] Figure 7 This is a functional block diagram illustrating an image processing method according to an exemplary embodiment. (Refer to...) Figure 7 The device includes:
[0177] Model 71 is used to acquire a first image to be processed, wherein the first image is captured by a first camera module, and the first camera module is installed below the display screen of the mobile terminal.
[0178] The first processing module 72 is used to process the first image using an image processing model to obtain a second image. The image processing model is trained using a preset image sample set. The sample images in the preset image sample set include images captured by the second camera module and the third camera module for the same subject, respectively. The second camera module is covered by the display screen, and the third camera module is not covered by the display screen.
[0179] In some exemplary embodiments, the image processing model includes: a pre-trained model and a retrained model, wherein the model parameters of the pre-trained model and the retrained model are different;
[0180] The first processing module 72 is further configured to:
[0181] The first image is processed using the pre-trained model to obtain the third image;
[0182] The retrained model is used to process the third image to obtain the second image.
[0183] In some exemplary embodiments, the preset image sample set includes: a first image sample set and a second image sample set, wherein the sample images included in the first image sample set and the second image sample set are different;
[0184] The device includes:
[0185] The second processing model is used to train the pre-trained model using the first image sample set and to train the re-trained model using the second image sample set.
[0186] In some exemplary embodiments, the apparatus includes:
[0187] The third processing model is used to process the first sample image captured by the second camera module to obtain the second sample image, and the third sample image captured by the third camera module for the same subject as the first sample image, to obtain the second image sample set.
[0188] In some exemplary embodiments, the preset image sample set includes at least: a first training sample set, a first verification sample set, and a first test sample set;
[0189] The device includes:
[0190] The fourth processing module is used to train a preset model using the first training sample set and the first verification sample set to obtain a first model, wherein the first training sample set is used to train the preset model to obtain a training result; and the first verification sample set is used to verify the training result to indicate whether the training of the preset model has been successful.
[0191] The fifth processing module is used to test the first model using the first test sample set to obtain the first test result;
[0192] The sixth processing module is used to use the first model as the image processing model in response to the first test result indicating that the image quality of the image processed by the first model meets the image quality conditions.
[0193] In some exemplary embodiments, the first training sample set, the first verification sample set, and the first test sample set include different sample images.
[0194] In some exemplary embodiments, the device includes at least one of the following:
[0195] The seventh processing module is configured to, in response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, train the preset model using the Nth training sample set and the Nth validation sample set to obtain the Nth model, test the Nth model using the Nth test sample set to obtain the Nth test result, until the Nth test result indicates that the image quality of the image processed by the Nth model meets the image quality condition, and use the Nth model as the image processing model, wherein N is a positive integer greater than or equal to 2;
[0196] The eighth processing model is used to adjust the model parameters of the first model in response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, to obtain the Mth model, to test the Mth model using the first test sample set, to obtain the Mth test result, until the Mth test result indicates that the image quality of the image processed by the Mth model meets the image quality condition, and to use the Mth model as the image processing model, wherein M is a positive integer greater than or equal to 2.
[0197] In some exemplary embodiments, the first processing module 72 is further configured to perform at least one of the following:
[0198] The image processing model is used to perform dehazing on the first image;
[0199] The image processing model is used to denoise the first image;
[0200] The first image is de-diffraction processed using the image processing model described above;
[0201] The image processing model is used to perform color correction on the first image.
[0202] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0203] Figure 8 This is a block diagram illustrating a mobile terminal 800 according to an exemplary embodiment. For example, the mobile terminal 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.
[0204] Reference Figure 8 The mobile terminal 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.
[0205] Processing component 802 typically controls the overall operation of mobile terminal 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.
[0206] Memory 804 is configured to store various types of data to support operation on mobile terminal 800. Examples of this data include instructions for any application or method operating on mobile terminal 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0207] The power component 806 provides power to various components of the mobile terminal 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the mobile terminal 800.
[0208] Multimedia component 808 includes a screen that provides an output interface between the mobile terminal 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the mobile terminal 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0209] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when mobile terminal 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
[0210] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0211] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of mobile terminal 800. For example, sensor assembly 814 can detect the on / off state of mobile terminal 800, the relative positioning of components such as the display and keypad of mobile terminal 800, changes in position of mobile terminal 800 or a component of mobile terminal 800, the presence or absence of user contact with terminal 800, orientation or acceleration / deceleration of mobile terminal 800, and temperature changes of mobile terminal 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.
[0212] Communication component 816 is configured to facilitate wired or wireless communication between mobile terminal 800 and other devices. Terminal 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0213] In an exemplary embodiment, the mobile terminal 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.
[0214] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of a mobile terminal 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0215] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of a terminal, enables the terminal to perform the image processing methods described in the above embodiments.
[0216] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.
[0217] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. An image processing method, characterized in that, Applied to mobile terminals, including: Acquire a first image to be processed, wherein the first image is captured by a first camera module, and the first camera module is installed below the display screen of the mobile terminal; The first image is processed using an image processing model to obtain a second image. The image processing model is trained using a preset image sample set. The sample images in the preset image sample set include images captured by the second camera module and the third camera module for the same subject, respectively. The second camera module is covered by the display screen, while the third camera module is not covered by the display screen. The image processing model includes a pre-trained model and a re-trained model, wherein the model parameters of the pre-trained model and the re-trained model are different; the preset image sample set includes a first image sample set and a second image sample set; the pre-trained model is trained based on the first image sample set, and the re-trained model is trained based on the second image sample set; the first image sample set and the second image sample set include different sample images; at least one of the following is different: the shooting object, the shooting time, the shooting angle, and the shooting parameters of the different sample images; The step of processing the first image using an image processing model to obtain the second image includes: The first image is processed using the pre-trained model to obtain the third image; The retrained model is used to process the third image to obtain the second image.
2. The method according to claim 1, characterized in that, The method includes: The second image sample set is obtained by processing the first sample image captured by the second camera module using the pre-trained model to obtain the second sample image, and by collecting the third sample image from the third camera module for the same subject as the first sample image.
3. The method according to claim 1, characterized in that, The preset image sample set includes at least: a first training sample set, a first verification sample set, and a first test sample set; The method includes: A first model is obtained by training a preset model using the first training sample set and the first verification sample set. The first training sample set is used to train the preset model to obtain training results. The first verification sample set is used to verify the training results to indicate whether to end the training of the preset model. The first model is tested using the first test sample set to obtain the first test result; In response to the first test result indicating that the image quality of the image processed by the first model meets the image quality conditions, the first model is used as the image processing model.
4. The method according to claim 3, characterized in that, The first training sample set, the first verification sample set, and the first test sample set include different sample images.
5. The method according to claim 3 or 4, characterized in that, The method includes at least one of the following: In response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, the preset model is trained using the Nth training sample set and the Nth validation sample set to obtain the Nth model, and the Nth model is tested using the Nth test sample set to obtain the Nth test result, until the Nth test result indicates that the image quality of the image processed by the Nth model meets the image quality condition, and the Nth model is used as the image processing model, wherein N is a positive integer greater than or equal to 2; In response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, the model parameters of the first model are adjusted to obtain the Mth model. The Mth model is then tested using the first test sample set to obtain the Mth test result. This process continues until the Mth test result indicates that the image quality of the image processed by the Mth model meets the image quality condition. The Mth model is then used as the image processing model, where M is a positive integer greater than or equal to 2.
6. The method according to claim 1, characterized in that, The processing of the first image using the image processing model includes at least one of the following: The image processing model is used to perform dehazing on the first image; The image processing model is used to denoise the first image; The first image is de-diffraction processed using the image processing model described above; The image processing model is used to perform color correction on the first image.
7. An image processing apparatus, characterized in that, Applied to mobile terminals, including: An acquisition module is used to acquire a first image to be processed, wherein the first image is captured by a first camera module, and the first camera module is installed below the display screen of the mobile terminal; The first processing module is used to process the first image using an image processing module to obtain a second image. The image processing model is trained using a preset image sample set. The sample images in the preset image sample set include images captured by the second camera module and the third camera module for the same subject, respectively. The second camera module is covered by the display screen, and the third camera module is not covered by the display screen. The image processing model includes a pre-trained model and a re-trained model, wherein the model parameters of the pre-trained model and the re-trained model are different; the preset image sample set includes a first image sample set and a second image sample set; the pre-trained model is trained based on the first image sample set, and the re-trained model is trained based on the second image sample set; the first image sample set and the second image sample set include different sample images; at least one of the following is different: the shooting object, the shooting time, the shooting angle, and the shooting parameters of the different sample images; The first processing module is used for: The first image is processed using the pre-trained model to obtain the third image; The retrained model is used to process the third image to obtain the second image.
8. The apparatus according to claim 7, characterized in that, The image processing model includes a pre-trained model and a retrained model, wherein the model parameters of the pre-trained model and the retrained model are different; The first processing module is used for: The first image is processed using the pre-trained model to obtain the third image; The retrained model is used to process the third image to obtain the second image.
9. The apparatus according to claim 8, characterized in that, The preset image sample set includes: a first image sample set and a second image sample set, wherein the sample images included in the first image sample set and the second image sample set are different; The device includes: The second processing module is used to train the pre-trained model using the first image sample set and to train the re-trained model using the second image sample set.
10. The apparatus according to claim 9, characterized in that, The device includes: The third processing model is used to process the first sample image captured by the second camera module to obtain the second sample image, and the third sample image captured by the third camera module for the same subject as the first sample image, to obtain the second image sample set.
11. The apparatus according to claim 7, characterized in that, The preset image sample set includes at least: a first training sample set, a first verification sample set, and a first test sample set; The device includes: The fourth processing model is used to train a preset model using the first training sample set and the first verification sample set to obtain a first model, wherein the first training sample set is used to train the preset model to obtain a training result; and the first verification sample set is used to verify the training result to indicate whether to end the training of the preset model. The fifth processing module is used to test the first model using the first test sample set to obtain the first test result; The sixth processing module is used to use the first model as the image processing model in response to the first test result indicating that the image quality of the image processed by the first model meets the image quality conditions.
12. The apparatus according to claim 11, characterized in that, The first training sample set, the first verification sample set, and the first test sample set include different sample images.
13. The apparatus according to claim 11 or 12, characterized in that, The device includes at least one of the following: The seventh processing module is configured to, in response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, train the preset model using the Nth training sample set and the Nth validation sample set to obtain the Nth model, test the Nth model using the Nth test sample set to obtain the Nth test result, until the Nth test result indicates that the image quality of the image processed by the Nth model meets the image quality condition, and use the Nth model as the image processing model, wherein N is a positive integer greater than or equal to 2; The eighth processing model is used to adjust the model parameters of the first model in response to the first test result indicating that the image quality of the image processed by the first model does not meet the image quality condition, to obtain the Mth model, to test the Mth model using the first test sample set, to obtain the Mth test result, until the Mth test result indicates that the image quality of the image processed by the Mth model meets the image quality condition, and to use the Mth model as the image processing model, wherein M is a positive integer greater than or equal to 2.
14. The apparatus according to claim 7, characterized in that, The first processing module is used for at least one of the following: The image processing model is used to perform dehazing on the first image; The image processing model is used to denoise the first image; The first image is de-diffraction processed using the image processing model described above; The image processing model is used to perform color correction on the first image.
15. A mobile terminal, characterized in that, include: A processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, implements the method steps of any one of claims 1 to 6.
16. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method steps of any one of claims 1 to 6.