Image processing method and related equipment

Abstract

The embodiment of the present application provides an image processing method and related equipment, which relate to the field of terminal technology. The method includes: receiving an operation for taking a photo; in response to the operation for taking a photo, obtaining a first image; inputting the first image into an image processing model to obtain a second image; wherein the resolution of the second image is greater than the resolution of the first image, and the image processing model is used to output images of different resolutions according to the quality of the input image, and the input image is an image input to the image processing model, and the image resolution output by the image processing model is positively correlated with the quality of the input image. In this way, when the resolution of the first image is low, the second image will not have a scene with texture distortion due to the high resolution, thereby improving the quality of the second image and improving the user's photo-taking experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and related equipment. Background Art

[0002] The electronic device may support a photo taking function. A user may use the electronic device to take photos. For example, upon receiving a trigger operation for taking photos, the electronic device may obtain a photo of the object being photographed.

[0003] However, in a possible implementation, in a photo-taking scenario, the quality of the photo taken by the electronic device may be poor, for example, the photo taken is blurry, the texture of the photographed object is distorted, and so on. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and related devices, which are applied to the field of terminal technology. In a photo-taking scene, the texture distortion in the photo can be reduced, the photo quality can be improved, and the user's photo-taking experience can be improved.

[0005] In the first aspect, the embodiment of the present application proposes an image processing method, which is applied to an electronic device, and the method includes: receiving an operation for taking a photo; in response to the operation for taking a photo, obtaining a first image; inputting the first image into an image processing model to obtain a second image; wherein the resolution of the second image is greater than the resolution of the first image, and the image processing model is used to output images of different resolutions according to the quality of the input image, and the input image is an image input to the image processing model, and the image resolution output by the image processing model is positively correlated with the quality of the input image. In this way, when the resolution of the first image is low, the second image will not have a scene with texture distortion due to the high resolution, thereby improving the quality of the second image and improving the user's photo-taking experience.

[0006] In a possible implementation, the image processing model includes a first model and a second model; the first model is used to output a predicted quality score of an input image; the second model is used to output images of different resolutions according to the predicted quality score output by the first model and the input image. In this way, the image processing model can implement the function of scoring the input image based on the first model, and can also implement the function of adaptively adjusting the output image resolution according to the quality score based on the second model, so that the electronic device can use the image processing model to output high-quality images without texture distortion.

[0007] In a possible implementation, the image processing model is a model trained based on the first sample set, and any first sample in the first sample set includes a first-category image, a second-category image, and a label quality score of the first-category image; the resolution of the first-category image is smaller than the resolution of the second-category image, and the display content in the first-category image is the same as the display content in the second-category image. In this way, the image processing model to be trained can be trained with the first sample set to obtain the image processing model, so that the electronic device can use the image processing model to output high-quality images without texture distortion.

[0008] In a possible implementation, the image processing model is trained based on the following process: obtaining a first sample set; for any first sample in the first sample set, inputting the first class image of the first sample into the first model to obtain the predicted quality score of the first class image; inputting the predicted quality score of the first class image and the first class image into the second model to obtain the predicted image; completing the training when the value calculated by the target loss function converges; wherein the target loss function is respectively related to the loss function of the first model and the loss function of the second model; the loss function of the first model is related to the predicted quality score of the first class image and the labeling quality score of the first class image; the loss function of the second model is related to the predicted image and the second class image. In this way, when training the image quality assessment model, the super-resolution reconstruction model that needs to be used with the image quality assessment model in the future can be combined, and the output of the image quality assessment model can be used as the feedback factor of the super-resolution reconstruction model, so that the output of the trained image quality assessment model can better meet the needs of the subsequent super-resolution reconstruction model that uses the output. Therefore, the image quality assessment model and the super-resolution reconstruction model are jointly trained to obtain a better processing effect.

[0009] In a possible implementation, when calculating the target loss function, the weight of the first model is negatively correlated with the number of iterations, and the weight of the second model is positively correlated with the number of iterations. In this way, during the training process of the adaptive super-resolution network model, the effect of training the adaptive super-resolution network model based on the target loss function can be improved by adjusting the weight of the first model and the weight of the second model.

[0010] In one possible implementation, the objective loss function satisfies the following formula:

[0011] loss total =a×loss 1 +b×loss 2

[0012] Among them, the loss function of the first model is loss 1 , the loss function of the second model is loss 2, the weight of the first model is a, and the weight of the second model is b. In this way, the image quality evaluation model and the super-resolution reconstruction model can be jointly trained through the target loss function to obtain a better processing effect.

[0013] In a possible implementation, after the first image is input into the image processing model, the method further includes: obtaining a predicted quality score of the first image; wherein the predicted quality score of the first image is negatively correlated with the image quality of the first image, and the predicted quality score of the first image is negatively correlated with the resolution of the second image; or, the predicted quality score of the first image is positively correlated with the image quality of the first image, and the predicted quality score of the first image is positively correlated with the resolution of the second image. In this way, the electronic device can obtain a quality score after obtaining a second image with a higher resolution; the electronic device can post-process the second image based on the quality score to improve the quality of the image; and / or, the electronic device can also use the quality score to perform other functions, such as adding the quality score to the exchangeable image file format EXIF ​​information or other paths of the second image so that the user can view the quality of the photo.

[0014] In a possible implementation, the predicted quality score of the first image is negatively correlated with the image quality of the first image, and the predicted quality score of the first image is negatively correlated with the resolution of the second image; after obtaining the predicted quality score of the first image, it also includes: when the predicted quality score of the first image is greater than or equal to the preset score, the first image and the second image are fused to obtain a third image; the resolution of the third image is less than the resolution of the second image, and the resolution of the third image is greater than the resolution of the first image. In this way, when the quality of the first image is poor, the electronic device can further process the second image after obtaining the second image to reduce the resolution of the second image, weaken the algorithm traces in the image, and make the image more natural without distorted texture.

[0015] In a possible implementation, the predicted quality score of the first image is negatively correlated with the image quality of the first image, and the predicted quality score of the first image is negatively correlated with the resolution of the second image; after obtaining the predicted quality score of the first image, it also includes: when the predicted quality score of the first image is less than the preset score, edge enhancement processing is performed on the second image to obtain a fourth image; the clarity of the fourth image is higher than the clarity of the second image. In this way, when the quality of the first image is good, the electronic device can further enhance the sharpness and resolution of the second image through edge enhancement after obtaining the second image, thereby further improving the clarity of the image and improving the image quality.

[0016] In a possible implementation, before inputting the first image into the image processing model, it also includes: obtaining the magnification when taking the photo; inputting the first image into the image processing model, including: when the magnification is greater than the preset magnification, inputting the first image into the image processing model. In this way, the electronic device can use the adaptive super-resolution network model in the photo scene where the image resolution needs to be improved; when the output image quality is high, the adaptive super-resolution network model can be omitted to improve the photo efficiency and reduce the computing pressure caused by the super-resolution processing.

[0017] In a possible implementation manner, the first image, the second image, the third image, and the fourth image are all images in RBG format.

[0018] In a second aspect, an embodiment of the present application provides an electronic device, which may also be referred to as a terminal device, a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with a wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.

[0019] The electronic device comprises: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the method as in the first aspect.

[0020] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method of the first aspect is implemented.

[0021] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the method of the first aspect.

[0022] In the fifth aspect, an embodiment of the present application provides a chip system, which includes at least one processor and a communication interface, the communication interface and at least one processor are interconnected through lines, and the at least one processor is used to run computer programs or instructions to execute the method of the first aspect.

[0023] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application;

[0025] Figure 2 A software structure block diagram of the electronic device 100 provided in an embodiment of the present application;

[0026] Figure 3 A flowchart of the use process of the adaptive super-resolution network model provided in the embodiment of the present application;

[0027] Figure 4 A schematic diagram of internal interaction of an image processing method provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of a process flow of a training process of an adaptive super-resolution network model provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the process of jointly training the image quality assessment model and the super-resolution reconstruction model provided in the embodiment of the present application;

[0030] Figure 7 A schematic diagram of the training process of the image quality assessment model provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of the training process of the super-resolution reconstruction model provided in an embodiment of the present application;

[0032] Fig. 9 A flowchart of an image processing method provided in an embodiment of the present application;

[0033] Fig.10 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0035] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0036] It should be noted that the "at..." in the embodiment of the present application can be the instant when a certain situation occurs, or a period of time after a certain situation occurs, and the embodiment of the present application does not specifically limit this. In addition, the display interface provided in the embodiment of the present application is only an example, and the display interface can also include more or less content.

[0037] In a possible implementation, in a photo-taking scenario, the quality of the photo taken by the electronic device may be poor, for example, the photo taken is blurry, the texture of the photographed object is distorted, and so on.

[0038] This is because, when a trigger operation for taking a photo is received, the electronic device can obtain an image to be processed, and the electronic device processes the image to be processed by denoising, super-resolution reconstruction algorithms, etc. In a possible implementation, the resolution of the image processed by the super-resolution reconstruction algorithm is relatively fixed.

[0039] For example, when the shooting environment is poor (for example, cloudy, rainy, dim, etc.), the image quality of the image to be processed obtained by the electronic device is low, and the texture details in the image to be processed are relatively blurred; the electronic device performs super-resolution reconstruction on the image to be processed with poor image quality, and the texture details in the image to be processed are not sufficient to support a very high resolution; therefore, the processed image will show traces of algorithm processing, and the texture in the image will be distorted.

[0040] In view of this, an embodiment of the present application provides an image processing method. In a photo-taking scenario, an electronic device can obtain an image to be processed. For an image to be processed with poor image quality, the resolution of the processed image can be reduced by reducing the difference in resolution between the images before and after processing. In this way, the electronic device can improve the resolution of the image to be processed, but will not cause a scene with texture distortion in the processed image due to excessively improving the resolution of the image to be processed. This will improve image quality and enhance user experience.

[0041] In order to better understand the embodiments of the present application, the structure of the electronic device in the embodiments of the present application is introduced below:

[0042] Figure 1 The structure diagram of the electronic device 100 is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, and an embedded secure element (eSE) 196. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0043] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0044] The electronic device 100 implements the display function through the GPU, the display screen 194, and the application processor. The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In the embodiment of the present application, the electronic device 100 can display a processed high-quality image through the display screen 194.

[0045] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0046] ISP is used to process data fed back by camera 193. For example, when taking a photo, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, ISP can be set in camera 193. In the embodiment of the present application, electronic device 100 can obtain shooting parameters based on ISP, such as magnification when taking pictures.

[0047] The camera 193 is used to capture static images or videos. In the embodiment of the present application, the camera can collect images when the electronic device 100 receives an operation for taking a photo.

[0048] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0049] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0050] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system may include: application layer (applications), application framework layer (application framework), hardware abstract layer (hardware abstract layer, HAL) and kernel layer (kernel), where the kernel layer may become the driver layer.

[0051] The application layer may include a series of application packages.

[0052] like Figure 2As shown, the application package may include applications such as a camera and a gallery. The application package may also include an application for image processing, and the image processing application includes algorithms or models for image processing. It is understandable that the image recognition application may exist independently or may be part of any application in the application layer, and the embodiments of the present application are not specifically limited thereto.

[0053] In the embodiment of the present application, the application for image processing may be an image processing module. The image processing module may include an adaptive super-resolution network model and an image enhancement module. The adaptive super-resolution network model may be used to adaptively adjust the resolution of the output image according to the image quality of the input image; the image enhancement module may be used to post-process the image output by the adaptive super-resolution network model to further improve the clarity of the image.

[0054] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0055] like Figure 2 As shown, the application framework layer may include a camera access interface, etc.

[0056] The camera access interface allows applications to manage the camera and access the camera device, such as managing the camera to capture images.

[0057] The hardware abstraction layer may include multiple library modules, such as a camera library module. The Android system may load the corresponding library module for the device hardware, thereby enabling the application framework layer to access the device hardware.

[0058] The kernel layer is a layer between hardware and software. The kernel layer is used to drive the hardware to make the hardware work. The kernel layer may include a camera driver and a display driver, etc. The camera driver can be used to instruct the camera to capture images, and the display driver can be used to instruct the display screen to display images. This embodiment of the application does not limit this.

[0059] Combine the following Figure 3 The image processing method in the embodiment of the present application is introduced. Figure 3 As shown:

[0060] S301: When an electronic device receives an operation for taking a photo, the electronic device uses a camera to collect an original image.

[0061] The trigger operation for taking a photo may be, for example: in a photo taking scenario, a camera application is running in the foreground of the electronic device, and the electronic device receives a click operation on a shutter button in the camera application interface. In response to the trigger operation for taking a photo, the electronic device obtains the original image captured by the camera. The image type of the original image may be a raw image.

[0062] In the embodiment of the present application, the number of original images may be one frame or multiple frames. The multiple frames of original images may be multiple frames of images obtained by the electronic device when the electronic device receives a single trigger operation for taking a photo; the multiple frames of original images may also include images in the preview stream before and after partial exposure; the electronic device may fuse the multiple frames of original images, which is not limited in the embodiment of the present application.

[0063] S302 : The electronic device processes the original image to obtain image 1 .

[0064] Among them, image 1 can be an image in RGB format, and the electronic device can fuse multiple original images into multiple frames, use the second to Nth original images to register the first original image, obtain an original image, and convert the original image in RAW format into image 1 in RGB format. Alternatively, the electronic device can use a chip or algorithm to process the first original image after obtaining the first original image to obtain a single-frame image in RGB format. The embodiment of the present application does not limit the method of obtaining image 1.

[0065] S303 , the electronic device inputs image 1 into an adaptive super-resolution network model to obtain a quality score of image 1 and image 2.

[0066] The resolution of image 2 is greater than that of image 1. The quality score may be a numerical value, for example, the numerical value of the quality score may be between 0 and N, where N is a positive integer. The quality score is used to evaluate the quality of an image.

[0067] An adaptive super-resolution network model is set up in the electronic device. The adaptive super-resolution network model has the ability to improve image resolution and the ability to score image quality.

[0068] The adaptive super-resolution network model may include a super-resolution reconstruction model and an image quality assessment model, wherein the image quality assessment model may be used to score the image quality, and the super-resolution reconstruction model may be used to improve the image resolution capability according to the quality score. The adaptive super-resolution network model may be a model obtained by jointly training the image quality assessment model and the super-resolution reconstruction model, or the adaptive super-resolution network model may be a model obtained by non-jointly training the image quality assessment model and the super-resolution reconstruction model. The embodiments of the present application are not limited to this.

[0069] In the embodiment of the present application, after image 1 is input into the adaptive super-resolution network model, the adaptive super-resolution network model can use the image quality assessment model to score image 1 to obtain a quality score of image 1. The super-resolution reconstruction model processes image 1 according to the quality score of image 1 to obtain image 2.

[0070] In some embodiments, the image quality of image 1 is positively correlated with the resolution of image 2 .

[0071] Specifically, in a possible implementation, the quality score of image 1 is negatively correlated with the image quality of image 1; the resolution of image 2 is negatively correlated with the quality score of image 1. For example, the value of the quality score can be 0, 1, 2, 3, etc.; the lower the score, the higher the image quality of image 1, for example, 0 represents image 1 with high image quality; the higher the score, the lower the image quality of image 1, for example, 3 represents image 1 with low image quality. When image 1 has a low score, the resolution of image 2 is higher than the resolution of image 2 when image 1 has a high score.

[0072] In another possible implementation, the quality score of image 1 is positively correlated with the image quality of image 1; the resolution of image 2 is positively correlated with the quality score of image 1. For example, the value of the quality score can be 0, 1, 2, 3, etc.; the higher the score, the higher the image quality of image 1, for example, 3 represents image 1 with high image quality; the lower the score, the lower the image quality of image 1, for example, 0 represents image 1 with low image quality. When image 1 has a low score, the resolution of image 2 is lower than the resolution of image 2 when image 1 has a high score. The embodiment of the present application does not limit the scoring rules for image quality.

[0073] The electronic device can output image 2 of corresponding resolution according to the image quality of image 1. For example, if the image quality of image 1 is poor, the resolution of image 2 is low; if the image quality of image 1 is good, the resolution of image 2 is high. In this way, when the image quality of image 1 is poor, the electronic device will not output image 2 with too high resolution, thereby reducing the scene where traces of algorithm processing such as texture distortion appear in image 2. At the same time, when the image quality of image 1 is good, the electronic device can output image 2 with higher resolution, thereby obtaining a high-quality image.

[0074] Taking the case where the quality score of image 1 is negatively correlated with the image quality of image 1 and the resolution of image 2 is negatively correlated with the quality score of image 1 as an example, after step S303, the electronic device may further optimize image 2, as shown in step S304 or S305:

[0075] Optionally, S304 , when the quality score of image 1 is less than a preset score, the electronic device performs image edge enhancement on image 2 to obtain image 3 .

[0076] The definition of image 3 is higher than that of image 2. The preset score is, for example, 2. When the quality score of image 1 is 0 or 1, the electronic device executes step S304.

[0077] The electronic device may select a corresponding post-processing process according to the quality score of the image 1. In one possible implementation, when the quality score of the image 1 is less than the preset score, it means that the quality of the image 1 is high, and the quality of the obtained image 2 is also high. The electronic device further improves the clarity of the image 2 to obtain a higher quality image 3; at the same time, there will be no texture distortion in the image 3.

[0078] In step S303, the electronic device may improve the clarity of the image by improving the resolution; in addition, the electronic device may further improve the clarity of the image by improving the sharpness and / or resolution. Specifically, the electronic device may process the image 2 by an image edge enhancement algorithm to improve the contrast of the image edge in the image 2. The image edge enhancement algorithm may perform edge enhancement on the image 2 based on any one or more operators such as the Laplacian operator, the Sobel operator, and the Canny operator, and the embodiments of the present application are not limited to this.

[0079] It is understandable that, in possible implementations, the super-resolution reconstruction algorithm of the electronic device will not involve the image quality, so the resolution of the image output by the super-resolution reconstruction algorithm is relatively fixed; when the quality of the input image is high, the super-resolution reconstruction algorithm does not significantly improve the resolution of the image, and the sharpness and resolution of the output image are also not obvious. In the embodiment of the present application, after the electronic device improves the resolution of the image to be processed, it can further improve the sharpness and resolution of the image by edge enhancement, thereby further improving the clarity of the image and improving the image quality.

[0080] Or, optionally, S305 , when the quality score of image 1 is greater than or equal to a preset score, the electronic device merges image 2 with image 1 to obtain image 4 .

[0081] The resolution of image 4 is higher than that of image 1, and the resolution of image 4 is lower than that of image 2. The preset score is, for example, 2. When the quality score of image 1 is 2 or 3, the electronic device executes step S305.

[0082] The electronic device may select a corresponding post-processing process according to the quality score of the image 1. In another possible implementation, when the quality score of the image 1 is greater than or equal to the preset score, it indicates that the quality of the image 1 is low; the electronic device may further reduce the scene where texture distortion occurs in the image by reducing the resolution of the image 2.

[0083] In some embodiments, the resolution of image 2 may be reduced by, for example, fusing image 1 with a low resolution with image 2 with a high resolution. The electronic device may fuse a pixel region of image 1 with a pixel region of image 2. The pixel region may include one pixel or multiple pixels.

[0084] Specifically, taking the granularity of the pixel area as a pixel point as an example, the electronic device can obtain the pixel value of the pixel point of image 1 and the pixel value of the pixel point of image 2, and the electronic device can perform weighted fusion on the two pixel values ​​to obtain the pixel value of the pixel area of ​​image 4. The embodiment of the present application does not limit the method of image fusion.

[0085] In this way, for image 1 with lower quality, the electronic device can reduce the resolution of image 2 after obtaining image 2 through the adaptive super-resolution network model, thereby ensuring that the resolution of the photo is not too high to cause texture distortion.

[0086] Optionally, before step S303, the following steps may also be included:

[0087] S306, the electronic device detects the magnification ratio when taking a photo; when the magnification ratio is greater than a preset magnification ratio, the electronic device executes step S303.

[0088] In the photo-taking scenario, the electronic device may display a camera application interface, in which the electronic device may display an icon of the magnification of the image when taking the photo, for example, magnifications of 0.5X, 1X, 2X, and 5X. The electronic device may detect the magnification when receiving an operation for taking a photo; when the magnification is greater than a preset magnification, the electronic device processes the image 1 using an adaptive super-resolution network model.

[0089] The preset magnification ratio may be, for example, 1X. It is understandable that when the electronic device uses a magnification ratio less than or equal to 1X for shooting, the clarity of the captured image is high, and the electronic device may not process the image; and when the electronic device uses a magnification ratio greater than 1X for shooting, the clarity of the captured image is low, and the electronic device needs to perform super-resolution processing on the image to improve the clarity of the image and output a high-quality image.

[0090] In some embodiments, when receiving an operation for taking a photo, the electronic device may obtain shooting parameters and image 1 based on image signal processing (ISP). The shooting parameters may be, for example, a magnification. After obtaining the magnification, the electronic device determines whether the magnification is greater than a preset magnification; if the magnification is greater than the preset magnification, image 1 is input into the adaptive super-resolution network model to obtain image 2. If the magnification is less than or equal to the preset magnification, the electronic device outputs image 1 without performing adaptive super-resolution processing on image 1.

[0091] In this way, electronic devices can use adaptive super-resolution network models in photo-taking scenarios where image resolution needs to be improved; when the image quality is high, the adaptive super-resolution network model can be omitted to improve photo-taking efficiency and reduce the computational pressure brought by super-resolution processing.

[0092] Combine the following Figure 4 The internal interaction process of the software in the image processing method in the embodiment of the present application is explained.

[0093] The electronic device may include an application layer and a kernel layer, wherein the application layer includes a camera application and an image processing module, wherein the image processing module is provided with an adaptive super-resolution network model and an image enhancement module; and the kernel layer includes a camera driver. Figure 4 As shown:

[0094] S401: A camera application of an electronic device receives an operation for taking a photo.

[0095] S402: In response to an operation for taking a photo, a camera application of the electronic device instructs a camera to capture an original image.

[0096] After receiving an operation for taking a photo, the camera application can instruct the camera to collect the original image through the camera driver, and the camera driver can interact with the hardware to obtain the original image.

[0097] For steps S401-S402, reference may be made to the relevant description in step S301, which will not be repeated here.

[0098] S403 : The electronic device processes the original image to obtain image 1 .

[0099] This step may refer to the relevant description in step S302, wherein the electronic device may process the original image based on the algorithm in the algorithm library of the hardware abstraction layer to obtain image 1; the electronic device may also process the original image based on other modules in the image processing module of the application layer to obtain image 1; the electronic device may also process the original image based on the chip to obtain image 1. The processing of the original image may be a process of converting a RAW image into an RBG image and / or a denoising process, etc.

[0100] It should be noted that the embodiments of the present application are Figure 4 The specific module used to process step S403 is not shown, and the specific module may be any one or more modules such as the above-mentioned algorithm library, other modules in the image processing module and / or hardware chip, and the embodiment of the present application does not limit this.

[0101] S404: When the magnification during photographing is greater than a preset magnification, the electronic device inputs image 1 into an adaptive super-resolution network model to obtain a quality score of image 1 and image 2.

[0102] For this step, reference may be made to the relevant descriptions in step S306 and step S303, which will not be repeated here.

[0103] S405: The electronic device transmits the quality score of image 1 and image 2 to the image enhancement module.

[0104] S406 . When the quality score of image 1 is less than a preset score, the image enhancement module of the electronic device performs edge enhancement on image 2 to obtain image 3 .

[0105] The embodiment of the present application takes the quality score of 0, 1, 2, 3... as an example, wherein the smaller the value of the quality score, the higher the image quality. The quality score of image 1 is less than the preset score, which can be understood as: the image quality is higher than the preset image quality. In the embodiment of the present application, the value of the quality score may also be positively correlated with the image quality. For example, the larger the value of the quality score, the higher the image quality. In this case, the judgment condition in step S406 may also be "when the quality score of image 1 is greater than or equal to the preset score". The embodiment of the present application does not specifically elaborate on the scenario where the value of the quality score is positively correlated with the image quality.

[0106] For this step, please refer to the relevant description in step S304, which will not be repeated here.

[0107] S407 . The image enhancement module of the electronic device transmits the image 3 to the camera application.

[0108] Among them, the camera application can obtain image 3. In a possible implementation, after taking the image, the camera application can display the icon of image 3 in the thumbnail of the camera application interface, wherein the icon of image 3 can be the same as the proportionally reduced image 3.

[0109] In some embodiments, the electronic device may also transfer the image 3 to a gallery application (also called an album application), and the gallery application saves the image 3 .

[0110] Alternatively, after step S405, the method further includes:

[0111] S408 . When the quality score of image 1 is greater than or equal to a preset score, the image enhancement module of the electronic device fuses image 2 and image 1 to obtain image 4 .

[0112] The embodiment of the present application takes the case where the value of the quality score is negatively correlated with the image quality as an example to describe step S408. The embodiment of the present application can also be set to the case where the value of the quality score is positively correlated with the image quality. In this case, the determination condition in step S408 can also be "when the quality score of image 1 is less than the preset score". The embodiment of the present application does not further elaborate on the scenario where the value of the quality score is positively correlated with the image quality.

[0113] For this step, please refer to the relevant description in step S305, which will not be repeated here.

[0114] S409: The image enhancement module of the electronic device transmits the image 4 to the camera application.

[0115] Among them, the camera application can obtain image 4. In a possible implementation, after taking the image, the camera application can display the icon of image 4 in the thumbnail of the camera application interface, wherein the icon of image 4 can be the same as the proportionally reduced image 4.

[0116] In some embodiments, the electronic device may also transfer the image 4 to a gallery application (also called an album application), and the gallery application saves the image 4 .

[0117] Alternatively, after step S403, the method further includes:

[0118] S410: When the magnification ratio during photographing is less than or equal to a preset magnification ratio, the electronic device transmits the image 1 to a camera application.

[0119] When the magnification is less than or equal to the preset magnification, the electronic device may not perform super-resolution processing on the image 1 , and the electronic device outputs the image 1 .

[0120] The above embodiments illustrate the use process of the adaptive super-resolution network model in the embodiments of the present application. The following describes the training process of the adaptive super-resolution network model.

[0121] Figure 5 The training process of the adaptive super-resolution network model in the embodiment of the present application is shown as follows: Figure 5 As shown:

[0122] S501. Obtain sample image pairs, wherein any sample image pair includes a high-resolution (HR) image, a low-resolution (LR) image, and a quality score.

[0123] The first electronic device may be an electronic device with large computing power and memory that can process massive amounts of data. For example, the first electronic device may be a client-side server. In the embodiment of the present application, the first electronic device may also be an electronic device in the above embodiment, and the embodiment of the present application does not limit this.

[0124] The sample image pair includes an HR image, an LR image, and a quality score. The HR image may be obtained by taking a photo based on a second electronic device, and the second electronic device may be a device capable of taking high-resolution images, such as a SLR camera, and the first electronic device obtains the HR image from the second electronic device. The HR image may also be an HR image obtained by the first electronic device from the network side. The embodiment of the present application does not limit the source of the HR image.

[0125] The LR image corresponds to the HR image, for example, the object in the HR image is the same as the object in the LR image, and the resolution of the HR image is higher than that of the LR image. The object may be a person, an animal, a plant, or an inanimate object. In some embodiments, the LR image may be obtained by performing degradation or blurring processing on the HR image, and the degradation or blurring processing may reduce the resolution of the image, thereby converting the HR image into the LR image.

[0126] The quality score can be a quality score of the LR image or a quality score of the HR image. It is understandable that the algorithm used by the first electronic device to perform degradation blurring processing on multiple HR images is the same, and the quality of the LR image obtained from a high-quality HR image is also high; for example, the resolution of HR image A is higher than the resolution of HR image B, and the resolution of LR image A is higher than the resolution of LR image B. Therefore, the quality scores of the LR image and the HR image in a sample image pair are the same, and the quality score can be used to characterize the quality score of the LR image, and can also characterize the quality score of the HR image.

[0127] Specifically, in the embodiment of the present application, the quality score may be in the form of a numerical value, for example, the quality score may be 0, 1, 2, 3...; the value of the quality score may be negatively correlated with the image quality, for example, the smaller the value of the quality score, the higher the image quality. In the training process of the adaptive super-resolution network model shown in the embodiment of the present application, the value of the quality score may be negatively correlated with the image quality; however, the value of the quality score may also be positively correlated with the image quality, and the embodiment of the present application does not specifically limit this.

[0128] S502: Using the sample image pairs and the target loss function, an adaptive super-resolution network model is trained.

[0129] The adaptive super-resolution network model to be trained may include an image quality evaluation model to be trained and a super-resolution reconstruction model to be trained. The image quality evaluation model is used to score the image quality, and the super-resolution reconstruction model is used to process the low-resolution image into a high-resolution image. The target loss function is positively correlated with the loss function 1 and the loss function 2, respectively. The loss function 1 may be the loss function of the image quality evaluation model, and the loss function 2 may be the loss function of the super-resolution reconstruction model.

[0130] The first electronic device may use sample images to jointly train the image quality assessment model to be trained and the super-resolution reconstruction model to be trained to obtain an adaptive super-resolution network model.

[0131] The following takes the joint training of two models as an example to illustrate the training process of the adaptive super-resolution network model in step S502. Figure 6 As shown:

[0132] S601: Input the LR image into the image quality assessment model to be trained to obtain a predicted quality score of the LR image.

[0133] The first electronic device can obtain a sample image pair, wherein the quality scores of the LR image, the HR image, and the LR image belong to the same sample image pair. The structure of the image quality assessment model to be trained can be a classification model based on a convolutional neural network. The image quality assessment model to be trained has the ability to score image quality; after the first electronic device inputs the LR image into the image quality assessment model to be trained, the image quality assessment model to be trained can process the LR image based on multiple convolutional layers and output a predicted quality score corresponding to the LR image.

[0134] For example, step S601 may be as follows: Figure 7 As shown, the image quality assessment model to be trained includes, for example, 3 convolution layers. The image quality assessment model to be trained obtains an LR image; the image quality assessment model to be trained converts the LR image into a vector 1 based on the convolution layer, and operates the vector 1 using the hyperparameters of the neural network; the image quality assessment model to be trained converts the operated vector 1 into a predicted quality score of the LR image.

[0135] The predicted quality score of the LR image can be a map table with the same size as the LR image, and the quality score of the LR image is mapped in the map table. Figure 7 As shown, the quality score of the LR image is, for example, 0, the number of rows and columns of the map table is the same as the resolution of the LR image, for example, the resolution of the LR image is 1280×720, and the size of the map table is also 1280×720; the value corresponding to any pixel in the LR image is 0.

[0136] S602: Obtain loss function 1 based on the predicted quality score of the LR image and the quality score of the LR image.

[0137] The loss function 1 may be a classification loss function, for example, a cross entropy loss function. In some embodiments, the image quality assessment model to be trained may output a predicted probability value of a predicted quality score of the LR image.

[0138] Loss function 1 1 ) can satisfy the following formula:

[0139]

[0140] Among them, x i is the true probability value of the quality score of the LR image, is the predicted probability value of the predicted quality score of the LR image; n can be the number of values ​​of the quality score.

[0141] For example, when n is 4, the numerical values ​​of the quality scores can be 0, 1, 2, and 3, and the predicted probability values ​​of each quality score can be [0.7, 0.1, 0.1, 0.1]; the true probability values ​​of each quality score can be [1, 0, 0, 0], and the image quality assessment model to be trained can obtain the calculation result of the loss function 1 as 0.3.

[0142] S603: Input the LR image and the predicted quality score of the LR image into the super-resolution reconstruction model to be trained to obtain a predicted image.

[0143] Among them, the resolution of the predicted image is higher than that of the LR image. The structure of the super-resolution reconstruction model to be trained can be a model based on a convolutional neural network. The super-resolution reconstruction model to be trained has the ability to increase the image resolution according to the quality score; after the first electronic device inputs the LR image and the predicted quality score of the LR image into the super-resolution reconstruction model to be trained, the super-resolution reconstruction model to be trained can process the LR image based on multiple convolutional layers and output the predicted image corresponding to the LR image.

[0144] For example, step S603 may be as follows: Figure 8 As shown, the super-resolution reconstruction model to be trained may include a convolutional layer, a convolutional residual block (resblock), a down-sampling layer (down sample with conv2d), an up-sampling layer (conv2d transpose) and a connection layer (also called a splicing layer).

[0145] Specifically, 1) the super-resolution reconstruction model to be trained obtains the LR image in RGB format and a map table of quality scores.

[0146] 2) The super-resolution reconstruction model to be trained converts the map table of LR images and quality scores into vector 2 based on the convolutional layer.

[0147] Among them, the convolution layer can be used to convert the image into a vector. For any pixel in the LR image, the super-resolution reconstruction model to be trained can obtain the vector value of the RGB three-way channel based on the pixel value of the pixel, and obtain the vector value of one channel based on the quality score corresponding to the pixel in the map table. Vector 2 can be a combination of the vector values ​​of the above four channels.

[0148] 3) The super-resolution reconstruction model to be trained compresses vector 2 based on the downsampling layer to obtain vector 3.

[0149] The downsampling layer can be used to compress the vector value, reduce the subsequent calculation amount and increase the receptive field. For example, the downsampling layer can compress vector 2 according to the convolution kernel m×m, for example, the size of the convolution kernel is 3×3. The super-resolution reconstruction model to be trained obtains vector 3.

[0150] 4) The super-resolution reconstruction model to be trained operates on vector 3 based on the convolution residual block to obtain vector 4.

[0151] The convolution residual block may include multiple convolution layers, for example, the convolution residual block includes two 3×3 convolution layers. The super-resolution reconstruction model to be trained can use the hyperparameters in the neural network and the convolution residual block to extract the features of vector 3 and participate in the operation, so that the super-resolution reconstruction model to be trained learns the mapping relationship between the quality score and the resolution of the super-resolution reconstructed image.

[0152] 5) The super-resolution reconstruction model to be trained processes vector 4 based on the upsampling layer to obtain vector 5.

[0153] The upsampling layer is used to increase the resolution of the image output from the convolution residual block. For example, the resolution of the image corresponding to vector 4 is smaller than the resolution of the image corresponding to vector 5. In this way, it is convenient to convert vector 5 into a predicted image with a higher resolution later.

[0154] 6) The super-resolution reconstruction model to be trained processes vector 5 based on the convolutional layer to obtain a predicted image.

[0155] Convolutional layers can also be used to convert vectors into images, for example, converting vector 5 into a predicted image.

[0156] Optionally, the connection layer in the super-resolution reconstruction model to be trained can fuse the shallow network and the deep network to improve the expression ability of the extracted features of the super-resolution reconstruction model to be trained.

[0157] S604: Obtain loss function 2 based on the HR image and the predicted image.

[0158] The loss function 2 may be a regression loss function, such as a mean absolute error (MAE; also referred to as L1) loss function and a mean squared error (MSE; also referred to as L2) loss function. In some embodiments, the super-resolution reconstruction model to be trained may output an image after super-resolution reconstruction of the LR image, such as a predicted image. The loss function 2 may be calculated using the pixel values ​​of the HR image and the pixel values ​​of the predicted image.

[0159] Loss function 2 2 ) can satisfy the following formula:

[0160]

[0161] Among them, y i is the pixel value of any pixel in the HR image; is the pixel value of the pixel in the predicted image corresponding to the pixel in the HR image; l can be the number of pixels in the HR.

[0162] In a possible implementation, the first electronic device obtains loss 2 After calculating the results, we can 2 The calculation results are normalized so that loss 2 The calculation result is between 0 and 1.

[0163] It should be noted that the loss function 2 may also be a deformed formula related to the above formula, and the embodiment of the present application does not limit this.

[0164] S605: Obtain a trained adaptive super-resolution network model based on the calculation result of the target loss function.

[0165] Objective loss function total ) can satisfy the following formula:

[0166] loss total =a×loss 1 +b×loss 2

[0167] Wherein, a is the weight of loss function 1, and b is the weight of loss function 2. In some embodiments, the first electronic device may set the values ​​of a and b according to the number of iterations in the training process of the adaptive super-resolution network model; for example, the weight of loss function 1 is negatively correlated with the number of iterations, and the weight of loss function 2 is positively correlated with the number of iterations.

[0168] For example, when the number of iterations is small, the values ​​of a and b can both be set to 0.5. As the number of iterations increases, when the number of iterations reaches the first number threshold, the first electronic device adjusts the values ​​of a and b, setting a to 0.4 and b to 0.6. When the number of iterations reaches the second number threshold, the first electronic device adjusts the values ​​of a and b again, setting a to 0.3 and b to 0.7; and so on.

[0169] When the calculated value of the target loss function converges, the calculated value of the target loss function is less than a fixed value and / or reaches the maximum number of iterations, a trained adaptive super-resolution network model is obtained.

[0170] In this way, when training the image quality assessment model, the super-resolution reconstruction model that needs to be used with the image quality assessment model can be combined, and the output of the image quality assessment model can be used as a feedback factor of the super-resolution reconstruction model. The output of the trained image quality assessment model can better meet the needs of the super-resolution reconstruction model that uses the output. Therefore, the joint training of the image quality assessment model and the super-resolution reconstruction model can achieve better processing results.

[0171] Optionally, in step S501, the first electronic device may score the LR image in the sample image pair to obtain a quality score of the LR image. The scoring process may be as shown in step 7) to step 9):

[0172] 7) The first electronic device obtains a plurality of LR images.

[0173] 8) The first electronic device distinguishes the LR images according to the resolution.

[0174] The first electronic device is provided with a resolution threshold, which may be, for example, 1000×1000. When the resolution of the LR image is greater than the resolution threshold, the first electronic device classifies the LR image into a high-resolution image set; when the resolution of the LR image is less than or equal to the resolution threshold, the first electronic device classifies the LR image into a low-resolution image set.

[0175] Take N=3, and the value of the quality score is between 0 and 3 as an example, wherein the order of image quality that can be represented by each quality score can be: 0, 1, 2, 3, wherein 0 represents the highest image quality, and 3 represents the lowest image quality. Different sets correspond to different multiple quality scores, for example, the quality score in the high-resolution image set can be 0 or 1; the quality score in the low-resolution image set can be 2 or 3.

[0176] It can be understood that resolution can be used to reflect image quality, and the first electronic device can first distinguish between high-resolution images and low-resolution images through a resolution threshold; the resolution threshold can be one or more, and the image sets of different resolutions can also be two or more, and the embodiments of the present application do not limit this.

[0177] It should be noted that resolution can be a dimension to measure image quality, and other parameters such as sharpness, resolution, etc. can also be used to reflect image quality. The first electronic device can also perform further image quality evaluation on images in different sets in step 9).

[0178] 9) The first electronic device performs edge detection on the LR image, and determines a quality score of the image according to the clarity of the edge of the LR image.

[0179] The edge clarity may be negatively correlated with the quality score; the clearer the image edge, the higher the image quality, and correspondingly, the lower the quality score. Edge detection may be performed on parameters such as gradient values, differences, variance values, and contrast between pixel regions in the LR image; this is not limited in the present application.

[0180] Taking the gradient value as an example, when the gradient value of the LR image in the high-resolution image set is greater than the first gradient threshold, the electronic device marks the quality score of the LR image as 0; when the gradient value of the LR image in the high-resolution image set is less than or equal to the first gradient threshold, the electronic device marks the quality score of the LR image as 1. When the gradient value of the LR image in the low-resolution image set is greater than the second gradient threshold, the electronic device marks the quality score of the LR image as 2; when the gradient value of the LR image in the low-resolution image set is less than or equal to the second gradient threshold, the electronic device marks the quality score of the LR image as 3.

[0181] The embodiment of the present application only shows a method for scoring LR images. In one possible implementation, in order to improve the accuracy of the sample image pair, in step 3), the first electronic device may perform multiple edge detections on any LR image to obtain multiple quality scores. For example, the LR image may be scored according to parameters such as the gradient value, variance value and / or contrast of the LR image. In some embodiments, the first electronic device may obtain multiple quality scores for any LR image. For example, the first electronic device performs three detections on the LR image respectively to obtain three quality scores; wherein the three quality scores may be, for example, 0, 0, and 1. The first electronic device may sort the multiple quality scores and use the median in the quality score sequence as the quality score of the LR image, for example, marking the quality score of the LR image as 0.

[0182] Optionally, to further improve the accuracy of the sample image pair, the score of the LR image can also be marked based on the experience of the tester. For example, in step 9), multiple testers score the same LR image to obtain multiple quality scores. The first electronic device sorts the multiple quality scores and takes the median to obtain the LR quality score. The marking principle and process can be referred to steps 7) to 9) in this embodiment, and the embodiment of the present application will not be repeated here.

[0183] Optionally, the first electronic device may also perform non-joint training on the image quality assessment model to be trained and the super-resolution reconstruction model to be trained to obtain an adaptive super-resolution network model.

[0184] Taking the non-joint training of two models as an example, the training process of the adaptive super-resolution network model is briefly introduced, as shown in steps S701-S704:

[0185] S701: Obtain a first training sample set.

[0186] The first training sample set is used to train the image quality assessment model. In the first training sample set, any first training sample includes an LR image and a quality score of the LR image. The process of the first electronic device obtaining the LR image and the quality score of the LR image can refer to the relevant description in step S501, which will not be repeated here.

[0187] S702: Train the image quality assessment model to be trained based on the first training sample set to obtain a trained image quality assessment model.

[0188] During training, the LR image of the first training sample can be input into the image quality assessment model to be trained, and the predicted quality score output by the image quality assessment model to be trained and the quality score of the corresponding LR in the first training sample are subjected to loss calculation, and then the parameters of the image quality assessment model to be trained are updated based on the result of the loss calculation, and such a process is repeated until a predetermined maximum number of iterations is reached, or the loss converges, or the loss value is less than a certain value, and the model training can be considered to be completed, and a trained image quality assessment model is obtained. Among them, the loss calculation can refer to the calculation formula of the loss function 1, which will not be repeated here.

[0189] S703: Obtain a second training sample set.

[0190] The second training sample set is used to train the super-resolution reconstruction model. In the second training sample set, any second training sample includes an LR image and an HR image. The process of the first electronic device acquiring the LR image and the HR image can refer to the relevant description in step S501, which will not be repeated here.

[0191] S704: Input the second training sample set into the image quality assessment model, and use the output of the image quality assessment model to train and obtain a super-resolution reconstruction model.

[0192] In an embodiment of the present application, the second training sample set is first input into the image quality assessment model, and then the predicted quality score obtained by the image quality assessment model is used as the input for training the super-resolution reconstruction model. The trained super-resolution reconstruction model can use the predicted quality score obtained by the image quality assessment model; so that the super-resolution reconstruction model can learn the mapping relationship between the quality score and the resolution of the output image.

[0193] During training, the LR image of the second training sample can be input into the trained image quality assessment model, and the predicted quality score output by the image quality assessment model can be used as the input of the super-resolution reconstruction model. The predicted quality score and the LR image are input into the super-resolution reconstruction model to be trained, and the loss is calculated between the predicted image with a higher resolution output by the super-resolution reconstruction model to be trained and the HR image in the second training sample. Then, based on the result of the loss calculation, the parameters of the super-resolution reconstruction model to be trained are updated, and this process is repeated until the predetermined maximum number of iterations is reached, or the loss converges, or the loss value is less than a certain value. The model training can be considered to be completed, and a trained super-resolution reconstruction model is obtained. Among them, the loss calculation can refer to the calculation formula of loss function 2, which will not be repeated here.

[0194] It can be understood that, compared with the method of obtaining the sample image pair in step S501, in the embodiment of the present application, HR images and LR images can be obtained first, and some of the LR images can be marked and scored. After the super-resolution reconstruction model training process, the trained image quality evaluation model is used to score the LR images, thereby reducing the workload of evaluating the quality of LR images through algorithms or manual experience; at the same time, the LR images in the first training sample set are obtained by blurring the HR images, so the first training sample set can also include LR images and HR images. When training the super-resolution reconstruction model, the first training sample set can be reused, which can reduce the time for a large number of image acquisitions and image blurring.

[0195] It should be noted that the specific numerical values ​​in the embodiments of the present application are intended to exemplify the setting rules of some parameters of the adaptive super-resolution network model during the training process or use process, but the specific numerical values ​​do not impose any limitations on the technical solutions of the embodiments of the present application.

[0196] Based on the above embodiments, the present application provides an image processing method. For example, Fig. 9 A flowchart of an image processing method provided in an embodiment of the present application.

[0197] like Fig. 9As shown, the image processing method may include the following steps:

[0198] S901: Receive an operation for taking a photo.

[0199] S902: In response to an operation for taking a photo, acquire a first image.

[0200] For steps S901-S902, reference may be made to the relevant description in steps S301-S302. The first image may correspond to image 1. When receiving an operation for taking a photo, the electronic device may obtain image 1 with a lower resolution.

[0201] S903. Input the first image into an image processing model to obtain a second image; wherein the resolution of the second image is greater than the resolution of the first image, the image processing model is used to output images of different resolutions according to the quality of the input image, the input image is an image input into the image processing model, and the resolution of the image output by the image processing model is positively correlated with the quality of the input image.

[0202] Step S903 may refer to the relevant description in step S303. The second image may correspond to image 2. The resolution of image 2 is greater than the resolution of image 1. The image processing model may correspond to an adaptive super-resolution network model. The adaptive super-resolution network model has the ability to adaptively output images of different resolutions according to the quality of the input image. When the image quality of the input image is high, the resolution of the output image is high; when the image quality of the input image is low, the resolution of the output image is low.

[0203] In this way, when the resolution of the first image is low, the second image will not have a scene with texture distortion due to the high resolution, thereby improving the quality of the second image and enhancing the user's photo-taking experience.

[0204] Optionally, the image processing model includes a first model and a second model; the first model is used to output a predicted quality score of an input image; and the second model is used to output images of different resolutions based on the predicted quality score output by the first model and the input image.

[0205] The first model may correspond to an image quality assessment model, and the second model may correspond to a super-resolution reconstruction model. The image quality assessment model may score the quality of an input image and output a predicted quality score for the input image; the super-resolution reconstruction model may output images of different resolutions based on the predicted quality score and the input image. In the use scenario of the adaptive super-resolution network model, the predicted quality score may be Figure 3 and Figure 4 Quality ratings in the illustrated examples.

[0206] In this way, the image processing model can implement the function of scoring the input image based on the first model, and can also implement the function of adaptively adjusting the output image resolution according to the quality score based on the second model, so that the electronic device can use the image processing model to output high-quality images without texture distortion.

[0207] Optionally, the image processing model is a model trained based on the first sample set, and any first sample in the first sample set includes a first category of images, a second category of images, and a marking quality score of the first category of images; the resolution of the first category of images is smaller than the resolution of the second category of images, and the display content in the first category of images is the same as the display content in the second category of images.

[0208] The first sample set may correspond to Figure 5 and Figure 6 In the set of sample image pairs in the illustrated embodiment, the first sample may correspond to any sample image pair, the first type of image may correspond to the LR image, the second type of image may correspond to the HR image, and the labeled quality score of the first type of image may correspond to the quality score of the LR image. It can be understood that the labeled quality score can be used to characterize the quality score in the sample image pair, which is the quality score of the LR image labeled by the first electronic device based on an algorithm or manual experience. The test quality score can be used to characterize the quality score of the image quality predicted by the image quality evaluation model or the adaptive super-resolution network model.

[0209] In this way, the image processing model to be trained can be trained using the first sample set to obtain the image processing model, so that the electronic device can use the image processing model to output high-quality images without texture distortion.

[0210] Optionally, the image processing model is trained based on the following process: obtaining a first sample set; for any first sample in the first sample set, inputting the first category image of the first sample into the first model to obtain a predicted quality score of the first category image; inputting the predicted quality score of the first category image and the first category image into the second model to obtain a predicted image; the training is completed when the value calculated using the target loss function converges; wherein the target loss function is respectively related to the loss function of the first model and the loss function of the second model; the loss function of the first model is related to the predicted quality score of the first category image and the labeling quality score of the first category image; the loss function of the second model is related to the predicted image and the second category image.

[0211] Among them, this step can refer to the relevant description in steps S601-S605. The training is completed when the value calculated by the target loss function converges, which can be understood as: when the value calculated by the target loss function converges, the value calculated by the target loss function is less than the fixed value and / or reaches the maximum number of iterations, the trained adaptive super-resolution network model is obtained. The loss function of the first model can correspond to loss function 1 (loss1 ), the loss function of the second model can correspond to loss function 2 (loss 2 ).

[0212] In this way, when training the image quality assessment model, the super-resolution reconstruction model that needs to be used with the image quality assessment model can be combined, and the output of the image quality assessment model can be used as a feedback factor of the super-resolution reconstruction model. The output of the trained image quality assessment model can better meet the needs of the super-resolution reconstruction model that uses the output. Therefore, the joint training of the image quality assessment model and the super-resolution reconstruction model can achieve better processing results.

[0213] Optionally, when calculating the target loss function, the weight of the first model is negatively correlated with the number of iterations, and the weight of the second model is positively correlated with the number of iterations.

[0214] In this way, during the training process of the adaptive super-resolution network model, the effect of training the adaptive super-resolution network model based on the target loss function can be improved by adjusting the weight of the first model and the weight of the second model.

[0215] Optionally, the objective loss function satisfies the following formula:

[0216] loss total =a×loss 1 +b×loss 2

[0217] Among them, the loss function of the first model is loss 1 , the loss function of the second model is loss 2 , the weight of the first model is a, and the weight of the second model is b.

[0218] In this way, the image quality evaluation model and the super-resolution reconstruction model can be jointly trained through the target loss function to obtain better processing results.

[0219] Optionally, after inputting the first image into the image processing model, it also includes: obtaining a predicted quality score of the first image; wherein the predicted quality score of the first image is negatively correlated with the image quality of the first image, and the predicted quality score of the first image is negatively correlated with the resolution of the second image; or, the predicted quality score of the first image is positively correlated with the image quality of the first image, and the predicted quality score of the first image is positively correlated with the resolution of the second image.

[0220] This step may refer to the relevant description in step S303, wherein the smaller the value of the predicted quality score is, the better the quality of the first image is, and the higher the resolution of the second image is; or, the larger the value of the predicted quality score is, the better the quality of the first image is, and the higher the resolution of the second image is. The embodiment of the present application does not limit the setting rules of the predicted quality score.

[0221] In this way, the electronic device can obtain a quality score after obtaining a second image with a higher resolution; the electronic device can post-process the second image based on the quality score to improve the quality of the image; and / or, the electronic device can also use the quality score to perform other functions, for example, adding the quality score to the exchangeable image file format (EXIF) information or other path of the second image, so that the user can view the quality of the photo.

[0222] Optionally, the prediction quality score of the first image is negatively correlated with the image quality of the first image, and the prediction quality score of the first image is negatively correlated with the resolution of the second image; after obtaining the prediction quality score of the first image, it also includes: when the prediction quality score of the first image is greater than or equal to a preset score, fusing the first image and the second image to obtain a third image; the resolution of the third image is smaller than the resolution of the second image, and the resolution of the third image is greater than the resolution of the first image.

[0223] For this step, reference may be made to the relevant description in step S305 . The third image may correspond to image 4 ; the resolution of image 4 is lower than that of image 2 , and higher than that of image 1 .

[0224] In this way, when the quality of the first image is poor, the electronic device can further process the second image after obtaining the second image to reduce the resolution of the second image and weaken the algorithm traces in the image, making the image more natural and without distorted texture.

[0225] Optionally, the predicted quality score of the first image is negatively correlated with the image quality of the first image, and the predicted quality score of the first image is negatively correlated with the resolution of the second image; after obtaining the predicted quality score of the first image, it also includes: when the predicted quality score of the first image is less than a preset score, performing edge enhancement processing on the second image to obtain a fourth image; the clarity of the fourth image is higher than the clarity of the second image.

[0226] For this step, reference may be made to the relevant description in step S304 . The fourth image may correspond to image 3 ; the definition of image 3 is higher than that of image 2 .

[0227] In this way, when the quality of the first image is good, the electronic device can further improve the sharpness and resolution of the second image through edge enhancement after obtaining the second image, thereby further improving the clarity of the image and improving the image quality.

[0228] Optionally, before inputting the first image into the image processing model, it also includes: obtaining the magnification when taking the photo; inputting the first image into the image processing model, including: when the magnification is greater than a preset magnification, inputting the first image into the image processing model.

[0229] For this step, please refer to the relevant description in step S304.

[0230] In this way, electronic devices can use adaptive super-resolution network models in photo-taking scenarios where image resolution needs to be improved; when the output image quality is high, the adaptive super-resolution network model can be omitted to improve photo-taking efficiency and reduce the computational pressure brought by super-resolution processing.

[0231] Optionally, the first image, the second image, the third image and the fourth image are all images in RBG format.

[0232] The device image processing method of the embodiment of the present application has been described above, and the device for executing the above image processing method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above image processing method.

[0233] like Fig.10 As shown, the image processing apparatus 1000 can be used in a communication device, a circuit, a hardware component or a chip, and the image processing apparatus includes: a display unit 1001 and a processing unit 1002. The display unit 1001 is used to support the display step performed by the image processing apparatus 1000; the processing unit 1002 is used to support the image processing apparatus 1000 to perform the information processing step.

[0234] In a possible implementation, the image processing apparatus 1000 may also include a communication unit 1003. Specifically, the communication unit is used to support the image processing apparatus 1000 to perform the steps of sending and receiving data. The communication unit 1003 may be an input or output interface, a pin or a circuit.

[0235] In a possible embodiment, the image processing device may further include: a storage unit 1004. The processing unit 1002 and the storage unit 1004 are connected via a line. The storage unit 1004 may include one or more memories, and the memory may be a device used to store programs or data in one or more devices or circuits. The storage unit 1004 may exist independently and be connected to the processing unit 1002 of the image processing device via a communication line. The storage unit 1004 may also be integrated with the processing unit 1002.

[0236] The storage unit 1004 can store computer-executable instructions of the method in the terminal device so that the processing unit 1002 executes the method in the above embodiment. The storage unit 1004 can be a register, a cache, or a RAM, etc. The storage unit 1004 can be integrated with the processing unit 1002. The storage unit 1004 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and the storage unit 1004 can be independent of the processing unit 1002.

[0237] The image processing method provided in the embodiment of the present application can be applied to electronic devices with communication functions. The electronic device includes an electronic device, and the specific device form of the electronic device can refer to the above related description, which will not be repeated here.

[0238] An embodiment of the present application provides an electronic device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the above method.

[0239] The embodiment of the present application provides a chip system, which includes at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used to run a computer program or instruction to execute the above method. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, and will not be repeated here.

[0240] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0241] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0242] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0243] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0244] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: Used in electronic equipment, including: An operation for taking a photo is received; In response to the operation for taking a picture, acquiring a first image; Inputting the first image into an image processing model to obtain a second image; wherein the resolution of the second image is greater than the resolution of the first image, the image processing model is used to output images of different resolutions according to the quality of the input image, the input image is an image input into the image processing model, and the resolution of the image output by the image processing model is positively correlated with the quality of the input image; The image processing model includes a first model and a second model; the first model is used to output a predicted quality score of the input image; and the second model is used to output images of different resolutions based on the predicted quality score output by the first model and the input image.

2. The method according to claim 1, characterized in that The image processing model is a model trained based on a first sample set, wherein any first sample in the first sample set includes a first category of images, a second category of images, and a marking quality score of the first category of images; the resolution of the first category of images is smaller than the resolution of the second category of images, and the display content in the first category of images is the same as the display content in the second category of images.

3. The method according to claim 2, characterized in that The image processing model is trained based on the following process: obtaining the first sample set; for any first sample in the first sample set, inputting the first category image of the first sample into the first model to obtain a predicted quality score of the first category image; inputting the predicted quality score of the first category image and the first category image into the second model to obtain a predicted image; and completing the training when the value calculated using the target loss function converges; wherein the target loss function is respectively related to the loss function of the first model and the loss function of the second model; the loss function of the first model is related to the predicted quality score of the first category image and the labeling quality score of the first category image; and the loss function of the second model is related to the predicted image and the second category image.

4. The method according to claim 3, characterized in that When calculating the target loss function, the weight of the first model is negatively correlated with the number of iterations, and the weight of the second model is positively correlated with the number of iterations.

5. The method according to claim 4, characterized in that The objective loss function satisfies the following formula: loss total =a×loss1+b×loss2 Among them, the loss function of the first model is loss1, the loss function of the second model is loss2, the weight of the first model is a, and the weight of the second model is b.

6. The method according to claim 1, characterized in that After inputting the first image into the image processing model, the method further includes: Obtaining a predicted quality score of the first image; wherein the predicted quality score of the first image is negatively correlated with the image quality of the first image, and the predicted quality score of the first image is negatively correlated with the resolution of the second image; Alternatively, the predicted quality score of the first image is positively correlated with the image quality of the first image, and the predicted quality score of the first image is positively correlated with the resolution of the second image.

7. The method according to claim 6, characterized in that The predicted quality score of the first image is negatively correlated with the image quality of the first image, and the predicted quality score of the first image is negatively correlated with the resolution of the second image; after obtaining the predicted quality score of the first image, the method further includes: When the predicted quality score of the first image is greater than or equal to a preset score, the first image and the second image are fused to obtain a third image; the resolution of the third image is smaller than the resolution of the second image, and the resolution of the third image is greater than the resolution of the first image.

8. The method according to claim 6, characterized in that The predicted quality score of the first image is negatively correlated with the image quality of the first image, and the predicted quality score of the first image is negatively correlated with the resolution of the second image; after obtaining the predicted quality score of the first image, the method further includes: When the predicted quality score of the first image is less than a preset score, edge enhancement processing is performed on the second image to obtain a fourth image; the clarity of the fourth image is higher than the clarity of the second image.

9. The method according to claim 1, characterized in that: Before inputting the first image into the image processing model, the method further includes: Get the magnification when taking photos; The step of inputting the first image into an image processing model comprises: When the magnification is greater than a preset magnification, the first image is input into an image processing model.

10. The method according to any one of claims 1 to 6, characterized in that: The first image and the second image are both images in RBG format.

11. The method according to claim 7, characterized in that The third image is an image in RGB format.

12. The method according to claim 8, characterized in that The fourth image is an image in RGB format.

13. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. A chip system, characterized in that: The method comprises at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is used to run a computer program or instruction to execute the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The method comprises a computer program, which, when being executed, enables a computer to execute the method according to any one of claims 1 to 12.

Images