Image processing method, electronic device and computer readable storage medium

Abstract

The present application provides an image processing method, an electronic device, and a computer-readable storage medium, which relate to the field of image processing technology. The method is applied to an electronic device, and the method comprises: displaying a first interface, the first interface comprising a first control; in response to the operation of the first control, acquiring a first image to be processed, the clarity of the first image to be processed being less than a first threshold; displaying a priori reconstructed image after enhancing the first image to be processed in a second interface, the clarity of the priori reconstructed image being greater than the clarity of the first image to be processed; wherein the priori reconstructed image is an image obtained by enhancing the first image to be processed based on the priori image, the priori image is an image including the same photographed object as the first image to be processed, and the clarity is greater than the second threshold, and the priori reconstructed image includes the features of the photographed object contained in the priori image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the scene of taking photos, the captured image is often degraded due to low light, movement of the lens or the object being photographed. Degradation here includes noise, blur, JPEG compression and downsampling. In order to capture clear images, current electronic devices such as smart phones and computers are equipped with degrading algorithms, which can restore the degraded images to clear images.

[0003] Current de-degradation algorithms generally reduce or eliminate image degradation effects by restoring missing or damaged information in the image, thereby making the image clearer. However, degraded images usually contain noise or too much missing image information, which makes it impossible to accurately restore the missing or damaged information in the image, especially the details in the image, so there is a problem of poor de-degradation effect. Summary of the invention

[0004] In view of this, the present application provides an image processing method, an electronic device, and a computer-readable storage medium, which can restore a degraded image to a clear image with image details, thereby improving image quality and display effects.

[0005] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, and the method includes: displaying a first interface, the first interface including a first control; in response to an operation on the first control, acquiring a first image to be processed, the clarity of the first image to be processed being less than a first threshold; displaying a priori reconstructed image after enhancing the first image to be processed in a second interface, the clarity of the priori reconstructed image being greater than the clarity of the first image to be processed; wherein the priori reconstructed image is an image obtained by enhancing the first image to be processed based on the prior image, the prior image is an image that includes the same photographed object as the first image to be processed and has a clarity greater than a second threshold, and the priori reconstructed image includes features of the photographed object contained in the prior image.

[0006] The clarity of the first image to be processed being less than the first threshold value may mean that the degraded image is used as an image that needs to be reconstructed a priori. Based on the above process, the electronic device can obtain the first image to be processed whose clarity is less than the first threshold value, and enhance the first image to be processed based on the a priori image to obtain a clearer a priori reconstructed image, thereby improving the image quality and display effect.

[0007] In a possible implementation of the first aspect, the process of enhancing the first image to be processed based on the prior image to obtain the prior reconstructed image includes: aligning the prior image with the first image to be processed to obtain the second image to be processed; and fusing the first image to be processed and the second image to be processed to obtain the prior reconstructed image.

[0008] Among them, image alignment may include key point alignment and / or optical flow alignment.

[0009] After aligning the prior image with the first image to be processed, it can be ensured that the second image to be processed includes both the content in the prior image and the content in the first image to be processed, and the content in the prior image that is the same or similar to the content in the first image to be processed has the same position in the second image to be processed, so as to better fuse the first image to be processed and obtain a clear prior reconstructed image.

[0010] In a possible implementation of the first aspect, the prior image is aligned with the first image to be processed to obtain the second image to be processed, including: aligning the prior image with the first image to be processed at key points to obtain the third image to be processed; and aligning the third image to be processed with the first image to be processed at optical flow to obtain the second image to be processed.

[0011] By aligning the key points of the prior image with the first image to be processed, physical alignment can be achieved. After that, the third image to be processed obtained by the key point alignment is optically aligned with the first image to be processed, so that the image texture in the obtained second image to be processed can be clearer.

[0012] In a possible implementation of the first aspect, fusing the first image to be processed and the second image to be processed to obtain a priori reconstructed image includes: fusing the first image to be processed and the second image to be processed based on an attention mechanism to obtain a priori reconstructed image.

[0013] The attention mechanism can fuse multiple features together and focus on the parts that need attention during the fusion process to reduce the influence from other parts.

[0014] In another possible implementation of the first aspect, the process of enhancing the first image to be processed based on the prior image to obtain the prior reconstructed image includes: inputting the prior image and the first image to be processed into a priori reconstruction model to obtain the prior reconstructed image; the priori reconstruction model is trained based on a training sample set, and the priori reconstruction model has the function of restoring the low-resolution image to a high-resolution image based on the prior image; wherein the training sample set includes multiple groups of training samples, each group of training samples includes a first image with a clarity less than a third threshold, a first image with a clarity greater than a fourth threshold, and a second image that includes the same photographed object as the first image and has a clarity greater than a fifth threshold.

[0015] In some examples, the low-resolution image refers to a degraded image with lower image clarity or the first image to be processed, and the high-resolution image refers to a priori reconstructed image with higher image clarity.

[0016] Among them, the first image with a clarity less than the third threshold and the first image with a clarity greater than the fourth threshold are both first images, and the difference between them is that the clarity is different. The third threshold and the fourth threshold here are used to distinguish the clarity between the two first images, and the relationship between the third threshold and the fourth threshold can be determined according to actual conditions. For example, in order to ensure that the first image with a clarity greater than the fourth threshold is clearer than the first image with a clarity less than the third threshold, the third threshold can be made less than or equal to the fourth threshold.

[0017] The prior reconstruction model is pre-trained based on the training sample set, and the electronic device can directly use the prior reconstruction model to restore the low-resolution image to the high-resolution image, thereby improving efficiency. In addition, in order to ensure that the prior reconstruction model can accurately restore the low-resolution image to the high-resolution image, the prior reconstruction model can be trained based on multiple sets of training samples including a degraded first image, a clear first image, and a clear second image, thereby improving the accuracy of the prior reconstruction model.

[0018] In a possible implementation manner of the first aspect, the first image and the second image are captured under different lighting conditions.

[0019] By using the first image and the second image captured under different lighting conditions to train the priori reconstruction model, the accuracy and robustness of the output of the priori reconstruction model can be guaranteed.

[0020] In a possible implementation manner of the first aspect, a posture of the subject in the first image is different from a posture of the subject in the second image.

[0021] When the posture of the subject in the first image is different from that in the second image, it can be ensured that the prior reconstruction model accurately outputs a high-resolution image based on the input low-resolution image including the subject in different postures and the prior image, thereby improving the image quality and display effect.

[0022] In a possible implementation of the first aspect, the process of obtaining a priori image includes: identifying the identity of a person being photographed in a first image to be processed; and determining a priori image in a third image based on the identity of the person, wherein the third image is an image having a clarity greater than a second threshold.

[0023] In some embodiments, the photographed object may be a person, an animal, etc. In some cases, the photographed object may also be other recognizable objects, such as iconic landscapes, buildings, puppets, etc.

[0024] The electronic device can identify the identity of the subject according to face recognition technology. When the subject is an animal or other object, the electronic device can implement the identification process of the subject according to image recognition technology.

[0025] After the electronic device acquires the first image to be processed, it can determine the corresponding prior image based on the identity of the subject in the first image to be processed, thereby ensuring that the subject in the prior image is the same as the subject in the first image to be processed, and then ensuring that the electronic device can use the prior image to perform a priori reconstruction process on the first image to be processed to obtain a clear priori reconstructed image.

[0026] In a possible implementation of the first aspect, the first interface is used to display images, the first interface includes a first image to be processed, the first control is an image enhancement control, and the image enhancement control is used to enhance the first image to be processed based on a priori image to obtain a priori reconstructed image.

[0027] The user can operate the first control in the first interface so that the electronic device can enhance the first image to be processed based on the prior image to obtain a priori reconstructed image, thereby improving image quality and display effect.

[0028] In a second aspect, the present application provides an electronic device, comprising a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in the first aspect and any possible design method thereof.

[0029] In a third aspect, the present application provides a computer-readable storage medium, comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible design thereof.

[0030] In a fourth aspect, the present application provides a computer program product, including a computer program / instructions, which, when the computer program / instructions are executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible design thereof.

[0031] In a fifth aspect, the present application provides a device, which is included in an electronic device, and the device has the function of implementing the behavior of the electronic device in any of the above aspects and possible implementation methods. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above function. For example, an allocation module or unit, a scanning module or unit, a recycling module or unit, a moving module or unit, and a storage module or unit, etc.

[0032] In a sixth aspect, an embodiment of the present application provides a chip system, which includes a processor and may also include a memory, for implementing any one of the methods provided in the first aspect. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0033] It can be understood that the electronic device described in the second aspect and any possible design thereof provided above, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a scene for capturing a degraded image provided by an embodiment of the present application;

[0035] Figure 2 A schematic diagram of the principle analysis of a de-degradation algorithm provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of a face super-resolution module for degrading provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0039] Figure 6 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 1 ;

[0041] Figure 8 A schematic diagram of an image processing method in a photo-taking scenario provided by an embodiment of the present application;

[0042] Fig. 9 A schematic diagram of an image processing method in a scene where an image has been obtained provided by an embodiment of the present application;

[0043] Fig.10 A schematic diagram of another image processing method in a scenario where an image has been obtained provided by an embodiment of the present application;

[0044] Fig.11 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 2 ;

[0045] Fig.12 A schematic diagram of facial key points provided in an embodiment of the present application;

[0046] Fig.13 A schematic diagram of key point alignment provided in an embodiment of the present application;

[0047] Fig.14 A schematic diagram of optical flow alignment provided in an embodiment of the present application;

[0048] Fig.15 A schematic diagram of an optical flow alignment calculation provided in an embodiment of the present application;

[0049] Fig.16 A schematic diagram of image fusion provided in an embodiment of the present application Figure 1 ;

[0050] Fig.17 A schematic diagram of image fusion provided in an embodiment of the present application Figure 2 ;

[0051] Fig.18 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0052] Fig.19 A schematic diagram of training data provided in an embodiment of the present application;

[0053] Fig. 20 A schematic diagram of a priori reconstruction model provided in an embodiment of the present application;

[0054] Fig.21 A schematic diagram of an image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the following, the terms "first" and "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0056] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] Before introducing the embodiments of the present application, the technologies involved in the embodiments of the present application are first introduced in detail.

[0058] The word a priori originated from the terminology of scholastic philosophy. In the usual sense, it is understood as the opposite of "experience", meaning that it precedes experience but is indispensable to the formation of experience.

[0059] In the field of image processing, prior information generally refers to information that can be directly obtained through a series of image processing, which is unique information that can be distinguished from other images. For example, in the related art, an RGB image is a natural image, and the RGB image is processed into a grayscale image, which only includes brightness information. The brightness information is the unique information that distinguishes the RGB image from other images. Therefore, the grayscale image can be called the prior information of the RGB image.

[0060] In the embodiments of the present application, the description of "prior image" is used. The prior image in the embodiments of the present application may refer to an image including information that can be used to restore a degraded image to a clear image. Exemplarily, the portrait included in the prior image and the portrait included in the degraded image are portraits of the same person, and the prior image is a clear image. The specifics of the prior image are not described in detail here, and please refer to the detailed description in the subsequent embodiments.

[0061] Taking a mobile phone as an example, the following describes the scenario where degraded images are captured. When a user opens the camera application on the mobile phone to capture a photo, Figure 1 As shown, the face of the subject is displayed in the viewfinder of the mobile phone. In general, users hold the mobile phone to take pictures. Then, when the user presses the shutter button, the hand shaking may cause the mobile phone to shake; or, in a dark photo scene, the image may contain noise due to inappropriate shooting parameters. In this case, the captured image may be degraded. For example, in response to the user clicking on the thumbnail icon for opening the gallery, the mobile phone can display the image taken when the user presses the shutter button. Figure 1 As shown, it can be seen that the face in the captured image is degraded.

[0062] To avoid taking pictures like Figure 1 The degraded image shown in the figure, the idea of ​​the de-degradation algorithm in the current related art is to obtain a high resolution (HR) face image by enhancing a low resolution (LR) face image. Therefore, the de-degradation algorithm focuses on analyzing the LR face image to restore the corresponding HR face image based on the LR face image and the analysis result.

[0063] Furthermore, in the process of restoring a degraded image (such as the LR face image mentioned above) to a clear image (such as the HR face image mentioned above), it is necessary to find the cause of image degradation, and then establish a mathematical model related to the cause of image degradation according to the cause of image degradation. After that, the degraded image is restored to a clear image using the established mathematical model.

[0064] Specifically, the de-degradation algorithm in the current related art uses a face super-resolution model to restore the LR face image. Before determining the face super-resolution model, the relationship between the degraded image and the clear image can be analyzed based on the HR face image and the LR face image, and the relationship between the two can be mathematically converted into a degradation model. For example, the related art determines the degradation model based on the analysis between the HR face image and the LR face image. The degradation model Φ can be mathematically converted into the following formula (1):

[0065] Formula (1)

[0066] in, represents the LR face image, represents HR face image, Represents the parameters of the degradation model, which includes the blur kernel, downsampling parameters, and noise.

[0067] The degradation model Φ is used to represent the degradation relationship between the LR face image and the HR face image. Figure 2 As shown, through the degradation model, the clear image I2 can be degraded into the degraded image I1.

[0068] According to the above degradation model Φ, the inverse process for simulating the degradation model can be determined, thereby obtaining the corresponding face super-resolution (FSR) model, that is, the F model in the following formula (2):

[0069] Formula (2)

[0070] in, represents the LR face image, represents a super-resolution (SR) face image, represents the parameters of the F model. It can be solved by the following formula (3):

[0071] Formula (3)

[0072] in, represents the optimal training parameters of the F model, and L represents and The loss between.

[0073] After the F model is determined according to the degradation model, Figure 2 The degraded image I1 in is restored to the clear image I2.

[0074] In this way, the obtained F model can be used to process the degraded image into a clear reconstructed image. Figure 3As shown in the figure, if a mobile phone captures a degraded image, the mobile phone can input the degraded image into a predetermined F model to obtain a reconstructed image output by the F model. However, since the F model is actually obtained in advance based on deep learning of multiple training samples including portraits, and the portraits in the training samples are different from the portraits in the degraded image, after the degraded image is processed using the F model, some details of the portrait in the reconstructed image are missing or different from the details of the portrait captured in the degraded image. As a result, the final image quality is not high and the display effect is not good.

[0075] To this end, the present application embodiment provides an image processing method, such as Figure 4 As shown, a clear image can be used as a priori image, and the priori image and the degraded image are used to perform priori reconstruction processing to obtain a priori reconstructed image. Among them, the portraits included in the priori image and the degraded image are portraits of the same person, the priori image is a clear image, and the priori image includes image details of the portrait. Since the priori reconstructed image is obtained by performing priori reconstruction processing based on the priori image and the degraded image, the posture of the portrait included in the priori reconstructed image is consistent with that in the degraded image, and the priori reconstructed image also includes part or all of the image details of the portrait included in the prior image. The priori reconstructed image obtained by the above method is not only clearer than the degraded image, but also includes more image details of the portrait, which can effectively improve the image quality and display effect.

[0076] In order to better understand the embodiments of the present application, the electronic device provided by the embodiments of the present application is first introduced.

[0077] The image processing method provided in the embodiment of the present application can be applied to electronic devices, which can be mobile phones, tablet computers, smart screens, laptop computers, vehicle-mounted devices, wearable devices (such as smart watches), ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), artificial intelligence devices, and other terminal devices. The embodiment of the present application does not limit the specific type of electronic devices or the installed operating system.

[0078] The following introduces the hardware structure of the electronic device provided in the embodiments of the present application.

[0079] Figure 51 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0080] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0081] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0082] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0083] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0084] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0085] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0086] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0087] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0088] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0089] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0090] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0091] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194.

[0092] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0093] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology.

[0094] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0095] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0096] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0097] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0098] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0099] When the camera 193 captures a degraded image, the image processing method provided in the embodiment of the present application can be used to enhance the degraded image based on a priori image corresponding to the degraded image to obtain a priori reconstructed image, thereby improving image quality and display effect.

[0100] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0101] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0102] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0103] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0104] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0105] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, speakers, receivers, microphones, headphone interfaces, and application processors.

[0106] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110. The speaker is used to convert the audio electrical signal into a sound signal. The receiver is used to convert the audio electrical signal into a sound signal. The microphone is used to convert the sound signal into an electrical signal. The headphone jack is used to connect a wired headset.

[0107] The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0108] The buttons 190 include a power button, a volume button, etc. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power change, messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communications.

[0109] The following introduces the software structure of the electronic device provided in the embodiments of the present application.

[0110] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes an operating system of a layered architecture as an example to exemplify the software structure of the electronic device 100.

[0111] Figure 6 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0112] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into four layers, from top to bottom: the application layer, the application framework layer, the runtime and system library, and the kernel layer.

[0113] It should be noted that the embodiment of the present application is illustrated by taking an operating system as an example. In other operating systems, as long as the functions implemented by each functional module are similar to those of the embodiment of the present application, the solution of the present application can also be implemented.

[0114] The application layer can include a series of application packages. Figure 6 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0115] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0116] like Figure 6 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, a priori reconstruction module, and the like.

[0117] The window manager is used to manage window programs. The content provider is used to store and obtain data and make the data accessible to applications. The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The phone manager is used to provide communication functions for the electronic device 100. The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction.

[0118] The priori reconstruction module is used to perform priori reconstruction processing on the degraded image based on the priori image to obtain a clear image.

[0119] Runtime includes core libraries and virtual machines. Runtime is responsible for the scheduling and management of the operating system.

[0120] The core library consists of two parts: one part is the function that the Java language needs to call, and the other part is the core library of the operating system.

[0121] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0122] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0123] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0124] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0125] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0126] A 2D graphics engine is a drawing engine for 2D drawings.

[0127] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0128] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with a photo-taking scenario.

[0129] When the touch sensor receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, timestamp of the touch operation, and other information). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a single-click operation and the control corresponding to the single-click operation is a shooting control displayed on the camera viewfinder interface, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture the image through the camera 193. In the case where the captured image is a degraded image, the priori reconstruction module in the application framework layer can determine the priori image corresponding to the degraded image, and perform priori reconstruction processing on the degraded image based on the priori image to obtain a clear image.

[0130] In the above-mentioned photo-taking scenario, when the user presses the shooting control, after the electronic device obtains a degraded image through the camera, it can execute the following image processing method to obtain a clear image.

[0131] In other scenarios, the user can also select a degraded image in the gallery, and after clicking the corresponding prior reconstruction control, the following image processing method can be executed to obtain a clear image.

[0132] The above scenario is described in detail in the subsequent introduction to the image processing method and will not be repeated here.

[0133] The image processing method provided by the embodiments of the present application is described below with reference to the accompanying drawings.

[0134] Figure 7 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 1 , the image processing method is executed by an electronic device.

[0135] S701: The electronic device obtains a degraded image and a priori image corresponding to the degraded image.

[0136] The degraded image and the prior image corresponding to the degraded image include the same photographed object, so that the electronic device can use the photographed object in the prior image to repair the photographed object in the degraded image. The above-mentioned degraded image can be the first image to be processed in the embodiment of the present application, and the clarity of the degraded image is less than the first threshold.

[0137] Furthermore, in order to ensure the restoration effect of the degraded image, an image with a clarity greater than a threshold can be used as a prior image. In this way, the degraded image can be effectively restored to a clear image by utilizing the features of the photographed object in the clear prior image. For example, Figure 4 It can be clearly seen that the face and other parts of the body of the subject in the degraded image have overlapping contours, while the subject in the prior image is clear and has no overlapping contours.

[0138] In some examples, since the clarity of the prior image is greater than the threshold, the prior image includes image details such as moles, wrinkles, hair, etc. For example, Figure 4 In the prior image, it can be clearly seen that the subject has a mole below the right eye and on the left side of the chin, and there are wrinkles at the corners of the mouth, but these facial features are not reflected in the degraded image. In this way, it can be ensured that the prior reconstructed image obtained using the clear prior image also includes these facial features, thereby improving the image quality.

[0139] In the case where the definition of the degraded image is less than the first threshold, the definition of the a priori image in the present application is greater than the second threshold. There may be no magnitude relationship between the first threshold and the second threshold.

[0140] In some embodiments, the posture of the subject in the prior image and the posture of the subject in the degraded image may be the same or different. In most cases, the posture of the subject in the prior image and the posture of the subject in the degraded image are different.

[0141] The posture of the photographed object may refer to the action or body part of the photographed object in the image. Figure 4 As shown, the subject in the degraded image is the same as the subject in the prior image, the body parts of the subject in the degraded image are from the head to the shoulders, and the subject has no movement, while the body parts of the subject in the prior image are from the head to the waist, and the movement of the subject is to cross his hands and place them in front of his chest.

[0142] The acquisition method of the degraded image and the prior image corresponding to the degraded image corresponds to different application scenarios. For example, in the scenario of taking a photo, the degraded image and the prior image corresponding to the degraded image may be determined by the electronic device itself. For another example, in the scenario of an image that has been obtained, the degraded image and the prior image corresponding to the degraded image may be selected by the user, or the degraded image is selected by the user, and the prior image corresponding to the degraded image may be determined by the electronic device itself.

[0143] The following introduces the specific implementation of S701 in the scene of taking pictures.

[0144] In the scene of taking a photo, taking the electronic device as a mobile phone as an example, the display screen of the mobile phone shows the following Figure 8 The preview interface shown in the figure includes a viewfinder, a shooting control, etc. The image displayed in the viewfinder is the preview data obtained by the current camera in real time. When the user presses the shooting control, the mobile phone will obtain the image captured by the camera at that moment to complete the shooting action.

[0145] In some embodiments, in a photo-taking scenario, the electronic device may use the image captured by the camera when the user presses the shooting control as a degraded image. Alternatively, in a photo-taking scenario, and when the electronic device turns on the automatic image processing function, the electronic device may use the image captured by the camera when the user presses the shooting control as a degraded image.

[0146] Among them, the electronic device turning on the automatic image processing function may mean that the electronic device can execute the image processing method provided in the embodiment of the present application to process the degraded image into a priori reconstructed image based on the prior image, thereby obtaining a clear image.

[0147] The above-mentioned preview interface can be the first interface in the embodiment of the present application, and the shooting control can be the first control in the embodiment of the present application.

[0148] It is understandable that the image acquired at the moment the user presses the shooting control may not be of low quality. For example, the image acquired by the electronic device at the moment the user presses the shooting control is not actually of low quality in the eyes of the human eye. For another example, the image degradation degree is calculated by an image quality evaluation algorithm, and the degradation degree indicates that the image is less degraded. Therefore, in other embodiments, in a photo-taking scenario, the electronic device may determine whether the image is a degraded image after acquiring the image captured by the camera at the moment the user presses the shooting control. Alternatively, in a photo-taking scenario, and when the electronic device turns on the automatic image processing function, the electronic device may determine whether the image is a degraded image after acquiring the image captured by the camera at the moment the user presses the shooting control.

[0149] In some examples, the electronic device may determine whether the image is a degraded image based on the clarity of the image. For example, after the electronic device calculates the degradation degree of the image through an image quality evaluation algorithm, it determines whether the degradation degree of the image is greater than a preset threshold. When the degradation degree of the image is greater than the preset threshold, it indicates that the image has a degradation problem, and the electronic device may treat the image as a degraded image and continue to perform subsequent steps. Correspondingly, when the degradation degree of the image is less than the preset threshold, it indicates that the degradation problem of the image is minor, and the electronic device may not treat the image as a degraded image.

[0150] In the above embodiment, the image quality evaluation algorithm can be any one algorithm or a combination of multiple algorithms such as grayscale variance algorithm, grayscale difference absolute value sum variance algorithm, grayscale difference square sum variance algorithm and Laplace gradient algorithm. The specific algorithm content can be referred to the relevant technology and will not be described here.

[0151] In some other examples, the electronic device can detect the shaking of the electronic device and determine whether the image is a degraded image based on the detection result. For example, if the electronic device detects that the electronic device is shaking when the camera captures the image, it means that the image obtained at this time may be degraded, and therefore the image can be regarded as a degraded image.

[0152] In the above-mentioned photo-taking scenario, after the electronic device determines the degraded image, it can confirm the prior image corresponding to the degraded image according to the identity of the person in the degraded image. For example, the electronic device can identify the identity information of the person in the degraded image, and then select an image including the person with the same identity information as the prior image in the existing image library.

[0153] Among them, the way in which the electronic device recognizes the identity information of the person can be technology such as face recognition. For details, please refer to the relevant technology, which will not be repeated here.

[0154] The following describes the specific implementation of S701 in a scenario where an image has been obtained.

[0155] The scene where an image has been obtained refers to a scene where the electronic device has obtained a degraded image by shooting, communicating with other electronic devices, etc., and the image has been stored in the electronic device. Fig. 9 As shown, a plurality of stored images are displayed in the display interface of the mobile phone.

[0156] In some embodiments, in a scene where an image has been captured, the electronic device may use the image selected by the user as a degraded image. Fig. 9 As shown in FIG. 1 , the mobile phone displays an album display interface, which includes zoom icons for multiple images. Fig. 9 As shown, after the user clicks the zoom icon corresponding to an image, the mobile phone displays the display interface of the image. The image display interface may include an image enhancement control. When the user clicks the image enhancement control, the electronic device may treat the image as a degraded image.

[0157] The above-mentioned image display interface may be the first interface in the embodiment of the present application, the image enhancement control may be the first control in the embodiment of the present application, and the above-mentioned album display interface may be the second interface in the embodiment of the present application.

[0158] It should be noted that the electronic device may also obtain multiple degraded images at one time, and then obtain a priori images corresponding to each degraded image. Fig.10 As shown, the user long presses the zoom icon of one of the images in the album display interface displayed on the mobile phone to start batch selection of degraded images. In the batch selection of degraded images interface, as shown Fig.10 As shown, there may be a select all control in the upper right corner. When the user clicks the select all control, the electronic device may use all images in the album as degraded images. Fig.10 As shown, a selection control is set in the upper left corner of the zoom icon of each degraded image. The selection control is in the shape of a hollow circle, indicating that the degraded image is not selected, and the selection control is in the shape of a circle with a check mark in the center, indicating that the degraded image is selected. Fig.10 As shown, when the zoom icons of the three degraded images are in the shape of a circle with a check mark in the center, in response to the user clicking the image enhancement control, the electronic device uses the corresponding three degraded images as degraded images to be processed.

[0159] The above-mentioned album display interface can be the first interface in the embodiment of the present application, and the image enhancement control can be the first control in the embodiment of the present application.

[0160] Among them, when the user clicks Figure 8In the case of a thumbnail icon for opening the gallery in the viewfinder of the mobile phone shown in the figure, or when the user clicks the icon of the gallery application, the mobile phone can display the following Fig. 9 The album display interface shown.

[0161] After the electronic device determines the degraded image, the electronic device may determine the prior image corresponding to the degraded image according to the user's selection, or the electronic device may determine the prior image corresponding to the degraded image by itself. Fig. 9 As shown, after the user clicks on the image enhancement control included in the image display interface, the mobile phone can display a priori image selection interface, which includes zoom icons for multiple images. After the user clicks to select the zoom icon corresponding to one of the images and clicks the confirmation control, the electronic device can use the image corresponding to the zoom icon as the prior image. For another example, in the case of batch selection of multiple degraded images, the user can also select a corresponding prior image for each degraded image in the corresponding prior image selection interface. For another example, the electronic device can also confirm the prior image corresponding to the degraded image based on the identity of the person in the degraded image. For details, please refer to the above embodiment, which will not be repeated here.

[0162] In the above embodiment, the personal information used in the image processing method is limited to information for which the individual's separate consent is obtained, including but not limited to notifying and reminding the user to read the relevant user agreement or notice and sign an agreement including authorization of relevant user information before the user uses the function.

[0163] For example, when an electronic device implements the image processing method provided in the embodiment of the present application, it is necessary to read the image and other information stored in the memory and identify the personal information in the image. Therefore, before the electronic device implements the image processing method provided in the embodiment of the present application, the electronic device can display authorization prompt information, which includes prompt text, an authorization consent option, and a authorization rejection option. Among them, the prompt text is used to introduce the personal information that the electronic device needs to use when implementing the above-mentioned image processing method and the scope of use of the personal information.

[0164] In some examples, the electronic device displays a prompt text to notify and remind the user to read the relevant user agreement or notification, and after the user signs the agreement including the authorization of relevant user information, the electronic device can execute the image processing method provided in the embodiment of the present application. For example, if the user reads the prompt text and selects the option to agree to the authorization, the electronic device can execute the image processing method provided in the embodiment of the present application. If the user reads the prompt text and selects the option to refuse authorization, the electronic device cannot execute the image processing method provided in the embodiment of the present application.

[0165] After the electronic device acquires the degraded image and the prior image, the following S702 may be performed:

[0166] S702: The electronic device reconstructs the degraded image based on the prior image to obtain a clear image.

[0167] The clear image mentioned above may be the a priori reconstructed image in the present application, wherein the electronic device may implement a priori reconstruction by fusing the a priori image and the degraded image.

[0168] In some embodiments, the electronic device may fuse image details related to the person in the prior image into the degraded image to obtain a clear image with clear image details. For example, the electronic device determines image details related to the person from the prior image, then finds the location of these image details in the degraded image, and superimposes the image details to the location to obtain a clear image.

[0169] In some other embodiments, the electronic device may fuse features in the prior image to corresponding positions in the degraded image to obtain a clear image. The image features may refer to color features, texture features, shape features, etc. The color feature describes the color of a certain position in the image, the texture feature describes the surface properties of an object at a certain position in the image, and the shape feature describes the outline of an object at a certain position in the image.

[0170] In some examples, the electronic device determines the image feature under the right eye of the subject from the prior image as a mole, then determines the corresponding position under the right eye of the subject from the degraded image, and then fuses the mole under the right eye of the subject to the corresponding position in the degraded image to obtain a clear image.

[0171] The algorithms used for fusion involved in the above embodiments, such as the Poisson fusion algorithm, the pyramid fusion algorithm, etc., can be found in related technologies and will not be described in detail here.

[0172] After the electronic device executes S702, a clear image may be obtained. The electronic device may store the clear image in a memory of the electronic device, and display the clear image on a display screen when a user needs to view the clear image.

[0173] In some embodiments, in a photo-taking scenario, such as Figure 8 As shown in FIG. 1 , after the user presses the shooting control, the mobile phone shakes. After the above image processing method, the thumbnail icon of the obtained clear image is displayed in the lower left corner of the preview interface. Figure 8 The thumbnail icon of the clear image in the lower left corner of the preview interface is a display interface corresponding to the mobile phone display gallery, and the display interface includes the clear image processed by the above-mentioned image processing method.

[0174] In some other embodiments, in the scenario where an image has been obtained, such as Fig. 9 As shown, in response to the user selecting a degraded image, the mobile phone displays a selection interface for a priori images. Thereafter, in response to the user selecting a priori image and clicking OK, the mobile phone executes the above-mentioned image processing method to obtain a clear image, and displays the clear image on the display screen of the mobile phone for the user to view.

[0175] In order to explain the image processing method provided by the embodiment of the present application in more detail, the following is combined with Figure 11-Figure 18 Make an introduction.

[0176] Fig.11 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 2 , the image processing method is executed by an electronic device.

[0177] S1101. The electronic device obtains an image to be processed LQ1 and a priori image REF.

[0178] The prior image REF may be a prior image obtained by the method in the above S701 related embodiment. Based on the common understanding of the word "prior" and the role played by the prior image in the embodiment of the present application, the present application uses the English word "reference" to refer to the prior image REF.

[0179] In some embodiments, the image to be processed in S1101 of this embodiment corresponds to the degraded image in S701 of the above embodiment, and therefore, the image to be processed can also be referred to as a low quality image. Therefore, the image to be processed in S1101 is distinguished from the images to be processed in subsequent steps by LQ1. The above image to be processed LQ1 can be the first image to be processed in this application.

[0180] In some other embodiments, the image to be processed LQ1 may also be a low-quality image that has been preprocessed. The low-quality image may be the degraded image in the above-mentioned embodiment S701. The preprocessing may remove part of the noise in the low-quality image, or improve the clarity of the low-quality image, thereby ensuring that the priori reconstructed image REC can be more accurately determined by subsequently using the image to be processed LQ1 and the priori image REF.

[0181] Herein, this application uses the English word “reconstruction” to refer to the a priori reconstructed image REC.

[0182] In some examples, preprocessing may include filtering. The filtering method may include one or more of a spatial domain-based filtering method and a frequency domain-based filtering method. Among them, the spatial domain-based filtering method may include one or more of mean filtering, termination filtering, and Gaussian filtering. The frequency domain-based filtering method includes the following process: first Fourier transform the image, then filter the image in the frequency domain, and finally, inverse Fourier transform the filtered image to obtain a denoised image, that is, the image to be processed LQ1. For details of the above method, please refer to the relevant technology, which will not be repeated here.

[0183] In some other examples, preprocessing may also include, for example, processing of low-quality images using a de-degradation algorithm in the related art, and reference may be made to the description of the above-mentioned related art, which will not be repeated here.

[0184] It is understandable that other preprocessing methods may also be used, and no specific preprocessing method is limited here.

[0185] After obtaining the image to be processed LQ1 and the prior image REF, in order to be able to achieve prior reconstruction based on the fusion of the image to be processed LQ1 and the prior image REF, thereby obtaining a clear prior reconstructed image, the electronic device can first extract the key points in the image through the following S1102 and S1103, and perform preliminary fusion processing on the image to be processed LQ1 and the prior image REF based on the key points.

[0186] S1102: The electronic device obtains facial key points of the image to be processed LQ1 and facial key points of the prior image REF.

[0187] Since the image to be processed LQ1 and the prior image REF are not exactly the same, the positions of certain features of the photographed object in the image to be processed LQ1 are not exactly the same as the positions of the same features in the prior image REF. In order to ensure that the image to be processed LQ1 and the prior image REF can be fused, the electronic device can find facial key points from the image to be processed LQ1 and the prior image REF respectively, so as to facilitate subsequent image alignment based on the facial key points.

[0188] Among them, the facial key points may include facial key points, human face key points, etc. In some examples, the number of facial key points is generally six, such as Fig.12 As shown, there is one key point at each pupil of the two eyes, one key point at the tip of the nose, and three key points at the mouth, one of which is in the middle of the mouth, and the other two are at the corners of the mouth. In some other examples, facial key points may also refer to human face key points, which are located at eyebrows, eyes, nose bridge, nose wings, lips, etc. For details, please refer to the relevant technology, which will not be repeated here.

[0189] It should be noted that in S1102, the electronic device can determine the key points in the image to be processed LQ1 and the prior image REF according to the current actual image processing requirements. In the embodiment of the present application, face alignment is taken as an example, so the electronic device extracts the facial key points of the image to be processed LQ1 and the facial key points of the prior image REF in S1102. If it is actually necessary to align the human body posture, the electronic device can extract the human body key points of the image to be processed LQ1 and the human body key points of the prior image REF in S1102, and no specific restrictions are imposed on the key points extracted by the electronic device.

[0190] In some embodiments, the postures of the characters in the image to be processed LQ1 and the prior image REF may be different, or the image sizes of the image to be processed LQ1 and the prior image REF may be different, so the positions of the same features in the two images may be different. For example, when the image sizes of the image to be processed LQ1 and the prior image REF are exactly the same, a two-dimensional coordinate system is established in the same way to describe the position of each pixel in the image. At this time, the position of the pixel corresponding to the pupil center of the subject's eye in the image to be processed LQ1 is (1, 2), and the position of the pixel corresponding to the pupil center of the subject's eye in the prior image REF is (2, 3). If the electronic device directly fuses the image to be processed LQ1 and the prior image REF, it will result in the presence of images of the subject's eyes at positions (1, 2) and (2, 3) in the fused prior reconstructed image, resulting in unclear prior reconstructed images.

[0191] To this end, after executing S1102, the electronic device aligns the same facial key points at different positions in the two images through the following S1103 to ensure that a clear a priori reconstructed image is finally obtained.

[0192] S1103: The electronic device aligns the facial key points of the image to be processed LQ1 and the facial key points of the prior image REF to obtain the image to be processed LQ2 after the key points are aligned.

[0193] According to the above embodiments, the position of the same facial key point in the image to be processed LQ1 and the position in the prior image REF may be different. Therefore, the alignment of the same facial key point at different positions in different images can be achieved through topological transformation or rotation and translation shearing.

[0194] Here, alignment refers to the process of rotating, flipping, shearing, scaling and translating the content in the prior image REF based on the content in the image to be processed LQ1 to make it as close as possible to the content in the image to be processed LQ1.

[0195] In some examples, the content in the prior image REF can be rotated, flipped, sheared, scaled and translated by an affine matrix. For example, taking the facial key point as the eye, the position of the pixel corresponding to the pupil center of the eye in the prior image REF is (x, y), and this position is recorded as the matrix M1. The position of the pixel corresponding to the pupil center of the eye in the image to be processed LQ1 is (x', y'), and this position is recorded as the matrix M2. Theoretically, the relationship between the matrix M1 and the matrix M2 can be calculated through the rotation principle of the matrix, that is, M1=A·M2. Among them, A is an unknown affine matrix, which is used to represent the relationship between the rotation, flipping, shearing, scaling and translation of the pixel position in the matrix M1 and the pixel position in the matrix M2.

[0196] For the above facial key points of the eyes, two equations related to A can be obtained. Then, for the other four facial key points, four equations related to A can be obtained accordingly. In this way, a set of equations can be obtained based on the six equations related to A, and A can be solved using the idea of ​​least squares solution. Afterwards, the electronic device can convert the positions of other pixels in the prior image REF into the positions of the pixels in the image to be processed LQ1 according to the known affine matrix A, and superimpose the converted pixels with the pixels at the corresponding positions in the image to be processed LQ1 to obtain the image to be processed LQ2 after key point alignment. For example, Fig.13 As shown, the electronic device aligns the facial key points of the image to be processed LQ1 and the facial key points of the prior image REF, and obtains the image to be processed LQ2 after the key points are aligned. The image to be processed LQ2 includes the content of the image to be processed LQ1 and the content of the prior image REF, for example, Fig.13 It can be seen that the mole under the right eye and the mole on the left side of the chin of the subject in the prior image both appear in the image to be processed LQ2, and the wrinkles at the corners of the mouth of the subject in the prior image also appear in the image to be processed LQ2.

[0197] In some embodiments, after obtaining the image to be processed LQ2 after key point alignment, the electronic device may process the image to be processed LQ2 to obtain a priori reconstructed image.

[0198] Optionally, after executing S1103, the electronic device may also execute S1104 to ensure that the image to be processed LQ2 after key point alignment and the prior image REF can be more accurately matched, so that the pixels in the image to be processed LQ2 after key point alignment and the pixels in the prior image REF can be better integrated, thereby improving the clarity of the prior reconstructed image. The above-mentioned image to be processed LQ2 may be the third image to be processed in the present application.

[0199] S1104: The electronic device performs optical flow alignment on the image to be processed LQ2 after key point alignment and the prior image REF to obtain an image to be processed LQ3 after optical flow alignment.

[0200] like Fig.14 The figure shows the image to be processed LQ3 after optical flow alignment. It can be seen that the feature part corresponding to the prior image REF in the image to be processed LQ3 is fused with the feature corresponding to the image to be processed LQ2. Fig.13 In the image to be processed LQ2 shown, it can be seen that the contour of the eyebrow of the photographed subject retains the contour of the image to be processed LQ1 and the prior image REF, resulting in the overall contour of the eyebrow of the photographed subject in the image to be processed LQ2 being larger, and the effect is similar to two different eyebrow shapes directly superimposed on each other. In the image to be processed LQ3, the eyebrow contour of the photographed subject is more similar to the eyebrow contour of the photographed subject in the image to be processed LQ1, and part of the eyebrow contour of the photographed subject in the prior image REF is retained, making the eyebrow contour of the photographed subject clearer.

[0201] The above-mentioned image to be processed LQ3 may be the second image to be processed in the present application.

[0202] The main idea of ​​optical flow alignment is to match each pixel in turn, and then move the pixel vector in the prior image to the position of the matching pixel vector in the image to be processed LQ2. In some examples, each image is regarded as a three-dimensional matrix, the height of the three-dimensional matrix is ​​H (height), the width is W (width), and the number of channels is C (channel). Then, the three-dimensional matrix can be considered to be composed of multiple pixel blocks. Fig.15 As shown, the prior image REF and the image to be processed LQ2 are both represented as three-dimensional matrices. Each three-dimensional matrix is ​​composed of multiple pixel vectors. For example, Fig.15 The pixel vector p in the prior image REF is represented as a vector , where R represents a real number. Afterwards, the electronic device performs a dot multiplication operation on the pixel vector p and each pixel vector in the image to be processed LQ2. The value obtained by each dot multiplication operation can represent the correlation between the pixel vector p and the corresponding pixel vector in the image to be processed LQ2. Finally, the maximum value obtained by all dot multiplication operations is selected, and the corresponding pixel vector in the image to be processed LQ2 used to calculate the value is used as the pixel vector p' corresponding to the pixel vector p in the prior image REF, as shown in Fig.15 Afterwards, the electronic device moves the pixel vector p in the prior image REF to the position of the corresponding pixel vector p' in the image to be processed LQ2, so that optical flow alignment can be achieved.

[0203] The electronic device can achieve physical pixel-level alignment of the image to be processed LQ1 and the prior image REF through the key point alignment process, i.e., executing S1103. After that, the electronic device can better fuse the images to be processed LQ2 and the prior image REF that have been physically pixel-level aligned through optical flow alignment, i.e., executing S1104, so that the image texture in the image to be processed LQ3 obtained is clearer.

[0204] Understandably, Fig.15 The position of the pixel vector p' is an example. In fact, the position of the pixel vector p' can be a pixel vector at any position in the image to be processed LQ2, which is specifically related to the result of the dot multiplication operation.

[0205] Afterwards, the electronic device executes S1105:

[0206] S1105 . The electronic device fuses the image to be processed LQ1 and the image to be processed LQ3 to obtain a priori reconstructed image REC.

[0207] In some embodiments, the Transformer attention mechanism can be used to fuse the image to be processed LQ1 and the image to be processed LQ3. For example, the image to be processed LQ1 and the image to be processed LQ3 are still regarded as three-dimensional matrices. Among them, the image to be processed LQ1 and are represented as HWC, and the image to be processed LQ3 is represented as H'W'C'. If the three-dimensional matrix of the image to be processed LQ3 is rearranged and written in the form of a two-dimensional matrix, C' is taken as one-dimensional data alone, and the H'W' array is arranged in sequence as one-dimensional data, it is equivalent to the rearranged matrix having H'×W' C', which is written as matrix Q. Matrix Q represents each pixel vector in the image to be processed LQ1. Then, if Fig.16 As shown, the image to be processed LQ1 can be expressed as Q=HWC. Correspondingly, if the three-dimensional matrix of the image to be processed LQ1 is rearranged and written in the form of a two-dimensional matrix, C is taken as one-dimensional data alone, and the HW array is arranged in sequence as one-dimensional data, it is equivalent to having H×W Cs, which are written as a matrix K, where K represents each pixel vector in the prior image REF, and the image to be processed LQ3 can be expressed as K=H'W'C'. The Transformer's attention mechanism calculates the similarity of all pixels in the Q matrix and the K matrix, and calculates the correlation matrix B=QKT=HWC×(H'W'C')T. The correlation matrix B is used to represent the similarity of all pixel vectors in the Q matrix and the K matrix, that is, it represents the similarity relationship between each pixel vector in the image to be processed LQ1 and each pixel vector in the image to be processed LQ3, as shown Fig.16 Then, the correlation matrix B is multiplied by the two-dimensional matrix form H'W'C' of the three-dimensional matrix of the image to be processed LQ3, that is, Fig.16The matrix V in can complete the weighting process, and after weighting, the prior reconstructed image REC can be obtained. Corresponding to the attention mechanism, such as Fig.16 As shown, the three-dimensional matrix H'W'C' of the image to be processed LQ3 used in the weighting process can be expressed as a matrix V. In the above process, V=K.

[0208] In the above process, taking the first pixel vector of the image to be processed LQ1 as an example, the first pixel vector of the image to be processed LQ1 is matched with multiple pixel vectors in the image to be processed LQ3, and the weighted sum of multiple pixel vectors of the prior image is performed based on the similarity principle to obtain the reconstruction result of the first pixel vector of the image to be processed LQ1. According to the above process, the reconstruction result of each pixel vector in the image to be processed LQ1 can be calculated, so that the fusion of the image to be processed LQ1 and the image to be processed LQ3 is completed based on the attention mechanism to obtain the prior reconstructed image.

[0209] In other embodiments, Fig.17 As shown, the electronic device can fuse the features of the image obtained after key point alignment, the features of the high-definition prior image, and the features of the low-quality image to be processed by concatenation. Among them, concate is the name of a fusion function, which uses a convolution kernel for fusion. In other words, the information of all channels of the above features can be fused together through the above fusion function, and the fusion of the attention mechanism can be completed, so as to obtain the features of the prior reconstructed image REC.

[0210] It should be noted that the above fusion of the image to be processed LQ3 and the prior image REF can be achieved through any one of the self-attention mechanism or the cross-attention mechanism, and this embodiment does not limit this.

[0211] The following, combined Fig.18 Introduce image processing methods, Fig.18 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 3 , the image processing method is executed by an electronic device.

[0212] After acquiring the low-quality image and the prior image REF, the electronic device can first input the low-quality image into the image preprocessing module for preprocessing to obtain the image to be processed LQ1. Afterwards, the electronic device inputs the image to be processed LQ1 and the prior image REF into the image processing module. The image processing module includes a key point alignment processing module, an optical flow alignment processing module and a fusion processing module, wherein the key point alignment processing module can execute S1102 and S1103 in the above embodiment, the optical flow alignment processing module is used to execute S1104 in the above image processing method, and the fusion processing module is used to execute S1105 in the above image processing method. Finally, the image processing module can output a priori reconstructed image REC.

[0213] In some embodiments, the electronic device can train a priori reconstruction model using multiple sets of training data. The electronic device can train a priori reconstruction model based on multiple sets of training data in advance, so that after the high-definition priori image REF and the image to be processed LQ1 are input into the trained priori reconstruction model, the priori reconstruction model can directly output a priori reconstruction image REC.

[0214] In other embodiments, a trained prior reconstruction model is configured in the electronic device, and the process of obtaining the prior reconstructed image using the image to be processed LQ and the prior image REF can be completed. The prior reconstruction model can be trained by other electronic devices or servers, and the electronic device can obtain the trained prior reconstruction model from other electronic devices or servers. Alternatively, the trained prior reconstruction model can be configured in a cloud server or cloud platform, and the electronic device can send the image to be processed LQ and the prior image REF to the cloud server or cloud platform, and then the electronic device receives the prior reconstructed image sent by the cloud server or cloud platform.

[0215] Among them, in the process of training the prior reconstruction model of the electronic device, the multiple sets of training data used may include the images to be processed, the prior images and the prior reconstructed images under different lighting conditions and postures, so as to improve the robustness of the trained prior reconstruction model, ensure that the trained prior reconstruction model focuses on introducing the texture details of the prior image without changing the color tone and facial posture of the low-quality image, and improve the accuracy of obtaining the prior reconstructed image using the prior reconstruction model.

[0216] For example, multiple sets of training data may include Fig.19 The image shown. Fig.19As shown, degraded image A is the image to be processed, and clear image A' is the prior image corresponding to degraded image A. Degraded image B is the image to be processed, and clear image B' is the prior image corresponding to degraded image B. Among them, the light in degraded image A is brighter, and the light in degraded image B is darker. The person in degraded image A is a frontal face, and the person in clear image A' is a side face. The posture of the person in degraded image A is quite different from that in clear image A'. The person in degraded image B and the person in clear image B' are both side faces, and the posture of the person in degraded image B is quite different from that in clear image B'.

[0217] In some embodiments, the electronic device may use the acquired high-definition image as a priori reconstructed image in the training data. Afterwards, one or more operations such as sharpening, contrast, blurring, noise, JPEG compression, downsampling, etc. may be performed on the high-definition image to obtain an image with reduced display quality, and the acquired image with reduced quality may be used as the image to be processed in the training data. In addition, the electronic device may obtain an image that includes the same photographed object as the high-definition image and has a clarity greater than a preset threshold as a priori image.

[0218] Among them, the high-definition image can be the first image in the embodiment of the present application whose clarity is greater than the fourth threshold, the image with reduced quality, that is, the image to be processed can be the first image in the embodiment of the present application whose clarity is less than the third threshold, and the prior image can be the second image in the embodiment of the present application that includes the same photographed object as the first image and has a clarity greater than the fifth threshold.

[0219] It should be noted that the first image with a clarity less than the third threshold and the first image with a clarity greater than the fourth threshold are both first images, and the difference between them is the different clarity. The third threshold and the fourth threshold are used to distinguish the clarity between the two first images, and the relationship between the third threshold and the fourth threshold can be determined according to actual conditions. For example, in order to ensure that the first image with a clarity greater than the fourth threshold is clearer than the first image with a clarity less than the third threshold, the third threshold can be made less than or equal to the fourth threshold.

[0220] Furthermore, the photographed objects included in the images in different groups of training data may be different, so as to ensure that the prior reconstruction model can obtain a clear image based on the image to be processed including any photographed object and the corresponding prior image, thereby improving the effect of restoring the degraded image to a clear image.

[0221] like Fig. 20As shown, when a trained prior reconstruction model is configured in an electronic device, after obtaining the image to be processed LQ1 and the prior image REF, the image to be processed LQ1 and the prior image REF can be input into the trained prior reconstruction model, and then the prior reconstruction model can output a prior reconstructed image REC.

[0222] like Fig. 20 As shown, the trained prior reconstruction model at least includes a facial key point alignment module, an optical flow alignment module, and an attention fusion module. The facial key point alignment module is used to execute S1102 and S1103 in the above image processing method, the optical flow alignment module is used to execute S1104 in the above image processing method, and the attention fusion module is used to execute S1105 in the above image processing method.

[0223] Below, from the perspective of the interaction between each module, combined with Fig.21 The image processing method provided by the embodiment of the present application is introduced. When the camera application is running in a mobile phone, a preview interface is displayed in the mobile phone, and the preview interface displays a shooting control. In response to the user clicking the shooting control, the camera captures a degraded image. Afterwards, the camera sends a degraded image display instruction to the gallery so that the mobile phone displays a display interface including the degraded image and a priori reconstruction control. In response to the user clicking the a priori reconstruction control, the gallery sends a priori reconstruction instruction to the a priori reconstruction module so that the a priori reconstruction module obtains the degraded image and the a priori image corresponding to the degraded image, and obtains the a priori reconstructed image based on the degraded image and the a priori image corresponding to the degraded image. Specifically, the a priori reconstruction module can execute the steps in the above embodiment based on the a priori reconstruction instruction to obtain the a priori reconstructed image, which will not be repeated here. Afterwards, the a priori reconstruction module can send the a priori reconstructed image to the memory so that the memory stores the a priori reconstructed image.

[0224] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step in the above-mentioned method embodiment.

[0225] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program runs on an electronic device, the electronic device executes each function or step in the above method embodiment.

[0226] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0228] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0229] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0230] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0231] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Display a first interface, wherein the first interface includes a first control; In response to an operation on the first control, a first image to be processed is acquired, wherein the clarity of the first image to be processed is less than a first threshold; Displaying, in a second interface, a priori reconstructed image after enhancing the first image to be processed, the clarity of the priori reconstructed image is greater than the clarity of the first image to be processed; wherein the priori reconstructed image is an image obtained by enhancing the first image to be processed based on the priori image, the priori image is an image including the same photographed object as the first image to be processed and having a clarity greater than a second threshold, and the priori reconstructed image includes features of the photographed object contained in the priori image; The process of performing enhancement processing on the first image to be processed based on the prior image to obtain the prior reconstructed image includes: Aligning the prior image with the first image to be processed to obtain a second image to be processed; fusing the first image to be processed and the second image to be processed to obtain the priori reconstructed image; The step of aligning the prior image with the first image to be processed to obtain a second image to be processed includes: Aligning the key points of the prior image with the first image to be processed to obtain a third image to be processed; each pixel in the third image to be processed is obtained by superimposing each pixel in the prior image with the corresponding pixel in the first image to be processed after key point alignment; The third image to be processed is optically aligned with the first image to be processed to obtain a second image to be processed; the second image to be processed is obtained by moving each pixel in the third image to be processed to the position of the pixel in the first image to be processed corresponding to the pixel in the third image to be processed, and the pixel in the first image to be processed corresponding to the pixel in the third image to be processed is the pixel in the first image to be processed corresponding to the maximum value in the dot product calculation results between the pixel in the third image to be processed and each pixel in the first image to be processed.

2. The method according to claim 1, characterized in that The first to-be-processed image and the second to-be-processed image are fused to obtain the a priori reconstructed image, including: The first image to be processed and the second image to be processed are fused based on the attention mechanism to obtain the priori reconstructed image.

3. The method according to claim 1, characterized in that The process of performing enhancement processing on the first image to be processed based on the prior image to obtain the prior reconstructed image includes: Inputting the prior image and the first image to be processed into a priori reconstruction model to obtain the priori reconstructed image; the priori reconstruction model is trained based on a training sample set, and the priori reconstruction model has the function of restoring a low-resolution image to a high-resolution image based on the prior image; The training sample set includes multiple groups of training samples, each group of training samples includes a first image with a clarity less than a third threshold, a first image with a clarity greater than a fourth threshold, and a second image that includes the same photographed object as the first image and has a clarity greater than a fifth threshold.

4. The method according to claim 1, characterized in that: The process of obtaining the prior image includes: Identifying the identity of the person photographed in the first image to be processed; The priori image is determined in a third image based on the identity of the person, and the third image is an image with a clarity greater than a second threshold.

5. The method according to claim 1, characterized in that The first interface is used to display images, and the first interface includes a first image to be processed. The first control is an image enhancement control, and the image enhancement control is used to enhance the first image to be processed based on a priori image to obtain the priori reconstructed image.

6. An electronic device, characterized in that: The electronic device includes a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 5.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 5.

Images