Image processing method, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides an image processing method, electronic equipment, a storage medium and a program product, relates to the technical field of image processing, and can improve the effect of a picture containing a portrait. The image processing method comprises the following steps: acquiring a reference picture corresponding to a portrait picture to be processed; uploading the portrait picture to be processed; receiving a repaired portrait picture corresponding to the to-be-processed portrait picture; and carrying out fusion processing on the repaired portrait picture and the reference picture to obtain a fused portrait picture.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, electronic equipment, storage medium and program product. Background Art

[0002] When taking photos with mobile phones and other electronic devices, if there is a portrait in the scene, in order to increase the clarity of the picture and the portrait, the image will be enhanced and repaired through an algorithm. However, the current portrait enhancement and repair is mainly aimed at portraits taken at close range. For portraits taken at a distance, the effect of portrait enhancement and repair is poor due to the large degree of blurring of the face in the picture, loss of details, artifacts and other problems. Summary of the invention

[0003] In view of this, the present application provides an image processing method, an electronic device, a storage medium and a program product, which can improve the effect of pictures containing portraits.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, comprising: obtaining a reference image corresponding to a portrait image to be processed; uploading the portrait image to be processed;

[0005] A restored portrait image corresponding to the portrait image to be processed is received; and the restored portrait image and the reference image are fused to obtain a fused portrait image.

[0006] The portrait image to be processed is uploaded to the cloud for blind portrait enhancement to obtain a restored portrait image with better clarity and less noise. The restored portrait image is then received from the cloud and fused with the reference image, which improves the clarity of the portrait and the background, reduces pseudo-texture, and can greatly improve the clarity of the distant portrait while maintaining the original facial features as much as possible. Since the reference image is uploaded to the cloud for blind portrait enhancement, the powerful training and computing power of the cloud can be utilized. Since the fusion between the reference image and the restored portrait image is performed locally, the reference image does not need to be uploaded to the cloud, which avoids user privacy issues involving the reference image.

[0007] In a possible implementation, before obtaining the reference image corresponding to the portrait image to be processed and uploading the portrait image to be processed, the method further includes: in response to a photo taking instruction, obtaining the portrait image to be processed taken by the camera; obtaining the reference image corresponding to the portrait image to be processed includes: selecting the reference image corresponding to the portrait image to be processed from the local storage; after obtaining the fused portrait image, the method further includes: storing the fused portrait image. This allows the portrait photo to be enhanced and repaired instantly when the mobile phone takes the portrait photo.

[0008] In a possible implementation, the method further includes: if a reference image corresponding to the portrait image to be processed is stored locally and the portrait image to be processed meets a preset condition, then a process of uploading the portrait image to be processed is performed. The efficiency of image processing can be improved by executing the corresponding process based on whether there is a reference image and whether the preset condition is met.

[0009] In a possible implementation, the locally stored reference image corresponding to the portrait image to be processed includes: the locally stored image has a reference image that is consistent with the identity recognition result of the portrait in the portrait image to be processed; the portrait image to be processed meets the preset condition including: the distance between the person corresponding to the portrait in the portrait image to be processed and the camera exceeds the distance threshold. This can further improve the efficiency of image processing.

[0010] In a possible implementation, the portrait image to be processed satisfies the preset condition further including: the difference in face angle between the portrait in the portrait image to be processed and the portrait in the reference image is less than a face angle threshold; the difference in sight angle between the portrait in the portrait image to be processed and the portrait in the reference image is less than a sight angle threshold; and the clarity of the portrait in the portrait image to be processed is less than a clarity threshold. This can further improve the efficiency of image processing.

[0011] In a possible implementation, before the restored portrait image and the reference image are fused, the method further includes: preprocessing the reference image based on the portrait image to be processed to obtain a preprocessed reference image; fusing the restored portrait image and the reference image includes: fusing the restored portrait image and the preprocessed reference image. Before fusion, the reference image is preprocessed based on the portrait image to be processed so that it can be better fused with the restored portrait image later and the quality of the fused image is improved.

[0012] In a possible implementation, based on the portrait image to be processed, the reference image is preprocessed to obtain the preprocessed reference image, including: based on the portrait image to be processed, the reference image is subjected to deformation registration of the face part and skin color conversion of the face part to obtain the preprocessed reference image. The reference image is spatially and color-registered to improve the subsequent fusion effect.

[0013] In a possible implementation, uploading the portrait image to be processed includes: compressing and encoding the portrait image to be processed, and uploading the compressed and encoded portrait image to be processed; receiving the restored portrait image corresponding to the portrait image to be processed includes: receiving the compressed and encoded restored portrait image corresponding to the portrait image to be processed; decoding the compressed and encoded restored portrait image to obtain the decoded restored portrait image. This can improve the protection effect of user privacy.

[0014] In a second aspect, an electronic device is provided, including: a processor and a memory, the memory being used to store at least one instruction, and when the instruction is loaded and executed by the processor, the electronic device executes the above method.

[0015] According to a third aspect, a computer-readable storage medium is provided, comprising a program or an instruction. When the program or the instruction is run on a computer, the above method is executed.

[0016] According to a fourth aspect, a computer program product is provided. The computer program product comprises executable instructions. When the executable instructions are executed on a computer, the computer is enabled to perform the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 This is a structural block diagram of an electronic device in an embodiment of the present application;

[0019] Figure 2 A software structure block diagram of an electronic device in an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of a system architecture in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a flow chart of an image processing method in an embodiment of the present application;

[0022] Figure 5 Schematic diagram of another image processing method in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0024] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0025] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0026] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0027] Before describing the embodiments of the present application, the related technologies and their technical problems are described first. One portrait enhancement and restoration method is to input the original image into a neural network model, perform blind portrait enhancement and restoration on the image through the model, and obtain the restored image. This method has a good enhancement and restoration effect on portrait images in close-up shooting scenes, but for portrait images in long-distance shooting scenes, blind portrait enhancement and restoration have a poor effect on maintaining facial features. Facial features include eyes, nose, mouth, skin, and other detailed features on the face. In other words, the portrait images in long-distance shooting scenes are significantly different from the real facial features after blind portrait enhancement and restoration. Another portrait enhancement and restoration method is to introduce a relatively high-definition reference image, input the reference image and the portrait image to be processed into a neural network model, fuse the reference image and the portrait image to be processed, and perform portrait enhancement and restoration through the model, but this method requires a large amount of training data for the model, requires a large amount of training data and computing power, and is not suitable for general electronic devices such as mobile phones. In view of the above problems, a technical solution of an embodiment of the present application is provided, and the technical solution of an embodiment of the present application is described below.

[0028] Figure 1 A schematic structural diagram of an electronic device 100 involved in an embodiment of the present application is shown.

[0029] The electronic device 100 may include a processor 110 , an external memory interface 120 , an internal memory 121 , a camera 193 , a display screen 194 , an antenna 1 , an antenna 2 , a mobile communication module 150 , a wireless communication module 160 , and the like.

[0030] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0031] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The processor 110 may be used for functions such as encoding and decoding of audio and video streams.

[0032] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0033] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0034] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0035] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0036] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0037] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device, or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0038] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0039] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0040] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0041] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0042] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0043] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0044] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0045] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0046] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0047] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0048] In some embodiments, the electronic device 100 may set a touch sensor on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor is used to detect a touch operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0049] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0050] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0051] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0052] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0053] The application layer can include a series of application packages.

[0054] like Figure 2 As shown, the application package may include camera, local, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0055] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0056] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, and the like.

[0057] The window manager is used to manage window programs. The content provider is used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0058] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. The display interface can be composed of one or more views.

[0059] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0060] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0061] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0062] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0063] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0064] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0065] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0066] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0067] A 2D graphics engine is a drawing engine for 2D drawings.

[0068] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, Bluetooth driver, audio driver, and sensor driver.

[0069] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0070] When the touch sensor receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into raw input events (including touch coordinates, timestamp of the touch operation, and other information). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input events from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0071] The electronic devices involved in the embodiments of the present application may be smart TVs, mobile phones, tablet computers, personal computers (PCs), personal digital assistants (PDAs), smart watches, wearable electronic devices, augmented reality (AR) devices, virtual reality (VR) devices, vehicle-mounted devices, drone devices, smart cars, smart speakers, robots, smart glasses, and the like.

[0072] The system architecture involved in the embodiment of the present invention is as follows Figure 3 As shown, the system architecture includes an electronic device 100 and a cloud device 200, wherein the electronic device 100 is a local device. The cloud device 200 is implemented by one or more servers, and optionally cooperates with other computing devices, such as data storage, routers, load balancers, etc. The cloud device 200 can be arranged at one physical site, or distributed at multiple physical sites.

[0073] The electronic device 100 can interact with the cloud device 200 through a communication network with any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0074] like Figure 4 As shown, the embodiment of the present application provides an image processing method, which can be applied to the above-mentioned system architecture. The image processing method includes steps applied to the electronic device 100 and steps applied to the cloud device 200. The steps applied to the electronic device 100 include steps 101 to 104, and the execution subject is the electronic device 100. The steps applied to the cloud device 200 include steps 201 to 203, and the execution subject is the cloud device 200. The image processing method includes:

[0075] The electronic device 100 executes step 101, obtaining a reference image corresponding to the portrait image to be processed;

[0076] Among them, the reference image can be a picture related to the portrait image to be processed. For example, the identity of the portrait in the reference image is the same as that of the portrait in the portrait image to be processed, that is, the portrait in the reference image and the portrait in the portrait image to be processed can have the same facial features.

[0077] The electronic device 100 executes step 102 and uploads the portrait image to be processed to the cloud device 200. It should be noted that the order between step 101 and step 102 is not limited in the embodiment of the present application;

[0078] The cloud device executes step 201 and receives the portrait image to be processed;

[0079] The cloud device executes step 202, performs blind image enhancement and restoration on the portrait image to be processed, and obtains a restored portrait image;

[0080] In step 202, for example, an image-to-image network such as U-Net can be used for enhancement processing to obtain a restored portrait image with better clarity and less noise. Blind portrait enhancement restoration may include, for example, deblurring, denoising, super-resolution and other processing processes, the purpose of which is to improve the clarity of the overall image. For the face area, the restored portrait image still shows problems such as pseudo texture, noise and the generated face is not similar.

[0081] The cloud device executes step 203 and sends the restored portrait image to the intelligent electronic device 100;

[0082] The electronic device 100 executes step 103, receiving a restored portrait image corresponding to the portrait image to be processed;

[0083] The electronic device 100 executes step 104 to fuse the restored portrait image and the reference image to obtain a fused portrait image.

[0084] Specifically, in step 104, the electronic device 100 can fuse the restored portrait image and the reference image through a fusion network. The fusion network can use two downsampling modules to extract features of various scales from the restored portrait image and the reference image. The fusion network can use an upsampling module to fuse the extracted features to obtain a fused portrait image. It should be noted that the number of downsampling modules and upsampling modules in the fusion network is only for example, and the embodiment of the present application does not limit this. It can be set according to the needs of the scene. Any one of the downsampling module and the upsampling module can use a convolutional neural network (CNN) structure or a transformer structure. In the fusion process, the background and the posture of the person in the restored portrait image will be retained. The detailed features of the portrait tend to use the details of the person in the reference image. The detailed features of the portrait are mainly facial features, including eyes, nose, mouth, skin and other detailed features on the face. In this way, the fused portrait image obtained by the blind image enhancement processing in the cloud and the local fusion can have better clarity while maintaining the correct facial feature attributes of the person.

[0085] The image processing method in the embodiment of the present application uploads the portrait image to be processed to the cloud for blind portrait enhancement to obtain a restored portrait image with better clarity and less noise, then receives the restored portrait image from the cloud and fuses the restored portrait image with the reference image, thereby improving the clarity of the portrait and the background, reducing pseudo-textures, and greatly improving the clarity of the distant portrait while maintaining the facial features of the original person as much as possible. Since the reference image is uploaded to the cloud for blind portrait enhancement processing, the powerful training and computing power of the cloud can be utilized, and since the fusion between the reference image and the restored portrait image is performed locally, the reference image does not need to be uploaded to the cloud, thus avoiding user privacy issues involving the reference image.

[0086] In some embodiments, Figure 5 As shown, before the electronic device 100 executes step 101, obtains the reference image corresponding to the portrait picture to be processed, and step 102, uploads the portrait picture to be processed, the electronic device 100 also includes: the electronic device 100 executes step 105, responds to the photo command, and obtains the portrait picture to be processed taken by the camera; at this time, the reference image corresponding to the portrait picture to be processed in step 101 includes: selecting the reference image corresponding to the portrait picture to be processed from the local storage; after the fused portrait picture is obtained in step 104, the electronic device 100 also includes: the electronic device 100 executes step 106, and stores the fused portrait picture. For example, the electronic device 100 is a mobile phone. After the mobile phone camera function is called and executed, the subsequent image processing process can be executed. The mobile phone can search for the corresponding reference image from the local gallery of the mobile phone according to the portrait picture to be processed obtained by the shooting, so as to facilitate the subsequent fusion of the reference image and the portrait picture repaired in the cloud to obtain the fused portrait picture, and then save the fused portrait picture in the local gallery of the mobile phone. This is to achieve the enhancement and repair of the portrait photo when the mobile phone takes the portrait photo. Local refers to the electronic device 100 itself.

[0087] In some embodiments, the above method further includes: if a reference image corresponding to the portrait image to be processed is stored locally and the portrait image to be processed meets a preset condition, then executing a process of uploading the portrait image to be processed.

[0088] Specifically, Figure 5As shown, after executing step 105, the electronic device 100 executes step 107 to determine whether a reference image corresponding to the portrait image to be processed is stored locally. If so, step 101 is executed. That is to say, if the electronic device 100 stores the corresponding reference image locally, the blind portrait enhancement and repair can be performed through the cloud, and the image can be fused with the reference image locally, so the subsequent process continues. In step 107, if it is determined to be no, step 108 is executed to store the portrait image to be processed. That is to say, if the electronic device 100 does not store the corresponding reference image locally, image fusion cannot be performed, so the image can be directly output based on the captured portrait image. It should be noted that in the image output process, that is, before step 108, although the cloud repair and local fusion process are no longer performed, some necessary image processing processes can still be performed before storage. After step 101, step 109 is executed to determine whether the portrait image to be processed meets the preset conditions. If so, step 102 is executed, that is, the portrait image to be processed is uploaded. If not, step 108 is executed. That is to say, in addition to determining whether to perform the cloud-based repair and local fusion process based on whether a reference image is stored locally, it is also possible to determine whether to perform the cloud-based repair and local fusion process based on whether the portrait image to be processed meets the requirements. If the preset conditions are met, it means that the portrait image to be processed is suitable for portrait enhancement repair through the cloud-based repair and local fusion process. If the preset conditions are not met, it means that the portrait image to be processed is not suitable for portrait enhancement repair through the cloud-based repair and local fusion process. The efficiency of image processing can be improved by executing the corresponding process based on whether there is a reference image and whether the preset conditions are met.

[0089] In some embodiments, a reference image corresponding to a portrait picture to be processed is stored locally, including: a reference image in the locally stored picture having an identity recognition result consistent with that of a portrait in the portrait picture to be processed; the portrait picture to be processed meets a preset condition, including: the distance between the person corresponding to the portrait in the portrait picture to be processed and the camera exceeds a distance threshold.

[0090] Specifically, being consistent with the identity recognition result of the portrait in the portrait picture to be processed means that the portrait in the portrait picture to be processed and the portrait in the reference picture belong to the same person. In this case, the subsequent fusion process is meaningful. Therefore, only when it is determined that there is a person who is the same person as the person photographed in the current shooting scene in the picture stored by the electronic device 100, the picture will be used as a reference picture, and the subsequent cloud repair and local fusion process will be performed. If there is no portrait in the picture stored by the electronic device 100 that is the same person as the person photographed in the current shooting scene, the captured photo cannot be repaired by fusion, so the image can be directly output based on the current photo. If the distance between the portrait in the captured photo and the camera is close, it is not necessary to use the cloud repair and local fusion methods in the embodiment of the present application to perform portrait enhancement and repair. Therefore, in the embodiment of the present application, the cloud repair and local fusion methods are only applied when it is determined that the person photographed in the shooting scene is far away from the camera. The embodiment of the present application does not limit the method for determining the distance between the person corresponding to the portrait in the portrait picture to be processed and the camera. For example, the distance between the person corresponding to the portrait and the camera can be determined based on the analysis of the portrait picture to be processed, or the distance between the person corresponding to the portrait in the portrait picture to be processed and the camera can be determined by a distance sensor on the electronic device 100. The image processing efficiency can be further improved.

[0091] In some embodiments, the portrait picture to be processed meets the preset conditions and also includes: the difference in face angle between the portrait in the portrait picture to be processed and the portrait in the reference picture is less than a face angle threshold; the difference in sight angle between the portrait in the portrait picture to be processed and the portrait in the reference picture is less than a sight angle threshold; the clarity of the portrait in the portrait picture to be processed is less than a clarity threshold.

[0092] Specifically, the smaller the difference between the reference image and the portrait in the portrait image to be processed, the better the subsequent fusion effect. Therefore, it can be determined whether to perform subsequent cloud-based repair and local fusion based on the facial angle difference and sight angle difference between the portrait in the portrait image to be processed and the portrait in the reference image to ensure the portrait repair effect. In addition, if the clarity of the portrait in the portrait image to be processed is high, there is no need for cloud-based repair and local fusion. If any of the distance, facial angle difference, sight angle difference and clarity involved in the above-mentioned portrait image to be processed does not meet the corresponding threshold, it is determined that the portrait image to be processed does not meet the preset conditions. The efficiency of image processing can be further improved.

[0093] In some embodiments, before the restored portrait image and the reference image are fused, the process further includes: the electronic device 100 executes step 110, pre-processes the reference image based on the portrait image to be processed, and obtains a pre-processed reference image; at this time, the above step 104, fusion processing of the restored portrait image and the reference image includes: fusion processing of the restored portrait image and the pre-processed reference image. The reference image is pre-processed based on the portrait image to be processed before fusion, so that it can be better fused with the restored portrait image later, thereby improving the quality of the fused image.

[0094] In some embodiments, the above-mentioned step 109, based on the portrait picture to be processed, preprocessing the reference image to obtain the preprocessed reference image, includes: based on the portrait picture to be processed, performing deformation alignment of the facial part and skin color conversion of the facial part on the reference image to obtain the preprocessed reference image.

[0095] Specifically, there may be differences in facial shape or skin color between the portrait in the reference image and the portrait in the portrait image to be processed. Therefore, in order to achieve better subsequent fusion effect, the reference image is spatially and color-aligned during the preprocessing process to improve the subsequent fusion effect.

[0096] In some embodiments, Figure 5 As shown, the electronic device 100 executes the above step 102 and uploads the portrait picture to be processed, including: the electronic device 100 executes step 1021, compresses and encodes the portrait picture to be processed; and executes step 1022, uploading the compressed and encoded portrait picture to be processed. The cloud device 200 executes step 201, including: the cloud device 200 executes step 2011, receives the compressed and encoded portrait picture to be processed; step 2012, decodes the compressed and encoded human image to be processed. Step 202 includes: performing blind portrait enhancement and restoration on the decoded portrait picture to be processed to obtain a restored portrait picture. Step 203 includes: step 2031, compresses and encodes the restored portrait picture; step 2032, sends the compressed and encoded restored portrait picture to the electronic device 100. The electronic device 100 executes the above-mentioned step 103 and receives the restored portrait picture corresponding to the portrait picture to be processed, which includes: the electronic device 100 executes step 1031 and receives the compressed and encoded restored portrait picture corresponding to the portrait picture to be processed; step 1032, decodes the compressed and encoded restored portrait picture to obtain the decoded restored portrait picture.

[0097] Specifically, before the electronic device 100 uploads the portrait picture to be processed, it is compressed and encoded, that is, the user information is compressed and encoded to protect the user information. After the cloud device 200 receives the portrait picture to be processed, it will be decoded, and then blind portrait enhancement and restoration will be performed based on the decoded portrait picture to be processed. The restored portrait picture is also compressed and encoded before being sent to the electronic device 100 to protect the user information. After the electronic device 100 receives the restored portrait picture, it is first decoded, and then the decoded restored portrait picture and the reference picture are fused to obtain the fused portrait picture. The protection effect of user privacy can be improved.

[0098] Several specific application scenarios are used as examples for illustration. For example, the portrait image to be processed currently captured contains the portrait of user A. In step 101, a picture containing the portrait of user A can be selected from the gallery as a reference picture. In the subsequent fusion process, the portrait features of user A in the portrait image to be processed can be repaired based on the portrait features of user A in the reference picture. For another example, the portrait image to be processed currently captured contains the portrait of user A. In step 101, multiple pictures containing the portrait of user A can be selected from the gallery as reference pictures. In the subsequent fusion process, multiple pictures containing the portrait of user A can be used as reference pictures to be fused with the portrait image to be processed. That is, the portrait features of user A in the portrait image to be processed can be repaired based on the portrait features of user A in multiple reference pictures. For another example, the portrait picture to be processed currently captured includes the portrait of user A and the portrait of user B. In step 101, a picture including the portrait of user A and a picture including the portrait of user B can be selected from the gallery as two reference pictures. In the subsequent fusion process, the two reference pictures are fused with the portrait picture to be processed, and the portrait features of user A in the portrait picture to be processed are repaired based on the portrait features of user A in the first reference picture, and the portrait features of user B in the portrait picture to be processed are repaired based on the portrait features of user B in the second reference picture. For another example, the portrait picture to be processed currently captured includes the portrait of user A and the portrait of user B. In step 101, a picture including the portrait of user A is selected from the gallery, but a picture including the portrait of user B is not found. In the subsequent fusion process, the reference picture is fused with the portrait picture to be processed, and the portrait features of user A in the portrait picture to be processed are repaired based on the portrait features of user A in the reference picture, and the portrait of user B in the portrait picture to be processed may not be fused.

[0099] The embodiment of the present application also provides an electronic device, including: a processor and a memory, the memory is used to store at least one instruction, and when the instruction is loaded and executed by the processor, the electronic device executes the method of any of the above embodiments. The specific process and principle of the method are the same as those of the above embodiments, and will not be repeated here. The electronic device can specifically be the above electronic device 100.

[0100] The electronic devices involved in this application may be smart TVs, mobile phones, tablet computers, personal computers (PCs), personal digital assistants (PDAs), smart watches, wearable electronic devices, augmented reality (AR) devices, virtual reality (VR) devices, vehicle-mounted devices, drone devices, smart cars, smart speakers, robots, smart glasses, and any other products.

[0101] An embodiment of the present application further provides a computer-readable storage medium, including a program or an instruction. When the program or the instruction runs on a computer, the method in any of the above embodiments is executed.

[0102] The embodiments of the present application also provide a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes the method in any of the above embodiments.

[0103] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0104] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0105] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, It is characterized in that include: Obtain a reference image corresponding to the portrait image to be processed; Upload the portrait image to be processed; Receiving a restored portrait image corresponding to the portrait image to be processed; The restored portrait image and the reference image are fused to obtain a fused portrait image.

2. The method according to claim 1, It is characterized in that Before obtaining the reference image corresponding to the portrait image to be processed and uploading the portrait image to be processed, the method further includes: in response to a photo taking instruction, obtaining the portrait image to be processed taken by a camera; The obtaining of the reference image corresponding to the portrait image to be processed comprises: selecting the reference image corresponding to the portrait image to be processed from local storage; After obtaining the fused portrait picture, the method further includes: storing the fused portrait picture.

3. The method according to claim 2, It is characterized in that Also includes: If a reference image corresponding to the portrait image to be processed is stored locally and the portrait image to be processed meets a preset condition, the process of uploading the portrait image to be processed is performed.

4. The method according to claim 3, It is characterized in that The reference image corresponding to the portrait image to be processed is stored locally and includes: The locally stored picture has a reference picture that is consistent with the identity recognition result of the portrait in the portrait picture to be processed; The portrait image to be processed meets the preset conditions including: The distance between the person corresponding to the portrait in the portrait image to be processed and the camera exceeds a distance threshold.

5. The method according to claim 4, It is characterized in that The portrait image to be processed meets the preset conditions and further includes: The difference in face angle between the portrait in the portrait image to be processed and the portrait in the reference image is less than a face angle threshold; The difference in sight angle between the portrait in the portrait image to be processed and the portrait in the reference image is less than a sight angle threshold; The clarity of the portrait in the portrait picture to be processed is less than a clarity threshold.

6. The method according to any one of claims 1 to 5, It is characterized in that Before fusing the restored portrait image with the reference image, the method further includes: Based on the portrait image to be processed, preprocessing the reference image to obtain a preprocessed reference image; The fusing process of the restored portrait image and the reference image comprises: The restored portrait image and the pre-processed reference image are fused.

7. The method according to claim 6, It is characterized in that The process of preprocessing the reference image based on the portrait image to be processed to obtain the preprocessed reference image includes: Based on the portrait image to be processed, deformation registration of the face part and skin color conversion of the face part are performed on the reference image to obtain a preprocessed reference image.

8. The method according to any one of claims 1 to 7, It is characterized in that The uploading of the portrait image to be processed comprises: Compressing and encoding the portrait image to be processed, and uploading the compressed and encoded portrait image to be processed; The receiving the restored portrait picture corresponding to the portrait picture to be processed comprises: Receiving a restored portrait picture after compression encoding corresponding to the portrait picture to be processed; The compressed and encoded restored portrait picture is decoded to obtain a decoded restored portrait picture.

9. An electronic device, It is characterized in that include: A processor and a memory, wherein the memory is used to store at least one instruction, and when the instruction is loaded and executed by the processor, the electronic device executes the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium, It is characterized in that The method comprises a program or an instruction. When the program or the instruction is run on a computer, the method according to any one of claims 1 to 8 is executed.

11. A computer program product, It is characterized in that The computer program product comprises executable instructions, and when the executable instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 8.

Images