Method executed by electronic device, electronic device, storage medium and program product

By obtaining image recovery style and recovery degree information, and using AI network to perform image recovery processing, the problem of image quality improvement in the prior art is solved, and a personalized and efficient image recovery effect is achieved, which is suitable for image processing of electronic devices.

CN119090774APending Publication Date: 2024-12-06BEIJING SAMSUNG TELECOM R&D CENT +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310629198.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to effectively improve the image quality presented by electronic devices, especially in the recovery and repair of low-quality images, and lacks personalized and efficient solutions.

Method used

By acquiring images, determining their recovery style and recovery degree information, using AI network to perform image recovery processing, combining the recovery degree instructions and image quality information input by users, personalized image recovery is achieved.

Benefits of technology

It significantly improves image quality, enhances the texture detail and clarity of the image, can be personalized according to user preferences and image attributes, and is suitable for the recovery force of different regions to generate natural and realistic images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090774A_ABST
    Figure CN119090774A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a method executed by electronic equipment, the electronic equipment, a storage medium and a program product, and relates to the fields of computer vision, image processing, artificial intelligence and the like, and meanwhile, the method executed by the electronic equipment can be executed by using an artificial intelligence model. The method comprises the steps of obtaining a first image; determining recovery style information of the first image; and based on the recovery style information, using an AI network to perform recovery processing on the first image to obtain a recovered image, thereby achieving the purpose of enhancing the image quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology. Specifically, the present application relates to a method executed by an electronic device, an electronic device, a storage medium, and a program product. Background Art

[0002] With the increasing popularity and replacement frequency of electronic devices such as mobile phones, users have higher and higher demands for the image quality presented by electronic devices. How to improve image quality has become a continuous goal in this field. Summary of the invention

[0003] The purpose of the embodiments of the present application is to solve the problem of how to improve image quality.

[0004] According to one aspect of an embodiment of the present application, a method performed by an electronic device is provided, the method comprising:

[0005] acquiring a first image;

[0006] Determining restored style information of the first image;

[0007] Based on the restored style information, the first image is restored using an AI network to obtain a restored image.

[0008] In an optional implementation, the restored style information is related to the attribute information of the target object in the first image.

[0009] In an optional embodiment, the method further includes:

[0010] Determining restoration degree information of the first image;

[0011] Based on the restored style information, the first image is restored using the AI ​​network, including:

[0012] Based on the restoration style information and the restoration degree information, the first image is restored using an AI network.

[0013] In an optional implementation, the restoration degree information is related to at least one of the following: image quality of the first image; and a restoration degree related instruction input by a user.

[0014] In an optional implementation, determining restoration degree information of the first image includes:

[0015] Providing a restoration degree confirmation interface for the first image;

[0016] In response to a restoration degree related instruction input by a user through a restoration degree confirmation interface, second restoration degree information of the first image is determined.

[0017] In an optional implementation, providing a restoration degree confirmation interface for the first image includes:

[0018] determining image quality information of the first image;

[0019] Based on the image quality information, determining corresponding first restoration degree information;

[0020] Based on the first restoration degree information, providing a restoration degree confirmation interface for the first image;

[0021] In response to a restoration degree related instruction input by a user through a restoration degree confirmation interface, determining second restoration degree information of the first image includes:

[0022] In response to an adjustment instruction for the first restoration degree information input by a user through the restoration degree confirmation interface, second restoration degree information of the first image is determined.

[0023] In an optional implementation, determining restoration degree information of the first image includes:

[0024] determining image quality information of the first image;

[0025] Based on the image quality information, corresponding first restoration degree information is determined.

[0026] In an optional embodiment, the recovery degree confirmation interface includes at least one of the following: a global recovery degree confirmation interface and at least one local recovery degree confirmation interface, wherein the at least one local recovery degree confirmation interface corresponds to at least one local area of ​​the target object.

[0027] In an optional implementation, in response to a restoration degree related instruction input by a user through a restoration degree confirmation interface, determining the second restoration degree information of the first image includes:

[0028] The second recovery degree information of the first image is determined in response to at least one of the following instructions: a global recovery degree related instruction input by a user through a global recovery degree confirmation interface, and at least one local recovery degree related instruction input by a user through at least one local recovery degree confirmation interface.

[0029] In an optional implementation, determining corresponding first restoration degree information based on the image quality information includes:

[0030] Based on the image quality information, determining first global restoration degree information;

[0031] Determine first local restoration degree information corresponding to the at least one local area based on the image quality information and the image mask of the at least one local area;

[0032] Based on the first restoration degree information, providing a restoration degree confirmation interface for the first image includes:

[0033] Based on the first global recovery degree information, providing a global recovery degree confirmation interface in the recovery degree confirmation interface;

[0034] Based on the first local recovery degree information respectively corresponding to the at least one local area, respectively providing at least one corresponding local recovery degree confirmation interface in the recovery degree confirmation interface;

[0035] In response to an adjustment instruction for the first restoration degree information input by a user through a restoration degree confirmation interface, determining the second restoration degree information of the first image includes:

[0036] The second recovery degree information of the first image is determined in response to at least one of the following instructions: an adjustment instruction for the first global recovery degree information input by the user through the global recovery degree confirmation interface, and an adjustment instruction for at least one first local recovery degree information input by the user through at least one local recovery degree confirmation interface.

[0037] In an optional implementation, in response to a restoration degree related instruction input by a user through a restoration degree confirmation interface, determining the second restoration degree information of the first image includes:

[0038] In response to a global recovery degree related instruction input by a user through a global recovery degree confirmation interface, determining second global recovery degree information;

[0039] In response to at least one local recovery degree related instruction input by a user through at least one local recovery degree confirmation interface, determining corresponding at least one second local recovery degree information;

[0040] The second global recovery degree information and at least one second local recovery degree information are merged to obtain the second recovery degree information.

[0041] In an optional implementation manner, determining the image quality information of the first image includes:

[0042] Acquiring depth information and / or brightness information of the first image;

[0043] Image quality information of the first image is determined based on the brightness information and / or the depth information.

[0044] In an optional implementation, determining the image quality information of the first image based on the brightness information and / or the depth information includes:

[0045] Obtaining a first image feature based on the brightness information and / or the depth information;

[0046] Extracting features from the first image to obtain features of the second image;

[0047] Image quality information of the first image is determined based on the first image characteristic and the second image characteristic.

[0048] In an optional implementation, obtaining the first image feature based on the brightness information and the depth information includes:

[0049] Feature extraction is performed on brightness information and depth information respectively to obtain brightness feature and depth feature;

[0050] Based on the brightness feature, determining a first weight map for compensating the depth feature;

[0051] Performing weighted processing on the depth feature according to the first weight map to obtain compensated depth feature;

[0052] The brightness feature and the compensated depth feature are fused to obtain the first image feature.

[0053] In an optional implementation, determining the image quality information of the first image based on the first image feature and the second image feature includes:

[0054] Based on the first image feature, determining a second weight map, the second weight map indicating a correlation between the image quality degradation degree of the first image and the depth information and / or the brightness information;

[0055] Processing the second image feature through a multi-head self-attention network to obtain a query feature, a key feature, and a value feature of the second image feature;

[0056] weighting the value features according to a second weight map;

[0057] Image quality information of the first image is determined based on the weighted value feature, the query feature, and the key feature.

[0058] In an optional embodiment, the method further includes:

[0059] Repeat the following update process at least once:

[0060] Determine a peak signal-to-noise ratio value of the first image according to the currently obtained restored image;

[0061] Determine the recovery degree information update parameter according to the peak signal-to-noise ratio value;

[0062] Based on the recovery degree information update parameter, the recovery degree information is updated;

[0063] Based on the updated restoration degree information, the first image is restored to obtain an updated restored image.

[0064] In an optional implementation, determining the restoration style information of the first image includes:

[0065] Determine attribute information of a target object in the first image according to the first image;

[0066] Based on the attribute information, restoration style information of the first image is determined.

[0067] In an optional implementation, determining the restoration style information of the first image based on the attribute information includes:

[0068] Encode attribute information into attribute vectors;

[0069] Through a multi-layer perception network, the attribute vector is mapped to restore the style information.

[0070] In an optional implementation, based on the restoration style information, the first image is restored using an AI network, including:

[0071] Based on the first image, extracting corresponding third image features;

[0072] extracting, from the third image feature, a fifth image feature corresponding to the attribute information of the target object in the first image based on the restored style information;

[0073] A restored image is obtained based on the third image feature and the fifth image feature.

[0074] In an optional implementation, extracting corresponding third image features based on the first image includes:

[0075] Based on the first image and the restoration degree information of the first image, corresponding third image features are extracted.

[0076] In an optional implementation, based on the restoration style information, extracting a fifth image feature corresponding to the attribute information of the target object in the first image from the third image feature includes:

[0077] The restored style information is integrated with the attribute information to obtain a fourth image feature;

[0078] Based on the fourth image feature and the attribute information, a fifth image feature is obtained.

[0079] In an optional implementation, obtaining a restored image based on the third image feature and the fifth image feature includes:

[0080] determining an image feature correction amount corresponding to the attribute information according to a correlation between the third image feature and the fifth image feature;

[0081] The third image feature is corrected according to the image feature correction amount to obtain a restored image.

[0082] In an optional implementation, the third image feature includes at least one scale, and determining the image feature correction amount corresponding to the attribute information according to the correlation between the third image feature and the fifth image feature includes:

[0083] For each scale of the third image feature, according to the correlation between the third image feature of the scale and the fifth image feature of the corresponding scale, determine the image feature correction amount of the scale corresponding to the attribute information;

[0084] Correcting the third image feature according to the image feature correction amount to obtain a restored image includes:

[0085] Correction processing is performed on the corresponding third image features according to the image feature correction amount of at least one scale to obtain a restored image.

[0086] In an optional implementation, acquiring a first image includes:

[0087] acquiring a second image;

[0088] detecting a target object in a second image;

[0089] The first image is determined based on at least one of the following:

[0090] The number of detected target objects;

[0091] Image quality of the image area corresponding to the detected target object;

[0092] The target object selection instruction input by the user.

[0093] In an optional implementation, acquiring the second image includes at least one of the following methods:

[0094] Acquire a second image based on the image acquired in real time by the acquisition device;

[0095] A second image is acquired based on an image stored in the electronic device.

[0096] In an optional implementation, determining the first image includes:

[0097] When the number of detected target objects is not less than a first set threshold, a first image is determined in the image area corresponding to the target object based on at least one of image quality of the image area corresponding to the detected target object and a target object selection instruction input by a user.

[0098] In an optional implementation, determining a first image in the image region corresponding to the target object based on the image quality of the detected image region corresponding to the target object includes:

[0099] determining an image quality of an image region corresponding to at least one target object;

[0100] An image region whose image quality related value is lower than a second set threshold is determined as a first image.

[0101] In an optional implementation, if the second image is an image acquired in real time based on an acquisition device, the method further includes:

[0102] When the number of detected target objects is not less than a third set threshold, the second image is stored.

[0103] In an optional embodiment, the method further includes:

[0104] determining a corresponding third image based on the restored image and the second image;

[0105] The third image is stored or displayed.

[0106] In an optional implementation, the first image is a facial image.

[0107] In an optional implementation, the attribute information includes at least one of the following: age information, gender information, skin color information, hair information, makeup information, and glasses information.

[0108] According to another aspect of an embodiment of the present application, an electronic device is provided, which includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the method provided in the embodiment of the present application.

[0109] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0110] According to another aspect of an embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0111] The method, electronic device, storage medium and program product executed by the electronic device provided in the embodiments of the present application can achieve the purpose of enhancing image quality by acquiring a first image; determining restoration style information of the first image; and restoring the first image based on the restoration style information using an AI network to obtain a restored image. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0113] Figure 1 A flowchart of a method performed by an electronic device provided in an embodiment of the present application;

[0114] Figure 2 A schematic diagram of the relationship between character attributes and restoration direction provided in an embodiment of the present application;

[0115] Figure 3 A schematic diagram of the relationship between image quality and restoration strength provided in an embodiment of the present application;

[0116] Figure 4 A schematic diagram of guiding a neural network to restore an image using appropriate restoration force and in a correct restoration direction provided by an embodiment of the present application;

[0117] Figure 5a A schematic diagram of a method for determining image quality information provided by an embodiment of the present application;

[0118] Figure 5b A schematic diagram of another method for determining image quality information provided by an embodiment of the present application;

[0119] Figure 6 A schematic diagram of a user controlling the degree of recovery according to preferences provided in an embodiment of the present application;

[0120] Figure 7 A schematic diagram of a method for determining restoration style information provided in an embodiment of the present application;

[0121] Figure 8 A schematic diagram of a personalized image restoration method provided in an embodiment of the present application;

[0122] Fig. 9 A schematic diagram of an attribute feature enhancement module provided in an embodiment of the present application;

[0123] Fig.10 A schematic diagram of attribute feature enhancement data provided in an embodiment of the present application;

[0124] Fig.11 A schematic diagram of a correction amount weight map provided in an embodiment of the present application;

[0125] Fig.12 A schematic diagram of a correction module provided in an embodiment of the present application;

[0126] Fig.13a A schematic diagram of a personalized face restoration method provided in an embodiment of the present application;

[0127] Fig.13b A schematic diagram of another personalized face restoration method provided in an embodiment of the present application;

[0128] Fig.14 A schematic diagram of the law of changes in visually perceived image quality provided by an embodiment of the present application;

[0129] Fig.15 A schematic diagram of a relationship mapping between a PSNR value and a user's subjective feeling provided in an embodiment of the present application;

[0130] Fig.16 A schematic diagram of a method for optimizing recovery degree information provided in an embodiment of the present application;

[0131] Fig.17 A schematic diagram of a personalized face redrawing method provided in an embodiment of the present application;

[0132] Fig.18 A schematic diagram of another personalized face redrawing method provided in an embodiment of the present application;

[0133] Fig.19 A schematic diagram of another personalized face redrawing method provided in an embodiment of the present application;

[0134] Fig. 20 A schematic diagram of a typical camera pipeline of an AI camera provided in an embodiment of the present application;

[0135] Fig.21 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0136] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present application as defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should be considered as exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present application. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0137] The terms and expressions used in the following specification and claims are not limited to their dictionary meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present application. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present application is provided for illustration purposes only and not to limit the purpose of the present application as defined in the appended claims and their equivalents.

[0138] It should be understood that the singular forms "a", "an", and "the" may also include plural references unless the context clearly indicates otherwise. Thus, for example, reference to a "component surface" includes reference to one or more such surfaces. When we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or the one element and the other element may be connected through an intermediate element. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling.

[0139] The term "include" or "may include" refers to the presence of corresponding disclosed functions, operations, or components that can be used in various embodiments of the present application, rather than limiting the presence of one or more additional functions, operations, or features. In addition, the term "include" or "have" may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0140] The term "or" used in various embodiments of the present application includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple, or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" may be implemented as parameter A including A1 or A2 or A3, and may also be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0141] Unless defined differently, all terms (including technical terms or scientific terms) used in this application have the same meanings understood by those skilled in the art of this application. Common terms as defined in dictionaries are interpreted as having meanings consistent with the context in the relevant technical field, and should not be interpreted ideally or overly formally, unless clearly defined in this application.

[0142] At least some functions of the device or electronic device provided in the embodiments of the present application can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory and processor.

[0143] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0144] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0145] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0146] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0147] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0148] According to the present application, in a method performed in an electronic device, a method for image restoration can obtain output data of a restored image or image quality in an image, attributes of a target object, etc. by using image data as input data of an artificial intelligence model. The artificial intelligence model can be obtained by training. Here, "obtained by training" means obtaining a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm.

[0149] The method of the present application may relate to the field of visual understanding of artificial intelligence technology, which is a technology used to recognize and process things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / positioning or image enhancement.

[0150] The following describes several optional embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0151] In an embodiment of the present application, a method executed by an electronic device is provided, such as Figure 1 As shown, the method includes:

[0152] Step S101: acquiring a first image;

[0153] For the embodiment of the present application, the source of the first image is not specifically limited. As an example, the first image may be derived from a stored image, such as an image selected in an album, etc.; or, the first image may be derived from an image to be stored, such as an image taken in real time using a camera, an image downloaded from the network, an image received from another device, etc. Optionally, the first image may directly use a stored image or an image to be stored, or it may be that the stored image or the image to be stored is processed to a certain extent and then used as a to-be-processed image for subsequent steps S102 to S103, which is not limited in the embodiment of the present application. As an example, if a stored image or an image to be stored contains multiple target objects, the multiple target objects may be divided, and the image where each target object is located may be used as a first image. As an example, for each image where the target object is located, it may be screened to determine whether it can be used as a first image for subsequent steps S102 to S103.

[0154] For the embodiments of the present application, the type of the first image is not specifically limited. For different application scenarios, the type of the first image may be different. As an example, the first image may be a facial image (such as a human face image, an animal face image), a human image, an animal image, or an image of another object. For ease of description, the following may be introduced using a human face image as an example.

[0155] Step S102: determining restoration style information of the first image;

[0156] In an embodiment of the present application, for an input low-quality first image, a restoration style information (for ease of description, it may be referred to as restoration style, or style, or style information, or style direction hereinafter) is calculated. The restoration style information may correspond to the characteristics of the first image and be used to personalize and guide subsequent steps to use an appropriate restoration method to restore the image.

[0157] Optionally, the restored style information may be related to the attribute information of the target object in the first image. Among them, the restored style information refers to the texture detail features consistent with the given attributes. Simply put, the restored style information refers to what texture or what type of texture features to add in different areas of the input low-quality first image. The purpose of restoring style information prediction is to predict various attributes of the target object, and map the predicted one or more attributes into the restored style information as the restoration direction (the direction consistent with the attribute) to guide the neural network to restore the texture details of different areas. Taking the first image as a face image as an example, the target object is a face. For a low-quality face image, reasonable and real textures can be added to the area where the texture details are missing. The restored style information predicts various attributes of the face, and maps the predicted one or more attributes into the restored style information. Therefore, the restored style information can guide the neural network to add reasonable and real textures to the missing areas of the face image. In practical applications, those skilled in the art can set the attribute type corresponding to the target object according to actual conditions, and the embodiments of the present application are not limited here. For example, if the target object is a human face, the available attribute information includes but is not limited to at least one of the following: age information, gender information, skin color information, hair information (such as hair color, hair curl), makeup information (such as whether makeup is worn), glasses information (such as whether glasses are worn, or the type of glasses worn), etc. In one example, male eyebrows are wide and thick, female eyebrows are narrow and thin, the elderly have wrinkles on their skin, and children have smooth skin. In another example, Figure 2 As shown in the figure, for an input face image, textures of corresponding directions can be added to the input face image according to the styles of young people, young women, middle-aged women, old women, old people, old men, middle-aged men, young men, etc. The added texture detail features are consistent with the given attributes, so that high-quality face images of different styles may be output. Among them, according to the attributes of the face in the input image, a style direction can usually be selected for restoration, that is, Figure 2 In the example, the attribute of the face is a middle-aged male, and based on the predicted restored style information, a high-quality face image of the style direction can be output. It can be understood that only two attributes are considered in this example: age and gender. In actual scenarios, more attributes can be considered. Since the embodiment of the present application improves the quality of the image based on the consideration of the image style, the omitted details can also be restored.

[0158] For the embodiments of the present application, some implementations of this step may use an AI (Artificial Intelligence) network. As an example, the restored style information of the first image is determined by a first neural network. In practical applications, those skilled in the art may set the specific neural network used in this step according to actual conditions, and the embodiments of the present application are not limited here.

[0159] Step S103: Based on the restoration style information, the AI ​​network is used to restore the first image to obtain a restored image.

[0160] Among them, image restoration is an important branch of image quality enhancement technology. Taking the first image as a face image as an example, face restoration is a technology specifically for improving the image quality of the face area, which can also be called face redrawing or face hallucination.

[0161] The embodiment of the present application aims to restore (also referred to as restore) an input low-quality first image into a high-quality image, and make the restored texture details correspond to the first image. Low quality refers to the degradation of image quality, which may include but is not limited to noise pollution, image blur, loss of texture details, artifacts and other degradation conditions or a combination of degradation conditions, such as photos taken with a smartphone in low light. In contrast, high quality means that the image has rich texture details and is clean and clear. That is, in the embodiment of the present application, the restoration process includes but is not limited to the restoration and generation of details (for example, interpolating finer textures in places with blurred textures, generating semantically reasonable, real and natural textures in places without textures, etc.), reducing image noise, eliminating blur to make the image clear, and so on.

[0162] For the embodiments of the present application, some implementations of this step may also use an AI network. As an example, based on the restoration information, the first image is restored through a second neural network to obtain a restored image. Among them, the restoration style information can guide the second AI network to restore the image in the correct restoration direction, and finally the second AI network can output a high-quality face image. In practical applications, those skilled in the art can set the specific neural network used in this step according to actual conditions, and the embodiments of the present application are not limited here.

[0163] The solution provided in the embodiment of the present application can restore the input low-quality first image into a high-quality restored image, and make the restored information correspond to the characteristics of the first image, so as to guide the neural network model to use an appropriate restoration method to restore the first image, thereby achieving the purpose of personalized enhancement of image quality.

[0164] In an optional implementation, the method provided by the electronic device in the embodiment of the present application may further include step S102b: determining restoration degree information of the first image. Further, step S103 may specifically include: restoring the first image using an AI network based on the restoration style information and the restoration degree information.

[0165] Optionally, the restoration degree information (for ease of description, it may be referred to as restoration degree, restoration strength or restoration force hereinafter) is related to at least one of the following: the image quality of the first image; the restoration degree related instructions input by the user. Among them, the restoration degree information may be related to the degree of image quality degradation of the first image, and the restoration degree information may be understood as a comprehensive reflection of the degree of image quality degradation (such as the degree of texture detail loss, the degree of noise pollution, etc.), and has nothing to do with the image content. The greater the degree of quality degradation, the greater the strength of image restoration. Taking the first image as a face image as an example, for a face image input (Input) with severely degraded quality, a greater restoration force is required to restore it to a high-quality output (Output), such as Figure 3 As shown in the upper part of ; for the face image input (Input) with less quality degradation, a smaller restoration effort is required to restore it to a high-quality output (Output), such as Figure 3 As shown in the middle part of ; for a face image input with little quality degradation (Input), only a small restoration effort is required to restore it to a high-quality output (Output), as shown in Figure 3 The restoration degree information may be related to the restoration degree related instruction input by the user, and the user may set the restoration degree information according to his / her own preference (preference for the image restoration degree). Optionally, a restoration degree confirmation interface for adjusting or determining the image restoration degree is provided to the user, and the adjustment information of the restoration degree information is displayed, and the user may adjust it, or the user may directly enter the restoration degree value in the restoration degree confirmation interface.

[0166] For the embodiments of the present application, some implementations of step S102b may also use an AI network. As an example, the restoration degree information of the first image is determined by a third neural network. In practical applications, those skilled in the art may set the specific neural network used in this step according to actual conditions, and the embodiments of the present application are not limited here.

[0167] It should be noted that the first neural network for determining the degree of recovery information and the third neural network for determining the recovery style may be different neural networks, or the first neural network and the third neural network may be encapsulated into one neural network. Those skilled in the art may set the architecture of the adopted neural network according to actual conditions, and the embodiments of the present application are not limited thereto.

[0168] For the embodiment of the present application, taking the first image as an example, a face image, such as Figure 4 As shown, for the input low-quality face image, the restoration degree can be first calculated to guide the neural network (such as the AI ​​generation model) to use appropriate restoration power to restore the image. At the same time, the restoration style information is calculated based on the input first image to guide the neural network to restore the image in the correct restoration direction, and finally the neural network can output a high-quality face image.

[0169] In the embodiment of the present application, a feasible implementation is provided for the step of “determining restoration degree information of the first image” (step S102b), which may specifically include:

[0170] Step SA1: providing a restoration degree confirmation interface for the first image;

[0171] Step SA2: In response to a restoration degree related instruction input by the user through the restoration degree confirmation interface, second restoration degree information of the first image is determined.

[0172] In an embodiment of the present application, a restoration degree confirmation interface of the first image is provided to the user. Optionally, the restoration degree confirmation interface is an interface that can adjust the image restoration degree. The user can adjust the restoration degree information according to his or her own preferences (preferences for the image restoration degree). For example, the default restoration degree information is displayed in the restoration degree confirmation interface, and the user can adjust it. Alternatively, the restoration degree confirmation interface is an interface that can directly input the image restoration degree value. The user can input the restoration degree value according to his or her own preferences (preferences for the image restoration degree) to restore the first image using the corresponding restoration degree value.

[0173] After detecting a restoration degree related instruction input by the user through the restoration degree confirmation interface (such as an instruction completed by user adjustment or an instruction completed by inputting a restoration degree value), in response to the restoration degree related instruction, based on the instruction result (the restoration degree information adjusted by the user or directly input), the second restoration degree information of the first image used to indicate the image restoration strength is determined. The second restoration degree information can be used to guide the neural network to restore the first image using the restoration force controlled by the user.

[0174] In the embodiment of the present application, another feasible implementation is provided for the step of “determining restoration degree information of the first image” (step S102b), which may specifically include:

[0175] Step SB1: determining image quality information of the first image;

[0176] Step SB2: determining corresponding first restoration degree information based on the image quality information;

[0177] Step SB3: Based on the first restoration degree information, providing a restoration degree confirmation interface for the first image;

[0178] Furthermore, step SA2 may specifically include:

[0179] Step SB4: In response to an adjustment instruction for the first restoration degree information input by the user through the restoration degree confirmation interface, second restoration degree information of the first image is determined.

[0180] Since the quality of an image reflects the quality of the image (a good image quality means a clear image, rich texture details, and low noise, while a poor image quality means a large noise, blurred image, and loss of texture details, etc.), a good quality image requires a small restoration effort, while a poor quality image requires a large restoration effort. That is, the prediction of image quality or its degradation can further realize the prediction of image restoration degree information.

[0181] Therefore, in the embodiment of the present application, the image quality information of the first image can be predicted first, and the recommended initial recovery degree (first recovery degree information) can be determined based on the predicted image quality information, and the recommended initial recovery degree can be displayed in the interface provided to the user (the recovery degree confirmation interface of the first image). That is, in this step, a reasonable initial recovery degree can be recommended to the user first, and on this basis, the user can adjust it according to his or her own preferences (including confirming the use of the recommended initial recovery degree) as guidance information for the final recovery degree.

[0182] After detecting that the user's adjustment is completed, in response to the user's input of an adjustment instruction for the first restoration degree information through the restoration degree confirmation interface, based on the adjustment result of the adjustment instruction (the restoration degree information adjusted by the user), the second restoration degree information of the first image indicating the image restoration strength is determined. The second restoration degree information can be used to guide the neural network to use an appropriate restoration force to restore the first image.

[0183] In the embodiment of the present application, another feasible implementation is provided for the step of “determining restoration degree information of the first image” (step S102b), which may specifically include:

[0184] Step SC1: determining image quality information of a first image;

[0185] Step SC2: Based on the image quality information, determine corresponding first restoration degree information.

[0186] In the embodiment of the present application, the image quality information of the first image can be predicted first, and based on the predicted image quality information, the first restoration degree information for indicating the image restoration strength is determined. The first restoration degree information is used to directly guide the neural network to use an appropriate restoration force to restore the first image. The first restoration degree information already contains information about using different restoration strengths for texture details in different regions.

[0187] In the embodiment of the present application, a feasible implementation is provided for the step of “determining the image quality information of the first image” (in step SB1 or step SC1), which may specifically include:

[0188] Step SD1: Acquire depth information and / or brightness information of the first image;

[0189] In the embodiment of the present application, there is no specific limitation on the method for obtaining the depth information and brightness information of the first image. Optionally, the depth information can directly use the sparse depth information obtained by the depth camera (TOF, Time of flight), or the depth information obtained by other methods. Optionally, the brightness information can be obtained by converting the first image to (for example, the rgb2lab algorithm can be used, but not limited to) the LAB color space (composed of the brightness channel L (Luminosity), color channel a (from red to green) and color channel b (from yellow to blue)), and then obtaining the brightness information of the L channel, or the brightness information obtained by other methods.

[0190] Step SD2: determining image quality information of the first image based on brightness information and / or depth information;

[0191] This step is used to explore the relationship between depth and brightness and spatial position, because the inventor of the present application found that the degree of image quality degradation varies at positions of different depths or different brightness in the image. Therefore, the image quality information of the first image can be determined based on the brightness information and / or the depth information.

[0192] In the embodiment of the present application, an optional implementation manner is provided for step SD2, which may specifically include:

[0193] Step SD21: Obtain a first image feature based on brightness information and / or depth information.

[0194] Optionally, after acquiring the brightness information and / or the depth information, features may be extracted from the brightness information and / or the depth information to obtain corresponding features (first image features).

[0195] Optionally, to make the depth information more accurate, it can be combined with the brightness information of the first image to make some corrections. Specifically, after obtaining the brightness, the depth (information or feature) and the brightness (information or feature) can be fused, and the depth information can be optimized using the brightness information to obtain more reliable depth information, and the fused feature map (first image feature) can be output.

[0196] Step SD22: extracting features from the first image to obtain features of the second image;

[0197] In practical applications, those skilled in the art may select a suitable feature extraction method, such as performing multiple convolution operations on the first image, etc., which is not limited in the embodiments of the present application.

[0198] Step SD23: Determine the image quality information of the first image based on the first image feature and the second image feature.

[0199] In the embodiment of the present application, the first image feature and the second image feature are combined to explore the correlation between the image quality degradation distribution and the depth information and the brightness information, and the image quality information is assisted in predicting the image quality distribution based on the depth and brightness information to predict a more accurate image quality distribution.

[0200] In the embodiment of the present application, an optional implementation method is provided for the solution of "obtaining the first image feature based on the brightness information and the depth information" in step SD21. Specifically, step SD21 may include:

[0201] Step SD211: extract features from brightness information and depth information respectively to obtain brightness features and depth features.

[0202] In practical applications, those skilled in the art may select a suitable feature extraction method, such as performing multiple convolution operations on the depth information and brightness information of the first image, etc., which is not limited in the embodiments of the present application.

[0203] Optionally, the purpose of making the depth information more accurate in step SD21 can be achieved by a spatial adaptive convolution module (SA module). Specifically, the brightness feature and the depth feature are input to the SA module, and the steps shown in steps SD212 to SD214 are executed.

[0204] Step SD212: Based on the brightness feature, determine a first weight map for compensating the depth feature.

[0205] In practical applications, those skilled in the art may select a suitable weight determination method, such as performing a predetermined convolution process on the brightness feature to obtain a first weight map for compensating the depth feature, etc. The embodiments of the present application are not limited thereto.

[0206] In the embodiment of the present application, considering that pixels with the same brightness are likely to have similar depths, optionally, a large kernel convolution (e.g., 5×5) is used to process the brightness features to obtain a first weight map (also referred to as a weighted map) that compensates for the sparse depth information.

[0207] Step SD213: weighting the depth feature according to the first weight map to obtain compensated depth feature.

[0208] In an embodiment of the present application, the depth feature is weighted according to the first weight map to compensate for sparse depth information, that is, a weighted value of a pixel is calculated over a large perceptual field (such as the above-mentioned 5×5) and applied to the depth feature of the pixel to achieve correction of the depth information.

[0209] Step SD214: Fuse the brightness feature and the compensated depth feature to obtain a first image feature.

[0210] In the embodiment of the present application, the compensated depth information and brightness information are fused to obtain a fused first image feature, which is used to predict the image quality information of the first image.

[0211] In the embodiment of the present application, an optional implementation manner is provided for step SD23. Specifically, step SD23 may include:

[0212] Step SD231: Determine a second weight map based on the first image feature.

[0213] The second weight map indicates the correlation between the image quality degradation degree of the first image and the depth information and / or brightness information.

[0214] In practical applications, those skilled in the art can select a suitable weight determination method, such as performing a channel attention operation on the first image feature to obtain a second weight map. Specifically, assuming that the size of the first image feature is h×w×c, where c is the number of channels in the channel dimension, a channel attention operation is performed on the fused first image feature, that is, for each pixel, the attention is transferred to the most important channel among the c channels, and a second weight map (also called a spatial weight map) of size h×w×1 is obtained.

[0215] Step SD232: Process the second image feature through a multi-head self-attention network to obtain a query feature, a key feature, and a value feature of the second image feature;

[0216] That is, in the embodiment of the present application, a multi-head self-attention network model is used to obtain three branches of KQV (Key, Query, Value, key feature, query feature, value feature) based on the extracted second image features.

[0217] Step SD233: weighting the value features according to the second weight map;

[0218] That is, in the embodiment of the present application, the second weight map and the above-mentioned V branch are weightedly fused.

[0219] Step SD234: Determine the image quality information of the first image based on the weighted value features, query features and key features.

[0220] That is, in the embodiment of the present application, the image quality information (which may be an image quality distribution diagram) is obtained by using the fused V, K and Q. After the image quality information is predicted, the restoration degree (distribution) information of the first image can be further obtained.

[0221] Optionally, step SD23 can be implemented by an image quality prediction module (Depth and Lightness wised Multi-head Self-Attention block, DL-MSA module). Specifically, the first image feature and the second image feature are input into the DL-MSA module, and the steps shown in steps SD231 to SD234 are executed to output the image quality information of the first image.

[0222] Considering that depth and brightness have a nonlinear relationship with image quality degradation, the main function of DL-MSA is to apply this correlation to predict the spatial distribution of image degradation with depth and brightness. Specifically, the first image feature (depth and / or brightness) is converted into a spatial weight map (second weight map) through the channel attention operation. In the multi-head self-attention of the second image feature, this weight map is applied as a spatial weight to the (V) value of the second image feature (mining depth and brightness information). The process can be expressed as:

[0223]

[0224] Among them, X 0 represents the first image feature; X 0 W represents the feature of the first image feature after performing channel attention operation; softmax(X 0 W) represents the second weight map (activated by the softmax activation function); X 1 represents the second image feature; X 1 W V V represents the feature of the second image; ⊙ represents weighted processing; SelfAttention(X 1 ) represents the self-attention feature after the K feature and Q feature of the second image feature are fused; Indicates fusion processing.

[0225] Based on at least one of the above embodiments, the embodiment of the present application takes the first image as an example of a face image, and provides a complete process example for determining the image quality information of the first image, such as Figure 5a As shown, it may specifically include:

[0226] Step 5.1: Obtain a depth map (ie, depth information) of the input face image, extract features from the depth map through a depth encoder, and obtain depth features.

[0227] Step 5.2: Convert the input face image to the LAB color space through the rgb2lab algorithm, obtain the brightness information of the L channel, extract features from the brightness information through the brightness encoder, and obtain brightness features.

[0228] Step 5.3: Combine the depth features and brightness features through n 1 SA modules are used for processing. The processing process of the SA module may be: convolution processing is performed on the brightness feature using a 5×5 large kernel convolution, and the brightness convolution processing result is processed through an activation function (such as sigmoid, but not limited thereto) to obtain a first weight map; convolution processing is performed on the depth feature using a 1×1 kernel convolution, and the depth convolution processing result is weighted through the first weight map (element multiplication), and the brightness feature and the weighted result are fused through a correction module to realize the correction of the depth feature; and the fusion result is convoluted through a 1×1 kernel convolution to obtain the first image feature X 0 .

[0229] Step 5.4: Extract features from the input face image through the shared encoder to obtain the second image feature X 1 .

[0230] Step 5.5: Set the first image feature X 0 and the second image feature X 1 By 2 The processing process of the DL-MSA module is as follows: the first image feature X of size h×w×c 0 Perform channel attention operation to obtain image features W of size h×w×1; process image features W through activation function (such as softmax, but not limited to this) to obtain spatial weight map (second weight map) of size h×w×1; convert the second image features X of size h×w×c into 1 The multi-head self-attention network model is used to obtain the three branches of KQV, and the features W are all hw×c. K , feature W Q and feature W V ; The feature W VUse the second weight map to perform weighted processing (multiply elements) to obtain the weighted V feature; K and feature W Q The weighted V feature and the self-attention feature are fused (matrix multiplication) to obtain the image quality information X of h×w×c. out .

[0231] In another example, the number of SA modules n 1 The number of DL-MSA modules that can be used is n 2 Same, such as Figure 5b As shown, n 1 =n 2 =2 as an example, the specific processing flow may include:

[0232] Get the depth map (i.e. depth information) of the input image, extract features from the depth map through a depth encoder to obtain depth features. Convert the input image to the LAB color space through the rgb2lab algorithm, and obtain the brightness information of the L channel from it, extract features from the brightness information through a brightness encoder to obtain brightness features. Convolve the depth features and brightness features separately and input them into the first SA module (SA1 module) and the second SA module (SA2 module) for processing to obtain the first image features respectively. and the first image feature The input image is passed through a shared encoder to extract features, and the second image features are obtained after convolution (conv) The first image feature and the second image feature The first DL-MSA module (DL-MSA1 module) is used for processing, and the processing result is convolved (conv) to obtain the updated second image feature The first image feature and the second image feature The image quality information X is obtained by processing through the second DL-MSA module (DL-MSA2 module) OUT It is not difficult to understand that the number of SA modules and DL-MSA modules can be deduced in the same way until the last DL-MSA module outputs image quality information X OUT , similar processes will not be repeated here.

[0233] In the embodiment of the present application, the above recovery degree confirmation interface may include at least one of the following functions: a global adjustment function and a local adjustment function. Both the global adjustment function and the local adjustment function can be controlled by the user according to his or her own preferences.

[0234] In an optional implementation, the recovery degree confirmation interface includes at least one of the following: a global recovery degree confirmation interface and at least one local recovery degree confirmation interface, wherein the at least one local recovery degree confirmation interface corresponds to at least one local area of ​​the target object.

[0235] In actual applications, technical personnel in this field can set the display method of the global recovery degree confirmation interface and at least one local recovery degree confirmation interface according to actual conditions. For example, they can be displayed simultaneously, such as being displayed as sub-interfaces in different areas of an interface; or they can be displayed in sequence, for example, the global recovery degree confirmation interface is displayed first, and when the relevant instructions of the global recovery degree confirmation interface are input, one or more local recovery degree confirmation interfaces are displayed, etc. The embodiments of the present application are not limited here.

[0236] That is, in the embodiment of the present application, the control of the image restoration degree provided to the user includes global control and local control. Optionally, the default restoration degree recommended to the user may include a global default restoration degree (displayed in the global restoration degree confirmation interface) and a local default restoration degree for a local area (displayed in the corresponding local restoration degree confirmation interface); or optionally, the entry for the user to input a restoration degree value may include a global default restoration degree entry (displayed in the global restoration degree confirmation interface) and a local default restoration degree entry for a local area (displayed in the corresponding local restoration degree confirmation interface). In actual applications, those skilled in the art may set the type of restoration degree information (for example, corresponding to which local area or areas of the target object), quantity, and specific numerical conditions according to actual conditions, and the embodiment of the present application does not limit this.

[0237] Furthermore, step SA2 may specifically include:

[0238] Step SA21: determining the second restoration degree information of the first image in response to at least one of the following instructions: a global restoration degree related instruction input by a user through a global restoration degree confirmation interface, at least one local restoration degree related instruction input by a user through at least one local restoration degree confirmation interface;

[0239] That is, in this step, the user can enter relevant instructions only in one recovery degree confirmation interface (such as the global recovery degree confirmation interface or any local recovery degree confirmation interface), and keep other interfaces unchanged or no longer enter, or can also enter corresponding relevant instructions in multiple recovery degree confirmation interfaces respectively.

[0240] In an embodiment of the present application, in response to relevant instructions input in one or more restoration degree confirmation interfaces, second restoration degree information of the first image is determined to indicate the degree of restoration required at different spatial positions of the first image, and this degree is used to guide the degree of image restoration in the personalized face restoration process.

[0241] In the embodiment of the present application, a feasible implementation method is provided for step SB2, which may specifically include:

[0242] Step SB21: determining first global restoration degree information based on the image quality information;

[0243] Step SB22: Based on the image quality information and the image mask of at least one local area, determine the first local restoration degree information corresponding to at least one local area.

[0244] That is, in the embodiment of the present application, the adjustment of the image restoration degree provided to the user includes global adjustment and local adjustment. Specifically, the initial restoration degree (first restoration degree information) recommended to the user may include the global initial restoration degree (first global restoration degree information) and the local initial restoration degree of the local area (first local restoration degree information).

[0245] Then step SB3 may specifically include:

[0246] Step SB31: providing a global recovery degree confirmation interface in the recovery degree confirmation interface based on the first global recovery degree information;

[0247] Step SB32: Based on the first local recovery degree information respectively corresponding to at least one local area, providing at least one corresponding local recovery degree confirmation interface in the recovery degree confirmation interface;

[0248] Similarly, those skilled in the art may set the display method of the global recovery degree confirmation interface and at least one local recovery degree confirmation interface according to actual conditions. For example, they may be displayed simultaneously, such as being displayed as sub-interfaces in different areas of an interface; or they may be displayed sequentially, for example, the global recovery degree confirmation interface may be displayed first, and when the relevant instructions of the global recovery degree confirmation interface are input, one or more local recovery degree confirmation interfaces may be displayed in a certain manner, etc. The embodiments of the present application are not limited here.

[0249] In an embodiment of the present application, the local area can be determined by the corresponding image mask. In practical applications, those skilled in the art can set the type and number of local areas (image masks) according to actual conditions, and the embodiments of the present application are not limited here. Taking the first image as a face image as an example, the types of local areas (image masks) may include but are not limited to eyes, mouth, skin, hair, etc. Among them, each local area (image mask) can be obtained through a variety of ways, for example, obtaining each local area of ​​the face image according to a face analysis graph. In practical applications, the global area can also be understood as a mask image corresponding to a global area.

[0250] Specifically, in step SB21, the overall average level of the image quality information may be calculated, that is, the average value of all pixels in the image quality information may be calculated, and the first global restoration degree information recommended to the user may be determined based on the overall average level.

[0251] In step SB22, the average level corresponding to at least one local area of ​​the image quality information can be calculated, that is, for each local area in the at least one local area in the image quality information, the average value of all pixels in the local area is calculated (it can also be understood as the average value of all pixels in the image quality information after multiplying by the local mask image), and based on the average level corresponding to at least one local area, the first local recovery degree information corresponding to at least one local area recommended to the user is determined.

[0252] Furthermore, a feasible implementation method is provided for step SB4, which may specifically include:

[0253] Step SB41: Determine the second recovery degree information of the first image in response to at least one of the following instructions: an adjustment instruction for the first global recovery degree information input by the user through the global recovery degree confirmation interface, and an adjustment instruction for at least one first local recovery degree information input by the user through at least one local recovery degree confirmation interface.

[0254] That is, in the embodiment of the present application, the first global restoration degree information and the first local restoration degree information corresponding to at least one local area can be controlled by the user according to his or her preferences. The control variable (the amount of change before and after adjustment) can be interpreted as a weighted factor of the degree of image restoration in the global area or the local area. These variables can build a bridge between the model and the user, allowing the user to control the degree of image restoration according to his or her preferences, thereby achieving personalized face image enhancement.

[0255] Among them, the adjustable user interface includes: a global adjustment function (corresponding to the first global recovery degree information, the user can adjust the overall recovery degree of the image, which is displayed in the global recovery degree confirmation interface) and a local adjustment function (corresponding to the first local recovery degree information corresponding to at least one local area, the user can adjust the image recovery degree for different local areas, such as the eyes, mouth, skin, hair, etc. in the human face, which is displayed in the corresponding local recovery degree confirmation interface).

[0256] In this step, the user can input adjustment instructions in only one recovery degree confirmation interface (eg, the global recovery degree confirmation interface or any local recovery degree confirmation interface), while other interfaces remain unchanged, or can also input adjustment instructions in multiple recovery degree confirmation interfaces respectively.

[0257] That is, in an embodiment of the present application, in response to an adjustment instruction input in one or more restoration degree confirmation interfaces, second restoration degree information of the first image is determined, which is used to indicate the degree of restoration required at different spatial positions of the first image, and this degree is used to guide the degree of image restoration in the personalized face restoration process.

[0258] In the embodiment of the present application, a feasible implementation method is provided for step SA2 (which may also refer to SA21, step SB4, or step SB41), which may specifically include:

[0259] Step SAa: In response to a global recovery degree related instruction input by a user through the global recovery degree confirmation interface, determining second global recovery degree information;

[0260] In some of the above embodiments, the global recovery degree related instruction may also refer to an adjustment instruction for the first global recovery degree information input by the user through the global recovery degree confirmation interface.

[0261] Step SAb: in response to at least one local recovery degree related instruction input by the user through at least one local recovery degree confirmation interface, determining corresponding at least one second local recovery degree information;

[0262] In some of the above embodiments, any local recovery degree related instruction may also refer to an adjustment instruction for the corresponding first local recovery degree information input by the user through the corresponding local recovery degree confirmation interface.

[0263] Step SAc: Fusing the second global recovery degree information and at least one second local recovery degree information to obtain the second recovery degree information.

[0264] In an optional implementation, according to the second global restoration degree information and at least one second local restoration degree information, a global adjustment value and a local adjustment value are obtained as the adjustment result, and the image quality information (such as the image quality distribution map) is adjusted globally and locally based on the global adjustment value and the local adjustment value to obtain the second global restoration degree information and at least one second local restoration degree information, and after fusion, an image restoration degree control map (second restoration degree information) is obtained. The control map is a weight map with the same scale as the first image, indicating the degree of restoration required at different spatial positions of the image, and the degree is used to guide the degree of image restoration in the personalized face restoration process.

[0265] It can be understood that for the recovery degree confirmation interface where the user has not entered any adjustment instructions, the corresponding first (global or local) recovery degree information and the second (global or local) recovery degree information may be the same, that is, the (global or local) adjustment value is zero.

[0266] In the following, the first image is a face image as an example. Figure 6 An example of a user controlling the restoration degree information according to his / her own preference is shown, specifically including:

[0267] Step S6.1: Obtain input image quality information;

[0268] Step S6.2: Calculate the overall average level of the image quality information (optionally, superimpose the image mask of the global area), determine the first global restoration degree information α recommended to the user based on the overall average level, and display it in the provided interface (global restoration degree confirmation interface);

[0269] Step S6.3: Calculate the average level of the eye area of ​​the image quality information using the image mask of the eye area, and determine the first local restoration degree information β recommended to the user based on the average level of the eye area. 1 , and display it in the provided interface (the local recovery degree confirmation interface corresponding to the eye area);

[0270] Step S6.4: Using the image mask of the mouth area, calculate the average level of the mouth area of ​​the image quality information, and determine the first local restoration degree information β recommended to the user based on the average level of the mouth area. 2 , and displayed in the provided interface (the local recovery degree confirmation interface corresponding to the mouth area);

[0271] Step S6.5: Calculate the average level of the image quality information of the skin area using the image mask of the skin area, and determine the first local restoration degree information β recommended to the user based on the average level of the skin area. 3, and displayed in the provided interface (the local recovery degree confirmation interface corresponding to the skin area);

[0272] Step S6.6: Using the image mask of the hair area, calculate the average level of the image quality information of the hair area, and determine the first local restoration degree information β recommended to the user based on the average level of the hair area. 4 , and displayed in the provided interface (the local recovery degree confirmation interface corresponding to the skin area);

[0273] Step S6.7: α, β 1 , β 2 , β 3 and β 4 The user can control the first global recovery degree information α and the first local recovery degree information β in the provided interface. 1 , β 2 , β 3 and β 4 At least one of the control operations (input adjustment instructions for the first recovery degree information) is performed to obtain the corresponding adjustment amounts Δα, Δβ 1 , Δβ 2 , Δβ 3 and Δβ 4 ;

[0274] Step S6.8: Based on the image quality information, adjustment values ​​Δα, Δβ 1 , Δβ 2 , Δβ 3 and Δβ 4 , calculate the final second recovery degree information. Optionally, the calculation method can be expressed as the following formula:

[0275]

[0276] Wherein, finaldegree represents the final second restoration degree information; Q represents the image quality information; Δ represents the difference between the recommended value and the user feedback value; Δα represents the adjustment amount of the global restoration degree information; Δβ 1 , Δβ 2 , Δβ 3 , Δβ 4 The adjustment amounts representing the local restoration degree information of the eye area, mouth area, skin area and hair area respectively; Indicates the second global recovery degree information; mask eye 、mask mouth 、mask skin 、mask hair Image masks representing the eye area, mouth area, skin area, and hair area respectively; Respectively represent the second local restoration degree information of the eye area, mouth area, skin area and hair area; ⊕ represents the addition of each element; Δα, Δβ 1 , Δβ 2 , Δβ 3 , Δβ 4 ∈[0,1].

[0277] It can be understood that the user can achieve different global control effects by adjusting only the global restoration degree information α, for example, adjusting α to α=0.1, α=0.4, α=0.5, α=0.7, α=0.9, etc. Alternatively, the user can adjust the image quality of a certain area according to his or her preferences to achieve spatial decoupling of face semantic areas such as eyes, mouth, skin, and hair. For example, the local restoration degree information β1 of the eye area is adjusted to β1=0.1, β1=0.3, etc. to achieve different control effects for the eye area.

[0278] In the embodiment of the present application, a feasible implementation is provided for the step of “determining the restoration style information of the first image” (in step S102), which may specifically include:

[0279] Step SE1: determining attribute information of a target object in the first image according to the first image;

[0280] Specifically, taking the first image as a face image and the target object as a face as an example, Figure 7 As shown, the input first image can be passed through a decoder to predict various attributes of the target object, such as age prediction, gender prediction, makeup prediction, glasses prediction, hair color prediction, hair curl prediction, etc., but not limited to these.

[0281] Step SE2: Based on the attribute information, determine the restoration style information of the first image.

[0282] Optionally, the restored style information may be a style control code, which is a reflection of personalized facial features. Such style information may be used as prior knowledge to provide personalized direction for image restoration.

[0283] Optionally, step SE2 may specifically include:

[0284] Step SE21: Encode the attribute information into an attribute vector;

[0285] Step SE22: Map the attribute vector to restore style information through a multi-layer perception network.

[0286] Among them, if there are multiple attributes, the predicted multiple attributes can be jointly encoded into an attribute vector.

[0287] Furthermore, the attribute vector can be mapped to restore style information through a multilayer perceptron (MLP), and the layers of the multilayer perceptron are fully connected (FC).

[0288] Therefore, the embodiment of the present application predicts the attribute information of the face, calculates the face restoration style information according to the face attributes, and is used to guide the style / direction of face restoration. During the face restoration process, it can guide the restoration of personalized features consistent with the character attributes.

[0289] In the embodiment of the present application, when the restoration information includes restoration degree information and restoration style information, an optional implementation method is provided for step S103, and personalized restoration of the first image is performed using the predicted restoration degree information and restoration style information. Specifically, it may include:

[0290] Step S1031: extracting corresponding third image features based on the first image;

[0291] Step S1032: extracting a fifth image feature corresponding to the attribute information of the target object in the first image from the third image feature based on the restoration style information;

[0292] Step S1033: obtaining a restored image based on the third image feature and the fifth image feature.

[0293] In step S1031, a deep feature extraction network (eg, an encoder) is used to extract deep feature codes (third image features) of the first image.

[0294] Optionally, if the restoration degree information of the first image is determined, step S1031 may specifically include: extracting corresponding third image features based on the first image and the restoration degree information of the first image.

[0295] Optionally, the calculated restoration degree information of the first image (for example, a restoration degree control map) is superimposed on the first image (which can also be understood as correction), and then a deep feature extraction network (for example, an encoder) is used to extract deep feature encoding (third image features), such as Figure 8 As shown in Figure 2, since the degree of restoration is related to the spatial position and the image quality is different in different regions, alignment and superposition can extract more meaningful image features and more accurately reflect the degree to which the image quality needs to be restored.

[0296] In practical applications, the encoder can extract one or more scales of third image features ( Figure 8 4 scales are taken as an example), those skilled in the art may set the number of scales extracted by the encoder according to actual conditions, and the embodiments of the present application are not limited thereto.

[0297] Furthermore, in step S1032, the modulation information (fifth image feature) is extracted from the third image feature extracted in step S1031 using the restored style information (for example, it may be a style control code). The restored style information is obtained after encoding the attributes of the image target object. Find features that match the style based on the style in the encoded features, and then calculate the correction amount using the decoded features of the first image at the decoder, which is equivalent to calculating how much of the style content is needed in the current feature, and then performing a reasonable restoration. If the third image feature includes multiple scales, the extracted fifth image feature may also correspond to multiple scales, for example Figure 8 In , the fifth image features of 4 scales are extracted.

[0298] Optionally, the extraction of modulation information from the third image features using the restored style information can be achieved through an attribute feature enhancement module, where the extracted modulation information is essentially a deep image feature consistent with the associated attribute (a fifth image feature corresponding to the attribute information of the target object in the first image). The attributes are related to texture features, but these features are not clear. The core functionality of this module is two-fold: (1) selecting features related to the associated attribute, and (2) adapting the selected features to the associated attribute. The output of this module enhances features that are only related to the associated attribute and is used as modulation information in the decoding process. Optionally, in order to improve computational efficiency, the attribute feature enhancement module can be performed only at the minimum resolution scale.

[0299] Furthermore, in step S1033, the modulation information (fifth image feature) is used to correct the decoding feature of the third image feature, for example, through a correction module, and finally a personalized restored image is restored. The function of the correction module is to correct the deviation of the restored feature according to the associated attribute. This module is applicable to all resolution ratios except the minimum decoder. Among them, if the third image feature includes multiple scales, the fifth image feature of the corresponding scale is used to correct the third image features of different scales (the third image feature starting from the second scale will be added to the correction result of the previous scale as the third image feature of the current scale), such as Figure 8 shown.

[0300] In this way, it is possible to use the restoration degree information (such as the restoration degree control map) and the restoration style information (such as the style control code) to perform personalized restoration on the input low-quality first image.

[0301] First, the predicted restoration degree information is related to the spatial position in the first image. The image quality in different spatial regions is different, and the degree of image restoration is also different. Aligning and superimposing the restoration degree information with the first image in spatial position facilitates the exploration of the spatial distribution of image quality and guides the model to achieve different restoration strengths in different regions.

[0302] Secondly, the recovery degree information can enable users to control their preferences based on the recovery degree of the region.

[0303] Third, the recovered style information is used to extract modulation information from the deep features of the first image. This information can then modulate the decoded features at different scales and guide the model to add plausible and realistic textures (consistent with the predicted multiple attributes) in areas where texture details are missing.

[0304] In the embodiment of the present application, an optional implementation is provided for step S1032, which may specifically include:

[0305] Step S10321: Fusing the restored style information with the attribute information to obtain a fourth image feature;

[0306] Step S10322: Based on the fourth image feature and the attribute information, obtain the fifth image feature.

[0307] Optionally, in step S10321, based on the restored style information (which may be style control coding), an attribute feature enhancement module is used to enhance the features consistent with the predicted attributes in the third image features (encoded image features) to obtain a fourth image feature with enhanced associated attributes, such as Fig. 9 shown.

[0308] Optionally, AdaIN (Adaptive Instance Normalization) is used to continuously inject the restored style information into the third image feature, the common information is weakened, and the image features associated with the attributes are strengthened. The process can be expressed as the following expression:

[0309] AdaIN(x,y)=σ(y)((x-μ(x)) / σ(x))+μ(y)

[0310] Among them, AdaIN(x,y) represents the fourth image feature, x represents the third image feature, y represents the restored style information, σ and μ are the mean and standard deviation respectively, and the mean and standard deviation can represent the style of the image; (x-μ(x)) / σ(x) represents normalizing the mean and standard deviation of the third image feature (destyling), σ(y)((x-μ(x)) / σ(x))+μ(y) represents denormalizing the image feature to the mean and standard deviation corresponding to the restored style information (stylized according to the restored style information). Further, in step S10322, features corresponding to the associated attributes are selected and extracted from the fourth image features after the associated attributes are strengthened. It is not difficult to understand that the selected and extracted features are sensitive to user attributes, and the values ​​of these features will change accordingly if the user attributes are different. For example Fig.10 As shown, there is a difference between the upper and lower branches. The upper branch is the result of further convolution of the image feature itself (the third image feature), and the lower branch is the result of further convolution of the image feature mixed with the attribute coding (the fourth image feature) (due to the combination with the style control coding, the features related to the associated attributes are highlighted through further feature extraction). Therefore, the difference between the two branches represents the texture features related to the associated attributes. In short, in this step S10322, the related attribute features are selected.

[0311] Optionally, step S10322 may specifically include: determining a difference image feature between the third image feature and the fourth image feature; and adjusting the difference image feature to be consistent with the attribute information to obtain a fifth image feature.

[0312] That is, in the embodiment of the present application, the feature corresponding to the associated attribute can be selected and extracted from the fourth image feature after the associated attribute is enhanced by determining the difference image feature between the third image feature and the fourth image feature.

[0313] Furthermore, the selected and extracted features (determined difference image features) can be further adjusted. In other words, the features need to be consistent with the attributes of the guidance. For example, the expression of wrinkles is different between 30-year-olds and 70-year-olds.

[0314] Optionally, continue as Fig. 9 and Fig.10As shown, using a transformer network (module), the values ​​of the selected and extracted features (determined difference image features) are adjusted to match the relevant attribute values. The transformer network is used to mine the long-distance semantics of attribute features through the overall global area, that is, the relatively distant features in the first image can establish associations. The correlation of texture features related to the associated attributes becomes very large, while the correlation of unrelated texture features decreases. For example, taking the first image as a face image, for the feature matching the age, the feature value of the feature is adjusted according to the predicted user age. If the predicted user age is older, the tiny wrinkles can be emphasized.

[0315] Optionally, multiple transformer networks can be used ( Fig. 9 Taking four transformer networks as an example) to adjust the difference image features to correspond to the attribute information. Starting from the second transformer network, the input difference image features can be the difference image features between the third image features after the corresponding number of convolutions and the fourth image features obtained by injecting the corresponding number of restored style information into the third image features, and the difference image features are corrected based on the adjustment result output by the previous transformer network, and then input into the current transformer network, until the last transformer network outputs the adjusted fifth image features.

[0316] It should be noted that Fig.10 The values ​​of the image features shown in are for reference only. The specific values ​​are subject to actual implementation. Fig.10 The numerical examples in the figures should not be construed as limiting the present application.

[0317] In the embodiment of the present application, an optional implementation is provided for step S1033, which may specifically include:

[0318] Step S10331: determining an image feature correction amount corresponding to the attribute information according to the correlation between the third image feature and the fifth image feature;

[0319] Step S10332: Correct the third image feature according to the image feature correction amount to obtain a restored image.

[0320] That is, in the embodiment of the present application, the three image features are modulated based on the extracted modulation information (fifth image feature), specifically, a correction module (see Figure 8 ), correct the decoded third image feature, that is, calculate the image feature correction amount of the associated attribute and add it to the decoded feature of the third image feature to implement the modulation process and obtain the restored image.

[0321] This is because the third image feature itself contains attribute features, and directly superimposing the modulation information (the fifth image feature, related to the associated attribute) on the decoded third image feature for image restoration may cause excessive texture restoration, that is, it is impossible to control the reasonable restoration of the associated texture. Therefore, an embodiment of the present application proposes the use of a correction module to calculate the image feature correction amount corresponding to the associated attribute feature based on the existing features and modulation information in the current third image feature, and correct the image feature corresponding to the attribute information in the third image feature based on the image feature correction amount. Specifically, the function of the correction module is to correct the deviation of the restored features in the guiding attributes.

[0322] Optionally, the correction process described above can be achieved by calculating a correction amount weight map. Since the adjusted attribute feature (fifth image feature) only contains features related to the attribute, and the third image feature reflects the comprehensive features of the first image. The correlation between the two inputs can reveal how many additional attribute features are needed in the current image feature. The higher the correlation, the fewer attribute features are needed, and the lower the correlation, the more attribute features are needed. In an embodiment of the present application, this correlation is displayed as a correction amount weight map, which reflects the size of the correlation and is also an indication of the amount of correction of the attribute features in the current image feature, such as Fig.11 Specifically, Fig.12 As shown, the result of the deformation (Reshape) of the third image feature after convolution is fused (matrix multiplication) with the result of the convolution, deformation and transposition (Transpose) of the fifth image feature, and then processed by an activation function (such as softmax, but not limited to this) to obtain a correction amount weight map, which is then fused with the convolution of the fifth image feature (matrix multiplication) to extract a suitable image feature correction amount from the fifth image feature, and finally added to the third image feature to obtain the correction result of the third image feature.

[0323] Among them, if the third image feature includes at least one scale, step S10331 may specifically include: for the third image feature of each scale, according to the correlation between the third image feature of the scale and the fifth image feature of the corresponding scale, determining the image feature correction amount of the scale corresponding to the attribute information.

[0324] Furthermore, step S10332 may specifically include: performing correction processing on corresponding third image features according to an image feature correction amount of at least one scale to obtain a restored image.

[0325] The third image feature of each scale starting from the second scale of at least one scale is the third image feature after correction processing is performed on the image feature correction amount of the previous scale.

[0326] That is, in the embodiment of the present application, the correction module can correct the third image features of different scales during the decoding process, achieve reasonable restoration of attribute-related textures at each resolution scale, and finally obtain a restored face image.

[0327] like Figure 8 As shown, the correction module works on each scale feature of the decoder. The calibration amount for each scale feature can be calculated in an incremental manner. The number of correction modules (N) is related to the number of decoder scales (M), N = M. Calibration at different scales helps improve the stability of texture restoration.

[0328] The correction module provided in the embodiment of the present application has the following advantages:

[0329] 1) It provides a feasible solution for embedding attributes (such as the age and gender of the person in the face image) into image features.

[0330] 2) Utilize the local feature extraction capability of convolution and the long-range semantic capture capability of transformer to enhance attribute-related features.

[0331] 3) From deep semantic features to shallow texture features, a progressive correction strategy is adopted to gradually restore features related to attributes (e.g., person attributes of face images).

[0332] Based on at least one of the above embodiments, the embodiment of the present application takes the first image as an example of a face image to be redrawn, and provides a complete process example of personalized face restoration, such as Fig.13a As shown, it may specifically include:

[0333] The low-quality face image is input into the module for determining the restoration degree information and the module for determining the restoration style information respectively.

[0334] In the module for determining the degree of restoration information, the face image will go through the encoder, decoder and other processing processes to predict the image quality information of the face image, which is used to determine the degree of restoration information recommended to the user. High image quality means that the image is clearer, the texture details are richer and the noise points are fewer. On the contrary, poor image quality means that the image is more blurred, the texture details are lost and there are more noise points. High-quality images require a smaller restoration strength, while poor-quality images require a larger restoration strength. Further, based on the predicted image quality information, the initial restoration degree control value (restoration degree information) is recommended to the user and displayed on the user adjustment interface. Different users adjust the image restoration degree control value according to their own preferences. The user's control of the restoration degree information includes global control, as well as local control such as eye control, mouth control, skin control, and hair control. According to the restoration degree control value fed back by the user, the user image restoration degree control diagram (adjusted restoration degree information) is obtained.

[0335] In the module for determining the restored style information, the face image is passed through a decoder to predict various attributes of the person, such as age prediction, gender prediction, makeup prediction, etc., but not limited to these. Personalized features are closely related to human attributes. The predicted multiple attributes are jointly encoded into an attribute vector, and then the attribute vector is mapped to the attribute control code through a multi-layer perceptron (multiple FC layers) to obtain the restored style information, which is used to provide a restoration direction for the personalized face redrawing model to restore personalized features consistent with human attributes.

[0336] Furthermore, the obtained restoration degree information and restoration style information are input into the personalized face redrawing model together to guide the model to use appropriate restoration strength and restore the image in the correct restoration direction (the direction consistent with the attribute). Specifically, the restoration degree information is used to extract the deep feature code (the third image feature) by the encoder. Through the attribute feature enhancement module, the modulation information is used to extract the modulation information (the fifth image feature) from the deep feature code. Through the correction module, the modulation information is used to correct the decoded features after the decoder decodes the deep feature code; the correction result is superimposed on the deep feature code and then decoded by the decoder to output the restored face image.

[0337] Based on at least one of the above embodiments, the embodiment of the present application takes the first image as a face image as an example to provide another complete process example of personalized face restoration, such as Fig.13b As shown, it may specifically include:

[0338] The low-quality face image is input into the module for determining the restoration degree information and the module for determining the restoration style information. In the module for determining the restoration degree information, the face image is processed by the encoder, decoder and other processing processes to predict the image quality information of the face image, which is used to determine the restoration degree information recommended to the user and displayed in the user adjustment interface, including the user's control of the restoration degree information, including global control, and local control such as eye control, mouth control, skin control, and hair control, to obtain the restoration degree information adjusted by the user. In the module for determining the restoration style information, the face image is passed through the decoder to predict various attributes of the face, such as age prediction, gender prediction, makeup prediction, etc., but not limited thereto. The predicted multiple attributes are jointly encoded into an attribute vector, and then the attribute vector is mapped into the restoration style information through a multi-layer perceptron (multiple FC layers). Further, the obtained restoration degree information and restoration style information are input into the personalized face redrawing model together with the face image, so as to guide the model to use appropriate restoration strength and restore the image in the correct redrawing direction (the direction consistent with the attribute) through the restoration degree information and restoration style information. Specifically, the restoration degree information is superimposed on the face image (which can also be understood as correction), and the encoder is used to extract the deep feature codes of four scales (third image features). Through the attribute feature enhancement module, the modulation information of four scales is extracted from the deep feature codes using the restored style information (fifth image features). Through the correction module corresponding to the first scale, the modulation information of the first scale is used to correct the decoded features of the deep feature codes of the first scale; the correction result is superimposed on the deep feature codes of the second scale and then decoded, and the correction module corresponding to the second scale is used to correct the modulation information of the second scale, and so on, until the correction modules corresponding to the four scales output high-quality face images.

[0339] The personalized face redrawing method provided in the embodiment of the present application can restore an input low-quality face image into a high-quality face image, and make the restored texture details consistent with the actual attributes of the portrait (for example, women's lips are rosy, young people's skin is tighter and smoother than that of the elderly, etc.).

[0340] In the embodiment of the present application, considering that the subjective visual perception factor is not considered in the image restoration process, the restored image may still not be natural enough in subjective visual perception. Therefore, a method for optimizing selectable restoration degree information (such as a restoration degree control map) is provided. Specifically, it may include: repeating the following update process at least once:

[0341] Step S201: determining a Peak Signal to Noise Ratio (PSNR) of a first image according to a currently obtained restored image;

[0342] Step S202: determining a recovery degree information update parameter according to the peak signal-to-noise ratio value;

[0343] Step S203: updating the recovery degree information based on the recovery degree information update parameter;

[0344] Step S204: Based on the updated restoration degree information, the first image is restored to obtain an updated restored image.

[0345] The PSNR value is an objective evaluation index. In the embodiment of the present application, after performing one or more image restorations to obtain a restored image, the restoration degree information can be optimized according to the restored image.

[0346] Specifically, referring to the currently obtained restored image, the PSNR value of the first image can be determined, and according to the PSNR value, the restoration degree information update parameter (that is, the image quality degree that needs to be adjusted) is determined:

[0347]

[0348] in, represents the recovery degree information update parameter, and γ comes from the user's control (e.g. Figure 6 The output result of the image is f(PSNR), and f(PSNR) represents the image subjective perception adjustment parameter determined according to the PSNR value, which remaps the PSNR value to [0, 1].

[0349] Optionally, the image subjective perception adjustment parameter may be determined according to the above PSNR value and the user's subjective perception sensitivity rule:

[0350]

[0351] Among them, PSNR is the above PSNR value, and a and b are related to the user's subjective sensitivity.

[0352] Optionally, you can follow the Fig.14According to the law shown, a mapping function for the relationship between the above PSNR value and the user's subjective feeling is determined. Specifically, when the PSNR is between 15db and 35db, human vision is sensitive to changes in image quality. If the PSNR of the first image is within this range, the mapping function can adopt a larger change rate, so that the image subjective feeling adjustment parameter changes more; when the PSNR is lower than 15db or higher than 35db, human vision is not sensitive to changes in image quality, and the mapping function can adopt a slower change rate, so that the image subjective feeling adjustment parameter changes less. In the embodiment of the present application, a and b are used to control the steepness of the curve and determine the range of the sensitive area. The default values ​​can be set to a=5 and b=5. These two parameters can also be adjusted according to the user's preferences. For example, if the user likes richer details and clearer edges, these two values ​​can be increased. The mapping function f takes into account the subjective perception of PSNR by human vision, and f(PSNR) better reflects the subjective perception of the image by human vision.

[0353] For the embodiment of the present application, the above mapping function maps the sensitivity range of human vision to image changes to most of the new value range, so that the mapping function focuses on the subjective feeling of the human eye. Then, the subjective feeling adjustment parameter can be calculated according to the mapping function, such as Fig.15 shown.

[0354] Furthermore, if Fig.16 As shown in Fig. 1, the predicted image restoration degree information (quality map) can be updated in an iterative manner through the self-reference quality re-estimation of the model (determining the restoration degree information update parameters). Specifically, q 0 is the predicted restoration degree information. In the personalized image restoration model (see Figure 8 ), the output restored image is used as a reference to re-evaluate the input first image and output a re-evaluated restoration degree information After the fusion gate, the new quality map q i Used to restore the personalized image again:

[0355]

[0356] in, Fig.16 For details not included in the Figure 8 The introduction of , I will not go into details here.

[0357] Furthermore, based on the restoration degree information update parameter determined by the subjective feeling adjustment parameter, the restoration degree information (for example, the image restoration degree control map) is adjusted.

[0358] In the embodiment of the present application, the subjective adjustment parameter is generated by a mapping function, which reflects the degree of restoration of the restored image output under the current setting, but takes more into account the subjective sensitivity of the human eye, and can be weighted and fused with the historical restoration degree information to obtain new restoration degree information. The restoration degree information takes into account the distribution of image quality and the user's preference for the restoration degree, and also takes into account the universal subjective visual perception.

[0359] Since the subjective perception of human vision is fully considered, the updated image restoration degree can guide the network to restore textures that are more in line with human perception. As an example, the inventor of this application used the eye area of ​​a face image as input and experimented with two rounds of image restoration degree information estimation. 0 = 0.567, the strength of the network restored image is slightly smaller than human perception; based on the restoration degree information of the first round of re-evaluation Updated recovery level information 1 =0.451, the strength of the restored image is slightly larger than human perception; based on the second round of re-evaluation of the restoration degree information Updated recovery level information 1 =0.498, the intensity of the restored image is more consistent with human perception.

[0360] In the embodiment of the present application, an optional implementation method is provided for step S101, specifically, it may include at least one of the following methods:

[0361] Step S1011: Acquire a second image;

[0362] In the embodiment of the present application, the first image may be from the second image. For example, the second image is a stored image or a captured image, and the first image is an image of a partial area in the second image.

[0363] Step S1012: Detecting a target object in the second image;

[0364] That is, in the embodiment of the present application, the first image is related to the area where the target object in the second image is located.

[0365] Step S1013: Determine the first image based on at least one of the following:

[0366] (1) The number of detected objects;

[0367] Optionally, if the second image contains only one target object, the area corresponding to the target object can be used as the first image; if the second image contains multiple target objects, the area corresponding to each target object can be used as a first image, or the areas corresponding to some target objects can be used as a first image.

[0368] Optionally, when the number of detected target objects is less than a first set threshold, the area corresponding to each target object can be used as a first image, and when the number of detected target objects is not less than the first set threshold, the areas corresponding to some target objects can be used as a first image, and the subsequent image restoration process. In practical applications, those skilled in the art can set the first set threshold according to actual conditions, and the embodiments of the present application are not limited thereto.

[0369] (2) Image quality of the image area corresponding to the detected target object;

[0370] That is, in the embodiment of the present application, to determine whether an image area corresponding to a target object needs to be redrawn, the image quality of the image area corresponding to the target object can be used to make the determination.

[0371] Optionally, based on the image quality of the image area corresponding to the detected target object, determining the first image in the image area corresponding to the target object may include: determining the image quality of at least one image area corresponding to the target object; determining the image area whose image quality related value is lower than the second set threshold as the first image. If the image quality related value is lower than the second set threshold, the corresponding image area is used as a first image, and the subsequent image restoration process can be activated. This setting can save computing resources and improve real-time processing performance. In actual applications, those skilled in the art can set the value of the second set threshold according to actual conditions, such as 90, etc., and the embodiments of the present application are not limited here.

[0372] Optionally, there may be a variety of judgment criteria for image quality related values. For example, signal processing technology may be used to check the clarity of the image area as an initial quick judgment criterion. Other judgment criteria may also be used, which is not limited in the embodiments of the present application.

[0373] (3) Target object selection instructions input by the user.

[0374] In an embodiment of the present application, for image areas corresponding to one or more target objects included in the second image, the user can determine whether the image area corresponding to each target object is used as the first image to perform a subsequent image restoration processing flow; alternatively, the user can select the image area corresponding to one or more target objects as the first image to perform a subsequent image restoration processing flow. If the user selects an image area corresponding to a target object, the image area corresponding to the target object is used as the first image; if the user selects image areas corresponding to multiple target objects, each image area corresponding to the target object selected by the user is used as a first image.

[0375] It can be understood that the above-mentioned several methods can be combined to determine the first image according to the settings.

[0376] As an example, in an optional embodiment, the method of determining the first image may include: when the number of detected target objects is not less than a first set threshold, based on the image quality of the image area corresponding to the detected target object and at least one of the target object selection instructions input by the user, determining the first image in the image area corresponding to the target object.

[0377] For example, when the number of detected target objects is not less than a first set threshold, it can be determined which image regions are to be processed as the first image based on the image quality of the image region corresponding to each target object.

[0378] Alternatively, when the number of detected target objects is not less than a first set threshold, the user may select image areas corresponding to which target objects are to be processed as the first image.

[0379] In practical applications, those skilled in the art can set the combination of the above methods according to actual conditions, and the embodiments of the present application are not limited here.

[0380] Optionally, for image areas that have not been used as the first image to perform subsequent image recovery processing procedures, when the user is not using the electronic device, or when a time period is set, or when the device status is set, the image area corresponding to each target object (which may be unprocessed) can be automatically used as the first image to perform subsequent image recovery processing procedures.

[0381] In the embodiment of the present application, a feasible implementation method is provided for step S1011. Specifically, obtaining the second image includes at least one of the following methods:

[0382] (1) Acquire a second image based on the image acquired in real time by the acquisition device.

[0383] For example, when a user takes a photo using an electronic device such as a mobile phone, the processing flow of at least one of the above embodiments may be triggered. The acquisition device may be built-in or external to the electronic device, and the embodiments of the present application do not limit this.

[0384] Optionally, the image collected by the collection device in real time can be selected by the user as the second image to trigger the processing flow of at least one of the above embodiments. If the user selects no, it can be deleted or directly stored in the electronic device based on the user's instructions.

[0385] Optionally, if the second image is an image acquired in real time based on the acquisition device, it may also include: when the number of detected target objects is not less than the third set threshold value, the second image is stored. That is, first determine whether to process in real time based on the number of target objects. For the second image that is not processed in real time, the processing flow of at least one of the above embodiments can be performed offline (when the user is not using the electronic device, or in a set time period, or when the device state is set, but not limited to this) or at the back end. Optionally, if it is not processed in real time, it can be determined offline or at the back end which image areas are processed as the first image based on the image quality of the image area corresponding to each target object. Alternatively, the user can choose offline or at the back end whether to process the image area corresponding to the target object as the first image, or the user can choose which image area or areas corresponding to which target objects are processed as the first image.

[0386] Optionally, each directly stored captured image can be used as a second image to trigger the processing flow of at least one of the above embodiments. For example, each directly stored image can be processed separately in an offline state or automatically at the back end. In other words, each stored image can be processed to obtain a high-quality restored image.

[0387] Optionally, in addition to triggering the processing flow of at least one of the above embodiments, a remaster (image re-recording) function may also be triggered. The remaster function includes but is not limited to image processing functions such as denoising, color processing, blur processing, and image restoration processing of at least one of the above embodiments.

[0388] (2) Acquire a second image based on an image stored in the electronic device.

[0389] For example, when a user selects an image in a photo album of an electronic device, the image may be used as the second image to trigger the processing flow of at least one of the above embodiments.

[0390] Optionally, each stored image can be used as a second image to trigger the processing flow of at least one of the above embodiments, for example, each directly stored image can be processed separately in an offline state or automatically at the back end. In other words, each stored image can be processed to obtain a high-quality restored image.

[0391] Optionally, in addition to triggering the processing flow of at least one of the above embodiments, a remaster function may also be triggered.

[0392] Furthermore, after step S103, the method may further include the steps of: determining a corresponding third image based on the restored image and the second image; and storing or displaying the third image.

[0393] For example, the photos taken by the user are redrawn and stored in the photo album of the electronic device.

[0394] For another example, the photos selected by the user from the photo album are redrawn and displayed in the interface of the electronic device.

[0395] For another example, the photos in the album are redrawn by the back end and then stored in the album of the electronic device.

[0396] In practical applications, those skilled in the art can set the timing of storage or display according to actual conditions, and the embodiments of the present application are not limited thereto.

[0397] In an embodiment of the present application, a corresponding third image is determined based on the restored image and the second image, that is, an image area corresponding to the target object in the second image is restored as the first image, and after obtaining the restored image, it is fused with the second image to obtain a third image after the target object is redrawn.

[0398] In the embodiment of the present application, the method of fusing the restored image and the second image is not specifically limited. As an example, a binary mask image of the first image is obtained, in which 1 represents the first image area, 0 represents the background area, the pixel values ​​of the area with a mask value of 0 keep the original pixel values ​​unchanged, and the pixel values ​​of the area with a mask value of 1 can use an updated pixel value (corresponding to the restored image). A filter band can be set at the junction of 0 and 1 (i.e., the edge of the first image), and the mask value in the filter band can be between 0 and 1 to achieve a seamless fusion effect. Optionally, for the pixels in the filter band, the mask weight can be determined according to the semantics of the pixels to achieve a more refined seamless fusion effect. In practical applications, other restored image fusion methods, such as Poisson image fusion technology, etc., can also be used, which is not limited in the embodiment of the present application.

[0399] Based on at least one of the above embodiments, some process examples of personalized face redrawing are provided.

[0400] In an example, Fig.17 As shown, an example of a process of personalized face redrawing is shown.

[0401] Specifically, it may include:

[0402] Step 17.1: The user uses the mobile phone camera to take a photo or the user selects a photo from the mobile phone album to edit, and obtains an image (second image);

[0403] Step 17.2: Perform face (target object) recognition to determine whether the image is a face image with at least one face (determine the number of target objects). If not, end; if so, proceed to the next step.

[0404] Step 17.3: split the face in the face image into individual face regions;

[0405] Step 17.4: For at least one face region (first image), ask the user whether the face needs to be redrawn. If the user chooses not to, the process ends; if the user chooses to redraw the face (the target object selection instruction input by the user), execute steps 17.5 and 17.7;

[0406] Step 17.5: Providing an image restoration degree adjustment (confirmation) interface;

[0407] Step 17.6: receiving restoration degree information input by the user through the adjustment interface, and determining the image restoration degree information based on the image quality information of the face area and / or the user preference, so as to guide the restoration strength required for face redrawing;

[0408] Step 17.7: Predicting facial attribute information corresponding to the face region, such as age, gender, makeup, and other attribute information;

[0409] Step 17.8: Based on the facial attribute information, determine the restoration style information to guide the restoration direction required for personalized face redrawing;

[0410] Step 17.9: Based on the restoration degree information obtained in step 17.6 and the restoration style information obtained in step 17.8, the face area is restored using the AI ​​network to achieve personalized face redrawing;

[0411] Step 17.10: Output the updated face region.

[0412] For each face region shown in step 17.3, under normal circumstances, the maximum number of faces n (first set threshold) that can be processed can be determined based on the computing power of the electronic device. If the number of faces in the image is greater than n, it can be determined which faces need to be processed. Possible solutions include:

[0413] Solution 1: User selection, for example, providing a selectable interface for users to click or circle.

[0414] Solution 2: The learned neural network model can automatically select based on the state of the face in the photo.

[0415] For the activation condition of face redrawing shown in step 17.4, it is also possible to decide whether to redraw the face based on the initial image quality criterion (the image quality of the image area corresponding to the target object). A possible solution is to use a signal processing method to score by measuring the clarity of the face. This strategy helps save computing resources and improve real-time performance.

[0416] In an example, Fig.18 As shown, another process example of personalized face redrawing is shown.

[0417] Specifically, it may include:

[0418] Step 18.1: Select an image (second image) from the mobile phone album to edit;

[0419] Step 18.2: Perform face recognition to determine whether there is at least one face in the image (determine the number of target objects). If so, proceed to the next step.

[0420] Step 18.3: Split the face in the image into individual face regions;

[0421] Step 18.4: For each face region (first image), detect the image quality score (image quality related value) to determine whether the face is redrawn. If the quality score is > 90 (second set threshold), no processing is performed. If the quality score is < 90, activate face redrawing and execute steps 18.5 and 18.7;

[0422] Step 18.5: Determine the image quality of the face area that needs face redrawing;

[0423] Step 18.6: receiving the restoration degree information controlled by the user, and determining the image restoration degree information (restoration strength) based on the quality information of the face area and the user control result;

[0424] Step 18.7: Predict the facial attribute information corresponding to the face area, such as age, gender, etc.;

[0425] Step 18.8: Determine restoration style information based on face attribute information;

[0426] Step 18.9: Based on the restoration degree information obtained in step 18.6 and the restoration style information obtained in step 18.8, the face area is restored using the AI ​​network to achieve personalized face redrawing;

[0427] Step 18.10: Output the updated face region.

[0428] Optionally, the personalized face redrawing process provided in the embodiment of the present application can be applied to an album image editing solution, and whether to redraw the face is determined based on an image quality measurement standard as an activation condition used in the album.

[0429] Among them, the face redrawing effect of the embodiment of the present application can achieve a significant improvement in image quality after the image is redrawn compared to before the image is redrawn, and can make everyone in the image clear and beautiful.

[0430] In an example, Fig.19 As shown, another process example of personalized face redrawing is shown.

[0431] Specifically, it may include:

[0432] Step 19.1: Take a photo using a mobile phone camera and obtain raw data (second image) before ISP (Image Signal Processing, Image Signal Processing) software (Software, SW);

[0433] Step 19.2: Perform face recognition to determine whether there is a face in the image (determine the number of target objects). If yes, proceed to the next step. If not, save the image to the album.

[0434] Step 19.3: Determine whether there is one face in the image (determine the number of target objects) (third set threshold, which can also be considered as the first set threshold). If yes, execute step 19.4 (real-time processing process, for one face). If not, save the image to the album and then execute step 19.8 (back-end processing process, for multiple faces).

[0435] Step 19.4: Determine whether the face is redrawn (the target object selection instruction input by the user or the image quality of the image area corresponding to the detected target object). If not, save the image to the album. If yes, activate the face (first image) redrawing and execute the next step;

[0436] Step 19.5: Determine restoration degree information and restoration style information of the face image;

[0437] Step 19.6: Based on the restoration degree information and restoration style information, use the AI ​​network to restore the face area to achieve personalized face redrawing;

[0438] Step 19.7: Save the redrawn face image to the photo album;

[0439] Step 19.8: For each face in the image (first image), determine whether the face is redrawn. If not, do not process it. If so, activate face redrawing and execute the next step;

[0440] Step 19.9: Determine restoration degree information and restoration style information of the face image;

[0441] Step 19.10: Based on the restoration degree information and restoration style information, use the AI ​​network to restore the face area to achieve personalized face redrawing;

[0442] Step 19.11: Update the redrawn face image to the album.

[0443] Optionally, the personalized face redrawing process provided in the embodiment of the present application can be applied to a camera photo-taking solution, and whether to redraw the face is determined based on user choice and / or image quality measurement criteria as an activation condition used in the camera.

[0444] It should be emphasized that the purpose of the face redrawing technology provided in at least one of the above embodiments is to restore real and natural facial details, which is dominated by objective factors. It does not improve the visual effect by changing the features of the face (for example, by changing the shape of the face to obtain a "big eyes" or "thin face" effect, or using a simple filter to make the skin smoother), but by interpolating the existing texture prior information to generate high-frequency detail information, and also includes learning a large amount of high-definition face prior knowledge to generate semantically reasonable and real and natural texture details in texture-free areas. In the camera pipeline, face redrawing is located in the front position. In general, the face redrawing technology provided in the embodiments of the present application generates real features rather than changing real features. Among them, the typical camera pipeline of an AI camera can be as follows Fig. 20 As shown, it can be divided into normal mode and night scene mode. For normal mode, the image output by the camera sensor is first processed by demosaicing, and then processed by the ISP software of the normal mode, and then beauty, background blur and other processing can be performed. Optionally, the technical solution of the embodiment of the present application can be applied to the night scene mode. For the night scene mode, the image output by the camera sensor is processed by demosaicing, and then processed by the ISP software of the night scene mode. The solution of the embodiment of the present application can be used for face redrawing, and then beauty, background blur and other processing can be performed. It can be seen that in the camera pipeline, face redrawing is located in the front position. In practical applications, the technical solution of the embodiment of the present application can also be applied to other modes besides the night scene mode, and the embodiment of the present application is not limited here.

[0445] Compared with the prior art, the personalized face redrawing solution provided in the embodiment of the present application can produce natural and realistic textures and achieve true image redrawing. For example, for images taken under low light conditions, the image quality can still be guaranteed.

[0446] Compared with the prior art, the personalized face redrawing solution provided in the embodiment of the present application can restore personalized facial features consistent with the user attributes, such as women's red lips (gender attribute), young people's skin is firmer and smoother than that of the elderly (age attribute), etc.

[0447] Compared with the prior art, the personalized face redrawing solution provided in the embodiment of the present application can achieve different restoration strengths in different areas through restoration degree information in view of the different image qualities in different areas of the image. Among them, the image restoration degree refers to the image restoration strength, and a high restoration degree means that the restored image is clearer and has richer texture details. A low restoration degree means that the restored image texture is more missing, blurry, and noisy. The image restoration degree includes the global restoration degree and the local restoration degree. The local restoration degree refers to the restoration degree of the local area, such as the restoration degree of the eyes, eyebrows, and mouth areas.

[0448] Compared with the prior art, the personalized face redrawing solution provided in the embodiment of the present application uses depth information and brightness information to predict image quality information (see Figure 5a ), which directly reflects the degree of image restoration and can more accurately predict the degree of restoration.

[0449] Compared with the prior art, the personalized face redrawing solution provided in the embodiment of the present application uses the restored style information to extract the depth image features corresponding to the given attributes of the character and uses it as the modulation information (see Fig. 9 ), which can use the complex style information more accurately to guide the recovery direction.

[0450] Compared with the prior art, the personalized face redrawing solution provided in the embodiment of the present application performs adaptive correction on image features of different scales during decoding based on modulation information (see Fig.12 ), the restored face image can generate more realistic and reasonable texture.

[0451] The personalized face redrawing solution provided in the embodiment of the present application can be applied in the mobile phone camera mode. When the user selects the mobile phone's camera function to take a photo, the mobile phone camera will self-detect the people in the photo and perform personalized quality improvement on the face area. It can also be applied to the image editing function of the mobile phone album. When the edited image contains a face, this solution will be activated and the face area will be personalized. Quality improvement.

[0452] The technical solution provided in the embodiments of the present application can be applied to various electronic devices, including but not limited to mobile terminals, smart terminals, etc., such as smart phones, tablet computers, laptops, smart wearable devices (such as watches, glasses, etc.), smart speakers, vehicle terminals, personal digital assistants, portable multimedia players, navigation devices, etc., but not limited to these. It can be understood by those skilled in the art that, in addition to components specifically used for mobile purposes, the structure according to the embodiments of the present application can also be applied to fixed-type terminals, such as digital televisions, desktop computers, etc.

[0453] The technical solution provided in the embodiments of the present application can also be applied to image recovery in a server, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0454] Take smartphones as an example. The camera function of smartphones is an important selling point of smartphones and has become one of the core competitiveness of smartphones of major brands. With the increasing popularity and replacement frequency of smartphones, the camera functions of mobile phones are becoming more and more powerful. AI-based image quality improvement technology has greatly improved the imaging quality of mobile phone cameras.

[0455] In addition, users have increasingly higher demands for camera imaging quality. In addition to imaging quality, they also expect the portraits in the photos to be more personalized and to reflect characteristics consistent with the attributes of the people in the photos, such as women's rosy lips and firm skin.

[0456] The technical solution provided in the embodiments of the present application can meet such demands and introduce more personalized features into mobile phone cameras.

[0457] In some examples, the visual effects of the technical solution provided by the embodiments of the present application may include but are not limited to: restoring the pores of the character's skin to be more consistent with the real gender; restoring the wrinkled skin of the character to be consistent with the real age of humans; restoring the character's texture to be more reasonable without artifacts; by adjusting the global control variable α, the user can globally control the degree of restoration, for example, the user sets α=0.5, the user sets α=0.8, etc., and output images with different degrees of restoration can be obtained; by adjusting the local control variable β 1 , the user can only locally control the degree of eye recovery, for example, the user sets β 1 =0.4, user set β 1 =0.9, etc., can all obtain output images with different degrees of eye restoration.

[0458] An embodiment of the present application also provides an electronic device, which includes a processor and, optionally, may also include a transceiver and / or a memory coupled to the processor, and the processor is configured to execute the steps of the method provided in any optional embodiment of the present application.

[0459] Fig.21 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present invention is shown in FIG. Fig.21 As shown, Fig.21The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0460] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0461] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.21 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0462] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0463] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0464] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0465] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0466] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0467] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0468] The above text and drawings are provided only as examples to help readers understand the present application. They are not intended to and should not be interpreted as limiting the scope of the present application in any way. Although certain embodiments and examples have been provided, based on the contents disclosed herein, it is obvious to those skilled in the art that the embodiments and examples shown can be changed without departing from the scope of the present application, and other similar implementation means based on the technical ideas of the present application are adopted, which also fall within the protection scope of the embodiments of the present application.

Claims

1. A method performed by an electronic device, It is characterized in that include: acquiring a first image; Determining restoration style information of the first image; Based on the restoration style information, an artificial intelligence (AI) network is used to restore the first image to obtain a restored image.

2. The method according to claim 1, It is characterized in that The restoration style information is related to attribute information of a target object in the first image.

3. The method according to claim 1 or 2, It is characterized in that Also includes: Determining restoration degree information of the first image; The restoring the first image using an AI network based on the restoration style information includes: Based on the restoration style information and the restoration degree information, the first image is restored using an AI network.

4. The method according to claim 3, It is characterized in that The restoration degree information is related to at least one of the following: image quality of the first image; and a restoration degree related instruction input by a user.

5. The method according to claim 3 or 4, It is characterized in that The determining the restoration degree information of the first image includes: Providing a restoration degree confirmation interface for the first image; In response to a restoration degree related instruction input by a user through the restoration degree confirmation interface, second restoration degree information of the first image is determined.

6. The method according to claim 5, It is characterized in that The providing of the restoration degree confirmation interface of the first image includes: determining image quality information of the first image; Based on the image quality information, determining corresponding first restoration degree information; Based on the first restoration degree information, providing a restoration degree confirmation interface for the first image; The determining, in response to a restoration degree related instruction input by a user through the restoration degree confirmation interface, second restoration degree information of the first image includes: In response to an adjustment instruction for the first restoration degree information input by a user through the restoration degree confirmation interface, second restoration degree information of the first image is determined.

7. The method according to claim 3 or 4, It is characterized in that The determining the restoration degree information of the first image includes: determining image quality information of the first image; Based on the image quality information, corresponding first restoration degree information is determined.

8. The method according to claim 5 or 6, It is characterized in that The recovery degree confirmation interface includes at least one of the following: a global recovery degree confirmation interface and at least one local recovery degree confirmation interface, wherein the at least one local recovery degree confirmation interface corresponds to at least one local area of ​​the target object.

9. The method according to claim 8, It is characterized in that The determining, in response to a restoration degree related instruction input by a user through the restoration degree confirmation interface, second restoration degree information of the first image includes: The second recovery degree information of the first image is determined in response to at least one of the following instructions: a global recovery degree related instruction input by the user through the global recovery degree confirmation interface, and at least one local recovery degree related instruction input by the user through the at least one local recovery degree confirmation interface.

10. The method according to claim 6, It is characterized in that The determining, based on the image quality information, corresponding first restoration degree information includes: Based on the image quality information, determining first global restoration degree information; Based on the image quality information and the image mask of at least one local area, determining first local restoration degree information respectively corresponding to the at least one local area; The providing a restoration degree confirmation interface of the first image based on the first restoration degree information includes: Based on the first global recovery degree information, providing a global recovery degree confirmation interface in the recovery degree confirmation interface; Based on the first local recovery degree information respectively corresponding to the at least one local area, respectively providing at least one corresponding local recovery degree confirmation interface in the recovery degree confirmation interface; The determining, in response to an adjustment instruction for the first restoration degree information input by a user through the restoration degree confirmation interface, second restoration degree information of the first image comprises: The second recovery degree information of the first image is determined in response to at least one of the following instructions: an adjustment instruction for the first global recovery degree information input by the user through the global recovery degree confirmation interface, and an adjustment instruction for at least one first local recovery degree information input by the user through the at least one local recovery degree confirmation interface.

11. The method according to claim 9 or 10, It is characterized in that The determining, in response to a restoration degree related instruction input by a user through the restoration degree confirmation interface, second restoration degree information of the first image includes: In response to a global recovery degree related instruction input by a user through the global recovery degree confirmation interface, determining second global recovery degree information; In response to at least one local recovery degree related instruction input by the user through the at least one local recovery degree confirmation interface, determining corresponding at least one second local recovery degree information; The second global recovery degree information and the at least one second local recovery degree information are merged to obtain the second recovery degree information.

12. The method according to claim 6 or 7, It is characterized in that The determining the image quality information of the first image includes: Acquiring depth information and / or brightness information of the first image; Image quality information of the first image is determined based on the brightness information and / or the depth information.

13. The method according to claim 12, It is characterized in that The determining, based on the brightness information and / or the depth information, image quality information of the first image includes: Obtaining a first image feature based on the brightness information and / or the depth information; Extracting features from the first image to obtain features of a second image; Image quality information of the first image is determined based on the first image feature and the second image feature.

14. The method according to claim 13, It is characterized in that Obtaining a first image feature based on the brightness information and the depth information includes: Extracting features from the brightness information and the depth information respectively to obtain brightness features and depth features; Based on the brightness feature, determining a first weight map for compensating the depth feature; Performing weighted processing on the depth feature according to the first weight map to obtain a compensated depth feature; The brightness feature and the compensated depth feature are fused to obtain the first image feature.

15. The method according to claim 13 or 14, It is characterized in that The determining the image quality information of the first image based on the first image feature and the second image feature includes: Based on the first image feature, determining a second weight map, wherein the second weight map indicates a correlation between a degree of image quality degradation of the first image and the depth information and / or the brightness information; Processing the second image feature through a multi-head self-attention network to obtain a query feature, a key feature, and a value feature of the second image feature; Performing weighted processing on the value feature according to the second weight map; Image quality information of the first image is determined based on the weighted value feature, the query feature, and the key feature.

16. The method according to any one of claims 3 to 15, It is characterized in that Also includes: Repeat the following update process at least once: Determining a peak signal-to-noise ratio value of the first image according to the currently obtained restored image; Determining a recovery degree information update parameter according to the peak signal-to-noise ratio value; Based on the recovery degree information update parameter, the recovery degree information is updated; Based on the updated restoration degree information, the first image is restored to obtain an updated restored image.

17. The method according to any one of claims 1 to 16, It is characterized in that The determining the restoration style information of the first image includes: Determining attribute information of a target object in the first image according to the first image; Based on the attribute information, restoration style information of the first image is determined.

18. The method according to claim 17, It is characterized in that The determining, based on the attribute information, restoration style information of the first image includes: Encoding the attribute information into an attribute vector; The attribute vector is mapped to the restored style information through a multi-layer perception network.

19. The method according to any one of claims 1 to 18, It is characterized in that The restoring the first image using an AI network based on the restoration style information includes: Based on the first image, extracting corresponding third image features; extracting, based on the restoration style information, a fifth image feature corresponding to the attribute information of the target object in the first image from the third image feature; A restored image is obtained based on the third image feature and the fifth image feature.

20. The method according to claim 19, It is characterized in that If the restoration degree information of the first image is determined, extracting corresponding third image features based on the first image includes: Based on the first image and the restoration degree information of the first image, corresponding third image features are extracted.

21. The method according to claim 19 or 20, It is characterized in that The step of extracting, based on the restoration style information, a fifth image feature corresponding to the attribute information of the target object in the first image from the third image feature comprises: fusing the restored style information with the attribute information to obtain a fourth image feature; The fifth image feature is obtained based on the fourth image feature and the attribute information.

22. The method according to any one of claims 19 to 21, It is characterized in that The step of obtaining a restored image based on the third image feature and the fifth image feature includes: determining an image feature correction amount corresponding to the attribute information according to a correlation between the third image feature and the fifth image feature; The third image feature is corrected according to the image feature correction amount to obtain the restored image.

23. The method according to claim 22, It is characterized in that The third image feature includes at least one scale, and determining the image feature correction amount corresponding to the attribute information according to the correlation between the third image feature and the fifth image feature includes: For each scale of the third image feature, determining an image feature correction amount of the scale corresponding to the attribute information according to a correlation between the third image feature of the scale and the fifth image feature of the corresponding scale; The correcting the third image feature according to the image feature correction amount to obtain the restored image includes: Correction processing is performed on the corresponding third image features according to the image feature correction amount of the at least one scale to obtain the restored image.

24. The method according to any one of claims 1 to 23, It is characterized in that The acquiring of the first image comprises: acquiring a second image; detecting a target object in the second image; The first image is determined based on at least one of the following: The number of detected target objects; Image quality of the image area corresponding to the detected target object; The target object selection instruction input by the user.

25. The method according to claim 24, It is characterized in that The acquiring of the second image includes at least one of the following methods: Acquire a second image based on the image acquired in real time by the acquisition device; A second image is acquired based on the image stored in the electronic device.

26. The method according to claim 24 or 25, It is characterized in that The determining of the first image comprises: When the number of detected target objects is not less than a first set threshold, a first image is determined in the image area corresponding to the target object based on at least one of image quality of the image area corresponding to the detected target object and a target object selection instruction input by a user.

27. The method according to claim 26, It is characterized in that Determining a first image in the image region corresponding to the target object based on the image quality of the detected image region corresponding to the target object includes: determining an image quality of an image region corresponding to at least one target object; An image region whose image quality related value is lower than a second set threshold is determined as a first image.

28. The method according to claim 24, It is characterized in that If the second image is an image acquired in real time based on an acquisition device, the method further includes: When the number of detected target objects is not less than a third set threshold, the second image is stored.

29. The method according to any one of claims 24 to 28, It is characterized in that Also includes: determining a corresponding third image based on the restored image and the second image; The third image is stored or displayed.

30. The method according to any one of claims 1 to 29, It is characterized in that The first image is a facial image.

31. The method according to any one of claims 2, 17-23, It is characterized in that The attribute information includes at least one of the following: age information, gender information, skin color information, hair information, makeup information, and glasses information.

32. An electronic device comprising a memory, a processor and a computer program stored in the memory, It is characterized in that The processor executes the computer program to implement the method according to any one of claims 1 to 31.

33. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 31 is implemented.

34. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 31 is implemented.