Method, device and equipment for face enhancement

Abstract

The present application provides a method and device for face enhancement, which relate to the field of image processing technology. The method comprises: using a face feature extraction module in a face enhancement model to split an original face image based on face key point information to obtain multiple key point feature vectors and face feature vectors, wherein the face feature extraction module performs objective similarity constraints on the face feature vectors; using a face enhancement module in the face enhancement model to generate an enhanced face image corresponding to the original face image, wherein the face enhancement module performs redundant similarity constraints on face information of the enhanced face image; inputting the enhanced face image into a face modification module in the face enhancement model, wherein the face modification module adjusts face modification perception information to generate a high-realism face enhanced image, wherein the face modification module performs sensory similarity constraints on the face modification perception information to improve the clarity of the photographed face while ensuring the authenticity of the face.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and equipment for face enhancement. Background Art

[0002] Face enhancement technology is a technology based on computer vision and image processing, which aims to improve the quality of facial images, enhance the visual effects of faces, and enhance facial feature information to meet the needs of various practical applications.

[0003] Face enhancement technology usually detects and locates the input face image, determines the position and boundaries of the face in the image, and then uses basic image processing techniques such as image filtering, color correction, contrast enhancement, and sharpening to improve the overall quality of the image.

[0004] However, this method only enhances the characteristic information of the face, without considering the authenticity of the enhanced face, and does not solve the abnormal problems such as low recognition rate, fat and thin changes, templateization, and loss of age sense generated during the face enhancement process.

[0005] Therefore, there is an urgent need for a face enhancement technology to solve the technical problem of poor authenticity of the face after face enhancement. Summary of the Invention

[0006] The purpose of this application is to provide a method for face enhancement to solve the technical problem of poor authenticity of the enhanced face.

[0007] In a first aspect, the present application provides a method for face enhancement, comprising:

[0008] Obtain an original face image, and use the face feature extraction module in the face enhancement model to split the original face image based on facial key point information to obtain multiple key point feature vectors and face feature vectors, wherein the face feature extraction module imposes objective similarity constraints on the face feature vectors;

[0009] The face enhancement module in the face enhancement model generates an enhanced face image corresponding to the original face image based on the face feature vector, wherein the face enhancement module imposes redundant similarity constraints on the face information of the enhanced face image;

[0010] The enhanced face image is input into the face modification module in the face enhancement model, and the face modification module adjusts the face modification perception information of the enhanced face image to generate a high-realism face enhanced image, wherein the face modification module imposes sensory similarity constraints on the face modification perception information.

[0011] Optionally, splitting the original face image based on the face key point information by a face feature extraction module in the face enhancement model to obtain a plurality of key point feature vectors and face feature vectors, including: inputting the original face image into the face feature extraction module, and the face feature extraction module splitting the original face image into a plurality of key point images based on the face key point information;

[0012] The first deep learning network computing layer in the face feature extraction module processes multiple key point images and extracts key point feature vectors corresponding to each key point image based on the channel attention mechanism;

[0013] Multiple key point feature vectors and the original face image are input into the second deep learning network operation layer of the face feature extraction module to generate a face feature vector.

[0014] Optionally, the facial key point information includes at least two of the following information:

[0015] Left eye, right eye, nose, mouth, left ear, right ear;

[0016] The facial feature extraction module imposes objective similarity constraints on facial feature vectors, including:

[0017] Determine a first difference between facial key point information of the true face image and a facial feature vector corresponding to the original face image;

[0018] Determine a second difference between a key point feature vector of the true face image and a key point feature vector corresponding to the original face image;

[0019] The first difference and the second difference are constrained by the loss function of the facial feature extraction module to achieve objective similarity constraints on the facial information of the facial feature vector.

[0020] Optionally, the face enhancement module performs redundant similarity constraints on the face information of the enhanced face image, including:

[0021] Calculating a first Euclidean distance between the enhanced face image and a ground-truth face image of the same face, and a second Euclidean distance between the enhanced face image and ground-truth face images of other faces;

[0022] Based on the first Euclidean distance and the second Euclidean distance, the redundant similarity constraint of the enhanced face image is realized through the redundant face similarity loss function.

[0023] Optionally, the face modification module performs sensory similarity constraints on the face modification perception information, including:

[0024] The face modification module uses a sliding window method to sequentially obtain local areas of the high-realism face enhancement image, and combines it with a mean filter to constrain the local brightness difference between the high-realism face enhancement image and the true face image of the same face;

[0025] The face modification module uses intermediate frequency filters and high frequency filters based on Manhattan distance to achieve the facial detail loss constraint between the high-realism face enhanced image and the true face image of the same face.

[0026] Optionally, the face modification module performs sensory similarity constraints on the face modification perception information, including:

[0027] The face modification module implements the perceptual similarity constraint on the face modification perception information through the following formula: Characterizing the sensory similarity constraints of the face modification module:

[0028]

[0029] in, represents a high-realism face enhanced image, y represents the ground-truth face image corresponding to the same face in the high-realism face enhanced image, * represents the convolution operation, A represents the mean filter, k represents the local area of the high-realism face enhanced image, B represents the intermediate frequency filter designed for facial features, and C represents the high frequency filter designed for facial features. The above B and C are used to extract skin detail information.

[0030] Optionally, the method for face enhancement further includes:

[0031] The face enhancement model achieves overall consistency constraints on the original image by weighted superposition of the loss function corresponding to the objective similarity constraint, the loss function corresponding to the redundant similarity constraint, the loss function corresponding to the sensory similarity constraint, and the Manhattan distance between the high-realism face enhanced image and the true face image of the same face.

[0032] Optionally, the method for face enhancement further includes:

[0033] A true face image of the same face as the original face image and the enhanced face image is obtained by capturing the image sensor; and / or

[0034] Through the semi-supervised learning model, a true face image that belongs to the same face as the original face image and the enhanced face image is obtained.

[0035] On the other hand, an embodiment of the present application further provides a device for face enhancement, the device comprising:

[0036] A facial feature extraction module is used to obtain an original facial image and split the original facial image based on facial key point information to obtain multiple key point feature vectors and facial feature vectors. The facial feature extraction module imposes objective similarity constraints on the facial feature vectors.

[0037] A face enhancement module is used to generate an enhanced face image corresponding to the original face image based on the face feature vector, wherein the face enhancement module performs redundant similarity constraints on the face information of the enhanced face image;

[0038] The face modification module is used to adjust the face modification perception information of the enhanced face image and generate a high-fidelity face enhanced image, wherein the face modification module imposes sensory similarity constraints on the face modification perception information.

[0039] On the other hand, a device for face enhancement is provided, the device for face enhancement comprising:

[0040] processor;

[0041] A memory stores computer-readable instructions, which, when executed by a processor, implement the above-mentioned method for face enhancement.

[0042] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0043] The embodiment of the present application adopts a method that can both enhance the face and improve the authenticity of the enhanced face. Through the constraint of objective similarity, the consistency of the information parts of the key points of the face is guaranteed. Through the constraint of redundant similarity, not only the naked eye clarity of the face image is improved through face enhancement, but also the template effect problem after face enhancement is avoided. The sensory authenticity is improved through the constraint of sensory similarity. The portrait enhanced by this method can also improve the comparison rate for facial feature comparison, eliminating the problems of inconsistent facial features, templated facial features, changes in fatness and thinness, and disappearance of age after face enhancement. Applying this method to images captured by a camera can improve the clarity of the captured face while ensuring the authenticity of the face. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 This is a processing flow chart of a face enhancement model provided in an embodiment of the present application;

[0046] Figure 2 This is a flowchart of a method for face enhancement provided by an embodiment of the present application;

[0047] Figure 3 This is a flowchart of a method for splitting and fusing facial key point information provided by an embodiment of the present application;

[0048] Figure 4 Schematic diagram of a method for generating a channel feature vector provided in an embodiment of the present application;

[0049] Figure 5 This is a schematic diagram of the processing process of a face enhancement module provided in an embodiment of the present application;

[0050] Figure 6 This is a schematic diagram of the processing process of a face modification module provided in an embodiment of the present application;

[0051] Figure 7 This is a structural block diagram of a device for face enhancement provided by an embodiment of the present application;

[0052] Figure 8 This is a schematic diagram of the structure of a device for face enhancement provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The present application will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings, but these embodiments do not limit the present application. Structural, methodological, or functional changes made by ordinary technicians in this field based on these embodiments are included in the scope of protection of the present application.

[0054] Due to changes in ambient lighting, weather, and object motion, the images captured by cameras can be noisy and blurry. In particular, captured facial images often have low recognition. This low recognition significantly limits the utility of captured facial images, making them difficult to apply in other areas, such as age recognition.

[0055] To address this issue, face enhancement is usually used to increase the recognizability of images captured by cameras. However, facial images enhanced using face enhancement technology often lack authenticity. For example, faces often have brightness changes that cause distortion of facial fatness or thinness, and faces are too smooth, resulting in the loss of skin information such as facial wrinkles, skin spots, and skin moles.

[0056] In order to solve the above problems, the embodiment of the present application provides an end-to-end modular face enhancement model to improve the authenticity of face enhancement. Figure 1As shown. The face enhancement model of the embodiment of the present application is mainly composed of a face feature extraction module, a face enhancement module and a face modification module. The face enhancement model of the embodiment of the present application is an end-to-end deep learning model, which splits the deep learning network operation layer into different modules. The functions of different modules can be constrained by different loss functions, and the effects of different modules can be targeted to achieve the purpose of improving the authenticity of the face image output by the overall face enhancement model.

[0057] For example, Figure 2 As shown, an embodiment of the present application provides a method for face enhancement, and the method for face enhancement includes:

[0058] S201. Obtain an original face image, and use a face feature extraction module in a face enhancement model to split the original face image based on face key point information to obtain multiple key point feature vectors and face feature vectors, wherein the face feature extraction module imposes objective similarity constraints on the face feature vectors.

[0059] Among them, the original face image is a low-definition portrait.

[0060] The original face image is split by facial key point information, and the facial feature vector is generated by the convolution operation of the deep learning model based on the split facial key point information.

[0061] This face enhancement model inputs a low-resolution portrait into a facial feature extraction module to extract facial key point information from the low-resolution image. A face enhancement module then generates a higher-resolution enhanced face image based on the low-resolution image. The higher-resolution enhanced face image is then refined based on the facial key point information from the low-resolution image, ultimately yielding a highly realistic enhanced face image. This approach improves the low face authenticity comparison success rate of the face enhancement model, a key evaluation metric. A significant decrease in the comparison success rate after face enhancement can reduce the practicality of face enhancement algorithms. During the learning and sharpening process, the face enhancement model is guided by the input original face image and the ground-truth face image corresponding to the face in the original face image. Different loss functions are used to constrain different components of the face enhancement model to achieve different functions. The ground-truth face image is either a face image corresponding to the current portrait captured by a high-resolution camera or a high-resolution face image corresponding to the current portrait generated by a specific face enhancement algorithm.

[0062] For example, the facial feature extraction module can improve the accuracy of the enhanced facial key point information and thus improve the enhanced face comparison success rate by, but not limited to, the following methods:

[0063] Since facial images of different sizes have less facial feature information and are more blurred in small-sized images after being scaled to a uniform size, in order to improve the adaptability of the face enhancement model to faces of different sizes and facial features, the original facial image can be first split into facial features and then subjected to subsequent convolution operations together with the original facial image, thereby improving the comparison success rate of the enhanced face.

[0064] Exemplarily, the facial key point information includes at least two of the following information:

[0065] Left eye, right eye, nose, mouth, left ear, right ear.

[0066] For example, taking the facial key point information including the left eye, right eye, nose and mouth as an example, the splitting and fusion method of the facial key point information can be but is not limited to the following: Figure 3 As shown in the figure, the facial feature extraction module of the face enhancement model splits the original face image into four parts: left eye, right eye, nose and mouth according to the facial key point information, as shown in the figure. Figure 4 As shown, these four parts of facial key point information are processed through convolutional layers to extract channel-wise key point feature vectors. These extracted key point feature vectors are then integrated into the subsequent facial feature extraction module to improve the authenticity of the facial feature extraction module's reconstruction of important facial parts. This operation increases the model's attention to facial features, while refining the key point information of facial features, thereby improving the face comparison rate. This module ensures the effectiveness of feature extraction through a loss function.

[0067] Wherein, exemplarily, the original face image is split based on the face key point information by the face feature extraction module in the face enhancement model to obtain multiple key point feature vectors and face feature vectors, including: inputting the original face image into the face feature extraction module, and the face feature extraction module splitting the original face image into multiple key point images according to the face key point information;

[0068] The first deep learning network computing layer in the face feature extraction module processes multiple key point images and extracts key point feature vectors corresponding to each key point image based on the channel attention mechanism;

[0069] Multiple key point feature vectors and the original face image are input into the second deep learning network operation layer of the face feature extraction module to generate a face feature vector.

[0070] The channels in the above-mentioned channel attention mechanism can be channels classified according to the five senses or channels classified according to key areas.

[0071] The above-mentioned deep learning network operation layer may include, but is not limited to, a convolutional layer, a probability distribution layer, etc.

[0072] In some embodiments, the facial feature extraction module is constrained by the objective similarity of facial information.

[0073] Exemplarily, the facial feature extraction module imposes objective similarity constraints on facial feature vectors, including:

[0074] Determine a first difference between facial key point information of the true face image and a facial feature vector corresponding to the original face image;

[0075] Determine a second difference between a key point feature vector of the true face image and a key point feature vector corresponding to the original face image;

[0076] The first difference and the second difference are constrained by the loss function of the facial feature extraction module to achieve objective similarity constraints on the facial information of the facial feature vector.

[0077] For example, a loss function of a face feature extraction module may be the following loss function:

[0078] ,in, Represents the facial feature vector corresponding to the original face image, Characterize the key point information of the label face, Characterize the left eye feature vector corresponding to the original face image, Characterize the left eye feature vector of the label, Represents the right eye feature vector corresponding to the original face image, Characterize the label right eye feature vector, The nose feature vector representing the original face image, Characterize the label nose feature vector, Characterize the mouth feature vector corresponding to the original face image, Characterize the mouth feature vector of the label, Characterizes the loss function of the facial feature extraction module. The above-mentioned labeled facial key point information can be the facial key point information in the manually calibrated true face image, or the facial key point information in the true face image obtained through semi-supervised learning. The labeled left eye feature vector can be the left eye feature vector in the manually calibrated true face image, or the left eye feature vector in the true face image obtained through semi-supervised learning. The labeled right eye feature vector can be the right eye feature vector in the manually calibrated true face image, or the right eye feature vector in the true face image obtained through semi-supervised learning. The labeled nose feature vector can be the nose feature vector in the manually calibrated true face image, or the nose feature vector in the true face image obtained through semi-supervised learning. The labeled mouth feature vector can be the mouth feature vector in the manually calibrated true face image, or the mouth feature vector in the true face image obtained through semi-supervised learning. The above process can optimize the facial feature extraction module based on the rough facial features segmentation method, and improve the objective accuracy of the facial feature vector by combining the segmented facial features and the loss function of the facial feature extraction module.

[0079] S202, using the face enhancement module in the face enhancement model, based on the face feature vector, generates an enhanced face image corresponding to the original face image, wherein the face enhancement module imposes redundant similarity constraints on the face information of the enhanced face image. The processing of the face enhancement module can be as follows: Figure 5 As shown in the figure, the face enhancement module obtains the face feature vector from the face feature extraction module. The face feature vector is first convolved by the deep learning network operation layer and then reconstructed into an enhanced face image ( Figure 5 The face enhanced by the face enhancement model often has a template effect. The template effect refers to the high degree of similarity between the facial features of different faces. To address the problem of high facial feature similarity, the embodiment of the present application performs end-to-end training of the face enhancement model based on a similarity loss function between the same face and different faces.

[0080] Exemplarily, the face enhancement module performs redundant similarity constraints on the face information of the enhanced face image, including:

[0081] Calculating a first Euclidean distance between the enhanced face image and a ground-truth face image of the same face, and a second Euclidean distance between the enhanced face image and ground-truth face images of other faces;

[0082] Based on the first Euclidean distance and the second Euclidean distance, the redundant similarity constraint of the enhanced face image is realized through the redundant face similarity loss function.

[0083] Exemplarily, the redundant face similarity loss function includes:

[0084] ,in, Represents the enhanced face image, y represents the true face image corresponding to the face in the enhanced face image, 、 、 、…… Represents the true face image of other faces, S represents the structural similarity function SSIM between images, Characterizes the redundant face similarity loss function, which can be used to calculate the similarity loss function between the same face and different faces.

[0085] In an optional embodiment of the present application, S may also represent the Euclidean distance between facial key points of two images. The algorithm represented by S only needs to be able to calculate the structural similarity between two facial images, and is not limited here.

[0086] Here, by comparing the similarity between the true face and the enhanced face image, the template effect in the enhanced face image can be removed. The structural similarity algorithm can be used to remove the template effect mainly based on the structural features of the face, which is more efficient and accurate.

[0087] S203. Input the enhanced face image into the face modification module in the face enhancement model, and the face modification module adjusts the face modification perception information of the enhanced face image to generate a high-realism face enhanced image, wherein the face modification module imposes sensory similarity constraints on the face modification perception information.

[0088] The processing of the face modification module can be as follows Figure 6 As shown, the face modification module improves the facial brightness and contour clarity of the enhanced facial image obtained from the face enhancement module after the convolution operation of the deep learning network operation layer, thereby obtaining a high-fidelity face enhanced image with higher authenticity. The role of this part can include improving the sensory authenticity of the face enhancement model, because the face enhanced by the face enhancement model often has problems with facial brightness changes and excessive facial smoothness. The problem of facial brightness changes refers to the change in the fatness and thinness of the face caused by brightness changes. The problem of excessive facial smoothness refers to the loss of skin information such as facial wrinkles, skin spots and skin moles. Excessive facial smoothness will cause the face to subjectively lose its sense of age and reality, thereby reducing the naturalness of the face. In order to solve the problems of facial brightness changes and excessive facial smoothness, the face modification module of the embodiment of the present application can be used for end-to-end training of the face enhancement model through a loss function for sensory similarity.

[0089] In some embodiments, the face modification module performs sensory similarity constraints on the face modification perception information, which may include:

[0090] The face modification module uses a sliding window method to sequentially obtain local areas of the high-realism face enhancement image, and combines it with a mean filter to constrain the local brightness difference between the high-realism face enhancement image and the true face image of the same face;

[0091] The face modification module uses intermediate frequency filters and high frequency filters based on Manhattan distance to achieve the facial detail loss constraint between the high-realism face enhanced image and the true face image of the same face.

[0092] Exemplarily, the face modification module imposes sensory similarity constraints on the face modification perception information, including:

[0093] The face modification module implements the perceptual similarity constraint on the face modification perception information through the following formula: Characterizing the sensory similarity constraints of the face modification module:

[0094]

[0095] in, represents a high-fidelity face enhancement image, y represents the true face image corresponding to the face in the enhanced face image, * represents the convolution operation, A represents the mean filter, Characterizes the loss of brightness and darkness changes of the face, which can be The brightness difference between the local regions and y is the Euclidean distance of the convolution operation A, where the size of each local region is k, and the local regions are obtained in sequence in a sliding window manner. B represents the intermediate frequency filter designed for facial features, and C represents the high frequency filter designed for facial features. B and C are used to extract skin detail information.

[0096]

[0097] Characterization determination After the Manhattan distance of the local area between and y is calculated with the convolution operation of B and C respectively, the skin detail information extracted based on the Manhattan distance is used as the face detail loss. This is the loss function for the face enhancement module. Using a mean filter in this module's loss function prevents the impact of extreme facial brightness on the solution and prevents recognition errors caused by strong light and shadows. This loss function, based on brightness and facial details, addresses issues such as weight changes and loss of age in facial skin detail after face enhancement.

[0098] Since the face enhancement model is a deep learning network model, its learning process is end-to-end learning. In addition to ensuring the effect of each module, it is also necessary to ensure the consistency of the overall effect. The overall effect can be achieved by high-fidelity face enhancement images. Calculate the Manhattan distance with the clear label image y to get the loss function L ( , y).

[0099] Optionally, the face enhancement model ensures the overall consistency between the enhanced image and the original image through the following loss function, where the loss function includes:

[0100] ,in, Characterize the loss function of the face modification module, Characterize the loss function of the face enhancement module, Characterize the loss function of the face feature extraction module, The loss function that characterizes the overall effect of the face enhancement model is obtained by calculating the Manhattan distance between the enhanced face image x and the true face image y corresponding to the face in the enhanced face image. Characterization The weight of Characterization The weight of Characterization The weight of Represents the weight of L.

[0101] The value of each weight can be set by an empirical value, or the initial value of each weight can be preset. During the training process of the above-mentioned face enhancement model, each weight is updated to obtain a more realistic face enhancement effect.

[0102] Exemplarily, the method for face enhancement further includes a face enhancement model training process. In one embodiment, the face enhancement model training process includes:

[0103] An original face enhancement model and training data are obtained, wherein the training data includes a low-definition face image and a true face image corresponding to the low-definition face image. The low-definition face image is used as input data of the original face enhancement model, and the true face image corresponding to the low-definition face image is used as output data of the original face enhancement model. The original face enhancement model is trained to obtain a face enhancement model.

[0104] For example, after setting the original face enhancement model, the loss function of each module in the original face enhancement model, and the loss function of the overall model of the original face enhancement model, the original face enhancement model can be trained based on the training data. During the training process, the original face enhancement model is constrained by the loss function of each module in the original face enhancement model and the loss function of the overall model of the original face enhancement model to obtain the final face enhancement model.

[0105] The loss function of each module in the original face enhancement model and the loss function of the overall model of the original face enhancement model can be adopted from the previous article. , , , ,and The loss function shown in Figure 2 is not described here.

[0106] For example, the method provided in the embodiment of the present application may further include:

[0107] A true face image of the same face as the original face image and the enhanced face image is obtained by capturing the image sensor; and / or

[0108] Through the semi-supervised learning model, a true face image that belongs to the same face as the original face image and the enhanced face image is obtained.

[0109] The embodiment of the present application can adopt an end-to-end learning method of the deep learning network operation layer. By designing different loss functions, the face enhancement model can sequentially complete the functions of face feature extraction, face enhancement and face modification, thereby solving the problems of facial features being templated after face enhancement by other face enhancement methods, the disappearance of the sense of age such as facial wrinkles and skin moles, changes in facial fatness and thinness, and low comparison rate, thereby improving the authenticity of low-definition face enhancement.

[0110] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0111] The embodiment of the present application adopts a method that can both enhance the face and improve the authenticity of the enhanced face. Through the constraint of objective similarity, the consistency of the information parts of the key points of the face is guaranteed. Through the constraint of redundant similarity, not only the naked eye clarity of the face image is improved through face enhancement, but also the template effect problem after face enhancement is avoided. The sensory authenticity is improved through the constraint of sensory similarity. The portrait enhanced by this method can also improve the comparison rate for facial feature comparison, eliminating the problems of inconsistent facial features, templated facial features, changes in fatness and thinness, and disappearance of age after face enhancement. Applying this method to images captured by a camera can improve the clarity of the captured face while ensuring the authenticity of the face.

[0112] like Figure 7 As shown, the present invention also provides a device for face enhancement, the device comprising:

[0113] The facial feature extraction module 701 is used to obtain an original facial image, split the original facial image based on facial key point information to obtain multiple key point feature vectors and facial feature vectors, wherein the facial feature extraction module imposes objective similarity constraints on the facial feature vectors;

[0114] A face enhancement module 702 is configured to generate an enhanced face image corresponding to the original face image based on the face feature vector, wherein the face enhancement module imposes redundant similarity constraints on the face information of the enhanced face image;

[0115] The face modification module 703 is used to adjust the face modification perception information of the enhanced face image to generate a high-fidelity face enhanced image, wherein the face modification module imposes sensory similarity constraints on the face modification perception information.

[0116] Figure 8 FIG. 1 is a structural diagram of a device for face enhancement provided in an embodiment of the present application. Figure 8 As shown, exemplarily, the device 810 for face enhancement may include a first processor 8001.

[0117] Exemplarily, the device 810 for face enhancement may further include a memory 8002 and a transceiver 8003 .

[0118] The first processor 8001, the memory 8002 and the transceiver 8003 may be connected via a communication bus.

[0119] The following combination Figure 8 The example of FIG. 8 is used to introduce various components of the device 810 for face enhancement:

[0120] The device 810 for face enhancement may include the following components: the first processor 8001 may be a single processor or a collective term for multiple processing elements. For example, the first processor 8001 may be one or more central processing units (CPUs), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0121] Exemplarily, the first processor 8001 can perform various functions of the device 810 for face enhancement by running or executing a software program stored in the memory 8002 and calling data stored in the memory 8002.

[0122] In a specific implementation, as an embodiment, the first processor 8001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.

[0123] As an optional embodiment, the device 810 for face enhancement may also include multiple processors, such as Figure 8 1 and 2. The first processor 8001 and the second processor 8004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0124] Among them, the memory 8002 is used to store the software program that executes the solution of the present invention, and is controlled by the first processor 8001 to execute. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0125] Exemplarily, the memory 8002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 8002 may be integrated with the first processor 8001 or exist independently and accessed through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 8001, which is not specifically limited in this embodiment of the present application.

[0126] The transceiver 8003 is used to communicate with a network device or a terminal device.

[0127] For example, the transceiver 8003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0128] For example, the transceiver 8003 may be integrated with the first processor 8001 or may exist independently and communicate with the first processor 8001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 8001, which is not specifically limited in this embodiment of the present application.

[0129] In addition, the technical effects of the device 810 for face enhancement can refer to the technical effects of the method for face enhancement in the above-mentioned method embodiment, and will not be repeated here.

[0130] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), but may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0131] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0132] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. A computer program product comprises one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions according to the embodiments of the present application are fully or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via infrared, microwave, or other means. A computer-readable storage medium can be any available medium accessible by a computer, or a data storage device such as a server or data center that contains a collection of one or more available media. Available media can include magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), etc.

[0133] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0134] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0135] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0136] The communication interface is used for communication between the above-mentioned electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0137] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0138] Each embodiment in this specification is described in a related manner. Similar portions between the embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so their description is relatively simple. For related portions, refer to the description of the method embodiments.

[0139] The above is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application. Based on the same inventive concept, an embodiment of the present application also provides an electronic device, including: at least one memory and at least one processor, the at least one memory stores executable code, and the at least one processor is used to execute the executable code in the at least one memory to implement the above-mentioned image restoration model training method, and / or image restoration method.

Claims

1. A method for face enhancement, characterized in that: The method comprises: Obtain an original face image, and use a face feature extraction module in a face enhancement model to split the original face image based on face key point information to obtain multiple key point feature vectors and face feature vectors, wherein the face feature extraction module imposes objective similarity constraints on the face feature vectors, including: Determine a first difference between facial key point information of a true face image and the facial feature vector corresponding to the original face image; Determine a second difference between the key point feature vector of the true face image and the key point feature vector corresponding to the original face image; Constraining the first difference and the second difference by using the loss function of the facial feature extraction module to achieve objective similarity constraint on the facial information of the facial feature vector; Generate an enhanced face image corresponding to the original face image based on the face feature vector by a face enhancement module in a face enhancement model, wherein the face enhancement module imposes redundant similarity constraints on face information of the enhanced face image, including: Calculating a first Euclidean distance between the enhanced face image and a true face image of the same face, and a second Euclidean distance between the enhanced face image and the true face images of other faces; Based on the first Euclidean distance and the second Euclidean distance, redundant similarity constraints on the enhanced facial image are implemented by a redundant face similarity loss function, wherein the redundant face similarity loss function includes: ,in, Characterize the redundant face similarity loss function, Represents the enhanced face image, y represents the true face image corresponding to the face in the enhanced face image, 、 、 、…… Represents the true face image of other faces, S represents the structural similarity function SSIM between images; Inputting the enhanced face image into a face modification module in the face enhancement model, and adjusting face modification perception information of the enhanced face image by the face modification module to generate a high-fidelity face enhanced image, wherein the face modification module imposes sensory similarity constraints on the face modification perception information, including: The face modification module sequentially obtains local areas of the highly realistic face enhancement image using a sliding window method, and combines a mean filter to constrain the local brightness difference between the highly realistic face enhancement image and the true face image of the same face; The face modification module uses a medium frequency filter and a high frequency filter based on the Manhattan distance to implement the facial detail loss constraint between the high-realism face enhanced image and the true face image of the same face.

2. The method for face enhancement according to claim 1, characterized in that: The face feature extraction module in the face enhancement model splits the original face image based on the face key point information to obtain multiple key point feature vectors and face feature vectors, including: inputting the original face image into the face feature extraction module, and the face feature extraction module splits the original face image into multiple key point images based on the face key point information; Processing multiple key point images through the first deep learning network computing layer in the facial feature extraction module and extracting key point feature vectors corresponding to each key point image based on a channel attention mechanism; Input multiple key point feature vectors and the original facial image into the second deep learning network operation layer of the facial feature extraction module to generate a facial feature vector.

3. The method for face enhancement according to claim 1, characterized in that: The facial key point information includes at least two of the following information: Left eye, right eye, nose, mouth, left ear, right ear.

4. The method for face enhancement according to claim 1, characterized in that: The face modification module performs sensory similarity constraints on the face modification perception information, including: The face modification module implements the face perception similarity constraint on the face modification perception information through the following formula: Characterizing the sensory similarity constraints of the face modification module: in, represents a high-realism face enhanced image, y represents the ground-truth face image corresponding to the same face in the high-realism face enhanced image, * represents the convolution operation, A represents the mean filter, k represents the local area of the high-realism face enhanced image, B represents the intermediate frequency filter designed for facial features, and C represents the high frequency filter designed for facial features. The above B and C are used to extract skin detail information.

5. The method for face enhancement according to claim 1, characterized in that: The method further comprises: The face enhancement model achieves overall consistency constraints on the original image by weighted superposition of the loss function corresponding to the objective similarity constraint, the loss function corresponding to the redundant similarity constraint, the loss function corresponding to the sensory similarity constraint, and the Manhattan distance between the high-realism face enhanced image and the true face image of the same face.

6. The method for face enhancement according to any one of claims 1 to 5, characterized in that: The method further comprises: Obtaining a true face image of the same face as the original face image and the enhanced face image by capturing the image with an image sensor; and / or A true face image that belongs to the same face as the original face image and the enhanced face image is obtained through a semi-supervised learning model.

7. A device for face enhancement, characterized in that: The device comprises: A facial feature extraction module is configured to obtain an original facial image and split the original facial image based on facial key point information to obtain a plurality of key point feature vectors and facial feature vectors. The facial feature extraction module applies objective similarity constraints to the facial feature vectors, including: Determine a first difference between facial key point information of a true face image and the facial feature vector corresponding to the original face image; Determine a second difference between the key point feature vector of the true face image and the key point feature vector corresponding to the original face image; Constraining the first difference and the second difference by using the loss function of the facial feature extraction module to achieve objective similarity constraint on the facial information of the facial feature vector; A face enhancement module is configured to generate an enhanced face image corresponding to the original face image based on the face feature vector, wherein the face enhancement module performs redundant similarity constraints on the face information of the enhanced face image, including: Calculating a first Euclidean distance between the enhanced face image and a true face image of the same face, and a second Euclidean distance between the enhanced face image and the true face images of other faces; Based on the first Euclidean distance and the second Euclidean distance, redundant similarity constraints on the enhanced facial image are implemented by a redundant face similarity loss function, wherein the redundant face similarity loss function includes: ,in, Characterize the redundant face similarity loss function, Represents the enhanced face image, y represents the true face image corresponding to the face in the enhanced face image, 、 、 、…… Represents the true face image of other faces, S represents the structural similarity function SSIM between images; A face modification module is configured to adjust the face modification perception information of the enhanced face image to generate a highly realistic face enhanced image, wherein the face modification module imposes sensory similarity constraints on the face modification perception information, including: The face modification module sequentially obtains local areas of the highly realistic face enhancement image using a sliding window method, and combines a mean filter to constrain the local brightness difference between the highly realistic face enhancement image and the true face image of the same face; The face modification module uses a medium frequency filter and a high frequency filter based on the Manhattan distance to implement the facial detail loss constraint between the high-realism face enhanced image and the true face image of the same face.

8. A device for face enhancement, characterized in that: The device for face enhancement comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

Images