Face image processing method, face image processing device, medium and electronic equipment
By utilizing skin feature information from high-resolution reference face images to optimize low-resolution face images, the problems of false texture and dirt recognition in existing technologies are solved, achieving more accurate and efficient image optimization.
Patent Information
- Application Number
- CN202310596833.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
Existing technologies are prone to producing pseudo-texture or no-texture issues when performing super-resolution processing on low-resolution face images. Furthermore, dirty areas in dark regions of the original image are easily misidentified as textures, resulting in unsatisfactory face image optimization effects.
By acquiring a high-resolution reference face image, performing de-featureization processing to extract skin feature information, and applying it to the skin optimization processing of a low-resolution face image, a target image is generated.
It effectively avoids the phenomenon of false textures, improves the accuracy and efficiency of image optimization, reduces the difficulty of obtaining reference images, and ensures the intuitive optimization effect of face images.
Smart Images

Figure CN119027345A_ABST
Abstract
Description
Background Art
[0002] At present, terminal devices have been widely used in daily photography. In order to meet users' requirements for face image quality, how to optimize face images to enhance the shooting details of face images has become one of the hot topics.
[0003] In the prior art, super-resolution processing is usually used to process low-resolution facial images to optimize the clarity or details of facial images. However, in practical applications, when the original facial image is of low quality, pseudo-texture or no-texture problems often occur during super-resolution processing; and after texture supplementation, the "dirty" parts of the original image with more dark areas are easily misidentified as textures, and unnecessary enhancement is performed, resulting in pseudo-textures, which makes it impossible to obtain an ideal facial image.
[0004] Therefore, how to accurately and effectively optimize the facial images to effectively improve the display effect of the facial images is a problem that needs to be solved urgently in the prior art. Summary of the invention
[0005] The present disclosure provides a facial image processing method, a facial image processing device, a computer-readable storage medium and an electronic device, thereby at least to a certain extent improving the problem that the prior art cannot accurately and effectively optimize facial images.
[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0007] According to a first aspect of the present disclosure, a facial image processing method is provided, comprising: acquiring a facial image to be processed and a reference facial image, wherein the clarity of the reference facial image is higher than that of the facial image to be processed; performing de-feature processing on a non-skin area of the reference facial image, and using the obtained reference facial skin image as skin feature information of the reference facial image; and performing skin optimization processing on the facial image to be processed according to the skin feature information to generate a target image.
[0008] According to a second aspect of the present disclosure, a facial image processing device is provided, comprising: an image acquisition module, used to acquire a facial image to be processed and a reference facial image, wherein the clarity of the reference facial image is higher than that of the facial image to be processed; a de-featured processing module, used to perform de-featured processing on a non-skin area of the reference facial image, and use the obtained reference facial skin image as skin feature information of the reference facial image; and an image optimization module, used to perform skin optimization processing on the facial image to be processed according to the skin feature information, and generate a target image.
[0009] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the face image processing method of the first aspect and its possible implementation methods are implemented.
[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the facial image processing method of the first aspect and possible implementation thereof by executing the executable instructions.
[0011] A face image to be processed and a reference face image are obtained, wherein the clarity of the reference face image is higher than that of the face image to be processed; a non-skin area of the reference face image is de-featured, and the obtained reference face skin image is used as skin feature information of the reference face image; skin optimization processing is performed on the face image to be processed according to the skin feature information to generate a target image. On the one hand, this exemplary embodiment proposes a new facial image processing method. Compared with directly performing super-resolution processing on the facial image to be processed, which can only be optimized based on the feature information of the original image, this exemplary embodiment optimizes the facial image to be processed based on the feature information of the reference facial image, which can avoid the situation where the target image has no texture or pseudo-texture, and improve the accuracy and effectiveness of image optimization; on the other hand, considering the characteristics of low recognition difference of skin features and wide coverage area, this exemplary embodiment performs skin optimization processing on the facial image to be processed based on the skin feature information extracted from the reference facial image, which can conveniently and intuitively optimize the facial image to be processed; on another hand, the reference facial skin image obtained by de-featured processing of the non-skin area of the reference facial image can realize the optimization processing of the facial image to be processed and generate the target image, which has low requirements on the reference facial image, reduces the difficulty of obtaining the reference facial image, and further ensures the processing efficiency of the facial image.
[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0014] Figure 1 A schematic diagram showing a system architecture in this exemplary embodiment;
[0015] Figure 2 A flowchart of a method for processing a face image in this exemplary embodiment is shown;
[0016] Figure 3 A sub-flow chart of a face image processing method in this exemplary embodiment is shown;
[0017] Figure 4 A schematic diagram showing a face image to be processed in this exemplary embodiment;
[0018] Figure 5 (a)~ Figure 5 (b) is a schematic diagram showing a comparison of a reference face image before and after alignment processing in this exemplary embodiment;
[0019] Figure 6 A schematic diagram of a reference face skin image in this exemplary embodiment is shown;
[0020] Figure 7 A sub-flow chart of another face image processing method in this exemplary embodiment is shown;
[0021] Figure 8 A schematic diagram of the structure of a face image optimization network in this exemplary embodiment is shown;
[0022] Fig. 9 A flow chart of a face image optimization network training in this exemplary embodiment is shown;
[0023] Fig.10 A schematic diagram of the architecture of a face image optimization network training in this exemplary embodiment is shown;
[0024] Fig.11 A flow chart for determining a true face image in this exemplary embodiment is shown;
[0025] Fig.12 (a)~ Fig.12 (c) shows a comparative schematic diagram of face image optimization in this exemplary embodiment;
[0026] Fig.13 (a)~ Fig.13 (c) shows a comparative schematic diagram of another face image optimization in this exemplary embodiment;
[0027] Fig.14 A structural block diagram of a facial image optimization device in this exemplary embodiment is shown;
[0028] Fig.15 A structural diagram of an electronic device in this exemplary embodiment is shown. DETAILED DESCRIPTION
[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.
[0030] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0031] In an exemplary embodiment of the present disclosure, a reference face image can be used to perform face image restoration processing on a face image to be processed, but this method has high requirements on the similarity between the reference face image and the face image to be processed, and the similarity can include the degree of matching of facial features, the degree of matching of facial position and posture, and the degree of matching of skin color, brightness, light and shadow, etc. Ideally, the reference face is a high-definition image of the face image to be processed. However, in actual applications, it is very difficult to obtain a reference face image with a high degree of similarity to the face image to be processed to ensure the effectiveness of the migration of facial feature information. This method not only increases the workload of obtaining the reference face image, but also because it is difficult to obtain an effective reference face image, it is difficult to guarantee the processing effect of the face image to be processed.
[0032] In view of one or more of the above problems, the exemplary embodiments of the present disclosure first provide a method for processing a face image. Figure 1 The system architecture of the operating environment of this exemplary embodiment will be described.
[0033] refer to Figure 1As shown, the system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be an electronic device such as a mobile phone, a tablet computer, or a smart wearable device. The server 120 generally refers to a background system that provides services related to face image processing in this exemplary embodiment, and may be a server or a cluster of multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to exchange data.
[0034] In one implementation, the facial image processing method in this exemplary implementation may be executed by the terminal 110. Exemplarily, a facial image taken by a user using the terminal 110 or a facial image selected by a user in an album of the terminal 110 is used as a facial image to be processed, and the terminal 110 outputs an image processing result of the facial image to be processed by executing the facial image processing method.
[0035] In one implementation, the facial image processing method in this exemplary implementation may be executed by the server 120. Exemplarily, an image processing App (Application) is installed on the terminal 110, and the user selects a facial image to be processed in the App, such as a facial image taken by the terminal 110, or a facial image in a local album. After that, the terminal 110 uploads the facial image to be processed to the server 120, and the server 120 executes the facial image processing method to obtain a target image, which may be returned to the terminal 110 later.
[0036] As can be seen from the above, in this exemplary embodiment, the execution subject of the face image processing method may be the above-mentioned terminal 110 or server 120, and the present disclosure does not limit this.
[0037] Combine the following Figure 2 The process of the face image processing method is described below. Figure 2 As shown, the face image processing method may include the following steps S210 to S230:
[0038] Step S210, obtaining a face image to be processed and a reference face image, wherein the definition of the reference face image is higher than the definition of the face image to be processed.
[0039] The face image to be processed refers to the face image to be optimized, and the reference face image refers to the reference image used to optimize the face image to be processed. Clarity is a sensory indicator of human vision. The clearer the feeling seen by the human eye, the higher the clarity. Clarity can be expressed by one or more indicators reflecting the visual perception of the human eye, such as gradient, resolution, etc. In this exemplary embodiment, the clarity of the reference face image needs to be higher than that of the face image to be processed, so as to optimize the image of the face image to be processed based on the clearer reference face image and improve the display effect of the face image to be processed.
[0040] In this exemplary embodiment, the faces in the face image to be processed and the reference face image may belong to the same person. For example, a relatively clear face image of the same user is used as a reference face image, and other face images to be processed of the user with lower clarity are optimized. In particular, the faces in the face image to be processed and the reference face image in this exemplary embodiment may belong to different people, as long as the face in the reference face image has good skin details or display effects. For example, the same reference face image may be used to optimize different face images to be processed. In this exemplary embodiment, by using a face image of a person different from that in the face image to be processed as a reference face image, the face image to be processed is optimized. On the one hand, the requirements for obtaining the reference face image are reduced, and the convenience and efficiency of optimizing the face image to be processed are improved; on the other hand, the face relationship between the reference face image and the face image to be processed is decoupled, and a targeted reference face image is obtained from the perspective of image optimization feasibility, which expands the applicable scenarios of face image processing.
[0041] This exemplary embodiment can obtain the face image to be processed in a variety of ways, such as real-time acquisition through a terminal device equipped with a camera, selection from a local photo album, download from the network or a gallery, etc. The reference face image can also be obtained in a variety of ways, such as acquisition through a high-pixel camera, or acquisition from a specific high-definition image library, etc.
[0042] In an exemplary embodiment, the above step S210 may include:
[0043] Acquire a face image to be processed, and identify feature information of a face in the face image to be processed;
[0044] According to the feature information, a reference face image matching the face image to be processed is searched in the face image library.
[0045] Considering that the faces in the face image to be processed may have differences in age, gender, facial structure or angle, etc., in order to improve the accuracy and effectiveness of image optimization processing, this exemplary embodiment can identify the facial feature information therein after obtaining the face image to be processed. Among them, the facial feature information may include age (such as age range), gender, distance or size of facial features, facial rotation angle, etc. Further, based on the feature information, a reference face image that best matches the face image to be processed is searched from a preset face image library. The face image library can be a pre-built library for storing reference face images, which can include face images with high clarity under multiple types of feature information, such as male face images, female face images, elderly face images, baby face images, or side face images, etc. Searching for a reference facial image that matches the facial image to be processed in a facial image library may include a variety of methods, such as searching for a reference facial image in the facial image library by calculating image feature similarity, or processing feature information through a pre-trained model to predict or map the reference facial image in the facial image library, or determining a reference facial image in the facial image library by matching feature information as an index with pre-configured information, etc. The present disclosure does not make any specific limitations on this.
[0046] Step S220, performing de-feature processing on the non-skin area of the reference face image, and using the obtained reference face skin image as skin feature information of the reference face image.
[0047] In a face image, the features with strong representation ability are generally the facial features, face shape, hairstyle, decoration, etc., and although the skin has differences in factors such as skin quality, it has a lower impact on the observation of face recognition, that is, when the skin of a face image is adjusted, it often does not affect the recognition of other facial features (such as facial features, etc.). Considering that the skin area occupies a large area in a face image, and may even be the main area, the optimization of the skin area has a small impact on the recognition of the person itself, but can have a better repair effect on the face image as a whole. In addition, the skin area is also the most likely to lose details in the face image, such as skin texture. Therefore, this exemplary embodiment can extract skin feature information from the reference face image as reference information for optimizing the face image to be processed.
[0048] Skin features may include facial skin features, neck skin features, and other skin features, such as ear skin features, lip skin features, etc. Specifically, in practical applications, skin features may be expanded according to actual needs, for example, areas that do not have obvious face recognition differences may be considered as skin features, such as lip texture features, pure color features, eyebrow texture, eyebrow color, and eyebrows themselves.
[0049] In this exemplary embodiment, the skin feature information may be data reflecting skin features, for example, the skin feature information may be a reference face skin image generated based on the face skin area in the reference face image. This exemplary embodiment may obtain the reference face skin image by performing a de-feature processing on the non-skin area of the reference face image. Specifically, the de-feature processing of the non-skin area of the reference face image may be implemented in a variety of ways, for example, the reference face image may be processed by a pre-trained skin feature extraction model to extract skin feature information; a skin mask may also be pre-set, and the reference face image may be processed by the skin mask to obtain skin feature information, etc.
[0050] In an exemplary embodiment, before the above step S220, the above facial image processing method may further include:
[0051] Align the facial key points in the reference face image with the facial key points in the face image to be processed.
[0052] Considering the differences in facial structure, angle, facial features, etc. between the reference facial image and the facial image to be processed, in order to improve the effectiveness of feature migration in the reference facial image, this exemplary embodiment may first align the facial key points in the reference facial image with the facial key points in the facial image to be processed, so as to achieve alignment between the reference facial image and the facial image to be processed. The facial key points may refer to any feature points in the face, such as key points of multiple parts such as eyes, nose, lips, etc.
[0053] This exemplary embodiment can first detect the key points of the face in the reference face image and the face image to be processed through the key point detection algorithm. For example, multiple key points of the eyes, nose, lips and other parts can be extracted from the face image. The more key points of the face, the more accurate the alignment. The specific number can be determined according to the actual operation performance. Then, according to the transformation parameters, such as the affine transformation matrix, through the image transformation, the eyes, nose and lips of the face in the reference face image are aligned to the preset positions in the face image to be processed, so as to realize the alignment process.
[0054] In addition, after detecting the key points of the face, this exemplary embodiment can also determine auxiliary points based on the detected key points of the face, for example, determine the difference points between the cheek area and the forehead area as auxiliary points to further ensure the accuracy of the alignment.
[0055] In an exemplary embodiment, if Figure 3 As shown, the above-mentioned alignment of the facial key points in the reference facial image with the facial key points in the face image to be processed may include the following steps:
[0056] Step S310, dividing the reference face image into a plurality of polygonal regions based on the face key points in the reference face image;
[0057] Step S320, adjusting the positions of the facial key points in the reference facial image to make them consistent with the positions of the facial key points in the facial image to be processed;
[0058] Step S330, performing affine transformation on the polygonal area according to the positions of the facial key points in the reference facial image before and after the adjustment.
[0059] After obtaining the key points of the face, the exemplary embodiment can divide the reference face image into multiple polygonal areas based on the key points of the face. The polygonal areas can be used to assist in the alignment of the key points of the face. According to the actual transformation scene and transformation requirements, the polygonal areas can be of various shapes, for example, multiple triangular areas can be divided. The specific division method can also include multiple methods, for example, the delaunay triangulation algorithm can be used to implement the division of the triangular area.
[0060] Then, the positions of the facial key points in the reference facial image are adjusted to be consistent with the positions of the facial key points in the facial image to be processed. According to the positions of the facial key points in the reference facial image before and after the adjustment, an affine transformation is performed on the polygonal area. For example, an affine transformation of the corresponding triangular area in the reference facial image and the facial image to be processed is performed to align the reference facial image to the facial image to be processed.
[0061] Figure 4 A schematic diagram of a face image to be processed in this exemplary embodiment is shown. Figure 5 FIG. 1 is a schematic diagram showing a comparison of a reference face image before and after alignment processing in this exemplary embodiment. In this exemplary embodiment, the reference face image and the face image to be processed are different face images. In order to ensure the migration effect of skin feature information, the reference face image can be aligned before the optimization processing of the face image to be processed. For example, Figure 4 As shown in FIG. 1 , the mouth of the face image to be processed has the corners of the mouth turned up due to the smile, and the nose tip is located lower than the nostrils. Figure 5 In (a), the mouth of the person in the reference face image is relaxed, the curvature of the mouth is small, and the position of the nose tip is substantially level with the position of the nostrils. Figure 5 (a) The reference face image and Figure 4 By aligning the face images to be processed in Figure 5 (b) shows the aligned reference face image. Figure 5 (b) The face is still Figure 5(a) The facial expression and facial features have changed to some extent. The state of the mouth and nose is different from Figure 4 The mouth and nose of the face image to be processed are similar, thus completing the alignment process between the reference face image and the face image to be processed. Figures 4-5 This is only used as an example for schematic explanation. In actual applications, after the reference face image is aligned with the face image to be processed, the change state may include more, such as eye size, distance between the eyes, position of the eye beginning / end, and changes in eyebrow position, curvature, etc., which are not specifically limited in the present disclosure.
[0062] Step S230, performing skin optimization processing on the face image to be processed according to the skin feature information to generate a target image.
[0063] Finally, the skin optimization processing can be performed on the face image to be processed according to the extracted skin feature information, so as to transfer the skin features in the reference face image to the face image to be processed, and the face image to be processed can be optimized to obtain the target image. Compared with the face image to be processed, the skin area of the target image has been optimized in detail, so that the whole image can obtain an intuitive and effective optimization processing effect.
[0064] This exemplary embodiment can perform skin optimization processing on the facial image to be processed through an algorithm, or can process the facial image to be processed through a pre-trained facial image optimization network. For example, skin feature information and the facial image to be processed are used as input data of the facial image optimization network, and after processing, a target image after skin optimization processing is output.
[0065] In summary, in this exemplary embodiment, a facial image to be processed and a reference facial image are obtained, and the clarity of the reference facial image is higher than that of the facial image to be processed; the non-skin area of the reference facial image is de-featured, and the obtained reference facial skin image is used as skin feature information of the reference facial image; the skin of the facial image to be processed is optimized according to the skin feature information to generate a target image. On the one hand, this exemplary embodiment proposes a new facial image processing method. Compared with directly performing super-resolution processing on the facial image to be processed, which can only be optimized based on the feature information of the original image, this exemplary embodiment optimizes the facial image to be processed based on the feature information of the reference facial image, which can avoid the situation where the target image has no texture or pseudo-texture, and improve the accuracy and effectiveness of image optimization; on the other hand, considering the characteristics of low recognition difference of skin features and wide coverage area, this exemplary embodiment performs skin optimization processing on the facial image to be processed based on the skin feature information extracted from the reference facial image, which can conveniently and intuitively optimize the facial image to be processed; on another hand, the reference facial skin image obtained by de-featured processing of the non-skin area of the reference facial image can realize the optimization processing of the facial image to be processed and generate the target image, which has low requirements on the reference facial image, reduces the difficulty of obtaining the reference facial image, and further ensures the processing efficiency of the facial image.
[0066] In an exemplary embodiment, the above step S230 may include:
[0067] The face image to be processed and the reference face skin image are input into a pre-trained face image optimization network to perform skin optimization processing on the face image to be processed and generate a target image.
[0068] In this exemplary embodiment, the skin feature information may be a reference face skin image generated based on the face skin area in the reference face image. The reference face skin image may be obtained by de-featured processing the non-skin area of the reference face image, that is, retaining the features of the skin area in the reference face image to obtain the reference face skin image. Figure 5 (b) Taking the aligned reference face image as an example, the reference face image can be converted into a grayscale image first, and then the grayscale value of the non-skin area is set to a preset value, such as "127", to obtain Figure 6 The reference facial skin image shown is used to achieve de-characterization of the non-skin area of the reference facial image.
[0069] Furthermore, the face image to be processed and the reference face skin image can be input into a pre-trained face image optimization network for processing to obtain a target image after skin optimization processing. The face image optimization network can be a neural network model, which can be trained by sample face images and corresponding high-definition face images, as well as reference face images.
[0070] In an exemplary embodiment, the above-mentioned face image optimization network may include a main image encoding subnetwork, a reference image encoding subnetwork, and a decoding subnetwork, wherein the main image encoding subnetwork may include n main image downsampling modules connected in sequence, the reference image encoding module includes n-1 reference image downsampling modules connected in sequence, and the decoding subnetwork includes n upsampling modules connected in sequence;
[0071] like Figure 7 As shown, the above-mentioned inputting the face image to be processed and the reference face skin image into the pre-trained face image optimization network to perform skin optimization processing on the face image to be processed and generate the target image may include the following steps:
[0072] Step S710, inputting the face image to be processed into the main image encoding sub-network, and the n main image down-sampling modules sequentially output the first down-sampled image to be processed to the nth down-sampled image to be processed;
[0073] Step S720, inputting the reference face skin image into the reference image encoding subnetwork, and the n-1 reference image downsampling modules sequentially output the first downsampled reference image to the n-1th downsampled reference image;
[0074] Step S730, inputting the nth down-sampled image to be processed into the decoding subnetwork, merging the i-th up-sampled image output by the i-th up-sampling module among the n up-sampling modules, the ni-th down-sampled image to be processed and the ni-th down-sampled reference image as the input of the i+1-th up-sampling module; i represents any positive integer from 1 to n-1;
[0075] Step S740, obtaining a target image according to the nth up-sampling image output by the nth up-sampling module.
[0076] by Figure 8 Taking the schematic diagram of the face image optimization network structure shown in FIG. 1 as an example, the processing process of the face image optimization network is illustrated. Figure 8The face image optimization network shown may include a main image encoding subnetwork 820, a reference image encoding subnetwork 840, and a decoding subnetwork 850, wherein, exemplarily, the main image encoding subnetwork 820 may include a first main image downsampling module 821, a second main image downsampling module 822, a third main image downsampling module 823, and a fourth main image downsampling module 824 connected in sequence; the reference image encoding module 840 may include a first reference image downsampling module 841, a second reference image downsampling module 842, and a third reference image downsampling module 843 connected in sequence; the decoding subnetwork 850 may include a first upsampling module 851, a second upsampling module 852, a third upsampling module 853, and a fourth upsampling module 854 connected in sequence. It should be noted that only the first and second upsampling modules are used here. Figure 8 The structure shown is for explanation only. In actual applications, the number of modules in the main image encoding subnetwork, the reference image encoding subnetwork, and the decoding subnetwork can be set according to actual network requirements or performance.
[0077] In this exemplary embodiment, after obtaining the face image 810 to be processed, it can be input into the main image encoding subnetwork 820, and each main image downsampling module therein sequentially outputs the downsampled images to be processed, for example, the first main image downsampling module 821, the second main image downsampling module 822, the third main image downsampling module 823, and the fourth main image downsampling module 824 sequentially output the first downsampled image to be processed s1, the second downsampled image to be processed s2, the third downsampled image to be processed s3, and the fourth downsampled image to be processed s4; obtain a reference face image 830, which can be a reference face skin image that has been de-featured in the non-skin area, and then input it into the reference image encoding subnetwork 840, and the first reference image downsampling module 841, the second reference image downsampling module 842, the third main image downsampling module 823, and the fourth main image downsampling module 824 sequentially output the first downsampled image to be processed s1, the second downsampled image to be processed s2, the third downsampled image to be processed s3, and the fourth downsampled image to be processed s4. The third reference image downsampling module 843 outputs the first downsampled reference image r1, the second downsampled reference image r2, and the third downsampled reference image r3 in sequence; then, the fourth downsampled image to be processed is input into the decoding subnetwork 850, the first upsampled image output by the first upsampling module 851 is merged with the third downsampled image to be processed s3 and the third downsampled reference image r3 as the input of the second upsampling module 852, the second upsampled image output by the second upsampling module 852 is merged with the second downsampled image to be processed s2 and the second downsampled reference image r2 as the input of the third upsampling module 853, the third upsampled image output by the third upsampling module 853 is merged with the first downsampled image to be processed s1 and the first downsampled reference image r1 as the input of the fourth upsampling module 854, and so on.
[0078] Finally, the target image can be obtained according to the fourth up-sampled image output by the fourth up-sampling module 854. For example, the fourth up-sampled image can be directly used as the target image, or Figure 8 In the structure shown, the 4th up-sampled image output by the 4th up-sampling module 854 can be merged with the face image 810 to be processed to obtain a target image 860 and so on.
[0079] In this exemplary embodiment, when training the face image optimization network of the above structure, the loss function can be calculated and the model parameters can be adjusted according to the difference between the target image output by the network and the true value image to complete the training process.
[0080] In an exemplary embodiment, if Fig. 9 As shown, the above face image optimization network can be trained in the following ways:
[0081] Step S910, obtaining a sample low-definition face image, a sample high-definition face image corresponding to the sample low-definition face image, and a sample reference face image; the non-skin area of the sample reference face image is subjected to de-feature processing;
[0082] Step S920, fusing the sample reference face image and the sample high-definition face image to generate a true face image;
[0083] Step S930, training a face image optimization network using the sample low-definition face images, the sample reference face images, and the sample high-definition face images.
[0084] In this exemplary embodiment, the facial image optimization network can be trained by using sample low-definition facial images, sample high-definition facial images corresponding to the sample low-definition facial images, and sample reference facial images, wherein the non-skin area of the sample reference facial image is de-featured, that is, the sample reference facial image can be considered to be a skin image of the sample reference face. The sample high-definition facial image and the sample reference facial image can be obtained in a variety of ways, such as obtaining the sample high-definition facial image and the sample reference facial image from different image libraries, or obtaining the sample high-definition facial image and the sample reference facial image from the same image library, etc. In addition, the sample low-definition facial image can also include a variety of acquisition methods, such as being directly obtained from an image library, or being obtained by degrading the obtained sample high-definition facial image, etc., and the present disclosure does not make specific limitations on this.
[0085] In this exemplary embodiment, the determination of sample low-definition facial images and sample high-definition facial images may include multiple methods, for example, when the resolution of the facial image is greater than a first threshold, it is considered to be a high-definition facial image, and when the resolution of the facial image is less than the first threshold, it is considered to be a low-definition facial image; or when the resolution of the facial image is greater than the first threshold, it is considered to be a high-definition facial image, and when the resolution of the facial image is less than a second threshold, it is considered to be a low-definition facial image, wherein the second threshold is less than the first threshold; or when the resolution of the facial image is a first preset resolution, such as a resolution of 256*256, it is considered to be a high-definition facial image, and when the resolution of the facial image is a second preset resolution, such as a resolution of 128*128, it is considered to be a low-definition facial image; or the facial images obtained from the first image library are set to be high-definition facial images, and the facial images obtained from the second image library are set to be low-definition facial images, wherein the first image library stores images with higher resolution or better image quality, and the second image library stores images with lower resolution or poorer image quality, etc.
[0086] In particular, considering that the present exemplary embodiment pays more attention to the skin feature information in the reference face image, such as the texture information of the skin. If only the sample high-definition face image is used as the true value image, it may not be possible for the trained face image optimization network to learn the skin feature information in the reference face image well. Therefore, the present exemplary embodiment may fuse the sample reference face image with the sample high-definition face image, and use the fused image as the face true value image to train the neural network. The specific network training process may be to input the sample low-definition face image and the sample reference face image into the face image optimization network, and adjust the parameters of the face image optimization network according to the difference between the output image and the face true value image, so as to train the face image optimization network for face image optimization.
[0087] Fig.10FIG. 1 shows a schematic diagram of the architecture of a face image optimization network training in the exemplary embodiment, which may specifically include: inputting a sample high-definition face image 1010 into a degradation module 1020 to perform degradation processing on the sample high-definition face image 1010 to obtain a corresponding sample low-definition face image 1030; inputting a sample reference face image 1040 into a de-feature processing module 1050 to perform de-feature processing on the non-skin area of the sample reference face image 1040 to obtain a sample reference face skin image 1060; and inputting a sample high-definition face image 1010 into a de-feature processing module 1050 to perform de-feature processing on the non-skin area of the sample reference face image 1040 to obtain a sample reference face skin image 1060. The sample reference facial skin image 1060 is input into the information fusion module 1070 to fuse the sample reference facial skin image 1060 and the sample high-definition facial image 1010 to generate a true face image 1080; further, the sample low-definition facial image 1030 and the sample reference facial skin image 1060 are used as input data of the facial image optimization network 1090, and based on the difference between the output result of the facial image optimization network 1090 and the true face image 1080, the loss function is calculated and the network parameters are adjusted to train the facial image optimization network 1090.
[0088] In an exemplary embodiment, the above step S920 may include:
[0089] Extract high-frequency information of the sample reference face image;
[0090] Extracting low-frequency information of the sample high-definition face image and high-frequency information of the sample high-definition face image;
[0091] The high-frequency information of the sample reference face image and the high-frequency information of the sample high-definition face image are weightedly fused to obtain the target high-frequency information;
[0092] The low-frequency information of the sample high-definition face image and the target high-frequency information are fused to obtain the true face image.
[0093] Generally, areas with more texture features in an image have a higher frequency, while areas with softer images have a lower frequency. In this exemplary embodiment, the high-frequency information of the face image may be skin texture information, such as cheek skin texture information, lip texture information, and so on.
[0094] This exemplary embodiment can first extract high-frequency information from the sample reference face image, and extract low-frequency information and high-frequency information from the sample high-definition face image. The extraction of low-frequency information and high-frequency information can include a variety of methods, such as extracting low-frequency information or high-frequency information of a face image through a preset filter, or extracting low-frequency information of a face image through a low-pass filter, and determining the high-frequency information of the face image by calculating the face image before filtering and the image after filtering. In addition, when filtering with a low-pass filter, mean filtering, Gaussian filtering, or a beauty skin smoothing algorithm can be used, and the present disclosure does not make specific limitations on this.
[0095] After obtaining the high-frequency information of the sample reference face image and the sample high-definition face image, the two can be weightedly fused to obtain the target high-frequency information, and then the low-frequency information of the sample high-definition face image is fused with the target high-frequency information to obtain the face truth image. This exemplary embodiment fuses the sample reference face image with the sample high-definition face image, so that the generated face truth image carries some skin features in the reference face image, making the face truth image more tractional to the reference face image, so that when training the face image optimization network, the network can learn or extract the effective features of the reference face image input, thereby improving the optimization effect and optimization targeting of the face image.
[0096] In addition, when the high-frequency information of the sample reference face image and the high-frequency information of the sample high-definition face image are weighted and fused to obtain the target high-frequency information, a reference weight α can be set to measure the reference components of the two input images of the high-definition face image and the reference face image. Among them, the reference weight α can be set or adjusted according to actual needs. Specifically, when the reference weight α is 1, the texture information of the target high-frequency information is completely derived from the sample reference face image. As the network learning proceeds, the task of the final reference face image is equivalent to the skin-changing operation of the processed face image; when the reference weight α>0 and α<1, it is the fusion effect of the skin quality of the reference face image and the high-definition face image. At the same time, for different scenes, such as high-brightness environment and low-brightness environment, different α values can be selected to adapt to the scene requirements, and the effect can be adapted to the scene regulation. For example, if a scene requires that the face skin after processing is more delicate, the skin quality of the reference face can be selected to be delicate and smooth, and the α value can be increased at the same time to make the face optimization produce a delicate skin effect.
[0097] Fig.11A flowchart for determining a true face image in this exemplary embodiment is shown, which may specifically include the following steps: obtaining a sample high-definition face image 1102; processing the sample high-definition face image through a first low-pass filter 1104 to obtain low-frequency information 1106 of the sample high-definition face image; determining high-frequency information 1108 of the sample high-definition face image, that is, the texture component in the sample high-definition face image, based on the difference between the sample high-definition face image and the low-frequency information of the sample high-definition face image; obtaining a sample reference face image that has been de-characterized in a non-skin area, that is, a sample reference face skin image 1110; processing the sample reference face skin image through a second low-pass filter 1112 to obtain low-frequency information 1114 of the sample reference face skin image; determining the high-frequency information 1118 of the sample high-definition face image based on the difference between the sample high-definition face image and the low-frequency information of the sample high-definition face image; obtaining a sample reference face image that has been de-characterized in a non-skin area, that is, a sample reference face skin image 1110; processing the sample reference face skin image through a second low-pass filter 1112 to obtain low-frequency information 1114 of the sample reference face skin image; determining the high-frequency information 1119 of the sample high-definition face image based on the difference between the sample reference face skin image and the sample The difference of the low-frequency information of the reference facial skin image is used to determine the high-frequency information 1116 of the sample reference facial skin image, that is, the texture component in the sample reference facial skin image; the high-frequency information of the sample reference facial image and the high-frequency information of the sample high-definition facial image are weightedly fused through the information fusion module 1118 to obtain the target high-frequency information 1120, wherein the weighted fusion can be calculated according to the pre-configured reference weight α, for example, the target high-frequency information = α high-frequency information of the sample reference facial skin image + (1-α) high-frequency information of the sample high-definition facial image. When more skin texture information of the sample reference facial skin image needs to be supplemented, it can be achieved by increasing the reference weight α; the low-frequency information of the sample high-definition facial image and the target high-frequency information are fused to obtain the true face image 1122.
[0098] In an exemplary embodiment, the above-mentioned face image processing method may further include:
[0099] Determine the application scenario requirements of the face image optimization network and determine the reference weights based on the application scenario requirements;
[0100] Furthermore, the weighted fusion of the high-frequency information of the sample reference face image and the high-frequency information of the sample high-definition face image may include:
[0101] Based on the reference weights, the high-frequency information of the sample reference face image and the high-frequency information of the sample high-definition face image are weightedly fused.
[0102] Among them, the application scenario requirements may refer to the requirements of what kind of target image needs to be generated in the current application scenario, for example, the application scenario requirements may be the requirements for the skin fineness of the target image generated in the application scenario, or the requirements for the degree of attention to the reference face skin image in image processing, etc. This exemplary embodiment may determine different reference weights according to different application scenario requirements, and then generate a face image optimization network adapted to different application scenarios, for example, in a scenario with a lower requirement for the skin fineness of the target image, a lower reference weight is set to train the face image optimization network, and in a scenario with a higher requirement for the skin fineness of the target image, a higher reference weight is set to train the face image optimization network, etc.
[0103] Based on the above-mentioned face image processing method, this exemplary embodiment can use a face image optimization network to obtain better face image optimization effects in various application scenarios. In different application scenarios, the same face image optimization network can be used. In particular, different face image optimization networks can also be used. For example, for application scenarios with large differences in ambient light, when processing the face image to be processed, the degree of attention paid to the reference face image is different. Therefore, different reference weights α can be configured respectively, and different face image optimization networks can be trained to process the face image to be processed. For example, in a darker environment, a higher reference weight α is configured to train the face image optimization network, so that when the face image optimization network is used to optimize the image to be processed, it relies more on the skin feature information of the reference face image. Or, in a brighter environment, a moderate reference weight is configured to train the face image optimization network, so that when the face image optimization network is used to optimize the image to be processed, it can integrate the skin feature information of the face image to be processed and the reference face image. In addition, in a scene with brighter environment, but when there is a dependency on the skin of the reference face image, for example, when the skin of the optimized target image is desired to be more delicate, a higher reference weight α may be configured for the face image optimization network. The specific reference weight α may be set based on factors such as the demand for the optimized target image in the specific application scenario and the requirement for the skin delicacy of the optimized target image, and the present disclosure does not make any specific limitation on this.
[0104] Through this exemplary embodiment, the face image optimization network can be trained in the application scenarios of bright environment and dark environment respectively, so that it can optimize the face skin quality better in the application scenario of bright environment, such as Fig.12 As shown, Fig.12 (a) When the face image to be processed is optimized by the method of this exemplary embodiment, the following can be obtained: Fig.12 (b) is the target image shown in FIG. 1 , and after being optimized by single-frame image super-resolution processing in the prior art, the following can be obtained: Fig.12As shown in the optimized image (c), it can be seen that compared with the prior art, this exemplary embodiment can refine the skin texture while increasing the skin texture details; in addition, using the face image optimization network corresponding to the application scene with a darker environment to process the image to be processed can also better restore the texture effect of the face skin in the application scene with a darker environment, such as Fig.13 As shown, Fig.13 (a) When the face image to be processed is optimized by the method of this exemplary embodiment, the following can be obtained: Fig.13 (b) is the target image shown in FIG. 1 , and after being optimized by single-frame image super-resolution processing in the prior art, the following can be obtained: Fig.13 From the optimized image shown in (c), it can be seen that compared with the prior art, this exemplary embodiment is able to restore realistic skin texture effects in dark light scenes such as night scenes or dim light, by using the texture information of the reference face image, and avoid the problem of pseudo-texture enhancement of "noise" or "dirt".
[0105] The exemplary embodiment of the present disclosure also provides a face image processing device. Fig.14 As shown, the facial image processing device 1400 may include: an image acquisition module 1410, used to acquire a facial image to be processed and a reference facial image, the clarity of the reference facial image being higher than the clarity of the facial image to be processed; a de-featured processing module 1420, used to perform de-featured processing on the non-skin area of the reference facial image, and use the obtained reference facial skin image as skin feature information of the reference facial image; an image optimization module 1430, used to perform skin optimization processing on the facial image to be processed according to the skin feature information to generate a target image.
[0106] In an exemplary embodiment, the image optimization module includes: a network processing unit, which is used to input the face image to be processed and the reference face skin image into a pre-trained face image optimization network to perform skin optimization processing on the face image to be processed and generate a target image.
[0107] In an exemplary embodiment, a face image optimization network includes a main image encoding subnetwork, a reference image encoding subnetwork, and a decoding subnetwork, wherein the main image encoding subnetwork includes n main image downsampling modules connected in sequence, the reference image encoding module includes n-1 reference image downsampling modules connected in sequence, and the decoding subnetwork includes n upsampling modules connected in sequence; a network processing unit includes: a first processing subunit for inputting a face image to be processed into the main image encoding subnetwork, and the n main image downsampling modules sequentially output the first downsampled image to be processed to the nth downsampled image to be processed; a second processing subunit for inputting the reference image downsampling modules into the main image encoding subnetwork, and the n main image downsampling modules sequentially output the first downsampled image to be processed to the nth downsampled image to be processed; A face skin image is input into a reference image encoding subnetwork, and n-1 reference image downsampling modules sequentially output the 1st downsampled reference image to the n-1th downsampled reference image; a third processing subunit is used to input the nth downsampled image to be processed into a decoding subnetwork, and the i-th upsampled image output by the i-th upsampling module among the n upsampling modules is combined with the ni-th downsampled image to be processed and the ni-th downsampled reference image as the input of the i+1-th upsampling module; i represents any positive integer from 1 to n-1; a fourth processing subunit is used to obtain a target image according to the n-th upsampled image output by the n-th upsampling module.
[0108] In an exemplary embodiment, a facial image optimization network is trained in the following manner: a sample image acquisition unit is used to acquire a sample low-definition facial image, a sample high-definition facial image corresponding to the sample low-definition facial image, and a sample reference facial image; the non-skin area of the sample reference facial image is de-featured; a true image generation unit is used to fuse the sample reference facial image and the sample high-definition facial image to generate a true facial image; a network training unit is used to train the facial image optimization network using the sample low-definition facial image, the sample reference facial image, and the sample high-definition facial image.
[0109] In an exemplary embodiment, the true value image generation unit includes: a first information extraction unit, used to extract high-frequency information of a sample reference face image; a second information extraction unit, used to extract low-frequency information of a sample high-definition face image and high-frequency information of the sample high-definition face image; a target information generation unit, used to weightedly fuse the high-frequency information of the sample reference face image and the high-frequency information of the sample high-definition face image to obtain target high-frequency information; an information fusion unit, used to fuse the low-frequency information of the sample high-definition face image with the target high-frequency information to obtain a true value face image.
[0110] In an exemplary embodiment, the face image processing device also includes: a reference weight determination unit, which is used to determine the application scenario requirements of the face image optimization network and determine the reference weight according to the application scenario requirements; a target information generation unit, which is used to weightedly fuse the high-frequency information of the sample reference face image and the high-frequency information of the sample high-definition face image based on the reference weight.
[0111] In an exemplary embodiment, the face image processing apparatus further includes: an alignment unit for aligning the face key points in the reference face image with the face key points in the face image to be processed before de-feature processing is performed on the non-skin area of the reference face image.
[0112] In an exemplary embodiment, the alignment unit includes: a region division subunit, used to divide the reference face image into multiple polygonal regions based on the facial key points in the reference face image; a key point position adjustment subunit, used to adjust the positions of the facial key points in the reference face image to make them consistent with the positions of the facial key points in the face image to be processed; and an affine transformation subunit, used to perform affine transformation on the polygonal region according to the positions of the facial key points in the reference face image before and after the adjustment.
[0113] In an exemplary embodiment, the image acquisition module includes: a feature information recognition unit, used to acquire a facial image to be processed and identify feature information of a face in the facial image to be processed; and an image search unit, used to search a reference facial image that matches the facial image to be processed in a facial image library based on the feature information.
[0114] The specific details of each part of the above device have been described in detail in the implementation method part, so they will not be repeated here.
[0115] The exemplary embodiments of the present disclosure further provide a computer-readable storage medium, which can be implemented in the form of a program product, including program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification, for example, to execute Figure 2 , Figure 3 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 or Fig.11 The program product may be in the form of a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.
[0116] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0117] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0118] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0119] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0120] The exemplary embodiment of the present disclosure further provides an electronic device. The electronic device may include a processor and a memory, wherein the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned face image processing method by executing the executable instructions.
[0121] Below Fig.15The mobile terminal 1500 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig.15 The construction in can also be applied to fixed type equipment.
[0122] like Fig.15 As shown, the mobile terminal 1500 may specifically include: a processor 1501, a memory 1502, a bus 1503, a mobile communication module 1504, an antenna 1, a wireless communication module 1505, an antenna 2, a display screen 1506, a camera module 1507, an audio module 1508, a power module 1509 and a sensor module 1510.
[0123] Processor 1501 may include one or more processing units, for example, processor 1501 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.
[0124] The encoder can encode (i.e. compress) an image or video to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data. The mobile terminal 1500 can support one or more encoders and decoders, for example: image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and video formats such as MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, and HEVC (High Efficiency Video Coding).
[0125] The processor 1501 may be connected to the memory 1502 or other components via a bus 1503 .
[0126] The memory 1502 may be used to store computer executable program codes, which may include instructions. The processor 1501 executes various functional applications and data processing of the mobile terminal 1500 by running the instructions stored in the memory 1502. The memory 1502 may also store application data, such as images, videos, and other files.
[0127] The communication function of the mobile terminal 1500 can be implemented by the mobile communication module 1504, antenna 1, wireless communication module 1505, antenna 2, a modulation and demodulation processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1504 can provide 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1500. The wireless communication module 1505 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 1500.
[0128] The display screen 1506 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 1507 is used to implement shooting functions, such as shooting images, videos, etc. The audio module 1508 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 1509 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 1510 may include one or more sensors for implementing corresponding sensing detection functions. For example, the sensor module 1510 may include an inertial sensor, which is used to detect the motion posture of the mobile terminal 1500 and output inertial sensing data.
[0129] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other implementations of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
[0130] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A facial image processing method, characterized in that: include: Acquire a face image to be processed and a reference face image, wherein the definition of the reference face image is higher than the definition of the face image to be processed; Performing de-feature processing on the non-skin area of the reference face image, and using the obtained reference face skin image as the skin feature information of the reference face image; The skin optimization processing is performed on the face image to be processed according to the skin feature information to generate a target image.
2. The method according to claim 1, characterized in that The step of performing skin optimization processing on the face image to be processed according to the skin feature information to generate a target image includes: The face image to be processed and the reference face skin image are input into a pre-trained face image optimization network to perform skin optimization processing on the face image to be processed to generate a target image.
3. The method according to claim 2, characterized in that The face image optimization network includes a main image encoding subnetwork, a reference image encoding subnetwork, and a decoding subnetwork, wherein the main image encoding subnetwork includes n main image downsampling modules connected in sequence, the reference image encoding module includes n-1 reference image downsampling modules connected in sequence, and the decoding subnetwork includes n upsampling modules connected in sequence; The step of inputting the face image to be processed and the reference face skin image into a pre-trained face image optimization network to perform skin optimization processing on the face image to be processed to generate a target image includes: Inputting the face image to be processed into the main image encoding subnetwork, and the n main image downsampling modules sequentially output the first downsampled image to be processed to the nth downsampled image to be processed; Inputting the reference face skin image into the reference image encoding subnetwork, and the n-1 reference image downsampling modules sequentially output the first downsampling reference image to the n-1th downsampling reference image; Input the nth down-sampled image to be processed into the decoding subnetwork, and combine the i-th up-sampled image output by the i-th up-sampling module among the n up-sampling modules with the ni-th down-sampled image to be processed and the ni-th down-sampled reference image as the input of the i+1-th up-sampling module; i represents any positive integer from 1 to n-1; The target image is obtained according to the nth up-sampling image output by the nth up-sampling module.
4. The method according to claim 2, characterized in that: The face image optimization network is trained in the following way: Acquire a sample low-definition face image, a sample high-definition face image corresponding to the sample low-definition face image, and a sample reference face image; the non-skin area of the sample reference face image is subjected to de-feature processing; Fusion of the sample reference face image and the sample high-definition face image to generate a true face image; The facial image optimization network is trained using the sample low-definition facial images, the sample reference facial images and the sample high-definition facial images.
5. The method according to claim 4, characterized in that The fusing the sample reference face image and the sample high-definition face image to generate a true face image includes: Extracting high-frequency information of the sample reference face image; Extracting low-frequency information of the sample high-definition face image and high-frequency information of the sample high-definition face image; Performing weighted fusion on the high-frequency information of the sample reference face image and the high-frequency information of the sample high-definition face image to obtain target high-frequency information; The low-frequency information of the sample high-definition face image and the target high-frequency information are fused to obtain the true face image.
6. The method according to claim 5, characterized in that The method further comprises: Determine application scenario requirements of the face image optimization network, and determine reference weights according to the application scenario requirements; The weighted fusion of the high-frequency information of the sample reference face image and the high-frequency information of the sample high-definition face image includes: Based on the reference weight, the high-frequency information of the sample reference face image and the high-frequency information of the sample high-definition face image are weightedly fused.
7. The method according to claim 1, characterized in that Before performing de-feature processing on the non-skin area of the reference face image, the method further includes: Align the facial key points in the reference facial image with the facial key points in the face image to be processed.
8. The method according to claim 7, characterized in that The step of aligning the facial key points in the reference facial image with the facial key points in the facial image to be processed comprises: Based on facial key points in the reference facial image, dividing the reference facial image into a plurality of polygonal areas; Adjusting the positions of the key points of the face in the reference face image to make them consistent with the positions of the key points of the face in the face image to be processed; The polygonal area is subjected to affine transformation according to the positions of the facial key points in the reference facial image before and after the adjustment.
9. The method according to claim 1, characterized in that: The step of obtaining the face image to be processed and the reference face image comprises: Acquire a face image to be processed, and identify feature information of a face in the face image to be processed; According to the feature information, a reference face image matching the face image to be processed is searched in a face image library.
10. A facial image processing device, characterized in that: include: An image acquisition module, used to acquire a face image to be processed and a reference face image, wherein the definition of the reference face image is higher than that of the face image to be processed; a de-feature processing module, used for performing de-feature processing on the non-skin area of the reference face image, and using the obtained reference face skin image as the skin feature information of the reference face image; The image optimization module is used to perform skin optimization processing on the face image to be processed according to the skin feature information to generate a target image.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
12. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 9 by executing the executable instructions.
