Face defect repair method, device and computer-readable storage medium

Abstract

An embodiment of the present application provides a method, device, and computer-readable storage for face defect repair, which relates to the field of live broadcast technology. In the above method, after converting the first face image into a second face image with low resolution, an image processing model is used to extract features from the second face image to infer linear transformation parameters. Finally, based on the linear transformation parameters, a local linear transformation is applied to the first face image with high resolution to automatically detect and repair the defective areas in the face. In addition, the above method can also fix the input resolution and output resolution of the image processing model by reducing the resolution, ensuring that the performance consumption of the image processing model does not increase with the increase in resolution, so that the image processing model can still maintain excellent performance when processing defective faces with high resolution.

Description

Technical Field

[0001] The present application relates to the technical field of live broadcast, and in particular, to a method, device and computer-readable storage medium for repairing facial defects. Background Art

[0002] With the development of mobile Internet technology and network communication technology, network live broadcast has developed rapidly and been applied to people's daily work and life. Currently, the live broadcast interaction method with the host as the carrier is deeply favored by the audience. However, when the host's face is in poor condition and has defects, it may affect the viewing experience of the live broadcast.

[0003] Facial defect repair is a technology that removes the defective area of the face while maintaining the overall authenticity and consistency of the face image to achieve a beauty effect. However, the existing automatic facial defect repair methods either cannot adapt to complex lighting scenarios, resulting in missed detection and false detection of defective areas, making the defect removal incomplete or the non-defective areas affected, or use complex deep learning models for detection and repair, which easily leads to excessive performance resource consumption, especially difficult to run in real time at high resolutions. Summary of the Invention

[0004] In order to at least overcome the above deficiencies in the prior art, the purpose of the present application is to provide a method, device and computer-readable storage medium for repairing facial defects.

[0005] In a first aspect, an embodiment of the present application provides a method for repairing facial defects, and the method for repairing facial defects includes:

[0006] Obtain a first face image to be repaired;

[0007] Preprocess the first face image to obtain a second face image after preprocessing, where the resolution of the first face image is greater than the resolution of the second face image;

[0008] Input the second face image into a pre-trained image processing model to obtain a linear transformation parameter map corresponding to the preprocessed face image, and the resolution of the linear transformation parameter map is lower than the resolution of the second face image;

[0009] Repair the first face image based on the linear transformation parameter map to obtain a face image after defect repair.

[0010] In a possible implementation manner, the step of repairing the first face image based on the linear transformation parameter map to obtain a face image after defect repair includes:

[0011] Bilinearly interpolate and magnify the linear variation parameter map to obtain a repaired parameter map with the same resolution as the first face image, where the repaired parameter map includes a multiplication parameter map and a bias parameter map;

[0012] Apply a linear variation corresponding to the multiplication parameter map and the bias parameter map to each pixel of the first face image to obtain a face image with the blemishes repaired.

[0013] In a possible implementation, the step of applying a linear variation corresponding to the multiplication parameter map and the bias parameter map to each pixel of the first face image to obtain a face image with the blemishes repaired includes:

[0014] Multiply the corresponding pixel value in the first face image by the scaling factor in the multiplication parameter map and then add the offset in the bias parameter map to obtain the repaired pixel value, and obtain a face image with the blemishes repaired from the repaired pixel values.

[0015] In a possible implementation, the step of preprocessing the first face image to obtain a preprocessed second face image includes:

[0016] Reduce the resolution of the first face image, and perform normalization processing on the first face image with the reduced resolution, and map the pixel values of the pixel points in the first face image to a preset pixel value range to obtain a preprocessed second face image.

[0017] In a possible implementation, before the step of inputting the second face image into a pre-trained image processing model to obtain a linear variation parameter map corresponding to the preprocessed face image, the method further includes a step of training to obtain an image processing model, and this step includes:

[0018] Obtain a sample image pair, where the sample image pair includes a blemished face sample image to be repaired and a corresponding flawless face sample image with the blemishes repaired;

[0019] Preprocess the blemished face sample image to obtain a preprocessed blemished face sample image, and the resolution of the preprocessed blemished face sample image is lower than the original resolution of the blemished face sample image;

[0020] Input the preprocessed blemished face sample image into an image training model for training, and train to obtain a training linear variation parameter map corresponding to the preprocessed blemished face sample image, and the resolution of the training linear variation parameter map is less than the resolution of the preprocessed blemished face sample image, where the image training model is composed of a downsampling module, an upsampling module, and an output layer connected in series;

[0021] Repair the defective face sample image based on the training linear transformation parameter map to obtain a sample face image after defect repair;

[0022] Calculate the loss function value of the image training model based on the perfect face sample image and the sample face image, compare the loss function value with a preset loss function threshold, and when the loss function value is greater than the preset loss function threshold, update the model parameters in the image training model and repeat the above steps until the loss function value is less than the preset loss function threshold or the number of iterative updates of the model parameters reaches a preset number, end the training of the image training model, and use the image training model corresponding to the completion of training as the trained image processing model.

[0023] In a possible implementation manner, before the step of inputting the preprocessed defective face sample image into the image training model for training to obtain a training linear transformation parameter map corresponding to the preprocessed defective face sample image, the method further includes:

[0024] Construct a downsampling module with four layers of convolution, an upsampling module with two layers of convolution, and an output layer formed by a 1x1 convolution layer respectively;

[0025] Connect the downsampling module, the upsampling module, and the output layer in series to obtain the image training model.

[0026] In a possible implementation manner, the step of repairing the defective face sample image based on the training linear transformation parameter map to obtain a sample face image after defect repair includes:

[0027] Perform bilinear interpolation magnification on the training linear transformation parameter map to obtain a training repair parameter map with the same resolution as the defective face sample image, where the training repair parameter map includes a training multiplication parameter map and a training bias parameter map;

[0028] Apply a linear transformation corresponding to the training multiplication parameter map and the training bias parameter map to each pixel of the defective face sample image to obtain a sample face image after defect repair.

[0029] In a possible implementation manner, the step of applying a linear transformation corresponding to the training multiplication parameter map and the training bias parameter map to each pixel of the defective face sample image to obtain a sample face image after defect repair includes:

[0030] Multiply the corresponding pixel value in the defective face sample image by the scaling factor in the training multiplication parameter map and then add the offset in the training bias parameter map to obtain the repaired pixel value, and obtain the sample face image with the defective repair completed from the repaired pixel value.

[0031] In a second aspect, an embodiment of the present application further provides a face defect repair device, including:

[0032] An acquisition module, configured to acquire a first face image to be repaired;

[0033] A preprocessing module, configured to preprocess the first face image to obtain a second face image after preprocessing, where the resolution of the first face image is greater than the resolution of the second face image;

[0034] An image processing module, configured to input the second face image into a pre-trained image processing model to obtain a linear transformation parameter map corresponding to the preprocessed face image, and the resolution of the linear transformation parameter map is lower than the resolution of the second face image;

[0035] A repair module, configured to repair the first face image based on the linear transformation parameter map to obtain a face image after defect repair.

[0036] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the face defect repair method described in any one of the first aspects.

[0037] Based on any of the above aspects, the face defect repair method, device, and computer-readable storage provided by the embodiments of the present application, in the above method, after converting the first face image into a second face image with a low resolution, then using an image processing model to perform feature extraction on the second face image to infer linear transformation parameters, and finally applying local linear transformation to the first face image with a high resolution based on the linear transformation parameters to automatically detect and repair the defective area in the face. In addition, the input resolution and output resolution of the image processing model can be fixed by reducing the resolution, ensuring that part of the performance consumption of the image processing model does not increase with the increase of the resolution, so that the image processing model can still maintain excellent performance when processing defective faces with high resolution. Description of the Drawings

[0038] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the attached drawings required in the embodiments. It should be understood that the following attached drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation of the scope. For those of ordinary skill in the art, without creative efforts, other related attached drawings can also be obtained based on these attached drawings.

[0039] Figure 1 Schematic diagram of the interaction scenario of the face defect repair method provided by the embodiment of the present application;

[0040] Figure 2 Schematic flow chart of a face defect repair method provided by the embodiment of the present application;

[0041] Figure 3 For Figure 2 Sub - flow chart of step S140 in

[0042] Figure 4 Multiplication parameter diagram and bias parameter diagram during the repair process of the embodiment of the present application;

[0043] Figure 5 Comparison diagram of the face image before repair and the face image after repair in the embodiment of the present application;

[0044] Figure 6 Another schematic flow chart of the face defect repair method provided by the embodiment of the present application;

[0045] Figure 7 For Figure 6 Sub - flow chart of step S240 in

[0046] Figure 8 Functional module diagram of a face defect repair device provided by the embodiment of the present application;

[0047] Figure 9 Schematic block diagram of the structure of a computer device provided by the embodiment of the present application. Specific embodiments

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. It should be understood that the accompanying drawings in this application are only for the purposes of illustration and description, and are not used to limit the protection scope of this application. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the embodiments of this application. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without logical context relationships may be reversed or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of this application.

[0049] In addition, the described embodiments are only some embodiments of this application, rather than all embodiments. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of this application claimed, but only represents the selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative efforts belong to the scope of protection of this application.

[0050] It should be noted that, without conflict, different features in the embodiments of this application can be combined with each other.

[0051] To solve the above-mentioned technical problems in the prior art, the embodiments of this application provide a method for repairing facial flaws. To facilitate understanding of the solution of this application, a facial flaw repair system to which this application may be applied will be introduced first. It can be understood that the following-introduced facial flaw repair system is only for explaining the possible application scenarios of the solution of this application, and the solution of this application can also be applied to other application scenarios outside the following scenarios.

[0052] Please refer to Figure 1 , Figure 1 , which shows a schematic diagram of a possible interaction scenario of the facial flaw repair system provided by the solution of this application. The facial flaw repair system 10 may include a computer device 100, a host terminal 200, and an audience terminal 300 that are communicatively connected. The computer device 100 may provide video image processing support for the host terminal 200. For example, an algorithm for repairing facial flaws may be stored in the computer device 100 to obtain a host image without facial flaws by processing the live video.

[0053] In the embodiments of this application, the host terminal 200 and the audience terminal 300 may be, but are not limited to, smart phones, personal digital assistants, tablet computers, personal computers, laptop computers, virtual reality terminal devices, augmented reality terminal devices, etc. In the specific implementation process, multiple host terminals 200 and audience terminals 300 may be connected to the computer device 100. Figure 1Only one host terminal 200 and two viewer terminals 300 are shown. Among them, live broadcast service programs can be installed in the host terminal 200 and the viewer terminals 300. For example, the service program can be an application APP or a small program related to Internet live broadcast used in a computer or a smart phone, etc.

[0054] In the embodiment of the present application, the computer device 100 can be a server or other electronic devices with image data processing capabilities. When the computer device 100 is a server, the computer device 100 can be a single physical server, or can be a server group composed of multiple physical servers for performing different data processing functions, where the server group can be centralized or distributed (for example, the server can be a distributed system). In some possible implementation manners, if the server adopts a single physical server, different logical server components can be allocated to the physical server based on different service functions.

[0055] It can be understood that Figure 1 The shown live broadcast scenario is only a feasible example. In other feasible embodiments, this live broadcast scenario may also only include Figure 1 a part of the shown components or may also include other components.

[0056] Next, in combination with Figure 1 the shown application scenario, an exemplary description of the face defect repair method provided by the embodiment of the present application will be given. Please refer to Figure 2 , the face defect repair method provided by the embodiment of the present application can be executed by the aforementioned computer device 100. The order of some steps in the face defect repair method in the embodiment of the present application can be interchanged according to actual needs, or some of the steps can also be omitted or deleted. The detailed steps of the face defect repair method executed by the computer device 100 are introduced as follows.

[0057] Step S110: Obtain a first face image to be repaired.

[0058] In this step, the first face image can be obtained by the image processing unit (GPU) in the computer device 100 processing the video stream in real time, or can be actively uploaded to the computer device 100 by the host terminal 200. The first face image includes the face image of the host or other objects.

[0059] Step S120: Preprocess the first face image to obtain a second preprocessed face image.

[0060] In this step, the resolution of the second face image is lower than that of the first face image, and the resolution of the second face image can be a fixed value, which can reduce the computational complexity while fixing the input resolution of the image processing model, ensuring that some performance consumption of the image processing model does not increase with the increase of the resolution, so as to realize the repair of the defective face image with high resolution. In addition, the second face image with low resolution can enable the image processing model to focus more on extracting face features, which helps to reduce the influence of illumination. Exemplarily, in this step, a pre-processor can be constructed to pre-process the first face image to obtain a second face image with a resolution of 256×192.

[0061] Step S130: Input the second face image into a pre-trained image processing model to obtain a linear transformation parameter map corresponding to the pre-processed face image.

[0062] In this step, the resolution of the linear transformation parameter map is lower than that of the second face image, and the resolution of the linear transformation parameter map can be a fixed value, which can further improve the processing rate of the model while retaining the linear transformation parameters and enhance the model performance. Exemplarily, the resolution of the linear transformation parameter map can be 64×48.

[0063] Step S140: Repair the first face image based on the linear transformation parameter map to obtain a face image after defect repair.

[0064] In this step, a linear transformation parameter map including linear transformation parameters is inferred on the fixed low-resolution second face image and applied to the high-resolution first face image. A linear transformation is applied to each pixel in the first face image to adjust its original value to achieve the effect of defect repair.

[0065] In this embodiment, first, the first face image is converted into a second face image with low resolution, then the image processing model is used to extract features from the second face image to infer linear transformation parameters, and finally, a local linear transformation is applied to the high-resolution first face image based on the linear transformation parameters to automatically detect and repair the defective area in the face. In addition, the input resolution and output resolution of the image processing model can be fixed by reducing the resolution, ensuring that some performance consumption of the image processing model does not increase with the increase of the resolution, so that the image processing model can still maintain excellent performance when processing high-resolution defective faces.

[0066] As a possible implementation manner of the embodiment of the present application, step S120 can be implemented by the following method.

[0067] First, the resolution of the first face image can be reduced, and the first face image with reduced resolution can be normalized to remap the pixel value range of the pixel points in the first face image to a preset pixel value interval. Exemplarily, the preset pixel value interval can be [-1, 1] to obtain the preprocessed second face image. Specifically, a preprocessor can be constructed to reduce the resolution of the first face image to a fixed value. Exemplarily, the resolution of the second face image can be 256×192, and then the first face image with reduced resolution can be normalized according to the formula. Specifically, please refer to the following expression (1):

[0068]

[0069] where Px is the preprocessor, I is the first face image, and resize() is used to adjust the resolution of the image.

[0070] In this embodiment, by reducing the resolution of the second face image to a fixed value, not only can the computational complexity be reduced, but also the input resolution of the image processing model can be fixed to ensure that some performance consumption of the image processing model does not increase with the increase of the resolution, so as to realize the repair of high-resolution defective face images.

[0071] Further, please refer to Figure 3 , step S140 can be implemented by the following method.

[0072] Step S141: Bilinearly interpolate and magnify the linear transformation parameter map to obtain a repair parameter map with the same resolution as the first face image. The repair parameter map includes a multiplication parameter map and a bias parameter map.

[0073] In this step, a processor can be constructed to perform bilinear interpolation and magnification processing on the linear transformation parameter map to obtain a repair parameter map with the same resolution as the first face image. The repair parameter map can be composed of a three-channel multiplication parameter map and a bias parameter map merged. Among them, the multiplication parameter map includes the multiplication scaling factors applied to each channel of the input, which can be understood as the learned weights, and the bias parameter map includes the biases applied to each channel of the input, which can be understood as the learned bias values.

[0074] Step S142: Apply a linear transformation corresponding to the multiplication parameter map and the bias parameter map to each pixel of the first face image to obtain a face image with the defect repaired.

[0075] In this step, the corresponding pixel value in the first face image can be multiplied by the scaling factor in the multiplication parameter map and then added to the offset in the bias parameter map to obtain the repaired pixel value, and the face image with the defect repaired can be obtained from the repaired pixel values. Specifically, please refer to Figure 4and Figure 5 , Figure 4 schematically shows a multiplication parameter map and a bias parameter map, Figure 5 schematically shows a comparison diagram of the first face image (left face image) and the face image with defects repaired (right face image). In this embodiment, the processor Py can apply a corresponding linear transformation to each pixel point in the first face image. For details, please refer to the following expression (2):

[0076]

[0077] where resize bilinear () is used to adjust the resolution of the image to the resolution of the image to be repaired. I is the first face image, A is the linear transformation parameter map, A′ is the repair parameter map, A1′ is the multiplication parameter map, A2′ is the bias parameter map, and O is the face image with defects repaired.

[0078] In this embodiment, on the second face image with low resolution, the linear transformation parameters are inferred by using the feature extraction of deep learning, and then the local linear transformation is applied to the first face image with high resolution, so as to automatically detect and repair the defective area in various complex lighting scenarios.

[0079] As a possible implementation manner of the embodiment of the present application, please refer to Figure 6 , and before step S130, it may further include the step of training an image processing model, and this step can be implemented by the following method.

[0080] Step S210: Obtain a pair of sample images.

[0081] In this step, a dataset of pairs of sample images containing defects and defect repairs can be collected by manual annotation or generation methods. Among them, the pair of sample images includes a defective face sample image to be repaired and a defect-free face sample image corresponding to the completed defect repair.

[0082] Step S220: Preprocess the defective face sample image to obtain a preprocessed defective face sample image.

[0083] In this step, the resolution of the preprocessed defective face sample image is lower than the original resolution of the defective face sample image, and the resolution of the preprocessed defective face sample image can be a fixed value, which can reduce the computational complexity while fixing the input resolution of the image processing model, ensuring that some performance consumption of the image processing model does not increase with the increase of the resolution, so as to realize the restoration of high-resolution defective face images. In addition, the defective face sample image with reduced resolution can enable the image processing model to focus more on extracting face features, which helps to reduce the influence of illumination. Exemplarily, in this step, a preprocessor Px can be constructed to preprocess the defective face sample image, and the resolution of the preprocessed defective face image can be 256×192.

[0084] Step S230: Input the preprocessed defective face sample image into the image training model for training, and obtain the training linear transformation parameter map corresponding to the preprocessed defective face sample image.

[0085] In this step, the resolution of the training linear transformation parameter map is smaller than the resolution of the preprocessed defective face sample image, and the resolution of the training linear transformation parameter map can be a fixed value, which can retain the linear transformation parameters while further improving the processing speed of the model and enhancing the model performance. Exemplarily, the resolution of the training linear transformation parameter map can be 64×48. In addition, the image training model is composed of a downsampling module, an upsampling module, and an output layer connected in series.

[0086] Step S240: Repair the defective face sample image based on the training linear transformation parameter map to obtain the sample face image after defect repair.

[0087] In this step, the training linear transformation parameter map including linear transformation parameters is inferred on the preprocessed low-resolution defective face sample image and applied to the high-resolution defective face sample image. Exemplarily, a processor py can be constructed to apply a linear transformation to each pixel in the defective face sample image to adjust its original value to achieve the effect of defect repair.

[0088] Step S250: Calculate the loss function value of the image training model based on the flawless face sample image and the sample face image, compare the loss function value with the preset loss function threshold. When the loss function value is greater than the preset loss function threshold, update the model parameters in the image training model and repeat the above steps until the loss function value is less than the preset loss function threshold or the number of iterative updates of the model parameters reaches the preset number, then end the training of the image training model, and use the image training model corresponding to the completion of training as the trained image processing model.

[0089] In this step, the image perception similarity between the flawless face sample image and the sample face image can be used as the loss function value to train the image processing model to generate a more realistic and natural defect repair image. For details, please refer to the following expression (3):

[0090]

[0091] Where I fix represents the flawless face sample image in the sample image pair dataset, Px(I) represents the preprocessor Px in step S220, Py represents the processor Py in step S240, M θ represents the parameter θ learned by the image processing model during training, represents the j-th layer LPIPS perceptual feature of the input image I.

[0092] As a possible implementation manner of the embodiment of the present application, step S220 can be implemented by the following method.

[0093] First, the resolution of the defective face sample image can be reduced, and the reduced-resolution defective face sample image can be normalized to remap the pixel value range of the pixel points in the defective face sample image to a preset pixel value interval. Exemplarily, the preset pixel value interval can be [-1, 1] to obtain the preprocessed defective face sample image. Specifically, a preprocessor Px is constructed to reduce the resolution of the defective face sample image to a fixed value. Exemplarily, the resolution of the preprocessed defective face sample image can be 256×192, and then the reduced-resolution defective face sample image is normalized according to the formula. For details, please refer to the above expression (1), which will not be elaborated here.

[0094] Furthermore, before step S230, there is also a step of constructing an image training model, which can be implemented by the following method.

[0095] First, a downsampling module with four layers of convolution, an upsampling module with two layers of convolution, and an output layer formed by a 1x1 convolution layer are respectively constructed, and then the downsampling module, the upsampling module, and the output layer are connected in series to obtain the image training model.

[0096] Furthermore, please refer to Figure 7 , step S240 can be implemented by the following method.

[0097] Step S241: Bilinearly interpolate and enlarge the training linear transformation parameter map to obtain a training repair parameter map with the same resolution as the defective face sample image. The training repair parameter map includes a training multiplication parameter map and a training bias parameter map.

[0098] In this step, the processor Py can perform bilinear interpolation magnification on the linearly varying parameter map to obtain a repaired parameter map with the same resolution as the first face image. The repaired parameter map can be formed by merging a three-channel multiplication parameter map and a bias parameter map. Among them, the multiplication parameter map includes the multiplication scaling factors applied to each channel of the input, which can be understood as the learned weights, and the bias parameter map includes the biases applied to each channel of the input, which can be understood as the learned bias values.

[0099] Step S242: Apply a linear transformation corresponding to the trained multiplication parameter map and the trained bias parameter map to each pixel of the defective face sample image to obtain a sample face image after defect repair.

[0100] In this step, the corresponding pixel value in the defective face sample image can be multiplied by the scaling factor in the trained multiplication parameter map and then added to the offset in the trained bias parameter map to obtain the repaired pixel value, and the sample face image with defect repair completed can be obtained from the repaired pixel values. Specifically, the processor Py can apply the corresponding linear transformation to each pixel point in the defective face sample image. For details, please refer to the above expression (2), which will not be elaborated here.

[0101] In this embodiment, linear transformation parameters are inferred by using deep learning feature extraction on the preprocessed low-resolution defective face sample image, and then local linear transformation is applied to the high-resolution defective face sample image based on the linear transformation parameters, which can adapt to various complex lighting scenarios for automatic detection and repair of defective areas.

[0102] Based on the same inventive concept, the present application also provides a face defect repair device. Please refer to Figure 8 , Figure 8 which is a schematic diagram of a functional module of the face defect repair device provided in the embodiment of the present application. The embodiment of the present application can divide the face defect repair device 400 according to the method embodiments executed by the computer device, that is, the following respective functional modules corresponding to the face defect repair device 400 can be used to execute the above respective method embodiments. Among them, the face defect repair device 400 can include an acquisition module 410, a preprocessing module 420, an image processing module 430, and a repair module 440. The functions of each functional module of the face defect repair device will be elaborated in detail below.

[0103] The acquisition module 410 is used to acquire a first face image to be repaired.

[0104] The first face image can be obtained by the image processing unit (GPU) in the computer device 100 in real time by processing a video stream, or actively uploaded by the host device 200 to the computer device 100. The first face image includes the face image of the host or other objects.

[0105] In this embodiment, the obtaining module 410 can be used to execute the above step S11. For the detailed implementation manner of the obtaining module 410, reference can be made to the detailed description of step S11 above.

[0106] The preprocessing module 420 is used to preprocess the face image to be repaired to obtain a second face image after preprocessing. Among them, the resolution of the first face image is greater than that of the second face image.

[0107] The resolution of the second face image is less than that of the first face image, and the resolution of the second face image can be a fixed value, which can reduce the computational complexity and also fix the input resolution of the image processing model, ensuring that part of the performance consumption of the image processing model will not increase with the increase of the resolution, and realizing the repair of the defective face image with high resolution. In addition, the second face image with low resolution can enable the image processing model to focus more on extracting face features, which helps to reduce the influence of light. Exemplarily, in this step, a preprocessor Px can be constructed to preprocess the first face image to obtain a second face image with a resolution of 256×192.

[0108] In this embodiment, the preprocessing module 420 can be used to execute the above step S12. For the detailed implementation manner of the preprocessing module 420, reference can be made to the detailed description of step S12 above.

[0109] The image processing module 430 is used to input the second face image into a pre-trained image processing model to obtain a linear transformation parameter map corresponding to the face image after preprocessing. The resolution of the linear transformation parameter map is lower than that of the second face image.

[0110] The resolution of the linear transformation parameter map is lower than that of the second face image, and the resolution of the linear transformation parameter map can be a fixed value, which can retain the linear transformation parameters while further improving the processing rate of the model and enhancing the model performance. Exemplarily, the resolution of the linear transformation parameter map can be 64×48.

[0111] In this embodiment, the image processing module 430 can be used to execute the above step S13. For the detailed implementation manner of the image processing module 430, reference can be made to the detailed description of step S13 above.

[0112] The repair module 440 is used to repair the first face image based on the linear transformation parameter map to obtain a face image after defect repair.

[0113] Infer a linear transformation parameter map including linear transformation parameters on a fixed low-resolution second face image, and apply it to a high-resolution first face image. Apply a linear transformation to each pixel in the first face image to adjust its original value to achieve the effect of defect repair.

[0114] In this embodiment, the repair module 440 can be used to execute the above step S14. For the detailed implementation manner of the repair module 440, reference can be made to the detailed description of step S14 above.

[0115] It should be noted that it should be understood that the division of each module in the above device or system is only a logical function division. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software (for example, open source software) that can be called by a processor; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processor, and some modules can be implemented in the form of hardware. As an example, the image processing module 430 can be implemented by a separate processor running, and can be stored in the memory of the above device or system in the form of program code, and called and executed by a certain processor of the above device or system to perform the functions of the above image processing module 430. The implementation of other modules is similar and will not be elaborated here. In addition, these modules can be fully or partially integrated together or independently implemented. The processor described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step or each module in the above technical solution can be completed through the integrated logic circuit in the processor or in the form of executing a software program.

[0116] Please refer to Figure 9 , Figure 9 shows a schematic hardware structure diagram of a computer device 100 provided by an embodiment of the present disclosure for implementing the above-mentioned face defect repair method. As Figure 9 shown, the computer device 100 may include a processor 110, a computer-readable storage medium 120, a bus 130, and a communication unit 140.

[0117] In a specific implementation process, the processor 110 executes computer execution instructions stored in the computer-readable storage medium 120 (for example, Figure 8 each module in the face defect repair device 400 shown), so that the processor 110 can execute the face defect repair method in the above method embodiment. Among them, the processor 110, the computer-readable storage medium 120, and the communication unit 140 can be connected through the bus 130.

[0118] For the specific implementation process of the processor 110, reference can be made to the various method embodiments executed by the above computer device 100. Their implementation principles and technical effects are similar, and will not be elaborated here in the embodiments of the present application.

[0119] The computer-readable storage medium 120 can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. Among them, the memory 111 is used to store programs or data.

[0120] The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, the buses in the drawings of the present application are not limited to only one bus or one type of bus.

[0121] In the interaction scenario provided by the embodiments of the present application, the communication unit 140 can be used to communicate with the host end 200 and the viewer end 300 to achieve data interaction between the computer device 100, the host end 200, and the viewer end 300.

[0122] In addition, the embodiments of the present application also provide a readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the above face flaw repair method is implemented.

[0123] In summary, the embodiments of the present application provide a face flaw repair method, device, and computer-readable storage. In the above method, after converting the first face image into a second face image with a low resolution, the image processing model is used to extract features from the second face image to infer the linear transformation parameters, and finally, based on the linear transformation parameters, a local linear transformation is applied to the first face image with a high resolution to automatically detect and repair the flaw area in the face. In addition, the input resolution and output resolution of the image processing model can be fixed by reducing the resolution, ensuring that some performance consumption of the image processing model does not increase with the increase in resolution, so that the image processing model can still maintain excellent performance when processing flaw faces with a high resolution.

[0124] The above are only the preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A method for repairing facial flaws, characterized in that, The method includes: Obtain a first face image to be repaired; Preprocess the first face image to obtain a second face image after preprocessing, wherein the resolution of the first face image is greater than that of the second face image; Input the second face image into a pre-trained image processing model to obtain a linear transformation parameter map corresponding to the preprocessed face image, and the resolution of the linear transformation parameter map is lower than that of the second face image, wherein the linear transformation parameter map is composed of linear transformation parameters inferred by the image processing model for feature extraction of the second face image; Perform bilinear interpolation magnification on the linear transformation parameter map to obtain a repair parameter map with the same resolution as the first face image, and the repair parameter map includes a multiplication parameter map and a bias parameter map, wherein the multiplication parameter map includes multiplication scaling factors applied to each channel of the input, and the bias parameter map includes biases applied to each channel of the input; Apply a linear transformation corresponding to the multiplication parameter map and the bias parameter map to each pixel of the first face image to obtain a face image with blemishes repaired.

2. The face defect repair method according to claim 1, characterized in that The step of applying a linear transformation corresponding to the multiplication parameter map and the bias parameter map to each pixel of the first face image to obtain a face image with blemishes repaired includes: Multiply the corresponding pixel value in the first face image by the scaling factor in the multiplication parameter map and then add the offset in the bias parameter map to obtain the repaired pixel value, and obtain a face image with blemishes repaired from the repaired pixel values.

3. The face defect repair method according to claim 1, wherein, The step of preprocessing the first face image to obtain a second face image after preprocessing includes: Reduce the resolution of the first face image, perform normalization processing on the first face image with reduced resolution, and map the pixel values of the pixel points in the first face image to a preset pixel value range to obtain a second face image after preprocessing.

4. The face defect repair method according to claim 1, wherein Before the step of inputting the second face image into a pre-trained image processing model to obtain a linear transformation parameter map corresponding to the preprocessed face image, the method further includes a step of training to obtain an image processing model, and this step includes: Obtain a sample image pair, wherein the sample image pair includes a blemished face sample image to be repaired and a corresponding flawless face sample image after blemishes are repaired; Preprocess the blemished face sample image to obtain a preprocessed blemished face sample image, and the resolution of the preprocessed blemished face sample image is lower than the original resolution of the blemished face sample image; Input the preprocessed blemished face sample image into an image training model for training to obtain a training linear transformation parameter map corresponding to the preprocessed blemished face sample image, and the resolution of the training linear transformation parameter map is less than that of the preprocessed blemished face sample image, wherein the image training model is composed of a downsampling module, an upsampling module, and an output layer connected in series; Repair the defective face sample image based on the trained linear transformation parameter map to obtain a sample face image after defect repair; Calculate the loss function value of the image training model based on the flawless face sample image and the sample face image, compare the loss function value with a preset loss function threshold, and when the loss function value is greater than the preset loss function threshold, update the model parameters in the image training model and repeat the above steps until the loss function value is less than the preset loss function threshold or the number of iterative updates of the model parameters reaches a preset number, end the training of the image training model, and use the image training model corresponding to the completion of training as the trained image processing model.

5. The face defect repair method according to claim 4, wherein, Before the step of inputting the preprocessed defective face sample image into the image training model for training to obtain the trained linear transformation parameter map corresponding to the preprocessed defective face sample image, the method further includes: Construct a downsampling module with four convolutional layers, an upsampling module with two convolutional layers, and an output layer formed by a 1x1 convolutional layer respectively; Connect the downsampling module, the upsampling module, and the output layer in series to obtain the image training model.

6. The face defect repair method according to claim 4, characterized in that, The step of repairing the defective face sample image based on the trained linear transformation parameter map to obtain a sample face image after defect repair includes: Perform bilinear interpolation magnification on the trained linear transformation parameter map to obtain a trained repair parameter map with the same resolution as the defective face sample image, where the trained repair parameter map includes a trained multiplication parameter map and a trained bias parameter map; Apply a linear transformation corresponding to the trained multiplication parameter map and the trained bias parameter map to each pixel of the defective face sample image to obtain a sample face image after defect repair.

7. The face flaw repair method according to claim 6, characterized in that The step of applying a linear transformation corresponding to the trained multiplication parameter map and the trained bias parameter map to each pixel of the defective face sample image to obtain a sample face image after defect repair includes: Multiply the corresponding pixel value in the defective face sample image by the scaling factor in the trained multiplication parameter map and then add the offset in the trained bias parameter map to obtain the repaired pixel value, and obtain a sample face image with defect repair completed from the repaired pixel value.

8. A face flaw repair device, characterized in that, Includes: An acquisition module for acquiring a first face image to be repaired; A preprocessing module for preprocessing the first face image to obtain a preprocessed second face image, where the resolution of the first face image is greater than the resolution of the second face image; An image processing module for inputting the second face image into a pre-trained image processing model to obtain a linear transformation parameter map corresponding to the preprocessed face image, where the resolution of the linear transformation parameter map is lower than the resolution of the second face image, and where the linear transformation parameter map is composed of linear transformation parameters inferred by the image processing model for feature extraction of the second face image; A repair module is configured to perform bilinear interpolation magnification on the linear variation parameter map to obtain a repaired parameter map with the same resolution as that of the first face image. The repaired parameter map includes a multiplication parameter map and a bias parameter map. Among them, the multiplication parameter map includes multiplication scaling factors applied to each channel of the input, and the bias parameter map includes biases applied to each channel of the input. A linear variation corresponding to the multiplication parameter map and the bias parameter map is applied to each pixel of the first face image to obtain a face image with repaired defects.

9. A computer-readable storage medium, characterized in that, A computer program is stored thereon, characterized in that when the computer program is executed by a processor, it implements the face defect repair method according to any one of claims 1-7.

Images