Makeup migration method and device, face alignment method and device
Through global and local alignment technology, the problem of poor makeup migration caused by the difference in face angle and morphology is solved, the precise alignment of the face and the accurate migration of makeup are achieved, and the effect of makeup migration is improved.
Patent Information
- Application Number
- CN202110881316.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-08-02
AI Technical Summary
The existing global makeup transfer method based on deep learning is difficult to achieve accurate alignment when dealing with face angle and morphology differences, resulting in poor makeup transfer results.
By obtaining the global key points of the face to be made up and the face to be made up, performing global alignment and local alignment, constructing transformation formulas for affine, perspective or thin plate spline function transformation, generating intermediate images and face alignment images, Realize the precise migration of makeup.
It realizes accurate alignment of human faces, improves the accuracy and effect of makeup migration, and reduces errors at different angles and forms.
Smart Images

Figure CN113793248B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a face alignment method and device, a makeup migration method and device, an electronic device, and a computer-readable storage medium. Background Art
[0002] With the development of AI technology, the application scenarios of facial technology are becoming more and more diversified. In addition to common face recognition and other problems, deep learning is combined with various facial technologies. For two pictures, one is the original picture without makeup, and the other is a makeup reference picture with makeup added. Through the neural network, the makeup of the reference picture is transferred to the original face without makeup.
[0003] This makeup transfer technology has certain technical feasibility and high practical value. However, due to the differences in the angle and face shape of the original image and the reference image, the global makeup transfer method based on deep learning has certain deficiencies in capturing the key makeup areas. Therefore, it is necessary to better deform the face data to assist in more accurate matching of the makeup areas during training.
[0004] In practical applications, if we want to transfer makeup between two faces, we must hope that the lighting, angle and other conditions of the faces in the two pictures are as similar as possible, so that there will be fewer problems in the migration process and the effect will be better. However, in practical applications, it is difficult to ensure that the angles and shapes of the faces in the two photos are aligned. At present, the existing alignment methods based on traditional operators are generally rough and it is difficult to align accurately at the pixel level. Summary of the invention
[0005] The embodiment of the present application provides a makeup migration method for achieving accurate makeup migration through precise alignment.
[0006] The present application embodiment provides a makeup migration method, comprising:
[0007] Obtaining the global key point positions of the face image to be made up and the global key point positions of the face image already made up;
[0008] According to the global key point positions of the face image to be made-up and the global key point positions of the face image with made-up, the face image with made-up is globally aligned to obtain an intermediate image;
[0009] According to the key point positions of the target part in the face image to be made-up and the key point positions of the target part in the intermediate image, local alignment is performed on the target part of the intermediate image to obtain a face alignment image corresponding to the face image with made-up;
[0010] Makeup migration is performed based on the face image to be made-up and the face alignment image corresponding to the face image with makeup, so as to obtain a makeup image corresponding to the face image to be made-up.
[0011] In one embodiment, obtaining the global key point positions of the face image to be made up and the global key point positions of the face image already made up includes:
[0012] Obtaining a face image to be made-up and a face image already made-up;
[0013] Global key point detection is performed on the face image to be made-up and the face image with made-up, respectively, to obtain the global key point positions of the face image to be made-up and the global key point positions of the face image with made-up.
[0014] In one embodiment, a face image to be made up and a face image already made up are obtained;
[0015] Get the original image and reference image;
[0016] Performing face detection on the original image and the reference image respectively, to obtain a face image to be made-up corresponding to the original image and a face image with made-up corresponding to the reference image.
[0017] In one embodiment, the globally aligning the made-up face images according to the global key point positions of the face images to be made-up and the global key point positions of the face images with made-up to obtain the intermediate images includes:
[0018] Constructing a first transformation formula according to the correspondence between the key point positions of the global face image to be made-up and the key point positions of the global face image already made-up;
[0019] According to the first transformation formula, an affine transformation, a perspective transformation or a thin plate spline function transformation is performed on the makeup-applied facial image to obtain the intermediate image.
[0020] In one embodiment, locally aligning the target part of the intermediate image according to the key point positions of the target part in the face image to be made-up and the key point positions of the target part in the intermediate image includes:
[0021] constructing a second transformation formula according to the correspondence between the key point positions of the target part of the face image to be made-up and the key point positions of the target part of the intermediate image;
[0022] According to the second transformation formula, an affine transformation, a perspective transformation or a thin plate spline function transformation is performed on the target part of the intermediate image.
[0023] In one embodiment, performing makeup migration according to the face image to be made-up and the face alignment image corresponding to the face image with makeup to obtain the makeup image corresponding to the face image to be made-up includes:
[0024] The face alignment images corresponding to the face image to be made-up and the face image with made-up are used as inputs of a makeup transfer model, and a makeup image corresponding to the face image to be made-up outputted by the makeup transfer model is obtained.
[0025] In one embodiment, before using the face alignment images corresponding to the to-be-made-up face image and the made-up face image as inputs to the makeup transfer model, the method further includes:
[0026] Obtaining a user image to be made-up, an image of a user who has already made-up, and a makeup reference image;
[0027] Performing global alignment and local alignment on the makeup reference image to obtain a makeup aligned image;
[0028] The image of the user to be made-up and the makeup alignment image are used as inputs of a neural network, and the parameters of the neural network are adjusted so that the error between the makeup image output by the neural network and the image of the user who has already made-up is minimized, thereby obtaining the makeup transfer model.
[0029] The present application embodiment provides a face alignment method, including:
[0030] Obtain a template face image and a target face image;
[0031] Performing global key point detection on the template face image and the target face image respectively to obtain the global key point positions of the template face image and the global key point positions of the target face image;
[0032] According to the global key point positions of the template face image and the global key point positions of the target face image, the target face image is globally aligned to obtain a first aligned image;
[0033] According to the key point positions of the target part in the template face image and the key point positions of the target part in the first aligned image, the target part of the intermediate image is locally aligned to obtain a face aligned image corresponding to the target face image.
[0034] The embodiment of the present application provides a makeup migration device, comprising:
[0035] A key point detection module, used to obtain the key point positions of the global face image to be made up and the key point positions of the global face image with made up;
[0036] A global alignment module, used to globally align the makeup-applied face images according to the global key point positions of the to-be-applied face images and the global key point positions of the makeup-applied face images, to obtain an intermediate image;
[0037] A local alignment module, configured to locally align the target part of the intermediate image according to the key point positions of the target part in the face image to be made-up and the key point positions of the target part in the intermediate image, so as to obtain a face alignment image corresponding to the face image with made-up;
[0038] The makeup migration module is used to perform makeup migration according to the face image to be made-up and the face alignment image corresponding to the face image with makeup, so as to obtain the makeup image corresponding to the face image to be made-up.
[0039] The embodiment of the present application provides a face alignment device, comprising:
[0040] An image acquisition module is used to acquire a template face image and a target face image;
[0041] A key point detection module is used to perform global key point detection on the template face image and the target face image respectively, to obtain the global key point positions of the template face image and the global key point positions of the target face image;
[0042] A global alignment module, used to globally align the target face image according to the global key point positions of the template face image and the global key point positions of the target face image, to obtain a first aligned image;
[0043] The local alignment module is used to locally align the target part of the first aligned image according to the key point positions of the target part in the template face image and the key point positions of the target part in the first aligned image, so as to obtain the face aligned image corresponding to the target face image.
[0044] An embodiment of the present application provides an electronic device, the electronic device comprising:
[0045] processor;
[0046] a memory for storing processor-executable instructions;
[0047] Wherein, the processor is configured to execute the above-mentioned makeup migration method or face alignment method.
[0048] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor to complete the above-mentioned makeup migration method or face alignment method.
[0049] The technical solution provided by the above-mentioned embodiments of the present application detects the global key point positions of the face image to be made-up and the global key point positions of the face image with makeup, and globally aligns the face images with makeup to obtain an intermediate image. Then, according to the key point positions of the target parts in the face image to be made-up and the key point positions of the target parts in the intermediate image, the intermediate image is locally aligned to obtain a face alignment image. Thus, precise alignment can be achieved through global alignment and local alignment, and the makeup of the aligned face alignment image can be migrated to the face image to be made-up, so as to obtain the makeup image corresponding to the face image to be made-up, thereby achieving accurate makeup migration. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.
[0051] Figure 1 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;
[0052] Figure 2 A schematic diagram of a makeup migration method according to an embodiment of the present application;
[0053] Figure 3 yes Figure 2 Detailed flowchart of step S210 in the corresponding embodiment;
[0054] Figure 4 A schematic diagram of the training process of the makeup transfer model provided in one embodiment of the present application;
[0055] Figure 5 A schematic diagram of a flow chart of a face alignment method provided in an embodiment of the present application;
[0056] Figure 6 A block diagram of a makeup migration device provided in one embodiment of the present application;
[0057] Figure 7 A block diagram of a face alignment device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0058] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0059] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0060] Figure 1 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 can be used to execute the makeup migration method or face alignment method provided in an embodiment of the present application. Figure 1 As shown, the electronic device 100 includes: one or more processors 102, and one or more memories 104 storing processor executable instructions. The processor 102 is configured to execute the makeup migration method or face alignment method provided in the following embodiments of the present application.
[0061] The processor 102 can be a gateway, a smart terminal, or a device including a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic device 100 and control other components in the electronic device 100 to perform desired functions.
[0062] The memory 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the makeup migration method or face alignment method described below. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the application.
[0063] In one embodiment, Figure 1 The electronic device 100 shown may also include an input device 106, an output device 108, and a data acquisition device 110, which are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device 100 may also have other components and structures as required.
[0064] The input device 106 may be a device for a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, etc. The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, etc. The data acquisition device 110 may acquire an image of an object, and store the acquired image in the memory 104 for use by other components. Exemplarily, the data acquisition device 110 may be a camera.
[0065] In one embodiment, the various components in the example electronic device 100 for implementing the makeup migration method or face alignment method of the embodiment of the present application can be integrated or dispersed, such as integrating the processor 102, memory 104, input device 106 and output device 108 into one, and separately arranging the data acquisition device 110.
[0066] In one embodiment, the example electronic device 100 for implementing the makeup migration method or face alignment method of the embodiment of the present application can be implemented as a smart terminal such as a smart phone, a tablet computer, a smart watch, a car-mounted device, etc.
[0067] Figure 2 is a flow chart of the makeup migration method provided in the embodiment of the present application. Figure 2 As shown, the method includes the following steps S210-S240.
[0068] Step S210: Obtaining the global key point positions of the face image to be made-up and the global key point positions of the face image already made-up.
[0069] The face image to be made up refers to an image of a face without makeup, and the face image with makeup refers to an image of a face with makeup. Makeup transfer refers to copying the makeup of the face image with makeup to the face image to be made up, such as transferring lipstick, skin color, eye shadow, etc. to the face image to be made up.
[0070] The global key point position refers to the position of all key points on the entire face. The number of key points can be 68, including the corners of the eyes, the corners of the mouth, the corners of the eyebrows, the tip of the nose, etc. The position can be expressed by the coordinates of the key points in the image coordinate system. For example, a two-dimensional coordinate system can be established with a vertex of the image as the origin of the coordinates.
[0071] In one embodiment, the global key point positions of the facial image to be made-up and the global key point positions of the facial image with makeup can be obtained in advance and stored locally or on the server side, so that when the makeup is transferred, the stored global key point positions of the facial image to be made-up and the global key point positions of the facial image with makeup can be directly obtained.
[0072] In one embodiment, if Figure 3 As shown, the above step S210 specifically includes the following steps S211-S212.
[0073] Step S211: Acquire the face image to be made-up and the face image with made-up.
[0074] The face image to be made up can be an image taken by a local camera, and the face image already made up can be an image stored in advance locally or on the server.
[0075] In one embodiment, an original image and a reference image may be obtained, and face detection may be performed on the original image and the reference image respectively to obtain a face image to be made-up corresponding to the original image and a face image with made-up corresponding to the reference image.
[0076] The original image refers to an image containing a face image to be made up, and the reference image refers to an image containing a face image that has been made up. The face image to be made up can be considered as a face area cut out from the original image. The face image that has been made up can be considered as a face area cut out from the reference image.
[0077] For example, the trained RetinaFace (face detector), DenseBox and various cascade convolutional neural networks can be used to perform face detection on the original image and the reference image respectively, so as to obtain the face image to be made up contained in the original image and the face image with makeup contained in the reference image.
[0078] Step S212: performing global key point detection on the face image to be made-up and the face image with made-up respectively, to obtain the global key point positions of the face image to be made-up and the global key point positions of the face image with made-up.
[0079] Global key point detection refers to finding all the key points in the face image to be made up and the face image with makeup. The key point detection method can be to call the dlib library (dlib library is a very classic open source library for image processing) for detection, or to use neural network methods such as DCNN (deep convolutional neural network) and MTCNN (multi-task convolutional neural network). The neural network is trained with sample images with known key point positions, and then the trained neural network model is used to detect all the key point positions of the face image to be made up and the face image with makeup.
[0080] Step S220: performing global alignment on the made-up face images according to the global key point positions of the to-be-made-up face images and the global key point positions of the made-up face images to obtain an intermediate image.
[0081] The intermediate image is the image obtained after global alignment of the makeup face image. Global alignment is relative to local alignment. Global alignment can be to transform the entire makeup face image so that the key point positions of the intermediate image obtained after the transformation are aligned with the key point positions of the makeup face image.
[0082] In one embodiment, a first transformation formula can be constructed based on the correspondence between the global key point positions of the face image to be made-up and the global key point positions of the face image with makeup; and based on the first transformation formula, the face image with makeup is subjected to affine transformation, perspective transformation or thin plate spline function transformation.
[0083] The corresponding relationship refers to the correspondence between a key point in the face image to be made up and a key point in the face image with makeup. For example, the position of the nose tip in the face image to be made up corresponds to the position of the nose tip in the face image with makeup, and the position of the left corner of the mouth in the face image to be made up corresponds to the position of the left corner of the mouth in the face image with makeup. The first transformation formula can be considered as a mapping relationship between the key point position in the face image with makeup and the key point position in the face image to be made up.
[0084] In one embodiment, the first transformation formula may be an affine transformation matrix, a perspective transformation matrix or a thin plate spline function, and the position of each pixel in the makeup face image is transformed according to the affine transformation matrix, the perspective transformation matrix or the thin plate spline function to map the position of the pixel in the intermediate image, wherein the RGB color value of the pixel is retained.
[0085] Thin plate spline function transformation is an interpolation method. The final transformation is based on the key points in the two images. It seeks to minimize the transformation of the whole image energy by aligning the key points under the premise of considering the two-dimensional image as a deformable thin plate. 2 logr(.) is used as the radial basis function kernel, and the distance after radial basis function difference represents the total deformation energy from all pixel positions of the original image to the transformed image. In this process, the total deformation energy is constrained to a minimum, and the overall change amplitude is minimized, and the image registration is completed so that the corresponding key points of the face image to be made up and the face image with made up are aligned.
[0086] Step S230: performing local alignment on the target part of the intermediate image according to the key point positions of the target part in the face image to be made-up and the key point positions of the target part in the intermediate image, so as to obtain a face alignment image corresponding to the face image with made-up;
[0087] The target part may be an eye area, a mouth area, a skin area, a nose area, etc. The face alignment image may be considered as an image obtained by aligning the target parts of the intermediate image.
[0088] Local alignment is relative to global alignment. It can be considered as aligning the eye area of the intermediate image, or aligning the mouth area of the intermediate image, or aligning other preset areas as needed, such as the upper and lower lips, left and right eye sockets, etc. Each part can be locally aligned separately, and the next part can be locally aligned after one part is aligned.
[0089] Among them, the local alignment method can refer to the global alignment method, and can first construct a second transformation formula based on the correspondence between the key point positions of the target part of the face image to be made-up and the key point positions of the target part of the intermediate image; according to the second transformation formula, the target part of the intermediate image is transformed by affine transformation, perspective transformation or thin plate spline function transformation.
[0090] To distinguish it from the first transformation formula above, the mapping relationship between the key points of a certain part in the face image to be made up and the corresponding key points of the part in the intermediate image can be called the second transformation formula. Similar to the first transformation formula, the second transformation formula can also be an affine transformation matrix, a perspective transformation matrix or a thin plate spline function. The position of each pixel point of the target part in the intermediate image is transformed according to the affine transformation matrix, the perspective transformation matrix or the thin plate spline function, and the position of the pixel point in the face alignment image is mapped. Among them, the GRB color information of the pixel point remains unchanged. Among them, local alignment can be completed after global alignment, and direct local alignment may cause distortion of edge pixel areas.
[0091] Step S240: performing makeup migration according to the face image to be made-up and the face alignment image corresponding to the face image with makeup, to obtain a makeup image corresponding to the face image to be made-up.
[0092] Makeup transfer refers to copying the makeup color of a makeup-applied face image to the face image to be made up. In one embodiment, the face image to be made up and the face alignment image corresponding to the makeup-applied face image can be used as inputs of the makeup transfer model to obtain the makeup image corresponding to the face image to be made up.
[0093] Among them, the makeup transfer model can be trained in advance, that is, the existing makeup transfer model can be directly used. The input of the makeup transfer model is the face alignment image corresponding to the face image to be made up and the face image with makeup. The output of the makeup transfer model is the makeup image of the face image to be made up. The makeup image can be considered as the makeup of the face image with makeup transferred to the face image to be made up, and the made-up face image is obtained. Because after the transformation, the coordinates of each part of the different faces in the same set of face images to be made up and the face images with makeup have been aligned, in the calculation of the neural network, the coordinate position can be used to index all aligned pixels. Therefore, the face image to be made up and the face alignment image can be input into the makeup transfer model in the form of data pairs.
[0094] In one embodiment, before the above step S240, a makeup transfer model may be trained. Figure 4 As shown, the makeup transfer model can be trained through the following steps S601 to S603.
[0095] Step S601: Acquire an image of a user who is about to put on makeup, an image of a user who has put on makeup, and a makeup reference image.
[0096] The user image to be made up can be considered as the face image of user A without makeup. The user image with makeup can be considered as the face image of user A after makeup. The makeup reference image can be considered as the face image of user B after makeup.
[0097] Step S602: globally aligning and locally aligning the makeup reference image to obtain a makeup aligned image.
[0098] Among them, the makeup alignment image can be considered as the image obtained after global alignment and local alignment of the makeup reference image. For details, please refer to the alignment method of the face image to be made up and the face image with makeup above. First, global key point detection is performed on the user image to be made up and the makeup reference image. Then, based on the key point positions in the user image to be made up and the key point positions in the makeup reference image, global alignment and local alignment of the target parts are performed to obtain the makeup alignment image of the makeup reference image.
[0099] Step S603: using the image of the user to be made-up and the makeup alignment image as inputs of the neural network, adjusting the parameters of the neural network so that the error between the makeup image output by the neural network and the image of the user who has already made up is minimized, thereby obtaining the makeup transfer model.
[0100] The error can be represented by the distance between the image data of the makeup image output by the neural network and the image data of the user image with makeup. The optimization goal of the neural network is to minimize the difference between the result output by the neural network and the known user image with makeup. The neural network after parameter optimization is the makeup transfer model.
[0101] Figure 5 FIG. 1 is a flow chart of the face alignment method provided in the embodiment of the present application. Figure 5 As shown, the method includes the following steps S710-S740.
[0102] Step S710: Acquire a template face image and a target face image.
[0103] The template face image and the target face image may be face images obtained from the original image through face detection. The template face image and the target face image may be images of different persons or taken at different angles. The face shape or angle of the person in the template face image and the target face image is different.
[0104] Step S720: performing global key point detection on the template face image and the target face image respectively to obtain the global key point positions of the template face image and the global key point positions of the target face image.
[0105] The global key point detection may be the process of performing global key point detection on the face image to be made-up and the face image with made-up in step S212 in the above embodiment.
[0106] Step S730: performing global alignment on the target face image according to the global key point positions of the template face image and the global key point positions of the target face image to obtain a first aligned image.
[0107] The first aligned image refers to an image obtained after the target face image is globally aligned. The global alignment process can refer to the global alignment method of step S220 in the above embodiment.
[0108] Step S740: performing local alignment on the target part of the first aligned image according to the key point positions of the target part in the template face image and the key point positions of the target part in the first aligned image to obtain a face aligned image corresponding to the target face image.
[0109] The local alignment method can refer to the local alignment method of step S230 in the above embodiment. The target face image is globally aligned to obtain a first aligned image, and the first aligned image is locally aligned to each part to obtain a face alignment image corresponding to the target face image. Figure 2The makeup transfer process of the corresponding embodiment can also be applied to face recognition tasks.
[0110] In one embodiment, the face alignment images of the template face image and the target face image can be used as neural network inputs to train the neural network to identify whether the template face image and the target face image belong to the same face. The above face alignment method can better solve the problem of processing faces at different angles and postures.
[0111] The following is an embodiment of the device of the present application, which can be used to execute the above-mentioned makeup transfer method and face alignment method embodiments of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiments of the makeup transfer method and face alignment method of the present application.
[0112] Figure 6 FIG. 1 is a block diagram of a makeup migration device according to an embodiment of the present application. Figure 6 As shown, the device includes: a key point detection module 810, a global alignment module 820, a local alignment module 830 and a makeup migration module 840.
[0113] Key point detection module 810, used to obtain the key point positions of the global face image to be made up and the key point positions of the global face image with made up;
[0114] A global alignment module 820, configured to globally align the makeup-applied facial images according to the global key point positions of the to-be-applied facial images and the global key point positions of the makeup-applied facial images, to obtain an intermediate image;
[0115] A local alignment module 830 is used to locally align the target part of the intermediate image according to the key point positions of the target part in the face image to be made-up and the key point positions of the target part in the intermediate image, so as to obtain a face alignment image corresponding to the face image with made-up;
[0116] The makeup migration module 840 is used to perform makeup migration according to the face alignment image corresponding to the face image to be made-up and the face image with makeup, so as to obtain the makeup image corresponding to the face image to be made-up.
[0117] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned makeup migration method, and will not be repeated here.
[0118] Figure 7 FIG. 1 is a block diagram of a face alignment device according to an embodiment of the present application. Figure 7 As shown, the device includes: an image acquisition module 910, a key point detection module 920, a global alignment module 930 and a local alignment module 940.
[0119] An image acquisition module 910 is used to acquire a template face image and a target face image;
[0120] A key point detection module 920 is used to perform global key point detection on the template face image and the target face image respectively, to obtain the global key point positions of the template face image and the global key point positions of the target face image;
[0121] A global alignment module 930, configured to globally align the target face image according to the global key point positions of the template face image and the global key point positions of the target face image, to obtain a first aligned image;
[0122] The local alignment module 940 is used to locally align the target part of the first aligned image according to the key point positions of the target part in the template face image and the key point positions of the target part in the first aligned image to obtain a face alignment image corresponding to the target face image.
[0123] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned face alignment method, and will not be repeated here.
[0124] In several embodiments provided in the present application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a part of a module, a program segment or a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0125] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0126] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.
Claims
1. A makeup migration method, characterized in that: include: Obtaining the global key point positions of the face image to be made up and the global key point positions of the face image already made up; According to the global key point positions of the face image to be made-up and the global key point positions of the face image with made-up, the face image with made-up is globally aligned to obtain an intermediate image; According to the key point positions of the target part in the face image to be made up and the key point positions of the target part in the intermediate image, the target part of the intermediate image is locally aligned to obtain a face alignment image corresponding to the face image with made up; wherein the global alignment is to align the entire face image with made up, and the local alignment is to align the target part of the intermediate image; Makeup migration is performed based on the face image to be made-up and the face alignment image corresponding to the face image with makeup, so as to obtain a makeup image corresponding to the face image to be made-up.
2. The method according to claim 1, characterized in that The step of obtaining the global key point positions of the face image to be made up and the global key point positions of the face image already made up includes: Obtaining a face image to be made-up and a face image already made-up; Global key point detection is performed on the face image to be made-up and the face image with made-up, respectively, to obtain the global key point positions of the face image to be made-up and the global key point positions of the face image with made-up.
3. The method according to claim 2, characterized in that Obtaining a face image to be made up and a face image already made up; Get the original image and reference image; Performing face detection on the original image and the reference image respectively, to obtain a face image to be made-up corresponding to the original image and a face image with made-up corresponding to the reference image.
4. The method according to claim 1, characterized in that: The step of globally aligning the makeup-applied face images according to the global key point positions of the face images to be made-up and the global key point positions of the makeup-applied face images to obtain the intermediate images comprises: Constructing a first transformation formula according to the correspondence between the key point positions of the global face image to be made-up and the key point positions of the global face image already made-up; According to the first transformation formula, an affine transformation, a perspective transformation or a thin plate spline function transformation is performed on the makeup-applied facial image to obtain the intermediate image.
5. The method according to claim 1, characterized in that The locally aligning the target part of the intermediate image according to the key point position of the target part in the face image to be made-up and the key point position of the target part in the intermediate image comprises: constructing a second transformation formula according to the correspondence between the key point positions of the target part of the face image to be made-up and the key point positions of the target part of the intermediate image; According to the second transformation formula, an affine transformation, a perspective transformation or a thin plate spline function transformation is performed on the target part of the intermediate image.
6. The method according to claim 1, characterized in that Performing makeup migration according to the face alignment image corresponding to the face image to be made-up and the face image already made-up to obtain the makeup image corresponding to the face image to be made-up, including: The face alignment images corresponding to the face image to be made-up and the face image with made-up are used as inputs of a makeup transfer model, and a makeup image corresponding to the face image to be made-up outputted by the makeup transfer model is obtained.
7. The method according to claim 6, characterized in that Before using the face alignment images corresponding to the to-be-made-up face image and the made-up face image as inputs of the makeup transfer model, the method further includes: Obtaining a user image to be made-up, an image of a user who has already made-up, and a makeup reference image; Performing global alignment and local alignment on the makeup reference image to obtain a makeup aligned image; The image of the user to be made-up and the makeup alignment image are used as inputs of a neural network, and the parameters of the neural network are adjusted so that the error between the makeup image output by the neural network and the image of the user who has already made-up is minimized, thereby obtaining the makeup transfer model.
8. A face alignment method, characterized in that: include: Obtain a template face image and a target face image; Performing global key point detection on the template face image and the target face image respectively to obtain the global key point positions of the template face image and the global key point positions of the target face image; According to the global key point positions of the template face image and the global key point positions of the target face image, the target face image is globally aligned to obtain a first aligned image; According to the key point positions of the target part in the template face image and the key point positions of the target part in the first aligned image, the target part of the first aligned image is locally aligned to obtain the face aligned image corresponding to the target face image; wherein, the global alignment is to align the entire made-up face image, and the local alignment is to align the target part of the intermediate image.
9. A makeup migration device, characterized in that: include: A key point detection module, used to obtain the key point positions of the global face image to be made up and the key point positions of the global face image with made up; A global alignment module, used to globally align the makeup-applied face images according to the global key point positions of the to-be-applied face images and the global key point positions of the makeup-applied face images, to obtain an intermediate image; A local alignment module is used to locally align the target part of the intermediate image according to the key point positions of the target part in the face image to be made up and the key point positions of the target part in the intermediate image, so as to obtain a face alignment image corresponding to the face image with made up; wherein, global alignment is to align the entire face image with made up, and local alignment is to align the target part of the intermediate image; The makeup migration module is used to perform makeup migration according to the face image to be made-up and the face alignment image corresponding to the face image with makeup, so as to obtain the makeup image corresponding to the face image to be made-up.
10. A face alignment device, characterized in that: include: An image acquisition module is used to acquire a template face image and a target face image; A key point detection module is used to perform global key point detection on the template face image and the target face image respectively, to obtain the global key point positions of the template face image and the global key point positions of the target face image; A global alignment module, used to globally align the target face image according to the global key point positions of the template face image and the global key point positions of the target face image, to obtain a first aligned image; The local alignment module is used to locally align the target part of the first aligned image according to the key point positions of the target part in the template face image and the key point positions of the target part in the first aligned image, so as to obtain the face aligned image corresponding to the target face image; wherein, the global alignment is to align the entire made-up face image, and the local alignment is to align the target part of the intermediate image.
11. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the makeup migration method described in any one of claims 1-7 or the face alignment method described in claim 8.
12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program can be executed by a processor to complete the makeup migration method described in any one of 1-7 or the face alignment method described in claim 8.
