A CNN-based portrait face migration method, device and equipment

Abstract

The present invention discloses a CNN-based portrait face shape migration method, apparatus, device, and storage medium, which includes: preprocessing a face image to be migrated and a model reference image to obtain corresponding face points and a face shape mask of the model reference image, wherein the model reference image is used to migrate the face shape of the portrait in the model reference image to the face shape reference image of the face image to be migrated; using the image obtained by splicing the face image to be migrated and the face shape mask of the model reference image as input data; inputting the input data into a pre-built face migration network model for migration processing to obtain a migrated face image. The method can achieve the purpose of portrait face shape migration, so that the facial skin of the migrated image is generated in a more natural migration effect.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a CNN-based portrait face shape migration method, device, and equipment. Background Art

[0002] Portrait face shape migration refers to migrating the face shape of any model portrait to the user portrait without changing the facial features of the user image, so that everyone can have a unique face shape, thereby meeting the user's experience needs in face shape switching, and can be widely used in portrait beauty software, beauty cameras and other fields. The existing portrait face shape migration mainly uses traditional image algorithms to migrate face shapes. The traditional image algorithm uses the facial points of the user image and the model image to form a series of grids, and then warps (affine transforms the image) the user image to the same face shape as the model image. However, in actual applications, the main problem of traditional face migration image algorithms is that the facial skin of the migrated image is not natural enough, resulting in a poor user experience. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to propose a CNN-based portrait face migration method, device and equipment, aiming to solve the problem that the existing portrait face migration methods have problems such as the facial skin of the migrated image is not natural enough.

[0004] To achieve the above object, the present invention provides a CNN-based portrait face shape migration method, the method comprising:

[0005] Preprocessing the face image to be migrated and the model reference reference to obtain corresponding face points and a face shape mask of the model reference reference, wherein the model reference reference is used to migrate the face shape of the portrait in the model reference reference to the face shape reference of the face image to be migrated;

[0006] An image obtained by splicing the face image to be migrated and the face mask of the model reference image is used as input data;

[0007] The input data is input into a pre-built face migration network model for migration processing to obtain a migrated face image.

[0008] Preferably, the step of preprocessing the face image to be transferred and the model reference image to obtain corresponding face points and a face mask of the model reference image comprises:

[0009] Using face detection technology to detect and process the face image to be transferred and the model reference image respectively, and then output corresponding face points;

[0010] The faces in the face image to be migrated and the model reference image are respectively cut out according to the face points, and the face mask of the model reference image is segmented by using image segmentation technology.

[0011] Preferably, the training process of the pre-built face transfer network model includes:

[0012] Using the facial points, faces with the same contour in the image training set are classified to obtain N sets based on the image training set;

[0013] The same face shape adjustment is performed on the image data in each of the N sets using a grid deformation algorithm, and paired training data is generated to train the face migration network model.

[0014] Preferably, the step of training the face transfer network model further includes:

[0015] A loss function is calculated for the result image output by the face transfer network model and the target image, and the face transfer network model is optimized according to the loss function.

[0016] Preferably, the loss function includes L1 loss, perceptual VGG loss and facial contour local loss function, wherein the facial contour local loss function includes the L1 loss obtained by cropping the target image of the face points and the face edge pixel area of ​​the result image output by the face transfer network model.

[0017] Preferably, the network framework of the face transfer network model includes a gated convolution layer, a convolution layer and an upsampling layer.

[0018] Preferably, the step of using the image obtained by splicing the face image to be migrated and the face mask of the model reference image as input data includes:

[0019] The face image to be migrated and the face mask of the model reference image are spliced ​​into a four-channel image as input data.

[0020] To achieve the above object, the present invention further provides a CNN-based portrait face shape migration device, the device comprising:

[0021] A processing unit, configured to pre-process the face image to be transferred and the model reference reference to obtain corresponding face points and a face shape mask of the model reference reference, wherein the model reference reference is used to transfer the face shape of the portrait in the model reference reference to the face shape reference of the face image to be transferred;

[0022] A splicing unit, configured to splice the face image to be migrated and the face mask of the model reference image to obtain an image as input data;

[0023] The migration unit is used to input the input data into a pre-built face migration network model for migration processing to obtain a migrated face image.

[0024] In order to achieve the above-mentioned objectives, the present invention also proposes a device, including a processor, a memory, and a portrait face migration program stored in the memory, wherein the portrait face migration program is executed by the processor to implement the steps of a CNN-based portrait face migration method as described in the above embodiment.

[0025] In order to achieve the above-mentioned objectives, the present invention also proposes a computer-readable storage medium, on which a portrait face migration program is stored. The portrait face migration program is executed by a processor to implement the steps of a CNN-based portrait face migration method as described in the above embodiment.

[0026] Beneficial effects:

[0027] The above scheme pre-processes the face image to be migrated and the model reference image, splices the face masks of the face image to be migrated and the model reference image into a four-channel image as input data, and inputs it into the designed face migration network model for migration processing, thereby achieving the purpose of portrait face migration, making the generation of facial skin in the image after migration more natural, thereby enhancing the convenience and scope of use in practical applications, and improving the user experience.

[0028] The above scheme calculates the loss function on the output image and target image of the face transfer network model during the training process, and continuously optimizes the face transfer network model under the supervision of the loss function. In addition, by adding a local loss function for the facial contour, the facial contour of the transferred image is made smoother, greatly improving the natural effect of the face in the transferred image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 A flowchart of a CNN-based portrait face shape migration method provided by one embodiment of the present invention.

[0031] Figure 2 A schematic diagram of the network framework of a face migration network model provided by another embodiment of the present invention.

[0032] Figure 3 A schematic diagram of the structure of a CNN-based portrait face shape migration device provided by one embodiment of the present invention.

[0033] The realization of the objectives of the invention, the functional features and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] The present invention is described in detail below with reference to the embodiments.

[0036] Reference Figure 1 FIG2 is a flow chart of a CNN-based portrait face shape migration method provided by one embodiment of the present invention.

[0037] In this embodiment, the method includes:

[0038] S11, pre-processing the face image to be migrated and the model reference reference to obtain corresponding face points and a face shape mask of the model reference reference, wherein the model reference reference is used to migrate the portrait face shape in the model reference reference to the face shape reference of the face image to be migrated.

[0039] The step of preprocessing the face image to be transferred and the model reference image to obtain corresponding face points and the face mask of the model reference image comprises:

[0040] S11-1, using face detection technology to detect and process the face image to be transferred and the model reference image respectively, and then output corresponding face points;

[0041] S11-2, cutting out faces from the face image to be migrated and the model reference image respectively according to the face points, and segmenting the face mask of the model reference image using image segmentation technology.

[0042] In this embodiment, image data requires preprocessing, primarily through the use of face detection technology to output facial points. These points are then used to crop the faces in the face image to be transferred and the model reference image from their respective images. Image segmentation technology is then used to segment the face mask of the model reference image. Furthermore, using face detection technology to perform facial point detection on an image involves locating key facial regions, including eyebrows, eyes, nose, mouth, and facial contours, within a given face image, thereby obtaining the facial points corresponding to the face image. Face detection technology involves employing a specific strategy to search a given image to determine whether it contains a face. If so, the location, size, and pose of the face are returned. Image segmentation technology involves dividing an image into several specific regions with unique properties and identifying objects of interest. It is a key step in the transition from image processing to image analysis. Existing image segmentation methods are primarily categorized into threshold-based segmentation, region-based segmentation, edge-based segmentation, and segmentation methods based on specific theories. From a mathematical perspective, image segmentation is the process of dividing a digital image into disjoint regions. Image segmentation is also a labeling process, assigning the same number to pixels belonging to the same region. Furthermore, image masking is the process of blocking (completely or partially) the image being processed with a selected image, graphic, or object to control the region or process of image processing.

[0043] S12: Using an image obtained by splicing the face image to be migrated and the face mask of the model reference image as input data.

[0044] Furthermore, the step of using the image obtained by splicing the face image to be migrated and the face mask of the model reference image as input data includes:

[0045] The face image to be transferred and the face mask of the model reference are spliced ​​into a four-channel image as input data, wherein the four channels include the face image to be transferred as an RGB three-channel image and the face mask of the model reference as a single-channel image.

[0046] S13, inputting the input data into a pre-built face transfer network model for transfer processing to obtain a transferred face image.

[0047] Among them, the network framework of the face migration network model includes a gated convolution layer, a convolution layer and an upsampling layer.

[0048] like Figure 2The network framework of the face transfer network model shown in Figure 1 is a schematic diagram. In this embodiment, the network framework of the face transfer network model mainly consists of a gated convolution layer, a convolution layer, and an upsampling layer. There are four downsampling and upsampling processes. The feature layer of the front and the upsampled features of the back are concatenated together through concat, which facilitates more natural facial skin generation. The gated convolution can effectively guide the network to learn the face shape of the model reference image. The entire network can achieve face shape transfer without changing the facial features of the face image to be transferred, thereby ensuring a more natural face shape transfer result and no deformation of the facial features.

[0049] The training process of the pre-built face transfer network model includes:

[0050] Using the facial points, faces with the same contour in the image training set are classified to obtain N sets based on the image training set;

[0051] The same face shape adjustment is performed on the image data in each of the N sets using a grid deformation algorithm, and paired training data is generated to train the face migration network model.

[0052] Furthermore, the step of training the face migration network model further includes:

[0053] A loss function is calculated for the result image output by the face transfer network model and the target image, and the face transfer network model is optimized according to the loss function.

[0054] The loss function includes L1 loss, perceptual VGG loss and facial contour local loss function; wherein the facial contour local loss function includes the L1 loss obtained by cropping the target image of the face points and the face edge pixel area of ​​the result image output by the face transfer network model.

[0055] In this embodiment, first, the faces with the same contour in the training set are put together by using the facial points, and finally the training set is divided into N sets; the image data in each set are adjusted to the same face shape by the grid deformation algorithm, and paired training data are produced for network training. Furthermore, the training loss function specifically includes: calculating L1 loss, perceptual vgg loss, and facial contour local loss function for the network output result and the target image. Among them, the target image is the image data in the image training set, and the target image indicates that the face shape is the same as the model reference image and the person is the user image. Further, using the formula Find L1loss, where y represents the network output migration result graph, Represents the target graph; use the vgg-19 network to calculate the vgg loss, that is use Calculate the local loss function of the facial contour, where y i Represents the pixel area cropped around the i-th face point in the network output. Represents the pixel area cropped around the target i-th facial point. Among them, the facial contour local loss function can make the generated facial contour smoother. The facial contour local loss function refers to calculating the L1 loss of the face edge pixel area of ​​the target image and the network output result image by cropping the facial points. Finally, by inputting data into the network for training, the model is continuously optimized under the supervision of the loss function, and finally an optimal face migration network model is output. In this embodiment, a new face migration network framework is designed to produce paired data for CNN network training. At the same time, in order to solve the problem of uneven facial contour, a contour local loss function is added. Therefore, the face can be migrated while making the migrated facial skin more natural and the facial features not deformed.

[0056] Reference Figure 3 FIG2 is a schematic structural diagram of a CNN-based portrait face shape migration device provided by one embodiment of the present invention.

[0057] In this embodiment, the device 30 includes:

[0058] A processing unit 31 is configured to pre-process the face image to be transferred and the model reference reference to obtain corresponding face points and a face shape mask of the model reference reference, wherein the model reference reference is used to transfer the face shape of the portrait in the model reference reference to the face shape reference of the face image to be transferred;

[0059] A splicing unit 32 is configured to splice the face image to be migrated and the face mask of the model reference image to obtain an image as input data;

[0060] The migration unit 33 is used to input the input data into a pre-built face migration network model for migration processing to obtain a migrated face image.

[0061] Further preferably, the processing unit 31 further includes:

[0062] A face point detection unit, configured to use face detection technology to detect and process the face image to be transferred and the model reference image respectively, and then output corresponding face points;

[0063] The image cropping unit is used to crop the faces in the face image to be migrated and the model reference image respectively according to the face points, and to segment the face mask of the model reference image using image segmentation technology.

[0064] Further preferably, the training process of the pre-built face transfer network model includes:

[0065] Using the facial points, faces with the same contour in the image training set are classified to obtain N sets based on the image training set;

[0066] The same face shape adjustment is performed on the image data in each of the N sets using a grid deformation algorithm, and paired training data is generated to train the face migration network model.

[0067] Further preferably, a loss function is calculated for the result image output by the face transfer network model and the target image, and the face transfer network model is optimized according to the loss function.

[0068] Further preferably, the loss function includes L1 loss, perceptual VGG loss and facial contour local loss function, wherein the facial contour local loss function includes the L1 loss obtained by cropping the target image of the face points and the face edge pixel area of ​​the result image output by the face transfer network model.

[0069] Further preferably, the network framework of the face transfer network model includes a gated convolution layer, a convolution layer and an upsampling layer.

[0070] Further preferably, the splicing unit 32 is further used for:

[0071] The face image to be migrated and the face mask of the model reference image are spliced ​​into a four-channel image as input data.

[0072] Each unit module of the device 30 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the corresponding steps above for details.

[0073] The embodiment of the present invention further provides a device, which includes the above-mentioned CNN-based portrait face transfer device, wherein the CNN-based portrait face transfer device can be used Figure 3 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments and will not be repeated here.

[0074] The device includes: a mobile phone, digital camera, tablet computer, or other device with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0075] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0076] The input unit can be used to receive input digital, character, or image information, and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0077] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel. Optionally, the display panel can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Furthermore, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event. The processor then provides a corresponding visual output on the display panel based on the type of touch event.

[0078] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The computer-readable storage medium may be a read-only memory, a disk, or an optical disk.

[0079] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For similar or identical parts between the various embodiments, reference can be made to each other. For the apparatus embodiments, device embodiments, and storage medium embodiments, since they are generally similar to the method embodiments, their descriptions are relatively simple. For relevant parts, reference can be made to the descriptions of the method embodiments.

[0080] Furthermore, in this document, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0081] While the foregoing description shows and describes preferred embodiments of the present invention, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments, and can be modified within the scope of the present invention by the teachings herein or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. A CNN-based portrait face transfer method, characterized in that: The method comprises: Preprocessing the face image to be migrated and the model reference reference to obtain corresponding face points and a face shape mask of the model reference reference, wherein the model reference reference is used to migrate the face shape of the portrait in the model reference reference to the face shape reference of the face image to be migrated; An image obtained by splicing the face image to be migrated and the face mask of the model reference image is used as input data; Inputting the input data into a pre-built face transfer network model for transfer processing to obtain a transferred face image; Furthermore, the training process of the pre-built face transfer network model includes: Using the facial points, faces with the same contour in the image training set are classified to obtain N sets based on the image training set; Performing the same face shape adjustment on the image data in each of the N sets using a grid deformation algorithm to generate paired training data for training the face transfer network model; Calculating a loss function on the result image output by the face transfer network model and the target image, and optimizing the face transfer network model according to the loss function; Furthermore, the loss function includes L1 loss, perceptual VGG loss and facial contour local loss function, wherein the facial contour local loss function includes L1 loss obtained by cropping the target image of the face points and the face edge pixel area of ​​the result image output by the face transfer network model; using Calculate, where y represents the result graph of the network output. Represents the target image; using to perform calculations; to use Calculate the local loss function of the facial contour, where y i Represents the pixel area cropped around the i-th face point in the network output result graph, Represents the pixel area cropped around the i-th face point in the target image; Furthermore, the network framework of the face transfer network model includes a gated convolution layer, a convolution layer and an upsampling layer.

2. The CNN-based portrait face shape migration method according to claim 1, characterized in that: The step of pre-processing the face image to be transferred and the model reference image to obtain corresponding face points and a face mask of the model reference image comprises: Using face detection technology to detect and process the face image to be transferred and the model reference image respectively, and then output corresponding face points; The faces in the face image to be migrated and the model reference image are respectively cut out according to the face points, and the face mask of the model reference image is segmented by using image segmentation technology.

3. The CNN-based portrait face shape migration method according to claim 1, characterized in that: The step of using the image obtained by splicing the face image to be migrated and the face mask of the model reference image as input data includes: The face image to be migrated and the face mask of the model reference image are spliced ​​into a four-channel image as input data.

4. A CNN-based portrait face transfer device, characterized in that: The device comprises: A processing unit, configured to pre-process the face image to be transferred and the model reference reference to obtain corresponding face points and a face shape mask of the model reference reference, wherein the model reference reference is used to transfer the face shape of the portrait in the model reference reference to the face shape reference of the face image to be transferred; A splicing unit, configured to splice the face image to be migrated and the face mask of the model reference image to obtain an image as input data; A migration unit, configured to input the input data into a pre-built face migration network model for migration processing to obtain a migrated face image; Furthermore, the training process of the pre-built face transfer network model includes: Using the facial points, faces with the same contour in the image training set are classified to obtain N sets based on the image training set; Performing the same face shape adjustment on the image data in each of the N sets using a grid deformation algorithm to generate paired training data for training the face transfer network model; Calculating a loss function on the result image output by the face transfer network model and the target image, and optimizing the face transfer network model according to the loss function; Furthermore, the loss function includes L1 loss, perceptual VGG loss and facial contour local loss function, wherein the facial contour local loss function includes L1 loss obtained by cropping the target image of the face points and the face edge pixel area of ​​the result image output by the face transfer network model; using Calculate, where y represents the result graph of the network output. Represents the target image; using to perform calculations; to use Calculate the local loss function of the facial contour, where y i Represents the pixel area cropped around the i-th face point in the network output result graph, Represents the pixel area cropped around the i-th face point in the target image; Furthermore, the network framework of the face transfer network model includes a gated convolution layer, a convolution layer and an upsampling layer.

5. A device, characterized in that The invention comprises a processor, a memory and a portrait face migration program stored in the memory, wherein the portrait face migration program is executed by the processor to implement the steps of a CNN-based portrait face migration method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a portrait face migration program, and the portrait face migration program is executed by a processor to implement the steps of a CNN-based portrait face migration method as described in any one of claims 1 to 3.

Images