Face fusion method, device and electronic equipment

Abstract

The application discloses a face fusion method and device and electronic equipment, and relates to the field of cloud computing. The specific implementation scheme is as follows: acquiring face key points in a to-be-fused image and face key points in a bottom image; acquiring a first face region according to the face key points in the to-be-fused image and a second face region according to the face key points in the bottom image; performing morphing transformation of the first face region to the second face region; and fusing the first face region and the second face region to obtain a face fusion image. The problem that a fused mouth is seriously deformed due to a large pose difference in the process of mouth fusion is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to the field of face fusion. BACKGROUND

[0002] At present, in the face effect application such as expression driving and virtual portrait generation, or in the scene where mouth fusion is needed alone, the mouth region in the material is often directly fused into the bottom image through Poisson fusion. The face fusion method of this simple processing leads to excessive deformation of the fused mouth region due to the difference of face posture, and the texture difference between the fused mouth region and the bottom image is large, which is relatively unnatural. SUMMARY

[0003] Embodiments of the present application provide a face fusion method and device and electronic equipment to solve one or more technical problems in the prior art.

[0004] In a first aspect, embodiments of the present application provide a face fusion method, comprising:

[0005] obtaining face key points in a to-be-fused image and face key points in a bottom image;

[0006] obtaining a first face region according to the face key points in the to-be-fused image, and obtaining a second face region according to the face key points in the bottom image;

[0007] transforming the first face region to the second face region by deformation;

[0008] fusing the first face region and the second face region to obtain a face fusion image.

[0009] In the embodiment, by transforming the first face region in the to-be-fused image to the second face region in the bottom image, and then fusing the first face region and the second face region in the bottom image, the mouth can be prevented from excessive deformation in the process of mouth fusion, and a better fused image can be obtained.

[0010] In one embodiment, before transforming the first face region to the second face region by deformation, the method further comprises:

[0011] selecting a first group of key points according to the face key points in the to-be-fused image, the first group of key points comprising a first eye key point, a first nose key point, and a first chin key point;

[0012] selecting a second group of key points according to the face key points in the bottom image, the second group of key points comprising a second eye key point, a second nose key point, and a second chin key point;

[0013] According to the first set of key points and the second set of key points, the face in the image to be fused is aligned with the face in the base image.

[0014] In the embodiment, by selecting the key points of the eyes, the nose and the chin, the face in the image to be fused can be aligned with the shape of the whole face in the base image.

[0015] In an embodiment, the first face region is obtained according to the face key points in the image to be fused, and the second face region is obtained according to the face key points in the base image, including:

[0016] The first region including the mouth is obtained according to the face key points in the image to be fused, and the first region includes the facial region below the bottom of the nose of the face in the image to be fused;

[0017] The first region is expanded at a first ratio to obtain the first face region;

[0018] The second region including the mouth is obtained according to the face key points in the base image, and the second region includes the facial region below the bottom of the nose of the face in the base image;

[0019] The second region is expanded at a second ratio to obtain the second face region.

[0020] In the embodiment, the first region in the image to be fused and the second region in the base image are respectively expanded, without the need for full-face fusion, which can reduce the amount of calculation and speed up the processing speed.

[0021] In an embodiment, the first face region is morphed and transformed to the second face region, including:

[0022] The first face region is morphed and transformed to the second face region by using a long-side triangle partition morphing algorithm.

[0023] In the embodiment, the long-side triangle partition morphing algorithm is used to transform the mouth region, and the purpose is to align the shape of the first face region in the image to be fused with the second face region in the base image.

[0024] In an embodiment, the first face region and the second face region are fused to obtain a face fusion image, including:

[0025] The first mask matrix of the first region and the second mask matrix of the second region are subjected to AND operation to obtain a third mask matrix;

[0026] The third mask matrix is subjected to erosion processing and Gaussian blur processing to obtain a fourth mask matrix;

[0027] Based on the fourth mask matrix, the image obtained after morphing and transformation is subjected to color transformation algorithm and Poisson fusion algorithm processing to obtain a face fusion image.

[0028] In the embodiment, the third mask matrix is obtained by performing an AND operation on the first mask matrix and the second mask matrix, so that mouth deformation after fusion caused by a large attitude difference can be prevented. Through the corrosion processing and the Gaussian blur processing on the third mask matrix, the first face region and the second face region will not have boundary region cracks during fusion, and the display effect is more natural. The color difference can be reduced through the color transformation processing mode.

[0029] In a second aspect, an embodiment of the present application provides a face fusion device, comprising:

[0030] a face key point acquisition module, configured to acquire face key points in a to-be-fused image and face key points in a base image;

[0031] a face region acquisition module, configured to acquire a first face region according to the face key points in the to-be-fused image, and acquire a second face region according to the face key points in the base image;

[0032] a face region transformation module, configured to transform the first face region to the second face region;

[0033] a face fusion module, configured to fuse the first face region and the second face region to obtain a face fusion image.

[0034] In an embodiment, the device further comprises:

[0035] a first key point acquisition module, configured to select a first group of key points according to the face key points in the to-be-fused image, the first group of key points comprising a first eye key point, a first nose key point and a first chin key point;

[0036] a second key point acquisition module, configured to select a second group of key points according to the face key points in the base image, the second group of key points comprising a second eye key point, a second nose key point and a second chin key point;

[0037] a face alignment module, configured to align the face in the to-be-fused image and the face in the base image according to the first group of key points and the second group of key points.

[0038] In an embodiment, the face region transformation module comprises:

[0039] a first region acquisition unit, configured to acquire a first region comprising a mouth according to the face key points in the to-be-fused image, the first region comprising a face region below a nose bottom of the face in the to-be-fused image;

[0040] a first region expansion unit, configured to expand the first region by a first proportion to obtain the first face region;

[0041] The second region acquisition unit is configured to acquire a second region including a mouth according to the face key points in the base image, and the second region includes a face region below a nose bottom of the face in the base image.

[0042] The second region expansion unit is configured to expand the second region by a second ratio to obtain a second face region.

[0043] In an embodiment, the face region transformation module comprises:

[0044] The transformation unit is configured to transform the first face region to the second face region by using a long-side triangle partition morphing algorithm.

[0045] In an embodiment, the face fusion module comprises:

[0046] The first processing unit is configured to perform an AND operation on the first mask matrix of the first region and the second mask matrix of the second region to obtain a third mask matrix.

[0047] The second processing unit is configured to perform erosion processing and Gaussian blur processing on the third mask matrix to obtain a fourth mask matrix.

[0048] The third processing unit is configured to perform color transformation algorithm and Poisson fusion algorithm processing on the image obtained after the morphing transformation based on the fourth mask matrix to obtain a face fusion image.

[0049] In another aspect, the embodiment of the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the method as described above.

[0050] The embodiment of the above application has the following advantages or beneficial effects: because the first face region in the image to be fused is morphed and transformed to the second face region in the base image, and then the first face region and the second face region in the base image are fused, the technical problem of excessive morphing of the mouth in the process of mouth fusion is effectively solved, and the fusion effect of the mouth region is improved.

[0051] The other effects of the above optional mode will be described in the following with respect to specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings serve to better understand the present application and do not constitute a limitation thereof. In the drawings:

[0053] Figure 1 is a flowchart of a face fusion method according to the present application;

[0054] Figure 2 is a flowchart of a specific embodiment of a face fusion method according to the present application;

[0055] Figure 3 is another face fusion method flowchart according to the present application;

[0056] Figure 4 is another face fusion method flowchart according to the present application;

[0057] Figure 5 is a face fusion device structure block diagram according to the present application;

[0058] Figure 6 is another face fusion device structure block diagram according to the present application;

[0059] Figure 7 is another face fusion device structure block diagram according to the present application;

[0060] Figure 8 is a block diagram of an electronic device for implementing a face fusion method according to an embodiment of the present application;

[0061] Figure 9 is an application scenario diagram for implementing the first embodiment of the present application. DETAILED DESCRIPTION

[0062] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are presented for the purpose of illustration and description. It is to be understood that the embodiments described herein are exemplary only, and that various changes and modifications can be made thereto without departing from the scope and spirit of the application. As such, the exemplary embodiments described herein are to be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment should typically be considered as available for combination with features or aspects of the other embodiments.

[0063] Embodiment One

[0064] In one specific embodiment, a face fusion method is provided, as shown in Figure 1 comprising:

[0065] Step S10: obtaining face key points in a to-be-fused image and face key points in a base image;

[0066] Step S20: obtaining a first face region according to the face key points in the to-be-fused image, and obtaining a second face region according to the face key points in the base image;

[0067] Step S30: morphing and transforming the first face region to the second face region;

[0068] Step S40: fusing the first face region and the second face region to obtain a face fusion image.

[0069] In one example, as shown in Figure 2As shown, firstly, a large number of images to be fused are acquired, each containing mouth shapes to be fused. These images can include video frame sequences, single images, etc., and can be denoted as `src_img`. Next, a large number of background images are acquired, and the mouth shapes from the images to be fused are merged into these background images. These background images can also include video frame sequences, single images, etc., and can be denoted as `dst_img`. The images to be fused and the background images can be input into a facial landmark detection model to extract multiple facial landmarks from both the images to be fused and the background images. The number of facial landmarks is typically greater than 50, and can be adaptively adjusted as needed. An anomaly detection criterion is used to determine if the extracted landmarks are correct. If incorrect, the current frame is returned, and a new image to be fused or a new background image is acquired.

[0070] Then, based on the facial key points in the image to be fused, key points including the mouth region are selected to form the first face region, which can be denoted as src_mouth_img. Similarly, based on the facial key points in the base image, key points including the mouth region are selected to form the second face region, which can be denoted as dst_mouth_img. It should be noted that the first and second face regions at least include the facial area below the nose, including the mouth body. This can be expanded to a certain extent as needed, all within the scope of this embodiment. Before performing mouth region deformation transformation, the faces in the image to be fused and the faces in the base image can be aligned using facial key points. Then, a long-side triangulation deformation algorithm can be used to transform the first face region in the image to be fused into the second face region in the base image, aligning the shapes of the first face region in the image to be fused with the second face region in the base image. Of course, other deformation algorithms can also be used for shape alignment, all within the scope of this embodiment.

[0071] Finally, the first and second face regions are merged to obtain a fused face image. The fusion process may include calibration of the merged mouth region to reduce pose differences, ensuring the merged mouth remains undistorted. Erosion and Gaussian blurring are applied to the merged mouth region to address the issue of unnatural boundary cracks during fusion. Furthermore, skin tone blending is performed on the merged mouth region to resolve significant color differences. Figure 9 As shown, the middle image is the image to be merged, containing a profile view of a person with their mouth closed. The left image is the background image, containing a frontal view of a person with their mouth open. After merging, the image on the right is formed, showing the frontal view of the person with their mouth closed. The mouth area of ​​the person is not excessively distorted and blends well with the background color.

[0072] In the embodiment, by morphing the first face region in the to-be-fused image to the second face region in the base image, and then fusing the first face region and the second face region in the base image, the technical problem of excessive morphing of the mouth in the process of fusing the mouth is effectively solved.

[0073] In one embodiment, as shown in FIG. 2, after step S20 and before step S30, the method further includes: Figure 3

[0074] Step S21: selecting a first group of key points according to the face key points in the to-be-fused image, the first group of key points including first eye key points, first nose key points, and first chin key points.

[0075] Step S22: selecting a second group of key points according to the face key points in the base image, the second group of key points including second eye key points, second nose key points, and second chin key points.

[0076] Step S23: aligning the face in the to-be-fused image with the face in the base image according to the first group of key points and the second group of key points.

[0077] In one example, before morphing the first face region to the second face region, the entire face in the to-be-fused image can be aligned with the entire face in the base image. The alignment can be performed by selecting key points of several key parts of the face in the to-be-fused image and key points of several key parts of the face in the base image. The eye key points, such as the key points of the two corners of the eyes, can be selected. The nose key points, such as the key point of the center of the nose, can be selected. The chin key points, such as the key point of the center of the chin, can be selected. The entire face in the to-be-fused image is aligned to the entire face in the base image by using the two eye corner key points, the center of the nose key point, and the center of the chin key point.

[0078] In one embodiment, as shown in FIG. 2, step S20 includes: Figure 4

[0079] Step S201: selecting a first region including the mouth according to the face key points in the to-be-fused image, the first region including a face region below the nose of the face in the to-be-fused image;

[0080] Step S202: expanding the first region by a first ratio to obtain a first face region;

[0081] Step S203: selecting a second region including the mouth according to the face key points in the base image, the second region including a face region below the nose of the face in the base image;

[0082] Step S204: expanding the second region by a second ratio to obtain a second face region.​​

[0083] In the embodiment, the face region below the nose bottom in the image to be fused can be selected as the first region, and the face region below the nose bottom in the base image can be selected as the second region. The first region and the second region can be expanded by 1.3 times, and can be expanded to above the nose. Of course, the expansion ratio can also be other values, and can be adjusted adaptively as needed. The expanded range can include the face region above the nose, which is better for morphing transformation. The first region in the image to be fused and the second region in the base image are expanded respectively, so that the whole face does not need to be fused, the calculation amount is reduced, and the processing speed is accelerated.

[0084] In an embodiment, as shown in Figure 4 , step S30 comprises:

[0085] Step S301: using a long-side triangular partition morphing algorithm to morph the first face region to the second face region.

[0086] In an example, the long-side triangular partition morphing algorithm is used to transform the mouth region. The purpose is to shape align the first face region in the image to be fused with the second face region in the base image. For example, using a two-dimensional morphing (2dimension morphing) algorithm to transform the key points of the first face region in the image to be fused to the positions of half of the key points of the second face region in the base image. In the case that the face in the image to be fused is a front face, and the face in the base image is a side face, the shape of the mouth region after morphing transformation is shape aligned, which solves the technical problem of excessive morphing of the mouth due to large attitude change.

[0087] In an embodiment, as shown in Figure 4 , step S40 comprises:

[0088] Step S401: performing AND operation on the first mask matrix of the first region and the second mask matrix of the second region to obtain a third mask matrix;

[0089] Step S402: performing erosion processing and Gaussian blur processing on the third mask matrix to obtain a fourth mask matrix;

[0090] Step S403: based on the fourth mask matrix, performing color transformation algorithm and Poisson fusion algorithm processing on the image obtained after morphing transformation to obtain a face fusion image.

[0091] In the embodiment, the key point outer contour of the first region generates a first mask matrix, the key point outer contour of the second region generates a second mask matrix, and a third mask matrix is obtained by performing an AND operation on the first mask matrix and the second mask matrix, that is, a mask of the fused mouth is obtained. The purpose is to prevent the problem of serious deformation of the fused mouth caused by large attitude difference.

[0092] Then, the size of the kernel convolution is adaptively calculated. The specific process includes: through the corrosion processing and Gaussian blur processing of the third mask matrix, that is, the mask of the fused mouth, the problems of boundary region cracks, boundary region unevenness and unnaturalness in the fusion process of the first face region and the second face region are solved. The kernel convolution of the corrosion and the kernel convolution of the Gaussian blur are too large, which causes the mask of the fused mouth to be too small after being corroded, resulting in obvious boundary texture, and the size of the kernel convolution is related to the proportion of the mouth region in the entire image. Therefore, the size of the kernel convolution in the Gaussian blur can be adaptively adjusted by multiplying the proportion of the fused mouth region by an odd coefficient, and the coefficient can be obtained by calculating the mean average.

[0093] Based on the mask after corrosion and Gaussian blur, that is, the fourth mask matrix, the color of the first region in the to-be-fused image before expansion is transformed to the color of the base image. When the color difference between the base image and the skin color of the fused mouth region is large, the linear color transformation processing method is used to transform the color of the first region in the to-be-fused image before expansion into the color of the second region before expansion, and the mouth region after color transformation is subjected to Poisson fusion, thereby solving the problem of large color difference and making the skin of the mouth region in the face fusion image have no color difference. Finally, a more natural result of the mouth region fusion in different scenes is obtained.

[0094] Embodiment two

[0095] In another specific embodiment, as shown in Figure 5 the embodiment of the present application provides a face fusion device 100, which comprises:

[0096] The face key point acquisition module 110 is configured to acquire face key points in a to-be-fused image and face key points in a base image.

[0097] The face region acquisition module 120 is configured to acquire a first face region according to the face key points in the to-be-fused image and acquire a second face region according to the face key points in the base image.

[0098] The face region transformation module 130 is configured to transform the first face region to the second face region.

[0099] The face fusion module 140 is configured to fuse the first face region and the second face region to obtain a face fusion image.

[0100] In an embodiment, as shown in FIG. 2, a face fusion device 200, based on the face fusion device 100, further comprises: Figure 6

[0101] The first key point acquisition module 121 is configured to select a first group of key points according to the face key points in the image to be fused, the first group of key points comprising a first eye key point, a first nose key point and a first chin key point.

[0102] The second key point acquisition module 122 is configured to select a second group of key points according to the face key points in the base image, the second group of key points comprising a second eye key point, a second nose key point and a second chin key point.

[0103] The face alignment module 123 is configured to align the face in the image to be fused with the face in the base image according to the first group of key points and the second group of key points.

[0104] In an embodiment, as shown in FIG. 3, a face fusion device 300, based on the face fusion device 100, the face region acquisition module 120 comprises: Figure 7

[0105] The first region acquisition unit 1201 is configured to extract a plurality of first face key points according to the face key points in the image to be fused, and acquire a first region according to the plurality of first face key points.

[0106] The first region expansion unit 1202 is configured to expand the first region by a first proportion to obtain a first face region.

[0107] The second region acquisition unit 1203 is configured to extract a plurality of second face key points according to the face key points in the base image, and acquire a second region according to the plurality of second face key points.

[0108] The second region expansion unit 1204 is configured to expand the second region by a second proportion to obtain a second face region.

[0109] In an embodiment, as shown in FIG. 4, a face fusion device 300, based on the face fusion device 100, the face region transformation module 130 comprises: Figure 7 The transformation unit 1301 is configured to transform the first face region to the second face region by using a long-side triangle partition morphing algorithm.

[0110] In an embodiment, as shown in FIG. 5, a face fusion device 300, based on the face fusion device 100, the face fusion module 140 comprises:

[0111] Figure 7 ​​​​

[0112] The first processing unit 1401 is configured to perform an AND operation on the first mask matrix of the first region and the second mask matrix of the second region to obtain a third mask matrix.

[0113] The second processing unit 1402 is configured to perform etching processing and Gaussian blur processing on the third mask matrix to obtain a fourth mask matrix.

[0114] The third processing unit 1403 is configured to perform color transform algorithm and Poisson fusion algorithm on the image obtained after the deformation transformation based on the fourth mask matrix to obtain a face fusion image.

[0115] According to the embodiments of the present application, the present application further provides an electronic device and a readable storage medium.

[0116] As shown in Figure 8 FIG. 1 is a block diagram of an electronic device according to a face fusion method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown in the FIG. 1, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0117] As shown in Figure 8 The electronic device includes one or more processors 801, a memory 802, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other by different buses, and can be installed on a common mainboard or in other ways as needed. The processor can process instructions executed within the electronic device, including graphical information stored in the memory or on the memory to display a graphical user interface (GUI) on an external input / output device, such as a display device coupled to the interface. In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple storage devices, if necessary. Similarly, multiple electronic devices can be connected, each providing part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 The processor 801 is taken as an example in the

[0118] The memory 802 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by the at least one processor, so that the at least one processor executes a face fusion method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute a face fusion method provided by the present application.

[0119] The memory 802 is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules (for example, face key point acquisition module 110, face region acquisition module 120, face region transformation module 130 and face fusion module 140 shown in the embodiment of the present application) of a face fusion method provided by the present application. Figure 5 The processor 801 executes various function applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 802, that is, implements a face fusion method in the above method embodiment.

[0120] The memory 802 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of a face fusion electronic device, etc. In addition, the memory 802 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 802 can optionally include a memory disposed remotely with respect to the processor 801, which can be connected to a face fusion electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0121] The electronic device of the face fusion method can further include an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803 and the output device 804 can be connected through a bus or other means, Figure 8 for example, by taking the bus connection as an example.

[0122] The input device 803 can receive input digital or character information, and generate key signal inputs corresponding to user manipulation of a touch panel, a key pad, a mouse, a track ball, a joystick, one or more buttons, a wheel, and the like. The output device 804 can include a display device, an audio output device, and a haptic feedback device. The display device can include a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display, among others. In some embodiments, the display device can be a touch screen.

[0123] The present application also provides a computer program product comprising computer program / instructions which, when executed by a processor, implement the method according to any of the above embodiments.

[0124] Various embodiments of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0125] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0126] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0127] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0128] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0129] According to the technical scheme of the embodiment of the present application, the first face region in the to-be-fused image is morphed and transformed to the second face region in the base image, and then the first face region and the second face region in the base image are fused, thereby effectively solving the technical problem of excessive deformation of the mouth in the process of mouth fusion. When morphing and transforming, the long-side triangle partition morphing algorithm is used to align the shapes of the first face region in the to-be-fused image and the second face region in the base image, thereby solving the technical problem of excessive deformation of the mouth caused by large pose changes. The fused mouth region is subjected to erosion processing and Gaussian blur processing, thereby solving the problems of cracks in the boundary region, non-smoothness of the boundary region, and unnaturalness. The mouth region after the erosion and Gaussian blur processing is subjected to color transformation and Poisson fusion, thereby solving the problem of large color difference and making the skin of the mouth region in the face-fused image free of color difference. Finally, a relatively natural result of mouth region fusion in different scenes is obtained.

[0130] It should be understood that the various forms of flow shown above can be reordered, added to, or deleted from without departing from the scope of the present application. For example, the steps described in the present application can be executed in parallel, in sequence, or in a different order, as long as the desired results of the technical scheme disclosed in the present application can be achieved, and the present application is not limited herein.

[0131] The specific embodiments described above do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A face fusion method, characterized in that, include: Obtain facial key points in the image to be fused and facial key points in the background image; A first face region is obtained based on the facial key points in the image to be fused, and a second face region is obtained based on the facial key points in the base image. The first face region is deformed and transformed into the second face region; The first face region and the second face region are merged to obtain a face fusion image; The step of fusing the first face region and the second face region to obtain a face fusion image includes: The first mask matrix of the first region and the second mask matrix of the second region are ANDed to obtain the third mask matrix. The first region includes the facial region below the bottom of the nose of the face in the image to be fused, and the second region includes the facial region below the bottom of the nose of the face in the base image. The third mask matrix is ​​subjected to erosion and Gaussian blurring to obtain the fourth mask matrix; Based on the fourth mask matrix, the image obtained after deformation transformation is processed by color transformation algorithm and Poisson fusion algorithm to obtain the face fusion image.

2. The method according to claim 1, characterized in that, Before transforming the first face region to the second face region, the method further includes: selecting a first set of key points based on the facial key points in the image to be fused, wherein the first set of key points includes a first eye key point, a first nose key point, and a first chin key point. Select a second set of key points based on the facial key points in the base image. The second set of key points includes a second eye key point, a second nose key point, and a second chin key point. Based on the first set of key points and the second set of key points, the faces in the image to be fused are aligned with the faces in the base image.

3. The method according to claim 1, characterized in that, Obtaining a first face region based on facial key points in the image to be fused, and obtaining a second face region based on facial key points in the base image, includes: obtaining a first region including the mouth based on facial key points in the image to be fused, wherein the first region includes the facial region below the base of the nose in the image to be fused. The first region is expanded by a first ratio to obtain the first face region; A second region, including the mouth, is obtained based on the facial key points in the base image. The second region includes the facial area below the base of the nose in the base image. The second region is expanded at a second ratio to obtain the second face region.

4. The method according to claim 1, characterized in that, Transforming the first face region into the second face region includes: transforming the first face region into the second face region using a long-side triangulation deformation algorithm.

5. A face fusion device, characterized in that, include: The facial landmark acquisition module is used to acquire facial landmarks in the image to be fused and facial landmarks in the base image; The face region acquisition module is used to acquire a first face region based on the face key points in the image to be fused, and to acquire a second face region based on the face key points in the base image. A face region transformation module is used to deform and transform the first face region to the second face region; The face fusion module is used to fuse the first face region and the second face region to obtain a face fusion image; The face fusion module includes: a first processing unit, used to perform a bitwise AND operation between the first mask matrix of the first region and the second mask matrix of the second region to obtain a third mask matrix, wherein the first region includes the facial region below the bottom of the nose of the face in the image to be fused, and the second region includes the facial region below the bottom of the nose of the face in the base image. The second processing unit is used to perform erosion and Gaussian blurring on the third mask matrix to obtain the fourth mask matrix. The third processing unit is used to process the image obtained after deformation transformation using color transformation algorithm and Poisson fusion algorithm based on the fourth mask matrix to obtain the face fusion image.

6. The apparatus according to claim 5, characterized in that, Also includes: The first key point acquisition module is used to select a first set of key points based on the facial key points in the image to be fused. The first set of key points includes a first eye key point, a first nose key point, and a first chin key point. The second key point acquisition module is used to select a second set of key points based on the facial key points in the base image. The second set of key points includes a second eye key point, a second nose key point, and a second chin key point. The face alignment module is used to align the face in the image to be fused with the face in the base image based on the first set of key points and the second set of key points.

7. The apparatus according to claim 5, characterized in that, The face region transformation module includes: a first region acquisition unit, used to acquire a first region including the mouth based on the facial key points in the image to be fused, wherein the first region includes the facial region below the bottom of the nose in the image to be fused; The first region expansion unit is used to expand the first region by a first ratio to obtain the first face region; The second region acquisition unit is used to acquire a second region including the mouth based on the facial key points in the base image. The second region includes the facial area below the bottom of the nose of the face in the base image. The second region expansion unit is used to expand the second region at a second ratio to obtain the second face region.

8. The apparatus according to claim 5, characterized in that, The face region transformation module includes a transformation unit, used to transform the first face region to the second face region using a long-side triangulation deformation algorithm.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

Images