Facial feature fusion method and apparatus, storage medium, and electronic device

Abstract

The present application discloses a facial feature fusion method and apparatus, a storage medium, and an electronic device. The method comprises: performing face alignment of a first target image with respect to an original image; on the basis of a facial feature fusion region marked in a mask, determining a first region to be fused in the original image and a second region to be fused in the first target image, and transferring to the second region to be fused a color of the first region to be fused, so as to obtain a color-transferred first target image as a second target image; according to pixel coordinates within a target region to be fused, a preset feathering parameter, and the shortest distance to a convex hull from each pixel within the target region to be fused, determining a first feathering weight of each pixel in the second region to be fused and a second feathering weight of each pixel in the first region to be fused; and according to a pixel value and the first feathering weight of each pixel in the second region to be fused, and a pixel value and the second feathering weight of each pixel in the first region to be fused, fusing the first region to be fused and the second region to be fused to obtain a fused image.

Description

Face feature fusion method and device, storage medium and electronic equipment TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a face feature fusion method and device, a storage medium and an electronic equipment. BACKGROUND

[0002] Face features refer to key features of a face image, including but not limited to facial contours, eyes, nose, mouth, etc. In the field of intelligent virtual digital people, it is often necessary to fuse a face feature dynamically changing video with a static face image to make the static face image become a dynamic face video. For example, a static character can be changed into a dynamic character who can speak through such a fusion method.

[0003] The above-mentioned "face feature fusion" refers to the fusion of an original face image and a face image with face feature changes. The fused image mainly inherits the face features or information of the face image with face feature changes within a specified range, and mainly inherits the features of the original face image, including key feature points, textures, colors, etc. outside the specified range.

[0004] However, simple image region replacement and other general fusion methods may cause distortion, unnatural artifacts, etc. Fusion methods based on deep learning have high data requirements and large resource calculations. Moreover, since face feature changes involve individual differences and diversity, the generalization ability may be poor when processing new faces, resulting in poor fusion effect. SUMMARY

[0005] The present application provides a face feature fusion method, device, storage medium and electronic equipment, which can seamlessly fuse an original image with a video frame image with face feature changes, ensure smooth transition in the face area, and achieve accurate fusion effect.

[0006] The specific technical solutions are as follows:

[0007] In a first aspect, the present application provides a face feature fusion method, which comprises:

[0008] aligning a face of a first target image to an original image, the first target image being a video frame image in a video with face feature changes;

[0009] determining a first fusion area in the original image and a second fusion area in the first target image based on the face feature fusion area marked in the mask, and transferring the color of the first fusion area to the second fusion area to obtain the first target image after color transfer as a second target image;

[0010] According to the pixel point coordinates in the target to-be-fused region, the preset feathering parameter, and the nearest distance from each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region, a first feathering weight of each pixel point in the second to-be-fused region is determined, and a second feathering weight of the same pixel point in the first to-be-fused region is determined according to each first feathering weight, wherein the target to-be-fused region includes any one of the face feature to-be-fused region, the first to-be-fused region, and the second to-be-fused region.

[0011] According to the pixel value of each pixel point in the second to-be-fused region and the first feathering weight, and the pixel value of each pixel point in the first to-be-fused region and the second feathering weight, the first to-be-fused region and the second to-be-fused region are fused to obtain a fused image, wherein the fused image includes a fused region and other regions in the original image except the first to-be-fused region.

[0012] According to the above scheme, the embodiment of the present application can first align the first target image to the face of the original image, then define the face feature to-be-fused region through the mask, apply the color in the first to-be-fused region of the original image to the second to-be-fused region of the first target image, and then fuse the first to-be-fused region and the second to-be-fused region according to the pixel value of each pixel point in the second to-be-fused region and the first feathering weight, and the pixel value of each pixel point in the first to-be-fused region and the second feathering weight. This method only performs color transfer in the marked region (i.e. the face feature to-be-fused region), can maintain color consistency, and can ensure smooth transition at the fusion edge, avoid harsh division lines, achieve precise fusion effect, make the overall effect more realistic. And since the present application directly performs fusion calculation without relying on deep learning, there is no need for a large amount of data and resource calculation.

[0013] In a possible implementation, the color of the first to-be-fused region is transferred to the second to-be-fused region to obtain the first target image after color transfer as a second target image, including:

[0014] The pixel mean value of the pixel points in the first to-be-fused region is taken as a first pixel mean value, and the pixel mean value of the pixel points in the second to-be-fused region is taken as a second pixel mean value;

[0015] According to the pixel value of each pixel point in the second to-be-fused region and the second pixel mean value, the pixel value of the second to-be-fused region is de-meaning processed to obtain the second to-be-fused region after de-meaning processing;

[0016] The pixel value of each pixel point in the second to-be-fused region after the de-meaning is superimposed with the first pixel mean value, and the superimposed pixel value is limited in a preset pixel value range, to obtain the first target image after color transfer, and the first target image after color transfer is taken as the second target image.

[0017] According to the above scheme, the second to-be-fused region is first de-meaned, and then the de-meaned second to-be-fused region is superimposed with the first pixel mean value of the first to-be-fused region, so that the accuracy of color transfer can be improved.

[0018] In a possible implementation, the first feathering weight of each pixel point in the second to-be-fused region is determined according to the pixel point coordinates in the target to-be-fused region, a preset feathering parameter, and the nearest distance from each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region, and includes:

[0019] The coordinate range of the target to-be-fused region in the horizontal and vertical coordinates is determined.

[0020] The preset feathering parameter is adjusted according to the coordinate range to obtain an adjusted preset feathering parameter.

[0021] The feathering weight of each pixel point in the second to-be-fused region is calculated according to the nearest distance from each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region and the adjusted preset feathering parameter, and the calculated feathering weight is limited in a preset weight range to obtain the first feathering weight of each pixel point in the second to-be-fused region.

[0022] According to the above scheme, compared with directly using the preset feathering parameter, the first feathering weight of each pixel point in the second to-be-fused region is determined by not only adjusting the preset feathering parameter through the coordinate range of the target to-be-fused region in the horizontal and vertical coordinates, but also combining the nearest distance from each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region, the adjusted preset feathering parameter, and the preset weight range, so that the accuracy of the feathering weight is improved, and the smoothness of the edge fusion is further improved.

[0023] In a possible implementation, the second feathering weight of the same pixel point in the first to-be-fused region is determined according to each first feathering weight, and includes:

[0024] For the same pixel point with the same coordinates in the first to-be-fused region and the second to-be-fused region, the difference between 1 and the first feathering weight is taken as the second feathering weight.

[0025] In a possible implementation, the fusing the first to-be-fused region and the second to-be-fused region according to the pixel value of each pixel point in the second to-be-fused region and the first feathering weight, the pixel value of each pixel point in the first to-be-fused region and the second feathering weight, to obtain a fused image, includes:

[0026] For each pixel point in the first to-be-fused region and the second to-be-fused region, the pixel value of a same pixel point at the same coordinate in the first to-be-fused region and the second to-be-fused region and the corresponding first feathering weight and the second feathering weight are weighted and calculated respectively, to obtain the fused image.

[0027] In a possible implementation, the face alignment of the first target image to the original image includes:

[0028] performing face key point detection on the original image based on a key point detection model to obtain first key points, and performing face key point detection on the first target image based on the key point detection model to obtain second key points;

[0029] aligning the first target image to the original image by performing a similarity transformation on the first key points and the second key points.

[0030] In a second aspect, an embodiment of the present application provides a face feature fusion device, and the device includes:

[0031] an alignment unit, configured to perform face alignment of a first target image to an original image, the first target image being a video frame image in a video in which a face feature changes;

[0032] a first determination unit, configured to determine a first to-be-fused region in the original image and a second to-be-fused region in the first target image based on a face feature to-be-fused region marked in a mask;

[0033] a transfer unit, configured to transfer a color of the first to-be-fused region to the second to-be-fused region, to obtain the first target image after color transfer as a second target image;

[0034] a second determination unit, configured to determine a first feathering weight of each pixel point in the second to-be-fused region according to a pixel point coordinate in a target to-be-fused region, a preset feathering parameter, and a nearest distance of each pixel point in the target to-be-fused region to a convex hull of the target to-be-fused region, and determine a second feathering weight of a same pixel point in the first to-be-fused region according to each first feathering weight, wherein the target to-be-fused region includes any one of the face feature to-be-fused region, the first to-be-fused region, and the second to-be-fused region.

[0035] a fusion unit, configured to fuse the first to-be-fused region and the second to-be-fused region according to the pixel value of each pixel point in the second to-be-fused region and the first feathering weight, the pixel value of each pixel point in the first to-be-fused region and the second feathering weight, to obtain a fused image, wherein the fused image includes a fused region and other regions in the original image except the first to-be-fused region.

[0036] In a possible implementation, the transferring unit includes:

[0037] a mean value calculation module, configured to take the mean value of the pixel points in the first to-be-fused region as a first pixel mean value, and take the mean value of the pixel points in the second to-be-fused region as a second pixel mean value;

[0038] a de-meaning module, configured to perform de-meaning processing on the pixel values of the second to-be-fused region according to the pixel values of the pixel points in the second to-be-fused region and the second pixel mean value, to obtain a de-meaned second to-be-fused region;

[0039] a superposition limiting module, configured to superimpose the pixel values of the pixel points in the de-meaned second to-be-fused region on the first pixel mean value respectively, and limit the superimposed pixel values in a preset pixel value range, to obtain a color-transferred first target image, and take the color-transferred first target image as the second target image.

[0040] In a possible implementation, the second determining unit includes:

[0041] a determining module, configured to determine a coordinate range of the target to-be-fused region in horizontal and vertical coordinates;

[0042] an adjusting module, configured to adjust the preset feathering parameter according to the coordinate range, to obtain an adjusted preset feathering parameter;

[0043] a first weight calculation module, configured to calculate the feathering weight of each pixel point in the second to-be-fused region according to the nearest distance of each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region and the adjusted preset feathering parameter, and limit the calculated feathering weight in a preset weight range, to obtain the first feathering weight of each pixel point in the second to-be-fused region.

[0044] In a possible implementation, the second determining unit includes:

[0045] The second weight calculation module is configured to, for a same pixel point with a same coordinate in the first to-be-fused region and the second to-be-fused region, take a difference between 1 and the first feathering weight as the second feathering weight.

[0046] In a possible implementation, the fusion unit is configured to, for each pixel point in the first to-be-fused region and the second to-be-fused region, respectively perform weighted calculation on a pixel value of a same pixel point with a same coordinate in the first to-be-fused region and the second to-be-fused region and the corresponding first feathering weight and the second feathering weight, to obtain the fused image.

[0047] In a possible implementation, the alignment unit includes:

[0048] The detection module is configured to perform face key point detection on the original image based on a key point detection model to obtain first key points, and perform face key point detection on the first target image based on the key point detection model to obtain second key points.

[0049] The alignment module is configured to align the first target image to the original image by performing a similarity transformation on the first key points and the second key points.

[0050] It can be learned from the above solution that the embodiments of the present application can first align a face of the first target image to the original image, then define a face feature to-be-fused region through a mask, apply a color in a first to-be-fused region of the original image to a second to-be-fused region of the first target image, and then fuse the first to-be-fused region and the second to-be-fused region according to a pixel value of each pixel point in the second to-be-fused region and the first feathering weight, and a pixel value of each pixel point in the first to-be-fused region and the second feathering weight. This method only performs color transfer in a marked region (i.e., the face feature to-be-fused region), can maintain color consistency, and can ensure smooth transition at a fusion edge, avoid a harsh division line, achieve accurate fusion effect, and make the overall effect more realistic. Moreover, since the present application directly performs fusion calculation without relying on deep learning, there is no large amount of data requirement and resource calculation.

[0051] In a third aspect, the embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method according to any possible implementation manner of the first aspect.

[0052] In a fourth aspect, the embodiments of the present application provide an electronic device, which includes:

[0053] one or more processors;

[0054] The processor is coupled with a storage device for storing one or more programs;

[0055] When the one or more programs are executed by the one or more processors, the electronic device implements the method as described in any possible implementation manner of the first aspect.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product, which contains instructions. When the instructions are run on a computer or a processor, the computer or the processor executes the method as described in any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some of the embodiments of the present application. For those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0058] Fig. 1 is a flow diagram of a face feature fusion method provided by an embodiment of the present application;

[0059] Fig. 2 is a flow diagram of another face feature fusion method provided by an embodiment of the present application;

[0060] Fig. 3 is a block diagram of a face feature fusion device provided by an embodiment of the present application;

[0061] Fig. 4 is a structural diagram of an electronic device or a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0063] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The terms "include" and "have" and any variations thereof in the embodiments of the present application and the drawings are intended to cover non-exclusive inclusion. For example, the processes, methods, systems, products or devices containing a series of steps or units are not limited to the listed steps or units, but optionally also include steps or units not listed, or optionally also include other steps or units inherent to these processes, methods, products or devices.

[0064] In order to improve the smoothness of the face feature fusion, and further improve the authenticity of the face feature fusion, an embodiment of the present application provides a face feature fusion method, which can be applied to an electronic device or a computer device. The implementation process of the fusion method is described below in combination with the detailed flowchart shown in FIG. 1 and the simple flowchart shown in FIG. 2:

[0065] S110: Aligning the first target image to the original image in terms of face.

[0066] The first target image is a video frame image in a video in which face features change, for example, a video frame image in a video in which mouth shape changes. The original image is a static image containing a face region.

[0067] The method of aligning the first target image to the original image in terms of face includes: detecting face key points of the original image based on a key point detection model to obtain first key points, and detecting face key points of the first target image based on the key point detection model to obtain second key points; and aligning the first target image to the original image by performing a similarity transformation on the first key points and the second key points.

[0068] The face key points detected based on the key point detection model include key points in regions such as eyes, nose, and mouth, for example, the center point of the eye, the tip of the nose, and the corner of the mouth. The first key points include each face key point in the original image, and the second key points include each face key point in the first target image.

[0069] The similarity transformation formula includes:

[0070] Wherein, x and y are the horizontal and vertical coordinates of the second key points, X and Y are the coordinates of the first key points, s is a scale factor, θ is an anti-clockwise rotation angle (in radians), and t x and t y are translation parameters in the x and y directions, respectively.

[0071] Therefore, the similarity transformation matrix of aligning the first target image to the original image includes:

[0072] S120: Determine a first to-be-fused region in the original image and a second to-be-fused region in the first target image based on the face feature to-be-fused region marked in the mask, respectively, and transfer the color of the first to-be-fused region to the second to-be-fused region to obtain a first target image after color transfer as a second target image.

[0073] After the face alignment of the first target image and the original image, a mask marking the face feature to-be-fused region can be created first, and then the first to-be-fused region in the original image and the second to-be-fused region in the first target image can be determined based on the face feature to-be-fused region marked in the mask, and finally the color of the first to-be-fused region is transferred to the second to-be-fused region to obtain the first target image after color transfer as the second target image.

[0074] The method for creating the mask includes: first creating a full zero matrix with the same size as the original image, which is used to represent the mask, and in the mask, the face feature to-be-fused region can be marked differently from other regions. For example, the face feature to-be-fused region can be marked as 255, and other regions are marked as 0. In addition, the face feature to-be-fused region can be the entire face region, or the region where the face feature changes, such as the mouth region and the eye region.

[0075] The specific implementation method for transferring the color of the first to-be-fused region to the second to-be-fused region includes: taking the pixel mean of the pixel points in the first to-be-fused region as the first pixel mean, and taking the pixel mean of the pixel points in the second to-be-fused region as the second pixel mean; performing de-meaning processing on the pixel values of the second to-be-fused region according to the pixel values of each pixel point in the second to-be-fused region and the second pixel mean, to obtain the second to-be-fused region after de-meaning; by superimposing the pixel values of each pixel point in the second to-be-fused region after de-meaning on the first pixel mean respectively, and limiting the superimposed pixel values within a preset pixel value range, the first target image after color transfer is obtained, and the first target image after color transfer is taken as the second target image.

[0076] The method for de-meaning processing includes: subtracting the second pixel mean from the pixel value of each pixel point in the second to-be-fused region to obtain the second to-be-fused region after de-meaning.

[0077] After obtaining the second to-be-fused region after de-meaning, the pixel values of each pixel point in the second to-be-fused region after de-meaning can be superimposed on the first pixel mean first, and then it is judged whether the pixel values of each pixel point in the second to-be-fused region after de-meaning after superimposing the first pixel mean are within the preset pixel value range, if not, the pixel value of the corresponding pixel point is adjusted to a target value according to the preset adjustment rule, such as the value closest to the preset pixel value range, and if it is within the preset pixel value range, the pixel value of the pixel point is not adjusted.

[0078] For example, the preset pixel value range can be [0, 255], when the pixel value after superimposing the first pixel mean value is less than 0, the corresponding pixel value can be adjusted to 0, and when the pixel value after superimposing the first pixel mean value is greater than 255, the corresponding pixel value can be adjusted to 255.

[0079] The embodiment of the present application can improve the accuracy of color transfer by first performing mean value removal processing on the pixel values of the second to-be-fused region, and then superimposing the first pixel mean value of the first to-be-fused region on the mean value removed second to-be-fused region.

[0080] S130: determining a first feathering weight of each pixel point in the second to-be-fused region according to the pixel point coordinates in the target to-be-fused region, the preset feathering parameter, and the nearest distance of each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region, and determining a second feathering weight of the same pixel point in the first to-be-fused region according to each first feathering weight.

[0081] The target to-be-fused region includes any one of the face feature to-be-fused region, the first to-be-fused region, and the second to-be-fused region.

[0082] The method for determining the first feathering weight includes: determining the coordinate range of the target to-be-fused region in the horizontal and vertical coordinates; adjusting the preset feathering parameter according to the coordinate range to obtain an adjusted preset feathering parameter; calculating the feathering weight of each pixel point in the second to-be-fused region according to the nearest distance of each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region and the adjusted preset feathering parameter, and limiting the calculated feathering weight in a preset weight range to obtain the first feathering weight of each pixel point in the second to-be-fused region.

[0083] For the same pixel point in the first to-be-fused region and the second to-be-fused region with the same coordinates, the difference between 1 and the first feathering weight is taken as the second feathering weight.

[0084] In an implementation, the maximum value and the minimum value of the target to-be-fused region in the horizontal coordinate are determined first, and the maximum value and the minimum value of the target to-be-fused region in the vertical coordinate are determined, and then the coordinate range is determined according to the maximum value and the minimum value of the horizontal coordinate and the maximum value and the minimum value of the vertical coordinate. For example, the coordinate range is [horizontal coordinate maximum value-horizontal coordinate minimum value, vertical coordinate maximum value-vertical coordinate minimum value].

[0085] For example, the coordinate information of the target to-be-fused region includes:

[0086] For example, the maximum values in the horizontal coordinate x and the vertical coordinate y are 4 and 7 respectively, and the minimum values in the x and y directions are 1 and 3 respectively. Then the coordinate range faceSize = [4-1, 7-3] = [3, 4].

[0087] The method for adjusting the preset feathering parameter according to the coordinate range comprises: multiplying the preset feathering parameter by the coordinate range to obtain the adjusted feathering parameter. The preset feathering parameter is generally an experience value between 0.05 and 0.2.

[0088] In an embodiment, the convex hull is the minimum circumscribed convex polygon of a region, which contains all the points in the fusion region and makes the boundary of the polygon minimum. The calculation of the convex hull ensures that the shape of the fusion region is convex, so that the edge transition is natural when the image is fused.

[0089] The embodiment of the application can use ConvexHull to calculate the convex hull, denoted as H:{maskPts}, wherein maskPts is a coordinate set of all pixel points in the target region to be fused. The convex hull H is the minimum convex polygon containing all points P1, P2, …, Pn in the target region to be fused. That is, for any P n belonging to the convex hull H, the following formula is satisfied: H=ConvexHull(P1, P2, …, Pn). i n ).

[0090] For each pixel point in the target region to be fused, the distance of the nearest point on the convex hull is calculated, which is usually the vertical distance (the distance of a point to a straight line), and can be expressed by the following formula:

[0091] wherein distance(P i , Q) represents the distance of point P i to the point Q on the convex hull, and dists[i] represents the nearest distance of the ith pixel point to the convex hull.

[0092] In an embodiment, the specific implementation method for calculating the feathering weight of each pixel point in the second region to be fused according to the nearest distance of each pixel point in the target region to be fused to the convex hull of the target region to be fused and the adjusted preset feathering parameter comprises: dividing the nearest distance of each pixel point in the target region to be fused to the convex hull of the target region to be fused by the adjusted preset feathering parameter to obtain the feathering weight of the pixel point, and when the feathering weight of the pixel point is not in the preset weight range, the feathering weight of the pixel point can be adjusted to the closest weight value in the preset weight range.

[0093] When the preset weight range is [0, 1], the above implementation process can be expressed by the following formula: weights[i]=dists[i] / featherAmount

[0094] ​Wherein, weights[i] represents the feathering weight of the i-th pixel point, and featherAmount represents the adjusted preset feathering parameter.

[0095] Compared with directly using the preset feathering parameter, the embodiment of the present application not only adjusts the preset feathering parameter through the coordinate range of the target to-be-fused region on the horizontal and vertical coordinates, but also determines the first feathering weight of each pixel point in the second to-be-fused region in combination with the nearest distance of each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region, the adjusted preset feathering parameter, and the preset weight range, thereby improving the accuracy of the feathering weight and the smoothness of the edge fusion.

[0096] S140: Fusing the first to-be-fused region and the second to-be-fused region according to the pixel value and the first feathering weight of each pixel point in the second to-be-fused region, and the pixel value and the second feathering weight of each pixel point in the first to-be-fused region, to obtain a fused image.

[0097] The fused image includes a fused region and other regions in the original image except the first to-be-fused region.

[0098] The specific fusion method includes: for each pixel point in the first to-be-fused region and the second to-be-fused region, respectively performing weighted calculation on the pixel value and the corresponding first feathering weight and second feathering weight of the same pixel point at the same coordinate in the first to-be-fused region and the second to-be-fused region, to obtain the fused image.

[0099] The formula for the weighted calculation of each pixel point includes: the pixel value of the i-th pixel point in the first to-be-fused region * the first feathering weight + the pixel value of the i-th pixel point in the second to-be-fused region * the second feathering weight = the pixel value of the i-th pixel point in the first to-be-fused region * the first feathering weight + the pixel value of the i-th pixel point in the second to-be-fused region * (1-the first feathering weight).

[0100] After obtaining the fused image, the first to-be-fused region in the original image can be replaced by the fused image, and other regions in the original image remain unchanged, thereby obtaining the final required image.

[0101] The face feature fusion method provided in the embodiments of the present application can first align a first target image to a face of an original image, then define a face feature to-be-fused region through a mask, apply colors in a first to-be-fused region of the original image to a second to-be-fused region of the first target image, and then fuse the first to-be-fused region and the second to-be-fused region according to a pixel value of each pixel point in the second to-be-fused region and a first feathering weight, and a pixel value of each pixel point in the first to-be-fused region and a second feathering weight. This method only performs color transfer in a marked region (i.e., the face feature to-be-fused region), can maintain color consistency, and can ensure smooth transition at a fusion edge, avoiding a harsh division line, achieving accurate fusion effect, and making the overall effect more realistic. Moreover, since the present application directly performs fusion calculation without relying on deep learning, there is no need for a large amount of data and resource calculation.

[0102] It should be noted that the face feature fusion method provided in the embodiments of the present application can be applied in various scenarios, such as a host of a vehicle-mounted navigation, and a scenario in which a virtual digital person is required to perform voice broadcast in an entertainment APP, a website, and the like of a mobile terminal such as a vehicle-mounted terminal, a mobile phone, and a computer.

[0103] Corresponding to the method embodiments, another embodiment of the present application provides a face feature fusion device, as shown in FIG. 3, which comprises:

[0104] An alignment unit 210 is configured to align a first target image to a face of an original image, the first target image being a video frame image in a video in which a face feature changes;

[0105] A first determination unit 220 is configured to determine a first to-be-fused region in the original image and a second to-be-fused region in the first target image based on a face feature to-be-fused region marked in a mask;

[0106] A transfer unit 230 is configured to transfer colors of the first to-be-fused region to the second to-be-fused region, and obtain the first target image after color transfer as a second target image;

[0107] A second determination unit 240 is configured to determine a first feathering weight of each pixel point in the second to-be-fused region according to a pixel point coordinate in a target to-be-fused region, a preset feathering parameter, and a nearest distance from each pixel point in the target to-be-fused region to a convex hull of the target to-be-fused region, and determine a second feathering weight of a same pixel point in the first to-be-fused region according to each first feathering weight, wherein the target to-be-fused region includes any one of the face feature to-be-fused region, the first to-be-fused region, and the second to-be-fused region;

[0108] The fusion unit 250 is configured to fuse the first to-be-fused region and the second to-be-fused region according to the pixel value of each pixel point in the second to-be-fused region and the first feathering weight, the pixel value of each pixel point in the first to-be-fused region and the second feathering weight, to obtain a fused image, wherein the fused image includes a fused region and other regions in the original image except the first to-be-fused region.

[0109] In a possible implementation, the transfer unit 230 includes:

[0110] The mean value calculation module is configured to take the pixel mean value of the pixel points in the first to-be-fused region as a first pixel mean value, and take the pixel mean value of the pixel points in the second to-be-fused region as a second pixel mean value.

[0111] The mean value removal module is configured to perform mean value removal processing on the pixel values of the second to-be-fused region according to the pixel values of the pixel points in the second to-be-fused region and the second pixel mean value, to obtain a second to-be-fused region after mean value removal.

[0112] The superposition restriction module is configured to superimpose the pixel values of the pixel points in the second to-be-fused region after mean value removal on the first pixel mean value respectively, and restrict the superimposed pixel values in a preset pixel value range, to obtain the first target image after color transfer, and take the first target image after color transfer as the second target image.

[0113] In a possible implementation, the second determination unit 240 includes:

[0114] The determination module is configured to determine a coordinate range of the target to-be-fused region in horizontal and vertical coordinates.

[0115] The adjustment module is configured to adjust the preset feathering parameter according to the coordinate range, to obtain an adjusted preset feathering parameter.

[0116] The first weight calculation module is configured to calculate the feathering weight of each pixel point in the second to-be-fused region according to the nearest distance of each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region and the adjusted preset feathering parameter, and restrict the calculated feathering weight in a preset weight range, to obtain the first feathering weight of each pixel point in the second to-be-fused region.

[0117] In a possible implementation, the second determination unit 240 includes:

[0118] The second weight calculation module is configured to, for a same pixel point with a same coordinate in the first to-be-fused region and the second to-be-fused region, take a difference between 1 and the first feathering weight as the second feathering weight.

[0119] In a possible implementation, the fusion unit 250 is configured to, for each pixel point in the first to-be-fused region and the second to-be-fused region, respectively perform weighted calculation on pixel values of a same pixel point with a same coordinate in the first to-be-fused region and the second to-be-fused region and corresponding first feathering weights and second feathering weights, to obtain the fused image.

[0120] In a possible implementation, the alignment unit 210 includes:

[0121] The detection module is configured to perform face key point detection on the original image based on a key point detection model to obtain first key points, and perform face key point detection on the first target image based on the key point detection model to obtain second key points.

[0122] The alignment module is configured to align the first target image to the original image by performing a similarity transformation on the first key points and the second key points.

[0123] The face feature fusion apparatus provided in the embodiments of the present application can first align a first target image to an original image, then define a face feature to-be-fused region by using a mask, apply colors in a first to-be-fused region of the original image to a second to-be-fused region of the first target image, and then fuse the first to-be-fused region and the second to-be-fused region according to pixel values of each pixel point in the second to-be-fused region and first feathering weights, and pixel values of each pixel point in the first to-be-fused region and second feathering weights. This method only performs color transfer in a marked region (i.e., the face feature to-be-fused region), can maintain color consistency, and can ensure smooth transition at a fusion edge, avoids harsh division lines, achieves precise fusion effect, and makes the overall effect more realistic. Moreover, the present application directly performs fusion calculation without relying on deep learning, so there is no need for a large amount of data and resource calculation.

[0124] Based on the above method embodiments, another embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method according to any of the above embodiments.

[0125] Based on the above method embodiments, another embodiment of the present application provides an electronic device or a computer device, as shown in FIG. 4, which includes:

[0126] one or more processors 310;

[0127] The processor 310 is coupled with a storage device 320 for storing one or more programs;

[0128] When the one or more programs are executed by the one or more processors 310, the electronic device or the computer device implements the method according to any one of the above embodiments.

[0129] Based on the above method embodiments, another embodiment of the present application provides a vehicle, which comprises the device according to any one of the above embodiments, or comprises the electronic device according to the above.

[0130] Based on the above embodiments, another embodiment of the present application provides a computer program product, which comprises instructions, when the instructions are run on a computer or a processor, the computer or the processor executes the method according to any one of the above embodiments.

[0131] The above device embodiments correspond to the method embodiments and have the same technical effects as the method embodiments. For specific descriptions, refer to the method embodiments. The device embodiments are based on the method embodiments, and specific descriptions can be referred to the method embodiments, which will not be repeated here. Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily required to implement the present application.

[0132] Those skilled in the art can understand that the modules in the device in the embodiments can be distributed in the device in the embodiments as described in the embodiments, or can be changed and located in one or more devices different from the embodiments. The modules in the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A face feature fusion method, characterized in that, The method comprises: aligning a first target image to an original image, the first target image being a video frame image in a video in which a human face feature changes; determining a first to-be-fused region in the original image and a second to-be-fused region in the first target image based on the marked human face feature to-be-fused region in the mask, and transferring the color of the first to-be-fused region to the second to-be-fused region to obtain the first target image after color transfer as a second target image; determining a first feathering weight of each pixel point in the second to-be-fused region according to the pixel point coordinates in the target to-be-fused region, a preset feathering parameter, and the nearest distance from each pixel point in the target to-be-fused region to the convex hull of the target to-be-fused region, and determining a second feathering weight of the same pixel point in the first to-be-fused region according to each first feathering weight, wherein the target to-be-fused region includes any one of the human face feature to-be-fused region, the first to-be-fused region, and the second to-be-fused region; fusing the first to-be-fused region and the second to-be-fused region according to the pixel value of each pixel point in the second to-be-fused region and the first feathering weight, and the pixel value of each pixel point in the first to-be-fused region and the second feathering weight, to obtain a fused image, wherein the fused image includes a fused region and other regions in the original image except the first to-be-fused region.

2. The method of claim 1, wherein, The method comprises: taking the pixel mean value of the pixel points in the first to-be-fused region as a first pixel mean value, and taking the pixel mean value of the pixel points in the second to-be-fused region as a second pixel mean value; performing mean value removal processing on the pixel values of the second to-be-fused region according to the pixel values of each pixel point in the second to-be-fused region and the second pixel mean value, to obtain the second to-be-fused region after mean value removal; stacking the pixel values of each pixel point in the second to-be-fused region after mean value removal with the first pixel mean value respectively, and limiting the stacked pixel values within a preset pixel value range, to obtain the first target image after color transfer, and taking the first target image after color transfer as the second target image.

3. The method of claim 1, wherein, The method comprises: determining the coordinate range of the target to-be-fused region in the horizontal and vertical coordinates; adjusting the preset feathering parameter according to the coordinate range to obtain an adjusted preset feathering parameter; According to the nearest distance of each pixel point in the target region to be fused to a convex hull of the target region to be fused and the adjusted preset feathering parameter, a feathering weight of each pixel point in the second region to be fused is calculated, and the calculated feathering weight is limited in a preset weight range to obtain the first feathering weight of each pixel point in the second region to be fused.

4. The method of claim 1, wherein, The second feathering weight of the same pixel point in the first region to be fused is determined according to each first feathering weight, including: The difference between 1 and the first feathering weight is taken as the second feathering weight for the same pixel point with the same coordinates in the first region to be fused and the second region to be fused.

5. The method of claim 1, wherein, The first region to be fused and the second region to be fused are fused according to the pixel value of each pixel point in the second region to be fused and the first feathering weight, and the pixel value of each pixel point in the first region to be fused and the second feathering weight, to obtain a fused image, including: For each pixel point in the first region to be fused and the second region to be fused, the pixel value of the same pixel point with the same coordinates in the first region to be fused and the second region to be fused and the corresponding first feathering weight and second feathering weight are weighted and calculated to obtain the fused image.

6. The method according to any one of claims 1-5, characterized in that, The face alignment of the first target image to the original image includes: Face key point detection is performed on the original image based on a key point detection model to obtain first key points, and face key point detection is performed on the first target image based on the key point detection model to obtain second key points; The first target image is aligned to the original image by performing a similarity transformation on the first key points and the second key points.

7. A face feature fusion device, characterized by, The device includes: An alignment unit is configured to perform face alignment of a first target image to an original image, the first target image being a video frame image in a video with face feature changes; A first determination unit is configured to determine a first region to be fused in the original image and a second region to be fused in the first target image based on face feature regions to be fused marked in a mask; A transfer unit is configured to transfer the color of the first region to be fused to the second region to be fused to obtain the first target image after color transfer as a second target image; A second determination unit is configured to determine a first feathering weight of each pixel point in the second region to be fused according to pixel point coordinates in a target region to be fused, a preset feathering parameter, and the nearest distance of each pixel point in the target region to be fused to a convex hull of the target region to be fused, and determine a second feathering weight of the same pixel point in the first region to be fused according to each first feathering weight, wherein the target region to be fused includes any one of the face feature region to be fused, the first region to be fused, and the second region to be fused. A fusion unit is configured to fuse the first to-be-fused region and the second to-be-fused region according to pixel values of each pixel point in the second to-be-fused region and the first feathering weight, pixel values of each pixel point in the first to-be-fused region and the second feathering weight, to obtain a fused image, wherein the fused image includes a fused region and other regions in the original image except the first to-be-fused region.

8. The apparatus of claim 7, wherein, The transfer unit includes: A mean value calculation module is configured to take a pixel mean value of pixel points in the first to-be-fused region as a first pixel mean value, and take a pixel mean value of pixel points in the second to-be-fused region as a second pixel mean value; A de-meaning module is configured to perform de-meaning processing on pixel values of the second to-be-fused region according to the pixel values of each pixel point in the second to-be-fused region and the second pixel mean value, to obtain the second to-be-fused region after de-meaning. A superposition limiting module is configured to superimpose the pixel values of each pixel point in the second to-be-fused region after de-meaning on the first pixel mean value respectively, and limit the superimposed pixel values in a preset pixel value range, to obtain the first target image after color transfer, and take the first target image after color transfer as the second target image.

9. The apparatus of claim 7, wherein, The second determination unit includes: A determination module is configured to determine a coordinate range of the target to-be-fused region in horizontal and vertical coordinates; An adjustment module is configured to adjust the preset feathering parameter according to the coordinate range, to obtain an adjusted preset feathering parameter; A first weight calculation module is configured to calculate a feathering weight of each pixel point in the second to-be-fused region according to a nearest distance of each pixel point in the target to-be-fused region to a convex hull of the target to-be-fused region and the adjusted preset feathering parameter, and limit the calculated feathering weight in a preset weight range, to obtain the first feathering weight of each pixel point in the second to-be-fused region.

10. The apparatus of claim 7, wherein, The second determination unit includes: A second weight calculation module is configured to take a difference between 1 and the first feathering weight as the second feathering weight for a same pixel point of the same coordinates in the first to-be-fused region and the second to-be-fused region.

11. The apparatus of claim 7, wherein, The fusion unit is configured to perform weighted calculation on pixel values of a same pixel point of the same coordinates in the first to-be-fused region and the second to-be-fused region and corresponding first feathering weights and second feathering weights of the same pixel point for each pixel point in the first to-be-fused region and the second to-be-fused region, to obtain the fused image.

12. The apparatus of any one of claims 7-11, wherein, The alignment unit includes: A detection module is configured to perform face key point detection on the original image based on a key point detection model to obtain first key points, and perform face key point detection on the first target image based on the key point detection model to obtain second key points; An alignment module is configured to align the first target image to the original image by performing similarity transformation on the first key points and the second key points.

13. A computer readable storage medium having stored thereon a computer program, characterized in that The program, when executed by the processor, implements the method according to any one of claims 1-6.

14. An electronic device, comprising: The electronic device includes: one or more processors; The processor is coupled with a storage device, and the storage device is configured to store one or more programs; The one or more programs, when executed by the one or more processors, cause the electronic device to implement the method according to any one of claims 1-6.

Images