A method, apparatus, and equipment for removing wrinkles from portrait videos.

Abstract

This invention discloses a method, apparatus, device, and storage medium for removing wrinkles from portrait videos. The method includes: selecting a video frame that meets preset conditions from the video to be processed as a first keyframe image; performing wrinkle segmentation on the first keyframe image to obtain a first wrinkle mask image; performing connected component detection on the first wrinkle mask image to obtain N connected components; assigning values ​​to corresponding pixel values ​​in the first wrinkle mask image based on the relationship between the centroid of each connected component and a preset range to obtain a second wrinkle mask image; performing 3D face reconstruction on the first keyframe image and other video frames to obtain a first MVP matrix and a second MVP matrix; performing mapping processing on the second wrinkle mask image based on the first MVP matrix and the second MVP matrix to obtain a third wrinkle mask image corresponding to the current video frame; and removing facial wrinkles from the current video frame based on the third wrinkle mask image to obtain a target video.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method, apparatus, and device for removing wrinkles from portrait videos. Background Technology

[0002] In the video editing industry, facial enhancement is a frequently used feature. Most beauty and editing software on the market offers functions to remove skin imperfections such as skin smoothing, dark circle removal, and blemish and acne removal. However, for wrinkles, they can only remove dull, non-wrinkled lines, such as nasolabial folds and marionette lines. There is still no effective solution for deeper wrinkles such as crow's feet and forehead wrinkles. While image wrinkle removal solutions are relatively mature, the time required for segmentation algorithms is difficult to achieve real-time standards, making direct application in video editing impossible. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method, apparatus and device for removing wrinkles from human faces in video, in order to solve the problem that there is no effective solution for removing wrinkles from human faces in video scenes.

[0004] To achieve the above objectives, the present invention provides a method for removing wrinkles from portrait videos, the method comprising:

[0005] The video to be processed is obtained, and video frames of the same face in the video to be processed that meet the preset conditions are selected as the first key frame images. Each first key frame image is matched and labeled based on the preset expression category label, and the first key frame image is segmented into wrinkles to obtain the corresponding first wrinkle mask image.

[0006] Connectivity detection is performed on the first wrinkle mask image to obtain N connected components. The corresponding pixel values ​​in the first wrinkle mask image are assigned according to the relationship between the centroid of each connected component and the preset range to obtain the second wrinkle mask image. The expression category label corresponding to the first keyframe image and the second wrinkle mask image are stored in the first set.

[0007] Perform 3D face reconstruction on the first keyframe image to obtain the corresponding 3D face model and the first MVP matrix;

[0008] Face 3D reconstruction is performed on other video frames in the video to be processed to obtain a second MVP matrix. Based on the first MVP matrix and the second MVP matrix, the second wrinkle mask image is mapped to obtain a third wrinkle mask image corresponding to the current video frame.

[0009] The target video is obtained by removing facial wrinkles from the current video frame based on the third wrinkle mask image.

[0010] Preferably, selecting video frames of the same face in the video to be processed that meet preset conditions as the first keyframe image includes:

[0011] Select video frames of the same face in the video to be processed, where the face is not obscured, as candidate keyframe images;

[0012] The candidate keyframe images are hierarchically clustered according to a preset aggregation strategy, divided into N categories based on the same expression, and the video frame with the smallest distance from the cluster center in each category is extracted as the first keyframe image.

[0013] Preferably, the determination that the face is not obscured is made by the ratio of the area of ​​the facial skin to the area of ​​the polygon formed by the points of the facial outline being greater than a threshold.

[0014] Preferably, when the expression category label corresponding to the first keyframe image is the first category; the step of assigning values ​​to the corresponding pixel values ​​in the first wrinkle mask image according to the relationship between the centroid of each connected component and a preset range to obtain the second wrinkle mask image includes:

[0015] When the center of gravity is inside the first circle and the maximum distance of the pixels in the connected domain from the center of gravity is greater than 0.75 * the first radius, the wrinkle type is determined to be a smile line, and the corresponding pixel value in the first wrinkle mask image is set to 128. Here, the first radius is the distance from the midpoint between the left wing point and the left corner of the mouth to the left wing point, and the first circle is a circle with the midpoint between the left wing point and the left corner of the mouth as the center and the first radius as the radius.

[0016] When the center of gravity is inside the second circle and the maximum distance of the pixels in the connected domain from the center of gravity is greater than 0.75 * the second radius, the wrinkle type is determined to be a smile line, and the corresponding pixel value in the first wrinkle mask image is set to 128. Here, the second radius is the distance from the midpoint between the right nostril point and the right corner of the mouth point to the right nostril point, and the second circle is a circle formed with the midpoint between the right nostril point and the right corner of the mouth point as the center and the second radius as the radius.

[0017] When the distance between the center of gravity and the left or right corner of the eye is less than the first threshold, the wrinkle type is determined to be an eye wrinkle, and the corresponding pixel value in the first wrinkle mask image is set to 128.

[0018] Preferably, when the expression category label corresponding to the first keyframe image is the second category; the step of assigning values ​​to the corresponding pixel values ​​in the first wrinkle mask image according to the relationship between the centroid of each connected component and a preset range to obtain the second wrinkle mask image includes:

[0019] Gaussian filtering is applied to the first keyframe image to obtain the filtered result image;

[0020] Calculate the difference between the pixel value of the first keyframe image and the pixel value of the filtered result image. When the difference is less than the second threshold, the wrinkle type is determined to be a shallow wrinkle, and the corresponding pixel value in the first wrinkle mask image is set to 64.

[0021] Preferably, the step of mapping the second wrinkle mask image based on the first MVP matrix and the second MVP matrix to obtain the third wrinkle mask image corresponding to the current video frame includes:

[0022] When it is determined that the expression category label of the current video frame exists in the first set, the second wrinkle mask image is back-projected onto the 3D face model according to the first MVP matrix to obtain the intermediate wrinkle mask image. The intermediate wrinkle mask image is then projected onto the current video frame through the second MVP matrix to obtain the third wrinkle mask image corresponding to the current video frame.

[0023] Preferably, the step of removing facial wrinkles from the current video frame based on the third wrinkle mask image to obtain the target video includes:

[0024] For the region with a pixel value of 255 in the third wrinkle mask image, texture replacement is performed by finding skin texture in adjacent non-wrinkle regions.

[0025] After applying Gaussian filtering to the region with a pixel value of 128 in the third wrinkle mask image, the Gaussian filtering result is fused with the corresponding first keyframe image at a degree of 50%.

[0026] A curve brightening operation is performed on the region with a pixel value of 64 in the third wrinkle mask image.

[0027] To achieve the above objectives, the present invention also provides a facial video wrinkle removal device, the device comprising:

[0028] The keyframe processing unit is used to acquire the video to be processed, select the video frame of the same face in the video to be processed that meets the preset conditions as the first keyframe image, match and label each first keyframe image based on the preset expression category label, and perform wrinkle segmentation on the first keyframe image to obtain the corresponding first wrinkle mask image.

[0029] The mask image analysis unit is used to perform connected component detection on the first wrinkle mask image to obtain N connected components, and assign values ​​to the corresponding pixel values ​​in the first wrinkle mask image according to the relationship between the centroid of each connected component and the preset range to obtain the second wrinkle mask image. The expression category label corresponding to the first keyframe image and the second wrinkle mask image are stored in the first set.

[0030] The first reconstruction unit is used to perform 3D face reconstruction on the first keyframe image to obtain the corresponding 3D face model and the first MVP matrix.

[0031] The second reconstruction unit is used to perform face 3D reconstruction on other video frames in the video to be processed to obtain a second MVP matrix, and to perform mapping processing on the second wrinkle mask image based on the first MVP matrix and the second MVP matrix to obtain a third wrinkle mask image corresponding to the current video frame.

[0032] The wrinkle removal unit is used to remove facial wrinkles from the current video frame based on the third wrinkle mask image to obtain the target video.

[0033] To achieve the above objectives, the present invention also proposes a portrait video wrinkle removal device, including a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a portrait video wrinkle removal method as described in the above embodiments.

[0034] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of a portrait video wrinkle removal method as described in the above embodiments.

[0035] Beneficial effects:

[0036] The above solution, by extracting keyframes for wrinkle segmentation, eliminates the need for wrinkle segmentation algorithms to be performed on every frame when removing wrinkles from videos. Wrinkle segmentation and analysis are only performed again when facial expressions change significantly. This effectively applies detection algorithms that were previously unable to operate in real time, thus meeting the performance and effectiveness requirements for wrinkle removal in video scenes.

[0037] The above solution, by performing 3D facial reconstruction on video frames and projecting based on the MVP matrix, can accurately map wrinkle segmentation masks when removing facial wrinkles from videos. It can locate the position of wrinkles on the same face of different sizes and poses in each video frame, ensuring that non-wrinkle areas are not misprocessed, thus improving the wrinkle removal effect.

[0038] The above method processes the same face in the video, enabling wrinkle removal for different faces in the video according to their individual facial features, thus improving the accuracy of facial wrinkle removal. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating a method for removing wrinkles from portrait videos according to an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram illustrating the process of processing a keyframe image according to an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the face wrinkle removal process in other video frames provided in an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of a facial video wrinkle removal device according to an embodiment of the present invention.

[0044] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0047] The present invention will be described in detail below with reference to the embodiments.

[0048] Reference Figure 1The diagram shown is a flowchart illustrating a method for removing wrinkles from portrait videos according to an embodiment of the present invention.

[0049] In this embodiment, the method includes:

[0050] S11, acquire the video to be processed, select the video frame of the same face in the video to be processed that meets the preset conditions as the first key frame image, match and label each first key frame image based on the preset expression category label, and perform wrinkle segmentation on the first key frame image to obtain the corresponding first wrinkle mask image.

[0051] The step of selecting video frames of the same face in the video to be processed that meet preset conditions as the first keyframe image includes:

[0052] S11-1, Select video frames of the same face in the video to be processed where the face is not obscured as candidate keyframe images;

[0053] S11-2, The candidate keyframe images are hierarchically clustered according to a preset aggregation strategy, divided into N categories according to the same expression, and the video frame with the smallest distance from the cluster center in each category is extracted as the first keyframe image.

[0054] In this embodiment, video frames featuring the same face where the face is not obscured are selected as candidate keyframes. The criterion for determining whether a face is not obscured is: the area of ​​the face skin / the area of ​​the polygon formed by the outer contour points of the face > a threshold. In this embodiment, the threshold is set to 0.95. Specifically, if multiple faces exist in the video, each identical face is processed separately, and the data of each face is subsequently stored in different sets Q1, Q2, etc. The face data of the video frames that meet the above requirements are hierarchically clustered using a bottom-up aggregation strategy, divided into several classes, and classes with the same expression are merged. Expressions are analyzed based on the face points at the class centers to obtain expression category labels: such as 0 - neutral expression, 1 - toothy smile, 2 - open mouth, 3 - surprised, 4 - other expressions. Then, the video frame with the smallest distance to the cluster center in each class is extracted as the keyframe. In this embodiment, there are a maximum of five keyframes (one for each expression). If an expression category label does not exist in the entire video, then its corresponding keyframe also does not exist. A deep learning model (which can be an existing model suitable for wrinkle segmentation, such as the U-Net model or other mature models) is used to perform wrinkle segmentation on the keyframe images, resulting in a single-channel wrinkle mask image M. The pixel values ​​of the wrinkled areas in this wrinkle mask image are labeled as 255, and the pixel values ​​of the non-wrinkled areas are labeled as 0.

[0055] S12, perform connected component detection on the first wrinkle mask image to obtain N connected components, assign values ​​to the corresponding pixel values ​​in the first wrinkle mask image according to the relationship between the centroid of each connected component and the preset range, obtain the second wrinkle mask image, and store the expression category label corresponding to the first keyframe image and the second wrinkle mask image in the first set.

[0056] Furthermore, when the expression category label corresponding to the first keyframe image is the first category; the step of assigning values ​​to the corresponding pixel values ​​in the first wrinkle mask image based on the relationship between the centroid of each connected component and a preset range to obtain the second wrinkle mask image includes:

[0057] S12-1, when the center of gravity is inside the first circle and the maximum distance of the pixels in the connected domain from the center of gravity is > 0.75 * the first radius, the wrinkle type is determined to be a smile line, and the corresponding pixel value in the first wrinkle mask image is set to 128. Here, the first radius is the distance from the midpoint between the left wing point and the left corner of the mouth to the left wing point, and the first circle is a circle with the midpoint between the left wing point and the left corner of the mouth as the center and the first radius as the radius.

[0058] S12-2, when the center of gravity is inside the second circle and the maximum distance of the pixels in the connected domain from the center of gravity is > 0.75 * the second radius, the wrinkle type is determined to be a smile line, and the corresponding pixel value in the first wrinkle mask image is set to 128. Here, the second radius is the distance from the midpoint between the right nostril point and the right corner of the mouth point to the right nostril point, and the second circle is a circle formed with the midpoint between the right nostril point and the right corner of the mouth point as the center and the second radius as the radius.

[0059] S12-3, when the distance between the center of gravity and the left or right corner of the eye is less than the first threshold, the wrinkle type is determined to be eye wrinkles, and the corresponding pixel value in the first wrinkle mask image is set to 128.

[0060] Furthermore, when the expression category label corresponding to the first keyframe image is the second category; the step of assigning values ​​to the corresponding pixel values ​​in the first wrinkle mask image based on the relationship between the centroid of each connected component and a preset range to obtain the second wrinkle mask image includes:

[0061] S12-4, Perform Gaussian filtering on the first keyframe image to obtain a filtered result image; calculate the difference between the pixel value of the first keyframe image and the pixel value of the filtered result image. When the difference is less than a second threshold, determine that the wrinkle type is a shallow wrinkle, and set the corresponding pixel value in the first wrinkle mask image to 64.

[0062] Reference Figure 2The diagram illustrates the image processing steps (in the image, the left side is the keyframe image, the middle side is the first wrinkle mask image, and the right side is the second wrinkle mask image). In this embodiment, since smile lines and crow's feet are dynamic wrinkles caused by facial expressions, completely removing them would look very unnatural. Therefore, it is necessary to further locate whether smile lines or crow's feet exist in the wrinkle mask image. Specifically:

[0063] Connectivity detection is performed on the wrinkle mask image to obtain N connected components. The centroid of each connected component is calculated by traversing these components. Here, the centroid refers to the geometric center. In n-dimensional space, the geometric center or centroid of an object X is the intersection point of all hyperplanes that divide X into two equal parts (it can also be described as the average of all points in X). If an object has a uniform mass distribution, its centroid is its center of mass.

[0064] The centroid of a connected component is calculated in the current scenario as follows:

[0065]

[0066] Where n is the number of pixels in a single connected component, and P(x,y) is the coordinate (x,y) of the i-th pixel in the connected component.

[0067] (1) If the corresponding expression category label of the keyframe image is 2 or 3, then the following analysis and processing are performed:

[0068] a. If the centroid falls within a circle with Mid_left as the center and Rleft as the radius, and the maximum distance of a pixel in the connected region from the centroid is >0.75*Rleft; then it is determined to be a smile line, and the corresponding pixel value in the texture mask image M is set to 128;

[0069] b. If the centroid falls within a circle with Mid_right as the center and Rright as the radius, and the maximum distance of a pixel in the connected region from the centroid is >0.75*Rright; then it is determined to be a smile line, and the corresponding pixel value in the texture mask image M is set to 128;

[0070] c. If the distance between the centroid and the left or right corner of the eye is less than the threshold A, then it is determined to be an eye pattern, and the corresponding pixel value in the pattern mask image M is set to 128.

[0071] Where Mid_left and Mid_right are the midpoints of the left and right nostril points and the corner of the mouth, respectively, and the radius Rleft is the distance from Mid_left to the left nostril point, and Rright is the distance from Mid_right to the right nostril point.

[0072] (2) For wrinkle connected components that do not belong to a, b, or c above, and when the corresponding expression category label of the keyframe image is 0, 1, or 4, the following analysis and processing are performed:

[0073] Gaussian filtering is applied to the keyframe image to obtain the filtered result image. The difference between the pixel values ​​of the original image and the filtered result image is calculated. If the difference between the original image and the filtered result image is greater than the threshold B, the wrinkle is identified as a deeper wrinkle and no further processing is performed; otherwise, it is identified as a shallower wrinkle, and the corresponding pixel value in the texture mask image M is set to 64.

[0074] S13, perform 3D face reconstruction on the first keyframe image to obtain the corresponding 3D face model and the first MVP matrix.

[0075] In this embodiment, the obtained 3D face reconstruction model is derived through 3D deformation modeling (3DMM). This method treats the face space as a linear space, establishes a face space basis based on pre-collected 3D face data, and approximates the face on the 2D image using the projection of a linear combination of the pre-established 3D face data. The face space basis includes a 3D average face, a face shape basis constituting the 3D face model, and an expression basis constituting the 3D expression model. The basic formula of 3DMM is expressed as follows:

[0076] Formula (1)

[0077] In the formula, M represents the final fitted 3D face model, M mean Represents the average three-dimensional face. The base shape of the face that constitutes the 3D face model is represented by s. i As the base coefficient for face shape, e represents the expression base that constitutes the three-dimensional expression model. j is the expression basis coefficient, and n and m represent the number of face shape basis and expression basis, respectively.

[0078] The MVP matrix refers to three transformation matrices: model, view, and projection, used to project a 3D face model onto a 2D image. Its initial parameters are estimated based on the feature points of the face space basis, including camera position, image plane rotation angle, components of direct and ambient light, and image contrast. Based on the extracted face feature points, the face space basis, and the initial parameters of the projection matrix, a 3D face model of the corresponding image is fitted. That is, based on 3D model data with the same number of vertices and topological structure, the parameters of the linear combination of the 3D model are obtained by minimizing the distance between the projected feature points on the 3D model and the 2D feature points. These parameters are then used to fit the 3D face model and projection matrix of the corresponding face image. The formula is as follows:

[0079] Formula (2)Error=MVP*MP 2d

[0080] In the formula, MVP represents the projection matrix of the 3D model, P 2d Let M represent the facial feature points detected by facial feature point detection, and M represent the 3D face model. The 3D face model and MVP matrix are obtained by minimizing Error using formula (1).

[0081] S14, perform face 3D reconstruction on other video frames in the video to be processed to obtain a second MVP matrix, and perform mapping processing on the second wrinkle mask image based on the first MVP matrix and the second MVP matrix to obtain a third wrinkle mask image corresponding to the current video frame.

[0082] The step of mapping the second wrinkle mask image based on the first MVP matrix and the second MVP matrix to obtain the third wrinkle mask image corresponding to the current video frame includes:

[0083] When it is determined that the expression category label of the current video frame exists in the first set, the second wrinkle mask image is back-projected onto the 3D face model according to the first MVP matrix to obtain the intermediate wrinkle mask image. The intermediate wrinkle mask image is then projected onto the current video frame through the second MVP matrix to obtain the third wrinkle mask image corresponding to the current video frame.

[0084] In this embodiment, the MVP matrix P' is obtained by performing 3D face reconstruction on other video frames in the video. First, it is determined whether the expression category label of the current video frame exists in the set Q:

[0085] (1) If it exists, back-project the corresponding wrinkle mask image onto the 3D face model according to the MVP matrix P of the key frame image to obtain the intermediate wrinkle mask image under the uv unfolded coordinates of the 3D face model; project the intermediate wrinkle mask image onto the current video frame image through the MVP matrix P' to obtain the wrinkle mask M' (third wrinkle mask image) of the current video frame.

[0086] (2) If it does not exist, the current video frame is processed according to the above steps S11-S13. The expression category label of the current video frame and the corresponding wrinkle mask image are also added to the set Q and then mapped.

[0087] S15, perform facial wrinkle removal on the current video frame based on the third wrinkle mask image to obtain the target video.

[0088] The step of removing facial wrinkles from the current video frame based on the third wrinkle mask image to obtain the target video includes:

[0089] S15-1, The region with a pixel value of 255 in the third wrinkle mask image is replaced by finding skin texture in the adjacent non-wrinkle region;

[0090] S15-2, After performing Gaussian filtering on the region with a pixel value of 128 in the third wrinkle mask image, the Gaussian filtering result is fused with the corresponding first keyframe image at a degree of 50%.

[0091] S15-3, Perform curve brightening operation on the area with a pixel value of 64 in the third wrinkle mask image.

[0092] In this embodiment, wrinkle removal is performed on the current video frame image based on the wrinkle mask M' of the current video frame, including:

[0093] (1) The area with a pixel value of 255 in the wrinkle mask M' is a deeper texture. The texture is directly replaced by finding the skin texture in the adjacent non-wrinkle area.

[0094] (2) The area with a pixel value of 128 in the wrinkle mask M' is a relatively deep dynamic texture that only needs to be dodged. Gaussian filtering is applied to it, and the Gaussian filtering result is fused with the original image at a degree of 50%. This can improve the expression lines when they are beautified to a certain extent. Since expression lines are dynamic textures that can exist at any age, they are not completely removed, so as to achieve a better expression line processing effect and improve the overall naturalness. That is, the fused result image = 0.5 * original image + 0.5 * Gaussian filtering result image.

[0095] (3) The area with a pixel value of 64 in the wrinkle mask M' is a relatively light, dark texture. A curve brightening operation is performed on this area. Furthermore, to avoid whitening, the brightened pixel value is limited to not exceeding the pixel value after Gaussian filtering. (Refer to...) Figure 3 The diagram shows the face wrinkle removal process for other video frames in the video (in the diagram, the left side is the original image of other video frames, the middle is the third wrinkle mask image, and the right side is the final wrinkle removal result of the corresponding original image).

[0096] Reference Figure 4 The diagram shown is a structural schematic of a human portrait video wrinkle removal device according to an embodiment of the present invention.

[0097] In this embodiment, the device 40 includes:

[0098] The keyframe processing unit 41 is used to acquire the video to be processed, select the video frame of the same face in the video to be processed that meets the preset conditions as the first keyframe image, match and label each first keyframe image based on the preset expression category label, and perform wrinkle segmentation on the first keyframe image to obtain the corresponding first wrinkle mask image.

[0099] The mask image analysis unit 42 is used to perform connected component detection on the first wrinkle mask image to obtain N connected components, and assign values ​​to the corresponding pixel values ​​in the first wrinkle mask image according to the relationship between the centroid of each connected component and the preset range to obtain the second wrinkle mask image, and store the expression category label corresponding to the first keyframe image and the second wrinkle mask image in the first set.

[0100] The first reconstruction unit 43 is used to perform 3D face reconstruction on the first keyframe image to obtain the corresponding 3D face model and the first MVP matrix.

[0101] The second reconstruction unit 44 is used to perform face 3D reconstruction on other video frames in the video to be processed to obtain a second MVP matrix, and perform mapping processing on the second wrinkle mask image based on the first MVP matrix and the second MVP matrix to obtain a third wrinkle mask image corresponding to the current video frame.

[0102] The wrinkle removal unit 45 is used to remove facial wrinkles from the current video frame based on the third wrinkle mask image to obtain the target video.

[0103] Each unit module of the device 40 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0104] This invention also provides a facial video wrinkle removal device, which includes the facial video wrinkle removal apparatus as described above, wherein the facial video wrinkle removal apparatus can employ... Figure 4 The structure of the embodiment, correspondingly, can be executed Figure 1 The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0105] The aforementioned facial video wrinkle removal device includes: devices with photographic functions such as mobile phones, digital cameras, or tablet computers; devices with image processing functions; or devices with image display functions. The device may include components such as a memory, processor, input unit, display unit, and power supply.

[0106] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as image playback function), etc.; the data storage area can store data created according to the use of the device. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0107] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0108] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0109] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The method for removing wrinkles from portrait videos is shown. The computer-readable storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0110] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0111] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0112] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method of wrinkle removal from a portrait video, characterized by, The method comprises: acquiring a to-be-processed video, selecting a video frame of a same face in the to-be-processed video as a first key frame image, matching and labeling each first key frame image based on a preset expression category label, and performing wrinkle segmentation on the first key frame image to obtain a corresponding first wrinkle mask image; performing connected domain detection on the first wrinkle mask image to obtain N connected domains, assigning values to corresponding pixel values in the first wrinkle mask image according to a relationship between a gravity center of each connected domain and a preset range to obtain a second wrinkle mask image, and storing the expression category label corresponding to the first key frame image and the second wrinkle mask image in a first set; wherein when the expression category label corresponding to the first key frame image is a first category, assigning values to corresponding pixel values in the first wrinkle mask image according to a relationship between a gravity center of each connected domain and a preset range to obtain a second wrinkle mask image comprises: when the gravity center is in a first circle and the maximum distance of pixels in the connected domain from the gravity center is greater than 0.75*the first radius, it is determined that the wrinkle type is a smile line, and the corresponding pixel value in the first wrinkle mask image is set to 128, wherein the first radius is the distance from the midpoint of the left ala nasi point and the left corner of the mouth to the left ala nasi point, and the first circle is a circle with the midpoint of the left ala nasi point and the left corner of the mouth as the center and the first radius as the radius; when the gravity center is in a second circle and the maximum distance of pixels in the connected domain from the gravity center is greater than 0.75*the second radius, it is determined that the wrinkle type is a smile line, and the corresponding pixel value in the first wrinkle mask image is set to 128, wherein the second radius is the distance from the midpoint of the right ala nasi point and the right corner of the mouth to the right ala nasi point, and the second circle is a circle with the midpoint of the right ala nasi point and the right corner of the mouth as the center and the second radius as the radius; when the distance between the gravity center and the left or right corner of the eye is less than a first threshold, it is determined that the wrinkle type is an eye line, and the corresponding pixel value in the first wrinkle mask image is set to 128; performing face 3D reconstruction on the first key frame image to obtain a corresponding 3D face model and a first mvp matrix; performing face 3D reconstruction on other video frames in the to-be-processed video to obtain a second mvp matrix, performing mapping processing on the second wrinkle mask image based on the first mvp matrix and the second mvp matrix to obtain a third wrinkle mask image corresponding to a current video frame; performing face wrinkle removal on the current video frame according to the third wrinkle mask image to obtain a target video.

2. The method of claim 1, wherein the step of applying the wrinkle removal process to the video image of the person is performed by a computer. The selection of the video frame of the same face in the to-be-processed video as the first key frame image comprises: screening the video frame of the same face in the to-be-processed video whose face is not occluded as a candidate key frame image; performing hierarchical clustering on the candidate key frame image according to a preset aggregation strategy, dividing into N categories according to the same expression, and extracting the video frame with the smallest distance to the clustering center in each same category as the first key frame image.

3. The method of claim 2, wherein the video image is a human image. The human face is not occluded by judging that a polygon area constituted by a human face skin area and an outer contour point of the human face is greater than a threshold value.

4. The method of claim 1, wherein the step of removing wrinkles from the video portrait is performed by using a method of claim 3. When the expression category label corresponding to the first key frame image is a second category, the assigning of the pixel value in the first wrinkle mask image according to the relationship between the barycenter of each connected domain and the preset range to obtain a second wrinkle mask image comprises: performing Gaussian filtering on the first key frame image to obtain a filtering result image; calculating a difference value between the pixel value of the first key frame image and the pixel value of the filtering result image, and when the difference value is less than a second threshold value, determining that the wrinkle type is a shallow wrinkle, and setting the corresponding pixel value in the first wrinkle mask image to 64.

5. The method for removing wrinkles from portrait videos according to claim 1, characterized in that, The mapping processing of the second wrinkle mask image based on the first mvp matrix and the second mvp matrix to obtain a third wrinkle mask image corresponding to the current video frame comprises: when the expression category label of the current video frame exists in the first set, projecting the second wrinkle mask image onto the 3D face model according to the first mvp matrix to obtain an intermediate wrinkle mask image, and projecting the intermediate wrinkle mask image onto the current video frame through the second mvp matrix to obtain the third wrinkle mask image corresponding to the current video frame.

6. The method of claim 1, wherein the step of removing wrinkles from the video portrait is performed by using a method of claim 5. The face wrinkle removal of the current video frame according to the third wrinkle mask image to obtain a target video comprises: performing texture replacement on the region with a pixel value of 255 in the third wrinkle mask image by searching for skin texture in the adjacent non-wrinkle region; performing Gaussian filtering on the region with a pixel value of 128 in the third wrinkle mask image, and fusing the Gaussian filtering result with the corresponding first key frame image according to a degree value of 50%; performing curve brightening operation on the region with a pixel value of 64 in the third wrinkle mask image.

7. A device for removing wrinkles from portrait videos, characterized in that, The device comprises: a key frame processing unit configured to acquire a to-be-processed video, select video frames of the same human face in the to-be-processed video that meet a preset condition as first key frame images, match and label each first key frame image based on a preset expression category label, and perform wrinkle segmentation on the first key frame images to obtain corresponding first wrinkle mask images; a mask image analysis unit configured to perform connected domain detection on the first wrinkle mask images to obtain N connected domains, assign a pixel value in the first wrinkle mask image according to the relationship between the barycenter of each connected domain and a preset range to obtain a second wrinkle mask image, and store the expression category label corresponding to the first key frame image and the second wrinkle mask image in a first set; wherein, when the expression category label corresponding to the first key frame image is a first category, the mask image analysis unit is further configured to: when the gravity center is in the first circle and the maximum distance between the pixels in the connected domain and the gravity center is greater than 0.75*the first radius, it is determined that the wrinkle type is a smile wrinkle, and the corresponding pixel value in the first wrinkle mask image is set to 128, wherein the first radius is the distance from the midpoint of the left ala nasi point and the left corner of the mouth to the left ala nasi point, and the first circle is a circle with the midpoint of the left ala nasi point and the left corner of the mouth as the center and the first radius as the radius; when the gravity center is in the second circle and the maximum distance between the pixels in the connected domain and the gravity center is greater than 0.75*the second radius, it is determined that the wrinkle type is a smile wrinkle, and the corresponding pixel value in the first wrinkle mask image is set to 128, wherein the second radius is the distance from the midpoint of the right ala nasi point and the right corner of the mouth to the right ala nasi point, and the second circle is a circle with the midpoint of the right ala nasi point and the right corner of the mouth as the center and the second radius as the radius; when the distance between the gravity center and the left or right corner of the eye is less than the first threshold, it is determined that the wrinkle type is an eye wrinkle, and the corresponding pixel value in the first wrinkle mask image is set to 128; a first reconstruction unit configured to perform 3D face reconstruction on the first key frame image to obtain a corresponding 3D face model and a first mvp matrix; a second reconstruction unit configured to perform 3D face reconstruction on other video frames in the to-be-processed video to obtain a second mvp matrix, and perform mapping processing on the second wrinkle mask image based on the first mvp matrix and the second mvp matrix to obtain a third wrinkle mask image corresponding to the current video frame; a wrinkle removal unit configured to perform face wrinkle removal on the current video frame according to the third wrinkle mask image to obtain a target video.

8. A portrait video wrinkle removal apparatus, characterized by, A computer program is stored in the memory and executed by the processor to implement the steps of the portrait video wrinkle removal method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored in the memory and executed by the processor to implement the steps of the portrait video wrinkle removal method according to any one of claims 1 to 6.

Images