Live video processing method, computer device and computer readable storage medium

Abstract

This application relates to a live video processing method, computer device, and storage medium, which can improve the efficiency of face beautification in live videos. The method includes: obtaining a target video frame containing a face from a live video; beautifying the face using a preset face beautification model with first beautification parameters to obtain a beautified video frame to be evaluated; obtaining a face evaluation result based on the similarity between the face in the beautified video frame and face images in a preset image library; if the face evaluation result does not meet a preset beautification termination condition, then beautifying the face again using a preset face beautification model with second beautification parameters until the face evaluation result after the second beautification meets the preset beautification termination condition; if the face evaluation result meets the preset beautification termination condition, then the beautified target video frame is obtained.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a live video processing method, computer device, and computer-readable storage medium. Background Technology

[0002] With the development of computer technology, live video applications are becoming increasingly widespread, allowing users to interact face-to-face with others in real time. To improve image quality, facial enhancement features can be applied when using live video applications.

[0003] In related technologies, when beautifying faces in live streams, users need to manually adjust each element one by one, such as adjusting the eyes first, then the nose, and finally the eyebrows. However, in order to obtain a good image effect, users often need to make repeated adjustments, which is time-consuming and laborious, resulting in low efficiency in beautifying faces in live video. Summary of the Invention

[0004] Therefore, it is necessary to provide a live video processing method, computer equipment, and storage medium that can improve the efficiency of facial beautification in live videos, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a method for processing live video. The method includes:

[0006] Extract target video frames containing faces from live video streams;

[0007] The face is beautified using a preset face beautification model and a first beautification parameter to obtain a beautified video frame to be evaluated.

[0008] A face evaluation result is obtained based on the similarity between the face in the beautified video frame and each face image in the preset image library; wherein each face image in the preset image library has a corresponding evaluation result.

[0009] If the face evaluation result does not meet the preset beautification termination condition, the preset face beautification model uses the second beautification parameter to beautify the face in the beautified video frame again until the face evaluation result after the second beautification process meets the preset beautification termination condition; the second beautification parameter is determined based on the face parameters of the target face image in the preset image library; the target face image is a face image in the preset image library whose similarity to the face in the beautified video frame reaches a similarity threshold;

[0010] If the face evaluation result meets the preset beautification termination condition, then the beautified target video frame is obtained.

[0011] In one embodiment, the face parameters include positional parameters of facial key points; before the face in the beautified video frame is further beautified by the preset face beautification model using the second beautification parameters, the method further includes:

[0012] Obtain the similarity between the faces in the beautified video frame and the face images in the preset image library;

[0013] Based on the similarity and the evaluation results of each face image, at least one target face image is obtained;

[0014] Based on the position parameters of each facial key point in the at least one target face image, the position adjustment parameters of each facial key point in the beautified video frame are obtained as the second beautification parameters of the face in the beautified video frame.

[0015] In one embodiment, obtaining at least one target face image based on the similarity and the evaluation results of each face image includes:

[0016] Based on a preset similarity threshold and preset evaluation result conditions, at least one candidate face image is filtered to obtain at least one target face image, wherein the similarity of the target face image reaches the preset similarity threshold and the evaluation result of the target face image satisfies the preset evaluation result conditions.

[0017] In one embodiment, obtaining the position adjustment parameters of each facial key point in the beautified video frame based on the position parameters of each facial key point in the at least one target facial image includes:

[0018] Obtain the parameter weights for each target face image;

[0019] For each facial key point in the beautified video frame, based on the position parameters of the facial key point in each target face image and the parameter weights of each target face image, the target position parameters of the facial key point in the beautified video frame are obtained.

[0020] Based on the target position parameters of each facial key point, the position adjustment parameters of each facial key point in the beautified video frame are obtained.

[0021] In one embodiment, obtaining the face evaluation result based on the similarity between the face in the beautified video frame and face images in a preset image library includes:

[0022] Based on the similarity between the faces in the beautified video frames and the face images in the preset image library, the most similar face image in the preset image library is determined.

[0023] Based on the evaluation results of the most similar face image, the face evaluation results of the faces in the beautified video frame are obtained.

[0024] In one embodiment, after the beautified target video frame, the method further includes:

[0025] If a face in a subsequent video frame of the live video is detected to match a face in the target video frame, then the face in the subsequent video frame is beautified according to the face parameters of the beautified target video frame.

[0026] In one embodiment, after the beautified target video frame, the method further includes:

[0027] If a face in a subsequent video frame of the live video is detected to not match the face in the target video frame, or if an update indication for the face parameters is detected, then the subsequent video frame is used as the new target video frame and the beautification parameters for the face in the new target video frame are redefined.

[0028] In one embodiment, the process of beautifying the face using a preset face beautification model and first beautification parameters includes:

[0029] The key points of the face shape and facial features of the person are determined, resulting in multiple key points of the face;

[0030] The parameters are adjusted according to the position of each facial key point in the face to obtain the first beautification parameter;

[0031] The preset face beautification model uses the first beautification parameters to adjust the key points of each face in the face.

[0032] Secondly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0033] Extract target video frames containing faces from live video streams;

[0034] Based on a preset face beautification model, the face is beautified using the first beautification parameter to obtain beautified video frames to be evaluated.

[0035] Based on the similarity between the face in the beautified video frame to be evaluated and each face image already evaluated in the preset image library, the corresponding face evaluation result is obtained.

[0036] If the face evaluation result does not meet the preset beautification termination condition, the face is beautified again using the second beautification parameter based on the preset face beautification model until the corresponding face evaluation result meets the preset beautification termination condition; the second beautification parameter is determined according to the face parameters of the target face image in the preset image library; the target face image is a face image in the preset image library whose similarity to the face in the beautified video frame to be evaluated reaches a similarity threshold;

[0037] If the face evaluation result meets the preset beautification end condition, the beautified target video frame is obtained, and the processed live video is obtained based on the face parameters of the beautified target video frame.

[0038] Thirdly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0039] Extract target video frames containing faces from live video streams;

[0040] The face is beautified using a preset face beautification model and a first beautification parameter to obtain a beautified video frame to be evaluated.

[0041] A face evaluation result is obtained based on the similarity between the face in the beautified video frame and each face image in the preset image library; wherein each face image in the preset image library has a corresponding evaluation result.

[0042] If the face evaluation result does not meet the preset beautification termination condition, the preset face beautification model uses the second beautification parameter to beautify the face in the beautified video frame again until the face evaluation result after the second beautification process meets the preset beautification termination condition; the second beautification parameter is determined based on the face parameters of the target face image in the preset image library; the target face image is a face image in the preset image library whose similarity to the face in the beautified video frame reaches a similarity threshold;

[0043] If the face evaluation result meets the preset beautification termination condition, then the beautified target video frame is obtained.

[0044] The aforementioned live video processing method, computer equipment, and computer-readable storage medium can acquire target video frames containing faces from live videos, perform beautification processing on the faces using a preset face beautification model with first beautification parameters, and obtain beautified video frames to be evaluated. Then, based on the similarity between the faces in the beautified video frames and face images in a preset image library, a face evaluation result can be obtained, where each face image in the preset image library has a corresponding evaluation result. If the face evaluation result does not meet the preset beautification termination condition, the preset face beautification model performs beautification processing on the faces in the beautified video frames again using a second beautification parameter, until the face evaluation result after the second beautification processing meets the preset beautification termination condition. The second beautification parameter is determined based on the face parameters of the target face image in the image library, where the target face image is a face image in the image library whose similarity to the face in the beautified video frame to be evaluated reaches a similarity threshold. If the face evaluation result meets the preset beautification termination condition, the beautified target video frame is obtained. In this application, on the one hand, the similarity between the face in the beautified video frame and the face image in the image library can be used to obtain the overall face evaluation result, thereby improving the reliability and accuracy of the face evaluation result. On the other hand, when it is determined from the face evaluation result that the face in the beautified video frame should continue to be optimized, a second beautification parameter that is adapted to the face in the beautified video frame can be determined based on the similar target face image as a whole, until a face evaluation result that meets the preset beautification termination condition is obtained. This allows the second beautification parameter to be adjusted adaptively and quickly during the overall face beautification process, without requiring the user to manually adjust the parameter or select the adjustment mode for different parts of the face, effectively improving the efficiency of face beautification. Attached Figure Description

[0045] Figure 1 This is an application environment diagram of a live video processing method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating a live video processing method in one embodiment;

[0047] Figure 3 In one embodiment, a subset of facial images is taken from a pre-defined image library.

[0048] Figure 4 This is a flowchart illustrating one step in determining a second beautification parameter in one embodiment;

[0049] Figure 5a This is a schematic diagram of facial key points in one embodiment;

[0050] Figure 5b This is a schematic diagram illustrating the adjustment of facial key points in one embodiment.

[0051] Figure 5cThis is a schematic diagram of a human face after beautification processing in one embodiment;

[0052] Figure 6 This is a flowchart illustrating another live video processing method in one embodiment;

[0053] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The live video processing method provided in this application embodiment can be applied to, for example, Figure 1 The application environment shown may include a terminal and a server. The terminal communicates with the server via a network, and can collect live video data and upload the obtained live video to the server.

[0056] In this application, the server can receive live video uploaded by the terminal and obtain target video frames containing faces from the live video. Then, a preset face beautification model can be used to beautify the faces using a first beautification parameter to obtain beautified video frames to be evaluated. Based on the similarity between the faces in the beautified video frames to be evaluated and the face images in a preset image library, a face evaluation result is obtained. Each face image in the preset image library has a corresponding evaluation result. If the face evaluation result does not meet the preset beautification termination condition, the preset face beautification model can be used to beautify the faces in the beautified video frames again using a second beautification parameter until the beautified face evaluation result meets the preset beautification termination condition. If the face evaluation result meets the preset beautification termination condition, the beautified target video frame can be obtained.

[0057] Among them, the terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices and portable wearable devices. IoT devices can be smart TVs or smart in-vehicle devices, etc., and portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc.

[0058] A server can be implemented using a standalone server or a server cluster consisting of multiple servers. A server can have a data storage system that can store the data that the server needs to process, such as live video uploaded by the terminal. The data storage system can be integrated into the server or placed in the cloud or on other network servers.

[0059] It is understood that the above application environment is merely an example of this application. The live video processing method in this application can also be applied to other environments. For example, after obtaining the live video, the first client performing live video can process the obtained live video based on the live video processing method of this application, and then upload the processed live video to the server so that the server can send the processed live video to the second client watching the live video for playback.

[0060] In one embodiment, such as Figure 2 As shown, a live video processing method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0061] Step S201: Obtain the target video frame containing a human face from the live video.

[0062] In this step, after acquiring the live video, image recognition can be performed on the video frames of the live video to identify the target video containing a human face.

[0063] Specifically, when users conduct live video streaming, they can record live video that includes faces. For example, the streamer can communicate face-to-face with other viewers in the virtual space (also known as the live streaming room) through the live video, or there can be video calls between multiple users.

[0064] It is understandable that a live video may contain multiple video frames that contain faces. After acquiring the live video (for example, after the server receives the video stream data of the live video), it can identify one or more video frames in the live video that contain faces, and determine the target video frame among the one or more video frames that contain faces.

[0065] In some embodiments, the first video frame containing a face in the live video can be used as the target video frame. In other embodiments, multiple video frames containing faces can be identified from the live video first, and a video frame whose image quality meets the quality conditions can be determined as the target video frame based on the image quality of the multiple video frames containing faces.

[0066] Step S202: The face is beautified using the first beautification parameter by the preset face beautification model to obtain the beautified video frame to be evaluated.

[0067] The face beautification model can be a model used to beautify faces, such as a face beautification algorithm. For example, the face beautification model can beautify at least one of the following objects: facial features, face shape, skin (such as skin color or roughness), hairstyle, and facial lighting effects.

[0068] The beautification parameters can be understood as the adjustment parameters used by the face beautification model when beautifying the face. To distinguish them from the beautification parameters mentioned in the subsequent steps of this embodiment, the beautification parameters used when beautifying the face in the target video frame for the first time can be called the first beautification parameters.

[0069] Specifically, after obtaining the target video frame, the face can be beautified using a preset face beautification model and a first beautification parameter; for example, the first beautification parameter can be a default value.

[0070] It is understandable that different beautification parameters will result in different face beautification effects. In this step, the video frames obtained after beautification can be used as beautified video frames to be evaluated, and the beautified faces in the beautified video frames can be evaluated to determine whether the current face beautification method of the face beautification model is appropriate.

[0071] Step S203: Obtain face evaluation results based on the similarity between the faces in the beautified video frames and the face images in the preset image library; each face image in the preset image library has its own evaluation result.

[0072] As an example, the evaluation results of faces in beautified video frames and the evaluation results of face images in a preset image library can be in the form of evaluation scores, evaluation text expressed in words, or expressions such as happy, smiling, or sad.

[0073] Specifically, an image library can be pre-set, which may include multiple facial images. In one possible embodiment, the faces in the image library may include images of public figures such as actors, artists, or singers, or images of faces provided with authorization from ordinary people. For example... Figure 3 The image shown is a partial image of human faces from an image library.

[0074] Each face image in the image library has a corresponding evaluation result. This evaluation result can be determined based on public opinion. For example, for each face image in the image library, the evaluation result of the face image can be obtained based on the evaluation of the face image by multiple people; or, the evaluation of the face image can be determined based on the aesthetic standards of human faces; or, users who upload live videos can pre-evaluate multiple face images. For example, based on users' habits of beautifying faces, multiple face images with similar facial features can be pre-selected, and users can be invited to score the selected face images.

[0075] Since each face image in the image library already has a corresponding evaluation result, and the face in the beautified video frame can be similar to one or more face images in the image library, the evaluation results of the face images in the image library can be used as a reference when evaluating the face in the beautified video frame. Specifically, after obtaining the beautified video frame to be evaluated, the face evaluation result can be obtained based on the similarity between the face in the beautified video frame and each evaluated face image in the image library.

[0076] Step S204: If the face evaluation result does not meet the preset beautification end condition, the preset face beautification model uses the second beautification parameter to beautify the face in the beautified video frame again until the face evaluation result after the second beautification process meets the preset beautification end condition.

[0077] The beautification parameters used in subsequent face beautification processes are referred to as the second beautification parameters. These second beautification parameters can be determined based on the facial parameters of a target face image in the image library. This target face image is defined as a face image in the image library whose similarity to the face in the beautified video frame to be evaluated reaches a similarity threshold.

[0078] In practical applications, beautification termination conditions can be preset, which can characterize the face achieving a preset beautification effect. In some embodiments, if the face evaluation result is a quantitative evaluation result, such as an evaluation score, the preset beautification termination condition can be a score threshold; or, if the face evaluation result is a qualitative evaluation result, such as an evaluation level or other evaluation text, the preset beautification termination condition can be a preset evaluation level or evaluation text containing preset text information, which can be text information expressing positive opinions, such as "very beautiful" or "good-looking".

[0079] After obtaining the face evaluation results for the faces in the beautified video frames, it can be determined whether the face evaluation results meet the preset beautification termination conditions. If not, a second beautification parameter can be obtained. The second beautification parameter can be determined based on the face parameters in the target face image. Then, the preset face beautification model can use the second beautification parameter to further beautify the faces in the beautified video frames. In this step, the second beautification parameter is determined using the target face image that has reached the similarity threshold. The faces in the beautified video frames can be optimized by using face images with similar appearances, so that the beautified faces can correspond to the faces of the people being filmed.

[0080] After the face beautification model uses a second beautification parameter to further beautify the faces in the beautified video frame, it can continue to obtain face evaluation results for the beautified video frame. If the face evaluation result still does not meet the preset beautification termination condition, a new second beautification parameter is obtained, and the face beautification model uses the second beautification parameter to further beautify the faces in the beautified video frame. This beautification process is repeated until the face evaluation result obtained after beautification meets the preset beautification termination condition, at which point the beautification process can be stopped.

[0081] In some related technologies, live streaming clients can provide multiple facial beautification functions. Each facial beautification function can adjust different parts of the face. For example, it can provide three facial beautification functions: "skin smoothing", "face slimming (e.g., oval face and heart-shaped face)" and "eye enlargement". The live streaming client can adjust the facial position targeted by the selected facial beautification function according to the user's selection using preset parameters. For example, if the user selects the "skin smoothing" function, the live streaming client will only adjust the facial skin.

[0082] Although the above methods do not require users to manually set specific beautification parameters, since the human face is a whole composed of multiple organs, the above methods only require users to select the parts to be adjusted based on experience or personal preferences. The resulting facial coordination is poor (for example, after "enlarging" the eyes, the proportions of the facial features are unbalanced), and it is often difficult to obtain good results. Users need to switch the facial parts to be adjusted or the adjustment mode multiple times.

[0083] In this embodiment, on the one hand, each time a face beautification is completed, a face evaluation result for the entire face can be obtained based on the similarity between the face in the beautified video frame and the face images in the image library, rather than an evaluation of a single part of the face. This evaluation method is compatible with how humans evaluate whether a face is beautiful in daily life, effectively improving the reliability and accuracy of the face evaluation result. On the other hand, when it is determined that the face in the beautified video frame needs further optimization based on the face evaluation result, the second beautification parameter is determined based on the overall similar target face images. This can improve the adaptability of the second beautification parameter with the face in the beautified video frame and the coordination of the face. Furthermore, the face beautification model continuously beautifies the face using the second beautification parameter until a face evaluation result that meets the preset beautification termination condition is obtained. This allows the second beautification parameter to be adaptively and quickly adjusted during the face beautification process. When adjusting the parameters of the entire face, no manual adjustment by the user is required, effectively improving the efficiency of face beautification.

[0084] Step S205: If the face evaluation result meets the preset beautification end condition, then the beautified target video frame is obtained.

[0085] If the face evaluation result at level two meets the preset beautification termination condition, the current beautified video frame can be determined as the target video frame after beautification processing. Specifically, if the beautified video frame to be evaluated has undergone multiple beautification processes through the face beautification model (i.e., the first beautification process based on the first beautification parameter and at least one subsequent beautification process based on the second beautification parameter), then the final beautified video frame obtained after multiple beautification processes can be used as the target video frame after beautification processing. Of course, if the face evaluation result of the beautified video frame to be evaluated has met the preset beautification termination condition after one beautification process (i.e., the first beautification process based on the first beautification parameter), then that beautified video frame can be used as the target video frame after beautification processing.

[0086] After obtaining the target video frame, other video frames in the live video can be processed based on the facial parameters of the beautified target video frame to obtain the processed live video.

[0087] In this embodiment, a target video frame containing a face can be obtained from a live video. A preset face beautification model uses a first beautification parameter to beautify the face, resulting in a beautified video frame to be evaluated. Then, a face evaluation result can be obtained based on the similarity between the face in the beautified video frame and face images in a preset image library, where each face image in the preset image library has its own evaluation result. If the face evaluation result does not meet the preset beautification termination condition, the preset face beautification model uses a second beautification parameter to beautify the face in the beautified video frame again until the face evaluation result after the second beautification process meets the preset beautification termination condition. The second beautification parameter is determined based on the face parameters of a target face image in the image library, where the target face image is a face image in the image library whose similarity to the face in the beautified video frame to be evaluated reaches a similarity threshold. If the face evaluation result meets the preset beautification termination condition, the beautified target video frame is obtained. In this application, on the one hand, the similarity between the face in the beautified video frame and the face image in the image library can be used to obtain the overall face evaluation result, thereby improving the reliability and accuracy of the face evaluation result. On the other hand, when it is determined from the face evaluation result that the face in the beautified video frame should continue to be optimized, a second beautification parameter that is adapted to the face in the beautified video frame can be determined based on the similar target face image as a whole, until a face evaluation result that meets the preset beautification termination condition is obtained. This allows the second beautification parameter to be adjusted adaptively and quickly during the overall face beautification process, without requiring the user to manually adjust the parameter or select the adjustment mode for different parts of the face, effectively improving the efficiency of face beautification.

[0088] In one embodiment, facial parameters include location parameters of facial landmarks. For example... Figure 4As shown, before the faces in the beautified video frames are further beautified using the second beautification parameter by the preset face beautification model, the method may further include the following steps:

[0089] Step S401: Obtain the similarity between the faces in the beautified video frame and the face images in the image library.

[0090] Specifically, after obtaining the beautified video frame, the faces in the beautified video frame can be compared with each evaluated face image in the image library to obtain the similarity between the faces in the beautified video frame and each face image in the image library.

[0091] Step S402: Based on the similarity and the evaluation results of each face image, obtain at least one target face image.

[0092] In practical applications, each evaluated face image in the image library can have a corresponding evaluation result. After determining the similarity, at least one target face image can be selected to determine the second beautification parameter based on the similarity between the face and each face image and the evaluation result of each face image.

[0093] In one embodiment, the evaluation result of a face image can be related to the popularity of the face in the image or the degree of user (such as the user providing the live video or the general public) liking (or attention). For example, when the evaluation result is a score, the score is positively correlated with the popularity of the face. Therefore, a face image with a corresponding evaluation result can be obtained as the target face image based on the similarity of the face images and the desired face beautification effect by the user.

[0094] Step S403: Based on the position parameters of each facial key point in at least one target face image, obtain the position adjustment parameters of each facial key point in the face, and use them as the second beautification parameters for the face in the beautification video frame.

[0095] Facial landmarks can be understood as key feature points on the face. By using multiple facial landmarks, the key areas of the face can be located. Figure 5a This shows multiple facial landmarks in a face. These landmarks are connected by lines to outline the important facial areas.

[0096] Location parameters can be parameters that characterize the location of facial key points, such as the two-dimensional coordinates of facial key points on a plane or the three-dimensional coordinates in a three-dimensional model.

[0097] Position adjustment parameters can be parameters used to adjust the position of facial key points. For example, position adjustment parameters can be specific locations, such as... Figure 5bThe image shows the adjusted positions of facial key points; of course, the position adjustment parameter can also be an offset.

[0098] In this step, after obtaining at least one target face image, the position adjustment parameters of each face key point in the beautified video frame can be determined based on the position parameters of each face key point in each target face image, and the position adjustment parameters are used as the second beautification parameters of the face in the beautified video frame.

[0099] In this embodiment, the position adjustment parameters of each facial key point in the face can be determined through the target face image, and the second beautification parameter can be obtained based on the position adjustment parameters. The parameters of the face in the video frame can be adjusted pixel by pixel through the facial key points, and more detailed face beautification can be achieved in the process of adaptively optimizing the face.

[0100] In one embodiment, at least one target face image is obtained based on similarity and the evaluation results of each face image, including:

[0101] Based on a preset similarity threshold and preset evaluation result conditions, at least one candidate face image is screened to obtain at least one target face image. The similarity of the target face image reaches the similarity threshold and the evaluation result of the target face image meets the preset evaluation result conditions.

[0102] In practical applications, after obtaining the similarity between a face and various face images, candidate face images can be filtered using a similarity threshold and preset evaluation result conditions. In one embodiment, candidate face images whose similarity reaches the similarity threshold can be obtained first. Then, among the candidate face images that reach the similarity threshold, the candidate face images whose face evaluation results meet the preset evaluation result conditions are identified as the target face images. For example, the preset evaluation result conditions can be the K highest-rated (K is a positive integer greater than or equal to 1) faces. If the evaluation result is an evaluation score, the preset evaluation result conditions can also be that the evaluation score is greater than a score threshold, or that the evaluation score belongs to a certain evaluation score range, which is determined based on the optimization effect the user intends to achieve.

[0103] In this embodiment, by filtering at least one candidate face image according to a preset similarity threshold and preset evaluation result conditions, it is possible to avoid the optimized face being too different from the user's actual face, ensuring the fit between the optimized face and the user's actual face, and also to obtain excellent face beautification effect.

[0104] In one embodiment, determining the position adjustment parameters of each facial key point in a face based on the position parameters of each facial key point in at least one target face image may include the following steps:

[0105] Obtain the parameter weights of each target face image; for each facial key point in the beautified video frame, based on the position parameters of the facial key point in each target face image and the parameter weights of each target face image, obtain the target position parameters of the facial key point in the beautified video frame; based on the target position parameters of each facial key point, obtain the position adjustment parameters of each facial key point in the beautified video frame.

[0106] Here, parameter weights can be understood as the proportion of position parameters.

[0107] In practical applications, after obtaining the target face image, the parameter weights of each target face image can be obtained.

[0108] In an optional embodiment, the parameter weight of each target face image can be determined based on the similarity between the target face image and the face in the beautified video frame, as well as the similarity between each of the target face images and that face. When determining the parameter weight, the similarity of all target face images can be summed first. For example, if target face images A, B, C, and D have similarities of 50%, 60%, 70%, and 80% with the faces in the beautified video frame, respectively, the summation result is 260%. Then, the parameter weight can be determined based on the ratio of the similarity of each target face image to the summation result. For example, for target face image A with a similarity of 50%, its corresponding parameter weight is 50% / 260%.

[0109] After determining the parameter weights for each target face image, since multiple facial key points can be extracted from each face image, the target position parameters of each facial key point in the beautified video frame can be calculated based on the position parameters of that facial key point in each target face image and the parameter weights of each face image. Specifically, for example, the position parameters of that facial key point in each target face image can be weighted and summed according to the parameter weights to obtain the corresponding result, and then the target position parameters can be obtained based on this result. After determining the position of each facial key point, the target position parameters of each facial key point can be used as the position adjustment parameters for each facial key point in the face.

[0110] In this embodiment, the position adjustment parameters of the face can be determined by combining the position parameters of the key facial points in each target face image. This allows for the comprehensive adjustment of the advantages of each target face image during the beautification process, thereby improving the beautification effect.

[0111] In one embodiment, beautifying a face using a preset face beautification model and first beautification parameters may include the following steps:

[0112] The key points of the face shape and facial features are determined to obtain multiple key points; the parameters are adjusted according to the position of each key point in the face to obtain the first beautification parameter; based on the preset face beautification model, the first beautification parameter is used to adjust each key point in the face.

[0113] In practice, the facial contour points and key facial features of the face in the beautified video frame can be determined, and these key points are identified as multiple facial key points for that face. Then, the position adjustment parameters for each facial key point can be obtained as the first beautification parameter. Using this first beautification parameter, the facial beautification model adjusts the positions of each facial key point to obtain the beautified face. For example, Figure 5a After facial landmark adjustments, a face can be obtained as follows: Figure 5c The face shown has been beautified.

[0114] In this embodiment, by acquiring multiple facial key points, including facial shape key points and facial feature key points, and using these multiple facial key points as position adjustment parameters as the first beautification parameter, the face can be adjusted. This allows for detailed, pixel-by-pixel adjustments to the face shape and facial features in the target video frame, facilitating matching and adjustment based on different faces in the target video frame and effectively improving the aesthetics of the face.

[0115] In one embodiment, obtaining a face evaluation result based on the similarity between a face in a beautified video frame and face images in a preset image library may include the following steps:

[0116] Based on the similarity between the faces in the beautified video frames and the face images in the preset image library, the most similar face image in the preset image library is determined; based on the evaluation results of the most similar face image, the face evaluation results of the faces in the beautified video frames are obtained.

[0117] Specifically, after obtaining the similarity between the face in the beautified video frame and each evaluated face image in the preset image library, the face image with the highest similarity can be determined from the multiple face images stored in the preset image library. This face image is then used as the most similar face image in the preset image library. The evaluation result of the most similar face image can then be obtained and used as the face evaluation result for the face in the beautified video frame.

[0118] In this embodiment, by using the evaluation result of the most similar face image in the image library as the face evaluation result, the evaluation complexity can be reduced, a suitable face evaluation result can be quickly matched, and it can be determined whether the beautification termination condition is met, thereby improving the beautification efficiency.

[0119] In one embodiment, after obtaining the beautified target video frame, this application may further include the following steps:

[0120] If a face in a subsequent video frame of the live video is detected to match a face in the target video frame, then the face in the subsequent video frame is beautified according to the face parameters of the beautified target video frame.

[0121] In practical applications, when capturing faces, the faces in the frame may not change over a period of time; for example, the same face may appear within a few seconds. Since video frames have a small temporal granularity, the content between multiple video frames is relatively stable. For video frames following the target video frame in a live stream, these subsequent frames can contain the same person's face. In this step, after obtaining the beautified target video frame, the facial parameters of the faces in the target video frame can be determined under the condition that the beautification ends. Using these facial parameters as a benchmark, beautification processing is applied to the faces in subsequent video frames. For example, the facial parameters of the faces in subsequent video frames can be adjusted to match those of the faces in the target video frame without recalculating the beautification parameters. This effectively improves the efficiency and speed of face beautification during live video streaming, enabling real-time processing of face beautification in live videos.

[0122] In one embodiment, after obtaining the beautified target video frame, this application may further include the following steps:

[0123] If a mismatch is detected between the face in a subsequent video frame and the face in the target video frame, or if an update indication for the face parameters is detected, then the subsequent video frame is used as the new target video frame, and the beautification parameters for the face in the new target video frame are redefined.

[0124] In practice, the faces captured in live video can change. For example, new faces may be added or existing faces may be replaced with other faces. If the original face parameters are used to beautify the changed faces, the beautification effect will be poor. Also, as the shooting scene changes, such as from indoor to outdoor shooting, the beautification parameters used will also differ.

[0125] In this embodiment, faces in subsequent video frames can be detected, or it can be detected whether an update instruction for face parameters has been received (e.g., triggered by the user). When a mismatch is detected between a face in a subsequent video frame and a face in a target video frame, or when an update instruction for face parameters is detected, subsequent video frames of the live video can be used as new target video frames. Based on the steps in any of the above embodiments, the beautification parameters of the faces in the new target video frames are redefined, so that during the beautification processing of the live video, the beautification parameters can be updated in a timely manner according to the shooting content or user instructions, thereby optimizing the beautification effect of faces in different live content.

[0126] To enable those skilled in the art to better understand the above steps, the following example illustrates the embodiments of this application, but it should be understood that the embodiments of this application are not limited thereto.

[0127] like Figure 6 As shown, after obtaining the input live video, a target video frame can be obtained from the live video and input into the face beautification model. The face beautification is performed using the first beautification parameter to obtain the beautified video frame to be evaluated. The face beautification model can also be called a face adjustment algorithm.

[0128] Then, the beautified video frames can be input into the scoring system, which has a preset image library. The scoring system can obtain the corresponding evaluation results by comparing the similarity between the faces in the beautified video frames and the evaluated face images in the preset image library. Based on the evaluation results, it can determine whether the beautification termination condition is met. For example, when the evaluation result is an evaluation score, it can determine whether the evaluation score has reached the optimal level (such as the evaluation score no longer changing after multiple adjustments) or whether it exceeds the score threshold.

[0129] If so, the beautified target video frame can be obtained. Based on the face parameters of the target video frame, subsequent video frames are then beautified, and the processed live video is output. If not, the position parameters of the facial key points in the beautified video frame can be adjusted, and a second beautification parameter can be determined based on the adjusted position parameters. The face beautification model then uses the second beautification parameter to process the face in the beautified video frame again until the evaluation result meets the beautification termination condition.

[0130] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0131] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores live video data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a live video processing method.

[0132] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0133] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0134] Extract target video frames containing faces from live video streams;

[0135] The face is beautified using a preset face beautification model and a first beautification parameter to obtain a beautified video frame to be evaluated.

[0136] A face evaluation result is obtained based on the similarity between the face in the beautified video frame and each face image in the preset image library; wherein each face image in the preset image library has a corresponding evaluation result.

[0137] If the face evaluation result does not meet the preset beautification termination condition, the preset face beautification model uses the second beautification parameter to beautify the face in the beautified video frame again until the face evaluation result after the second beautification process meets the preset beautification termination condition; the second beautification parameter is determined based on the face parameters of the target face image in the preset image library; the target face image is a face image in the preset image library whose similarity to the face in the beautified video frame reaches a similarity threshold;

[0138] If the face evaluation result meets the preset beautification termination condition, then the beautified target video frame is obtained.

[0139] In one embodiment, the processor also performs the steps described in the other embodiments when executing the computer program.

[0140] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0141] Extract target video frames containing faces from live video streams;

[0142] The face is beautified using a preset face beautification model and a first beautification parameter to obtain a beautified video frame to be evaluated.

[0143] A face evaluation result is obtained based on the similarity between the face in the beautified video frame and each face image in the preset image library; wherein each face image in the preset image library has a corresponding evaluation result.

[0144] If the face evaluation result does not meet the preset beautification termination condition, the preset face beautification model uses the second beautification parameter to beautify the face in the beautified video frame again until the face evaluation result after the second beautification process meets the preset beautification termination condition; the second beautification parameter is determined based on the face parameters of the target face image in the preset image library; the target face image is a face image in the preset image library whose similarity to the face in the beautified video frame reaches a similarity threshold;

[0145] If the face evaluation result meets the preset beautification termination condition, then the beautified target video frame is obtained.

[0146] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0147] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0148] Extract target video frames containing faces from live video streams;

[0149] The face is beautified using a preset face beautification model and a first beautification parameter to obtain a beautified video frame to be evaluated.

[0150] A face evaluation result is obtained based on the similarity between the face in the beautified video frame and each face image in the preset image library; wherein each face image in the preset image library has a corresponding evaluation result.

[0151] If the face evaluation result does not meet the preset beautification termination condition, the preset face beautification model uses the second beautification parameter to beautify the face in the beautified video frame again until the face evaluation result after the second beautification process meets the preset beautification termination condition; the second beautification parameter is determined based on the face parameters of the target face image in the preset image library; the target face image is a face image in the preset image library whose similarity to the face in the beautified video frame reaches a similarity threshold;

[0152] If the face evaluation result meets the preset beautification termination condition, then the beautified target video frame is obtained.

[0153] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing live video, characterized in that, The method includes: Extract target video frames containing faces from live video streams; The face is beautified using a preset face beautification model and a first beautification parameter to obtain a beautified video frame to be evaluated; the first beautification parameter is the adjustment parameter used by the face beautification model when beautifying the face. A face evaluation result is obtained based on the similarity between the face in the beautified video frame and each face image in the preset image library; wherein each face image in the preset image library has a corresponding evaluation result. If the face evaluation result does not meet the preset beautification termination condition, the preset face beautification model uses the second beautification parameter to beautify the face in the beautified video frame again until the face evaluation result after the second beautification process meets the preset beautification termination condition; the second beautification parameter is determined based on the face parameters of the target face image in the preset image library; the target face image is a face image in the preset image library whose similarity to the face in the beautified video frame reaches a similarity threshold; If the face evaluation result meets the preset beautification termination condition, then the beautified target video frame is obtained.

2. The method according to claim 1, characterized in that, The facial parameters include the positional parameters of facial key points; before the preset facial beautification model uses the second beautification parameters to further beautify the face in the beautified video frame, it also includes: Obtain the similarity between the faces in the beautified video frame and the face images in the preset image library; Based on the similarity and the evaluation results of each face image, at least one target face image is obtained; Based on the position parameters of each facial key point in the at least one target face image, the position adjustment parameters of each facial key point in the beautified video frame are obtained as the second beautification parameters of the face in the beautified video frame.

3. The method according to claim 2, characterized in that, The step of obtaining at least one target face image based on the similarity and the evaluation results of each face image includes: Based on a preset similarity threshold and preset evaluation result conditions, at least one candidate face image is filtered to obtain at least one target face image, wherein the similarity of the target face image reaches the preset similarity threshold and the evaluation result of the target face image satisfies the preset evaluation result conditions.

4. The method according to claim 2, characterized in that, The step of obtaining position adjustment parameters for each facial key point in the beautified video frame based on the position parameters of each facial key point in the at least one target facial image includes: Obtain the parameter weights for each target face image; For each facial key point in the beautified video frame, based on the position parameters of the facial key point in each target face image and the parameter weights of each target face image, the target position parameters of the facial key point in the beautified video frame are obtained. Based on the target position parameters of each facial key point, the position adjustment parameters of each facial key point in the beautified video frame are obtained.

5. The method according to claim 1, characterized in that, The step of obtaining a face evaluation result based on the similarity between the face in the beautified video frame and face images in a preset image library includes: Based on the similarity between the faces in the beautified video frames and the face images in the preset image library, the most similar face image in the preset image library is determined. Based on the evaluation results of the most similar face image, the face evaluation results of the faces in the beautified video frame are obtained.

6. The method according to claim 1, characterized in that, Following the beautified target video frame, the process also includes: If a face in a subsequent video frame of the live video is detected to match a face in the target video frame, then the face in the subsequent video frame is beautified according to the face parameters of the beautified target video frame.

7. The method according to claim 1, characterized in that, Following the beautified target video frame, the process also includes: If a face in a subsequent video frame of the live video is detected to not match the face in the target video frame, or if an update indication for the face parameters is detected, then the subsequent video frame is used as the new target video frame and the beautification parameters for the face in the new target video frame are redefined.

8. The method according to any one of claims 1 to 7, characterized in that, The process of beautifying the face using a preset face beautification model and first beautification parameters includes: The key points of the face shape and facial features of the person are determined, resulting in multiple key points of the face; The parameters are adjusted according to the position of each facial key point in the face to obtain the first beautification parameter; The preset face beautification model uses the first beautification parameters to adjust the key points of each face in the face.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images