Face key point detection method and device, live broadcast system, electronic equipment and medium

Abstract

The invention relates to a face key point detection method and device applied to video live broadcast, a live broadcast system, electronic equipment and a computer readable storage medium. The method comprises the following steps: extracting a current video frame from a live video stream; respectively transmitting the current video frame to a face detection tracking thread and a face key point detection thread; wherein the face detection tracking thread runs a face detection algorithm and a face tracking algorithm, and the face key point detection thread runs a face key point detection algorithm; recognizing a face frame of the current video frame through a face detection tracking thread, updating the face frame to a face frame cache, synchronously reading the face frame in the face frame cache in real time through a face key point detection thread, and performing face key point detection; according to the technical scheme, two asynchronous threads are used for parallel execution, the frame time consumption can be reduced, the frame loss problem can be avoided, especially in a multi-person group broadcast scene, real-time face key point detection can still be smoothly carried out, and the video live broadcast effect is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of video image technology, and in particular to a method, device, live broadcast system, electronic device and computer-readable storage medium for detecting key points of a face in live video. Background Art

[0002] With the development of network technology, live streaming has been used by most Internet users. With its intuitive, immediacy and interactivity in content and form, live streaming has played an important role in promoting flexible employment, promoting economic and social development, and enriching the spiritual and cultural life of the people. Anchors can better display their talents in live broadcasts, thereby helping more anchors realize their self-worth.

[0003] During live video streaming, in order to achieve various beauty and makeup functions, it is necessary to detect facial key points. The current mainstream facial key point detection method usually detects the face first and then detects the facial key points, that is, the two steps of face detection and facial key point detection are performed serially. This technical solution has the problems of poor real-time performance and high computational overhead. In order to reduce the computational overhead, some technical solutions will add face tracking algorithms to track faces in real time, which can reduce part of the computational overhead of face detection. However, since face detection / tracking and facial key point detection are performed serially, the problem of high frame time consumption and easy frame loss when using face detection cannot be solved. Especially in multi-person live streaming scenarios, frame time consumption and frame loss will become more and more obvious, seriously affecting the video live streaming effect. Summary of the invention

[0004] Based on this, it is necessary to provide a facial key point detection method, device, live broadcast system, electronic device and computer-readable storage medium that can reduce frame consumption and frame loss.

[0005] A method for detecting key points of a face, comprising:

[0006] Extract the current video frame from the live video stream;

[0007] Passing the current video frame to the face detection and tracking thread and the face key point detection thread respectively; wherein the face detection and tracking thread runs the face detection algorithm and the face tracking algorithm, and the face key point detection thread runs the face key point detection algorithm;

[0008] The face frame of the current video frame is identified through the face detection and tracking thread and updated into the face frame cache, and the face key point detection thread is synchronously used to read the face frame in the face frame cache in real time and perform face key point detection.

[0009] In one embodiment, identifying the face frame of the current video frame through the face detection and tracking thread and updating it into the face frame cache includes:

[0010] When the face detection condition is met, the face detection algorithm running in the face detection tracking thread is used to identify the face frame of the current video frame and update it to the face frame cache;

[0011] When the face detection condition is not met, the face tracking algorithm running in the face detection and tracking thread is used to identify the face frame of the current video frame and update it to the face frame cache.

[0012] In one embodiment, identifying the face frame of the current video frame through the face detection and tracking thread and updating it into the face frame cache includes:

[0013] Reading a target video frame from the current video frame according to a set frame interval, setting the target video frame as a face detection frame and setting other video frames as face tracking frames;

[0014] The face detection algorithm running in the face detection and tracking thread is called to sequentially identify the face frame of the current video frame and update it to the face frame cache; and the face tracking algorithm running in the face detection and tracking thread is called to sequentially identify the face frame of the current video frame and update it to the face frame cache.

[0015] In one embodiment, calling the face detection and tracking thread to run a face detection algorithm to identify a face frame of the current video frame includes:

[0016] Call the face detection algorithm running in the face detection and tracking thread and use the deep learning model to infer the current video frame to obtain the face frame;

[0017] The calling of the face detection and tracking thread to run the face tracking algorithm to identify the face frame of the current video frame includes:

[0018] The face detection and tracking thread is called to run the face tracking algorithm and estimate the face frame of the current video frame through the face key points identified in the previous video frame.

[0019] In one embodiment, reading a target video frame from a current video frame according to a set frame interval includes:

[0020] When there is a face in the picture, the target video frame is read from the current video frame according to the set frame interval;

[0021] When there is no human face in the picture or the number of human faces decreases, the frame interval is reduced to increase the density of reading the target video frame in a short time.

[0022] In one embodiment, the face frame in the face frame buffer is read in real time by the face key point detection thread and the face key point detection is performed, including:

[0023] When a face frame exists in the face frame cache, the face frame is read from the face frame cache in sequence by the face key point detection thread;

[0024] The face key points are detected from the current video frame using the face key point detection algorithm run by the face key point detection thread according to the face frame.

[0025] In one embodiment, the detecting of facial key points from the current video frame using a facial key point detection algorithm executed by a facial key point detection thread according to the facial frame includes:

[0026] For each face frame, the face key point detection algorithm running in the face key point detection thread is used to infer the current video frame in parallel to obtain the corresponding face key points;

[0027] The facial key points are screened and the facial key points with low confidence are removed to obtain the facial key point information of the current video frame.

[0028] In one embodiment, the face frame in the face frame buffer is read in real time by the face key point detection thread and face key point detection is performed, including:

[0029] If there are multiple face frames in the current video frame, the importance priority of each face frame is calculated according to the attribute parameters of the face frame and the evaluation function;

[0030] When the face frame is of high priority, a high-precision face key point detection model is used to detect the face key points of the face frame;

[0031] When the face frame has a low priority, a low-precision face key point detection model is used to detect face key points on the face frame.

[0032] In one embodiment, the facial key point detection method further includes:

[0033] When the number of face frames of the current video frame identified by the face detection and tracking thread exceeds a set number threshold, low-priority face frames are discarded according to the importance priority of each face frame.

[0034] In one embodiment, the face frame in the face frame buffer is read in real time by the face key point detection thread and face key point detection is performed, including:

[0035] Reading the face frame in the face frame buffer in real time through the face key point detection thread;

[0036] If there are multiple face frames in the current video frame, calculate the face importance parameters of each face frame, and calculate the maximum time for each face frame to perform face key point detection according to the face importance parameters;

[0037] When the processing time for facial key point detection of any face frame exceeds its maximum time, the face frame is discarded and the facial key point information detected by the previous video frame is used.

[0038] A facial key point detection device, comprising:

[0039] An extraction module, used to extract the current video frame from the live video stream;

[0040] A transfer module, used to transfer the current video frame to a face detection and tracking thread and a face key point detection thread respectively; wherein the face detection and tracking thread runs a face detection algorithm and a face tracking algorithm, and the face key point detection thread runs a face key point detection algorithm;

[0041] The detection module is used to identify the face frame of the current video frame through the face detection and tracking thread and update it to the face frame cache, and synchronously read the face frame in the face frame cache in real time through the face key point detection thread and perform face key point detection.

[0042] A live broadcast system, comprising: an anchor terminal, an audience terminal and a live broadcast server; wherein the anchor terminal and the audience terminal are respectively connected to the live broadcast server via a communication network;

[0043] The anchor terminal is used to access the anchor in the live broadcast room and collect the anchor's live video stream and upload it to the live broadcast server;

[0044] The live broadcast server is used to forward the live broadcast video stream of the anchor to the audience end, and use the facial key point detection method to detect the facial key points in the live broadcast video stream;

[0045] The audience terminal is used for audience users who access the live broadcast room to receive the live broadcast video stream of the anchor for playback.

[0046] An electronic device, comprising:

[0047] one or more processors;

[0048] Memory;

[0049] One or more application programs, wherein the one or more application programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to execute the steps of the facial key point detection method.

[0050] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded by the processor and executes the steps of the facial key point detection method.

[0051] The technical solution of the present application extracts the current video frame from the live video stream and transmits it to the face detection and tracking thread and the face key point detection thread respectively. The face detection and tracking thread runs the face detection algorithm and the face tracking algorithm, identifies the face frame of the current video frame and updates it to the face frame cache. The face key point detection thread runs the face key point detection algorithm, synchronously reads the face frame in the face frame cache in real time and performs face key point detection. This technical solution uses two asynchronous threads to execute in parallel, which can reduce frame time consumption and avoid frame loss problems. In particular, in a multi-person group broadcast scenario, real-time face key point detection can still be performed smoothly, greatly improving the video live broadcast effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of an example live broadcast service application scenario;

[0053] Figure 2 is a flow chart of a method for detecting key points of a face according to an embodiment;

[0054] Figure 3 This is a schematic diagram of facial key point detection in a multi-person scene;

[0055] Figure 4 This is a schematic diagram of the facial key point detection process;

[0056] Figure 5 This is a schematic diagram of an example of a video live broadcast platform extending the multi-person beauty mode;

[0057] Figure 6 is a schematic diagram of the structure of a facial key point detection device according to an embodiment;

[0058] Figure 7 is a block diagram of an example electronic device. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] In the embodiments of the present application, the words "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. "at least one" means one or more, and "multiple" means two or more. For example, multiple objects refer to two or more objects. "Include" or "comprises" and similar words mean that the information appearing before "include" or "comprises" covers the information listed after "include" or "comprises" and its equivalents, and does not exclude other information. "And / or" mentioned in the embodiments of the present application indicates that three types of relationships may exist. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0061] The technical solution provided in the embodiments of the present application can be applied to Figure 1 In the application scenario of the method related to this application shown in FIG. Figure 1 This is a schematic diagram of an example of a live broadcast service application scenario. The live broadcast system may include a live broadcast server, an anchor terminal and an audience terminal. The anchor terminal and the audience terminal communicate with the live broadcast server through a communication network, so that the anchor at the anchor terminal and the audience users at the audience terminal can perform real-time network live broadcast. Among them, for the anchor terminal and the audience terminal, their terminal devices can be but are not limited to various personal computers, laptops, smart phones and tablets. The live broadcast server can be implemented by an independent server or a server cluster composed of multiple servers.

[0062] In order to achieve better live video effects, various special effects are usually added to the host's face to provide beauty makeup, headgear special effects and other functions; in order to achieve the addition of these special effects, it is necessary to detect the key points of the face. Conventional face key point detection technology is based on the serial recognition process of face recognition / tracking and face key point detection, which is time-consuming and easily affects the live video effect. The technical solution of the embodiment of the present application utilizes synchronous threads to execute face detection / tracking and face key point detection in parallel, optimizes face key point recognition, reduces the real-time calculation amount, has the advantages of good real-time performance, stable time consumption, etc., and can achieve the effect of smooth real-time face key point detection in multi-person group broadcast scenes.

[0063] refer to Figure 2 As shown, Figure 2 The flowchart of a method for detecting key points of a face according to an embodiment includes the following steps:

[0064] Step S10, extracting the current video frame from the live video stream.

[0065] In this step, the video frame is extracted from the live video stream and the current video frame F is defined. t , the previous video frame F t-1 , the next video frame F t+1; Exemplarily, the video frames may be extracted frame by frame or at a certain interval; usually, when the FPS (Frames Per Second) of the video frame is high, the current video frame F t There is a strong correlation with the last acquired video frame; for example, in a live broadcast scene, at least 25 FPS can usually be met, so the correlation between video frames is strong, and the facial key points of two highly correlated video frames are close.

[0066] Step S20, passing the current video frame to the face detection and tracking thread and the face key point detection thread respectively; wherein the face detection and tracking thread runs the face detection algorithm and the face tracking algorithm, and the face key point detection thread runs the face key point detection algorithm.

[0067] In this step, two parallel threads are constructed, which are defined as face detection and tracking thread T1 and face key point detection thread T2. t It is passed to two threads T1 and T2 at the same time; among them, the face detection and tracking thread T1 is configured to run the face detection algorithm and the face tracking algorithm, and the face key point detection thread T2 is configured to run the face key point detection algorithm.

[0068] As for the face detection algorithm, it is an algorithm for identifying the position of a face in a video image or video frame. The face area is separated from the background by performing face detection on the input image. The face detection algorithm usually uses feature extraction technology, such as edge, texture, shape, etc., or machine learning models, such as support vector machine, convolutional neural network, etc. to identify facial features. Exemplarily, the face detection algorithm used in this embodiment includes Haar features combined with AdaBoost classifier, HOG (Histogram of Oriented Gradient) features combined with SVM (Support Vector Machine, Support Vector Machine), MTCNN (Multi-task Cascaded Convolutional Networks, face detection algorithm) based on deep learning, etc.

[0069] As for the face tracking algorithm, it is an algorithm for tracking the movement of faces in a video sequence, and can handle the changes of faces between different frames, such as position, size, posture, etc.; in general, the face tracking algorithm can combine face detection and motion model, use Kalman filter, particle filter, etc. to predict the position of the face in the next frame, and can also use optical flow method to analyze the motion information in the image sequence, or use deep learning model to directly predict the position of the face in the next frame. Exemplarily, the face tracking algorithm of this embodiment is an algorithm that predicts the face frame based on the correlation between the previous and next video frames through the face key point results of the previous frame.

[0070] For the face key point detection algorithm, it is an algorithm for identifying key feature points on the face, such as the positions of eyes, nose, mouth, eyebrows, etc.; face key points are mainly used for applications such as facial expression recognition, face alignment, and three-dimensional modeling. Usually, the face key point detection algorithm can use shape regression methods, such as ASM (Active Shape Model) or AAM (Active Appearance Model), or deep learning-based methods, such as using CNN (Convolutional Neural Network) to directly predict the position of key points. Exemplarily, the face key point detection algorithm used in this embodiment can perform face key point detection on the input face area image based on a deep learning model.

[0071] Step S30, identifying the face frame of the current video frame through the face detection and tracking thread and updating it into the face frame cache, and synchronously reading the face frame in the face frame cache in real time through the face key point detection thread and performing face key point detection.

[0072] In this step, two threads T1 and T2 are run in parallel. The face frame of the current video frame is identified by the face detection and tracking thread and updated to the face frame cache. The face detection algorithm or the face tracking algorithm can be used to identify the face frame from the current video frame according to actual needs. The face frame can be a rectangular frame surrounding the face range. At the same time, the face key point detection thread reads the face frame in the face frame cache in real time, and uses the face key point detection algorithm to combine the face frame and the current video frame F. t To detect the key points of the face.

[0073] In order to make the technical solution of the present application clearer, more embodiments are provided below for the technical solution of the face detection and tracking thread T1 and the face key point detection thread T2.

[0074] In one embodiment, considering that face detection may take longer time than face tracking, face detection may be performed on necessary video frames, and face tracking may be used for other video frames.

[0075] Accordingly, when the face frame of the current video frame is identified by the face detection and tracking thread in step S30, the face detection conditions can be set, and the following judgment and processing steps can be performed:

[0076] When the face detection condition is met, the face detection algorithm running in the face detection tracking thread is used to identify the current video frame F. t The face frame is obtained and updated to the face frame cache; when the face detection condition is not met, the face tracking algorithm running in the face detection and tracking thread is used to identify the current video frame F t The face frame is obtained and updated to the face frame cache.

[0077] As in the scheme of the above-mentioned embodiment, by constructing face detection conditions, on the face detection and tracking thread, when the face detection conditions are met, face detection is performed on the current video frame, otherwise face tracking is performed on the current video frame, thereby reducing the time consumption of face recognition and improving the processing efficiency of the face detection and tracking thread.

[0078] In one embodiment, based on the above-mentioned face detection condition technical solution, the technical solution of identifying the face frame of the current video frame through the face detection tracking thread in step S30 may include the following:

[0079] First, according to the set frame interval, the current video frame F t A target video frame is read from the t Set as face detection frame and other video frames as face tracking frames.

[0080] Then, the face detection algorithm running in the face detection and tracking thread is called to sequentially identify the face frame of the current video frame and update it to the face frame cache; and the face tracking algorithm running in the face detection and tracking thread is called to sequentially identify the face frame of the current video frame and update it to the face frame cache.

[0081] As an embodiment, when calling the face detection algorithm run by the face detection and tracking thread to identify the face frame of the current video frame, the face detection algorithm run by the face detection and tracking thread is called and the current video frame is inferred through a deep learning model to obtain the face frame; when calling the face tracking algorithm run by the face detection and tracking thread to identify the face frame of the current video frame, the face tracking algorithm run by the face detection and tracking thread can be called and the current video frame can be estimated through the face key points identified by the previous video frame to obtain the face frame.

[0082] For example, the time consumption of face detection is higher than that of face tracking. Face detection is called once every S frames. For video frames that do not call face detection, face tracking is called. Face detection uses a deep learning model to perform face detection on the current video frame F. t Perform inference to get the face frame BBox t , face tracking is done by the previous video frame F t-1 The facial key points of the current video frame F t Face frame BBox t Estimate and get the current video frame F t Face frame BBox t , update the global variable FBox to the face frame cache.

[0083] In one embodiment, the face detection condition may be set as a dynamically changing judgment process; accordingly, the judgment on whether the face detection condition is met may be based on the following logic:

[0084] When reading a target video frame from the current video frame according to the set frame interval, when there is a face in the picture, the target video frame is read from the current video frame according to the set frame interval; when there is no face in the picture or the number of faces decreases, the frame interval is reduced to increase the density of reading the target video frame in a short time.

[0085] Exemplarily, when there are faces in the picture, the detection interval is run according to the default detection interval, and face detection is run once every α video frames. For example, α=5, which means that face detection is run once every 5 video frames. When there are no faces in the picture or the number of faces decreases, faces may be lost. At this time, the density of face detection can be increased in a short time, such as shortening the face detection interval to once every γ video frames within β seconds. For example, β=3, γ=3, that is, shortening the face detection interval to once every 3 frames within 3 seconds, thereby ensuring that face detection will not be lost.

[0086] In one embodiment, the technical solution of reading the face frame in the face frame buffer in real time and performing face key point detection by the face key point detection thread in step S30 may include the following:

[0087] When a face frame exists in the face frame cache, the face frame is read from the face frame cache in sequence by the face key point detection thread; and face key points are detected from the current video frame according to the face frame using the face key point detection algorithm run by the face key point detection thread.

[0088] As an embodiment, when facial key points are detected, for each face frame, the facial key point detection algorithm running in the facial key point detection thread is used to infer the current video frame in parallel to obtain the corresponding facial key points; the facial key points are screened and the facial key points with low confidence are removed to obtain the facial key point information of the current video frame.

[0089] For example, the global variable FBox is obtained through the face frame cache. When the global variable FBox contains the face frame BBox t When each face frame FBox i The face key point detection algorithm is performed in parallel for inference, and then the face key point detection results are filtered, the face frames with low confidence are removed, and the face key point information of the current video frame is output.

[0090] In the current video frame F t After input, since the face detection and tracking thread T1 and the face key point detection thread T2 run in parallel, the global variable FBox obtained in the face key point detection thread T2 can be derived from the previous video frame F t-1 , that is, the video frame BBox t-1 Since the FPS of the live video stream is high, at least 25 FPS, the current video frame obtained is usually strongly associated with the previous video frame, that is, the video frame BBox t-1 with video frame BBox t The overlap between the frames is high, and the face detection frame usually has a certain degree of redundancy. Therefore, when the face changes slightly between video frames, the face frame can usually still contain the complete face part, so the face detection frame can be used to detect the face of the previous frame. t-1 To get the current video frame F t The global variable FBox.

[0091] As in the technical solution of the above-mentioned embodiment, by reading the face frame cache in real time and performing face key point detection in parallel, the detection efficiency can be guaranteed. At the same time, based on the high FPS characteristics of the live video stream, face detection adopts an alternate frame call method, and uses the previous video frame to obtain the global variables of the current video frame, thereby greatly reducing the time and calculation amount of the detection process, avoiding the problems of high frame time consumption and frame loss, and improving the efficiency of face key point detection.

[0092] In one embodiment, in order to further optimize the efficiency of facial key point detection, especially in the case of facial key point detection in a multi-person scene, the face frames are prioritized to optimize computing resources, thereby improving detection efficiency.

[0093] Accordingly, the technical solution of step S30 of reading the face frame in the face frame buffer in real time and performing face key point detection through the face key point detection thread may include the following:

[0094] 1) If there are multiple face frames in the current video frame, the importance priority of each face frame is calculated according to the attribute parameters of the face frame and using the evaluation function.

[0095] For example, in the case of facial key point detection in a multi-person scene, the importance of the face can be calculated through an evaluation function based on attribute parameters such as face position and face size; for example, Figure 3 As shown, Figure 3 This is a schematic diagram of facial key point detection in an example multi-person scene. A face frame located in the center of a video frame and having a larger face can be determined as a face frame of higher importance, while a face frame located at the edge of the video frame or having a face that is too small can be determined as a face frame of lower importance.

[0096] 2) When the face frame is of high priority, a high-precision face key point detection model is used to detect face key points of the face frame; when the face frame is of low priority, a low-precision face key point detection model is used to detect face key points of the face frame.

[0097] For example, when the importance of a face is high, multi-level performance optimization and effect optimization can be performed on the function. For example, a face frame with high importance uses an algorithm model with higher precision to detect facial key points, so that more detailed beauty operations and more exquisite beauty rendering can be performed. For faces with low importance, such as those located at the edge of the video frame or are too small, a low-precision algorithm model can be used to reduce the amount of calculation and simplify beauty operations and beauty rendering.

[0098] As in the solution of the above-mentioned embodiment, by calculating the importance priority of the face frame and executing differentiated processing strategies based on the importance priority, the overall performance of the face key point detection process can be optimized, which saves time and does not lose frames, thereby achieving better beauty operations and beauty rendering effects.

[0099] In one embodiment, based on the above-mentioned optimization scheme for facial key point detection, when detecting facial key points of the current video frame, a quantity limit can also be set for the number of face frames; accordingly, when the number of face frames of the current video frame identified by the face detection and tracking thread exceeds a set quantity threshold, low-priority face frames are discarded according to the importance priority of each face frame.

[0100] For example, the upper limit of the number of face frames for detecting facial key points may be limited to 10. When more than 10 faces are recognized in a live video stream, face frames with lower face importance priorities are discarded.

[0101] As in the solution of the above-mentioned embodiment, the number of faces that need to be processed by the face key point detection thread is limited, which can ensure smoothness even in multi-person scenarios and avoid frame loss problems.

[0102] In one embodiment, based on the above optimization scheme for facial key point detection, when detecting facial key points of the current video frame, the face frame processing time may also be limited.

[0103] Accordingly, in step S30, when the face frame in the face frame buffer is read in real time by the face key point detection thread and the face key point detection is performed, the following may be included:

[0104] The face frames in the face frame cache are read in real time through the face key point detection thread; if there are multiple face frames in the current video frame, the face importance parameters of each face frame are calculated, and the maximum time for face key point detection of each face frame is calculated according to the face importance parameters; when the processing time of face key point detection of any face frame exceeds its maximum time, the face frame is discarded, and the face key point information detected by the previous video frame is used.

[0105] Exemplarily, the processing time of a single face frame is dynamically adjusted in combination with the importance parameter of the face frame. Assuming that the importance parameter of the face frame is α, α∈(0,1), the maximum time for processing the face frame is M, and the upper limit calculation formula for the processing time of a single face frame can be F(x)=1 / α×M, that is, the maximum time value F(x). When the processing time of a single face frame exceeds its maximum time value F(x), it is discarded to avoid blocking and causing freezes and frame drops in the live video stream.

[0106] refer to Figure 4 As shown, Figure 4The present invention is a schematic diagram of an exemplary facial key point detection process. Based on the technical solutions of the above-mentioned embodiments, a technical solution for parallel execution of face detection tracking and facial key point detection is designed. After extracting the current video frame, the face detection tracking thread and the face key point detection thread are entered respectively. On the face detection tracking thread, the face detection condition is judged. When the condition is met, the face detection algorithm is called to perform face detection on the current video frame and updated to the face frame cache. Otherwise, the face tracking algorithm is called to perform face tracking on the current video frame and updated to the face frame cache. On the face key point detection thread, the face frame cache is read in real time. When an updated face frame is generated, face key point detection is performed on each face frame in parallel, and face key point information is output. This technical solution performs multiple facial key point detections in parallel and optimizes the facial key point detection process, especially in multi-person live broadcast scenarios, to achieve stable real-time and low frame rate fluctuations. It can meet the facial key point detection needs in multi-person live broadcast scenarios, with strong real-time performance, stable time consumption, and small frame rate fluctuations. In the scenario of multi-person group broadcast on the video live broadcast platform, it can still provide stable facial key point data, realizing the expansion of applicable live broadcast scenarios, such as Figure 5 As shown, Figure 5 This is an example of a live video streaming platform that extends the multi-person beauty mode. By using this mode, the live video streaming platform can enable multiple people to use beauty makeup and headgear special effects at the same time, enriching the live streaming format and content.

[0107] An embodiment of a facial key point detection device is described below.

[0108] like Figure 6 As shown, Figure 6 The structure diagram of a facial key point detection device according to an embodiment of the present invention comprises:

[0109] An extraction module 10, used to extract a current video frame from a live video stream;

[0110] The transmission module 20 is used to transmit the current video frame to the face detection and tracking thread and the face key point detection thread respectively; wherein the face detection and tracking thread runs the face detection algorithm and the face tracking algorithm, and the face key point detection thread runs the face key point detection algorithm;

[0111] The detection module 30 is used to identify the face frame of the current video frame through the face detection and tracking thread and update it to the face frame cache, and synchronously read the face frame in the face frame cache in real time through the face key point detection thread and perform face key point detection.

[0112] The face key point detection device of this embodiment can execute a face key point detection method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the face key point detection device in each embodiment of the present application correspond to the steps in the face key point detection method in each embodiment of the present application. For the detailed functional description of each module of the face key point detection device, please refer to the description in the corresponding face key point detection method shown in the previous text, which will not be repeated here.

[0113] An embodiment of the live broadcast system is described below.

[0114] The live broadcast system of this embodiment, the system framework may include an anchor end, a live broadcast server and an audience end. The audience end and the anchor end may enter the live broadcast platform through a client installed on an electronic device. Exemplarily, the anchor end and the audience end may be computer devices, such as PDAs, smart phones, tablet computers, desktop computers or laptop computers, etc., but there is no limitation to this. They may also be software modules of an application program. The live broadcast server may be implemented with an independent server or a server cluster consisting of multiple servers.

[0115] As in the above-mentioned live broadcast system, the anchor end is used to access the anchor in the live broadcast room and collect the anchor's live video stream and upload it to the live broadcast server; the live broadcast server is used to forward the anchor's live video stream to the audience end, and use the facial key point detection method of any of the above embodiments to detect the facial key points in the live video stream, and use the facial key points to add special effects; such as beauty, makeup, headgear special effects and other functions; the audience end is used to access the audience users in the live broadcast room, receive the anchor's live video stream for playback.

[0116] Exemplarily, in a multi-person live broadcast scenario, when the live broadcast server detects facial key points, face detection tracking and facial key point detection are performed in parallel. For the parallel execution of multiple facial key point detections, face detection can be called in alternate frames. The facial key point detection processing process is optimized, especially in a multi-person live broadcast scenario, to achieve a stable real-time and low frame rate fluctuation effect. This technical solution can meet the needs of facial key point detection in a multi-person live broadcast scenario, with strong real-time performance, stable time consumption, and small frame rate fluctuation. In a multi-person group broadcast scenario, it can provide stable facial key point data, realize the expansion of applicable live broadcast scenarios, reduce the number of live broadcast participants when using beauty makeup and headgear special effects, and enrich the live broadcast format and content.

[0117] Embodiments of electronic devices and computer-readable storage media are described below.

[0118] The present application provides a technical solution for an electronic device to implement functions related to a facial key point detection method. The electronic device of this embodiment includes one or more processors, a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more program configurations are used for the steps of the facial key point detection method of any embodiment.

[0119] like Figure 7 As shown, Figure 7 1 is a block diagram of an exemplary electronic device; the electronic device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc. The electronic device 100 may include one or more of the following components: a processing component 102, a memory 104, a power component 106, a multimedia component 108, an audio component 109, an input / output (I / O) interface 112, a sensor component 114, and a communication component 116.

[0120] The processing component 102 generally controls the overall operation of the electronic device 100, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations.

[0121] The memory 104 is configured to store various types of data to support operations in the electronic device 100, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0122] The power supply component 106 provides power to various components of the electronic device 100 .

[0123] The multimedia component 109 includes a screen that provides an output interface between the electronic device 100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). In some embodiments, the multimedia component 108 includes a front camera and / or a rear camera.

[0124] The audio component 109 is configured to output and / or input audio signals.

[0125] I / O interface 112 provides an interface between processing component 102 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0126] The sensor component 114 includes one or more sensors for providing various aspects of status assessment for the electronic device 100. The sensor component 114 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact.

[0127] The communication component 116 is configured to facilitate wired or wireless communication between the electronic device 100 and other devices. The electronic device 100 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof.

[0128] The present application provides a technical solution of a computer-readable storage medium, which is used to implement functions related to a facial key point detection method. The computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by a processor and executes the facial key point detection method of any embodiment.

[0129] In an exemplary embodiment, the computer-readable storage medium may be a non-temporary computer-readable storage medium including instructions, such as a memory including instructions. For example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0130] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for detecting key points of a face, characterized in that: include: Extract the current video frame from the live video stream; Passing the current video frame to the face detection and tracking thread and the face key point detection thread respectively; wherein the face detection and tracking thread runs the face detection algorithm and the face tracking algorithm, and the face key point detection thread runs the face key point detection algorithm; The face frame of the current video frame is identified through the face detection and tracking thread and updated into the face frame cache, and the face key point detection thread is synchronously used to read the face frame in the face frame cache in real time and perform face key point detection.

2. The method for detecting facial key points according to claim 1, characterized in that: The step of identifying the face frame of the current video frame through the face detection and tracking thread and updating the face frame cache includes: When the face detection condition is met, the face detection algorithm running in the face detection tracking thread is used to identify the face frame of the current video frame and update it to the face frame cache; When the face detection condition is not met, the face tracking algorithm running in the face detection and tracking thread is used to identify the face frame of the current video frame and update it to the face frame cache.

3. The method for detecting facial key points according to claim 2, wherein: The step of identifying the face frame of the current video frame through the face detection and tracking thread and updating the face frame cache includes: Reading a target video frame from the current video frame according to a set frame interval, setting the target video frame as a face detection frame and setting other video frames as face tracking frames; The face detection algorithm running in the face detection and tracking thread is called to sequentially identify the face frame of the current video frame and update it to the face frame cache; and the face tracking algorithm running in the face detection and tracking thread is called to sequentially identify the face frame of the current video frame and update it to the face frame cache.

4. The method for detecting facial key points according to claim 3, characterized in that: The calling of the face detection and tracking thread to run the face detection algorithm to identify the face frame of the current video frame includes: Call the face detection algorithm running in the face detection and tracking thread and use the deep learning model to infer the current video frame to obtain the face frame; The calling of the face detection and tracking thread to run the face tracking algorithm to identify the face frame of the current video frame includes: The face detection and tracking thread is called to run the face tracking algorithm and estimate the face frame of the current video frame through the face key points identified in the previous video frame.

5. The method for detecting facial key points according to claim 3, characterized in that: Read a target video frame from the current video frame according to the set frame interval, including: When there is a face in the picture, the target video frame is read from the current video frame according to the set frame interval; When there is no human face in the picture or the number of human faces decreases, the frame interval is reduced to increase the density of reading the target video frame in a short time.

6. The method for detecting facial key points according to claim 1, characterized in that: The face frame in the face frame buffer is read in real time by the face key point detection thread and the face key point detection is performed, including: When a face frame exists in the face frame cache, the face frame is read from the face frame cache in sequence by the face key point detection thread; The face key points are detected from the current video frame using the face key point detection algorithm run by the face key point detection thread according to the face frame.

7. The method for detecting facial key points according to claim 6, wherein: The method of detecting facial key points from the current video frame using a facial key point detection algorithm executed by a facial key point detection thread according to the facial frame includes: For each face frame, the face key point detection algorithm running in the face key point detection thread is used to infer the current video frame in parallel to obtain the corresponding face key points; The facial key points are screened and the facial key points with low confidence are removed to obtain the facial key point information of the current video frame.

8. The method for detecting facial key points according to any one of claims 1 to 7, characterized in that: The face key point detection thread reads the face frame in the face frame buffer in real time and performs face key point detection, including: If there are multiple face frames in the current video frame, the importance priority of each face frame is calculated according to the attribute parameters of the face frame and the evaluation function; When the face frame is of high priority, a high-precision face key point detection model is used to detect the face key points of the face frame; When the face frame has a low priority, a low-precision face key point detection model is used to detect face key points on the face frame.

9. The method for detecting key points of a face according to claim 8, characterized in that: Also includes: When the number of face frames of the current video frame identified by the face detection and tracking thread exceeds a set number threshold, low-priority face frames are discarded according to the importance priority of each face frame.

10. The method for detecting facial key points according to any one of claims 1 to 7, characterized in that: The face key point detection thread reads the face frame in the face frame buffer in real time and performs face key point detection, including: Reading the face frame in the face frame buffer in real time through the face key point detection thread; If there are multiple face frames in the current video frame, calculate the face importance parameters of each face frame, and calculate the maximum time for each face frame to perform face key point detection according to the face importance parameters; When the processing time for facial key point detection of any face frame exceeds its maximum time, the face frame is discarded and the facial key point information detected by the previous video frame is used.

11. A facial key point detection device, characterized in that: include: An extraction module, used to extract the current video frame from the live video stream; A transfer module, used to transfer the current video frame to a face detection and tracking thread and a face key point detection thread respectively; wherein the face detection and tracking thread runs a face detection algorithm and a face tracking algorithm, and the face key point detection thread runs a face key point detection algorithm; The detection module is used to identify the face frame of the current video frame through the face detection and tracking thread and update it to the face frame cache, and synchronously read the face frame in the face frame cache in real time through the face key point detection thread and perform face key point detection.

12. A live broadcast system, characterized in that: include: An anchor terminal, an audience terminal and a live broadcast server; wherein the anchor terminal and the audience terminal are respectively connected to the live broadcast server via a communication network; The anchor terminal is used to access the anchor in the live broadcast room and collect the anchor's live video stream and upload it to the live broadcast server; The live broadcast server is used to forward the live broadcast video stream of the anchor to the audience end, and use the facial key point detection method described in any one of claims 1 to 10 to detect the facial key points in the live broadcast video stream; The audience terminal is used for audience users who access the live broadcast room to receive the live broadcast video stream of the anchor for playback.

13. An electronic device, characterized in that: The electronic device comprises: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the steps of the facial key point detection method according to any one of claims 1-10.

14. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded by the processor and executes the steps of the facial key point detection method described in any one of claims 1-10.

Images