Face key point recognition method and device, and video live broadcast system

Abstract

The application relates to a face key point recognition method and device, a video live broadcast system, computer equipment and a computer readable storage medium. The method comprises the following steps: reading interval frames and detection frames in a video frame; setting a pre-stored tracking frame as a face frame of the interval frame; wherein the tracking frame is calculated based on face key points recognized in a previous video frame; performing face detection on the detection frame to obtain a detection frame, and calculating a face frame of the detection frame according to the detection frame and the pre-stored tracking frame; performing key point recognition on the face frame to obtain face key points of the video frame, calculating a new face frame according to position information of the face key points, and updating the tracking frame. The technical scheme improves the accuracy of face key point recognition of the interval frame on the basis of reducing the calculation amount; is particularly suitable for a video live broadcast scene with high real-time and accuracy requirements, meets the face key point recognition requirements of a beautifying and makeup algorithm, and improves special effect rendering efficiency and live broadcast effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a face key point recognition method and device, a video live streaming system, a computer device and a computer readable storage medium. BACKGROUND

[0002] In a live streaming process, in order to accurately locate a face position and add special effects, face key points need to be detected, thereby providing a basis for live streaming, short video special effect technologies such as beautifying, making up and reshaping.

[0003] Currently, a face key point recognition scheme mainly uses a cascaded technical solution, i.e., face detection is first performed and then face key point recognition is performed. Due to the picture content of a video frame in a complex scene, the performance of a face detection algorithm is reduced. In order to cope with the above problem, the industry proposes a scheme of recognizing a face by using a key frame of a video frame, without recognizing a face for each frame of face image. For example, in a disclosed technology, there is a face tracking scheme based on deep learning. Face detection is performed at an interval frame number. When a face is detected in a current frame, a face candidate box is output. Otherwise, the next frame is taken as a current frame to detect a face candidate box. Then, face key point positioning is performed on the face box image, and an external rectangle frame of the face key point is calculated. An external expansion rectangle frame is expanded to obtain an external expansion rectangle frame as a face candidate box of the next frame, thereby reducing face tracking time.

[0004] Although the above technical solution can reduce the amount of calculation by face detection at an interval frame number, it still cannot meet the face key point recognition requirement of a beautifying and making up algorithm when applied to a video live streaming scene with high real-time and accuracy requirements, thereby affecting the beautifying and making up special effect rendering efficiency in a live streaming scene and affecting the live streaming effect. SUMMARY

[0005] Therefore, it is necessary to provide a face key point recognition method, device, video live streaming system, computer device and computer readable storage medium in view of the above technical problems, to improve the recognition performance of face key points and improve the special effect rendering efficiency of video live streaming.

[0006] In a first aspect, the present application provides a face key point recognition method, comprising:

[0007] reading an interval frame and a detection frame in a video frame;

[0008] setting a pre-stored tracking box as a face box of the interval frame, wherein the tracking box is calculated based on face key points recognized in a previous frame of video frame;

[0009] performing face detection on the detection frame to obtain a detection frame, and calculating a face frame of the detection frame according to the detection frame and a pre-stored tracking frame;

[0010] performing key point recognition on the face frame to obtain face key points of the video frame, calculating a new face frame according to position information of the face key points, and updating the tracking frame.

[0011] In one embodiment, the calculating of the face frame of the detection frame according to the detection frame and the pre-stored tracking frame comprises:

[0012] calculating an intersection-over-union between the detection frame and the pre-stored tracking frame;

[0013] comparing the intersection-over-union with a set threshold, and drawing the face frame of the video frame according to a comparison result and the detection frame and the tracking frame.

[0014] In one embodiment, the drawing of the face frame of the video frame according to the comparison result and the detection frame and the tracking frame comprises:

[0015] if the intersection-over-union is less than a set first threshold, drawing the face frame of the video frame according to a union frame of the detection frame and the tracking frame;

[0016] if the intersection-over-union is greater than a set second threshold, setting the detection frame as the face frame of the video frame.

[0017] In one embodiment, the calculating of the new face frame according to the position information of the face key points comprises:

[0018] obtaining coordinate data of each face key point of the video frame;

[0019] traversing horizontal coordinates and vertical coordinates of all face key points respectively according to the coordinate data, and obtaining minimum and maximum values of the horizontal coordinates and minimum and maximum values of the vertical coordinates;

[0020] drawing the new face frame according to the minimum and maximum values of the horizontal coordinates and the minimum and maximum values of the vertical coordinates.

[0021] In one embodiment, the reading of the interval frames and the detection frames in the video frame comprises:

[0022] setting an interval frame detection number of the interval frame face detection mode according to picture content of the video frame;

[0023] dividing the video frames of the video file into the interval frames and the detection frames according to the interval frame detection number; wherein the detection frames are video frames that are integer multiples of the interval frame detection number, and the interval frames are video frames between the detection frames;

[0024] The interval frames and the detection frames are read in sequence according to frame numbers of the video file.

[0025] In one embodiment, the method for recognizing the face key point further comprises:

[0026] performing face detection on the 0th video frame of the video file to obtain a detection frame, setting the detection frame as a face frame of the 0th video frame, and calculating and saving a tracking frame according to face key points of the 0th video frame.

[0027] In a second aspect, the application provides a device for recognizing face key points, comprising:

[0028] a reading module configured to read interval frames and detection frames in a video frame;

[0029] a tracking module configured to set a pre-stored tracking frame as a face frame of the interval frame, wherein the tracking frame is calculated based on face key points recognized in a previous video frame;

[0030] a detection module configured to perform face detection on the detection frame to obtain a detection frame, and calculate a face frame of the detection frame according to the detection frame and the pre-stored tracking frame;

[0031] a recognition module configured to recognize face key points of the video frame by performing key point recognition on the face frame, calculate a new face frame according to position information of the face key points, and update the tracking frame.

[0032] In a third aspect, the application provides a video live streaming system, comprising a host end, a viewer end and a live streaming server.

[0033] The host end and the viewer end are connected to the live streaming server through a communication network, respectively.

[0034] The host end is configured to access a host in a live streaming room and collect a host live streaming video stream to upload to the live streaming server.

[0035] The live streaming server is configured to perform live streaming forwarding between the host end and the viewer end, and deliver live streaming video to the viewer end, obtain a target face image of the host from the host live streaming video stream, predict face key points of each video frame in the live streaming video by using the method for recognizing face key points according to any one of claims 1-6, and add special effects.

[0036] The viewer end is configured to access a viewer user in the live streaming room and receive and play the live streaming video.

[0037] In a fourth aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the face key point recognition method when executing the computer program.

[0038] In a fifth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the face key point recognition method when executed by a processor.

[0039] The above technical solution optimizes the cascade scheme of face key point recognition, adopts interval detection for the face detection link, uses the tracking frame calculated from the face key points of the previous video frame to obtain the face frame for the interval frame, and uses the detection frame and the tracking frame to calculate the face frame for the detection frame. After the face key points are recognized on the face frame, the tracking frame is updated using the new face frame calculated from the face key points, and the updated tracking frame is used in the subsequent video frames. This technical solution can reduce the calculation amount of the face detection algorithm by using the interval face detection method, thereby improving the real-time performance. The position information of the face key points of each video frame is used to obtain the tracking frame, and the detection frame and the tracking frame are used to calculate the face frame, thereby improving the accuracy of the face key point recognition of the interval frame on the basis of reducing the calculation amount of the algorithm. This technical solution is particularly suitable for video live streaming scenarios with high real-time performance and high accuracy requirements, and can meet the face key point recognition requirements of the beautification and makeup algorithms, thereby improving the beautification and makeup special effect rendering efficiency and the live streaming effect.

[0040] Further, when calculating the face frame of the detection frame, the intersection-over-union between the detection frame and the tracking frame is calculated, and the face frame of the video frame is drawn on the basis of the detection frame and the tracking frame according to the intersection-over-union size. The repeated face frame can be removed and the new face frame can be retained, thereby improving the accuracy of the detection frame used in the current video frame and the tracking frame used in the subsequent video frame, and improving the face key point recognition efficiency and accuracy as a whole.

[0041] Further, when calculating the new face frame, the minimum and maximum values of the vertical and horizontal coordinates of the coordinate data of each face key point of the video frame are used to redraw the new face frame, and the tracking frame is updated using the face frame, thereby improving the performance of the face key point recognition algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 FIG. 1 is a schematic diagram of a cascade scheme of face key point recognition;

[0043] Figure 2 FIG. 2 is a schematic diagram of an example live streaming business application scenario;

[0044] Figure 3Figure 1 is a flowchart of a face key point recognition method according to an embodiment;

[0045] Figure 4 Figure 1 is a flowchart of a face key point recognition method according to an embodiment;

[0046] Figure 5 Figure 1 is a flowchart of a face key point recognition method according to an embodiment;

[0047] Figure 6 Figure 1 is a flowchart of a face key point recognition method according to an embodiment;

[0048] Figure 7 Figure 1 is a flowchart of a face key point recognition method according to an embodiment;

[0049] Figure 8 Figure 1 is a flowchart of a face key point recognition method according to an embodiment;

[0050] Figure 9 Figure 1 is a flowchart of a face key point recognition method according to an embodiment; DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0052] The technical solution provided by the embodiments of the present application is an optimization of the cascade solution of face key point recognition, as shown in Figure 1 Figure 1 Figure 1 is a schematic diagram of a cascade solution of face key point recognition. Face key point recognition needs to pass through two links, respectively calling a face detection algorithm to detect a face frame, and then using a key point recognition algorithm to recognize face key points from the face frame and output to the subsequent process. Since the picture content of each frame of video is different, the face detection algorithm may not have sufficient performance. Therefore, the present application provides a face key point recognition method, which uses an interval detection method for the face detection link to reduce the detection times of the face detection algorithm. In order to meet the real-time and accuracy, the position information of the recognized face key points of each frame is used to calculate a tracking frame, thereby improving the face key point recognition performance.

[0053] The face key point recognition method provided by the present application is particularly suitable for video live application scenarios with high requirements for real-time and accuracy, as shown in Figure 2 Figure 2 ​​Fig. 1 is a schematic diagram of an example live broadcast service application scenario, and the video live broadcast system can include a live broadcast server, a host terminal and a viewer terminal. The host terminal and the viewer terminal perform data communication with the live broadcast server through a communication network, so that the host of the host terminal and the viewer of the viewer terminal can perform real-time network live broadcast. The terminal device of the host terminal and the viewer terminal can be various personal computers, notebook computers, smart phones and tablet computers, but is not limited to this. The live broadcast server can be implemented by an independent server or a server cluster composed of multiple servers.

[0054] The face key point recognition method of the present application is described below. Referring to Fig. 2, Figure 3 Figure 3 Fig. 3 is a flowchart of a face key point recognition method according to an embodiment, which can include the following steps:

[0055] Step S10, reading interval frames and detection frames in a video frame.

[0056] Specifically, the present application adopts an interval face detection method for face detection, according to which the video frames in the video file are divided into interval frames and detection frames. The interval frames refer to video frames that do not need to perform a face detection algorithm, and the detection frames refer to video frames that need to perform a face detection algorithm. Since the interval frames do not perform face detection, a tracking box is set as a face box for face key point recognition of the corresponding video frame.

[0057] In an embodiment, for interval frames and detection frames, first, the interval detection number of the interval face detection method is set according to the picture content of the video frame; then the video frames of the video file are divided into interval frames and detection frames according to the interval detection number; wherein the detection frames are video frames that are an integer multiple of the interval detection number, and the interval frames are video frames between the detection frames; finally, the interval frames and the detection frames are read in order according to the frame number sequence of the video file.

[0058] In the above-mentioned embodiment, the interval detection number SF of the interval face detection method can be set according to the complexity of the picture content of the current video file. Generally, the interval detection number SF ranges from 5 to 24. When the picture content is relatively complex, the interval detection number can take a small value, and when the picture content is relatively simple, the interval detection number can take a large value. Preferably, the interval detection number SF = 12; that is, the face detection algorithm is executed once every 12 video frames to detect the face box. The interval detection number SF is 12, which can effectively improve the performance of face key point recognition and provide a new face detection box when the tracking box cannot be accurately tracked or lost, thereby ensuring the effect of face key point recognition.

[0059] ​Since the 0th video frame of the video file must be detected for face frame, it can be considered as a detection frame, but since there is no pre-stored tracking frame for the last frame, the processing method for the 0th video frame of the video file is different from that of other detection frames, and the face detection algorithm and the key point detection algorithm processing flow need to be executed at the same time.

[0060] In step S20, the pre-stored tracking frame is set as the face frame of the interval frame; wherein the tracking frame is calculated based on the face key points identified in the last video frame.

[0061] In this step, if the current video is an interval frame, the pre-stored tracking frame is set as the face frame of the interval frame, and the tracking frame is calculated based on the face key points identified in the last video frame, which can be calculated and saved after the face key points are identified in each video frame.

[0062] In step S30, face detection is performed on the detection frame to obtain a detection frame, and the face frame of the detection frame is calculated based on the detection frame and the pre-stored tracking frame.

[0063] In this step, if the current video is a detection frame, i.e. the frame number is an integer multiple of the interval detection number, the face detection algorithm is called to perform face detection on the video frame to obtain a detection frame, and then the face frame of the detection frame is calculated based on the detection frame and the tracking frame calculated based on the face key points identified in the last video frame.

[0064] In one embodiment, the method for calculating the face frame of the detection frame can include the following:

[0065] (1) Calculate the intersection over union (IOU) between the detection frame and the pre-stored tracking frame; specifically, the IOU is the ratio of the intersection to the union between the detection frame and the tracking frame, so that the accuracy of the tracking frame can be verified.

[0066] (2) Compare the set threshold with the IOU, and draw the face frame of the video frame according to the comparison result and the detection frame and the tracking frame.

[0067] Preferably, if the IOU is less than a set first threshold, the face frame of the video frame is drawn according to the union frame of the detection frame and the tracking frame; if the IOU is greater than a set second threshold, the detection frame is set as the face frame of the video frame.

[0068] For example, for the comparison result, refer to Figure 4 , and Figure 4 is a schematic diagram of the comparison result and the drawn face frame, which can include the following:

[0069] ① the detection frame and the tracking frame have no intersection;

[0070] ② the detection frame and the tracking frame have intersection, and the intersection-over-union is less than a first threshold value;

[0071] ③ the detection frame and the tracking frame have intersection, and the intersection-over-union is greater than a second threshold value;

[0072] Based on the three comparison results, the face frame scheme corresponding to the drawing can be as follows:

[0073] As shown in Figure 4 , when the comparison result is ① and ②, the union of the detection frame and the tracking frame is used to draw the face frame of the video frame; when the comparison result is ③, the tracking frame is removed, and the detection frame is set as the face frame of the video frame; generally, the value range of the first threshold value and the second threshold value is 0.18 to 0.25, which can be set according to the actual situation, and preferably, the first threshold value can be 0.2, and the second threshold value can also be 0.2 or a higher value; when the first threshold value and the second threshold value are 0.2, redundant face frames can be effectively filtered, and new face frames can be retained.

[0074] In an embodiment, for the 0th video frame of the video file, face detection is needed to obtain the detection frame, and since there is no pre-stored tracking frame of the previous frame, the detection frame is directly set as the face frame of the 0th video frame.

[0075] As the technical scheme of the above embodiment, when calculating the face frame of the detection frame, the intersection-over-union between the detection frame and the tracking frame is calculated, and the face frame of the video frame is drawn based on the detection frame and the tracking frame according to the size of the intersection-over-union, which can remove redundant face frames and retain new face frames, thereby improving the accuracy of the detection frame used in the current video frame and the tracking frame used in the subsequent video frame, and improving the face key point recognition efficiency and accuracy as a whole.

[0076] In step S40, the face key points of the video frame are obtained by performing key point recognition on the face frame, a new face frame is calculated according to the position information of the face key points, and the tracking frame is updated.

[0077] In this step, the face key point information of the video frame is obtained by calling a key point algorithm to process the face frame obtained in the preceding interval frame or detection frame, and is output to a subsequent process for use; meanwhile, a new face frame is calculated using the position information of the face key points of the current video frame, and the tracking frame is updated using the new face frame, which is used in the face key point recognition process of the next video frame.

[0078] In an embodiment, the drawing method of the new face frame in step S40 can be as follows:

[0079] Obtaining coordinate data of each face key point of the video frame; traversing horizontal coordinates and vertical coordinates of all face key points respectively according to the coordinate data to obtain minimum and maximum values of the horizontal coordinates and minimum and maximum values of the vertical coordinates; and drawing a new face frame according to the minimum and maximum values of the horizontal coordinates and the minimum and maximum values of the vertical coordinates.

[0080] As an example, the scheme of the above embodiment is described with reference to Figure 5 , Figure 5 a new face frame drawing diagram; the face key points identified in the current video frame are shown in the diagram, the coordinate data of each face key point is obtained, and then the minimum and maximum values of the horizontal and vertical coordinates are obtained by traversing, as shown in the diagram, each face key point has a coordinate (x i , y i ), traversing the x coordinates of all face key points can obtain the minimum value x min and the maximum value x max of the x coordinates; similarly, traversing the y coordinates of all face key points can obtain the minimum value y min and the maximum value y max of the y coordinates; the above coordinates can be used to recalculate and draw a new face frame.

[0081] As described in the technical scheme of the above embodiment, when calculating a new face frame, the minimum and maximum values of the horizontal and vertical coordinates of the coordinate data of each face key point of the video frame are used to redraw a new face frame, which is used to update the tracking frame, thereby improving the performance of the face key point recognition algorithm.

[0082] In one embodiment, for the 0th video frame of a video file, since there is no pre-stored tracking frame of the previous frame, the tracking frame is empty, and therefore the tracking frame is calculated directly according to the face key points of the 0th video frame and is saved.

[0083] In order to make the technical scheme of the present application more clear, the following describes an example of the cascade scheme of the optimized face key point recognition of the present application.

[0084] As shown in Figure 6 , Figure 6 is an example of a face key point recognition algorithm flowchart, which mainly includes the following steps:

[0085] s1, setting a frame detection number, initializing the tracking frame to be empty;

[0086] s2, reading the current frame video frame;

[0087] s3, determining whether the video frame is the 0th frame or an integer multiple of the frame number, if not, executing s4, if yes, executing s5;

[0088] s4, set the pre-stored tracking frame as the face frame, and execute s9;

[0089] s5, call the face detection algorithm to perform face detection on the video frame to obtain a detection frame;

[0090] s6, determine whether the tracking frame is empty, if yes, execute s7, otherwise execute s8;

[0091] s7, set the detection frame as the face frame, and execute s9;

[0092] s8, calculate the intersection-over-union of the detection frame and the tracking frame, remove the repeated face frame according to the intersection-over-union, and draw a new face frame;

[0093] s9, call the key point algorithm to recognize the obtained face frame to obtain the face key points;

[0094] s10, calculate a new face frame according to the position information of the face key points to update the tracking frame, and output the face key point information.

[0095] In summary, the technical solutions of the above embodiments optimize the cascade scheme of face key point recognition, adopt an interval detection method for the face detection link, and use the tracking frame obtained by the face key points recognized in the previous video frame to obtain the face frame for the interval frame, and use the detection frame obtained by the face detection algorithm and the tracking frame to calculate the face frame for the detection frame. After recognizing the face key points on the face frame, the tracking frame is updated by using the new face frame calculated by the face key points, and is continuously used in the subsequent video frames. The interval face detection method can reduce the calculation amount of the face detection algorithm and improve the real-time performance. The position information of the face key points of each video frame recognized is used to obtain the tracking frame, and the detection frame and the tracking frame are used to calculate the face frame, which improves the accuracy of the face key point recognition of the interval frame on the basis of reducing the algorithm calculation amount. It is particularly suitable for video live streaming scenarios with high real-time and accuracy requirements, meets the face key point recognition requirements of beauty and makeup algorithms, and improves the beauty and makeup special effect rendering efficiency and live streaming effect.

[0096] It should be understood that although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0097] Based on the same inventive concept, the present application also provides a device for implementing the above-mentioned related method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more related device embodiments provided below can refer to the limitations of the related method described above, which will not be repeated here.

[0098] Reference Figure 7 As shown, Figure 7 is a structure diagram of a face key point recognition device of an embodiment, and the device comprises:

[0099] The reading module 10 is configured to read interval frames and detection frames in a video frame.

[0100] The tracking module 20 is configured to set a pre-stored tracking frame as a face frame of the interval frame, wherein the tracking frame is calculated based on the face key points recognized in the previous frame of the video frame.

[0101] The detection module 30 is configured to perform face detection on the detection frame to obtain a detection frame, and calculate a face frame of the detection frame according to the detection frame and the pre-stored tracking frame.

[0102] The recognition module 40 is configured to recognize the face key points of the video frame by recognizing the face key points of the face frame, calculate a new face frame according to the position information of the face key points, and update the tracking frame.

[0103] Each module in the above device can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0104] The facial landmark recognition device of this embodiment can execute a facial landmark recognition method provided in the embodiments of this application. The implementation principle is similar. The actions performed by each module in the facial landmark recognition device in each embodiment of this application correspond to the steps in the facial landmark recognition method in each embodiment of this application. For detailed functional descriptions of each module of the facial landmark recognition device, please refer to the descriptions in the corresponding facial landmark recognition methods shown above. They will not be repeated here.

[0105] The following describes an implementation example of a video live streaming system.

[0106] The video live streaming system provided in this embodiment is based on... Figure 8 As shown, Figure 8 This is a schematic diagram of an example video live streaming system, which includes: a broadcaster's end, a viewer's end, and a live streaming server; wherein the broadcaster's end and the viewer's end are connected to the live streaming server through a communication network.

[0107] For the broadcaster's end, it is used to connect to the live broadcast room and collect the broadcaster's live video stream and upload it to the live broadcast server; for the live broadcast server, it is used to forward the live broadcast between the broadcaster's end and the viewer's end and to send the live video to the viewer's end; the facial key point recognition method of any of the above embodiments is used to predict the facial key points of each video frame in the live video and add special effects; for the viewer's end, it is used to connect to the live broadcast room and receive the live video for playback.

[0108] like Figure 8 As shown, suppose viewers A, B, C... access the live stream through an app client to watch the streamer's live broadcast. When the streamer needs to use beauty filters, makeup effects, or other special effects, a facial landmark recognition method is called to predict the facial landmarks in each frame of the live video and add effects, such as 2D and 3D facial landmarks. At this time, the streamer can upload the video stream to the live streaming server through their client. The live streaming server then calls the facial landmark recognition method to predict the facial landmarks and accurately adds various beauty filters and makeup effects to the face according to the requirements. The live streaming server can then generate a live video stream with added effects and distribute it to the clients of each viewer A, B, C... for playback. Since the aforementioned video live streaming system uses the facial key point recognition method of this application to recognize facial key points, it can realize real-time and efficient beautification and other special effects in live streaming, improve the appearance of the anchor, enrich the gameplay in the live streaming scene, enhance the fun of the live streaming, attract new users and improve user retention rate, increase the daily active users of the online live streaming product, and better serve the video live streaming business.

[0109] Embodiments of a computer device and a computer readable storage medium of the present application are described below.

[0110] Reference Figure 9 As shown, Figure 9 is a schematic diagram of a computer device structure, which can be a device of a live server application, or a device of a viewer end and anchor end application. The computer device comprises a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store face image data sets and other data. The network interface of the computer device is configured to connect with external devices through a communication network. The computer program is executed by the processor to implement the related method provided by the embodiments of the present application.

[0111] Those skilled in the art can understand that the computer device structure provided by the above embodiments is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] The application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in the method of the above embodiments. A person of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above embodiments. Any reference to a memory, a database or other medium in the embodiments provided by the application can include at least one of a non-volatile memory and a volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., and is not limited thereto. The processor involved in the embodiments provided by the application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., and is not limited thereto.

[0113] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the application are information and data authorized by the user or authorized by all parties.

[0114] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0115] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for recognizing facial key points, characterized in that, Used to predict facial landmarks in each frame of a live video and add special effects, including: Read the interval frames and detection frames in the video frame; The pre-stored tracking box is set as the face box of the interval frame; wherein, the tracking box is calculated based on the facial key points identified in the previous video frame; The process involves performing face detection on the detection frame to obtain a detection bounding box, and calculating the face bounding box of the detection frame based on the detection bounding box and a pre-stored tracking bounding box. This includes: calculating the intersection-union ratio (IUGR) between the detection bounding box and the pre-stored tracking bounding box; comparing the IUGR with a set threshold; if the IUGR is less than a set first threshold, drawing the face bounding box of the video frame based on the union of the detection bounding box and the tracking bounding box; and if the IUGR is greater than a set second threshold, setting the detection bounding box as the face bounding box of the video frame. The face frame is identified by key point recognition to obtain the face key points of the video frame. A new face frame is calculated based on the position information of the face key points, and the tracking frame is updated.

2. The method for recognizing facial key points according to claim 1, characterized in that, Calculate a new face bounding box based on the location information of the facial key points, including: Obtain the coordinate data of each facial key point in the video frame; Based on the coordinate data, traverse the x and y coordinates of all facial key points respectively, and obtain the minimum and maximum values ​​of the x and y coordinates. Draw a new face bounding box based on the minimum and maximum values ​​of the horizontal and vertical coordinates.

3. The method for recognizing facial key points according to claim 1, characterized in that, Read the interval frames and detection frames in the video frame, including: The number of frames to be detected in the frame-by-frame face detection method is set according to the content of the video frames. The video frames of the video file are divided into interval frames and detection frames according to the number of interval detections; wherein, the detection frames are video frames that are integer multiples of the number of interval detections, and the interval frames are video frames between the detection frames; The interval frames and detection frames are read sequentially according to the frame number order of the video file.

4. The method for recognizing facial key points according to claim 2, characterized in that, Also includes: Perform face detection on the 0th frame of the video file to obtain a detection box, and set the detection box as the face box of the 0th frame of the video file; And calculate and save the tracking box based on the facial key points of the 0th video frame.

5. A facial landmark recognition device, characterized in that, Used to predict facial landmarks in each frame of a live video and add special effects, including: The reading module is used to read interval frames and detection frames from video frames. The tracking module is used to set the pre-stored tracking box as the face box of the interval frame; wherein the tracking box is calculated based on the facial key points identified in the previous video frame; A detection module is used to perform face detection on the detection frame to obtain a detection box, and to calculate the face box of the detection frame based on the detection box and a pre-stored tracking box, including: calculating the intersection-union ratio (IUGR) between the detection box and the pre-stored tracking box; comparing the IUGR with a set threshold; if the IUGR is less than a set first threshold, drawing the face box of the video frame based on the union of the detection box and the tracking box; if the IUGR is greater than a set second threshold, setting the detection box as the face box of the video frame. The recognition module is used to perform key point recognition on the face bounding box to obtain the face key points of the video frame, calculate a new face bounding box based on the position information of the face key points, and update the tracking box.

6. A video live streaming system, characterized in that, include: The broadcaster's client, the viewer's client, and the live streaming server; The broadcaster's terminal and the viewer's terminal are respectively connected to the live streaming server through a communication network; The broadcaster's client is used to connect to the broadcaster in the live broadcast room and to collect the broadcaster's live video stream and upload it to the live broadcast server. The live streaming server is used to forward live streams between the broadcaster and the viewer and to send live video to the viewer; to obtain the target face image of the broadcaster from the live video stream, and to predict the face key points of each video frame in the live video using the face key point recognition method described in any one of claims 1-4 and to add special effects. The viewer terminal is used to connect to the live broadcast room and to receive and play the live video.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the facial key point recognition method according to any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the facial key point recognition method according to any one of claims 1-4.

Images