Face detection method and device, terminal equipment and computer readable storage medium

Abstract

The application relates to the technical field of face detection, and discloses a face detection method, a face detection device, terminal equipment and a computer readable storage medium. The method comprises the following steps: acquiring a target video frame, wherein the target video frame is an image in a video stream; if the target video frame meets a first preset condition, performing a face detection operation on the target video frame to obtain a face detection result of the target video frame; and if the target video frame does not meet the first preset condition, determining a face detection result of a previous video frame of the target video frame as the face detection result of the target video frame. Through the arrangement, when the face detection is performed on the video stream, the face detection does not need to be performed on each image in the video stream, so that the time consumption of the face detection process can be reduced, and the timeliness can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face detection, and particularly relates to a face detection method and device, a terminal device and a computer readable storage medium. BACKGROUND

[0002] Face detection refers to locating a key region position of a face in a given face image. By using an existing face detection algorithm, a face position in a face image can be easily detected. However, for face detection of a video stream, a large amount of time is consumed in face detection of a plurality of frames of images in the video stream, resulting in poor timeliness of the face detection process. SUMMARY

[0003] Therefore, the present application provides a face detection method and device, a terminal device and a computer readable storage medium, which can improve the timeliness of face detection of a video stream.

[0004] A first aspect of the present application provides a face detection method, comprising:

[0005] obtaining a target video frame, the target video frame being one frame of image in a video stream;

[0006] if the target video frame meets a first preset condition, performing a face detection operation on the target video frame to obtain a face detection result of the target video frame; or

[0007] if the target video frame does not meet the first preset condition, determining a face detection result of a previous video frame of the target video frame as the face detection result of the target video frame.

[0008] After obtaining one frame of video frame in a video stream, the present application does not immediately perform face detection on the current video frame, but first judges whether the current video frame meets a preset condition. If the condition is met, the face detection operation is performed on the current video frame to obtain a corresponding face detection result. If the condition is not met, the face detection result of a previous frame of video frame of the current video frame is used as the face detection result of the current video frame, and the face detection operation is not triggered. By such setting, when face detection is performed on the video stream, face detection does not need to be performed on each frame of image in the video stream, so that the time consumption of the face detection process can be reduced and the timeliness can be improved.

[0009] A second aspect of the present application provides a face detection device, comprising:

[0010] a video frame obtaining module configured to obtain a target video frame, the target video frame being one frame of image in a video stream;

[0011] The face detection module is configured to perform a face detection operation on the target video frame to obtain a face detection result of the target video frame if the target video frame satisfies the first preset condition.

[0012] The face detection result acquisition module is configured to determine the face detection result of the previous video frame of the target video frame as the face detection result of the target video frame if the target video frame does not satisfy the first preset condition.

[0013] The third aspect of the embodiments of the present application provides a terminal device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the face detection method provided in the first aspect of the embodiments of the present application when executing the computer program.

[0014] The fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the face detection method provided in the first aspect of the embodiments of the present application.

[0015] The fifth aspect of the embodiments of the present application provides a computer program product, which, when executed on a terminal device, causes the terminal device to execute the face detection method provided in the first aspect of the embodiments of the present application.

[0016] It can be understood that the beneficial effects of the second aspect to the fifth aspect described above can be referred to the related description in the first aspect described above, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flow chart of one embodiment of the face detection method provided by the embodiments of the present application;

[0019] Figure 2 is a flow chart of one embodiment of another face detection method provided by the embodiments of the present application;

[0020] Figure 3 is a flow chart of one embodiment of another face detection method provided by the embodiments of the present application;

[0021] Figure 4 is an operation schematic diagram of the face detection method provided by the embodiments of the present application in an actual application scenario;

[0022] Figure 5 is a structural diagram of one embodiment of a face detection device provided by an embodiment of the present application.

[0023] Figure 6 is a schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following description, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the present application. However, persons skilled in the art will understand that the present application can be practiced without these specific details. In other instances, well-known structures, devices, circuits, and methods have not been described in detail in order to avoid obscuring the present application. In addition, the description of the present application and its accompanying drawings are not meant to limit the present application to the specific examples described. Other embodiments of the present application will be apparent to those skilled in the art from consideration of the description and accompanying drawings.

[0025] Face detection and face key point detection are important components of computer vision applications, and are widely used in various applications such as face verification, facial emotion recognition, human-computer interaction, and facial motion capture. Face key point detection is usually used together with face detection, and the existing deep learning method can easily realize the functions of face detection and face key point detection. However, for face detection in a video stream, a large number of video frame images need to be detected separately, which is very time-consuming and inefficient, resulting in poor timeliness of the face detection process.

[0026] To solve the above problems, the present application provides a face detection method, device, terminal device and computer readable storage medium, which can improve the timeliness of detecting face images in a video stream.

[0027] It should be understood that the execution subject of each method embodiment of the present application can be various types of terminal devices or servers, such as mobile phones, tablet computers, notebook computers, desktop computers, and wearable devices, etc.

[0028] Please refer to Figure 1 , which shows a face detection method provided by an embodiment of the present application, which includes:

[0029] 101, obtaining a target video frame, the target video frame being a frame image in a video stream;

[0030] First, a target video frame is obtained, which is any frame image in a video stream to be executed for face detection.

[0031] 102, judging whether the target video frame meets a first preset condition;

[0032] After the target video frame is acquired, it is determined whether the target video frame satisfies a first preset condition. Specifically, whether the target video frame satisfies the first preset condition can be determined according to image features (for example, pixel point feature values) or attribute information (for example, a video frame sequence number, an image shooting time point, and the like) of the target video frame.

[0033] In an embodiment of the present application, whether the target video frame satisfies the first preset condition can be determined by the following steps:

[0034] If the frame sequence number of the target video frame is divisible by the first value, it is determined that the target video frame satisfies the first preset condition.

[0035] The video stream can be pre-operated on, and each video frame in the video stream can be numbered in time sequence to obtain a frame sequence number corresponding to each video frame. For example, the frame sequence number of the first video frame is 0, which can be recorded as fream_id=0; the frame sequence number of the second video frame is 1, which can be recorded as fream_id=1, and so on. Before performing face detection on the target video frame, the frame sequence number of the target video frame is first acquired, and it is determined whether the frame sequence number is divisible by a first value (for example, 8 or 10, and the like). If yes, it is determined that the target video frame satisfies the first preset condition, and then the face detection operation is performed on the target video frame. As can be seen, by such a setting, face detection can be triggered for every frame of image, thereby reducing the calculation pressure and improving the timeliness of face detection. For example, if the first value is set to 10, face detection is triggered once every 10 frames of image.

[0036] In another embodiment of the present application, whether the target video frame satisfies the first preset condition can be determined by the following steps:

[0037] If a normalized distance between a center point of a face detection frame corresponding to the target video frame and a center point of a minimum frame circumscribing face key points is greater than a first threshold value, it is determined that the target video frame satisfies the first preset condition, and the minimum frame circumscribing face key points is a frame with the smallest area that encloses all face key points.

[0038] The method first acquires a face detection frame and a face key point minimum frame in the target video frame, wherein the face key point minimum frame is a frame that encloses all face key points and has the minimum area, for example, a rectangular frame that encloses the eye, nose, mouth corner, and other face key points and has the minimum area. Then, the normalized distance between the center point of the face detection frame and the center point of the face key point minimum frame is detected. If the normalized distance is greater than a certain set threshold, it is determined that the target video frame satisfies the first preset condition, and the operation of face detection is performed on the target video frame. The reason for such setting is that if the normalized distance is large, it indicates that the face moves too fast or there is a large change relative to the static state. At this time, the face needs to be repositioned to ensure the accuracy of subsequent face key point detection and other operations, that is, the operation of face detection needs to be performed on the target video frame to re-determine the position of the face. If the normalized distance is small, it indicates that the face moves slowly or there is no large change relative to the static state. At this time, the face does not need to be repositioned, so the operation of face detection does not need to be performed on the target video frame. It can be seen that by such setting, face detection can also be triggered for each frame of image, thereby reducing the calculation pressure and improving the timeliness of face detection.

[0039] In addition, the above-mentioned frame number limiting method, normalized distance limiting method, and other possible embodiments can be used alone or in combination, and the present application does not make any limitation in this regard.

[0040] If the target video frame satisfies the first preset condition, step 103 is performed; if the target video frame does not satisfy the first preset condition, step 104 is performed.

[0041] 103, performing face detection operation on the target video frame to obtain the face detection result of the target video frame;

[0042] The target video frame satisfies the first preset condition, and at this time, the face detection operation is performed on the target video frame. Specifically, various face detection algorithms can be executed to obtain the face position in the target video frame, thereby obtaining the face detection result.

[0043] 104, determining the face detection result of the previous video frame of the target video frame as the face detection result of the target video frame.

[0044] The previous video frame of the target video frame can be the previous video frame adjacent to the target video frame, which is not limited herein.

[0045] In a video stream, the face movement between consecutive video frames is often small, and the changes of the expression, illumination and occlusion of the face are very small. Therefore, it is unnecessary to detect the position of the face in each frame, which will cause the face detection algorithm to be triggered to detect in each frame of the video stream, which is very time-consuming and inefficient.

[0046] Therefore, if the target video frame does not satisfy the first preset condition, the face detection is not triggered, and the face detection result of the previous video frame of the target video frame is determined as the face detection result of the target video frame. For example, if the target video frame is the 9th frame in the video stream, the face detection result of the 8th frame in the video stream is taken as the face detection result of the target video frame. In particular, if the target video frame is the first video frame in the video stream, since the previous video frame does not exist, the face detection operation needs to be performed on the target video frame to obtain the corresponding face detection result, regardless of whether the target video frame satisfies the first preset condition.

[0047] After that, the next video frame of the target video frame is obtained, and the same processing process as the target video frame is performed until the entire video stream is processed, and finally the face detection result of the video stream is obtained.

[0048] The embodiment of the present application does not perform face detection on the current video frame immediately after obtaining a video frame in the video stream, but first judges whether the current video frame satisfies a preset condition. If the condition is satisfied, the face detection operation is performed on the current video frame to obtain the corresponding face detection result; if the condition is not satisfied, the face detection result of a video frame before the current video frame is used as the face detection result of the current video frame, and the face detection operation is not triggered. By such setting, when the face detection is performed on the video stream, the face detection does not need to be performed on each frame of the video stream, so the time consumption of the face detection process can be reduced, and the timeliness is improved.

[0049] Please refer to Figure 2 , another face detection method proposed by the embodiment of the present application is shown, which comprises:

[0050] 201, obtaining a target video frame, the target video frame being a frame of image in a video stream;

[0051] 202, judging whether the target video frame satisfies a first preset condition;

[0052] If the target video frame satisfies the first preset condition, step 203 is performed; if the target video frame does not satisfy the first preset condition, step 204 is performed.

[0053] 203, performing a face detection operation on the target video frame to obtain the face detection result of the target video frame;

[0054] 204、determine the face detection result of the previous video frame of the target video frame as the face detection result of the target video frame;

[0055] The previous video frame of the target video frame can be a previous video frame adjacent to the target video frame, which is not limited herein.

[0056] Steps 201-204 are the same as steps 101-104, and specific reference can be made to the related description of steps 101-104. In addition, step 205 is executed after step 203 is executed, and step 205 is also executed after step 204 is executed.

[0057] 205、detect whether the overlap degree of the face detection frame corresponding to the target video frame and the face detection frame corresponding to the previous video frame thereof is greater than a second threshold value;

[0058] In the face detection process of the video stream, due to the influence of face movement, expression or environmental change, the face detection or face key point detection often has a visual effect of shaking, resulting in poor user experience. In order to reduce the phenomenon of visual shaking, after obtaining the face detection result of the target video frame, it can be further detected whether the overlap degree of the face detection frame corresponding to the target video frame and the face detection frame corresponding to the previous video frame thereof is greater than a certain preset threshold value. The overlap degree, that is, the iou, represents the ratio of the intersection area of two regions to the union area of the two regions, and the overlap degree of two face detection frames can represent the correlation between them to a certain extent, that is, the higher the overlap degree of two face detection frames, the higher the correlation between them. Generally, the second threshold value can be set to a value between 0.5 and 0.8. If the overlap degree of the face detection frame corresponding to the target video frame and the face detection frame corresponding to the previous video frame thereof is greater than the second threshold value, step 206 is executed, otherwise step 207 is executed.

[0059] 206、perform filtering processing on the face detection result of the target video frame;

[0060] The overlap degree of two face detection frames is greater than the set threshold value, which indicates that the correlation of the two face detection frames is high, and at this time filtering processing needs to be performed on the face detection result of the target video frame. The essence is to perform smoothing processing on the face detection frame of the target video frame, and the result obtained will be closer to the two face detection frames and more smooth, so as to eliminate certain visual shaking phenomenon in the case of small face change.

[0061] In an embodiment of the present application, performing filtering processing on the face detection result of the target video frame can include:

[0062] (1) obtaining the first position coordinates of the face detection frame corresponding to the target video frame and the second position coordinates of the face detection frame corresponding to the previous video frame thereof;

[0063] (2) According to the first position coordinate, the second position coordinate and a preset filter factor, the position coordinate of the face detection frame of the target video frame after performing the filtering processing is calculated.

[0064] In the filtering processing, the first position coordinate of the face detection frame of the target video frame and the second position coordinate of the face detection frame of the previous video frame of the target video frame can be obtained. The position coordinate of a certain rectangular face detection frame can be represented by the vertex coordinate of the top-left corner and the vertex coordinate of the bottom-right corner (or the vertex coordinate of the top-right corner and the vertex coordinate of the bottom-left corner) of the face detection frame, for example, the first position coordinate can be represented as now_box(n_x1, n_y1, n_x2, n_y2), wherein n_x1 and n_y1 represent the position coordinate of the top-left corner of the face detection frame of the current frame (i.e. the target video frame), and n_x2 and n_y2 represent the position coordinate of the bottom-right corner of the face detection frame of the current frame. The second position coordinate can be represented as pervious_box(p_x1, p_y1, p_x2, p_y2), wherein p_x1 and p_y1 represent the position coordinate of the top-left corner of the face detection frame of the previous video frame of the current frame, and p_x2 and p_y2 represent the position coordinate of the bottom-right corner of the face detection frame of the previous video frame of the current frame.

[0065] Then, a certain preset filter factor (which can be a value between 0 and 1, for example, 0.8) is used to calculate the position coordinate of the face detection frame of the target video frame after performing the filtering processing in combination with the first position coordinate and the second position coordinate, so as to complete the filtering operation. Specifically, assuming that the position coordinate of the face detection frame of the target video frame after performing the filtering processing is new_box(x1, y1, x2, y2), and the filter factor is α, the position coordinate of the face detection frame after filtering can be calculated by using the following formula:

[0066]

[0067] After the position coordinate of the face detection frame after filtering is calculated, the position coordinate of the initial face detection frame of the target video frame is updated by using the position coordinate, i.e. now_box(n_x1, n_y1, n_x2, n_y2) is updated by new_box(x1, y1, x2, y2), so as to complete the filtering processing of the face detection result of the target video frame.

[0068] 207、keeping the face detection result of the target video frame unchanged.

[0069] The overlap degree of the two face detection boxes is small, which indicates that the correlation of the two face detection boxes is not high, and at this time, the visual jitter phenomenon is not easy to occur, so the filtering processing can not be performed, and the position coordinates now_box(n_x1, n_y1, n_x2, n_y2) of the initial face detection box of the target video frame are retained, that is, the face detection result of the target video frame is kept unchanged.

[0070] Compared with the previous embodiment of the present application, the embodiment further detects whether the overlap degree of the face detection box corresponding to the target video frame and the face detection box corresponding to the previous video frame is greater than a certain preset threshold after obtaining the face detection result of the target video frame, and performs filtering processing on the face detection result of the target video frame if yes. By such setting, the face detection box of the target video frame can be smoothed, and certain visual jitter phenomenon is eliminated in the case of small face change.

[0071] Please refer to Figure 3 , which shows another face detection method proposed in the embodiments of the present application, which comprises:

[0072] 301, obtaining a target video frame, the target video frame being an image in a video stream;

[0073] 302, judging whether the target video frame meets a first preset condition;

[0074] If the target video frame meets the first preset condition, step 303 is performed; if the target video frame does not meet the first preset condition, step 304 is performed.

[0075] 303, performing face detection operation on the target video frame to obtain a face detection result of the target video frame;

[0076] 304, determining the face detection result of the previous video frame of the target video frame as the face detection result of the target video frame;

[0077] The previous video frame of the target video frame can be the previous video frame adjacent to the target video frame, which is not limited herein.

[0078] Steps 301-304 are the same as steps 101-104, and the related description of steps 101-104 can be referred to. In addition, step 305 is performed after step 303 is performed, and step 305 is also performed after step 304 is performed.

[0079] 305, judging whether the target video frame meets a second preset condition;

[0080] After obtaining the face detection result of the target video frame, the embodiment of the present application further judges whether the target video frame satisfies another preset condition, so as to determine whether to perform the face key point detection operation on the target video frame. Similarly, whether the target video frame satisfies the second preset condition can also be determined according to the image characteristics (such as pixel feature value) or attribute information (such as video frame sequence number, image shooting time point, etc.) of the target video frame.

[0081] In an embodiment of the present application, whether the target video frame satisfies the second preset condition can be determined by the following steps:

[0082] If the frame sequence number of the target video frame is divisible by the second value, it is determined that the target video frame satisfies the second preset condition.

[0083] Before performing the face key point detection on the target video frame, the frame sequence number of the target video frame is first obtained, and it is judged whether the frame sequence number is divisible by a certain second value (such as 2). If yes, it is determined that the target video frame satisfies the second preset condition, and the face key point detection operation is performed on the target video frame. It can be seen that by such setting, it can be avoided to trigger the face key point detection for each frame of image, so as to reduce the calculation pressure and improve the timeliness of the face key point detection. For example, if the second value is set to 2, the face key point detection operation is triggered once every 2 frames of image.

[0084] In another embodiment of the present application, whether the target video frame satisfies the second preset condition can be determined by the following steps:

[0085] If the image difference between the target video frame and the previous video frame is greater than a third threshold value, it is determined that the target video frame satisfies the second preset condition.

[0086] Specifically, the image difference between the target video frame and the previous video frame thereof can be obtained by subtracting the feature values (e.g., RGB values, grayscale values, etc.) of the corresponding pixel points of the two video frames, and the image difference can be used to represent the difference degree of the contents of the two images. In addition, in order to reduce the calculation amount and improve the processing efficiency of the algorithm, the two video frames can be converted into grayscale images first and then the image difference is calculated. If the image difference between the two video frames is greater than a certain set threshold, it indicates that the image contents of the two video frames have a large difference (correspondingly, the positions of the face key points of the two video frames also have a large difference), and therefore it is necessary to determine the positions of the face key points in the target video frame again, and thus it is necessary to perform the face key point detection operation on the target video frame. If the image difference between the two video frames is small, it indicates that the image contents of the two video frames are approximately the same (correspondingly, the positions of the face key points of the two video frames are also approximately the same), and therefore the face key point detection result of the previous video frame can be directly determined as the face key point detection result of the target video frame, that is, it is not necessary to perform the face key point detection operation on the target video frame, thereby reducing the calculation pressure.

[0087] In addition, the above-mentioned frame number limiting manner, image difference limiting manner and other possible implementation manners can be used alone or in combination, and the present application does not make any limitation in this regard.

[0088] If the target video frame satisfies the second preset condition, step 306 is performed, and if the target video frame does not satisfy the second preset condition, step 307 is performed.

[0089] 306, performing a face key point detection operation on the target video frame to obtain a face key point detection result of the target video frame;

[0090] The target video frame satisfies the second preset condition, and at this time, the face key point detection operation is performed on the target video frame, specifically, various face key point detection algorithms can be performed to obtain the position coordinates of each face key point in the face detection frame corresponding to the target video frame, thereby obtaining the face key point detection result.

[0091] 307, determining the face key point detection result of the previous video frame of the target video frame as the face key point detection result of the target video frame.

[0092] If the target video frame does not satisfy the second preset condition, face key point detection is not triggered, and the face key point detection result of the previous video frame of the target video frame is determined as the face key point detection result of the target video frame. For example, if the target video frame is the ninth frame in the video stream, the face key point detection result of the eighth frame in the video stream is taken as the face key point detection result of the target video frame. In particular, if the target video frame is the first frame in the video stream, since the previous video frame does not exist, face key point detection needs to be performed on the target video frame to obtain the corresponding face key point detection result, regardless of whether the target video frame satisfies the second preset condition.

[0093] After that, visualization processing can be performed on the face key point detection result of the target video frame (for example, the face key points can be drawn out, so that the user can watch the positions of the face key points in the video stream), and the detection result of the current frame (including the face detection result and the face key point detection result) can also be updated to the detection result of the previous frame of the next frame, and then the next frame of the target video frame is obtained, the same processing process as the target video frame is performed, until the entire video stream is processed, and finally the detection result of the video stream is obtained.

[0094] In an embodiment of the present application, after obtaining the face key point detection result of the target video frame, the following steps can also be included:

[0095] If the overlap degree of the face key point bounding box of the target video frame and the face key point bounding box of the previous video frame is greater than a fourth threshold, the face key point detection result of the target video frame is filtered, and the face key point bounding box is the smallest frame that encloses all face key points.

[0096] In order to reduce the phenomenon of visual jitter, after obtaining the face key point detection result of the target video frame, the overlap degree of the face key point bounding box of the target video frame and the face key point bounding box of the previous video frame can be further detected. The overlap degree of the two face key point bounding boxes can represent the correlation between them to a certain extent, that is, the higher the overlap degree of the two face key point bounding boxes, the higher the correlation between them. Generally, the fourth threshold can be set to a value between 0.5 and 0.8.

[0097] If the overlap degree of the two face key point minimum bounding boxes is greater than the fourth threshold, it indicates that the relevance of the two face key point minimum bounding boxes is high, and filtering processing needs to be performed on the face key point detection result of the target video frame. The essence of filtering is to adjust the position coordinates of each face key point in the face key point minimum bounding box corresponding to the target video frame to eliminate certain visual jitter phenomenon.

[0098] If the overlap degree of the two face key point minimum bounding boxes is small, it indicates that the relevance of the two face key point minimum bounding boxes is not high, and visual jitter phenomenon is not easy to occur, so filtering processing can not be performed, and the position coordinates of each face key point in the face key point minimum bounding box corresponding to the target video frame remain unchanged, that is, the face key point detection result of the target video frame remains unchanged.

[0099] Specifically, performing filtering processing on the face key point detection result of the target video frame can include:

[0100] Performing filtering processing on the position coordinates of the target face key point of the target video frame, the distance between the position coordinates of the target face key point in the target video frame and the position coordinates of the target face key point in the previous video frame of the target video frame is less than a fifth threshold.

[0101] When performing filtering processing on the face key point detection result of the target video frame, it is first necessary to determine which face key points in the face key point detection result need to perform filtering processing and which face key points do not need to perform filtering processing. Specifically, the position distance of each face key point in the front and back two video frames can be calculated, and then the face key points with smaller distance are filtered, and the face key points with larger distance are not filtered. For example, there are 5 face key points corresponding to the left eye, the right eye, the nose center, the left corner of the mouth, and the right corner of the mouth. If the position distance of the left eye and the left corner of the mouth in the front and back two video frames is small, the two positions are determined as target face key points, and filtering and jitter elimination processing is performed on the two positions. As for the other three positions, no processing is performed. The specific way of filtering processing can refer to the related description in 206.

[0102] When the position of the face key point in the front and back two video frames changes greatly (that is, the distance is large), in order to ensure the stability when the face moves too fast or the position changes too much, no filtering processing is performed, and the current face key point detection result is directly used as the final result, which can ensure the accuracy. Since the picture is generally disturbed by light, environment, noise, etc., it may have a slight influence on the face key point detection result of the front and back frames. When the position of the face key point in the front and back two video frames changes little (that is, the distance is small), filtering is performed to eliminate part of the influence and improve stability.

[0103] In addition, when calculating the distance, the L2 distance of the two position coordinates can be calculated, assuming that the position coordinates of the left eye key point in the previous video frame of the target video frame are (pre x , pre y ), and the position coordinates of the left eye key point in the target video frame are (now x , now y ), then the L2 distance corresponding to the left eye key point can be calculated according to the following formula:

[0104]

[0105] where image_height represents the height of each video frame image in the video stream.

[0106] Compared with the first two embodiments of the present application, the embodiment further determines whether the target video frame satisfies another preset condition after obtaining the face detection result of the target video frame, so as to determine whether to perform the face key point detection operation on the target video frame. Through such a setting, it can be avoided to trigger the face key point detection for each frame of image in the video stream, so as to reduce the calculation pressure and improve the timeliness of the face key point detection.

[0107] In order to more clearly illustrate the face detection method proposed in the embodiments of the present application, an actual application scenario is shown below.

[0108] The operation schematic diagram of the face detection method provided by the embodiments of the present application in an actual application scenario is shown in Figure 4 . Figure 4 The specific operation steps in the embodiments of the present application are as follows:

[0109] (1) obtaining a frame of video frame image in a video stream, denoted as a current frame;

[0110] (2) detecting whether the frame number of the current frame is divisible by 10, or whether the normalized distance between the center point of the face detection box and the face key point in the current frame is greater than a set threshold; if yes, triggering the face detection module to perform the face detection operation on the current frame to obtain the face detection result of the current frame; if not, not triggering the face detection module, and obtaining the face detection result of the previous frame image of the current frame as the face detection result of the current frame;

[0111] The previous frame image of the current frame can be the previous frame image adjacent to the current frame, which is not limited herein.

[0112] (3) detecting whether the overlap degree of the face detection frame corresponding to the current frame and the face detection frame corresponding to the previous frame image of the current frame is greater than a set threshold; if yes, performing filtering processing on the face detection result of the current frame; if no, keeping the face detection result of the current frame unchanged;

[0113] (4) detecting whether the frame serial number of the current frame is divisible by 2 or whether the difference between the current frame and the previous frame image (after being converted into grayscale) of the current frame is greater than a set threshold; if yes, triggering the face key point detection module to perform face key point detection operation on the current frame to obtain the face key point detection result of the current frame; if no, not triggering the face key point detection module and obtaining the face key point detection result of the previous frame image of the current frame as the face key point detection result of the current frame;

[0114] (5) detecting whether the overlap degree of the face key point outermost frame corresponding to the current frame and the face key point outermost frame corresponding to the previous frame image of the current frame is greater than a set threshold; if yes, performing step (6) judgment, if no, keeping the face key point detection result of the current frame unchanged;

[0115] (6) detecting whether the L2 distance after normalization of each face key point corresponding to the current frame and the previous frame image of the current frame is greater than a set threshold, keeping the result unchanged for the face key point whose L2 distance is greater than the threshold; performing corresponding filtering processing for the face key point whose L2 distance is less than the threshold;

[0116] (7) finally, updating the detection result (including the face detection result and the face key point detection result) of the current frame to the detection result of the previous frame of the next frame image in the video stream, then obtaining the next frame image of the video stream as a new current frame, and repeating the same processing process until the detection result of all video frame images in the video stream is obtained.

[0117] The present application provides an efficient detection method for reducing the shaking of face key points in a video stream. By tracking stable logic, the face detection module and the face key point detection module are reasonably triggered in the entire video stream, the shaking of the face frame and the face key point between frames is stabilized, the algorithm complexity is low, the hardware requirement is low, and the method is easy to implement and use.

[0118] It should be understood that the size of the serial number of each step in the above-mentioned various embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0119] The above mainly describes a face detection method, and a face detection device will be described below.

[0120] Please refer to Figure 5An embodiment of a face detection device in the embodiments of the present application comprises:

[0121] The video frame acquisition module 501 is configured to acquire a target video frame, the target video frame being a frame of image in a video stream.

[0122] The face detection module 502 is configured to perform a face detection operation on the target video frame to obtain a face detection result of the target video frame if the target video frame meets a first preset condition.

[0123] The face detection result acquisition module 503 is configured to determine a face detection result of a previous video frame of the target video frame as the face detection result of the target video frame if the target video frame does not meet the first preset condition.

[0124] In an embodiment of the present application, the face detection device can further comprise:

[0125] The first condition determination module is configured to determine that the target video frame meets the first preset condition if a frame sequence number of the target video frame is divisible by a first value.

[0126] The second condition determination module is configured to determine that the target video frame meets the first preset condition if a normalized distance between a center point of a face detection frame corresponding to the target video frame and a center point of a minimum frame circumscribing face key points is greater than a first threshold value, the minimum frame circumscribing face key points being a frame with the smallest area that encloses all face key points.

[0127] In an embodiment of the present application, the face detection device can further comprise:

[0128] The first filtering module is configured to perform filtering processing on the face detection result of the target video frame if an overlapping degree of the face detection frame corresponding to the target video frame and a face detection frame corresponding to a previous video frame of the target video frame is greater than a second threshold value, the overlapping degree being a ratio of an intersection area of two regions to a union area of the two regions.

[0129] Further, the first filtering module can comprise:

[0130] The position coordinate acquisition unit is configured to acquire a first position coordinate of the face detection frame corresponding to the target video frame and a second position coordinate of the face detection frame corresponding to the previous video frame of the target video frame.

[0131] The position coordinate calculation unit is configured to calculate a position coordinate of the face detection frame corresponding to the target video frame after the filtering processing is performed according to the first position coordinate, the second position coordinate, and a preset filtering factor.

[0132] In an embodiment of the present application, the face detection device can further comprise:

[0133] The facial landmark detection module is used to perform facial landmark detection on the target video frame if the target video frame meets the second preset condition, and obtain the facial landmark detection result of the target video frame.

[0134] The facial landmark detection result acquisition module is used to determine the facial landmark detection result of the previous video frame of the target video frame as the facial landmark detection result of the target video frame if the target video frame does not meet the second preset condition.

[0135] Furthermore, the face detection device may also include:

[0136] The third condition determination module is used to determine that the target video frame satisfies the second preset condition if the frame number of the target video frame is divisible by the second value.

[0137] The fourth condition determination module is used to determine that the target video frame meets the second preset condition if the image difference between the target video frame and its previous video frame is greater than the third threshold.

[0138] In one embodiment of this application, the face detection device may further include:

[0139] The second filtering module is used to perform filtering on the facial key point detection results of the target video frame if the overlap between the minimum bounding box of the facial key points corresponding to the target video frame and the minimum bounding box of the facial key points corresponding to the previous video frame is greater than the fourth threshold. The minimum bounding box of the facial key points is the box that surrounds all facial key points and has the smallest area.

[0140] Furthermore, the second filtering module may include:

[0141] The filtering unit is used to perform filtering on the position coordinates of the target facial key points in the target video frame, wherein the distance between the position coordinates of the target facial key points in the target video frame and the position coordinates in the previous video frame is less than a fifth threshold.

[0142] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figures 1 to 3 The steps of any face detection method are represented.

[0143] This application also provides a computer program product that, when run on a terminal device, enables the terminal device to perform the following functions: Figures 1 to 3 The steps of any face detection method are represented.

[0144] Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of this application. For example... Figure 6As shown, the terminal device 6 of this embodiment includes a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. The processor 60 implements the steps in the above-described embodiments of the face detection method when executing the computer program 62, for example Figure 1 The processor 60 implements the functions of the modules / units in the above-described embodiments of the apparatus when executing the computer program 62, for example Figure 5 The functions of the modules 501 to 503.

[0145] The computer program 62 can be segmented into one or more modules / units, one or more of which are stored in the memory 61 and executed by the processor 60 to complete the present application. One or more modules / units can be a segment of computer program instructions capable of completing a specific function, which is used to describe the execution process of the computer program 62 in the terminal device 6.

[0146] The processor 60 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0147] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard disk or a memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 61 can include both the internal storage unit and the external storage device of the terminal device 6. The memory 61 is used to store computer programs and other programs and data required by the terminal device. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0150] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0151] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0152] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutually can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0154] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0155] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be instructed by a computer program to related hardware to complete. The computer program can be stored in a computer readable storage medium, and the computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0156] The above embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A face detection method, characterized by, The method comprises the following steps: obtaining a target video frame, the target video frame being a frame image in a video stream; if a frame sequence number of the target video frame is divisible by a first value, or a normalized distance between a center point of a face detection frame corresponding to the target video frame and a center point of a minimum frame circumscribing face key points is greater than a first threshold value, it is determined that the target video frame satisfies a first preset condition, the minimum frame circumscribing face key points being a frame with the smallest area that encloses all face key points; if the target video frame satisfies the first preset condition, performing a face detection operation on the target video frame to obtain a face detection result of the target video frame; if the target video frame does not satisfy the first preset condition, determining a face detection result of a previous video frame of the target video frame as the face detection result of the target video frame; after the face detection result of the target video frame is obtained, the method further comprises the following steps: if the target video frame satisfies a second preset condition, performing a face key point detection operation on the target video frame to obtain a face key point detection result of the target video frame; if the target video frame does not satisfy the second preset condition, determining a face key point detection result of a previous video frame of the target video frame as the face key point detection result of the target video frame; wherein if the frame sequence number of the target video frame is divisible by a second value, or an image difference between the target video frame and the previous video frame is greater than a third threshold value, it is determined that the target video frame satisfies the second preset condition.

2. The method of claim 1, wherein, after the face detection result of the target video frame is obtained and before the face key point detection operation on the target video frame is performed, the method further comprises the following steps: if an overlap degree between a face detection frame corresponding to the target video frame and a face detection frame corresponding to the previous video frame is greater than a second threshold value, performing filtering processing on the face detection result of the target video frame, the overlap degree being a ratio of an intersection area of two regions to a union area of the two regions.

3. The method of claim 2, wherein, the filtering processing on the face detection result of the target video frame comprises the following steps: obtaining a first position coordinate of the face detection frame corresponding to the target video frame and a second position coordinate of the face detection frame corresponding to the previous video frame; calculating a position coordinate of the face detection frame corresponding to the target video frame after the filtering processing is performed according to the first position coordinate, the second position coordinate and a preset filtering factor.

4. The method of claim 1, wherein, after the face key point detection result of the target video frame is obtained, the method further comprises the following steps: if an overlap degree between a minimum frame circumscribing face key points corresponding to the target video frame and a minimum frame circumscribing face key points corresponding to the previous video frame is greater than a fourth threshold value, performing filtering processing on the face key point detection result of the target video frame, the minimum frame circumscribing face key points being a frame with the smallest area that encloses all face key points.

5. The method of claim 4, wherein, the filtering processing on the face key point detection result of the target video frame comprises the following steps: The position coordinates of the target face key points of the target video frame are filtered, and a distance between the position coordinates of the target face key points in the target video frame and the position coordinates of the target face key points in a previous video frame of the target video frame is less than a fifth threshold value.

6. A face detection apparatus characterized by comprising: Comprise: A video frame acquisition module is configured to acquire a target video frame, the target video frame being an image in a video stream; A face detection module is configured to, if the target video frame meets a first preset condition, perform a face detection operation on the target video frame to obtain a face detection result of the target video frame; A face detection result acquisition module is configured to, if the target video frame does not meet the first preset condition, determine a face detection result of a previous video frame of the target video frame as the face detection result of the target video frame; A first condition determination module and a second condition determination module are configured to, if a frame number of the target video frame is divisible by a first value, or a normalized distance between a center point of a face detection box corresponding to the target video frame and a center point of a minimum face key point bounding box is greater than a first threshold value, determine that the target video frame meets the first preset condition, the minimum face key point bounding box being a box that encloses all face key points and has a minimum area; A face key point detection module is configured to, if the target video frame meets a second preset condition, perform a face key point detection operation on the target video frame to obtain a face key point detection result of the target video frame; A face key point detection result acquisition module is configured to, if the target video frame does not meet the second preset condition, determine a face key point detection result of a previous video frame of the target video frame as the face key point detection result of the target video frame; A third condition determination module is configured to, if the frame number of the target video frame is divisible by a second value, or an image difference between the target video frame and the previous video frame thereof is greater than a third threshold value, determine that the target video frame meets the second preset condition.

7. A terminal device, characterized by comprising: The terminal device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the face detection method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the face detection method according to any one of claims 1 to 5.

Images