Gesture recognition method, apparatus, and storage medium

Abstract

To provide a gesture recognition method, an apparatus, and a storage medium, which improve gesture recognition accuracy of an object person and reduce an effect on a recognition result due to an individual difference, when the profile of an object person faces a video capture device.SOLUTION: A gesture recognition method includes: sequentially extracting a plurality of image frames from a video stream including the image frame including the head of an object person; filtering the plurality of image frames on the basis of a head gesture model to acquire a plurality of recognition frames satisfying a predetermined requirement; calculating a ratio of the nose to the mouth equal to a ratio of the height of the nose to the height of the mouth of the object person in each recognition frame for each of the plurality of recognition frames; and determining whether or not the object person is performing a yawn action or a talk action on the basis of the ratio of the nose to the mouth in the plurality of recognition frames.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to image recognition, and to a gesture recognition method, device, and storage medium. [Background technology]

[0002] Many application scenarios require judging the concentration of a target. For example, in a distance / online education scenario, it is necessary to determine whether students are talking, yawning, etc., in order to judge whether students are paying attention during class. As another example, in a fatigue driving recognition scenario, it is necessary to recognize whether a driver is engaged in distracting behaviors, such as chatting with the person next to him or yawning, so as to remind the driver to ensure road safety. Summary of the Invention

[0003] The present disclosure relates to a gesture recognition method, an apparatus, and a non-transitory computer-readable storage medium that can improve the accuracy of gesture recognition of a subject when the subject's profile is facing a video capture device, and also reduce the impact of individual differences on the recognition results.

[0004] According to one aspect of the present disclosure, a gesture recognition method is provided, the method including: sequentially extracting a plurality of image frames from a video stream including image frames including a head of a subject, filtering the plurality of image frames based on a head gesture model to obtain a plurality of recognition frames that satisfy a predetermined requirement, calculating a mouth-to-nose ratio for each of the plurality of recognition frames, the mouth-to-nose ratio being equal to a ratio of a mouth height to a nose height of the subject in each recognition frame, and determining whether the subject is performing a yawning or talking behavior based on the mouth-to-nose ratio of the plurality of recognition frames.

[0005] In some embodiments of the present disclosure, determining whether a subject is yawning or speaking based on the mouth-to-nose ratio in multiple recognition frames includes determining whether a subject is yawning or speaking based on a trend of change in the mouth-to-nose ratio in multiple recognition frames.

[0006] According to some embodiments of the present disclosure, determining whether a subject is yawning or talking based on a change trend of the mouth-to-nose ratio of a plurality of recognition frames includes comparing the mouth-to-nose ratios of adjacent recognition frames among the plurality of recognition frames and recognizing one or more recognition intervals based thereon, the recognition interval including a first interval and a second interval immediately following the first interval, in which the mouth-to-nose ratio continues to rise and in which the mouth-to-nose ratio continues to fall, for each recognition interval, determining a difference between a maximum mouth-to-nose ratio and a minimum mouth-to-nose ratio in the first interval as an upward trend value and determining a difference between a minimum mouth-to-nose ratio and a maximum mouth-to-nose ratio in the second interval as a downward trend value, and determining whether a subject is yawning or talking based on the upward trend value and the downward trend value of one or more recognition intervals.

[0007] According to some embodiments of the present disclosure, determining whether the subject is yawning or talking based on the upward trend value and downward trend value of one or more identification intervals includes determining that the subject is yawning in response to an upward trend threshold of a first identification interval of the one or more identification intervals being greater than or equal to a first upward trend threshold and a downward trend value of the first identification interval being less than or equal to the first downward trend threshold.

[0008] According to some embodiments of the present disclosure, determining whether the subject is yawning or talking based on the upward trend value and downward trend value of the one or more identification intervals includes determining that the subject is talking in response to the upward trend value of a first identification interval of the one or more identification intervals being between a first upward trend threshold and a second upward trend threshold and the downward trend value of the first identification interval being between a first downward trend threshold and a second downward trend threshold, where the first upward trend threshold is greater than the second upward trend threshold and the first downward trend threshold is less than the second downward trend threshold.

[0009] According to some embodiments of the present disclosure, determining whether the subject is yawning or talking based on the upward trend values ​​and downward trend values ​​of one or more discrimination intervals includes determining that the subject is talking in response to a number of discrimination intervals whose upward trend values ​​are between a first upward trend threshold and a second upward trend threshold and whose downward trend values ​​are between a first downward trend threshold and a second downward trend threshold being equal to or greater than a threshold number, wherein the first upward trend threshold is greater than the second upward trend threshold and the first downward trend threshold is less than the second downward trend threshold.

[0010] In some embodiments of the present disclosure, determining whether a subject is yawning or speaking based on the mouth-to-nose ratio of a plurality of recognition frames includes comparing the mouth-to-nose ratio of each recognition frame to a first ratio threshold, and determining that the subject is yawning in response to determining that the number of recognition frames whose mouth-to-nose ratios are greater than the first ratio threshold is greater than a first number threshold.

[0011] In some embodiments of the present disclosure, determining whether the subject is yawning or speaking based on the mouth-to-nose ratio of a plurality of recognition frames includes comparing the mouth-to-nose ratio of each recognition frame to a first ratio threshold and a second ratio threshold, and determining that the subject is speaking in response to determining that the number of recognition frames whose mouth-to-nose ratio is greater than the second ratio threshold and less than the first ratio threshold is greater than a second number threshold.

[0012] In some embodiments of the present disclosure, filtering the plurality of image frames based on a head gesture model to obtain a plurality of recognition frames that satisfy a predetermined requirement includes determining a pitch angle of each of the plurality of image frames based on the head gesture model, and filtering the plurality of image frames based on the pitch angle of each image frame.

[0013] In some embodiments of the present disclosure, filtering the plurality of image frames based on the pitch angle of each image frame includes comparing the pitch angle of each image frame with a first head-lift threshold and a second head-lift threshold, and filtering out image frames whose pitch angle is greater than the first head-lift threshold and image frames whose pitch angle is less than the second head-lift threshold based on the comparison result.

[0014] In some embodiments of the present disclosure, filtering the plurality of image frames based on the head gesture model to obtain a plurality of recognition frames that satisfy a predetermined requirement includes determining a yaw angle of each of the plurality of image frames based on the head gesture model; and filtering the plurality of image frames based on the yaw angle of each image frame.

[0015] In some embodiments of the present disclosure, filtering the plurality of image frames based on a yaw angle of each image frame includes comparing the yaw angle of each image frame with a first deflection threshold and a second deflection threshold, and filtering out image frames whose yaw angle is greater than the first deflection threshold and image frames whose yaw angle is less than a second head-lift threshold based on a comparison result.

[0016] According to some embodiments of the present disclosure, the image frames are cropped before filtering to obtain image frames that include only the facial portion of the subject.

[0017] According to another aspect of the present disclosure, there is provided a gesture recognition device, the device including a processor and a memory in which one or more computer programs are stored, the one or more computer programs, when executed by the processor, causing the processor to perform the above-mentioned gesture recognition method.

[0018] According to yet another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the above-mentioned gesture recognition method.

[0019] In the embodiment of the present disclosure, the subject's gesture, i.e., whether the subject is yawning or speaking, can be determined based on the mouth to nose ratio. When the subject is in profile in the frame, the mouth height and nose height are relatively less affected by the profile compared with the mouth width, so the technical solution for determining whether the subject is yawning or speaking is less affected by the profile and improves the recognition accuracy of speaking / yawning with different head gestures. In addition, by considering multiple frames, the impact of individual differences on the recognition result can be further reduced.

[0020] Through the following description of the embodiments with reference to the accompanying drawings, aspects, features, and advantages of the present disclosure will become clearer and more understandable. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 shows a flowchart of a gesture recognition method according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 shows in detail the steps of the gesture recognition method according to FIG. [Diagram 3] FIG. 3 shows in detail the steps of the gesture recognition method according to FIG. [Figure 4] FIG. 4 shows in detail the steps of the gesture recognition method according to FIG. [Diagram 5] FIG. 5 shows in detail the steps of the gesture recognition method according to FIG. [Figure 6] FIG. 6 shows a schematic diagram of the mouth-nose ratio and the change trend of the mouth-nose ratio. [Figure 7] FIG. 7 shows specific details of determining whether a subject is yawning or talking based on the change trend of the mouth-to-nose ratio. [Figure 8] FIG. 8 illustrates diagrammatically the cropping of an image frame. [Figure 9] FIG. 9 illustrates a flow chart of a gesture recognition device according to an embodiment of the present disclosure. [Figure 10] FIG. 10 illustrates a schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be realized in various forms and should not be construed as being limited to the embodiments described herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and do not limit the scope of protection of the present disclosure.

[0023] It should be understood that the steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel, and method embodiments may include additional steps and / or omit certain steps.

[0024] As used herein, "include" and variations thereof are intended to be inclusive, similar to "including but not limited to." The term "based on" means "based at least in part on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms are provided in the description below.

[0025] It should be understood that concepts such as "first", "second", etc. referred to in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of functions performed by these devices, modules or units.

[0026] It should be noted that the modifications "one" and "multiple" referred to in this disclosure are exemplary and not limiting, and should be understood by one of ordinary skill in the art as "one or more" unless the context clearly dictates otherwise.

[0027] As mentioned above, in many application scenarios, it is necessary to judge the concentration of a subject, and recognize whether a subject, such as a student, a driver, or an operator, is speaking, yawning, etc. Current technical solutions are mainly divided into two types. One type of solution is to use a single frame image to calculate the ratio of the subject's mouth height and mouth width, set a threshold, and judge whether the subject is speaking, yawning, etc. based on the single frame image. The disadvantage of this solution is that when the subject's profile is facing the image / video capture device, misjudgment may occur due to inaccuracy of the mouth width data. In addition, because there are large individual differences between subjects (e.g., mouth size, mouth thickness, etc.), it is difficult to set an appropriate threshold to make an accurate judgment. Another type of solution is based on supervised video classification and recognition algorithms. The disadvantage of this solution is that many labels need to be added to the video to generate a training dataset.

[0028] The present application has been made in view of the above problems. The objective of the present disclosure is to provide a gesture recognition method, an apparatus, and a non-transitory computer-readable storage medium that can determine whether a subject is yawning or speaking, determine the subject's gesture based on the mouth-nose ratio, eliminate the influence of profile on gesture judgment, and improve the accuracy of speech / yawn recognition with different head gestures. In addition, the technical solution of the present disclosure can also reduce the influence of individual differences on gesture recognition results, and does not require large-scale labeling of videos.

[0029] 1 shows a flowchart of a gesture recognition method 100 according to an embodiment of the present disclosure. As shown in FIG 1, the gesture recognition method 100 according to an embodiment of the present disclosure includes steps S110 to S140.

[0030] In step S110, multiple image frames can be sequentially extracted from a video stream including an image frame that includes the subject's head. To determine whether the subject is performing a distracting behavior such as yawning, talking, etc., the video stream according to an embodiment of the present disclosure may be a video stream captured of a subject (e.g., a student, a driver, an operator, etc.), in particular a video stream captured of the subject's head. Thus, the video stream can include multiple image frames, each of which includes the subject's head.

[0031] Each image frame in the video stream may be sequentially extracted and analyzed to determine whether the subject is engaged in a distracting behavior such as yawning or speaking. However, instead of analyzing all frames in the video stream, some frames in the video stream may be sequentially extracted and analyzed at a certain frame interval to reduce consumption of computing and storage resources. In one embodiment, step S110 may include sequentially extracting a plurality of image frames from the video stream and analyzing only the extracted image frames, thereby reducing consumption of computing and storage resources. Extraction of the image frames may be performed based on a frame extraction rate.

[0032] In step S120, the multiple image frames extracted in step S110 can be filtered based on the head gesture model to obtain multiple recognition frames that meet the predetermined requirements. The multiple image frames extracted in step S110 may include image frames in which the subject raises or lowers his / her head too much. The technical solution of the present disclosure needs to use the ratio of the subject's mouth height to the nose height in the image frames (hereinafter referred to as the mouth-nose ratio) for gesture recognition, among which, the image frames in which the subject raises or lowers his / her head too much cannot obtain a usable mouth-nose ratio, so these image frames need to be filtered, thereby obtaining image frames that meet the predetermined requirements, herein referred to as the recognition frames.

[0033] According to an embodiment of the present disclosure, the head gesture model may be used to filter the plurality of image frames extracted in step S110, thereby retaining only the image frames of the front face and the profile face, and filtering out the image frames in which the head is raised or lowered too much. The head gesture model may be various deep learning models for gesture estimation known in the art, such as RepNet, HRNet, etc., but the present disclosure is not limited thereto. In one embodiment, as shown in FIG. 2, step S120 may further include steps S210 and S220. In step S210, a pitch angle of each of the plurality of image frames extracted in step S110 may be determined based on the head gesture model. The pitch angle may indicate a range in which the subject raises or lowers the head. The pitch angle of each image frame determined by the head gesture model may be within a normalized range of, for example, -99 to +99, respectively. For example, a positive pitch angle may mean raising the head, and a negative pitch angle may mean lowering the head. A larger positive pitch angle represents a larger range of head lift. A smaller negative pitch angle represents a larger range of head lowering. +99 represents the maximum range of the subject's head lift. -99 represents the maximum range of the subject's head lowering. In step S220, the multiple image frames extracted in step S110 can be filtered based on the pitch angle of each image frame. Based on the pitch angle of the image frame, the range of the subject's head lift or head lowering can be determined, and image frames with too much head lift or head lowering can be filtered.

[0034] In one embodiment, a threshold range can be set for the pitch angle such that only image frames whose pitch angle is within the threshold range are retained. Thus, as shown in FIG. 3, step S220 can further include steps S310 and S320. In step S310, the pitch angle of each image frame can be compared with a first head-lifting threshold and a second head-lifting threshold. The first head-lifting threshold and the second head-lifting threshold can be set based on experience or needs. The first head-lifting threshold can represent an upper limit of a head-lifting range beyond which a usable mouth-to-nose ratio cannot be obtained. If a positive pitch angle represents a head-lifting, the first head-lifting threshold can be a positive value. The second head-lifting threshold can represent an upper limit of a head-lowering range beyond which a usable mouth-to-nose ratio cannot be obtained. If a negative pitch angle represents a head-lowering, the second head-lifting threshold can be a negative value. For example, the first head-lifting threshold can be set to +30 and the second head-lifting threshold can be set to -28. The above threshold settings are merely examples, and users can pre-set these thresholds based on their experience and needs, and can also adjust these thresholds in real-time during operation.

[0035] In step S320, based on the comparison result in step S310, image frames whose pitch angle is greater than the first head-raising threshold and image frames whose pitch angle is less than the second head-raising threshold can be filtered out. As described above, the first head-raising threshold can represent the upper limit of the head-raising range, and the second head-lowering threshold can represent the upper limit of the head-lowering range. If these upper limits are exceeded, a usable mouth-to-nose ratio cannot be obtained. Therefore, in step S320, image frames whose pitch angles exceed these upper limits can be filtered out to obtain frontal face image frames or profile face image frames. These filtered image frames meet the predetermined requirements, i.e., the head-raising and -lowering ranges are not too large, and are referred to herein as recognition frames that meet the predetermined requirements.

[0036] The multiple image frames extracted in step S110 may include profile image frames in which the subject turns his / her head excessively to the left or right. As described above, the technical solution of the present disclosure needs to use the subject's mouth-to-nose ratio in the image frames to recognize gestures, and a usable mouth-to-nose ratio cannot be obtained from a profile image frame in which the subject turns his / her head excessively to the left or right. For such profile image frames in which the subject turns his / her head excessively to the left or right, these image frames can be automatically ignored in response to the inability to obtain the mouth-to-nose ratio when calculating the mouth-to-nose ratio in step S130 described later. In addition, since the yaw angle (yaw) of each image frame can also be determined based on the head gesture model, these image frames can also be directly filtered based on the yaw angle in step S120.

[0037] Therefore, in one embodiment, the above step S210 may determine a yaw angle of each of the multiple image frames extracted in step S110 based on the head gesture model. The yaw angle may indicate a range in which the subject turns his / her head to the left or right. The yaw angle of each image frame determined by the head gesture model may be, for example, within a normalized range of -99 to +99, respectively. For example, a positive yaw angle may mean turning the head to the left, and a negative yaw angle may mean turning the head to the right. A larger positive yaw angle represents a larger range in which the head turns to the left. A larger negative yaw angle represents a larger range in which the head turns to the right. Furthermore, the above step S220 may include filtering the multiple image frames extracted in step S110 based on the pitch angle of each image frame. A range in which the subject turns his / her head to the left or right may be determined based on the yaw angle of the image frame, and image frames in which the range in which the head turns to the left or right is too large may be filtered.

[0038] Similar to the pitch angle, a threshold range can also be set for the yaw angle such that only image frames whose yaw angle is within the threshold range are retained. Thus, the above step S310 can further include comparing the yaw angle of each image frame with a first deflection threshold and a second deflection threshold. The first deflection threshold and the second deflection threshold can also be set based on experience or needs. The first deflection threshold can represent an upper limit of a range of head turning to the left beyond which a usable mouth-to-nose ratio cannot be obtained. If a positive yaw angle represents a head turning to the left, the first deflection threshold can be a positive value. The second deflection threshold can represent an upper limit of a range of head turning to the right beyond which a usable mouth-to-nose ratio cannot be obtained. If a negative yaw angle represents a head turning to the right, the second deflection threshold can be a negative value. For example, the first deflection threshold can be set to +50 and the second deflection threshold can be set to -50. The above threshold settings are merely examples, and the user can pre-set these thresholds based on experience and needs, and can also adjust these thresholds in real time during operation. Accordingly, the above step S320 can further include filtering out image frames whose yaw angles are greater than the first deflection threshold and image frames whose yaw angles are smaller than the second deflection threshold based on the comparison result in step S310. As described above, the first deflection threshold can represent the upper limit of the range of turning the head to the left, and the second deflection threshold can represent the upper limit of the range of turning the head to the right. If these upper limits are exceeded, a usable mouth-to-nose ratio cannot be obtained. Therefore, in step S320, image frames whose yaw angles exceed these upper limits can be filtered to obtain frontal face image frames, or profile face image frames whose head turning range does not affect the determination of the usable mouth-to-nose ratio. These filtered image frames meet the predetermined requirements, i.e., the range of turning the head to the left or right is not too large, and are referred to herein as recognition frames that meet the predetermined requirements.

[0039] Speech / yawn recognition can be improved by filtering image frames that show excessive head lift or lowering, or excessive head rotation to the left or right.

[0040] Continuing to refer to FIG. 1, in step S130, a mouth-to-nose ratio is calculated for each of the multiple recognition frames acquired in step S120. As mentioned above, the mouth-to-nose ratio may be equal to the ratio of the subject's mouth height to nose height in each recognition frame. When the subject in the frame is in profile, the mouth height and nose height are relatively less affected by the profile compared with the mouth width, so the technical solution for determining whether the subject is yawning or speaking is less affected by the profile and improves the recognition accuracy of speaking / yawning with different head gestures.

[0041] In step S140, it can be determined whether the subject is yawning or speaking based on the mouth and nose ratios of multiple recognition frames. Compared with the prior art technical solution using a single frame image to determine whether the subject is yawning or speaking, the method according to the embodiment of the present disclosure uses multiple frames to make the determination, and by considering multiple image frames, the influence of individual differences (e.g., mouth size, mouth thickness, etc.) on the determination result can be reduced, and the accuracy of speech / yawn recognition can be improved.

[0042] Since the mouth-to-nose ratio reflects whether the subject has opened his / her mouth and the width of the mouth opening to some extent, it is possible to simply compare the mouth-to-nose ratios of multiple recognition frames with a threshold, and based on this, it is possible to determine whether the subject is yawning. As shown in FIG. 4, in one embodiment, step S140 can further include steps S410 and S420. In step S410, the mouth-to-nose ratio of each recognition frame can be compared with a first ratio threshold. The first ratio threshold may be a mouth-to-nose ratio threshold corresponding to the degree of mouth opening. If the mouth-to-nose ratio is greater than the first ratio threshold, it can be determined that the subject is opening his / her mouth wider and is yawning. The first ratio threshold can be set by a user based on experience. Through the comparison performed in step S410, it can be determined that the number of recognition frames in which the mouth-to-nose ratio is greater than the first ratio threshold. In step S420, if the number of recognition frames in which the mouth-to-nose ratio is greater than the first ratio threshold is greater than a first number threshold, it can be determined that the subject is yawning. The first number threshold may be 1 or more, and can be set by a user based on experience. By setting the first number threshold to be greater than 1, it is possible to determine whether a subject is yawning by comprehensively considering multiple frames, and to avoid misjudgment due to the influence of the shooting angle of a single frame, the distance between the subject and the video capture device in a single frame, etc. on the mouth-to-nose ratio.

[0043] Similarly, whether the subject is speaking can be simply determined based on comparing the mouth-to-nose ratio of multiple recognition frames with a threshold. As shown in FIG. 5, in one embodiment, step S140 can further include steps S510 and S520. In step S510, the mouth-to-nose ratio of each recognition frame can be compared with the above-mentioned first ratio threshold and second ratio threshold. The second ratio threshold is similar to the first ratio threshold in that it is a threshold of the mouth-to-nose ratio corresponding to the degree of mouth opening, but the second ratio threshold is smaller than the first ratio threshold. The first ratio threshold and the second ratio threshold reflect different degrees of mouth opening, respectively, and can be set by a user based on experience. In general, the degree of mouth opening when speaking is smaller than that when yawning, but larger than that when performing non-speech actions (e.g., breathing or licking the corners of the mouth). By setting the first ratio threshold and the second ratio threshold, a mouth opening interval is actually set, and the degree of mouth opening within this interval is considered to correspond to the possibility that the subject is speaking. Through the comparison performed in step S510, the number of recognition frames in which the mouth-to-nose ratio is greater than the second ratio threshold and less than the first ratio threshold can be determined. In step S520, if the number of recognition frames in which the mouth-to-nose ratio is greater than the second ratio threshold and less than the first ratio threshold is greater than the second number threshold, it can be determined that the subject is speaking. The second number threshold may be 1 or more and can be set by a user based on experience. By setting the second number threshold to be greater than 1, it is possible to determine whether the subject is speaking by comprehensively considering multiple frames, and to avoid misjudgment due to the influence of the shooting angle of a single frame, the distance between the subject and the video capture device in a single frame, etc. on the mouth-to-nose ratio.

[0044] In order to further eliminate the influence of individual differences (e.g., mouth size, mouth thickness, etc.) on gesture recognition and improve the accuracy of speech / yawning recognition, the change trend of the mouth-nose ratio can be further considered instead of the mouth-nose ratio itself. Thus, in one embodiment, step S140 can determine whether the subject is yawning or speaking based on the change trend of the mouth-nose ratio of multiple recognition frames. FIG. 6 shows a schematic diagram of the mouth-nose ratio and the change trend of the mouth-nose ratio. As shown in FIG. 6 and as described above, the mouth-nose ratio in this disclosure refers to the ratio between the height of the mouth and the height of the nose. As the subject performs actions such as speaking and yawning, the mouth-nose ratio may change over time. In addition, the subject may move his / her head, such as lifting his / her head, lowering his / her head, or turning his / her neck, and in these cases, the mouth-nose ratio does not change much unless the subject's mouth opening changes.

[0045] 7 shows specific details of determining whether the subject is yawning or talking based on the change trend of the mouth-to-nose ratio. As shown in FIG. 7, in one embodiment, the step of determining whether the subject is yawning or talking based on the change trend of the mouth-to-nose ratio can include steps S710 to S730.

[0046] In step S710, one or more recognition intervals can be recognized based on a comparison between the mouth and nose ratios of adjacent recognition frames among the plurality of recognition frames. Each time a subject speaks or yawns, there is a process in which the mouth and nose ratio first gradually increases (corresponding to the process of opening the mouth when speaking or yawning) and then gradually decreases (corresponding to the process of closing the mouth when speaking or yawning). In step S710, a recognition interval corresponding to the above process can be recognized based on a comparison between the mouth and nose ratios of adjacent recognition frames. Each recognition interval may include a first interval and a second interval immediately following the first interval. In the first interval, the mouth and nose ratio continues to increase. In the second interval, the mouth and nose ratio continues to decrease. Thus, the first interval corresponds to the process in which the mouth and nose ratio gradually increases in the above-mentioned process, and the second interval corresponds to the process in which the mouth and nose ratio gradually decreases in the above-mentioned process.

[0047] A number of methods can be used to recognize the above recognition interval. As an example, one or more local peak values ​​and a local valley value corresponding to each local peak value can be determined based on a comparison between the mouth and nose ratios of adjacent recognition frames among the plurality of recognition frames. Each local peak value can correspond to two local valley values. A recognition interval and a first interval and a second interval in the recognition interval can be determined based on the local peak value and the two local valley values ​​corresponding to the local peak value. As another example, a difference in the mouth and nose ratio between adjacent frames among the plurality of recognition frames can be calculated to obtain a difference sequence, i.e., a change trend sequence of the mouth and nose ratio. Furthermore, one or more difference sets can be determined from the difference sequence, each difference set including a number of consecutive positive differences and a number of consecutive negative differences. Each recognition interval can correspond to one of the above difference sets. A first interval in each recognition interval corresponds to a number of consecutive positive differences in the difference set, and a second interval in each recognition interval corresponds to a number of consecutive negative differences in the difference set.

[0048] In step S720, for each recognition interval, the difference between the maximum mouth-nose ratio and the minimum mouth-nose ratio in the first interval can be determined as an upward trend value, and the difference between the minimum mouth-nose ratio and the maximum mouth-nose ratio in the second interval can be determined as a downward trend value. It can be understood that the maximum mouth-nose ratio in the first interval is the same as the maximum mouth-nose ratio in the second interval. As an example, for each recognition interval, the difference between the one local peak value and a first local valley value of the corresponding two local valley values ​​can be determined as an upward trend value, and the difference between the second local valley value of the two local valley values ​​and the one local peak value can be determined as a downward trend value. As another example, for each recognition interval, the sum of a plurality of consecutive positive differences in the difference set can be determined as an upward trend value, and the sum of a plurality of consecutive negative differences in the difference set can be determined as a downward trend value.

[0049] The upward trend value and the downward trend value can reflect the maximum change range of the mouth opening and closing during the mouth opening and closing process (because the nose height hardly changes). In step S730, it can be determined whether the subject is yawning or talking based on the upward trend value and the downward trend value of one or more recognition sections. Step S730 may be performed based on comparison with various thresholds.

[0050] In the process of yawning, the change range of the mouth opening and the change range of the mouth closing are also relatively large. In one embodiment, a first upward trend threshold and a first downward trend threshold can be set. The first upward trend threshold can be a change trend threshold of the ratio of the mouth to the nose corresponding to the change range of the mouth opening, and the first trend value can be a positive value. The first downward trend threshold can be a change trend threshold of the ratio of the mouth to the nose corresponding to the change range of the mouth closing, and the first downward trend threshold can be a negative value. The first upward trend threshold and the first downward trend threshold can be set by a user based on experience. For example, the first upward trend threshold can be set to 0.36, and the first downward trend threshold can be set to -0.36. The upward trend value of each recognition section can be compared with the first upward trend threshold, and the downward trend value of each recognition section can be compared with the first downward trend threshold. As long as there exists a recognition interval in which the upward trend value is equal to or greater than the first upward trend threshold and the downward trend value is equal to or less than the first downward trend threshold, it can be determined that the subject is opening and closing his / her mouth wide, and therefore that the subject is yawning.

[0051] Generally, the change range of the mouth opening and closing during speaking is smaller than that during yawning, but larger than that during non-speech actions (e.g., breathing, licking the corners of the mouth, etc.). Thus, in one embodiment, in addition to the above-mentioned first upward trend threshold and first downward trend threshold, a second upward trend threshold and a second downward trend threshold can be additionally set, and a section of the change range of the mouth opening and a section of the change range of the mouth closing can be set. Similar to the first upward trend threshold, the second upward trend threshold may be a change trend threshold of the mouth to nose ratio corresponding to the change range of the mouth opening. The second upward trend threshold may be a positive value, and the first upward trend threshold may be larger than the second upward trend threshold. Similar to the first downward trend threshold, the second downward trend threshold may be a change trend threshold of the mouth to nose ratio corresponding to the change range of the mouth closing. The second downward trend threshold may be a negative value, and the first downward trend threshold may be smaller than the second downward trend threshold. The second upward trend threshold and the second downward trend threshold can be set by the user based on experience. For example, the second upward trend threshold can be set to 0.08, and the second downward trend threshold can be set to -0.08. The upward trend value of each recognition interval can be compared with the first upward trend threshold and the second upward trend threshold, and the downward trend value of each recognition interval can be compared with the first downward trend threshold and the second downward trend threshold. As long as there is a recognition interval whose upward trend value is between the first upward trend threshold and the second upward trend threshold and whose downward trend value is between the first downward trend threshold and the second downward trend threshold, it can be determined that the subject is engaged in speaking behavior.

[0052] In one embodiment, to determine whether the subject is speaking, the number of recognition intervals whose upward trend value is between a first upward trend threshold and a second upward trend threshold and whose downward trend value is between a first downward trend threshold and a second downward trend threshold is further considered. Only when the number of such recognition intervals is equal to or greater than the number of thresholds, the subject is deemed to be speaking. The number of thresholds can be set by a user based on experience and needs, so that it is possible to more flexibly determine whether the subject is speaking. By setting the number of thresholds, it is possible to determine whether the subject is speaking by comprehensively considering multiple frames, and to avoid misjudgment due to the influence of the shooting angle of a single frame, the distance between the subject and the video capture device in a single frame, etc. on the ratio of the mouth and nose.

[0053] In order to better utilize the head gesture model and improve the speech / yawn recognition accuracy, in one embodiment, before filtering the plurality of image frames in the above step S120, the plurality of image frames can be cropped to obtain a plurality of cropped image frames including only the facial portion of the subject. Then, in step S130, the plurality of cropped image frames can be filtered based on the head gesture model to obtain a plurality of recognition frames that meet a predetermined requirement. After cropping, information unrelated to the facial portion in the image frames has been removed, so that the recognition of the pitch angle and yaw angle by the head gesture model can be improved, and the speech / yawn recognition accuracy can be improved. FIG. 8 illustrates the cropping of the image frames. Specifically, FIG. 8 illustrates that the image frames include an image of the subject and its environment before cropping, and only an image of the facial portion of the subject after cropping.

[0054] Since the distance between the subject and the video capture device may vary, the size of the subject in different image frames may be different. In one embodiment, after cropping the multiple image frames to obtain multiple cropped image frames, the sizes of the multiple cropped image frames can be further adjusted to unify them to a fixed size, thereby eliminating the influence of the distance between the subject and the video capture device changing over time on the speech / yawn recognition, and further improving the recognition accuracy.

[0055] In the gesture recognition method according to the embodiment of the present disclosure, the subject's gesture, i.e., whether or not the subject is yawning or speaking, can be determined based on the mouth to nose ratio. When the subject in the frame is in profile, the influence of the profile on the mouth height and nose height is relatively small compared to the mouth width, so the technical solution for determining whether the subject is yawning or speaking is less affected by the profile and improves the recognition accuracy of speaking / yawning with different head gestures. In addition, by considering multiple frames, the influence of individual differences on the recognition result can be further reduced.

[0056] An embodiment of the present disclosure further provides a gesture recognition device. Figure 9 shows a schematic diagram of a gesture recognition device 900 according to an embodiment of the present disclosure.

[0057] 9, a gesture recognition device 900 according to an embodiment of the present disclosure includes a processor 910 and a memory 920. The memory 920 stores one or more computer programs.

[0058] The processor 910 is a program-controlled device such as a microprocessor, and operates, for example, according to a program installed in the memory 920. The memory 920 is, for example, a storage element such as a ROM or a RAM. The program executed by the processor 910 is stored in the memory 920. The gesture recognition device 900 shown in FIG. 9 can be used to realize the gesture recognition method disclosed in the present application.

[0059] In the device according to the embodiment of the present disclosure, the subject's gesture, i.e., whether or not the subject is yawning or speaking, can be determined based on the mouth to nose ratio. When the subject in the frame is in profile, the influence of the profile on the mouth height and nose height is relatively small compared to the mouth width, so the technical solution for determining whether the subject is yawning or speaking is less affected by the profile and improves the recognition accuracy of speaking / yawning with different head gestures. In addition, by considering multiple frames, the influence of individual differences on the recognition result can be further reduced.

[0060] An embodiment of the present disclosure further provides a non-transitory computer readable storage medium. Figure 10 shows a schematic diagram of a computer readable storage medium 1000 according to an embodiment of the present disclosure. The computer readable storage medium 1000 stores computer program instructions 1001, which, when executed by a processor, cause the processor to perform a gesture recognition method provided by the embodiment of the present disclosure.

[0061] The entire hardware computing device described in this disclosure or its components may be realized by various suitable hardware means, including but not limited to FPGA, ASIC, SoC, discrete gate or transistor logic, discrete hardware components, or any combination therebetween. The devices, apparatus, methods, and systems according to the present disclosure are not limited to any particular hardware architecture or configuration. The components of the disclosed devices, apparatus, and systems may be separate or integrated, may be combined in different ways, and / or may be replaced or supplemented by other components. It is understood that the present disclosure can be realized in various forms, such as hardware, software, firmware, dedicated processors, or combinations thereof.

[0062] Block diagrams of apparatus, devices, methods, and systems according to the present disclosure are merely examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these circuits, devices, devices, devices, and systems may be connected, arranged, or configured in any manner so long as the desired purpose is achieved.

[0063] In the above description, the present invention has been described based on the embodiment. The embodiment is merely for the purpose of explanation, and it should be understood by those skilled in the art that the components and combinations of the processes of the embodiment can be modified in various ways, and such modifications are also included in the scope of the present invention.

Claims

1. Sequentially extracting a plurality of image frames from a video stream including an image frame that includes a head of a subject; filtering the plurality of image frames based on a head gesture model to obtain a plurality of recognition frames that satisfy a predetermined requirement; calculating, for each of the plurality of recognition frames, a mouth-to-nose ratio that is equal to a ratio of a mouth height to a nose height of the subject in each recognition frame; and determining whether the subject is yawning or speaking based on a mouth-to-nose ratio in the plurality of recognition frames.

2. Determining whether the subject is yawning or talking based on a mouth-to-nose ratio of the plurality of recognition frames includes: The gesture recognition method according to claim 1 , further comprising: determining whether the subject is yawning or talking based on a change trend of a mouth-to-nose ratio in the plurality of recognition frames.

3. Determining whether the subject is yawning or talking based on a change trend of the mouth and nose ratio in a plurality of recognition frames, recognizing one or more recognition intervals based on a comparison between mouth-to-nose ratios of adjacent recognition frames among the plurality of recognition frames, the recognition intervals including a first interval and a second interval immediately following the first interval, the mouth-to-nose ratio continuing to increase in the first interval and the mouth-to-nose ratio continuing to decrease in the second interval; For each recognition section, determining a difference between a maximum mouth-to-nose ratio and a minimum mouth-to-nose ratio in the first section as an upward trend value, and determining a difference between a minimum mouth-to-nose ratio and a maximum mouth-to-nose ratio in the second section as a downward trend value; The gesture recognition method according to claim 2 , further comprising: determining whether the subject is yawning or talking based on the upward trend value and the downward trend value of the one or more recognition intervals.

4. Determining whether the subject is yawning or talking based on the upward trend value and the downward trend value of the one or more recognition intervals includes:

4. The gesture recognition method of claim 3, further comprising: determining that the subject is yawning in response to an upward trend value of a first discrimination interval among the one or more discrimination intervals being equal to or greater than a first upward trend threshold and a downward trend value of the first discrimination interval being equal to or less than a first downward trend threshold.

5. Determining whether the subject is yawning or talking based on the upward trend value and the downward trend value of the one or more recognition intervals includes: determining that the subject is speaking in response to an upward trend value of a first discrimination interval of the one or more discrimination intervals being between a first upward trend threshold and a second upward trend threshold and a downward trend value of the first discrimination interval being between a first downward trend threshold and a second downward trend threshold; The method of claim 3 , wherein the first upward trend threshold is greater than the second upward trend threshold and the first downward trend threshold is less than the second downward trend threshold.

6. Determining whether the subject is yawning or talking based on the upward trend value and the downward trend value of the one or more recognition intervals includes: determining that the subject is speaking in response to a number of recognition intervals in which the upward trend value is between a first upward trend threshold and a second upward trend threshold and the downward trend value is between a first downward trend threshold and a second downward trend threshold being equal to or greater than a threshold number; The method of claim 3 , wherein the first upward trend threshold is greater than the second upward trend threshold and the first downward trend threshold is less than the second downward trend threshold.

7. filtering the plurality of image frames based on the head gesture model to obtain a plurality of recognition frames that satisfy a predetermined requirement, determining a pitch angle for each of the plurality of image frames based on the head gesture model; and filtering the plurality of image frames based on a pitch angle of each image frame.

8. filtering the plurality of image frames based on a pitch angle of each image frame includes: comparing the pitch angle of each image frame to a first head-lift threshold and a second head-lift threshold; 8. The gesture recognition method of claim 7, further comprising: filtering out image frames whose pitch angle is greater than the first head-lifting threshold and image frames whose pitch angle is less than a second head-lifting threshold based on the compared result.

9. A processor; a memory in which one or more computer programs are stored; A gesture recognition device, wherein the one or more computer programs, when executed by the processor, cause the processor to perform the gesture recognition method according to any one of claims 1 to 8.

10. A non-transitory computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the gesture recognition method according to any one of claims 1 to 8.

Images