Posture recognition method and device and storage medium

Abstract

The invention relates to a posture recognition method and device and a storage medium. The posture recognition method comprises the following steps: sequentially extracting a plurality of image frames from a video stream comprising an image frame with a head of a target object; filtering the plurality of image frames based on a head posture model to obtain a plurality of recognition frames meeting preset requirements; for each identification frame in the plurality of identification frames, calculating a mouth-nose ratio which is equal to the ratio of the mouth height to the nose height of the target object in each identification frame; and determining whether the target object has a yawning behavior or a speaking behavior based on the mouth-nose ratio of the plurality of recognition frames.

Description

Technical Field

[0001] The present disclosure relates to image recognition, and more particularly to a gesture recognition method, device and storage medium. Background Art

[0002] Many application scenarios require judgments on the target object's concentration. For example, in the scenario of remote online education, it is necessary to identify whether students are talking or yawning to determine whether they are paying attention in class. For another example, in the scenario of fatigue driving recognition, it is necessary to identify whether the driver is chatting with the person next to him, yawning, and other distracting behaviors to remind the driver to concentrate and ensure road safety. Summary of the invention

[0003] The present disclosure relates to a gesture recognition method, device and non-transitory computer-readable storage medium, which can improve the accuracy of gesture recognition for a target object when the side face of the target object faces a video capture device, and can also reduce the influence of individual differences on the recognition result.

[0004] According to one aspect of the present disclosure, a gesture recognition method is provided, which includes: sequentially extracting multiple image frames from a video stream including image frames with a head of a target object; filtering the multiple image frames based on a head gesture model to obtain multiple recognition frames that meet predetermined requirements; calculating a mouth-to-nose ratio for each of the multiple recognition frames, the mouth-to-nose ratio being equal to the ratio of the mouth height to the nose height of the target object in each recognition frame; and determining whether the target object has a yawning behavior or a speaking behavior based on the mouth-to-nose ratios of the multiple recognition frames.

[0005] According to some embodiments of the present disclosure, determining whether the target object has yawning behavior or speaking behavior based on the mouth-to-nose ratio of multiple recognition frames includes: determining whether the target object has yawning behavior or speaking behavior based on the changing trend of the mouth-to-nose ratio of multiple recognition frames.

[0006] According to some embodiments of the present disclosure, determining whether a target object has yawning behavior or speaking behavior based on a changing trend of the mouth-to-nose ratio of multiple recognition frames includes: identifying one or more recognition intervals based on a comparison between the mouth-to-nose ratios of adjacent recognition frames in the multiple recognition frames, the recognition intervals including a first interval and a second interval immediately following the first interval, wherein the mouth-to-nose ratio continuously increases in the first interval, and the mouth-to-nose ratio continuously decreases in the second interval; for each recognition interval, determining the difference between the maximum mouth-to-nose ratio and the minimum mouth-to-nose ratio in the first interval as an increasing trend value, and determining the difference between the minimum mouth-to-nose ratio and the maximum mouth-to-nose ratio in the second interval as a decreasing trend value; and determining whether the target object has yawning behavior or speaking behavior based on the increasing trend value and the decreasing trend value of one or more recognition intervals.

[0007] According to some embodiments of the present disclosure, determining whether a target object has yawning behavior or speaking behavior based on the rising trend value and the falling trend value of one or more recognition intervals includes: in response to the rising trend value of the first recognition interval among the one or more recognition intervals being greater than or equal to the first rising trend threshold, and the falling trend value of the first recognition interval being less than or equal to the first falling trend threshold, determining that the target object has yawning behavior.

[0008] According to some embodiments of the present disclosure, determining whether a target object has yawning behavior or speaking behavior based on the rising trend value and the falling trend value of one or more recognition intervals includes: in response to the rising trend value of a first recognition interval of one or more recognition intervals being between a first rising trend threshold and a second rising trend threshold, and the falling trend value of the first recognition interval being between the first falling trend threshold and the second falling trend threshold, determining that the target object has speaking behavior, wherein the first rising trend threshold is greater than the second rising trend threshold, and the first falling trend threshold is less than the second falling trend threshold.

[0009] According to some embodiments of the present disclosure, determining whether a target object has yawning behavior or speaking behavior based on rising trend values ​​and falling trend values ​​of one or more identification intervals includes: in response to determining that the number of identification intervals whose rising trend values ​​are between a first rising trend threshold and a second rising trend threshold, and whose falling trend values ​​are between the first falling trend threshold and the second falling trend threshold is equal to or greater than a threshold number, determining that the target object has speaking behavior, wherein the first rising trend threshold is greater than the second rising trend threshold, and the first falling trend threshold is less than the second falling trend threshold.

[0010] According to some embodiments of the present disclosure, determining whether a target object is yawning or speaking based on the mouth-to-nose ratio of multiple recognition frames includes: comparing the mouth-to-nose ratio of each recognition frame with a first ratio threshold; in response to determining that the number of recognition frames whose mouth-to-nose ratios are greater than the first ratio threshold is greater than a first number threshold, determining that the target object is yawning.

[0011] According to some embodiments of the present disclosure, determining whether a target object is yawning or speaking based on the mouth-to-nose ratio of multiple recognition frames includes: comparing the mouth-to-nose ratio of each recognition frame with a first ratio threshold and a second ratio threshold; in response to determining that the number of recognition frames whose mouth-to-nose ratios are greater than the second ratio threshold and less than the first ratio threshold is greater than a second number threshold, determining that the target object is speaking.

[0012] According to some embodiments of the present disclosure, filtering multiple image frames based on a head posture model to obtain multiple recognition frames that meet predetermined requirements includes: determining the pitch angle of each image frame in the multiple image frames based on the head posture model; and filtering the multiple image frames based on the pitch angle of each image frame.

[0013] According to some embodiments of the present disclosure, filtering multiple image frames based on the pitch angle of each image frame includes: comparing the pitch angle of each image frame with a first head-up threshold and a second head-up threshold; and filtering out the following image frames based on the comparison: image frames whose pitch angles are greater than the first head-up threshold; and image frames whose pitch angles are less than the second head-up threshold.

[0014] According to some embodiments of the present disclosure, filtering multiple image frames based on a head posture model to obtain multiple identification frames that meet predetermined requirements also includes: determining the yaw angle of each image frame in the multiple image frames based on the head posture model; and filtering the multiple image frames based on the yaw angle of each image frame.

[0015] According to some embodiments of the present disclosure, filtering multiple image frames based on the yaw angle of each image frame includes: comparing the yaw angle of each image frame with a first deflection threshold and a second deflection threshold; and filtering out the following image frames based on the comparison: image frames whose yaw angle is greater than the first deflection threshold; and image frames whose yaw angle is less than the second deflection threshold.

[0016] According to some embodiments of the present disclosure, the multiple image frames are cropped before filtering the multiple image frames to obtain multiple image frames that only include the facial portion of the target object.

[0017] According to another aspect of the present disclosure, a device for gesture recognition is provided, which includes: a processor; and a memory storing one or more computer programs, wherein when the one or more computer programs are executed by the processor, the processor executes the above-mentioned gesture recognition method.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the above-mentioned gesture recognition method.

[0019] According to the embodiments of the present disclosure, the posture of the target object, that is, whether there is yawning or speaking behavior, can be determined based on the mouth-nose ratio. When the target object in the frame is in a side face position, the mouth height and nose height are relatively less affected by the side face than the mouth width, thereby reducing the influence of the side face on the technical solution for determining whether the target object has yawning or speaking behavior, and improving the recognition accuracy of speaking / yawning in different head postures. In addition, by considering multiple frames, the influence of individual differences on the recognition results can also be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The aspects, features and advantages of the present disclosure will become clearer and easier to understand through the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0021] Figure 1 A flowchart of a gesture recognition method according to an embodiment of the present disclosure is shown;

[0022] Figure 2 Shows Figure 1 Details of the steps of the gesture recognition method;

[0023] Figure 3 Shows Figure 1 Details of the steps of the gesture recognition method;

[0024] Figure 4 Shows Figure 1 Details of the steps of the gesture recognition method;

[0025] Figure 5 Shows Figure 1 Details of the steps of the gesture recognition method;

[0026] Figure 6 The trend of the mouth-to-nose ratio and the mouth-to-nose ratio is schematically shown;

[0027] Figure 7 The specific details of determining whether the target object yawns or speaks based on the change trend of the mouth-to-nose ratio are shown;

[0028] Figure 8 The cropping of the image frame is schematically shown;

[0029] Fig. 9 A schematic diagram of a device for gesture recognition according to an embodiment of the present disclosure is shown;

[0030] Fig.10 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0032] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include other steps and / or omit certain steps.

[0033] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0034] It should be understood that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0036] As mentioned above, many application scenarios require judgments on the concentration of the target object, and identification of the target object, such as students, drivers, operators, etc., to see whether they are talking, yawning, etc. The current technical solutions are mainly divided into two categories. One type of solution uses a single-frame image to calculate the ratio of the target object's mouth height to the mouth width, and sets a threshold to judge whether the target object is talking, yawning, etc. based on a single-frame image. The defect of this solution is that when the side face of the target object faces the image / video capture device, misjudgment may occur due to inaccurate mouth width data. In addition, due to the large individual differences of the target objects (for example, mouth size, mouth thickness, etc.), it is difficult to set a suitable threshold to make an accurate judgment. Another type of solution is based on a supervised video classification and recognition algorithm. The defect of this solution is that large-scale annotated videos are required to generate a training data set.

[0037] The present disclosure is made in view of the above problems. The purpose of the present disclosure is to provide a gesture recognition method, device and non-temporary computer-readable storage medium that can determine whether a target object is yawning or speaking, which determines the gesture of the target object based on the mouth-nose ratio, reduces the influence of the side face on gesture judgment, and improves the recognition accuracy of speaking / yawning under different head gestures. In addition, the technical solution according to the present disclosure can also reduce the influence of individual differences on gesture recognition results, and does not require large-scale annotation of videos.

[0038] Figure 1FIG. 1 is a flow chart of a gesture recognition method 100 according to an embodiment of the present disclosure. Figure 1 As shown, the gesture recognition method 100 according to the embodiment of the present disclosure includes steps S110 to S140.

[0039] In step S110, multiple image frames may be sequentially extracted from a video stream including an image frame having a head of the target object. In order to determine whether the target object has distracting behaviors such as yawning and talking, the video stream according to an embodiment of the present disclosure may be a video stream shot for the target object (e.g., a student, a driver, an operator, etc.), especially a video stream shot for the head of the target object. Therefore, the video stream may include multiple image frames, each of which has a head of the target object.

[0040] In order to determine whether the target object has distracting behaviors such as yawning and talking, each image frame in the video stream can be sequentially extracted for analysis. However, in order to reduce the consumption of computing and storage resources, several frames in the video stream can be sequentially extracted at a certain frame interval for analysis instead of analyzing every frame of the video stream. In one embodiment, step S110 may include sequentially extracting multiple image frames in the video stream and only analyzing the extracted image frames, thereby reducing the consumption of computing and storage resources. The extraction of image frames can be performed based on the frame extraction rate.

[0041] In step S120, the multiple image frames extracted in step S110 can be filtered based on the head posture model to obtain multiple recognition frames that meet predetermined requirements. The multiple image frames extracted in step S110 may include image frames in which the target object raises its head or lowers its head too much. Since the technical solution of the present disclosure needs to use the ratio of the mouth height to the nose height of the target object in the image frame (hereinafter referred to as the mouth-to-nose ratio) for posture recognition, and a usable mouth-to-nose ratio cannot be obtained in the image frames in which the target object raises its head or lowers its head too much, it is necessary to filter out these image frames to obtain image frames that meet predetermined requirements, which are referred to as recognition frames herein.

[0042] According to an embodiment of the present disclosure, a head posture model can be used to filter the multiple image frames extracted in step S110, so as to retain only the image frames of the front face and the side face, and filter out the image frames with a large head-up or head-down range. The head posture model can be any deep learning model for posture estimation known in the art, such as RepNet, HRNet, etc., which is not limited in the present disclosure. In one embodiment, as Figure 2As shown, step S120 may further include steps S210 and S220. In step S210, the pitch of each of the multiple image frames extracted in step S110 may be determined based on the head posture model. The pitch angle may represent the amplitude of the target object's head-up or head-down. The pitch angle of each image frame determined by the head posture model may be within a normalized range of, for example, -99 to +99. For example, a positive pitch angle may represent a head-up, and a negative pitch angle may represent a head-down. The larger the positive pitch angle, the larger the head-up amplitude. The smaller the negative pitch angle, the larger the head-down amplitude. +99 may represent the maximum amplitude that the target object can reach when it raises its head. -99 may represent the maximum amplitude that the target object can reach when it lowers its head. In step S220, the multiple image frames extracted in step S110 may be filtered based on the pitch angle of each image frame. The amplitude of the target object's head-up or head-down may be determined based on the pitch angle of the image frame, thereby filtering out image frames with excessive head-up or head-down amplitudes.

[0043] In one embodiment, a threshold range may be set for the pitch angle, so that only image frames whose pitch angles are within the threshold range are retained. Figure 3 As shown, step S220 may further include steps S310 and S320. In step S310, the pitch angle of each image frame may be compared with a first head-up threshold and a second head-up threshold. The first head-up threshold and the second head-up threshold may be set according to experience or demand. The first head-up threshold may represent an upper limit of the head-up amplitude, exceeding which an available mouth-nose ratio may not be obtained. In the case where a positive pitch angle represents a head-up, the first head-up threshold may be a positive value. The second head-up threshold may represent an upper limit of the head-down amplitude, exceeding which an available mouth-nose ratio may not be obtained. In the case where a negative pitch angle represents a head-down, the second head-up threshold may be a negative value. For example, the first head-up threshold may be set to +30, and the second head-up threshold may be set to -28. The above threshold settings are merely examples, and users may set these thresholds in advance according to experience or demand, and may also adjust these thresholds in real time during operation.

[0044] In step S320, the following image frames can be filtered out based on the comparison in step S310: image frames whose pitch angles are greater than the first head-raising threshold; image frames whose pitch angles are less than the second head-raising threshold. As described above, the first head-raising threshold can represent the upper limit of the head-raising amplitude, and the second head-raising threshold can represent the upper limit of the head-lowering amplitude. Exceeding these upper limits, a usable mouth-nose ratio cannot be obtained. Therefore, in step S320, image frames whose pitch angles exceed these upper limits can be filtered out to obtain frontal image frames or side face image frames. These filtered image frames meet the predetermined requirements, that is, the amplitude of raising and lowering the head cannot be too large, and are referred to herein as recognition frames that meet the predetermined requirements.

[0045] The multiple image frames extracted in step S110 may also include side face image frames in which the target object turns its head too far to the left or right. As described above, the technical solution of the present disclosure needs to utilize the mouth-to-nose ratio of the target object in the image frame for posture recognition, and a usable mouth-to-nose ratio cannot be obtained in the image frames in which the target object turns its head too far to the left or right. For these side face image frames in which the target object turns its head too far to the left or right, when calculating the mouth-to-nose ratio in the later-described step S130, these image frames can be automatically ignored in response to the inability to obtain the mouth-to-nose ratio. In addition, since the yaw angle (yaw) of each image frame can also be determined based on the head posture model, these image frames can also be directly filtered out based on the yaw angle in step S120.

[0046] Thus, in one embodiment, the above step S210 may further include determining the yaw angle of each of the multiple image frames extracted in step S110 based on the head posture model. The yaw angle may represent the amplitude of the target object turning its head to the left or right. The yaw angle of each image frame determined by the head posture model may be within a normalized range of, for example, -99 to +99. For example, a positive yaw angle may represent turning the head to the left, and a negative yaw angle may represent turning the head to the right. The larger the positive yaw angle, the larger the amplitude of turning the head to the left. The smaller the negative pitch angle, the larger the amplitude of turning the head to the right. Further, the above step S220 may further include filtering the multiple image frames extracted in step S110 based on the yaw angle of each image frame. The amplitude of the target object turning its head to the left or right may be determined based on the yaw angle of the image frame, thereby filtering out image frames with excessive amplitude of turning the head to the left or right.

[0047] Similar to the pitch angle, a threshold range can also be set for the yaw angle, so that only image frames whose yaw angles are within the threshold range are retained. Therefore, the above step S310 can also include comparing the yaw angle of each image frame with the first deflection threshold and the second deflection threshold. The first deflection threshold and the second deflection threshold can also be set according to experience or demand. The first deflection threshold can represent the upper limit of the amplitude of turning the head to the left, exceeding which the available mouth-nose ratio cannot be obtained. In the case where a positive yaw angle represents turning the head to the left, the first deflection threshold can be a positive value. The second deflection threshold can represent the upper limit of the amplitude of turning the head to the right, exceeding which the available mouth-nose ratio cannot be obtained. In the case where a negative yaw angle represents turning the head to the right, the second deflection threshold can be a negative value. For example, the first deflection threshold can be set to +50, and the second deflection threshold can be set to -50. The above threshold settings are only examples, and users can set these thresholds in advance according to experience or demand, and can also adjust these thresholds in real time during operation. Accordingly, the above step S320 may also include filtering out the following image frames based on the comparison in step S310: image frames whose yaw angles are greater than the first deflection threshold; image frames whose yaw angles are less than the second deflection threshold. As described above, the first deflection threshold may represent the upper limit of the amplitude of turning the head to the left, and the second deflection threshold may represent the upper limit of the amplitude of turning the head to the right. Exceeding these upper limits, it will not be possible to obtain a usable mouth-to-nose ratio. Therefore, in step S320, image frames whose yaw angles exceed these upper limits may be filtered out to obtain frontal image frames or side face image frames whose head turning amplitude does not affect the determination of the usable mouth-to-nose ratio. These filtered image frames meet the predetermined requirements, that is, the amplitude of turning the head to the left or to the right cannot be too large, and are referred to herein as recognition frames that meet the predetermined requirements.

[0048] By filtering out image frames in which the head is raised or lowered too much and / or turned too much to the left or right, the accuracy of speaking / yawning recognition can be improved.

[0049] Continue to refer to Figure 1 In step S130, the mouth-to-nose ratio may be calculated for each of the multiple recognition frames obtained in step S120. As described above, the mouth-to-nose ratio may be equal to the ratio of the mouth height to the nose height of the target object in each recognition frame. When the target object in the frame is in a side face position, the mouth height and the nose height are relatively less affected by the side face than the mouth width, thereby reducing the influence of the side face on the technical solution of determining whether the target object has a yawning behavior or a speaking behavior, and improving the recognition accuracy of speaking / yawning in different head postures.

[0050] In step S140, it is possible to determine whether the target object is yawning or speaking based on the mouth-to-nose ratio of multiple recognition frames. Compared with the technical solution of using a single frame image to determine whether there is a yawning or speaking behavior in the prior art, the method according to the embodiment of the present disclosure uses multiple frames for determination. By considering multiple image frames, the influence of individual differences (such as mouth size, mouth thickness, etc.) on the determination result can be reduced, and the recognition accuracy of speaking / yawning can be improved.

[0051] Since the mouth-to-nose ratio can reflect to a certain extent whether the target object has opened its mouth and the extent of its opening, it is possible to simply determine whether the target object has yawned based on the comparison of the mouth-to-nose ratios of multiple recognition frames with a threshold. Figure 4 As shown, in one embodiment, step S140 may further include steps S410 to S420. In step S410, the mouth-to-nose ratio of each recognition frame may be compared with a first ratio threshold. The first ratio threshold may be a threshold value for the mouth-to-nose ratio corresponding to the degree of mouth opening. When the mouth-to-nose ratio is greater than the first ratio threshold, it can be determined that the target object has a large mouth opening degree and may have yawning behavior. The first ratio threshold may be set by the user based on experience. By comparing in step S410, the number of recognition frames whose mouth-to-nose ratio is greater than the first ratio threshold may be determined. In step S420, if the number of recognition frames whose mouth-to-nose ratio is greater than the first ratio threshold is greater than the first number threshold, it may be determined that the target object has yawning behavior. The first number threshold may be equal to or greater than 1 and may be set by the user based on experience. By setting the first number threshold to be greater than 1, a judgment may be made as to whether the target object has yawning behavior in the case of comprehensive consideration of multiple frames, avoiding the influence of the shooting angle of a single frame, the distance of the target object from the video capture device in a single frame, etc. on the mouth-to-nose ratio, and making a wrong judgment.

[0052] Similarly, it is possible to simply determine whether the target object is speaking based on the comparison of the mouth-to-nose ratio of multiple recognition frames with a threshold. Figure 5As shown, in one embodiment, step S140 may further include steps S510 to S520. In step S510, the mouth-nose ratio of each recognition frame may be compared with the above-mentioned first ratio threshold and second ratio threshold. The second ratio threshold is similar to the first ratio threshold, and is also a threshold value for the mouth-nose ratio corresponding to the degree of mouth opening, but the second ratio threshold is less than the first ratio threshold. The first ratio threshold and the second ratio threshold respectively reflect different degrees of mouth opening, and can be set by the user based on experience. Generally, the degree of mouth opening when speaking is less than the degree of mouth opening when yawning, but greater than the degree of mouth opening when there is no language action (such as breathing, licking the corners of the mouth, etc.). By setting the first ratio threshold and the second ratio threshold, a mouth opening degree interval is actually set, and the mouth opening degree within the interval can be considered to correspond to the target object's possible speaking behavior. By comparing in step S510, the number of recognition frames whose mouth-nose ratio is greater than the second ratio threshold and less than the first ratio threshold can be determined. In step S520, if the number of recognition frames whose mouth-nose ratio is greater than the second ratio threshold and less than the first ratio threshold is greater than the second number threshold, the target object may have speaking behavior. The second quantity threshold may be equal to or greater than 1, and may be set by the user based on experience. By setting the second quantity threshold to be greater than 1, a judgment may be made as to whether the target object has a speaking behavior by comprehensively considering multiple frames, thereby avoiding misjudgment due to the influence of the shooting angle of a single frame, the distance of the target object from the video capture device in a single frame, etc. on the mouth-to-nose ratio.

[0053] In order to further eliminate the influence of individual differences (such as mouth size, mouth thickness, etc.) on gesture recognition and improve the recognition accuracy of speaking / yawning, the change trend of the mouth-to-nose ratio can be further considered instead of the mouth-to-nose ratio itself. Therefore, in one embodiment, step S140 can include determining whether the target object has yawning behavior or speaking behavior based on the change trend of the mouth-to-nose ratio of multiple recognition frames. Figure 6 The figure schematically shows the change trend of the mouth-nose ratio and the mouth-nose ratio. Figure 6 As shown in the figure and as described above, the mouth-to-nose ratio in the present disclosure refers to the ratio of the mouth height to the nose height. The mouth-to-nose ratio may change over time due to the target subject's behavior of talking, yawning, etc. In addition, the target subject's head may move, for example, the target subject may raise his head, lower his head, turn his head, etc. In these cases, if the target subject's mouth opening amplitude does not change, the mouth-to-nose ratio does not change much.

[0054] Figure 7 The specific details of determining whether the target object yawns or speaks based on the change trend of the mouth-to-nose ratio are shown. Figure 7As shown, in one embodiment, determining whether the target object has yawning behavior or speaking behavior based on the change trend of the mouth-to-nose ratio may include steps S710 to S730.

[0055] In step S710, one or more recognition intervals can be identified based on the comparison between the mouth-to-nose ratios of adjacent recognition frames in a plurality of recognition frames. Every time the target object speaks or yawns, there is a process in which the mouth-to-nose ratio first becomes larger (corresponding to the process of opening the mouth when speaking or yawning), and then becomes smaller (corresponding to the process of closing the mouth when speaking or yawning). In step S710, by comparing the mouth-to-nose ratios of adjacent recognition frames, the recognition interval corresponding to the above process can be identified. Each recognition interval may include a first interval and a second interval following the first interval. In the first interval, the mouth-to-nose ratio increases continuously. In the second interval, the mouth-to-nose ratio decreases continuously. Thus, the first interval corresponds to the process in which the mouth-to-nose ratio becomes larger and larger in the above process, and the second interval corresponds to the process in which the mouth-to-nose ratio becomes smaller and smaller in the above process.

[0056] There may be a variety of methods to identify the above-mentioned identification interval. As an example, one or more local peaks and local valleys corresponding to each local peak may be determined based on a comparison between the mouth-to-nose ratios of adjacent identification frames in a plurality of identification frames. Each local peak may correspond to two local valleys. Based on a local peak and two local valleys corresponding to the local peak, an identification interval and a first interval and a second interval in the identification interval may be determined. As another example, the difference in the mouth-to-nose ratio between adjacent frames in a plurality of identification frames may be calculated to obtain a difference sequence, i.e., a mouth-to-nose ratio change trend sequence. Further, one or more difference sets may be determined from the difference sequence, each difference set including a plurality of consecutive positive differences and a plurality of consecutive negative differences. Each identification interval may correspond to one of the above-mentioned difference sets. The first interval in each identification interval corresponds to a plurality of consecutive positive differences in the difference set, and the second interval in each identification interval corresponds to a plurality of consecutive negative differences in the difference set.

[0057] In step S720, for each identification interval, the difference between the maximum mouth-to-nose ratio and the minimum mouth-to-nose ratio in the first interval can be determined as an increasing trend value, and the difference between the minimum mouth-to-nose ratio and the maximum mouth-to-nose ratio in the second interval can be determined as a decreasing trend value. It can be understood that the maximum mouth-to-nose ratio in the first interval is the same as the maximum mouth-to-nose ratio in the second interval. As an example, for each identification interval, the difference between the above-mentioned local peak and the first local valley value of the two local valley values ​​corresponding thereto can be determined as an increasing trend value, and the difference between the second local valley value of the above-mentioned two local valley values ​​and the one local peak value can be determined as a decreasing trend value. As another example, for each identification interval, the sum of multiple consecutive positive difference values ​​in the above-mentioned difference set can be determined as an increasing trend value, and the sum of multiple consecutive negative difference values ​​in the above-mentioned difference set can be determined as a decreasing trend value.

[0058] The rising trend value and the falling trend value can reflect the maximum opening change range and the maximum closing change range of the mouth in a mouth opening and closing process (because the nose height hardly changes). In step S730, it can be determined whether the target object has yawning behavior or speaking behavior based on the rising trend value and the falling trend value of one or more identification intervals. Step S730 can be performed based on comparison with various thresholds.

[0059] During yawning, the opening and closing changes of the mouth are relatively large. In one embodiment, a first rising trend threshold and a first falling trend threshold may be set. The first rising trend threshold may be a threshold value for the mouth-to-nose ratio change trend corresponding to the opening change of the mouth, and the first trend value may be a positive value. The first falling trend threshold may be a threshold value for the mouth-to-nose ratio change trend corresponding to the closing change of the mouth, and the first falling trend threshold may be a negative value. The first rising trend threshold and the first falling trend threshold may be set by the user based on experience. For example, the first rising trend threshold may be set to 0.36, and the first falling trend threshold may be set to -0.36. The rising trend value of each identification interval may be compared with the first rising trend threshold, and the falling trend value of each identification interval may be compared with the first falling trend threshold. As long as there is an identification interval whose rising trend value is greater than or equal to the first rising trend threshold, and whose falling trend value is less than or equal to the first falling trend threshold, it may be determined that the target object has a large degree of opening and closing action, thereby determining that the target object has yawning behavior.

[0060] Generally, the opening and closing variation of the mouth when speaking is smaller than the opening and closing variation of the mouth when yawning, but larger than the opening and closing variation of the mouth when there is no language action (such as breathing, licking the corners of the mouth, etc.). Therefore, in one embodiment, in addition to the above-mentioned first rising trend threshold and the first falling trend threshold, a second rising trend threshold and a second falling trend threshold may be additionally set, so as to set an interval of the opening variation of the mouth and an interval of the closing variation of the mouth. Similar to the first rising trend threshold, the second rising trend threshold may be a threshold corresponding to the mouth opening variation of the mouth with respect to the mouth-nose ratio variation trend. The second rising trend threshold may be a positive value, and the first rising trend threshold may be greater than the second rising trend threshold. Similar to the first falling trend threshold, the second falling trend threshold may be a threshold corresponding to the mouth closing variation of the mouth with respect to the mouth-nose ratio variation trend. The second falling trend threshold may be a negative value, and the first falling trend threshold may be less than the second falling trend threshold. The second rising trend threshold and the second falling trend threshold may also be set by the user based on experience. For example, the second rising trend threshold may be set to 0.08, and the second falling trend threshold may be set to -0.08. The rising trend value of each recognition interval can be compared with the first rising trend threshold and the second rising trend threshold, and the falling trend value of each recognition interval can be compared with the first falling trend threshold and the second falling trend threshold. As long as there is a recognition interval whose rising trend value is between the first rising trend threshold and the second rising trend threshold, and whose falling trend value is between the first falling trend threshold and the second falling trend threshold, it can be determined that the target object has a speaking behavior.

[0061] In one embodiment, for determining whether the target object has a speaking behavior, the number of identification intervals whose rising trend value is between a first rising trend threshold and a second rising trend threshold, and whose falling trend value is between a first falling trend threshold and a second falling trend threshold is further considered. Only when the number of such identification intervals is equal to or greater than the threshold number, the target object is considered to have a speaking behavior. The threshold number can be set by the user based on experience or needs, so that it can be more flexible to determine whether the target object has a speaking behavior. By setting the threshold number, a judgment can be made on whether the target object has a speaking behavior while comprehensively considering multiple frames, avoiding the influence of the shooting angle of a single frame, the distance of the target object from the video capture device in a single frame, etc. on the mouth-to-nose ratio, and making a wrong judgment.

[0062] In order to better utilize the head posture model and improve the recognition accuracy of speaking / yawning, in one embodiment, before the above step S120, the multiple image frames can be cropped before filtering the multiple image frames to obtain multiple cropped image frames that only include the facial part of the target object. Then, in step S130, the multiple cropped image frames can be filtered based on the head posture model to obtain multiple recognition frames that meet the predetermined requirements. Since the information in the image frame that is not related to the facial part is removed after cropping, the head posture model can be used to improve the recognition of the pitch angle and yaw angle, thereby improving the recognition accuracy of speaking / yawning. Figure 8 The cropping of the image frame is schematically shown. Specifically, Figure 8 It shows that the image frame includes the image of the target object and its surroundings before cropping, and only includes the face part of the target object after cropping.

[0063] Since the distance of the target object from the video capture device may change, the size of the target object in different image frames may be different. In one embodiment, after cropping multiple image frames to obtain multiple cropped image frames, the sizes of the multiple cropped image frames may be further adjusted to make them uniform to a fixed size, thereby eliminating the influence on the recognition of speaking / yawning caused by the difference in the distance of the target object from the video capture device over time, and further improving the recognition accuracy.

[0064] In the gesture recognition method according to the embodiment of the present disclosure, the gesture of the target object, that is, whether there is yawning or speaking behavior, can be determined based on the mouth-nose ratio. When the target object in the frame is in a side face position, the mouth height and nose height are relatively less affected by the side face than the mouth width, thereby reducing the influence of the side face on the technical solution for determining whether the target object has yawning or speaking behavior, and improving the recognition accuracy of speaking / yawning in different head gestures. In addition, by considering multiple frames, the influence of individual differences on the recognition results can also be reduced.

[0065] An embodiment of the present disclosure also provides a device for gesture recognition. Fig. 9 A schematic diagram of a device 900 for gesture recognition according to an embodiment of the present disclosure is shown.

[0066] like Fig. 9 As shown, the device 900 for gesture recognition according to this embodiment includes a processor 910 and a memory 920. The memory 920 stores one or more computer programs.

[0067] The processor 910 is a program control device such as a microprocessor, for example, and operates according to a program installed in the memory 920. The memory 920 is, for example, a storage element such as a ROM or a RAM, etc. The program executed by the processor 910 or the like is stored in the memory 920. Fig. 9 The device 900 for gesture recognition shown can be used to implement the gesture recognition method disclosed in this application.

[0068] In the device according to the embodiment of the present disclosure, the posture of the target object, that is, whether there is yawning or speaking behavior, can be determined based on the mouth-nose ratio. When the target object in the frame is in a side face position, the mouth height and nose height are relatively less affected by the side face than the mouth width, thereby reducing the influence of the side face on the technical solution for determining whether the target object has yawning or speaking behavior, and improving the recognition accuracy of speaking / yawning in different head postures. In addition, by considering multiple frames, the influence of individual differences on the recognition results can also be reduced.

[0069] An embodiment of the present disclosure also provides a non-transitory computer-readable storage medium. Fig.10 A schematic diagram of a computer-readable storage medium 1000 according to an embodiment of the present disclosure is shown. The computer-readable storage medium 1000 stores computer program instructions 1001, wherein the computer program instructions 1001, when executed by a processor, enable the processor to perform the gesture recognition method provided by the embodiment of the present disclosure.

[0070] The hardware computing device described in the present disclosure may be implemented as a whole or its components by various suitable hardware means, including but not limited to FPGA, ASIC, SoC, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The devices, equipment, methods, and systems involved in the present disclosure are not limited to any specific hardware architecture or configuration. The components in the disclosed devices, equipment, and systems may be separated or integrated, may be combined in different ways, and / or replaced or supplemented by other components. It should be understood that the teachings of the present disclosure may be implemented in various forms of hardware, software, firmware, dedicated processors, or combinations thereof.

[0071] The block diagrams of the devices, equipment, methods, and systems involved in the present disclosure are only exemplary and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these circuits, devices, devices, equipment, and systems may be connected, arranged, and configured in any manner as long as the desired purpose can be achieved.

[0072] In the above description, the present invention has been described based on the embodiment. This embodiment is only illustrative, and those skilled in the art should understand that the combination of the constituent elements and processes of this embodiment can be modified in various ways, and such modifications are also within the scope of the present invention.

Claims

1. A gesture recognition method, comprising: sequentially extracting a plurality of image frames from a video stream including an image frame having a head of a target object; filtering the plurality of image frames based on the head posture model to obtain a plurality of recognition frames meeting predetermined requirements; For each recognition frame in the plurality of recognition frames, calculating a mouth-to-nose ratio, where the mouth-to-nose ratio is equal to a ratio of a mouth height to a nose height of the target object in each recognition frame; as well as Based on the mouth-to-nose ratios of the multiple recognition frames, it is determined whether the target object is yawning or talking.

2. The gesture recognition method according to claim 1, wherein: Determining whether the target object yawns or speaks based on the mouth-to-nose ratio of the multiple recognition frames includes: Based on the change trend of the mouth-to-nose ratio of the multiple recognition frames, it is determined whether the target object has yawning behavior or speaking behavior.

3. The gesture recognition method according to claim 2, wherein: Determining whether the target object yawns or speaks based on the change trend of the mouth-to-nose ratio of the multiple recognition frames includes: Based on a comparison between mouth-to-nose ratios of adjacent recognition frames in the plurality of recognition frames, identifying one or more recognition intervals, the recognition intervals comprising a first interval and a second interval immediately following the first interval, in which the mouth-to-nose ratio continuously increases, and in which the mouth-to-nose ratio continuously decreases; For each identification interval, determining the difference between the maximum mouth-to-nose ratio and the minimum mouth-to-nose ratio in the first interval as an increasing trend value, and determining the difference between the minimum mouth-to-nose ratio and the maximum mouth-to-nose ratio in the second interval as a decreasing trend value; and It is determined whether the target object has a yawning behavior or a talking behavior based on the increasing trend value and the decreasing trend value of the one or more recognition intervals.

4. The gesture recognition method according to claim 3, wherein: Determining whether the target object has a yawning behavior or a speaking behavior based on the rising trend value and the falling trend value of the one or more recognition intervals includes: In response to the rising trend value of a first identification interval among the one or more identification intervals being greater than or equal to a first rising trend threshold and the falling trend value of the first identification interval being less than or equal to the first falling trend threshold, it is determined that the target object has yawning behavior.

5. The gesture recognition method according to claim 3, wherein: Determining whether the target object has a yawning behavior or a speaking behavior based on the rising trend value and the falling trend value of the one or more recognition intervals includes: In response to an increasing trend value of a first recognition interval of the one or more recognition intervals being between a first increasing trend threshold and a second increasing trend threshold, and a decreasing trend value of the first recognition interval being between a first decreasing trend threshold and a second decreasing trend threshold, determining that the target object has a speaking behavior, The first rising trend threshold is greater than the second rising trend threshold, and the first falling trend threshold is less than the second falling trend threshold.

6. The gesture recognition method according to claim 3, wherein: Determining whether the target object has a yawning behavior or a speaking behavior based on the rising trend value and the falling trend value of the one or more recognition intervals includes: In response to determining that the number of identification intervals whose rising trend value is between a first rising trend threshold and a second rising trend threshold and whose falling trend value is between the first falling trend threshold and the second falling trend threshold is equal to or greater than a threshold number, determining that the target object has a speaking behavior, The first rising trend threshold is greater than the second rising trend threshold, and the first falling trend threshold is less than the second falling trend threshold.

7. The gesture recognition method according to claim 1, wherein: Filtering the plurality of image frames based on the head posture model to obtain a plurality of recognition frames meeting predetermined requirements includes: Determining a pitch angle of each of the plurality of image frames based on the head posture model; and The plurality of image frames are filtered based on a pitch angle of each image frame.

8. The gesture recognition method according to claim 7, wherein: Filtering the plurality of image frames based on the pitch angle of each image frame includes: Comparing the pitch angle of each image frame with a first head-up threshold and a second head-up threshold; and The following image frames are filtered out based on the comparison: An image frame whose pitch angle is greater than the first head-up threshold; and An image frame whose pitch angle is smaller than the second head-up threshold.

9. A device for gesture recognition, comprising: processor; and a memory storing one or more computer programs, When the one or more computer programs are executed by the processor, the processor executes the gesture recognition method according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having computer program instructions stored thereon, wherein: When the computer program instructions are executed by a processor, the processor is caused to perform the gesture recognition method according to any one of claims 1-8.

Images