Method, device, electronic device and storage medium for identifying gestures in the air
By setting preset periodic detection and conditional determination in the air-to-air gesture recognition system, the error detection and frame drop problems caused by individual differences and environmental factors are solved, and the accuracy and effectiveness of gesture recognition are improved.
Patent Information
- Application Number
- CN202111616232.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-27
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-12-27
AI Technical Summary
Due to individual differences and environmental factors, the existing air-to-air gesture recognition system is difficult to avoid misdetection and frame dropping problems, resulting in a high error rate of gesture recognition.
By setting preset periodic detection and condition determination, recording and analyzing the current gesture of each frame, determining whether the air-to-air gesture meets the preset conditions, including the proportion of frames, the type consistency of the start and end frames, etc. Determines the effectiveness and type of gestures.
It effectively avoids the error detection and frame drop problems of the air-to-air gesture recognition system, improves the accuracy and effectiveness of gesture recognition, and reduces the error rate.
Smart Images

Figure CN114546106B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of human-computer interaction technology, and in particular to a method, device, electronic device and storage medium for recognizing air gestures. Background Art
[0002] With the development of human-computer interaction and artificial intelligence technology, air gesture recognition technology has also developed rapidly. There are three types of gestures commonly used in air gesture recognition systems: static gestures, variable gestures, and mobile gestures. The air gesture recognition system can detect the position and hand shape of the operator in real time, but due to individual differences, the hand shape of each person is different for the same hand shape. In addition, due to interference from factors such as the environment and equipment, it is difficult for the air gesture recognition system to avoid problems such as false detection and frame loss. Summary of the invention
[0003] In view of this, the purpose of the present application is to propose a method, device, electronic device and storage medium for recognizing air gestures.
[0004] Based on the above purpose, the present application provides a method for recognizing air gestures, including:
[0005] Based on a preset first cycle, continuously and periodically detect the air gesture in time sequence, and record the current gesture of each frame in each of the first cycle;
[0006] In response to determining that the air gesture in the first period meets a first preset condition, determining that the air gesture is a valid air gesture;
[0007] The first preset condition includes: the proportion of the number of frames in which the current gesture is the target gesture in all frames meets a preset threshold.
[0008] Furthermore, the first preset condition also includes:
[0009] The current gesture of at least one frame of a first preset number of consecutive frames before the start frame or the first cycle is the target gesture;
[0010] and / or
[0011] In all frames, there are no consecutive frames greater than or equal to a preset number of frames in which the current gesture is not the target gesture;
[0012] and / or
[0013] The current gesture of the end frame is the target gesture or the current gestures of a second preset number of consecutive frames before the end frame are all the target gestures.
[0014] Furthermore, the air gesture includes a static gesture, a changing gesture or a moving gesture.
[0015] Further, based on a preset second period, the air gesture is continuously and periodically detected in time sequence and the current gesture of each frame in each second period is recorded; wherein the duration of the second period is greater than that of the first period;
[0016] In response to determining that each of the air gestures of the first period within the second period meets the first preset condition, the air gesture is determined to be a valid static gesture.
[0017] Furthermore, based on a preset third cycle, the air gesture is continuously and periodically detected in time sequence, and the current gesture of each frame in the end period of each third cycle is recorded; wherein the preset third cycle includes, according to time sequence: a preceding period and an end period;
[0018] In response to determining that the end time period of the third cycle meets the preset end condition, based on the preset detection cycle, continuously perform periodic detection in the preceding time period of the third cycle in reverse chronological order, and record the current gesture of each frame in each detection cycle;
[0019] In response to a recorded detection period meeting a preset start condition, and a duration between the detection period and the end period meeting a preset duration, determining that the air gesture is a valid variation gesture;
[0020] The preset end condition is: each of the first period of air gestures in the end period meets the first preset condition; the target gesture in the end period is a preset end gesture;
[0021] The preset start condition is: each of the first period's air gestures within the detection period meets the first preset condition; and the target gesture within the detection period is a preset start gesture.
[0022] Further, after determining that the air gesture is a valid air gesture, continue detecting until the air gesture reaches a preset termination condition;
[0023] Among them, the preset termination condition includes: the movement amount of the air gesture reaches a preset movement value and / or the air gesture in the first period does not meet the first preset condition.
[0024] Furthermore, the movement amount includes a movement distance and / or a rotation angle.
[0025] Furthermore, the duration of the first cycle is 0.1s to 0.8s.
[0026] Based on the same inventive concept, the present application also provides a device for identifying air gestures, including:
[0027] A detection and recording module is configured to continuously and periodically detect the air gesture in a time sequence based on a preset first period, and record the current gesture of each frame in each of the first period;
[0028] The determination module is configured to determine that the air gesture in the first period meets a first preset condition and determine that the air gesture is a valid target gesture; wherein the first preset condition includes: the proportion of the number of frames in which the current gesture is the target gesture in all frames meets a preset threshold.
[0029] Furthermore, in the determination module, the first preset condition further includes:
[0030] The current gesture of at least one frame of a first preset number of consecutive frames before the start frame or the first cycle is the target gesture;
[0031] and / or
[0032] In all frames, there are no consecutive frames greater than or equal to a preset number of frames in which the current gesture is not the target gesture;
[0033] and / or
[0034] The current gesture of the end frame is the target gesture or the current gestures of a second preset number of consecutive frames before the end frame are all the target gestures.
[0035] Furthermore, the identification device further includes:
[0036] A static gesture detection module is configured to continuously and periodically detect the air gesture in a time sequence based on a preset second period and record the current gesture of each frame in each second period; wherein the duration of the second period is greater than that of the first period;
[0037] The static gesture determination module is configured to determine that the air gesture is a valid static gesture in response to determining that each air gesture in the first period within the second period meets the first preset condition.
[0038] Furthermore, the identification device further includes:
[0039] The first variable gesture detection module is configured to continuously and periodically detect the air gesture in a time sequence based on a preset third cycle, and record the current gesture of each frame in the end period of each third cycle; wherein the preset third cycle includes, in time sequence, a preceding period and the end period;
[0040] A second variant gesture detection module is configured to, in response to determining that the end time period of the third cycle meets a preset end condition, perform continuous periodic detection in reverse chronological order within the preceding time period of the third cycle based on a preset detection cycle, and record the current gesture of each frame within each detection cycle;
[0041] The variable gesture determination module is configured to determine that the air gesture is a valid variable gesture in response to a recorded detection cycle meeting a preset start condition and the duration between the detection cycle and the end period meeting a preset duration.
[0042] Furthermore, the identification device further includes:
[0043] The moving gesture determination module is configured to continue detecting after determining that the air gesture is a valid air gesture until the air gesture reaches a preset termination condition; wherein the preset termination condition includes: the movement amount of the air gesture reaches a preset movement value and / or the air gesture of the first period does not meet the first preset condition.
[0044] Further, in the moving gesture determination module, the moving amount includes a moving distance and / or a rotation angle.
[0045] Based on the same inventive concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the methods described above is implemented.
[0046] Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the methods described above.
[0047] From the above, it can be seen that the air gesture recognition method, device, electronic device and storage medium provided by the present application, when the air gesture recognition system performs air gesture recognition, determines the air gesture according to preset conditions to determine whether the air gesture is valid, and can also determine whether it is a corresponding valid gesture type according to the type of the air gesture. By conditionally determining the validity of the air gesture, it is possible to avoid problems such as false detection and frame loss in the air gesture recognition system, as well as problems such as gesture recognition errors and triggering of erroneous instructions due to invalid air gestures, thereby improving the effectiveness and accuracy of air gesture recognition and reducing the error rate of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0049] Figure 1 A schematic diagram of a flow chart of a method for recognizing an air gesture according to an embodiment of the present application;
[0050] Figure 2 A schematic flow chart of an effective static gesture recognition method according to an embodiment of the present application;
[0051] Figure 3 A schematic flow chart of an effective method for identifying a variable gesture according to an embodiment of the present application;
[0052] Figure 4 A schematic diagram of the structure of a device for recognizing air gestures according to an embodiment of the present application;
[0053] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0055] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0056] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.
[0057] For ease of understanding, the terms involved in the embodiments of the present disclosure are explained below:
[0058] Gesture: refers to various movements or postures produced by fingers, palms, or palms and arms. According to the time-varying nature of gestures, gestures can be divided into static gestures, changing gestures, and moving gestures.
[0059] Gesture recognition: Natural gesture recognition technology based on computer vision, which uses mathematical algorithms to recognize human gestures. Gesture recognition can come from the movement of various parts of the human body, but generally refers to the movement of the hand. Users can use simple gestures to control or interact with the device, allowing the device to understand human behavior.
[0060] Static gesture: refers to a motionless hand making a special shape or posture, that is, the shape or posture of the hand is used to indicate the operator's intention, and the hand shape remains unchanged for a period of time. For example, keep the fist state and keep it still for 1 second; keep the palm state and keep it still for 2 seconds, etc.
[0061] Variable gesture: refers to a gesture that changes over time. Within a certain period of time, one gesture changes into another, forming a dynamic gesture, such as a palm changing into a fist; a palm changing into a thumb, etc.
[0062] Moving gesture: A dynamic gesture formed by moving the arm and / or rotating the wrist while keeping the shape of the hand unchanged, such as moving the hand in the shape of a scissors hand; rotating the hand in the shape of a thumbs up, etc.
[0063] False detection: The air gesture recognition system produces incorrect recognition results for the user's gesture.
[0064] Frame loss: The air gesture recognition system fails to detect the user's gesture, or some frames are lost during the detection process, resulting in inaccurate detection results.
[0065] Frame: The smallest unit of a single image in an image animation. Each frame is a still image.
[0066] The principle and spirit of the present application are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION
[0068] With the development of human-computer interaction and artificial intelligence technology, air gesture recognition technology has also developed rapidly. The purpose of air gesture recognition is to recognize human gestures through mathematical algorithms. Gestures can originate from any body movement or state, but usually from the hands. Users can use simple gestures to control or interact with devices without touching them. Gesture recognition can be seen as a way for computers to understand human body language, thereby building a richer bridge between machines and people than the original text user interface or even the graphical user interface.
[0069] Gesture recognition enables people to communicate with machines and interact naturally without any mechanical equipment. Therefore, gesture recognition can be applied in a variety of scenarios. For example, in the field of smart homes, remote control of smart home appliances, home robots, wearables and other hardware devices, based on gesture recognition technology, by inputting fixed gestures as smart hardware control instructions, makes the human-computer interaction more intelligent; in the field of smart cars, through car gestures, control left swipe, right swipe, volume increase and decrease, menu and other operations, can effectively supplement the shortcomings of voice recognition in some noisy environments, making the in-car interaction more convenient and natural; in the field of VR and AR, in a virtual environment, users are given gesture recognition navigation and control capabilities close to real life, switching from real scenes to virtual scenes, and establishing the most direct human-computer interaction method; other hardware devices such as wearable devices, children's teaching aids, etc., can all control the corresponding functions through user gestures, making the human-computer interaction method more intelligent and natural.
[0070] However, as described in the background technology section, the existing gesture recognition system can detect the position and hand shape of the operator in real time, but in actual applications, gestures are usually affected by individual differences and environmental factors. For example, for the same hand shape, each person's hand shape is different. Coupled with the interference of environmental and equipment factors, it is difficult for the remote gesture recognition system to avoid problems such as false detection and frame loss, and the error rate of gesture recognition is relatively high.
[0071] Based on the above problems, the present application provides a method, device, electronic device and storage medium for recognizing air gestures. First, based on a preset first period, the air gesture is continuously and periodically detected in chronological order, and the current gesture of each frame in each of the first period is recorded. Then, it is determined whether the air gesture is valid. When the air gesture of the first period meets the first preset condition, the air gesture is determined to be a valid air gesture. Among them, the first preset condition includes: the proportion of the number of frames in which the current gesture is the target gesture in all frames meets the preset threshold. In addition, according to the type of the air gesture (static gesture, changing gesture or moving gesture), for each different type of air gesture, it is continued to be determined whether the air gesture is a valid gesture type.
[0072] When the air gesture recognition system performs air gesture recognition, the air gesture is judged according to preset conditions to determine whether the air gesture is valid, and whether it is a corresponding valid gesture type can also be determined according to the type of the air gesture. By determining the validity of the air gesture through conditions, it is possible to avoid problems such as false detection and frame loss in the air gesture recognition system, as well as problems such as gesture recognition errors and triggering of wrong instructions due to invalid air gestures, thereby improving the effectiveness and accuracy of air gesture recognition and reducing the error rate of gesture recognition.
[0073] After summarizing the overall content of the present invention, various non-limiting embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.
[0074] refer to Figure 1 , the present application provides a method for recognizing an air gesture, comprising:
[0075] Step S101: based on a preset first cycle, continuously and periodically acquire air gestures in chronological order, and record the current gesture of each frame in each of the first cycle.
[0076] Specifically, the air gesture can be a static air gesture, a variable air gesture or a mobile air gesture. Before receiving the air gesture detection instruction, the air gesture recognition system has divided various valid gestures, and set relevant preset conditions for the determination of various valid gestures. For example, preset conditions corresponding to the determination of valid air gestures are set for air gestures, and preset conditions for determining valid static air gestures are set for static air gestures. After receiving the air gesture detection instruction, the air gesture recognition system obtains the air gesture of the operator, and detects and determines the air gesture according to the following steps based on the preset conditions corresponding to the air gesture.
[0077] After receiving the air gesture detection instruction, the air gesture is continuously and periodically scrolled and detected and recorded in chronological order, with the preset first period as the scrolling unit, and the detection process continues in chronological order.
[0078] The preset first period is a time period that has been preset in the initial stage, usually set according to experience, and the duration of the first period is set between 0.1s and 0.8s. When the air gesture recognition system is performing gesture recognition, the gesture of the operator's hand needs to remain unchanged and maintained for a certain period of time before the system can confirm the recognized gesture information. The preset first period is the shortest time that the operator's gesture needs to be maintained. Therefore, for the acquired air gesture, the same gesture must be maintained continuously for a time greater than or equal to the first period before the system can confirm that the gesture is valid, otherwise the system cannot confirm the acquired gesture information.
[0079] The current gesture recorded in each frame within the first period includes one or more of the hand shape information, position information, and posture information of the current gesture. When actually recording, the hand shape information of the current gesture must be recorded, and other information such as position information and posture information may be recorded or not based on actual needs, and are not limited here.
[0080] Among them, the hand shape information is the hand shape type detected in the current frame, such as the hand shape of the current frame is recorded as "fist", "palm" or "thumb", etc. The hand shape information can also be recorded with codes, for example, "fist" is recorded as "1", and "palm" is recorded as "2", etc.; the position information is the coordinate position of the hand in the current frame, such as data information of the three-dimensional coordinates of the hand; the posture information is data information of the posture angle of the hand in the current frame.
[0081] Among them, there are two methods for taking the preset first period as the scrolling unit and performing continuous periodic detection on the air gesture in chronological order: Method 1, taking the preset first period as the scrolling unit, and performing periodic continuous iteration based on the frame as the basic unit; Method 2, taking the first period as the scrolling unit, and performing continuous periodic detection based on the first period as the basic unit.
[0082] An example of method 1 is as follows:
[0083] Take the acquired air gesture of the operator as "fist" as an example. Assuming the preset first cycle is 0.5s, when the system acquires the air gesture "fist", it continuously detects the air gesture with a cycle of 0.5s. For every 0.5s, the system will recognize 5 frames of the current gesture and record the hand shape information of each frame of the current gesture.
[0084] Taking the start detection time of 0s as an example, based on the preset first cycle of 0.5s, continuous detection is performed. The system will detect the hand shapes from 0s to 0.5s as 1 to 5 frames in the first first cycle, and record the hand shape information of the current gesture in each frame in the first first cycle in sequence as: "fist", "fist", "palm", "fist", "palm"; the system will detect and record the hand shapes of 2 to 6 frames in the second first cycle, and record the hand shape information of the current gesture in each frame in the second first cycle in sequence as: "fist", "palm", "fist", "palm", "fist"; the system will detect and record the hand shapes of 3 to 7 frames in the third first cycle, and record the hand shape information of the current gesture in each frame in the third first cycle in sequence as: "palm", "fist", "palm", "fist", "fist"... Recursive iteration is performed based on frame units, and the current gesture in each frame in each first cycle is recorded in sequence.
[0085] An example of method 2 is as follows:
[0086] Take the acquired air gesture of the operator as "fist" as an example. Assuming the preset first cycle is 0.5s, when the system acquires the air gesture "fist", it continuously detects the air gesture with a cycle of 0.5s. For every 0.5s, the system will recognize 5 frames of the current gesture and record the hand shape information of each frame of the current gesture.
[0087] Taking the start detection time of 0s as an example, based on the preset first cycle of 0.5s, continuous detection is performed. The system will detect the current gesture from 0s to 0.5s in the first first cycle, and record the hand shape information of the current gesture in each frame in the first first cycle in turn: "fist", "fist", "palm", "fist", "palm"; the system will detect the current gesture from 0.5s to 1s in the second first cycle, and record the hand shape information of the current gesture in each frame in the second first cycle in turn: "fist", "palm", "fist", "palm", "fist"; the system will detect and record the hand shape from 1s to 1.5s in the third first cycle, and record the hand shape information of the current gesture in each frame in the third first cycle in turn: "palm", "fist", "palm", "fist", "fist"… Continuous detection is performed with the first cycle as the basic time unit, and the current gesture in each frame in each first cycle is recorded in turn.
[0088] Step S102, in response to determining that the air gesture in the first period meets a first preset condition, determining that the air gesture is a valid air gesture; wherein the first preset condition includes: the proportion of the number of frames in which the current gesture is the target gesture in all frames meets a preset threshold.
[0089] Specifically, in the process of continuous detection and recording, the current gesture of each frame of each first period is recorded in turn. When it is detected that the air gesture in a certain first period meets the first preset condition, the air gesture is determined to be a valid air gesture, and related operations corresponding to the air gesture can be executed.
[0090] The preset target gesture is pre-set by the system and is the correct gesture corresponding to the acquired air gesture. For example, when the air gesture that the operator is about to make is a "fist", the air gesture recognition system calls the preset target gesture corresponding to the air gesture made by the operator, which is a "fist".
[0091] In actual applications, gestures are usually affected by individual differences and environmental factors. For example, for the same gesture, each person's hand shape is different, and coupled with interference from factors such as the environment and equipment, the system may have false detections, missed detections, frame drops, or inaccurate recognition. For the same air gesture "fist", the system may recognize "fist", "slap" or "thumb", etc. Therefore, in this application, through the screening of the first preset condition and combined with the knowledge of probability, it is determined whether the air gesture is valid, thereby reducing the risk of false detection, missed detection, frame drops, etc. in the system and improving the accuracy of system recognition.
[0092] The first preset condition includes: the proportion of the number of frames in which the current gesture is the target gesture in all frames meets a preset threshold.
[0093] This condition determines from a probabilistic perspective whether the proportion of the current gesture in the first period that is the target gesture meets the preset threshold. If there are 5 frames of current gestures in the first period, but only 1 of the 5 frames is the target gesture, then from a probabilistic perspective it can be determined that the air gesture in the first period is not the corresponding target gesture. The preset threshold is determined based on an empirical value, and meeting the preset threshold means being greater than or equal to the preset threshold.
[0094] For example, the acquired air gesture is "fist", and the preset target gesture corresponding to "fist" is also "fist". For each first cycle, the system can detect 5 frames of current gestures. Assuming that the preset threshold is set to 0.6, if the current gesture detected in a first cycle is "fist", "fist", "slap", "thumb", "thumb", the current gesture in this first cycle is the target gesture. The proportion is 0.4, which is less than the preset threshold of 0.6, then it is proved that this first cycle does not meet the first preset condition, and the detection continues. If the current gesture detected in a first cycle is "fist", "fist", "slap", "fist", "fist", the current gesture in this first cycle is the target gesture. The proportion is 0.8, which is greater than the preset threshold of 0.6, then it is proved that this first cycle meets the first preset condition.
[0095] When a first period meets the first preset condition, the acquired air gesture is determined to be a valid air gesture.
[0096] In this embodiment, the acquired air gesture of the operator is still taken as an example. Assuming that the preset first period is 0.5s, when the system acquires the air gesture "fist", the air gesture is continuously detected with a period of 0.5s. For every 0.5s, the system will recognize 5 frames of current gestures and record the hand shape information of each frame of the current gesture, assuming that the preset threshold is set to 0.6.
[0097] Taking the start detection time as 0s as an example, based on the preset first cycle of 0.5s, continuous detection is performed. The system will detect gestures from 0s to 0.5s in the first first cycle, that is, gestures from 1 to 5 frames, and record the hand shape information of the current gesture in each frame in the first cycle in sequence: "thumb", "slap", "slap", "thumb", "slap". Among the current gestures in the first first cycle, none of the current gestures is the target gesture, so the gestures in the first first cycle do not meet the first preset condition, so the air gesture detection continues.
[0098] The system will detect and record the hand shapes of 2 to 6 frames in the second first cycle, and record the hand shape information of the current gesture in each frame in the second first cycle in turn as: "palm", "palm", "thumb", "palm", "fist". Among the current gestures in the second first cycle, there is 1 current gesture that is a target gesture, but the proportion of current gestures that are target gestures is 0.2, which is less than the preset threshold of 0.6 and does not meet the first preset condition, so it is necessary to continue to detect the air gesture.
[0099] Continue to record and detect until the system detects in the Nth first cycle, and record the hand shape information of the current gesture in each frame in the Nth first cycle in sequence: "fist", "fist", "fist", "slap", "fist". Among the current gestures in the Nth first cycle, 4 current gestures are target gestures, and the proportion of current gestures that are target gestures is 0.8, which is greater than the preset threshold value of 0.6 and meets the first preset condition. Therefore, the gesture in the Nth first cycle meets the first preset condition, and the acquired air gesture is determined to be a valid air gesture. After determining that the air gesture is a valid air gesture, you can stop detecting, perform related operations corresponding to the valid air gesture, or continue to detect other items.
[0100] For the related methods and systems for air gesture recognition, the determination of the air gesture within the preset first period is usually based on the type of the current gesture in the last frame within the first period. Determining only the last frame may cause the system to ignore frame loss or other situations in the intermediate process, resulting in the system's frame loss phenomenon not being taken seriously or handled. At the same time, since the last frame may have a recognition error, the probability of false detection in this type of gesture recognition method is very high. For example, if the current gesture recorded in a certain first period is "slap", "fist", "fist", "fist", "slap", then the related air gesture recognition system will determine that the air gesture in the first period is "slap" based on the last frame, but according to the gesture situation of these 5 frames, it is obvious that the air gesture in the first period is "fist", so the related air gesture recognition method will have a false detection. However, according to the gesture recognition method described in the present application, the air gesture in the first period is determined to be "fist", and the gesture recognition is accurate.
[0101] Using the air gesture recognition method described in the present application, when an air gesture recognition system performs air gesture recognition, the air gesture can be judged based on the first preset condition to determine whether the air gesture is valid. This can avoid problems such as false detection and frame loss in the air gesture recognition system, as well as gesture recognition errors caused by invalid air gestures and the triggering of erroneous instructions. This improves the effectiveness and accuracy of the recognized air gestures and reduces the error rate of gesture recognition.
[0102] In some embodiments, the first preset condition also includes: the gesture of at least one frame in a first preset number of consecutive frames before the start frame or the first cycle is the target gesture; and / or, in all frames, there are no consecutive current gestures greater than or equal to the preset number of frames that are not the target gesture; and / or, the gesture of the end frame is the target gesture or the gestures of a second preset number of consecutive frames before the end frame are all the target gestures.
[0103] Specifically, the first preset condition includes, in addition to the fact that the proportion of the number of frames in which the current gesture is the target gesture in all frames described in the above embodiment meets the preset threshold (hereinafter referred to as "Condition 1"), the following also includes: the current gesture in at least one frame of the first preset number of consecutive frames before the start frame or the first cycle is the target gesture (hereinafter referred to as "Condition 2"); and / or, there are no consecutive frames greater than or equal to the preset value in all frames in which the current gesture is not the target gesture (hereinafter referred to as "Condition 3"); and / or, the current gesture in the end frame is the target gesture or the current gesture in the second preset number of consecutive frames before the end frame are all the target gesture (hereinafter referred to as "Condition 4"). Among them, the first preset number of frames and the second preset number of frames are set according to actual needs, and the first preset number of frames and the second preset number of frames may be the same or different. In this embodiment, the first preset number of frames and the second preset number of frames are both two frames.
[0104] The air gestures in the first period can meet the conditions 1, 2, 3 and 4 at the same time, or some of the conditions. When the air gestures in the first period meet the above four conditions at the same time, the detection accuracy is relatively high and the detection result is more accurate. However, during actual detection, the specific conditions that the air gestures in the first period need to meet are determined according to actual needs and are not limited here.
[0105] Among them, the setting of condition 2 enables the recognition method to better and faster identify whether the acquired air gesture has entered the stage of valid gesture. If the current gesture of the starting frame in the first cycle is not the target gesture, and the current gestures of the two consecutive frames before the first cycle are not the target gesture, it proves that the acquired air gesture has not become the correct target gesture in this first cycle and has not entered the stage of valid target gesture.
[0106] For example, the acquired gesture is "fist", and the corresponding target gesture is also "fist". During the continuous detection process, if the current gesture of the starting frame of a first cycle is "slap", and the current gestures of the two consecutive frames before the first cycle are "slap" and "thumb" respectively, then it is proved that this first cycle does not meet the preset condition 2, and further detection is required. During the continued detection process, if the current gesture of the starting frame of a first cycle is "fist", then it is proved that this first cycle meets the preset condition 2; or if the current gestures of the two consecutive frames before a first cycle are "slap" and "fist" respectively, then one of the two frames is "fist", and this first cycle meets the preset condition 2.
[0107] Among them, the setting of condition 3 allows the recognition method to fully consider the situation of the intermediate frames, avoiding the changes in the acquired air gesture in the intermediate process, and ultimately affecting the effectiveness of the gesture. Fully considering the situation of the intermediate frames can also greatly avoid the impact of frame loss on the detection results and improve the accuracy of detection. When there are continuous frames greater than or equal to the preset value in all frames where the current gesture is not the target gesture, it proves that the acquired gesture has changed, and then the air gesture needs to be continued to be detected. The preset value is set based on the experience value.
[0108] For example, the acquired gesture is "fist", and the corresponding target gesture is also "fist", and the preset value is set to 2. During the continuous detection process, the current gestures of the 5 frames of a first cycle are: "fist", "slap", "slap", "slap", "fist", and since the current gesture is "slap" in 3 consecutive frames of the 5 frames in this first cycle, which is not the corresponding target gesture "fist", then this first cycle does not meet condition 3 and needs to be continuously detected; during the continued detection process, the current gestures of the 5 frames of a first cycle are: "fist", "slap", "fist", "slap", and "fist", and since there are no consecutive frames of the current gesture that is not "fist" in the 5 frames in this first cycle, then this first cycle meets condition 3.
[0109] Among them, the setting of condition 4 can reduce the impact of the end frame detection error on the entire detection result. As mentioned above, the end frame may be detected incorrectly, but this method not only considers the detection result of the end frame, but also considers the detection results of the two consecutive frames before the end frame, which is equivalent to comprehensively considering the detection results of the last 3 frames before obtaining the final detection result, thereby improving the accuracy and reliability of the detection.
[0110] For example, the acquired gesture is "fist", and the corresponding target gesture is also "fist". During the continuous detection process, the current gestures of the 5 frames in a first cycle are: "fist", "fist", "slap", "slap", "slap", and "slap" in sequence. Since the end frame of this first cycle is "slap", it is not the target gesture "fist", and the gestures of the two frames before the end frame are both "slap", which are not the target gesture "fist", then this first cycle does not meet condition 4 and needs to continue detection; during the continued detection process, the current gestures of the 5 frames in a first cycle are: "fist", "slap", "fist", "fist", and "slap" in sequence. Since the end frame of this first cycle is "slap", it is not the target gesture "fist", but the gestures of the two frames before the end frame are both "fist", which meets the target gesture "fist", then this first cycle meets condition 4.
[0111] In actual application, when using air gestures to perform air operations on a device, the device has been pre-set with the corresponding air gesture type, so the operator can only perform relevant operations on the device by making the corresponding valid air gesture. If the gesture type made by the operator is inconsistent with the preset air gesture type, or the air gesture made by the operator is not valid, the device will not perform the relevant operation. For example, the TV has preset the air gesture of "turning on" as a "valid static gesture", so the operator must make a valid static gesture to enable the TV to perform the power-on operation; or the car speaker has preset the air gesture of "increasing the volume by ten degrees" as a "valid mobile gesture translation 10cm", then the operator must make a valid mobile gesture and translate 10cm to enable the car speaker to perform the volume increase by ten degrees. Therefore, how to judge whether the static gesture made by the operator is a valid static gesture, or whether the mobile gesture is a valid mobile gesture, is a very important step in actual use. The following will describe in detail the process of judging valid static gestures, valid variable gestures, and valid mobile gestures.
[0112] In some embodiments, when the air gesture is a static gesture, it is necessary to continue to determine whether the static gesture is valid. A valid static gesture is to keep the hand shape unchanged and still for a period of time. The specific length of stillness is set based on experience, but it must be greater than the preset first period.
[0113] In this embodiment, a valid static gesture is to keep the shape of a valid air gesture unchanged and remain still for at least a preset second period, wherein the duration of the second period is greater than that of the first period.
[0114] Therefore, reference Figure 2 When the air gesture is a static gesture, it is necessary to continue to determine whether the static gesture is valid, and the following operations need to be performed:
[0115] Step S201: Based on a preset second cycle, the air gesture is continuously and periodically detected in chronological order and the current gesture of each frame in each second cycle is recorded; wherein the duration of the second cycle is greater than that of the first cycle.
[0116] Specifically, the preset second period is used as the scrolling unit, and the air gesture is continuously scrolled and recorded in chronological order, and the detection process is continuously performed in chronological order. The specific scrolling detection and recording process is similar to the continuous detection with the preset first period as the scrolling unit described above, and will not be repeated here.
[0117] Similarly, similar to the first cycle, the preset second cycle is used as a scrolling unit, and the continuous periodic detection of the air gesture in chronological order includes two methods: the first method is to use the preset second cycle as a scrolling unit, and to perform periodic continuous iteration based on the frame as the basic unit; the second method is to use the second cycle as a scrolling unit, and to perform continuous periodic detection based on the second cycle as the basic unit. In the actual detection process, the specific method to be used for continuous detection is determined according to actual needs, and is not limited here.
[0118] The preset second period is a time period corresponding to a valid static gesture that has been preset in the initial stage and is usually set based on experience.
[0119] Step S202: in response to determining that each of the air gestures in the first period within the second period meets the first preset condition, determining that the air gesture is a valid static gesture;
[0120] Specifically, in the process of continuous detection with the second period as the scrolling unit, it is determined whether each air gesture in the first period within the second period meets the first preset condition; if each air gesture in the first period within a second period meets the first preset condition, then the air gesture is determined to be a valid static gesture, and subsequent operations related to the valid static gesture can be executed.
[0121] In this embodiment, taking the static "fist" gesture of the operator as an example, the effective static gesture corresponding to the static "fist" is: keep the effective "fist" unchanged and stay still for at least one of the second periods, assuming that the preset second period is 1s and the preset first period is 0.5s.
[0122] When the system acquires the air gesture "fist", it continuously detects the air gesture in a cycle of 1 second. For every 1 second, the system will recognize 10 frames of the current gesture and record the hand shape information of each frame. For every 0.5 second, the system will recognize 5 frames of the current gesture and record the hand shape information of each frame.
[0123] Taking the start detection time as 0s as an example, continuous detection is performed based on the preset second cycle 1s. The system will detect the hand shape from 0s to 1s in the first of the second cycles, that is, record the current gestures of 1 to 10 frames, and record the hand shape information of the current gestures in each frame in the first second cycle in sequence: "fist", "palm", "fist", "thumb", "fist", "palm", "fist", "fist", "palm", "fist", "fist", "fist".
[0124] In the first second cycle, the detection time of the first first cycle is 0s to 0.5s, that is, the current gestures of 1 to 5 frames are recorded, which are "fist", "slap", "fist", "thumb", and "fist" in sequence; the second first cycle in the second cycle records the current gestures of 2 to 6 frames, which are "slap", "fist", "thumb", "fist", and "slap" in sequence; the third first cycle in the second cycle records the current gestures of 3 to 7 frames, which are "fist", "thumb", "fist", "slap", and "fist" in sequence; the fourth first cycle in the second cycle records the current gestures of 4 to 8 frames, which are "thumb", "fist", "slap", and "fist" in sequence; the fifth first cycle in the second cycle records the current gestures of 5 to 9 frames, which are "fist", "slap", "fist", "fist", and "slap" in sequence; the sixth first cycle in the second cycle records the current gestures of 6 to 10 frames, which are "slap", "fist", "fist", "slap", and "fist" in sequence.
[0125] According to the method described above, it is determined in turn whether the six air gestures in the first period meet the first preset condition.
[0126] If the six first-cycle air gestures in the first second cycle all meet the first preset condition at the same time, then the air gesture is determined to be a valid static gesture; if one of the six first-cycle air gestures in the first second cycle does not meet the first preset condition, then the air gesture needs to continue to be detected. When continuing to detect, the system will detect gestures of 2 to 11 frames in the second second cycle, and then continue to detect whether each first cycle meets the first preset condition in the second second cycle. If not, continue to detect; if satisfied, stop detecting, or perform related operations.
[0127] This embodiment adopts a process of periodically and continuously iterating with the second period as the rolling unit and the frame as the basic unit, and the detection of each first period within the second period is periodically and continuously iterated with the frame as the basic unit. It should be noted that in some other embodiments, the second period can also be used as the basic time unit for continuous detection, and the detection of each first period within the second period is carried out with the first period as the basic time unit for continuous periodic detection.
[0128] The recognition method described in the present application can not only judge the air gesture according to the preset conditions to determine whether the air gesture is valid, but also judge whether it is a corresponding valid gesture type according to the type of the air gesture. By determining the validity of the air gesture through conditions, it can avoid the problems of false detection and frame loss in the air gesture recognition system, and it can also avoid the problems of gesture recognition errors and triggering wrong instructions due to invalid air gestures, thereby improving the effectiveness and accuracy of air gesture recognition and reducing the error rate of gesture recognition.
[0129] In some embodiments, when the air gesture is a variable gesture, it is necessary to continue to determine whether the variable gesture is valid. A valid variable gesture is a variable gesture formed by changing from a start gesture to an end gesture within a preset time. The specific preset time length is set based on experience, but it must be greater than the preset first period.
[0130] In this embodiment, the effective change-type gesture is a moving gesture formed by changing from an effective "slap" (preset start gesture) to an effective "fist" (preset end gesture) within a preset third period. The third period is a preset time period corresponding to the change-type gesture, and the third period includes: a pre-sequence period and an end period according to the time sequence; wherein the pre-sequence period includes a start period and a middle period. When the third period is preset, the duration of the start period, the middle period and the end period has also been preset.
[0131] refer to Figure 3 When the air gesture is a modified gesture, it is necessary to continue to determine whether the modified gesture is valid, and the following operations need to be performed:
[0132] Step S301: Based on a preset third cycle, the air gesture is continuously and periodically detected in chronological order, and the current gesture of each frame within the end period of each third cycle is recorded.
[0133] Step S302, in response to determining that the end time period of the third cycle meets the preset end condition, based on the preset detection cycle, continuous periodic detection is performed in reverse chronological order within the preceding time period of the third cycle, and the current gesture of each frame in each detection cycle is recorded.
[0134] Step S303: In response to a recorded detection cycle meeting a preset start condition, and a duration between the detection cycle and the end period meeting a preset duration, determining that the air gesture is a valid variation gesture.
[0135] Among them, the preset end condition is: each of the first period's air gestures within the end time period meets the first preset condition; and the target gesture within the end time period is a preset end gesture.
[0136] Among them, the preset start condition is: each of the first period's air gestures within the detection period meets the first preset condition; and the target gesture within the detection period is a preset start gesture.
[0137] Specifically, the preset third period is used as a scrolling unit, and the air gesture is continuously scrolled and detected and recorded in chronological order, and the detection process continues in chronological order. Similar to the first period, the preset third period is used as a scrolling unit, and there are two ways to perform continuous periodic detection of the air gesture in chronological order: the first way is to use the preset third period as a scrolling unit, and perform periodic continuous iteration based on the frame as the basic unit; the second way is to use the third period as a scrolling unit, and perform continuous periodic detection based on the third period as the basic unit. In the actual detection process, the specific method to be used for continuous detection is determined according to actual needs, and is not limited here.
[0138] In the process of continuous detection with the third period as the scrolling unit, it is determined whether the air gestures in each first period within the end time period of the third period meet the first preset condition. If each air gesture in the first period within the end time period of a third period meets the first preset condition, then the next step S302 is executed; if at least one air gesture in the first period within the end time period of a third period does not meet the first preset condition, then step S301 needs to be repeated until the third period whose end time period meets the preset end condition is detected, and then step S302 is entered.
[0139] When the end time period of a certain third cycle meets the preset end condition, the end time period is determined to be the effective end time period of the change type gesture. After the effective end time period is determined, it is necessary to determine the effective start time period of the change type gesture. The duration of the effective start time period is the same as the duration of the start time period of the preset third cycle, but the position of the effective start time period is uncertain, so it is necessary to determine the position of the effective start time period according to steps S302 and S303.
[0140] Specifically, after the effective end time period detection is completed, in the preceding time period, in reverse chronological order, with the detection period as the scrolling unit, gestures in multiple detection periods are continuously detected and recorded. And continue to detect whether the gestures of each first period in each detection period meet the first preset condition. If the gestures of each first period in a certain detection period meet the first preset condition, then stop the detection in reverse chronological order, and determine that the detection period is the effective start time period of the variable gesture; if there is a first period of air gesture in a certain detection period that does not meet the first preset condition, then it is necessary to repeat this step until a detection period that meets the preset start condition is detected, and it is determined that this detection period is the effective start time period of the third period. At this time, the position of the effective start time period is determined. Continue to determine whether the duration between the effective start time period and the effective end time period meets the preset duration.
[0141] Among them, the duration of the detection cycle is the same as the duration of the starting period of the preset third cycle. Based on the continuous periodic detection of the detection cycle in reverse chronological order within the previous period, the detection cycle that meets the preset start conditions can be finally determined, that is, the effective starting period.
[0142] After the effective end time period and the effective start time period are determined in sequence, the duration of the transition time period between the effective start time period and the effective end time period (hereinafter referred to as transition duration) is calculated. The transition duration meets the preset duration, that is, the transition duration is less than or equal to the preset duration, and the preset duration is the duration of the preset intermediate time period.
[0143] When the transition duration is less than or equal to the preset duration, the air gesture is determined to be a valid change gesture; when the transition duration is greater than the preset duration, the transition duration is determined to be invalid, and re-detection starts from step S301.
[0144] In this embodiment, the acquired air gesture is a mobile gesture that changes from a "slap" to a "fist" as an example, and the corresponding valid mobile gesture is a change from a valid "slap" to a valid "fist" within the preset third period. Assuming that the preset third period is 4s, the start period (i.e., the detection period) is 1s, the middle period is 2s, the end period is 1s, and the preset first period is 0.5s.
[0145] When the system detects the "slap" gesture, it will continue to detect the gesture in a 4-second cycle. For every 4 seconds, the system will recognize 40 frames of the current gesture and record the hand shape information of each frame. For every 0.5 seconds, the system will recognize 5 frames of the current gesture.
[0146] Taking the start detection time as 0s as an example, based on the preset third cycle of 4s for continuous detection, the system will detect the hand shape from 0s to 4s in the first fourth cycle, and record a total of 1 to 40 frames of the current gesture.
[0147] In the first third cycle, detect the air gestures of each first cycle in the end period of the third cycle, that is, in the detected 0s to 4s, determine the air gestures of each 0.5s (i.e., each first cycle) in the end period of this 4s, i.e., 3s to 4s, and determine whether the air gestures in each 0.5s meet the first preset condition. Continue to detect until the third cycle with the end period meeting the condition is detected, and then proceed to the next step.
[0148] Assuming that it is detected that the end period of the first third cycle (i.e., 0s to 4s) meets the preset end condition, starting from the end period of the first third cycle (3s to 4s, recording the current gesture of frames 31 to 40), within the preceding time period, the detection cycle (1s) is used as the scrolling unit, and the detection is performed in reverse chronological order.
[0149] The first detection cycle is 3s to 2s (recording the current gesture of frames 30 to 21). Then, in the first detection cycle, each first cycle (0.5s, 5 frames) within the first detection cycle 3s-2s is detected in reverse chronological order to see if it meets the first preset condition. If it does, it is determined that this detection cycle is valid and proceeds to the next step. If it does not meet the condition, continue to detect until a detection cycle that meets the condition is detected, and then proceed to the next step.
[0150] The process of detecting each first cycle (0.5s, 5 frames) within the first detection cycle 3s-2s (recording the current gesture from 30 to 21 frames) in reverse chronological order is as follows:
[0151] During the first first cycle, the current gesture is detected and recorded from frames 30 to 26; during the second first cycle, the current gesture is detected and recorded from frames 29 to 25; during the third first cycle, the current gesture is detected and recorded from frames 28 to 24; during the fourth first cycle, the current gesture is detected and recorded from frames 27 to 23; during the fifth first cycle, the current gesture is detected and recorded from frames 26 to 22; during the sixth first cycle, the current gesture is detected and recorded from frames 25 to 21.
[0152] Only when the current gestures of the six first cycles within the first detection cycle all meet the first preset condition, the first detection cycle meets the preset start condition and the next step can be performed.
[0153] Assuming that the Nth detection period 2s-1s meets the preset start condition, the intermediate period is calculated. The detection period is 1s-2s, and the end period is 3s-4s, then the intermediate period is 2s-3s, that is, the intermediate period is 1s, which is less than the preset intermediate period 2s, then the intermediate period meets the preset intermediate condition, and the air gesture is determined to be a valid mobile gesture.
[0154] This embodiment adopts a process of periodically and continuously iterating with the third period as the rolling unit and the frame as the basic unit, and the detection of each first period within the third period is periodically and continuously iterated with the frame as the basic unit. It should be noted that in some other embodiments, the third period can also be used as the basic time unit for continuous detection, and the detection of each first period within the third period is carried out with the first period as the basic time unit for continuous periodic detection.
[0155] In the related mobile gesture recognition methods, the start time period is usually detected first, and after determining that the start time period is valid, the preset intermediate time period is continued to be detected. After determining that the preset intermediate time period is valid, the end time period is continued to be detected. However, in actual operation, the gesture of the operator in the start time period is more standard and easier to detect, but in the end time period, the operator may have a non-standard gesture due to time or human factors, which is more difficult to detect. Therefore, in the detection process of mobile gestures, it often occurs that the target start gesture is detected in the start time period, and the target end gesture of the end time period cannot be detected after continuing to detect for a long time, resulting in a long continuous detection time and low detection efficiency. However, using the recognition method described in the present application, the gesture of the end time period that is more difficult to detect is first detected. After detecting the target end gesture that meets the conditions, the gesture of the target start time period that is easier to detect is detected, so as to avoid wasting too much detection time and improve the detection efficiency and detection accuracy. In addition, the recognition method does not require the gesture in the intermediate time period, which simplifies the detection process and saves time cost.
[0156] In some embodiments, when the air gesture is a moving gesture, after determining that the air gesture is a valid air gesture, detection continues until the air gesture reaches a preset termination condition; wherein the preset termination condition includes: the movement amount of the air gesture reaches a preset movement value and / or the air gesture of the first period does not meet the first preset condition.
[0157] Specifically, an effective moving gesture is to keep the shape of the hand unchanged and perform actions such as rotation and translation for a period of time.
[0158] The movement amount includes the movement distance and / or the rotation angle. The preset movement value includes the preset movement distance and / or the preset rotation angle. When the movement distance of the air gesture reaches the preset movement distance, and / or the rotation angle reaches the preset rotation angle, the detection can be stopped. For example, when the movement distance of the preset mobile gesture is 10cm, the volume of the vehicle-mounted device can be increased by ten degrees. Then, when the movement distance of the mobile gesture of the operator reaches the preset 10cm, the preset termination condition is met and subsequent related operations can be performed.
[0159] When the air gesture is a mobile gesture, during the detection process, the current gesture that needs to be recorded includes the hand shape information, position information and posture information. During the continuous detection process, when the movement amount of the air gesture reaches the preset movement value and / or the air gesture of a first cycle no longer meets the first preset condition, it is determined that the mobile gesture stops. Then, the moving path information or rotation angle information of the mobile gesture is calculated based on the position information and posture information recorded for each frame. Subsequent related operations are performed based on the calculated moving path information or rotation angle information. For example, based on the calculated path movement information of 10 cm translation, the volume of the vehicle-mounted device controlled by the air gesture is adjusted to increase by ten degrees accordingly.
[0160] The method for recognizing air gestures provided in the present application determines the air gestures according to preset conditions when the air gesture recognition system performs air gesture recognition to determine whether the air gesture is valid, and can also determine whether it is a corresponding valid gesture type according to the type of the air gesture. By determining the validity of the air gesture through conditions, it is possible to avoid problems such as false detection and frame loss in the air gesture recognition system, as well as problems such as gesture recognition errors and triggering of wrong instructions due to invalid air gestures, thereby improving the effectiveness and accuracy of air gesture recognition and reducing the error rate of gesture recognition.
[0161] In some embodiments, the method for recognizing an air gesture described in the present application includes:
[0162] (1) Method for determining whether an operator's air gesture is effective
[0163] In view of the false detection and frame loss phenomenon in the air gesture system, this application proposes a strategy for determining whether the air gesture of the operator in the air gesture recognition system is valid by combining the knowledge of probability and based on the experimental analysis results: the same gesture must be made for Δt0 seconds (i.e., the first period preset above, such as 0.8 seconds) to confirm that the gesture made by the operator is valid. The specific steps are as follows (taking the fist hand shape as an example):
[0164] a. Rolling update records the gestures in each frame recognized by the air gesture recognition system within Δt0 seconds, and each gesture is recorded as P1, P2, P3, ..., P n-1 , P n . The “P1, P2, P3, ..., P n-1 , P n " is the hand shape information of the current gesture in each frame, such as "P1, P2, P3, ..., P n-1 , P n " can be "fist, fist, slap...slap, fist" or "1, 1, 1, ... 1, 0", where 1 or 0 is the code corresponding to each gesture.
[0165] b. Determine whether the gesture within Δt0 seconds meets the following four conditions:
[0166] Condition 1: At least one of the first frame or two consecutive frames before this period is a fist.
[0167] Condition 2: In the n frames, the ratio of the number of fist frames to the total number of frames is greater than a certain threshold. In this embodiment, the threshold is set to 1 / 2.
[0168] Condition 3: In the n frames, there are no consecutive m frames that are not fists. The value of m is adjusted and confirmed according to the system running speed. In this embodiment, it is assumed that there are 8 frames in Δt0 seconds, and m is set to 2.
[0169] Condition 4: P n Frame for fist or P n-1 , P n-2 Both frames are fists.
[0170] c. If the above four conditions are met, the operator's hand shape within Δt0 seconds is considered to be a stable fist shape, and the air gesture made by the operator is then determined to be a valid gesture.
[0171] (2) Method for determining effective static gestures
[0172] Take the fist state and stillness for Δt1 (i.e. the second period preset above) as an example:
[0173] a. Rolling update records the gestures in each frame recognized by the air gesture recognition system within Δt1 seconds (Note: Δt1>Δt0), and each gesture is recorded as PA1, PA2, PA3, ..., PA k-1 ,PA k ;
[0174] b. Using the method in step (1), iteratively update Δt0 within Δt1, and determine whether the operator's gesture within each Δt0 is a fist;
[0175] c. If the operator's gesture within each Δt0 in step b is judged to be a fist, then it is determined that the operator has completed a valid static fist gesture; if there is one or more gestures of the operator within Δt0 that are not fists, then it is determined that the gesture made by the operator is not a valid static gesture.
[0176] (3) Method for determining effective variable gestures
[0177] A valid change gesture is divided into three stages: the start period, the middle period, and the end period. A valid complete change gesture must meet four conditions: (1) the total time of the entire valid gesture is no more than Δt2 seconds (i.e., the third period preset above); (2) the start period is no less than Δt 2_1 seconds (i.e. the above-preset starting period); (3) the intermediate period is no greater than Δt 2_2 seconds (i.e. the above-preset intermediate period); (4) the end period is not less than Δt 2_3 seconds (i.e. the above preset end time period).
[0178] The specific steps for determining a valid moving gesture are as follows (taking the palm-to-fist gesture as an example):
[0179] a. Rolling update records the gestures in each frame recognized by the air gesture recognition system within Δt2 seconds (Note: Δt2>Δt0), and each frame gesture is recorded as PA1, PA2, PA3, ..., PA j-1 ,PA j .
[0180] b. Take the last Δt within Δt2 seconds 2_3 Seconds is the end period. Using the method in step (1), at Δt 2_3 Iterate and update Δt0 within the time, and determine whether the operator's gesture is a fist in each Δt0. If the operator's gesture is determined to be a fist in each Δt0, the end time period is determined to be valid, otherwise it is invalid. If the end time period is valid, go to step c, if not, return to step a.
[0181] c. If the end time period is valid, take the last Δt within Δt2 seconds 2_3 Frame reverse rolling update Δt outside seconds 2_1 , at Δt 2_1 Iterate and update Δt0 within the time, and judge whether the operator’s gesture within each Δt0 is a slap. 2_1 If the operator's gesture is judged as a slap within each Δt0 of the time, the reverse rolling update Δt is stopped. 2_1 , determine this Δt 2_1 If the valid start time period is found, the process goes to step d. If no valid start time period is found, the process goes back to step a.
[0182] d. Δt determined in steps b and c 2_3 , Δt 2_1 The period between is defined as the middle period, denoted as Δt 2_2_1 If Δt 2_2_1 >Δt 2_2 or Δt 2_1 +Δt 2_2_1 +Δt 2_3 >Δt2, the intermediate period is determined to be invalid, and the process returns to step a. 2_2_1 <Δt 2_2 And Δt 2_1 +Δt 2_2_1 +Δt 2_3 <Δt2, the middle period is determined to be valid, that is, the entire palm-to-fist gesture is a valid change gesture.
[0183] (4) Method for determining effective mobile gestures
[0184] In this application, the calculation method and steps of the moving gesture are as follows (taking the moving scissor hand gesture as an example):
[0185] a. Scroll and update the record of gesture and hand position coordinate information in each frame recognized by the air gesture recognition system within Δt0 seconds.
[0186] b. Use the same method as in step (1) to determine whether the operator's gesture within Δt0 is a scissor hand gesture.
[0187] c. If the operator's hand shape is determined to be a scissor hand for the first time within Δt0, it is determined to enter the moving scissor hand gesture, and the hand movement distance is calculated based on the position coordinate information of the hand in each frame. If the operator's hand shape is no longer a scissor hand within Δt0 during iterative update, it is determined that the moving scissor hand gesture is ended and the calculation of the movement distance is stopped.
[0188] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.
[0189] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0190] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a device for recognizing air gestures.
[0191] refer to Figure 4 , the air gesture recognition device comprises:
[0192] The detection and recording module 401 is configured to continuously and periodically detect the air gesture in a time sequence based on a preset first period, and record the current gesture of each frame in each of the first period;
[0193] The determination module 402 is configured to determine that the air gesture in the first period meets a first preset condition and to determine that the air gesture is a valid target gesture; wherein the first preset condition includes: the proportion of the number of frames in which the current gesture is the target gesture in all frames meets a preset threshold.
[0194] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0195] The device of the above-mentioned embodiment is used to implement the corresponding air gesture recognition method in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.
[0196] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for recognizing air gestures described in any of the above embodiments is implemented.
[0197] Figure 5A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.
[0198] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0199] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.
[0200] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0201] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0202] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).
[0203] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0204] The electronic device of the above-mentioned embodiment is used to implement the corresponding air gesture recognition method in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.
[0205] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method for recognizing air gestures as described in any of the above embodiments.
[0206] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0207] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method for recognizing air gestures as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0208] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.
[0209] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.
[0210] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
[0211] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.
Claims
1. A method for identifying air gestures, characterized in that: include: Based on a preset first cycle, continuously and periodically detect the air gesture in time sequence, and record the current gesture of each frame in each of the first cycle; In response to determining that the air gesture in the first period meets a first preset condition, determining that the air gesture is a valid air gesture; The first preset condition includes: the proportion of the number of frames in which the current gesture is the target gesture in all frames meets a preset threshold; The method further includes: based on a preset third cycle, continuously and periodically detecting the air gesture in a time sequence, and recording the current gesture of each frame in the end time period of each third cycle; wherein the preset third cycle includes, in time sequence, a preceding time period and the end time period; In response to determining that the end time period of the third cycle meets the preset end condition, based on the preset detection cycle, continuously perform periodic detection in the preceding time period of the third cycle in reverse chronological order, and record the current gesture of each frame in each detection cycle; In response to a recorded detection period meeting a preset start condition, and a duration between the detection period and the end period meeting a preset duration, determining that the air gesture is a valid variation gesture; The preset end condition is: each of the first period of air gestures in the end period meets the first preset condition; the target gesture in the end period is a preset end gesture; The preset start condition is: each of the first period's air gestures within the detection period meets the first preset condition; and the target gesture within the detection period is a preset start gesture.
2. The identification method according to claim 1, characterized in that: The first preset condition also includes: The current gesture of at least one frame of a first preset number of consecutive frames before the start frame or the first cycle is the target gesture; and / or In all frames, there are no consecutive frames greater than or equal to a preset number of frames in which the current gesture is not the target gesture; and / or The current gesture of the end frame is the target gesture or the current gestures of a second preset number of consecutive frames before the end frame are all the target gestures.
3. A device for identifying gestures in the air, characterized in that: include: A detection and recording module is configured to continuously and periodically detect the air gesture in a time sequence based on a preset first period, and record the current gesture of each frame in each of the first period; Based on a preset third cycle, the air gesture is continuously and periodically detected in time sequence, and the current gesture of each frame in the end time period of each third cycle is recorded; wherein the preset third cycle includes, in time sequence, a preceding time period and the end time period; A determination module, configured to determine that the air gesture in the first cycle meets a first preset condition, and to determine that the air gesture is a valid target gesture; wherein the first preset condition includes: the proportion of the number of frames in which the current gesture is the target gesture in all frames meets a preset threshold; in response to determining that the end period of the third cycle meets a preset end condition, based on a preset detection cycle, continuous periodic detection is performed in reverse chronological order within the preceding time period of the third cycle, and the current gesture of each frame in each detection cycle is recorded; in response to the recorded detection cycle meeting a preset start condition, and the duration between the detection cycle and the end period meets a preset duration, the air gesture is determined to be a valid variable gesture; The preset end condition is: each of the first period of air gestures in the end period meets the first preset condition; the target gesture in the end period is a preset end gesture; The preset start condition is: each of the first period's air gestures within the detection period meets the first preset condition; and the target gesture within the detection period is a preset start gesture.
4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 2 is implemented.
5. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 2.
