A method for gesture recognition and related products

Abstract

The embodiments of the present invention provide a method for gesture recognition and a related product thereof, the method comprising: acquiring a video frame containing a hand in real time; detecting whether the hand in the video frame contains a predetermined trigger action; and performing gesture recognition in response to detecting the predetermined trigger action. According to the embodiments of the present invention, gesture recognition is started only after the predetermined trigger action is detected, so that the gesture recognition after the trigger condition can be focused, so that the recognition efficiency and accuracy are high.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of computer vision and human-computer interaction. More specifically, embodiments of the present invention relate to a method for gesture recognition, an apparatus for gesture recognition, and a computer-readable storage medium. Background Art

[0002] This section is intended to provide background or context for embodiments of the invention set forth in the claims. The descriptions herein may include concepts that may be explored, but not necessarily concepts that have been previously thought of or explored. Therefore, unless noted herein, the content described in this section is not prior art with respect to the specification and claims of the present invention, and is not admitted to be prior art by inclusion in this section.

[0003] Gestures are hand postures, which refer to the specific movements and positions of people when they use their arms. Gestures can be used for expression. They are one of the earliest and still widely used communication tools used by humans. In the fields of human-computer interaction and computer vision, gesture interaction technology can control devices through gestures. Gesture interaction can be applied in interactive scenarios such as automobiles, home appliances, mobile phones, AR / VR, games, marketing, education, and government affairs. For example, in the application scenario of intelligent learning, gesture interaction can be used for intelligent reading, that is, the content to be read is determined by gestures.

[0004] The core technology of gesture interaction is gesture recognition, which generally includes: first, capturing images or videos through an optical system to collect gesture information; then, based on the image processing algorithm, identifying the user's intention based on the collected gesture information.

[0005] Specifically, gesture recognition technology may include one or more of hand / finger / hand key point detection, hand segmentation, target tracking, sequence prediction, sequence logic judgment and other technologies. For example, the Chinese patent application with publication number CN111324201A discloses a reading method, device and system based on somatosensory interaction. The technical solution of this patent document is used in the field of intelligent reading, in which a variety of interactive gestures are designed, including single-pointing, double-pointing, single-finger clicking twice in a row, single-finger circle, palm flat, etc.

[0006] The main drawback of this solution is that it can only recognize gestures of one hand. If there is a stationary hand in the recognition range, it may interfere with gesture recognition, thereby affecting the accuracy of recognition. In addition, even for the recognition of a single hand, there are many cases of misjudgment. In some cases, such as when an error occurs when executing a longer gesture (such as drawing a circle), the device cannot terminate the process in time and may require a long wait. Furthermore, some technical solutions in the prior art rely on higher quality images, so the requirements for the camera device are also higher, which increases the overall equipment cost. Summary of the invention

[0007] In the prior art, gesture recognition has the above-mentioned drawbacks. Therefore, a method for gesture recognition is very much needed to solve at least one of the above-mentioned drawbacks. At the same time, the present invention also provides a device for gesture recognition and a computer-readable storage medium.

[0008] In this context, embodiments of the present invention are intended to provide a method for gesture recognition and related products.

[0009] In a first aspect of an embodiment of the present invention, a method for gesture recognition is provided, comprising: acquiring a video frame containing a hand in real time; detecting whether the hand in the video frame contains a predetermined trigger action; and performing gesture recognition in response to detecting the predetermined trigger action.

[0010] In one embodiment, detecting whether the hand in the video frame includes a predetermined trigger action includes: detecting fingertip information in the video frame; and determining whether the hand in the video frame includes a predetermined trigger action based on the fingertip information.

[0011] In another embodiment, detecting the fingertip information in the video frame includes: detecting the position of the fingertip; and detecting the movement trajectory of the fingertip.

[0012] In another embodiment, detecting the movement trajectory of the fingertip includes: estimating the predicted position of the fingertip in the next frame based on the historical position of the fingertip; and determining the trajectory of the fingertip in time series based on the predicted position and the detected position of the fingertip in the next frame.

[0013] In yet another embodiment, estimating the predicted position of the fingertip in the next frame based on the historical position includes: estimating the predicted position using a second-order difference method or an extrapolation method.

[0014] In one embodiment, based on the predicted position and the detected position of the fingertip in the next frame, determining the trajectory of the fingertip in time sequence includes: in response to the difference between the predicted position and the detected position of the fingertip in the next frame being within a set range, adding the detected position of the fingertip in the next frame to the trajectory of the fingertip in time sequence; in response to the difference between the predicted position and the detected position of the fingertip in the next frame being outside the set range, adding the predicted position of the fingertip in the next frame to the trajectory of the fingertip in time sequence.

[0015] In another embodiment, detecting whether the hand in the video frame includes a predetermined trigger action includes: if it is detected that the fingertip stays within a predetermined range in a preset number of video frames, determining that the hand in the video frame includes a predetermined trigger action.

[0016] In yet another embodiment, if it is detected that a plurality of fingertips remain within a predetermined range in a preset number of video frames, it is determined that the predetermined trigger action included in the hand in the video frame is a pointing trigger action.

[0017] In yet another embodiment, in response to detecting the predetermined trigger action, performing gesture recognition includes: determining that the predetermined trigger action included in the hand in the video frame is a pointing trigger action, returning a current trigger state and returning current pointing coordinates.

[0018] In one embodiment, if it is detected that a single fingertip stays within a predetermined range within a preset number of video frames, it is determined that the predetermined trigger action included in the hand in the video frame is a circle drawing trigger action.

[0019] In another embodiment, in response to detecting the predetermined trigger action, performing gesture recognition includes: in response to detecting a cancel circle drawing condition, terminating the current gesture recognition; and / or in response to detecting a complete circle drawing condition, returning the gesture recognition result.

[0020] In yet another embodiment, the conditions for cancelling the circling include active conditions for cancelling the circling and passive conditions for cancelling the circling.

[0021] In yet another embodiment, the active cancellation of the circle drawing condition includes any one of the following: a multi-finger open gesture, a fist gesture, a swing gesture, and a move-out-of-recognition-area gesture.

[0022] In one embodiment, the passive cancellation of the circle drawing condition includes: detecting that the circle drawing action stops.

[0023] In another embodiment, the condition for completing the circle drawing includes any one of the following: the movement trajectory of the fingertip passes through the starting point of the movement trajectory; or the movement trajectory of the fingertip passes through any point of the movement trajectory.

[0024] In yet another embodiment, the movement trajectory of the fingertip passing through the starting point of the movement trajectory includes: the end point of the movement trajectory is within a preset distance from the starting point of the movement trajectory.

[0025] In yet another embodiment, in response to detecting that the circle drawing condition is completed, returning the gesture recognition result includes: returning the current trigger state and returning the current circle drawing trajectory.

[0026] In a second aspect of an embodiment of the present invention, there is provided an apparatus for gesture recognition, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, wherein when the program instructions are loaded and executed by the processor, the processor executes the method for gesture recognition according to any one of the first aspect of the embodiment of the present invention.

[0027] In a third aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which program instructions are stored. When the program instructions are loaded and executed by a processor, the processor executes the method for gesture recognition according to any one of the first aspects of the embodiments of the present invention.

[0028] According to an embodiment of the present invention, gesture recognition will only start after a predetermined trigger action is detected, that is, gesture recognition will only be performed when the trigger condition is met, and gesture recognition will not be performed before the trigger condition is met. In other words, before the trigger condition is met, only a specific trigger action needs to be detected, and other actions other than the trigger action do not need to be judged, so that the gesture recognition after the trigger condition can be focused, so the recognition efficiency and accuracy are high.

[0029] Furthermore, the design based on the cancellation logic can make users more flexible in the circle drawing process. For example, users can use the corresponding gesture to terminate the circle drawing process at any time, and the device can detect it in time to prepare for a new round of detection; and when the algorithm fails, the circle drawing gesture recognition process can be jumped out in time to restart the detection.

[0030] Furthermore, various logical judgments for gesture interaction only depend on the coordinates of the fingertips, so the application is simple and efficient. The fingertip coordinates can be obtained by a target detection model known in the prior art or developed in the future, so only an ordinary monocular camera is needed to meet the requirements of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0032] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0033] Figure 2 The flowchart of a method 200 for gesture recognition according to an embodiment of the present invention is schematically shown;

[0034] Figure 3 A diagram schematically shows a relationship between a detection means and corresponding fingertip information according to an embodiment of the present invention;

[0035] Figure 4 A flowchart of a method for gesture recognition according to an embodiment of the present invention is schematically shown; and

[0036] Figure 5 A schematic block diagram of a device 1000 for gesture recognition according to an embodiment of the present invention is schematically shown;

[0037] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0038] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0039] Figure 1 1 is a block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention. Figure 1As shown, the computing system 100 may include: a central processing unit (CPU) 101, a random access memory (RAM) 102, a read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, the RAM 102, the ROM 103, the hard disk controller 105, the keyboard controller 106, the serial controller 107, the parallel controller 108, and the display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that Figure 1 The structural block diagram is only for the purpose of illustration, rather than limiting the scope of the present invention. In some cases, some equipment may be added or reduced according to specific circumstances.

[0040] It is known to those skilled in the art that the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit", "module", "unit" or "system". In addition, in some embodiments, the present invention may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0041] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0042] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0043] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0044] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0045] The following will describe the implementation of the present invention with reference to the flowchart of the method of the embodiment of the present invention and the block diagram of the device (or system). It should be understood that each square frame of the flowchart and / or block diagram and the combination of square frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby producing a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the function / operation specified in the square frame in the flowchart and / or block diagram.

[0046] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0047] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0048] According to an embodiment of the present invention, a method for gesture recognition and a related product thereof are provided. In addition, any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0049] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention. SUMMARY OF THE INVENTION

[0051] The inventors found that the reason why there are many misjudgments in the prior art is that there are no clear trigger conditions in the existing gesture recognition schemes, so various meaningless gestures may be recognized, which ultimately leads to many misjudgments. If clear trigger conditions are added and recognition is performed after the clear trigger conditions are detected, the efficiency and accuracy of recognition can be improved.

[0052] The inventors also found that if an error occurs when executing a long gesture (such as drawing a circle), the device may not be able to terminate the process in time, which may cause a long waiting time and make the interactivity worse. If a termination condition is added and gesture recognition is stopped after the termination condition is detected, a new round of gesture recognition process can be started in time, and the interactivity is better.

[0053] The inventors also discovered that by tracking the hand that performs gesture interaction, the hand performing gesture interaction can be determined while eliminating interference from other hands, thereby allowing multiple hands to perform gesture interaction simultaneously without restricting the user to use only one hand.

[0054] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0055] Exemplary Methods

[0056] Reference below Figure 2 A method for gesture recognition according to an exemplary embodiment of the present invention is described. It should be noted that the embodiments of the present invention can be applied to any applicable scenario. For example, intelligent learning, assisted learning, and intelligent reading, etc.; as long as it is an application scenario that can use gesture interaction, the method introduced in the present invention can be used.

[0057] Figure 2A flowchart of a method 200 for gesture recognition according to an embodiment of the present invention is schematically shown, which includes step S201, step S202 and step S203.

[0058] In step S201, a video frame containing a hand is acquired in real time. "Acquisition in real time" means that the video frame needs to be continuously identified and judged, for example, each video frame is processed or the video frame is processed at intervals. In addition, depending on the different acquisition devices or sensors used, the acquired video frames can be infrared images, RGB images, RGB-D images, etc. In this embodiment, since the equipment cost for acquiring RGB images is relatively low and the recognition method does not have high requirements on image quality, RGB images are preferably used.

[0059] For example, the acquisition device may be a fixed camera that can capture a fixed recognition area, or a movable camera that can capture a recognition area corresponding to a moving target.

[0060] It should be noted that in step S201, the video frame containing the hand has hand information, but it does not mean that every video frame must contain hand information. For example, in some cases, the hand may quickly leave the recognition area, resulting in a lack of hand information in some video frames.

[0061] In step S202, it is detected whether the hand in the video frame includes a predetermined trigger action. The predetermined trigger action is a designed gesture or a group of gestures. As an example, the predetermined trigger action can be described in the product manual or in the product guide interface to help the user understand the predetermined trigger action.

[0062] In step S203, in response to detecting the predetermined trigger action, gesture recognition is performed. Detecting the predetermined trigger action means the start of gesture recognition. If there are other gestures before the predetermined trigger action is detected, gesture recognition will not be performed. In other words, when the user makes the predetermined trigger action, it means that the user is about to start making a gesture with a certain meaning.

[0063] Based on this, the present invention starts gesture recognition only after detecting a predetermined trigger action, that is, gesture recognition is performed only when the trigger condition is met, and gesture recognition is not performed before the trigger condition is met. In other words, before the trigger condition is met, only a specific trigger action needs to be detected, and no other actions other than the trigger action need to be judged, so that the gesture recognition after the trigger condition can be focused, so the recognition efficiency and accuracy are high.

[0064] In one embodiment, step S202 of detecting whether the hand in the video frame includes a predetermined trigger action may include: detecting fingertip information in the video frame; and determining whether the hand in the video frame includes a predetermined trigger action based on the fingertip information.

[0065] There are many types of hand information, for example, it may include fingers, palms, joints, etc. In the following embodiments, pointing gestures and circle gestures are mainly introduced, so the fingertips of the fingers are used as the recognition objects.

[0066] Figure 3 The relationship diagram of the detection means and the corresponding fingertip information according to an embodiment of the present invention is schematically shown. Figure 3 As shown, the fingertip information includes the position and category 302 of the fingertip, and the movement trajectory 304 of the fingertip. As an example, the position and category 302 of the fingertip can be obtained by using a target detection model. For a target detection model, a video frame is input to it, and the position and corresponding category of all targets in the video frame can be output. The position of the target can be represented by a horizontal rectangular box, and the category of the target can include six categories: hands and five fingertips. Using the target detection model, it can be obtained whether there are hands and fingers in each video frame, as well as the positions of all hands and corresponding fingers. The target detection model can be implemented using existing technology and will not be expanded in detail here.

[0067] like Figure 3 As shown, the position and category 302 of the fingertip are obtained through fingertip detection 301, and on the basis of fingertip detection 301, the motion trajectory 304 of the fingertip can be detected. Detecting the motion trajectory 304 of the fingertip can also be called fingertip tracking 303, and the motion trajectory 304 of the fingertip can be the trajectory of the fingertip in time series. Based on this, when the device collects multiple hands, the trajectories of different hands can be distinguished; and because the trajectory of the fingertip in time series has a certain continuity, some possible fingertip detection errors can also be eliminated based on this. The details are explained below.

[0068] Specifically, fingertip tracking 303 includes: estimating a predicted position of the fingertip in the next frame based on the historical position of the fingertip; and determining a temporal trajectory of the fingertip based on the predicted position and the detected position of the fingertip in the next frame.

[0069] As an example, the position of the fingertip can be represented by the fingertip coordinates, the trajectory of the fingertip in time series can be represented by the fingertip coordinate sequence, and the predicted position of the fingertip is represented by the fingertip coordinate estimation. Wherein, estimating the predicted position of the fingertip in the next frame based on the historical position may include: using an algorithm such as a second-order difference method or an extrapolation method to estimate the predicted position. In one embodiment, if the fingertip coordinates at time t are (x t,y t ), the fingertip coordinates at time t-1 are (x t-1 ,y t-1 ), the fingertip coordinates at time t-2 are (x t-2 ,y t-2 ), then the fingertip coordinates at time t+1 are estimated to be (3x t -3x t-1 +x t-2 , 3y t -3y t-1 +y t-2 ).

[0070] Among them, based on the predicted position and the detected position of the fingertip in the next frame, determining the trajectory of the fingertip in time sequence includes: in response to the difference between the predicted position and the detected position of the fingertip in the next frame being within a set range, adding the detected position of the fingertip in the next frame to the trajectory of the fingertip in time sequence; in response to the difference between the predicted position and the detected position of the fingertip in the next frame being outside the set range, adding the predicted position of the fingertip in the next frame to the trajectory of the fingertip in time sequence.

[0071] That is to say, in order to obtain the trajectory of the fingertip in the time series, it is necessary to compare the fingertip detection position and the fingertip predicted position of the next frame. If the difference between the fingertip detection position and the fingertip predicted position of the next frame is less than a certain threshold (set range), it is considered that the match is successful, and the fingertip detection position (fingertip coordinates) of the next frame is added to the end of the fingertip trajectory in the time series; if the difference between the fingertip detection position and the fingertip predicted position of the next frame is greater than a certain threshold, it is considered that the match fails, and the fingertip predicted position (estimate of fingertip coordinates) is added to the end of the fingertip trajectory in the time series. The above threshold can be set according to the general user's finger movement speed. Among them, the reason for the failure of the match may be an error in the fingertip detection or the user moves the hand out of the recognition area. Therefore, the above fingertip tracking method can eliminate some possible fingertip detection errors, so that the fingertip tracking has a certain fault tolerance and will not fail due to individual errors in the fingertip detection. Moreover, fingertip tracking can distinguish the movement trajectories of different hands in a scene where multiple hands exist at the same time, so there is no need to restrict the user to use only one hand when performing gesture interaction, and even allow the user to perform different gesture interactions with both hands at the same time.

[0072] In addition, in one embodiment, the fingertip tracking method further includes a determination of tracking failure. For example, the determination of tracking failure includes: if no matching is successful in a plurality of consecutive (e.g., three) video frames, then the tracking is determined to be a failure. At this time, the fingertip track in the time sequence can be discarded and the detection can be restarted. In addition, a prompt can be generated at this time to prompt the user to restart the triggering.

[0073] In one embodiment, detecting whether the hand in the video frame includes a predetermined trigger action in step S202 includes: if it is detected that the fingertip stays within a predetermined range in a preset number of video frames, determining that the hand in the video frame includes a predetermined trigger action.

[0074] As an example, the predetermined trigger action may include a pointing trigger action and a circle trigger action. Figure 3 The results of fingertip detection 301 and fingertip tracking 302 are shown. If multiple fingertips (two or more fingertips) are detected to stay within a predetermined range in a preset number (three or more) of video frames, the predetermined trigger action included in the hand in the video frame is determined to be a pointing trigger action. If a single fingertip is detected to stay within a predetermined range in a preset number of video frames, the predetermined trigger action included in the hand in the video frame is determined to be a circle trigger action.

[0075] If the predetermined trigger action included in the hand in the video frame is a pointing trigger action, it indicates that the user is about to perform a pointing gesture. If the predetermined trigger action included in the hand in the video frame is a circle trigger action, it indicates that the user is about to perform a circle gesture. In other words, different predetermined trigger actions reflect different intentions of the user. For example, in this embodiment, the pointing trigger action is that two fingers (which can be the index finger and the middle finger) stay within a predetermined range (a smaller range) for a period of time (in several consecutive video frames). The circle trigger action is that a single finger (which can be the index finger) stays within a predetermined range (a smaller range) for a period of time (in several consecutive video frames). Among them, the use of a two-finger gesture for the pointing trigger action is conducive to improving the success rate of recognition. Based on this, the embodiments of the present invention can simultaneously recognize both pointing and circling gestures with the help of different gesture designs.

[0076] The above-mentioned pointing trigger action and circle trigger action can be displayed in the user guidance interface of the product to teach the user. After the predetermined trigger action is detected, the pointing gesture and circle gesture can be further recognized respectively, which is described in detail below.

[0077] Figure 4 A flowchart of a method for gesture recognition according to an embodiment of the present invention is schematically shown; wherein, detecting a pointing trigger action is represented as a pointing trigger S402, and detecting a circle trigger action is represented as a circle trigger S403.

[0078] In step S401, the fingertip detection and fingertip tracking are performed on each video frame to obtain the fingertip position (eg fingertip coordinates) and the fingertip motion trajectory (eg fingertip coordinate sequence). If a corresponding predetermined trigger action is detected, corresponding gesture recognition is performed.

[0079] In step S402, if the predetermined trigger action is detected as a pointing trigger action, pointing gesture recognition is started; in step S403, if the predetermined trigger action is detected as a circle trigger action, circle gesture recognition is started; in step S404, if no predetermined trigger action is detected, return to step S401 to continue fingertip detection and fingertip tracking.

[0080] In step S4021, the pointing gesture recognition is performed and the pointing result is returned, including: returning the current trigger state and returning the current pointing coordinates. The trigger state includes pointing trigger or circle trigger. In step S4021, the returned trigger state is pointing trigger, and the returned current pointing coordinates are the fingertip coordinates at the time of pointing trigger (step S402).

[0081] In this embodiment, the fingertip coordinates at the time of the pointing trigger are directly returned in step S4021, which is equivalent to the pointing trigger action being not only a trigger condition but also the pointing gesture itself, thereby simplifying the pointing gesture recognition process. In addition, in other embodiments, the pointing trigger condition and the pointing gesture can also be split into different actions. For example, after a period of time after the pointing trigger, if the fingertip is detected to stay again, the fingertip coordinates at this time are used as the pointing result.

[0082] In step S403, if the predetermined trigger action is detected as a circle trigger action, circle gesture recognition is started. As described above, the circle trigger action is: a single fingertip stays within a predetermined range within a preset number of video frames. In one embodiment, in step S403, the current fingertip coordinates can be recorded as the trigger position, or the starting point of the fingertip motion trajectory, that is, the starting point of the circle trajectory.

[0083] As mentioned above, when an error occurs during the execution of a longer gesture (such as drawing a circle), if the device cannot terminate the process in time, the interactivity may deteriorate. Therefore, the embodiment of the present invention adds a cancel circle condition for terminating the current gesture recognition; after detecting the cancel circle condition, the gesture recognition is stopped, and a new round of effective gesture recognition process can be started in time to improve interactivity. Based on this, the circle gesture recognition of this embodiment includes: in response to detecting the cancel circle condition, terminating the current gesture recognition; and / or in response to detecting the complete circle condition, returning the gesture recognition result.

[0084] like Figure 4 As shown, the circle gesture recognition may include steps S4301 to S4304. In step S4301, fingertip detection and fingertip tracking are continued for the current frame, and further judgment is performed based on the obtained fingertip information.

[0085] In step S4302, if the circle-drawing cancellation condition is detected, the process directly jumps to step S401 to re-detect the fingertip information and wait for a new trigger condition to be generated.

[0086] For the user, the condition of canceling the circle drawing is also a preset gesture; for the device, the condition of canceling the circle drawing is a judgment logic (cancellation logic) in the circle drawing gesture recognition process, the purpose of which is to determine whether the current circle drawing process is canceled. When the condition of canceling the circle drawing is detected, the circle drawing gesture recognition process is jumped out, the fingertip information of the next video frame is processed and the trigger condition is waited for again. If the condition of canceling the circle drawing is not detected, step S4303 is executed to determine whether the circle drawing process is completed.

[0087] Specifically, the conditions for canceling the circle drawing include active cancellation of the circle drawing condition and passive cancellation of the circle drawing condition. Active cancellation of the circle drawing condition includes any of the following: multi-finger open gesture, fist gesture, swing gesture (left and right swing or up and down swing) and gesture of moving out of the recognition area. Passive cancellation of the circle drawing condition includes: detecting that the circle drawing action stops. Among them, detecting that the circle drawing action stops includes: fingertip tracking fails or the finger cannot be detected for a period of time.

[0088] The condition for cancelling the circle drawing is a kind of cancellation logic. The addition of the cancellation logic allows users to interrupt the circle drawing process at any time when they realize that the circle drawing process is wrong, which is more flexible for users. The passive cancellation of the circle drawing condition can terminate the circle drawing gesture recognition in time in the event of an algorithm error or some special circumstances. In addition, after detecting the passive cancellation of the circle drawing condition, a prompt can be given to the user (for example, through sound or display) to prevent the wrong result from causing trouble to the user, thereby helping to improve the user experience.

[0089] In step S4303, if the circle drawing condition is not detected, it means that the user is still in the process of drawing a circle, and then the process returns to step S4031 to continue detecting fingertip information. If the circle drawing condition is detected, step S4034 is executed to return the circle drawing result.

[0090] For the user, completing the circle drawing condition is also a predetermined gesture. For the device, completing the circle drawing condition is another judgment logic (completion logic) in the circle drawing gesture recognition process, the purpose of which is to determine whether the current circle drawing process has been completed; if the circle drawing process is completed, the circle drawing result needs to be returned, if the circle drawing process is not completed, continue to detect until the circle drawing process is completed. Specifically, when the circle drawing condition is detected, step S4304 is executed to return the circle drawing result. When the circle drawing condition is not detected, jump to step S4301 to continue processing the fingertip information of the next video frame, and then execute steps S4302 to S4303 to repeatedly judge the cancellation logic and completion logic until the circle drawing process is canceled or the circle drawing process is completed.

[0091] The conditions for completing the circle drawing include conditions (1) and (2), Condition (1): the motion trajectory of the fingertip passes through the starting point of the motion trajectory. Condition (2): the motion trajectory of the fingertip passes through any point of the motion trajectory.

[0092] Specifically, condition (1) indicates that the trajectory of the user's circle has returned to the vicinity of the starting point of the fingertip's motion trajectory, and the corresponding circle area is now the area surrounded by the entire fingertip coordinate sequence. The starting point of the fingertip's motion trajectory is the trigger position corresponding to step S403. In other words, condition (1) indicates that the user has drawn a relatively standard circle, that is, an "O"-shaped circle. Condition (2) indicates that the trajectory of the user's circle is self-intersecting, and the circle area is now a closed area surrounded by the self-intersection of part of the fingertip coordinate sequence. In other words, condition (2) allows the user to use a more flexible way of drawing circles, such as a "6"-shaped circle or a "9"-shaped circle.

[0093] The specific determination of condition (1) can be determined by comparing the end point of the motion trajectory with the starting point of the motion trajectory. For example, the fingertip positions in the most recent multiple (e.g., three) video frames can be detected to determine whether the fingertip positions in these video frames are all within a preset distance from the starting point of the motion trajectory. The preset distance can be set according to actual conditions, for example, given by a pre-set threshold. If the Euclidean distance between the current fingertip position and the starting point of the motion trajectory is less than the threshold, the current fingertip is considered to be near the starting point of the motion trajectory.

[0094] The specific determination of condition (2) can be achieved by determining whether the straight line segments that are approximate to the motion trajectory of the fingertip (i.e., the circle trajectory) intersect; for example, first, for the motion trajectory of the fingertip formed during the circle gesture recognition process, the adjacent coordinate points in the fingertip coordinate sequence corresponding to the trajectory are connected to form a line segment group composed of a series of straight line segments, and the line segment group is used as an approximation of the circle trajectory. Then, the last straight line segment is used to determine whether it intersects with each of the previous straight line segments one by one. If none of them intersect, the circle trajectory does not intersect itself; if it intersects with a certain straight line segment, the circle trajectory intersects itself, and the fingertip coordinate sequence between the two intersecting straight line segments is returned as the final circle trajectory. In other embodiments, other technical means in the prior art can also be used to determine conditions (1) and (2), and the present invention is not limited to this.

[0095] In step S4304, the circle drawing result is returned, including returning the current trigger state and returning the current circle drawing track. The current trigger state is the circle drawing trigger state, and the current circle drawing track can be the circle drawing track corresponding to condition (1) or the circle drawing track corresponding to condition (2).

[0096] According to the returned results of step S4201 and step S4304, these returned results can be used for subsequent functions, wherein step S4201 can return a coordinate, and step S4304 can return a coordinate sequence. For example, in an application scenario of assisted learning, the returned coordinates or coordinate sequence can be matched with the questions, and the questions in the exercise book pointed to or circled by the user can be collected; or the returned coordinates or coordinate sequence can be matched with the paragraph in the text, and the paragraph in the text pointed to or circled by the user can be read out using speech synthesis.

[0097] It should be noted that if Figure 4 In the embodiment, the cancellation logic is judged first, and then the completion logic is judged. In other embodiments, the completion logic may be judged first, and then the cancellation logic is judged.

[0098] In summary, the various judgment logics in the method for gesture recognition in the embodiment of the present invention only rely on the coordinates of the fingertips, so it is simple and efficient to apply. The fingertip coordinates can be obtained by the target detection model in the prior art or in the future, so only an ordinary monocular camera is needed to meet the requirements of the present invention.

[0099] Exemplary Devices

[0100] After introducing the method of the exemplary embodiment of the present invention, next, refer to Figure 5 The related products of the method for gesture recognition according to the exemplary embodiment of the present invention are described.

[0101] Figure 5 A schematic block diagram of a device 1000 for gesture recognition according to an embodiment of the present invention is schematically shown. Figure 5 As shown, the apparatus 1000 for gesture recognition may include a processor 1001 and a memory 1002. The memory 1002 stores computer instructions for executing the method for gesture recognition described in the embodiment of the present invention. When the computer instructions are executed by the processor 1001, the apparatus 1000 executes the above Figure 2 or Figure 4 For example, in some embodiments, the apparatus 1000 for gesture recognition may execute a method for gesture recognition to recognize a user's gesture.

[0102] In addition, in practical applications, the device 1000 for gesture recognition may also include a collection device for acquiring video frames. In order to display the results, the device 1000 for gesture recognition may also include a display or other device or module for interacting with the user. It should be noted that the embodiments of the present invention do not limit the structure of the device 1000 for gesture recognition.

[0103] It should be noted that although several steps of the method for gesture recognition are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more steps described above can be embodied in one step. Conversely, the features and functions of one step described above can be further divided into multiple steps to be embodied.

[0104] In addition, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0105] The use of the verbs "comprise", "include" and their conjugations mentioned in the application documents does not exclude the presence of elements or steps other than those recorded in the application documents. The article "a" or "an" before an element does not exclude the presence of a plurality of such elements.

[0106] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and this division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A method for gesture recognition, characterized in that: include: Acquire video frames containing hands in real time; Detecting whether the hand in the video frame includes a predetermined trigger action, wherein the predetermined trigger action includes a pointing trigger action and a circle trigger action, the pointing trigger action is that multiple fingertips stay within a predetermined range in a preset number of video frames, and the circle trigger action is that a single fingertip stays within a predetermined range in a preset number of video frames; and In response to detecting the predetermined trigger action, performing gesture recognition; In response to detecting the predetermined trigger action, performing gesture recognition includes: If the predetermined trigger action is detected to be a pointing trigger action, then starting the pointing gesture recognition; If it is detected that the predetermined trigger action is a circle drawing trigger action, circle drawing gesture recognition is started.

2. The method according to claim 1, characterized in that Detecting whether the hand in the video frame includes a predetermined trigger action includes: Detecting fingertip information in the video frame; Determine whether the hand in the video frame includes a predetermined trigger action according to the fingertip information.

3. The method according to claim 2, characterized in that Detecting fingertip information in the video frame includes: Detecting the position of the fingertips; and The movement trajectory of the fingertip is detected.

4. The method according to claim 3, characterized in that Detecting the motion trajectory of the fingertip includes: estimating a predicted position of the fingertip in the next frame based on the historical position of the fingertip; and Based on the predicted position and the detected position of the fingertip in the next frame, a trajectory of the fingertip in time sequence is determined.

5. The method according to claim 4, characterized in that Estimating the predicted position of the fingertip in the next frame based on the historical position includes: estimating the predicted position using a second-order difference method or an extrapolation method.

6. The method according to claim 4, characterized in that Based on the predicted position and the detected position of the fingertip in the next frame, determining the trajectory of the fingertip in time sequence includes: in response to the difference between the predicted position and the detected position of the fingertip in the next frame being within a set range, adding the detected position of the fingertip in the next frame to the trajectory of the fingertip in time sequence; in response to the difference between the predicted position and the detected position of the fingertip in the next frame being outside the set range, adding the predicted position of the fingertip in the next frame to the trajectory of the fingertip in time sequence.

7. The method according to any one of claims 2 to 6, characterized in that: Detecting whether the hand in the video frame includes a predetermined trigger action includes: If it is detected that the fingertip stays within a predetermined range in a preset number of video frames, it is determined that the hand in the video frames includes a predetermined trigger action.

8. The method according to claim 7, characterized in that In response to detecting the predetermined trigger action, performing gesture recognition includes: Determine that the predetermined trigger action included in the hand in the video frame is a pointing trigger action, return the current trigger state and return the current pointing coordinates.

9. The method according to claim 7, characterized in that: In response to detecting the predetermined trigger action, performing gesture recognition includes: In response to detecting a cancel circle condition, terminating the current gesture recognition; and / or In response to detecting that the circle drawing condition is completed, a gesture recognition result is returned.

10. The method according to claim 9, characterized in that The conditions for cancelling the circle drawing include active conditions for cancelling the circle drawing and passive conditions for cancelling the circle drawing.

11. The method according to claim 10, characterized in that The active cancellation of circle drawing conditions includes any one of the following: a multi-finger open gesture, a fist gesture, a swing gesture, and a gesture of moving out of the recognition area.

12. The method according to claim 10, characterized in that The passive cancellation of the circle drawing condition includes: detecting that the circle drawing action stops.

13. The method according to claim 9, characterized in that The circle drawing conditions include any of the following: The motion trajectory of the fingertip passes through the starting point of the motion trajectory; or The motion trajectory of the fingertip passes through any point of the motion trajectory.

14. The method according to claim 13, characterized in that The movement trajectory of the fingertip passes through the starting point of the movement trajectory includes: the end point of the movement trajectory is within a preset distance from the starting point of the movement trajectory.

15. The method according to claim 9, characterized in that In response to detecting that the circle drawing condition is completed, the gesture recognition result returned includes: Returns the current trigger state and the current circle track.

16. A device for gesture recognition, characterized in that: include: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, which, when loaded and executed by the processor, enables the processor to perform the method for gesture recognition according to any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that: Program instructions are stored therein, and when the program instructions are loaded and executed by a processor, the processor executes the method for gesture recognition according to any one of claims 1 to 15.

Images