Apparatus, method and storage medium for detecting action of hand in video

Abstract

The invention relates to a device, a method and a storage medium for detecting actions of a hand in a video. The device comprises an obtaining unit which obtains information related to three-dimensional key points of a hand in each frame of image in a video, and the information related to the three-dimensional key points of the hand at least comprises the position of each joint point of the hand in a three-dimensional space; the calculation unit obtains hand state information of each frame of image based on the information related to the three-dimensional key points of the hand, and the hand state information at least comprises a knuckle bending state; an analysis unit that obtains a change in hand state information in a time slice of the video; and a determination unit that determines an action performed by the hand in the time slice on the basis of a change in the hand state information and a rule relating to the action of the hand. According to the device, reusable hand action detection can be realized by analyzing the knuckle bending state frame by frame and making a hand action rule.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video analysis, and in particular to an apparatus, method, and machine-readable storage medium for detecting hand movements in a video. Background Art

[0002] This section provides background information related to the present disclosure which is not necessarily prior art.

[0003] Hand motion detection is a hot issue in computer vision, and it has a wide range of applications in human-computer interaction, virtual reality, intelligent production and life, and other fields. As for the data source for motion detection, common feature extraction methods include extracting RGB features and hand key point features from a single frame image as spatial domain features, and extracting optical flow features from continuous multi-frame images as temporal domain features. Among them, RGB features have rich texture information and scene context perception information; hand key point features have strong spatial structure constraints, which can avoid interference from factors such as hand appearance, lighting, and viewing angle, so they can more intuitively reflect hand movements; optical flow features convey the relative motion between the camera and the observed hand in the form of inter-frame motion.

[0004] Due to the continuous development and progress of deep neural network technology in recent years, it has been widely used in the field of action detection. The overall network structure of this type of method is as follows: Figure 1 As shown in Figure 1. Although motion detection based on deep neural networks can obtain good detection results, this type of method has many limitations in actual use scenarios. The production process in the factory is one of the common application scenarios of hand motion detection technology. By detecting the hand motions of production workers, it can be determined whether their movements meet the production requirements in terms of speed, process, posture, etc., thereby improving the work efficiency of workers. As we all know, the method based on deep neural networks needs to define the categories of hand motions before model training, and different factory users are concerned about different action categories, which makes it difficult to meet the needs of different users with a unified action category definition method. If the method of defining personalized action categories for different users and training models based on the defined actions is adopted, it will lead to inefficiency and waste of computing resources. In addition, due to the user's confidentiality requirements and data collection conditions, it is difficult to collect enough data for deep neural network model training in actual production scenarios. Summary of the invention

[0005] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0006] The object of the present disclosure is to provide a motion detection device, method and machine-readable storage medium for solving one or more of the above problems.

[0007] According to one aspect of the present disclosure, there is provided a device for detecting hand movements in a video, comprising: an acquisition unit, which acquires information related to three-dimensional key points of a hand in each frame image in the video, the information related to the three-dimensional key points of the hand at least including positions of various joints of the hand in three-dimensional space; a calculation unit, which obtains hand state information of each frame image based on the information related to the three-dimensional key points of the hand, the hand state information at least including knuckle bending state; an analysis unit, which obtains changes in hand state information in a time segment of the video; and a determination unit, which determines the movements of the hand in the time segment based on the changes in the hand state information and rules related to the movements of the hand.

[0008] According to another aspect of the present disclosure, a method for detecting hand movements in a video is provided, comprising: acquiring information related to three-dimensional key points of a hand in each frame image in the video, the information related to the three-dimensional key points of the hand including at least positions of various joints of the hand in three-dimensional space; obtaining hand state information of each frame image based on the information related to the three-dimensional key points of the hand, the hand state information including at least a knuckle bending state; obtaining changes in the hand state information in a time segment of the video; and determining the movements of the hand in the time segment based on the changes in the hand state information and rules related to the movements of the hand.

[0009] According to another aspect of the present disclosure, a machine-readable storage medium is provided, which carries a program product including machine-readable instruction codes stored therein, wherein the instruction codes, when read and executed by a computer, enable the computer to perform the detection method according to the present disclosure.

[0010] By using the motion detection device, motion detection method and machine-readable storage medium according to the present disclosure, reusable hand motion detection can be achieved by analyzing the knuckle bending state frame by frame and formulating hand motion rules.

[0011] Further areas of applicability will become apparent from the description provided herein.The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure. In the drawings:

[0013] Figure 1 To illustrate the overall network structure of action detection based on neural network;

[0014] Figure 2 A block diagram illustrating the structure of a detection device according to an embodiment of the present disclosure;

[0015] Figure 3 It is a schematic diagram showing the correspondence between the key points of the hand and the knuckles;

[0016] Figure 4 A diagram illustrating a detection device according to an embodiment of the present disclosure used to detect changes in the degree of bending of a knuckle during a “grasping” action;

[0017] Figure 5 A schematic diagram illustrating a process performed by a detection device according to an embodiment of the present disclosure;

[0018] Figure 6 A flowchart illustrating another process performed by the detection device according to an embodiment of the present disclosure;

[0019] Figure 7 A schematic diagram illustrating the calculation of another knuckle bending degree;

[0020] Figure 8 A schematic diagram illustrating a detection device according to an embodiment of the present disclosure for detecting a “twisting” action;

[0021] Fig. 9 A flowchart illustrating a detection method according to an embodiment of the present disclosure; and

[0022] Fig.10 1 is a block diagram of an exemplary structure of a general personal computer in which the detection apparatus and method according to the embodiments of the present disclosure can be implemented.

[0023] Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown as examples in the drawings and are described in detail herein. It should be understood, however, that the description of specific embodiments herein is not intended to limit the present disclosure to the specific forms disclosed, but rather, the present disclosure is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the present disclosure. It should be noted that throughout the several drawings, corresponding reference numerals indicate corresponding parts. DETAILED DESCRIPTION

[0024] Examples of the present disclosure will now be described more fully with reference to the accompanying drawings.The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0025] Example embodiments are provided so that the disclosure will be exhaustive and its scope will be fully conveyed to those skilled in the art. Numerous specific details such as examples of specific components, devices and methods are set forth to provide a detailed understanding of the embodiments of the present disclosure. It will be apparent to those skilled in the art that the example embodiments may be implemented in many different forms without the use of specific details, none of which should be construed as limiting the scope of the present disclosure. In certain example embodiments, well-known processes, well-known structures and well-known techniques are not described in detail.

[0026] In order to solve one or more of the problems raised in the background technology, the present application proposes a hand action detection method consisting of reusable frame-level basic hand states and rules defined according to user needs. The basic idea is: first, a certain method is used to describe the basic hand state of a single frame; then corresponding rules are formulated according to the actions that the user needs to detect. If the rules are met, the actions corresponding to these frames are the actions that need to be detected. Basic hand states include, for example, knuckle bending state, back of hand direction, finger direction, etc.

[0027] This application is based on the description of basic hand states at the frame level, which needs to meet two conditions: first, it is reusable between different users; second, the rules for action detection can be formulated based on the description of hand states. As mentioned above, there are usually three sources of hand features: RGB features, two-dimensional or three-dimensional hand key points, and optical flow information extracted between consecutive frames.

[0028] In terms of the selection of hand feature information, compared with the other two sources of information, hand key points are the most commonly used features for hand movement detection. The hand bones composed of hand key points can intuitively reflect hand movements without being affected by background and partial occlusion. In addition, affected by the shooting angle, the state of the hand cannot be accurately judged only by two-dimensional hand key points, while three-dimensional key points can give the absolute position of the key points in three-dimensional space, which will not change due to changes in shooting angle. Therefore, this application uses three-dimensional hand key points and information related to three-dimensional hand key points to describe the basic hand state.

[0029] In addition, regarding the formulation of rules for detecting hand movements, movement detection is performed on video clips, and the movement of the hand within a period of time is determined by analyzing the changing trend of the hand state information within a period of time.

[0030] Combine the following Figure 2 The basic structure of the detection device according to the embodiment of the present disclosure is described.

[0031] Figure 2 FIG. 1 is a block diagram showing the structure of a detection device according to an embodiment of the present disclosure. Figure 2As shown, the detection device 100 according to an embodiment of the present disclosure may include an acquisition unit 110 , a calculation unit 120 , an analysis unit 130 and a determination unit 140 .

[0032] The acquisition unit 110 may acquire information related to the 3D key points of the hand in each frame of the video, and the information related to the 3D key points of the hand may include at least the positions of the joints of the hand in the 3D space. For example, the acquisition unit 110 may acquire information related to the 3D key points of the hand through a pre-trained 3D hand key point detection model.

[0033] Furthermore, the calculation unit 120 may obtain the hand state information of each frame image based on the information related to the three-dimensional key points of the hand, where the hand state information at least includes the knuckle bending state.

[0034] Further, the analysis unit 130 may obtain changes in the hand state information in the time segment of the video.

[0035] Further, the determination unit 140 may determine the motion of the hand in the time segment based on the change of the hand state information and the rules related to the motion of the hand.

[0036] Therefore, the detection device 100 according to the embodiment of the present disclosure can realize the detection of reusable hand movements by analyzing the knuckle bending state frame by frame and formulating hand movement rules.

[0037] There are many uncertainties and diversity in hand movements in actual production scenarios. Next, the present application takes two hand movements ("grasping" and "twisting") widely present in production scenarios as examples to describe the various units of the detection device 100 in more detail.

[0038] <“Grab” action>

[0039] Since the action of "grasping" can be detected by analyzing the change of the degree of knuckle bending (the example of the knuckle bending state mentioned above), the degree of knuckle bending is selected as the basic hand state to detect the action of "grasping". More specifically, the bending state of the knuckle is used as the basic hand state.

[0040] The calculation unit 120 can calculate the knuckle bending degree of each frame image based on the information about the three-dimensional key points of the hand provided by the acquisition unit 110. Specifically, the calculation unit 120 can calculate the knuckle bending degree of each frame in the video clip as the basic hand state for detecting the action of "grasping".

[0041] As an example, the degree of knuckle flexion may be represented by the average value of the angles between the four fingers other than the thumb and the root skeleton.

[0042] Figure 3It is a schematic diagram illustrating the correspondence between the key points of the hand and the knuckles. There are 21 key points on the hand, and the joints between the adjacent key points of the finger part are usually called "knuckles", such as the joints between key points 5 and 6. From the perspective of application, this application broadens the definition of knuckles: for the four key points 5, 9, 13 and 17 at the base of the four fingers except the thumb, the joints between them and the wrist joint point 0 are also defined as knuckles.

[0043] based on Figure 3 In the definition of knuckles, the angle between adjacent knuckles in three-dimensional space is used to represent the bending degree of knuckles. Since the action of "grasping" usually includes the bending of knuckles where key points 5, 9, 13 and 17 are located, the action of "grasping" can be detected by analyzing the changing trend of the angle between two segments of knuckles where these four key points are located. For example, the knuckle angle of the index finger can be represented by the angle between vector (0, 5) and vector (5, 6). The knuckle angle of the middle finger can be represented by the angle between vector (0, 9) and vector (9, 10). The knuckle angle of the ring finger can be represented by the angle between vector (0, 13) and vector (13, 14). The knuckle angle of the little finger can be represented by the angle between vector (0, 17) and vector (17, 18).

[0044] Therefore, the degree of knuckle bending can be represented by the average of the angles between vector (0, 5) and vector (5, 6), the angle between vector (0, 9) and vector (9, 10), the angle between vector (0, 13) and vector (13, 14), and the angle between vector (0, 17) and vector (17, 18).

[0045] Taking into account the inherent errors in the detection of three-dimensional hand key points, in order to improve the robustness of the method, the average value of the three largest knuckle angles among the four knuckle angles (index finger, middle finger, ring finger and little finger) can also be calculated to indicate the bending state of the knuckles (the degree of knuckle bending).

[0046] Generally speaking, the degree of bending of the knuckles during the "grasping" action will gradually increase until it becomes constant after the object is grasped. Figure 4 FIG. 1 is a diagram illustrating a detection device according to an embodiment of the present disclosure used to detect changes in the degree of bending of the knuckles during the "grasping" action. In order to better illustrate how the hand changes during the "grasping" process, Figure 4 The upper part shows three representative frames of 3D hand key points and knuckles. Figure 4 The lower part shows the change of the knuckle angle in the whole video clip. It can be seen that as the knuckle bending degree increases, the knuckle angle also gradually increases. Therefore, the analysis unit 130 can obtain the change trend of the knuckle bending degree by analyzing the change trend of the knuckle angle.

[0047] Specifically, the analyzing unit 130 analyzes the changing trend of the knuckle bending degree of the multiple frames of the video in each sliding window. If the knuckle bending degree increases for each of the multiple consecutive sliding windows, the determining unit 140 determines that the video within the time length of the multiple consecutive sliding windows is related to the grasping action of the hand. Figure 4 As shown, the video within 0 to 50 frames is related to the grabbing action of the hand. Of course, this is just an example, and the present disclosure is not limited thereto.

[0048] The positioning of the action here can be achieved by analyzing whether the knuckle angle shows an upward trend in the sliding window, and the upward trend can be judged by the Mann-Kendall test method. Figure 4 Different grayscales are used to indicate that the sliding window slides at a certain step length. The length of the sliding window remains unchanged, and there is a certain overlap between the front and rear sliding windows. Figure 4 In the example, the length of the sliding window is 20 frames, and the overlap of the front and rear sliding windows is 10 frames. Of course, this is just an example, and the present disclosure is not limited thereto. A plurality of consecutive sliding windows (0 to 50 frames) with an increasing knuckle angle are detected as potential "grasping" actions.

[0049] At the same time, since the actual production scene is relatively complex and there are various uncertainties, rules for eliminating common interferences can also be formulated. In this case, the determination unit 140 can also exclude interferences of certain non-target actions based on the hand state information.

[0050] The determination unit 140 may exclude non-target action segments from potential "grasping" actions. As mentioned above, there are inherent errors in the three-dimensional hand key points, which may cause some non-"grasping" action segments to be detected as potential "grasping" actions by the determination unit 140. There are two steps to exclude false detection.

[0051] First, the determination unit 140 determines whether the hand has undergone radial twisting during the time period detected by analyzing. Usually, the hand will not twist during the "grasping" process, so judging whether it is twisted can exclude certain non-target action segments. When the hand is twisted radially, the direction of the back of the hand will change significantly, so whether it is twisted can be judged by detecting the change in the angle between the back of the hand and the coordinate axis. The specific process is as follows: Figure 5 shown.

[0052] Figure 5 Schematic diagram illustrating a process that can be performed by a detection device according to an embodiment of the present disclosure. Figure 5 As shown, the direction of the back of the hand is represented by the normal direction of the vector formed by the wrist (key point 0) and the base of the index finger (key point 5) and the vector formed by the wrist (key point 0) and the base of the little finger (key point 17), and the change in the direction of the back of the hand is represented by the standard deviation of the angles between the direction of the back of the hand and the x-axis, y-axis and z-axis of the coordinate system in which the hand is located.

[0053] Specifically, the determination unit 140 may be configured to: determine whether the change in the direction of the back of the hand in the sliding window is greater than a first threshold; and if greater than the first threshold, determine that the action is not grabbing.

[0054] Then, the determination unit 140 can exclude certain non-target actions by analyzing whether the upward trend is significant. The error in the three-dimensional hand key point detection will cause the knuckle angles of some non-"grabbing" segments to have a weak upward trend, while the upward trend of the real "grabbing" action is more significant. The size of the slope can be used to measure whether the upward trend is significant. The specific process is as follows: Figure 6 shown.

[0055] Figure 6 1 is a flowchart illustrating another process performed by the detection device according to an embodiment of the present disclosure. Specifically, the analysis unit 130 further analyzes the slope of the sequence of knuckle bending degrees (knuckle angle sequence) within the time length of multiple consecutive sliding windows, and only when the slope is greater than the second threshold, the determination unit 140 determines that the action is grasping.

[0056] In the above example, the degree of knuckle bending is represented by the average value of the angles between the four fingers other than the thumb and the root skeleton. The present application is not limited thereto. Figure 7 As shown, the degree of knuckle bending can also be represented by the sum of the bending degrees of the three joint points of the four fingers except the thumb. When the bending degrees are summed, the bending degrees of the joints closer to the base of the fingers have a higher weight.

[0057] In other words, the knuckle angles of other parts can be used to detect micro-grasping actions. When grasping certain larger objects, the hand posture changes little during the grasping process, and this type of action is called a micro-grasping action. Micro-grasping actions are not completed by bending the knuckles where the finger base nodes are located. To detect this type of action, it is necessary to analyze the changing trend of the knuckle angles of other parts. Figure 7 The figure shows the knuckles that are of concern when considering micro-grasping actions (i.e., the three knuckles on the fingers except the thumb). Different weights are assigned to the angles of each knuckle according to the probability of being used in the "grasping" action, and the weights satisfy the condition w1>w2>w3.

[0058] After the calculation unit 120 calculates the weighted knuckle bending degree by this method, the analysis unit 130 and the determination unit 140 may perform subsequent processing according to the above description, thereby detecting a video segment related to the “grasping” action in the video.

[0059] Therefore, the detection device 100 can formulate rules for action detection based on a full analysis of the changing trend of the basic hand state during the action process, and eliminate the interference of certain non-target actions, thereby improving the feasibility and robustness of the detection method.

[0060] <Twisting action>

[0061] The calculation unit 120 can calculate the finger direction of each frame in the video clip as the basic hand state for detecting the "twisting" action. Due to the limitation of the finger's physiological structure, each finger joint on the same finger can only move in one plane. Based on this condition, the finger direction can be defined as the normal vector of the plane where the finger is located, such as Figure 8 That is, the finger direction is represented by the normal vector of the plane where the finger is located. Specifically, any two adjacent knuckles in the finger (such as Figure 8 In (b), the vector (5, 6) and the vector (6, 7) are used to calculate the normal vector of the plane in which they are located, which is the defined finger direction.

[0062] The analysis unit 130 can obtain the finger direction deviation by calculating the deviation of the finger direction between each frame image and the initial frame image in the sliding window. Usually, the "twisting" action is accompanied by a change in the finger direction, and this change can be measured by the change in the finger direction deviation between each frame and the initial frame. If the finger direction deviation in multiple consecutive sliding windows shows an increasing trend, it can be considered that the time period is a potential "twisting" action.

[0063] In addition, the calculation unit 120 can also calculate the degree of knuckle bending and the direction of the back of the hand in the sliding window. Usually, during the "twisting" action, the hand is holding the tool tightly, so the change in the degree of knuckle bending is very small. In addition, during the "twisting" action, the hand mainly rotates within the plane rather than radially twisting, so the change in the direction of the back of the hand is also very small. By judging whether the degree of change in the degree of knuckle bending and the direction of the back of the hand is very small (by judging whether the standard deviation is less than a threshold), some non-target action segments in the potential "twisting" action detected by the determination unit 140 can be excluded.

[0064] It should be noted that the calculation of the degree of knuckle bending is the same as the calculation of the degree of knuckle bending in the "grasp" action above. That is, the degree of knuckle bending represented by the average value of the angles between the four fingers other than the thumb and the root skeleton can be calculated, and the degree of knuckle bending represented by the sum of the bending degrees of the three joint points of the four fingers other than the thumb (weighted knuckle bending) can also be calculated. Similarly, the back of the hand direction is represented by the normal direction of the vector formed by the wrist and the base of the index finger and the vector formed by the wrist and the base of the little finger, and the change in the back of the hand direction is represented by the standard deviation of the angles between the back of the hand direction and the x-axis, y-axis, and z-axis of the coordinate system in which the hand is located.

[0065] In addition, when the hand is doing the "twisting" action, it needs to hold the tool tightly, so some non-target action segments can be further excluded by judging whether the average knuckle bending degree in the sliding window is greater than the threshold. The average knuckle bending degree refers to the average value of the knuckle bending degree of all frames in the sliding window.

[0066] That is, the calculation unit 120 can calculate the finger direction, knuckle bending degree and hand back direction of multiple frames of the video in the sliding window, and the analysis unit 130 analyzes the changes in the finger direction, knuckle bending degree and hand back direction calculated by the calculation unit. The determination unit 140 determines that the video within the time length of the multiple consecutive sliding windows is related to the hand twisting action if the calculation results of each sliding window in the multiple consecutive sliding windows meet the following requirements: the finger direction deviation increases; the change in the knuckle bending degree is less than the third threshold; the change in the hand back direction is less than the fourth threshold; and the average knuckle bending degree in the sliding window is greater than the fifth threshold.

[0067] In addition, although not shown, in the detection of the "twisting" action, the selection and movement of the sliding window can be similar to the detection of the "grasping" action. For example, adjacent sliding windows in a plurality of continuous sliding windows overlap in time.

[0068] Therefore, the detection device 100 can formulate rules for action detection based on a full analysis of the changing trend of the basic hand state during the action process, and eliminate the interference of certain non-target actions, thereby improving the feasibility and robustness of the detection method.

[0069] Combine the following Fig. 9 To describe the detection method according to an embodiment of the present disclosure.

[0070] like Fig. 9 As shown, the detection method according to the embodiment of the present disclosure starts at step S110. In step S110, information related to the three-dimensional key points of the hand in each frame image in the video is obtained, and the information related to the three-dimensional key points of the hand at least includes the positions of each joint point of the hand in the three-dimensional space.

[0071] Next, in step S120, hand state information of each frame image is obtained based on information related to the three-dimensional key points of the hand, and the hand state information at least includes a knuckle bending state.

[0072] Next, in step S130, changes in the hand state information in the time segment of the video are obtained.

[0073] Next, in step S140, the hand motion in the time segment is determined based on the change of the hand state information and the rules related to the hand motion.

[0074] After this, the process ends.

[0075] Specifically, for the detection of the action of "grasping", the method may further include: setting sliding windows, calculating basic hand state information in each sliding window, such as the degree of knuckle bending; analyzing the trend of the degree of knuckle bending in multiple frames of video in the sliding window; if the degree of knuckle bending increases for each of multiple consecutive sliding windows, determining that the video within the time length of the multiple consecutive sliding windows is related to the grasping action of the hand, thereby locating the actions in multiple consecutive sliding windows that meet the rules as potential detected actions; and excluding non-target actions from potential detected actions.

[0076] For excluding non-target actions, it can also be determined whether the change in the direction of the back of the hand in the sliding window is greater than a first threshold, and if it is greater than the first threshold, it is determined that the action is not grasping.

[0077] For excluding non-target actions, the slope of the sequence of knuckle bending degrees within the time length of multiple consecutive sliding windows may also be analyzed, and only when the slope is greater than a second threshold, the action is determined to be grasping.

[0078] Specifically, for the detection of the action of "twisting", the method may further include: calculating the finger direction, knuckle bending degree and back of hand direction of multiple frames of video in a sliding window; and if the calculation results of each sliding window in multiple consecutive sliding windows meet the following requirements, then determining that the video within the time length of multiple consecutive sliding windows is related to the hand twisting action: the finger direction deviation increases; the change in the knuckle bending degree is less than the third threshold; the change in the back of the hand direction is less than the fourth threshold; and the average curvature in the sliding window is greater than the fifth threshold.

[0079] Therefore, the detection method according to the embodiment of the present disclosure can realize the detection of reusable hand movements by analyzing the knuckle bending state frame by frame and formulating hand movement rules.

[0080] Various specific implementations of the above steps of the detection method according to the embodiment of the present disclosure have been described in detail above and will not be repeated here.

[0081] Obviously, each operation process of the detection method according to the present disclosure can be implemented in the form of a computer executable program stored in various machine-readable storage media.

[0082] Moreover, the purpose of the present disclosure can also be achieved in the following manner: a storage medium storing the above-mentioned executable program code is directly or indirectly provided to a system or device, and a computer or a central processing unit (CPU) in the system or device reads out and executes the above-mentioned program code. In this case, as long as the system or device has the function of executing the program, the embodiment of the present disclosure is not limited to the program, and the program can also be in any form, for example, a target program, a program executed by an interpreter, or a script program provided to an operating system, etc.

[0083] The above-mentioned machine-readable storage media include, but are not limited to: various memories and storage units, semiconductor devices, disk units such as optical, magnetic and magneto-optical disks, and other media suitable for storing information.

[0084] In addition, the technical solution of the present disclosure can also be implemented by connecting a computer to a corresponding website on the Internet, downloading and installing the computer program code according to the present disclosure into the computer, and then executing the program.

[0085] Fig.10 1 is a block diagram of an exemplary structure of a general personal computer in which the detection apparatus and method according to the embodiments of the present disclosure can be implemented.

[0086] like Fig.10 As shown, a CPU 1301 performs various processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1308 to a random access memory (RAM) 1303. In the RAM 1303, data required when the CPU 1301 performs various processes and the like is also stored as needed. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output interface 1305 is also connected to the bus 1304.

[0087] The following components are connected to the input / output interface 1305: an input device 1306 (including a keyboard, a mouse, etc.), an output device 1307 (including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage device 1308 (including a hard disk, etc.), and a communication device 1309 (including a network interface card such as a LAN card, a modem, etc.). The communication device 1309 performs communication processing via a network such as the Internet. A drive 1310 may also be connected to the input / output interface 1305 as needed. A removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed on the drive 1310 as needed, so that a computer program read therefrom is installed in the storage device 1308 as needed.

[0088] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 1311 .

[0089] It should be understood by those skilled in the art that such storage media is not limited to Fig.10 The removable medium 1311 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1311 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 1302, a hard disk included in the storage device 1308, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0090] In the system and method of the present disclosure, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present disclosure. In addition, the steps of performing the above series of processes can naturally be performed in chronological order according to the order of description, but do not necessarily need to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0091] Although the embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings, it should be understood that the embodiments described above are only used to illustrate the present disclosure and do not constitute a limitation of the present disclosure. For those skilled in the art, various modifications and changes can be made to the above embodiments without departing from the essence and scope of the present disclosure. Therefore, the scope of the present disclosure is limited only by the attached claims and their equivalent meanings.

[0092] Regarding the implementation methods including the above embodiments, the following additional notes are also disclosed:

[0093] Note 1. A device for detecting hand movements in a video, comprising:

[0094] An acquisition unit, which acquires information related to the three-dimensional key points of the hand in each frame of the video, wherein the information related to the three-dimensional key points of the hand at least includes positions of various joints of the hand in three-dimensional space;

[0095] A calculation unit, which obtains hand state information of each frame image based on information related to the three-dimensional key points of the hand, wherein the hand state information at least includes a knuckle bending state;

[0096] an analyzing unit, which obtains changes in hand state information in a time segment of the video; and

[0097] A determination unit determines the hand movement in the time segment based on the change of the hand state information and the rules related to the hand movement.

[0098] Note 2. The device according to Note 1, wherein the knuckle bending state is the degree of knuckle bending, and

[0099] The calculation unit calculates the degree of knuckle bending of each frame image based on information related to the three-dimensional key points of the hand,

[0100] The analyzing unit analyzes, for each sliding window, a trend of the degree of knuckle bending of multiple frames of the video in the sliding window, and

[0101] If the degree of knuckle bending increases for each of a plurality of consecutive sliding windows, the determining unit determines that the video within the time length of the plurality of consecutive sliding windows is related to a grasping action of the hand.

[0102] Note 3. The device according to Note 2, wherein:

[0103] The determination unit determines whether a change in the back direction of the hand in the sliding window is greater than a first threshold, and if greater than the first threshold, determines that the action is not grasping.

[0104] Note 4. The device according to Note 2, wherein:

[0105] The analyzing unit analyzes the slope of the sequence of knuckle bending degrees within the time length of the plurality of continuous sliding windows, and

[0106] Only when the slope is greater than a second threshold, the determination unit determines that the action is grasping.

[0107] Note 5. The device according to Note 1, wherein the knuckle bending state is the degree of knuckle bending, and

[0108] The calculation unit calculates the finger direction, knuckle bending degree and hand back direction of multiple frames of the video in the sliding window,

[0109] The analyzing unit analyzes the changes in the finger direction, the degree of knuckle bending, and the back of the hand direction, and

[0110] If the calculation results of each sliding window in the plurality of continuous sliding windows meet the following requirements, the determining unit determines that the video within the time length of the plurality of continuous sliding windows is related to the hand twisting action:

[0111] Increased deviation in finger direction;

[0112] The change in the degree of knuckle bending is less than a third threshold;

[0113] The change in the hand back direction is less than a fourth threshold; and

[0114] The average knuckle bending degree within the sliding window is greater than a fifth threshold.

[0115] Note 6. The device according to Note 2 or 5, wherein:

[0116] The degree of knuckle flexion is represented by the average value of the angles between the four fingers other than the thumb and the root skeleton; or

[0117] The knuckle bending degree is represented by the sum of the bending degrees of the three joint points of the four fingers except the thumb, and when the bending degrees are summed, the bending degrees of the joints closer to the base of the fingers have a higher weight.

[0118] Note 7. The device according to Note 5, wherein the finger direction is represented by a normal vector of the plane where the finger is located, and the calculation unit obtains the finger direction deviation by calculating the deviation of the finger direction between each frame image in the sliding window and the initial frame image.

[0119] Note 8. An apparatus according to Note 3 or 5, wherein the back of the hand direction is represented by the normal direction of a vector formed by the wrist and the base of the index finger and a vector formed by the wrist and the base of the little finger, and the change in the back of the hand direction is represented by the standard deviation of the angles between the back of the hand direction and the x-axis, y-axis and z-axis of the coordinate system in which the hand is located.

[0120] Note 9. A method for detecting hand movements in a video, comprising:

[0121] Acquire information related to the three-dimensional key points of the hand in each frame of the video, wherein the information related to the three-dimensional key points of the hand at least includes positions of various joints of the hand in three-dimensional space;

[0122] Obtaining hand state information of each frame of image based on information related to three-dimensional key points of the hand, wherein the hand state information at least includes a knuckle bending state;

[0123] Obtaining changes in hand state information in a time segment of the video; and

[0124] The hand movement in the time segment is determined based on the change of the hand state information and the rules related to the hand movement.

[0125] Supplement 10. The method according to Supplement 9, wherein the knuckle bending state is the degree of knuckle bending, and the method further comprises:

[0126] The degree of knuckle bending in each frame is calculated based on the information related to the 3D key points of the hand.

[0127] For each sliding window, analyzing the trend of the knuckle bending degree of multiple frames of the video in the sliding window; and

[0128] If the degree of knuckle bending increases for each of a plurality of consecutive sliding windows, it is determined that the video within the time length of the plurality of consecutive sliding windows is related to a grasping action of the hand.

[0129] Note 11. The method according to Note 10 further includes:

[0130] Determining whether a change in the direction of the back of the hand in the sliding window is greater than a first threshold; and

[0131] If it is greater than the first threshold, it is determined that the action is not grabbing.

[0132] Note 12. The method according to Note 10 further includes:

[0133] analyzing the slope of the sequence of knuckle bending degrees within the time lengths of the plurality of consecutive sliding windows, and

[0134] Only when the slope is greater than a second threshold value is the action determined to be grabbing.

[0135] Supplement 13. The method according to Supplement 9, wherein the knuckle bending state is the degree of knuckle bending, and the method further comprises:

[0136] Calculate the finger direction, knuckle bending degree and hand back direction of multiple frames of the video in the sliding window,

[0137] Analyzing changes in the finger direction, the degree of knuckle bending, and the back of the hand direction, and

[0138] If the calculation result of each sliding window in the multiple continuous sliding windows meets the following requirements, it is determined that the video within the time length of the multiple continuous sliding windows is related to the hand twisting action:

[0139] Increased deviation in finger direction;

[0140] The change in the degree of knuckle bending is less than a third threshold;

[0141] The change in the hand back direction is less than a fourth threshold; and

[0142] The average knuckle bending degree within the sliding window is greater than a fifth threshold.

[0143] Note 14. The method according to Note 10 or 13, wherein:

[0144] The degree of knuckle flexion is represented by the average value of the angles between the four fingers other than the thumb and the root skeleton; or

[0145] The knuckle bending degree is represented by the sum of the bending degrees of the three joint points of the four fingers except the thumb, and when the bending degrees are summed, the bending degrees of the joints closer to the base of the fingers have a higher weight.

[0146] Note 15. The method according to Note 14, wherein the degree of flexion of the knuckles is represented by the average value of the first three angles between the four fingers other than the thumb and the root skeleton.

[0147] Note 16. A method according to Note 13, wherein the finger direction is represented by a normal vector of the plane where the finger is located, and the finger direction deviation is obtained by calculating the deviation of the finger direction between each frame image in the sliding window and the initial frame image.

[0148] Note 17. The method according to Note 11 or 13, wherein the back of the hand direction is represented by the normal direction of the vector formed by the wrist and the base of the index finger and the vector formed by the wrist and the base of the little finger, and the change of the back of the hand direction is represented by the standard deviation of the angles between the back of the hand direction and the x-axis, y-axis and z-axis of the coordinate system in which the hand is located.

[0149] Note 18. The method according to Note 9, wherein information related to the three-dimensional key points of the hand is detected by a pre-trained three-dimensional hand key point detection model.

[0150] Note 19. The method according to Note 10, wherein the Mann-Kendall test is used to determine the trend of the curvature in each sliding window.

[0151] Note 20. A machine-readable storage medium carrying a program product including machine-readable instruction codes stored therein, wherein the instruction codes, when read and executed by a computer, enable the computer to perform the detection method described in Notes 9-19.

Claims

1. A device for detecting hand movements in a video, comprising: An acquisition unit, which acquires information related to the three-dimensional key points of the hand in each frame of the video, wherein the information related to the three-dimensional key points of the hand at least includes positions of various joints of the hand in three-dimensional space; A calculation unit, which obtains hand state information of each frame image based on information related to the three-dimensional key points of the hand, wherein the hand state information at least includes a knuckle bending state; an analyzing unit, which obtains changes in hand state information in a time segment of the video; as well as A determination unit determines the hand movement in the time segment based on the change of the hand state information and the rules related to the hand movement.

2. The device according to claim 1, wherein: The knuckle flexion state is the degree of knuckle flexion, and The calculation unit calculates the degree of knuckle bending of each frame image based on information related to the three-dimensional key points of the hand, The analyzing unit analyzes, for each sliding window, a trend of the degree of knuckle bending of multiple frames of the video in the sliding window, and If the degree of knuckle bending increases for each of a plurality of consecutive sliding windows, the determining unit determines that the video within the time length of the plurality of consecutive sliding windows is related to a grasping action of the hand.

3. The device according to claim 2, wherein: The determination unit determines whether a change in the back direction of the hand in the sliding window is greater than a first threshold, and if greater than the first threshold, determines that the action is not grasping.

4. The device according to claim 2, wherein: The analyzing unit analyzes the slope of the sequence of knuckle bending degrees within the time length of the plurality of continuous sliding windows, and Only when the slope is greater than a second threshold, the determination unit determines that the action is grasping.

5. The device according to claim 1, wherein: The knuckle flexion state is the degree of knuckle flexion, and The calculation unit calculates the finger direction, knuckle bending degree and hand back direction of multiple frames of the video in the sliding window, The analyzing unit analyzes the changes in the finger direction, the degree of knuckle bending, and the back of the hand direction, and If the calculation results of each sliding window in the plurality of continuous sliding windows meet the following requirements, the determining unit determines that the video within the time length of the plurality of continuous sliding windows is related to the hand twisting action: Increased deviation in finger direction; The change in the degree of knuckle bending is less than a third threshold; The change in the hand back direction is less than a fourth threshold; and The average knuckle bending degree within the sliding window is greater than a fifth threshold.

6. The device according to claim 2 or 5, wherein: The degree of knuckle flexion is represented by the average value of the angles between the four fingers other than the thumb and the root skeleton; or The knuckle bending degree is represented by the sum of the bending degrees of the three joint points of the four fingers except the thumb, and when the bending degrees are summed, the bending degrees of the joints closer to the base of the fingers have a higher weight.

7. The device according to claim 5, wherein: The finger direction is represented by a normal vector of the plane where the finger is located, and the calculation unit obtains the finger direction deviation by calculating the deviation of the finger direction between each frame image in the sliding window and the initial frame image.

8. The device according to claim 3 or 5, wherein: The back of hand direction is represented by the normal direction of the vector formed by the wrist and the base of the index finger and the normal direction of the vector formed by the wrist and the base of the little finger, and the change of the back of hand direction is represented by the standard deviation of the angles between the back of hand direction and the x-axis, y-axis and z-axis of the coordinate system in which the hand is located.

9. A method for detecting hand movements in a video, comprising: Acquire information related to the three-dimensional key points of the hand in each frame of the video, wherein the information related to the three-dimensional key points of the hand at least includes positions of various joints of the hand in three-dimensional space; Obtaining hand state information of each frame of image based on information related to three-dimensional key points of the hand, wherein the hand state information at least includes a knuckle bending state; Obtaining changes in hand state information in a time segment of the video; as well as The hand movement in the time segment is determined based on the change of the hand state information and the rules related to the hand movement.

10. A machine-readable storage medium carrying a program product including machine-readable instruction codes stored therein, wherein: When the instruction code is read and executed by a computer, the computer can execute the method according to claim 9.

Images