Apparatus for, method of, and computer program for detecting movement of hand in video

Abstract

To provide an apparatus for, a method of, and a storage medium for detecting movement of a hand in a video.SOLUTION: An apparatus according to the present invention includes: a unit for acquiring information relating to a 3D key point of a hand in each frame image of a video, the information relating to the 3D key point of the hand including at least a position of each joint point of the hand in the 3D space; a unit for acquiring state information on the hand in each frame image based on the information relating to the 3D key point of the hand, the state information on the hand including at least a bent state of a finger segment; a unit for acquiring a change in the state information on the hand in a time fragment of the video; and a unit for determining movement of the hand in the time fragment based on rules relating to a change in the state information on the hand and movement of the hand.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to the technical field of video analysis, and in particular to an apparatus, method and computer program for detecting hand movements in a video. [Background technology]

[0002] Hand motion detection is a common problem in computer vision, which is widely applied in fields such as human-computer interaction, virtual reality, intelligent production and life. In terms of the source of data for motion detection, common feature extraction methods include extracting RGB features and hand keypoint features as spatial domain features from single-frame images, and extracting optical flow features as time domain features from consecutive multi-frame images. Among them, RGB features have rich texture information and scene context recognition (sensing) information, hand keypoint features have relatively strong spatial structure constraints, and can avoid interference caused by factors such as hand appearance, lighting, and viewing angle, so as to more intuitively reflect hand motion, and optical flow features convey the relative motion between the camera and the observed hand in the form of frame-to-frame motion.

[0003] In recent years, deep neural network technology has been developed and improved continuously, so that it has been widely used in the field of motion detection. The overall network structure of such a method is shown in Figure 1. Although motion detection based on deep neural network can obtain relatively good detection results, such a method has many limitations in actual application scenarios. The production process in a factory is one of the common application scenarios of hand motion detection technology, and by detecting the hand motion of production workers, it can be determined whether the motion meets the production requirements in terms of speed, workflow, posture, etc., so that the working efficiency of the workers can be improved. As is well known, in the method based on deep neural network, it is necessary to define the hand motion category (also called hand motion category) before training the model. However, since the hand motion categories of interest to users in different factories are different, the method of definition by unified hand motion category is difficult to meet the needs of different users. If a method of defining personalized hand motion categories for different users and training models based on the defined motions is adopted, it may result in a decrease in efficiency and a waste of computing resources. In addition, due to user confidentiality requirements and limited data collection conditions, it is difficult to collect enough data to train a deep neural network model in real production scenes. Summary of the Invention [Problem to be solved by the invention]

[0004] It is an object of the present invention to provide a hand gesture detection device, method and computer program for solving one or more of the above mentioned problems. [Means for solving the problem]

[0005] According to one aspect of the present invention, there is provided an apparatus for detecting hand movements in a video, comprising: an acquisition unit for acquiring information about 3D key points of a hand in each frame image in a video, where the information about the 3D key points of the hand includes at least a position of each joint point of the hand in a 3D space; a computing unit for obtaining hand state information for each frame image based on information about 3D key points of the hand, the hand state information including at least flexion (curvature) states of finger segments; an analysis unit for obtaining changes in hand state information in a time fragment of the video; and It includes a determination unit for determining hand movements in a time fragment based on changes in hand state information and rules regarding hand movements.

[0006] According to another aspect of the present invention, there is provided a method for detecting hand movements in a video, the method comprising: Obtaining information about 3D key points of the hand in each frame image in the video, the information about the 3D key points of the hand including at least a position in 3D space of each joint point of the hand; Obtain hand state information for each frame image based on information about the 3D key points of the hand, the hand state information including at least a flexion state of a finger segment; Obtaining changes in hand state information in a time fragment of the video; and It includes determining hand movements in the time fragment based on the changes in the hand state information and rules regarding hand movements.

[0007] According to another aspect of the invention, a computer program is provided, which when run on a computer causes the computer to carry out the detection method according to the invention. Effect of the Invention

[0008] Using the motion detection device, motion detection method and computer program according to the present invention, the bending state of finger segments can be analyzed frame by frame and hand motion rules can be formulated, thereby realizing reusable hand motion detection. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates the overall network structure of neural network-based motion detection. [Diagram 2] FIG. 1 is a configuration diagram of a detection device according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing correspondence between hand keypoints and finger segments. [Figure 4] 13A-13C are diagrams illustrating changes in the degree of bending of finger segments in the process of detecting a "grasping" motion by a detection device according to an embodiment of the present invention. [Diagram 5] FIG. 4 is a diagram showing a process performed by a detection device according to an embodiment of the present invention. [Figure 6] 11 is a flowchart of another process performed by a detection device according to an embodiment of the present invention. [Figure 7] FIG. 13 illustrates calculation of the degree of bending of another finger segment. [Figure 8] 1A to 1C are diagrams illustrating the principle by which a detection device according to an embodiment of the present invention detects a "twist" motion. [Figure 9] 2 is a flow chart of a detection method according to an embodiment of the present invention. [Figure 10] FIG. 1 is an exemplary block diagram of a general-purpose personal computer on which a detection apparatus and method according to an embodiment of the present invention may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the preferred embodiments are merely illustrative and are not intended to limit the present invention.

[0011] In order to solve one or more of the problems described in the background art, the present invention proposes a hand movement detection method consisting of reusable frame-level basic hand states and rules defined based on user needs. The basic idea is as follows: firstly, adopt a certain method to describe the basic hand states of a single frame, then formulate corresponding rules based on the actions that the user needs to detect, and when the rules are satisfied, the actions corresponding to these frames are the actions that need to be detected. The basic hand states include, for example, the bending states of finger segments, the orientation of the back of the hand, the orientation of the fingers, etc.

[0012] The present invention is based on a frame-level basic hand state description, which needs to satisfy the following two conditions: first, it is reusable among different users; second, action detection rules can be formulated based on the hand state description. As mentioned above, there are usually three sources of hand features, namely RGB features, 2D (two-dimensional) or 3D (three-dimensional) hand keypoints, and optical flow information extracted between consecutive frames.

[0013] Regarding the selection of hand feature information, compared with the other two types of information sources, hand keypoints are the most used features for hand movement detection. A hand skeleton consisting of hand keypoints can intuitively reflect hand movements without being affected by background or local occlusion. In addition, 2D hand keypoints alone cannot accurately determine the state of the hand due to the influence of the shooting viewing angle, but 3D keypoints can provide the absolute position of the keypoints in 3D space because it does not change with the change of the shooting viewing angle. Therefore, the present invention uses 3D hand keypoints and information about the 3D hand keypoints to describe the basic hand state.

[0014] In addition, with regard to formulating rules for detecting hand movements, motion detection is performed on video fragments (video clips), and the hand movements within a time fragment are determined by analyzing the changing trends of hand state information within the time fragment.

[0015] The basic configuration of a detection device according to an embodiment of the present invention will be described below with reference to FIG.

[0016] 2 is a block diagram of a detection device according to an embodiment of the present invention. As shown in FIG. 2, the detection device 100 according to an embodiment of the present invention may include an acquisition unit 110, a calculation unit 120, an analysis unit 130 and a determination unit 140.

[0017] The acquisition unit 110 can acquire information about the 3D key points of the hand in each frame image in the video, and the information about the 3D key points of the hand includes at least the position of each joint point of the hand in 3D space. For example, the acquisition unit 110 can acquire information about the 3D key points of the hand through a pre-trained 3D hand key point detection model.

[0018] The computing unit 120 can also obtain hand state information for each frame image based on information about the 3D key points of the hand, where the hand state information includes at least the bending states of the finger segments.

[0019] Additionally, analysis unit 130 can capture changes in hand state information over time fragments of the video.

[0020] Also, the determination unit 140 can determine hand movements in a time fragment based on the changes in the hand state information and the rules regarding hand movements.

[0021] As a result, the detection device 100 according to the embodiment of the present invention can analyze the flexion state of the finger segments for each frame and develop hand movement rules, thereby realizing reusable hand movement detection.

[0022] There is a lot of uncertainty and diversity in hand movements in actual production scenes. Below, we will explain each unit of the detection device 100 in more detail by taking two types of hand movements ("grasping" and "twisting") that are widely used in production scenes as examples.

[0023] <"Grab" action> Since the "grasping" motion can be detected by analyzing the change in the finger segment bending degree (example of the finger segment bending state mentioned above), the finger segment bending degree is selected as the basic hand state to detect the "grasping" motion. Specifically, the finger segment bending state is taken as the basic hand state.

[0024] The calculation unit 120 can calculate the bending degree of the finger segments in each frame image based on the information about the 3D key points of the hand provided by the acquisition unit 110. Specifically, the calculation unit 120 can calculate the bending degree of the finger segments in each frame in the video fragment as a basic hand state for detecting a "grabbing" action.

[0025] For example, the degree of bending of a finger segment can be represented by the average of the included angles between the four fingers other than the thumb and the bone structure at the base.

[0026] 3 is a diagram showing the correspondence between the key points of the hand and the finger segments. There are 21 key points (including joint points) on the hand, and here, the bones (line segments) between adjacent key points on the fingers are defined as "finger segments", for example, the bones (line segments) between key points 5 and 6 are defined as one "finger segment". In the present invention, from the viewpoint of application, the definition of the finger segment is expanded as follows, that is, for the four key points 5, 9, 13, and 17 at the bases of the four fingers other than the thumb, the bones (line segments) between them and the wrist joint point 0 are also defined as "finger segments".

[0027] Based on the definition of finger segments in FIG. 3, the included angle between adjacent finger segments in 3D space represents the degree of bending of the finger segments. Since a "grabbing" action usually involves bending of finger segments where key points 5, 9, 13, and 17 are located, the "grabbing" action can be detected by analyzing the change trend of the included angle between two finger segments where each of these four key points is located. For example, the included angle of the index finger segment can be represented by the included angle between vectors (0,5) and (5,6). The included angle of the middle finger segment can be represented by the included angle between vectors (0,9) and (9,10). The included angle of the ring finger segment can be represented by the included angle between vectors (0,13) and (13,14). The included angle of the little finger segment can be represented by the included angle between vectors (0,17) and (17,18).

[0028] Therefore, the degree of flexion of a finger segment can be expressed as the average value of the included angle between vectors (0,5) and (5,6), the included angle between vectors (0,9) and (9,10), the included angle between vectors (0,13) and (13,14), and the included angle between vectors (0,17) and (17,18).

[0029] Considering the inherent error in detecting 3D hand keypoints, in order to improve the robustness of the detection method, the average of the three largest included angles among the four finger segments (index finger, middle finger, ring finger and little finger) can also be calculated to represent the finger segment flexion state (degree of flexion of the finger segment).

[0030] Generally speaking, the bending degree of the finger segment gradually increases during the "grasping" action, and remains unchanged after the object is grasped. FIG. 4 is a diagram illustrating the change of the bending degree of the finger segment in the process of the detection device detecting the "grasping" action according to an embodiment of the present invention. In order to better illustrate how the hand changes during the "grasping" process, the upper part of FIG. 4 shows the 3D hand key points and finger segments of three representative frames, and the lower part of FIG. 4 shows the change of the included angle of the finger segment in the whole video fragment. As can be seen, as the bending degree of the finger segment increases, the included angle of the finger segment also gradually increases. Therefore, the analysis unit 130 can obtain the change trend of the bending degree of the finger segment by analyzing the change trend of the included angle of the finger segment.

[0031] Specifically, for each sliding window, the analysis unit 130 analyzes the change trend of the bending degree of the finger segment of the multi-frame image within the sliding window of the video. If the bending degrees of the finger segment all increase in each sliding window among the plurality of consecutive sliding windows, the determination unit 140 determines that the video within the time length of the plurality of consecutive sliding windows is related to the hand grasping action. As shown in FIG. 4, the video from frame 0 to frame 50 is related to the hand grasping action. Of course, this is merely an example, and the present invention is not limited thereto.

[0032] Here, the identification of the motion can be realized by analyzing whether the included angle of the finger segment is on an upward trend in the sliding window. The Mann-Kendall test method can be adopted to determine whether there is an upward trend. FIG. 4 uses different gray scales to show that the sliding window slides according to a predetermined step length, the length of the sliding window does not change, and there is a predetermined overlap between the previous and next sliding windows. In FIG. 4, the length of the sliding window is 20 frames, and the overlap between the previous and next sliding windows is 10 frames. Of course, this is merely an example, and the present invention is not limited thereto. A sliding window (frame 0 to frame 50) in which the included angle of the finger segment for multiple consecutive periods is on an upward trend is detected as a potential "grabbing" motion.

[0033] In addition, because the actual production scene is relatively complex and has various uncertainties, rules for eliminating common interferences can also be formulated. In such a case, the determination unit 140 can further eliminate interferences caused by some non-target actions based on the hand state information.

[0034] The determination unit 140 can eliminate non-target actions from potential "grab" actions. As mentioned above, due to inherent error in 3D hand keypoints, some non-"grab" actions may be detected as potential "grab" actions by the determination unit 140. There are two steps to eliminate false positives:

[0035] First, the determination unit 140 determines whether a radial twist of the hand occurs in the time fragment detected by the analysis. Normally, a twist of the hand does not occur during the "grasping" process, so determining whether a twist occurs can eliminate some non-target actions. When a radial twist of the hand occurs, the orientation of the back of the hand may change significantly, so the change in the included angle between the orientation of the back of the hand and the coordinate axis can be detected to determine whether a twist occurs. The specific flow is shown in FIG. 5.

[0036] Fig. 5 is a diagram showing a process executed by a detection device according to an embodiment of the present invention. As shown in Fig. 5, the orientation of the back of the hand is represented by the normal direction of the vector formed by the wrist (key point 0) and the base of the index finger (key point 5) and the vector formed by the wrist (key point 0) and the base of the little finger (key point 17), and the change in the orientation of the back of the hand is represented by the standard deviation of the included angle between the orientation of the back of the hand and the x-axis, y-axis, and z-axis of the coordinate system in which the hand is located.

[0037] Specifically, the determination unit 140 may be configured to: determine whether the change in orientation of the back of the hand within the sliding window is greater than a first threshold; and if it is greater than the first threshold, determine that the action is not a grab.

[0038] After that, the determination unit 140 can eliminate some non-target actions by analyzing whether the upward trend is significant. Due to the detection error of the 3D hand keypoints, the included angle of the finger segments of some non-"grasping" actions may also have a weak upward trend, but the upward trend of the real "grasping" action is relatively significant. The magnitude of the slope can be used to evaluate whether the upward trend is significant. The specific flow is as shown in Figure 6.

[0039] 6 is a flowchart of another process performed by the detection device according to an embodiment of the present invention. Specifically, the analysis unit 130 further analyzes the gradient of the sequence of finger segment bending degrees (sequence of finger segment included angles) within the time length of multiple consecutive sliding windows, and only when the gradient is greater than a second threshold, the determination unit 140 determines that the motion is a grab.

[0040] In the above example, the degree of curvature of a finger segment is represented by the average value of the included angles between the four fingers other than the thumb and the bone structure at the base. The present invention is not limited to this. As shown in FIG. 7, the degree of curvature of a finger segment may be represented by the sum of the curvatures of three joint points of the four fingers other than the thumb. When calculating the sum of the curvatures, the closer a joint is to the base, the higher the weight of that curvature.

[0041] In other words, the included angles of finger segments of other parts can be used to detect micro-grabbing movements. When grasping some relatively large objects, the change in hand posture during the grasping process is relatively small, and this type of movement is referred to as a micro-grabbing movement. A micro-grabbing movement is not realized by bending the joint where the base node is located, and to detect this movement, it is necessary to analyze the change tendency of the included angles of finger segments of other parts. Figure 7 shows finger segments that are of interest when considering micro-grabbing movements (i.e., the three finger segments of the fingers other than the thumb). Based on the probability of being used for the "grabbing" movement, different weights are given to the included angles of finger segments of each part, and the magnitude of the weights satisfies the condition w1>w2>w3.

[0042] After the calculation unit 120 calculates the weighted finger segment flexion degrees in this manner, the analysis unit 130 and the determination unit 140 can perform further processing as described above to detect video fragments related to a “grabbing” motion in the video.

[0043] This allows the detection device 100 to formulate rules for detecting movements after thoroughly analyzing the trends in changes in basic hand states during movements, and eliminate interference from some non-target movements, thereby improving the feasibility and robustness of the detection method.

[0044] <"Twist" motion> The computation unit 120 can compute the finger orientation of each frame in the video fragment as a basic hand state for detecting the "twist" motion. Due to the limitations of the physiological structure of the finger, each finger segment of the same finger can only move in one plane. Therefore, based on this condition, the finger orientation can be defined as the normal vector of the plane in which the finger is located, as shown in FIG. 8(a). In other words, the finger orientation is represented by the normal vector of the plane in which the finger is located. Specifically, any two adjacent finger segments of a finger (vector (5,6) and vector (6,7) in FIG. 8(b)) can be taken, and the normal vector of the plane in which they are located can be calculated to be the defined finger orientation.

[0045] The analysis unit 130 can obtain the deviation of finger orientation by calculating the deviation of finger orientation between each frame image in the sliding window and the initial frame image. Usually, a "twist" motion is accompanied by a change in finger orientation. This change can be evaluated by the change in the deviation of finger orientation between each frame and the initial frame. If the deviation of finger orientation in multiple consecutive sliding windows shows an increasing trend, the time fragment can be regarded as a potential "twist" motion.

[0046] In addition, the calculation unit 120 can further calculate the finger segment bending degree and the back of the hand orientation within the sliding window. Usually, during the "twist" motion, the finger segment bending degree changes very little because the hand grips the tool tightly. Also, during the "twist" motion, the hand mainly rotates in a plane, not in a radial twist, so the back of the hand orientation changes very little. By determining whether the finger segment bending degree and the back of the hand orientation change very little (determining whether the standard deviation is less than a threshold), some out-of-target motions among the potential "twist" motions detected by the determination unit 140 can be excluded.

[0047] The calculation of the bending degree of the finger segment is the same as the calculation of the bending degree of the finger segment during the above-mentioned "grasping" motion. That is, the bending degree of the finger segment represented by the average value of the included angles between the four fingers other than the thumb and the skeleton of the base can be calculated, and the bending degree of the finger segment represented by the sum of the bending degrees of the three joint points of the four fingers other than the thumb (weighted finger segment bending degree) can also be calculated. Similarly, the orientation of the back of the hand is represented by the normal direction of the vector formed by the wrist and the base of the index finger, and the vector formed by the wrist and the base of the little finger, and the change in the orientation of the back of the hand is represented by the standard deviation of the included angles between the orientation of the back of the hand and the x-axis, y-axis, and z-axis of the coordinate system in which the hand is located.

[0048] In addition, because the hand needs to grip the tool tightly when performing the "twist" motion, some out-of-target motions can be further excluded by determining whether the average finger segment flexion degree in the sliding window is greater than a threshold value. The average finger segment flexion degree refers to the average finger segment flexion degree of all frames in the sliding window.

[0049] In other words, the calculation unit 120 can calculate the finger orientation, the bending degree of the finger segment, and the back orientation of the hand of the multi-frame images in a sliding window of the video, and the analysis unit 130 can analyze the changes of the finger orientation, the bending degree of the finger segment, and the back orientation calculated by the calculation unit 120. If the calculation results of each sliding window among the plurality of consecutive sliding windows all satisfy the following conditions, the determination unit 140 determines that the video within the time length of the plurality of consecutive sliding windows is related to a hand twisting motion, namely, the deviation of the finger orientation increases; the change of the bending degree of the finger segment is smaller than a third threshold; the change of the back orientation is smaller than a fourth threshold; and the average value of the bending degree of the finger segment within the sliding window is greater than a fifth threshold.

[0050] Although not shown, in detecting a “twist” motion, the method of selecting and moving the sliding window may be similar to that of detecting a “grab” motion, for example, adjacent sliding windows among a plurality of consecutive sliding windows have an overlap in time.

[0051] This allows the detection device 100 to formulate rules for detecting movements after thoroughly analyzing the trends in changes in basic hand states during movements, and eliminate interference from some non-target movements, thereby improving the feasibility and robustness of the detection method.

[0052] A detection method according to an embodiment of the present invention will now be described in conjunction with FIG.

[0053] 9, the detection method according to the embodiment of the present invention starts in step S110. In step S110, information about 3D key points of the hand in each frame image in the video is obtained, and the information about the 3D key points of the hand at least includes the position of each joint point of the hand in 3D space.

[0054] Next, in step S120, hand state information for each frame image is obtained based on information about the 3D key points of the hand, where the hand state information includes at least the bending states of the finger segments.

[0055] Then, in step S130, the changes in the hand state information in the time fragment of the video are obtained.

[0056] Then, in step S140, the hand motion within the time fragment is determined based on the changes in the hand state information and the rules regarding the hand movement.

[0057] Thereafter, the process ends.

[0058] Specifically, with respect to detecting the "grabbing" action, the method may further include: setting sliding windows and calculating basic hand state information, such as the bending degree of finger segments, in each sliding window; analyzing the trend of the bending degree of finger segments of multi-frame images in the sliding window of the video; determining that the video within the time length of the multiple consecutive sliding windows is related to a hand grabbing action if the bending degrees of finger segments all increase in each sliding window among the multiple consecutive sliding windows, thereby determining that the actions within the multiple consecutive sliding windows that satisfy the rule are potential detected actions; and filtering out non-target actions from the potential detected actions.

[0059] The rejection of non-target movements further includes determining whether the change in orientation of the back of the hand within the sliding window is greater than a first threshold, and if so, determining that the movement is not a grab.

[0060] The rejection of non-targeted movements can further be achieved by analyzing the slope of the sequence of finger segment flexion degrees within a number of consecutive sliding window time lengths, and determining that the movement is a grasp only if the slope is greater than a second threshold.

[0061] Also, specifically, with regard to detecting the "twist" motion, the method may further include: calculating finger orientations, finger segment bending degrees, and back of hand orientations of multi-frame images within a sliding window of the video; and determining that the video within a time length of the plurality of consecutive sliding windows is related to a hand twist motion if the calculation results of each sliding window among the plurality of consecutive sliding windows all satisfy the following conditions: the deviation of the finger orientations increases; the change in the bending degrees of the finger segments is greater than a third threshold; the change in the back of hand orientation is less than a fourth threshold; and the average bending degree within the sliding window is greater than a fifth threshold.

[0062] As a result, the detection method according to the embodiment of the present invention can realize reusable hand movement detection by analyzing the bending state of finger segments frame by frame and formulating hand movement rules.

[0063] It should be noted that the various implementation manners of the above steps of the detection method according to the embodiment of the present invention have been described in detail above, so a detailed description thereof will be omitted here.

[0064] Of course, each operation process of the detection method according to the present invention may be realized by a computer executable program stored in various machine-readable storage media.

[0065] The object of the present invention can also be realized in the following manner: a storage medium storing the above-mentioned executable program code is provided directly or indirectly to a system or device, and a computer or central processing unit (CPU) in the system or device reads and executes the above-mentioned program code. In this case, as long as the system or device has a function of executing a program, the embodiment of the present invention is not limited to a program, and the program may be in any form, such as a target program, an interpreted executable program, a script program provided to an operating system, etc.

[0066] It should be noted that the machine-readable storage medium mentioned above may be various storage devices and storage units, semiconductor devices, magnetic disk units, such as optical, magnetic and magneto-optical disks, and other media suitable for storing information.

[0067] In addition, the technical solution of the present invention can also be realized by connecting a computer to a corresponding site on the Internet, downloading and installing the computer program code according to the present invention into the computer, and then running the program.

[0068] Each component or unit in the above-mentioned device may be configured by software, firmware, hardware, or a combination thereof. Note that the specific means and methods used for the configuration are well known to those skilled in the art, so detailed explanations are omitted here. When realized by software or firmware, the programs constituting the software are installed from a storage medium or network into a computer having a dedicated hardware configuration (e.g., the general-purpose computer 1300 shown in FIG. 10), and the computer can realize various functions when various programs are installed.

[0069] FIG. 10 is a diagram of a hardware configuration (general-purpose computer) 1300 capable of implementing the method and apparatus of an embodiment of the present invention.

[0070] The general-purpose computer 1300 may be, for example, a computer system. Note that the general-purpose computer 1300 is merely an example and does not limit the scope or functionality of the method and apparatus according to the present invention. Furthermore, the general-purpose computer 1300 does not depend on any module, assembly, etc., or combination thereof in the above-described method and apparatus.

[0071] 10, a central processing unit (CPU) 1301 performs various processes based on a program stored in a ROM 1302 or a program loaded from a storage unit 1308 to a RAM 1303. The RAM 1303 can also store data required when the CPU 1301 performs various processes, depending on the needs. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output interface 1305 is also connected to the bus 1304.

[0072] Further, the following components are connected to the input / output interface 1305: an input unit 1306 including a keyboard, an output unit 1307 including a display such as a liquid crystal display (LCD) and a speaker, a storage unit 1308 including a hard disk, and a communication unit 1309 including a network interface card such as a LAN card and a modem. The communication unit 1309 performs communication processing via a network such as the Internet or a LAN. A drive 1310 may be connected to the input / output interface 1305 according to needs. A removable medium 1311, such as a semiconductor memory, can be set in the drive 1310 as necessary, and a computer program read from the removable medium 1311 can be installed in the storage unit 1308.

[0073] The present invention further provides a program product including a machine-readable instruction code. When such instruction code is read and executed by a machine, it can execute the method in the above-mentioned embodiment of the present invention. Accordingly, various storage media that carry such a program product, such as magnetic disks (including floppy disks (registered trademark)), optical disks (including CD-ROMs and DVDs), magneto-optical disks (including MDs (registered trademark)), and semiconductor storage devices, are also included in the present invention.

[0074] The above-mentioned storage medium may include, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory device, and the like, but is not limited to these.

[0075] Moreover, each operation (process) in the above-described method may be realized in the form of a computer-executable program stored in various machine-readable storage media.

[0076] Furthermore, with respect to the above-mentioned embodiments, the following supplementary notes are disclosed.

[0077] (Appendix 1) 1. An apparatus for detecting hand movements in a video, comprising: an acquisition unit for acquiring information about 3D key points of a hand in each frame image in the video, the information about the 3D key points of the hand including at least a position of each joint point of the hand in a 3D space; a computing unit for obtaining hand state information for each frame image based on information about 3D key points of the hand, the hand state information including at least flexion states of finger segments; an analysis unit for obtaining changes in hand state information within a time fragment of said video; and and a determination unit for determining hand motion within the time fragment based on changes in the hand state information and rules regarding hand motion.

[0078] (Appendix 2) 2. The apparatus of claim 1, The bending state of the finger segment refers to the degree of bending of the finger segment, The computing unit computes a bending degree of the finger segment in each frame image based on information about the 3D key points of the hand; The analysis unit analyzes, for each sliding window, a trend of flexion degree of the finger segment in the multi-frame images within the sliding window of the video; If finger segment flexion degrees all increase in each sliding window of a plurality of consecutive sliding windows, the determination unit determines that the video within a time length of the plurality of consecutive sliding windows is associated with a hand grasping motion.

[0079] (Appendix 3) 3. The apparatus of claim 2, The determination unit determines whether a change in orientation of the back of the hand within the sliding window is greater than a first threshold, and if so, determines that the motion is not a grab.

[0080] (Appendix 4) 3. The apparatus of claim 2, The analysis unit analyzes a gradient of a sequence of finger segment flexion degrees within the plurality of consecutive sliding window time lengths; When the tilt is greater than a second threshold, the determination unit determines that the motion is a grab.

[0081] (Appendix 5) 2. The apparatus of claim 1, The bending state of the finger segment refers to the degree of bending of the finger segment, The computing unit computes finger orientations, finger segment flexion degrees, and back of hand orientations for multi-frame images within a sliding window of the video; The analysis unit analyzes changes in the finger orientation, the degree of flexion of the finger segments, and the orientation of the back of the hand; The determining unit determines that the video within the time length of the plurality of consecutive sliding windows is related to a hand twisting motion when the calculation results of each sliding window among the plurality of consecutive sliding windows all satisfy the following condition: Increased deviation in finger orientation; The change in the degree of flexion of the finger segment is less than a third threshold; The change in the orientation of the back of the hand is less than the fourth threshold; and The condition is that the average value of the degree of flexion of the finger segment within the sliding window is greater than a fifth threshold.

[0082] (Appendix 6) 6. The apparatus of claim 2 or 5, The degree of bending of the finger segment is represented by the average angle between the four fingers other than the thumb and the bone structure of the base; or The degree of flexion of the finger segment is represented by the sum of the flexion degrees of three joint points of the four fingers other than the thumb, and when calculating the sum of the flexion degrees, the closer a joint is to the base, the higher the weight of the flexion degree of that joint.

[0083] (Appendix 7) 6. The apparatus of claim 5, The device, wherein the finger orientation is represented by a normal vector of a plane on which the finger is located, and the calculation unit obtains the finger orientation deviation by calculating the finger orientation deviation between each frame image within a sliding window and an initial frame image.

[0084] (Appendix 8) 6. The apparatus of claim 3 or 5, A device in which the orientation of the back of the hand is represented by the normal directions of the vector formed by the wrist and the base of the index finger and the vector formed by the wrist and the base of the little finger, and a change in the orientation of the back of the hand is represented by the standard deviation of the included angle between the orientation of the back of the hand and the x-axis, y-axis, and z-axis of a coordinate system in which the hand is located.

[0085] (Appendix 9) 1. A method for detecting hand movements in a video, comprising: Obtaining information about 3D key points of a hand in each frame image in the video, the information about the 3D key points of the hand including at least a position in 3D space of each joint point of the hand; Obtain hand state information for each frame image based on information about the 3D key points of the hand, the hand state information including at least a flexion state of a finger segment; Obtaining changes in hand state information within a time fragment of the video; and determining hand motion within the time fragment based on the changes in hand state information and rules regarding hand movement.

[0086] (Appendix 10) 10. The method according to claim 9, The flexion state of the finger segment refers to a degree of flexion of the finger segment, and the method further comprises: Calculate the bending degree of finger segments in each image frame based on the information about the 3D key points of the hand; For each sliding window, analyzing a trend of flexion degree of the finger segment in the multi-frame images within the sliding window of the video; and determining that a video within a time length of a plurality of consecutive sliding windows is associated with a hand grasping motion if finger segment flexion rates all increase in each of the consecutive sliding windows.

[0087] (Appendix 11) 11. The method of claim 10, further comprising: determining whether a change in hand back orientation within the sliding window is greater than a first threshold; and determining that the motion is not a grab if the first threshold is greater than the first threshold.

[0088] (Appendix 12) 11. The method of claim 10, further comprising: Analyzing a gradient of a sequence of finger segment flexion degrees within the plurality of consecutive sliding window time lengths; and determining that the motion is a grab when the slope is greater than a second threshold.

[0089] (Appendix 13) 10. The method according to claim 9, The flexion state of the finger segment refers to a degree of flexion of the finger segment, and the method further comprises: Calculating finger orientation, finger segment flexion degree, and back of hand orientation for multi-frame images within a sliding window of said video; Analyzing changes in finger orientation, degree of flexion of the finger segments, and orientation of the back of the hand; and Determining that the video within a time length of the plurality of consecutive sliding windows is related to a hand twisting motion when the calculation results of each sliding window of the plurality of consecutive sliding windows all satisfy the following requirements (conditions): Increased deviation in finger orientation; The change in the degree of flexion of the finger segment is less than a third threshold; The change in the orientation of the back of the hand is less than the fourth threshold; and The method, wherein an average value of the degree of flexion of the finger segment within the sliding window is greater than a fifth threshold.

[0090] (Appendix 14) 14. The method according to claim 10 or 13, The degree of bending of the finger segment is represented by the average angle between the four fingers other than the thumb and the bone structure of the base; or the degree of flexion of the finger segment is represented by the sum of the degrees of flexion of three joint points of four fingers other than the thumb, and when calculating the sum of the degrees of flexion, the closer a joint is to the base, the higher the weight of the degree of flexion of that joint.

[0091] (Appendix 15) 15. The method according to claim 14, The degree of flexion of the finger segment is represented by an average value of the first three included angles between the four fingers other than the thumb and the bone structure at the base.

[0092] (Appendix 16) 14. The method according to claim 13, comprising: The finger orientation is represented by a normal vector of a plane on which the finger is located, and the finger orientation deviation is obtained by calculating the deviation of the finger orientation between each frame image in a sliding window and an initial frame image.

[0093] (Appendix 17) 14. The method according to claim 11 or 13, a direction of the back of the hand is represented by a normal direction of a vector formed by the wrist and the base of the index finger and a vector formed by the wrist and the base of the little finger, and a change in the direction of the back of the hand is represented by a standard deviation of the included angle between the direction of the back of the hand and the x-axis, y-axis, and z-axis of a coordinate system in which the hand is located.

[0094] (Appendix 18) 10. The method according to claim 9, A method for detecting information about 3D hand keypoints using a pre-trained 3D hand keypoint detection model.

[0095] (Appendix 19) 11. The method of claim 10, further comprising: The Mann-Kendall test is employed to determine the trend of tortuosity within each sliding window.

[0096] (Appendix 20) 1. A machine-readable storage medium, comprising: A computer program is stored in the A machine-readable storage medium, the computer program causing a computer to perform the method according to any one of claims 9-19.

[0097] Although the preferred embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and any modification to the present invention falls within the technical scope of the present invention as long as it does not depart from the spirit of the present invention.

Claims

1. 1. An apparatus for detecting hand movements in a video, comprising: an acquisition unit for acquiring information about 3D key points of the hand in each frame image in the video, the information about the 3D key points of the hand including at least a position in 3D space of each joint point of the hand; a computing unit for obtaining state information of the hand for each frame image based on information about 3D key points of the hand, the hand state information including at least a flexion state of finger segments; an analysis unit for obtaining changes in the hand state information within a time fragment of the video; and a determination unit for determining a motion of the hand within the time fragment based on changes in the hand state information and rules regarding the hand movement.

2. 2. The apparatus of claim 1, The bending state of the finger segment refers to the degree of bending of the finger segment, the computing unit computes a degree of flexion of the finger segments in each frame image based on information about the 3D key points of the hand; The analysis unit analyzes, for each sliding window, a trend of a flexion degree of the finger segment in the multi-frame images within the sliding window of the video; If the degree of flexion of the finger segments all increases in each sliding window of the plurality of consecutive sliding windows, the determination unit determines that the video within a time length of the plurality of consecutive sliding windows is associated with a hand grasping motion.

3. 3. The apparatus of claim 2, The determination unit determines whether a change in orientation of the back of the hand within the sliding window is greater than a first threshold, and if so, determines that the motion is not a grasping motion.

4. 3. The apparatus of claim 2, The analysis unit analyzes a gradient of a sequence of finger segment flexion degrees within a plurality of consecutive sliding window time lengths; When the inclination is greater than a second threshold, the determination unit determines that the motion is a grab motion.

5. 2. The apparatus of claim 1, The bending state of the finger segment refers to the degree of bending of the finger segment, The computing unit computes finger orientations, finger segment flexion degrees, and back of hand orientations for multi-frame images within a sliding window of the video; the analysis unit analyzes changes in finger orientation, degree of flexion of the finger segments, and orientation of the back of the hand; The calculation results of each sliding window among a plurality of consecutive sliding windows are all Increased deviation in finger orientation; the change in the degree of flexion of the finger segment is less than a third threshold; The change in the orientation of the back of the hand is less than a fourth threshold; and The average degree of flexion of the finger segment within the sliding window is greater than a fifth threshold. If the condition is satisfied, the determining unit determines that the video within a time length of a plurality of consecutive sliding windows is associated with a hand twisting motion.

6. 6. An apparatus according to claim 2 or 5, The degree of bending of the finger segment is represented by the average angle between the four fingers other than the thumb and the bone at the base; or The degree of flexion of the finger segment is represented by the sum of the degrees of flexion of three joint points of four fingers other than the thumb, and when calculating the sum of the degrees of flexion, the closer the joint point is to the base, the higher the weight of the degree of flexion of the joint point.

7. 6. The apparatus of claim 5, the orientation of the finger is represented by a normal vector of a plane on which the finger is located, The calculation unit obtains a finger orientation deviation by calculating a finger orientation deviation between each frame image in a sliding window and an initial frame image.

8. 6. An apparatus according to claim 3 or 5, the orientation of the back of the hand is represented by a vector formed by the wrist and the base of the index finger, and a normal direction of a vector formed by the wrist and the base of the little finger, The apparatus, wherein the change in orientation of the back of the hand is represented by the standard deviation of the included angle between the orientation of the back of the hand and the x-axis, y-axis, and z-axis of a coordinate system in which the hand is located.

9. 1. A method for detecting hand movements in a video, comprising: Obtaining information about 3D key points of a hand in each frame image in the video, the information about the 3D key points of the hand including at least a position in 3D space of each joint point of the hand; obtaining state information of the hand for each frame image based on information about 3D key points of the hand, the state information of the hand including at least a flexion state of finger segments; Obtaining changes in the hand state information within a time fragment of the video; and determining hand motion within the time fragment based on changes in the hand state information and rules relating to the hand movement.

10. A program for causing a computer to execute the method according to claim 9.

Images