Hand action recognition method based on 3D gesture sequence analysis

Abstract

The application discloses a hand action recognition method based on 3D gesture sequence analysis. For the 3D skeleton action sequence of the hand action, the speed, angle and relative position information of the hand joint are extracted frame by frame, the joint rate of each skeleton joint is calculated, the minimum value position of the joint rate waveform diagram is taken as the gesture action segmentation point, a plurality of action frame sequences are obtained, then the finger state is matched and coded frame by frame, and the coding result corresponding to the gesture action is obtained. After adding the pose information, the plurality of gesture actions are combined into an action matrix, and the matching of the hand action is completed according to the action matrix. The application uses the joint rate to replace the segmentation method of the fixed-length time window, so that the segmentation result can contain the complete gesture action and meet the actual situation. For each gesture action, the matching and coding are performed frame by frame, so that the interference frame or invalid information frame can be effectively removed, the matching difficulty and search time are reduced, and the recognition accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and relates to recognition of continuous hand actions, in particular to a hand action recognition method based on 3D gesture sequence analysis. BACKGROUND

[0002] A hand action can be regarded as a dynamic data sequence composed of multiple continuous gesture actions. The hand action recognition technology is to make a category determination on the hand action of a person in the dynamic data sequence. There are many reports on gesture recognition technology in the prior art, but there are few reports on hand action recognition. The difference between the two lies in that: gesture recognition is usually from the spatial dimension to recognize a static gesture object; and hand action recognition is from the time dimension to identify an action sequence composed of continuous gesture actions.

[0003] A common hand action recognition method first divides the dynamic data sequence containing multiple continuous gesture actions in time sequence, divides the long data sequence into data sequence segments each containing a single gesture action; then determines a fixed-length time window according to experience, selects frames of the data sequence segments with the fixed-length time window, and usually removes the head and tail of the data sequence segments to remove blank data frames or interference data frames. After frame selection by the fixed-length time window, a series of frame sequences with equal length are obtained, and finally the frame sequences are compared with gesture action templates stored in a database to complete the recognition process. The method of selecting data frames by a fixed-length time window is based on the assumption that the segmented frame sequence contains only one gesture action, so it can only achieve good recognition effect in the laboratory stage. However, in real application scenarios, it is difficult to ensure that different people complete a gesture action at the same speed, so when using a fixed-length time window to select data frames, there may be situations such as a frame sequence containing multiple gesture actions, interference frames not containing effective gesture actions, or partial frame data missing a complete gesture action, resulting in inaccurate recognition or slow recognition speed. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application proposes a hand action recognition method based on 3D gesture sequence analysis, which decomposes the action sequence according to the hand movement state, determines the start frame and end frame of a single action based on 3D action information, reduces the redundant processing of blank frames and interference frames, and improves the recognition efficiency and recognition performance.

[0005] The hand action recognition method based on 3D gesture sequence analysis includes template library establishment and action recognition, and specifically includes the following steps:

[0006] Step one, establish a matching template library

[0007] s1.1. Collect the 3D skeletal motion sequence during the execution of a hand movement, and extract the angle, velocity, and relative position of the hand bone joints in the sequence frame by frame.

[0008] s1.2. Divide the 3D skeleton motion sequence obtained in s1.1 according to the gesture action. After removing blank frames and interference frames, multiple motion frame sequences are obtained. Each motion frame sequence includes a motion frame of a complete gesture action.

[0009] s1.3 defines a 5-character string, where each character corresponds to the state of a finger. Based on the degree of finger bending, the relative position of the finger to the palm, and the relative position between different fingers, the state of each finger is assigned a value, resulting in an action frame code consisting of 5 characters, which is then stored in the action frame library.

[0010] s1.4. Concatenate the encoding X of the action frame in s1.3 according to the time sequence. If the encoding results of two adjacent frames are the same, merge them to obtain the encoding result α corresponding to a complete gesture action, α = X1X2X3…X m , and store in the gesture action library. Where m represents the number of frames in the action frame sequence corresponding to a complete gesture.

[0011] s1.5. Add pose information U and V to the encoded result of each gesture to obtain the sequence code corresponding to the gesture. Where U represents... coordinates (x) u ,y u ,z u ), It is the normal vector of the plane containing the palm, V represents coordinates (x) v ,y v ,z v ), It is the direction vector of the line connecting the palm and the metacarpophalangeal joint of the middle finger. Then, the sequence codes of multiple hand gestures are put into a matrix according to the time sequence, which is used as the action matrix ω corresponding to the hand gestures, to build a hand gesture library. Each row of the action matrix ω is a sequence code of a hand gesture. For hand gestures with a sequence code length less than k or with fewer than k hand gestures, zeros are used to pad the code, making the size of the action matrix ω k*k.

[0012] s1.6 Repeat steps s1.1 to s1.5 to collect 3D skeletal motion sequences of different hand movements, supplement the motion frame library, gesture motion library and hand motion library data, and complete the establishment of the matching template library.

[0013] Step 2: Hand Movement Recognition

[0014] s2.1, collect a 3D skeleton action sequence T of a hand action to be identified, extract the angle, speed and relative position of the hand skeleton joint in 3D space in the sequence frame by frame. Calculate the joint speed f(v) of all hand skeleton joints at different time points frame by frame:

[0015]

[0016] n represents the number of hand skeleton joints, v ix , v iy , v iz respectively represent the speed of the ith joint in the x-axis, y-axis and z-axis directions, w i represents the weight of the ith joint.

[0017] s2.2, according to the calculation result of the joint speed, the position where the joint speed reaches a minimum value or is less than a threshold value is taken as a segmentation point, and the 3D skeleton action sequence T is segmented into a plurality of action frame sequences, and one action frame sequence is taken as a gesture action to be estimated.

[0018] s2.3, for each action frame sequence, match frame by frame with the templates stored in the action frame library, and calculate the similarity. When the similarity is less than a threshold value, the frame is regarded as an interference frame and is discarded; otherwise, the code corresponding to the action frame with the highest similarity in the library is selected as the code of the frame.

[0019] s2.4, splice the code of each frame in time sequence, if the codes of two adjacent frames in an action frame sequence are the same, merge processing is performed, and the code result of a gesture action is obtained. Match the code result with the templates stored in the gesture action library to identify the gesture action.

[0020] s2.5, add pose information U, V to the code result of each gesture action obtained in s2.4 to obtain the sequence code corresponding to the gesture action. Then put the sequence codes of a plurality of gesture actions into a matrix with a size of k*k in time sequence. For gesture actions with a sequence code length less than k or gesture actions with a number less than k, use 0 to fill up, obtain an action matrix ω of the hand action to be estimated, and match it with the templates stored in the hand action library to complete the recognition of the hand action.

[0021] As a preferred, the 3D skeleton action sequence in the hand action process is obtained through a visual algorithm, a data glove or a Leap Motion.

[0022] As a preferred, a neural network is used to select the most interesting nodes from the hand skeleton joints, and different weights w i are given to each node.

[0023] The present application has the following beneficial effects:

[0024] The present application is directed to the 3D skeleton action sequence of hand action, the speed, angle and relative position information of hand joint are extracted frame by frame, then the segmentation point of gesture action is determined based on the joint rate, instead of using fixed time window, which can ensure that each action frame sequence after segmentation contains a complete gesture action, more in line with the actual situation, and can reduce the interference of false action in the movement process. For each gesture action, frame-by-frame matching and encoding can effectively remove interference frames or invalid information frames, reduce the matching difficulty and search time, and improve the recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 It is a 3D gesture skeleton node model schematic diagram;

[0026] Figure 2 It is an action frame encoding schematic diagram;

[0027] Figure 3 It is a palm pose vector schematic diagram;

[0028] Figure 4 It is a hand action matrix schematic diagram;

[0029] Figure 5 It is a joint rate waveform diagram;

[0030] Figure 6 It is a 3D skeleton action sequence segmentation schematic diagram based on joint rate;

[0031] Figure 7 It is a gesture action matching method flow chart;

[0032] Figure 8 It is a gesture action matching method schematic diagram;

[0033] Figure 9 It is a hand action matching method flow chart. DETAILED DESCRIPTION

[0034] The present application will be further explained in conjunction with the accompanying drawings;

[0035] The hand action recognition method based on 3D gesture sequence analysis includes template library establishment and action recognition, specifically including the following steps:

[0036] Step one, establish a matching template library

[0037] s1.1, through visual algorithm, data glove or Leap Motion, 3D skeleton action sequence in hand action process is obtained, 3D gesture skeleton node model is reconstructed, such as Figure 1The hand skeleton joint points in the sequence are extracted frame by frame, and the angle, speed and relative position of the hand skeleton joint points are extracted for subsequent analysis and recognition.

[0038] s1.2, the 3D skeleton action sequence obtained in s1.1 is divided according to gesture actions, and after removing blank frames and interference frames, a plurality of action frame sequences are obtained, each of which includes an action frame of a complete gesture action.

[0039] s1.3, a 5-bit string is defined, each bit in the string corresponds to the state of a finger, and the state of the finger is encoded according to the bending degree of the finger, the relative position of the finger and the palm, and the relative position between different fingers. The encoding rule defined in this embodiment is shown in the following table:

[0040]

[0041] According to the rule defined in the table, the number representing the state of the finger is filled into the string in the order of the finger, the action frame is encoded, and the action frame library is stored. Figure 2 Some finger states and their encodings are shown. In the action frame library, only the state of the finger itself and the relative position relationship between the fingers are concerned, and the position and posture of the hand relative to the space coordinate system are not concerned. For example, for the action of "fingers together, palm open", its encoding is "33333" regardless of the direction of the palm, which is not affected by the direction of the palm. In this way, the number of templates caused by the pose can be reduced as much as possible in the process of establishing the action frame library, the size of the action frame library is controlled, and the matching process is accelerated.

[0042] s1.4, the encoding X of the action frame in s1.3 is spliced in time sequence, and if the encoding results of adjacent two frames are the same, the merging processing is performed, to obtain the encoding result a corresponding to a complete gesture action, a=X1X2X3…X m , and is stored in the gesture action library. Wherein m represents the frame number of the action frame sequence corresponding to a complete gesture. For example, the gesture action of "hand clenched", after merging the same frames, includes "palm open", "four fingers half bent, thumb straight", "four fingers tightly held, thumb straight", and "five fingers clenched" four different action frames. The four action frames are encoded, and spliced in time sequence to obtain the encoding result a corresponding to the gesture action of "hand clenched", a=44434 22224 11114 11111.

[0043] s1.5, as Figure 3As shown, a camera coordinate system is established, and pose information U and V are added to the encoding result of each gesture to obtain the sequence code corresponding to the gesture. We assume that the poses of the same gesture should be similar, so we use X... m The pose of a frame represents the pose of the entire gesture. Where U represents... coordinates (x) u ,y u ,z u ), It is the normal vector of the plane containing the palm, V represents coordinates (x) v ,y v ,z v ), This is the direction vector of the line connecting the palm and the metacarpophalangeal joint of the middle finger. Then, the sequence codes of multiple hand gestures are placed into a matrix according to their temporal order, forming the action matrix ω corresponding to the hand gestures, thus establishing a hand gesture library. Each row of the action matrix ω is a sequence code of a hand gesture. For hand gestures with a sequence code length less than k or fewer than k hand gestures, zeros are used to pad the code, making the size of the action matrix ω k*k. Figure 4 As shown.

[0044] s1.6 Repeat steps s1.1 to s1.5 to collect 3D skeletal motion sequences of different hand movements, supplement the motion frame library, gesture motion library and hand motion library data, and complete the establishment of the matching template library.

[0045] Step 2: Hand Movement Recognition

[0046] s2.1 Acquire the 3D skeletal motion sequence T of the hand movement to be recognized, and extract the angles, velocities, and relative positions of the hand bone joints in the sequence in 3D space frame by frame. Calculate the joint velocity f(v) of all hand bone joints at different times frame by frame:

[0047]

[0048] n represents the number of joints in the hand bones, v ix v iy v iz Let w represent the velocities of the i-th joint in the x, y, and z axes, respectively. i The weight of the i-th joint is obtained through neural network training. Figure 5 The waveform of the calculated joint rate is shown.

[0049] s2.2、In the process of completing various hand gestures, the finger goes from static to motion, and then from motion to static. Each bone node will present different motion states and trends. The speed of each bone joint node in the three coordinate axis directions will change with the motion state, showing the process of acceleration and deceleration. Therefore, the complex process represented by a longer 3D bone motion sequence can be divided into several simple processes based on the change of the joint speed, and each process is regarded as a simple gesture action. For Figure 6 As shown in the speed waveform diagram, the position where the joint speed reaches the minimum value is taken as the segmentation point, and the 3D bone motion sequence T is segmented into multiple action frame sequences. An action frame sequence is regarded as a gesture action to be estimated.

[0050] s2.3、As Figure 7 shown, for each action frame sequence, the frame is matched with the template stored in the action frame library to calculate the similarity. When the similarity is less than the threshold, the frame is regarded as an interference frame and is discarded. Otherwise, the code corresponding to the action frame with the highest similarity in the library is selected as the code of the frame.

[0051] s2.4、The codes of each frame are spliced in time sequence. If the codes of two adjacent frames in an action frame sequence are the same, the merging process is performed to obtain the code result of a gesture action. The code result is matched with the template stored in the gesture action library to identify the gesture action, as Figure 8 shown.

[0052] s2.5、The pose information U and V is added to the code result of each gesture action obtained in s2.4 to obtain the sequence code corresponding to the gesture action. Then, the sequence codes of multiple gesture actions are placed in a k*k matrix in time sequence. For gesture actions with a sequence code length less than k or hand actions with a gesture action number less than k, 0 is supplemented to obtain the action matrix ω of the hand action to be estimated. The action matrix ω is matched with the template stored in the hand action library to complete the recognition of the hand action, as Figure 9 shown.

Claims

1. A hand action recognition method based on 3D gesture sequence analysis, characterized in that: Specifically comprising the following steps: Step one, establish matching template library s1.1, collect 3D skeleton action sequence in the process of performing a hand action, extract the angle, speed and relative position of the hand skeleton joint in the sequence by frame; s1.2, divide the 3D skeleton action sequence obtained by s1.1 according to the gesture action, and obtain a plurality of action frame sequences after removing the blank frames and interference frames; s1.3 define a 5-bit string, each bit in the string corresponds to the state of a finger, assign a state to each finger according to the bending degree of the finger, the relative position of the finger and the palm, and the relative position between different fingers, obtain an action frame code composed of 5-bit characters, and store it in the action frame library; s1.4, splice the coding X of s1.3 according to the time sequence, if the coding results of adjacent two frames are the same, merge processing is made, obtain the coding result a corresponding to an action frame sequence, and store it in the gesture action library; s1.5, add pose information U, V to the coding result of each gesture action, and obtain the sequence code corresponding to the gesture action; Where U represents the normal vector of the plane containing the palm. coordinates (x) u ,y u ,z u V represents the direction vector of the line connecting the palm and the metacarpophalangeal joint of the middle finger. coordinates (x) v ,y v ,z v Then, the sequence codes of multiple gesture actions are put into a matrix according to the time sequence and padded with 0s to make the size of the matrix k*k, and stored in the hand action library. s1.6, repeat steps s1.1-s1.5, collect 3D skeleton action sequences of different hand actions, supplement the data of the action frame library, the gesture action library and the hand action library, and complete the establishment of the matching template library; Step two, hand action recognition s2.1, collect the 3D skeleton action sequence T of the hand action to be recognized, extract the angle, speed and relative position of the hand skeleton joint in the sequence by frame, and calculate the joint rate f(v) of all hand skeleton joints at different times: n represents the number of hand skeletal joints, v ix , v iy , v iz wi, x, y, z represent the velocity of the ith joint in the x-axis, y-axis, z-axis direction, w i wi represents the weight of the ith joint; s2.2, the position where the joint rate reaches a minimum value or is less than a threshold value is taken as a segmentation point, and the 3D skeleton action sequence T is divided into a plurality of action frame sequences; s2.3, for each action frame sequence, match the frame with the template stored in the action frame library, calculate the similarity, when the similarity is less than a threshold value, the frame is regarded as an interference frame and discarded, otherwise, the coding of the action frame with the highest similarity in the library is selected as the coding of the frame; s2.4, splice the coding of each frame according to the time sequence, if the coding of adjacent two frames of an action frame sequence is the same, merge processing is made, and the coding result of an action frame sequence is obtained; match the coding result with the template stored in the gesture action library, and recognize the gesture action; s2.5, add pose information U, V to the coding result of each gesture action obtained in s2.4, obtain the sequence code corresponding to the gesture action; then put the sequence codes of a plurality of gesture actions into a matrix of size k*k according to the time sequence, for gesture actions with sequence code length less than k or hand actions with gesture action number less than k, fill with 0 to complete, obtain the action matrix ω of the hand action to be estimated, and match it with the template stored in the hand action library, complete the recognition of the hand action.

2. The method of claim 1, wherein the method further comprises: determining a hand pose sequence based on the captured image sequence; and determining the hand action based on the determined hand pose sequence. The 3D skeleton action sequence in the process of hand action is obtained by visual algorithm, data glove or Leap Motion. 3.The method of claim 1, wherein the method further comprises: determining a hand pose sequence based on the hand pose; and determining a hand action based on the hand pose sequence. using a neural network to select the most interesting nodes from the hand skeletal joints, giving each node a different weight w i .

Images