A mobile terminal real-time action detection and action segmentation method

Abstract

The present application relates to the technical field of computer vision, and provides a mobile terminal real-time action detection and action segmentation method, which comprises the following steps: collecting and defining a standard action video in advance, and extracting human skeleton information of each frame; obtaining a preliminary start frame and end frame through action recognition and classification, extracting human skeleton information of each frame in the interval (between the start frame and the end frame), comparing the skeleton information with that of the standard action, aligning the skeleton information by using a dynamic time warping (DTW) method, finding an optimal path, and determining the start frame and the end frame of the action according to the optimal path. Through the method, the action video is optimized and adjusted, the input image is output as a segmented action on a mobile phone, and the gpu can reach more than 40 fps in a single-person case. Since the start frame and the end frame of the action video are more accurate, the method is closer to the real action of a human body.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for real-time motion detection and motion segmentation on mobile devices. Background Technology

[0002] Locating and classifying actions in an unedited video is called action detection. Its task is to find the start and end frames of an action and classify them. Action segmentation goes a step further, dividing already classified actions into smaller segments and assigning action labels to each segment. There is no clear boundary between action detection and action segmentation; longer time segments can be called action detection, while shorter segments are considered action segmentation.

[0003] Action detection typically involves detecting actions in pre-recorded videos. For example, deep learning methods can be used to regress the positions of the start and end frames, while simultaneously classifying the video segments. Current action detection and segmentation methods rely on deep learning only for action classification (recognition), while the start and end frames are determined through statistical and logical reasoning. The advantage of this approach is the simplicity and lightweight nature of the deep learning model. It can achieve relatively good real-time action detection and segmentation even with limited resources on mobile devices. However, the start and end frames obtained using this method are not precise enough to accurately represent real-world human movements. Summary of the Invention

[0004] This invention primarily addresses the technical problem that in existing technologies, the start and end frames are not accurately determined through statistical and logical judgments. It proposes a real-time motion detection and motion segmentation method for mobile devices to improve the accuracy of the start and end frames, thereby making them more closely resemble real human movements.

[0005] This invention provides a method for real-time motion detection and motion segmentation on a mobile device, comprising:

[0006] Define a standard human motion video and extract the standard human skeleton information for each frame of the standard human motion video;

[0007] Perform preliminary motion recognition and motion segmentation on the current human motion video, and extract the current human skeleton information of each frame within the start frame and end frame interval of the current human motion video;

[0008] Based on the action recognition, the action sequence frames of the current human action video are aligned with the standard action sequence frames of the standard human action video using dynamic time warping, and the skeletal information of the current human body and the standard human body are compared.

[0009] Based on the alignment result, inaccurate action frames are discarded to obtain the optimal path, and the start and end frames of the action are selected based on the optimal path.

[0010] Action segmentation is performed based on the selected start and end frames.

[0011] Furthermore, the preliminary action recognition of the current human action video includes: performing action recognition every 24 frames of the current human action video; when the action recognition result is greater than a set threshold, it is pushed into the corresponding queue according to the action recognition type.

[0012] Furthermore, the step of pushing the action into the corresponding queue based on the action recognition type includes: at least k consecutive frames must be detected before pushing the action into the corresponding queue.

[0013] Furthermore, the preliminary action recognition of the current human action video also includes: n actions corresponding to n queues; when the i-th action among the n actions detects the m-th frame, and when m is greater than the minimum number of frames for the action, the m-th frame is determined to be a complete action sequence frame; n, i, and m are all variables greater than or equal to 1.

[0014] Furthermore, the ST-GCN action recognition network is used for action recognition.

[0015] Furthermore, the step of aligning the action sequence frames of the current human motion video with the standard action sequence frames of the human standard motion video using dynamic time warping based on the action recognition includes: performing dynamic time warping (DTW) on the X, which includes n1 current action sequence frames, and the standard action Y, which includes n2 standard action sequence frames, to obtain the path.

[0016] Further, the dynamic time warping (DTW) is performed on the action-identified X, which includes n1 current action sequence frames, and the standard action Y, which includes n2 standard action sequence frames, to obtain the path, which includes: x i y j Let i ∈ (1,2,...,n1) and j ∈ (1,2,...,n2) be vectors composed of v-1 dimensional cosine values. The cosine values ​​are obtained by the vector cosine formula, involving formulas such as (1) and (2):

[0017] dis,R,D=DTW(X,Y) (1)

[0018] path = find_path(R,D) (2)

[0019] In the formula, v is the number of human joints; dis is the shortest path value; R is the distance accumulation matrix; D is the distance matrix; X = (x1, x2, ..., x n1Y = (y1, y2, ..., y) n2 ).

[0020] Further, based on the alignment result, inaccurate action frames are discarded to obtain the optimal path, and the start and end frames of the action are selected based on the optimal path, including:

[0021] By working backwards from R, we can obtain an optimal path (p1, q1);

[0022] Calculate the average distance of p1. If the value at the corresponding position in the distance matrix D is greater than the average distance, discard this path node. After filtering, the path (p2, q2) is obtained, which can be expressed by equations (3) and (4):

[0023] n = len(p1) (3)

[0024] mean_dis = dis / len(p1) (4)

[0025] In the formula, n is the number of path nodes of p1, and dis is the shortest path obtained by Dynamic Time Warping (DTW).

[0026] If p2 has i nodes that correspond to a node q in q2 j If i > 1, then find the sum node q from the i nodes. j The node with the smallest distance is selected as a candidate node, and other nodes are discarded; multiple nodes in q2 correspond to a single node in P2, and the node with the smallest distance is also selected from the i nodes. j The node with the smallest distance is selected as the candidate node, and other nodes are discarded. Finally, the path (p3, q3) is obtained.

[0027] The final output path is (p3, q3), i.e., path = (p3, q3).

[0028] The action frame sequence corresponding to the first and last path nodes of p3 is used as the start and end frames of the action.

[0029] This invention provides a real-time motion detection and segmentation method for mobile devices. It pre-collects and defines a video of a standard motion and extracts the human skeleton information from each frame. Then, using the initially obtained start and end frames, it extracts the human skeleton information from each frame within this interval. By comparing the two sets of human skeleton information, aligning them using Dynamic Time Warping (DTW), and performing a final fine segmentation, it improves the accuracy of the start and end frames, resulting in a more accurate reflection of real human motion. Attached Figure Description

[0030] Figure 1 This is a flowchart of the mobile real-time motion detection and motion segmentation method of the present invention;

[0031] Figure 2 The subgraph structure of GCN in the embodiment represents the motion characteristics of centripetal motion, centrifugal motion, and stationary motion from left to right;

[0032] Figure 3 This is a schematic diagram of the action sequence frame in this invention;

[0033] Figure 4 This is a diagram illustrating the cosine evaluation method in this invention;

[0034] Figure 5 This is a diagram of the human skeleton in the embodiment. Detailed Implementation

[0035] To make the technical problems solved by this invention, the technical solutions adopted, and the technical effects achieved clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings, not all of them.

[0036] like Figure 1 As shown, the mobile real-time motion detection and motion segmentation method provided in this embodiment of the invention includes:

[0037] 101. Define a standard human motion video and extract the standard human skeleton information of each frame of the standard human motion video;

[0038] Specifically, to address the issue of inaccurate start and end frames, a standard motion video was pre-collected and defined, and the human skeleton information for each frame was extracted. The human skeleton information is as follows: Figure 5 As shown.

[0039] Figure 5 In the code, the mapping between key node IDs and names is as follows:

[0040] (0,"nose"),(1,"left eye"),(2,"right eye"),(3,"left ear"),(4,"right ear")

[0041] (5,"left shoulder"),(6,"right shoulder"),(7,"left elbow"),(8,"right elbow"),(9,"left wrist"),

[0042] (10,"Right Wrist"), (11,"Left Hip"), (12,"Right Hip"), (13,"Left Knee"), (14,"Right Knee")

[0043] (15,"Left ankle"),(16,"Right ankle"),(17,"Top of head"),(18,"Neck"),(19,"Hip")

[0044] (20,"Left Big Toe"),(21,"Right Big Toe"),(22,"Left Little Toe"),(23,"Right Little Toe"),(24,"Left Heel")

[0045] (25,"Right Heel"), (26,"Left Thumb"), (27,"Right Thumb"), (28,"Left Middle Finger"), (29,"Right Middle Finger")

[0046] 102. Perform preliminary motion recognition and motion segmentation on the current human motion video, and extract the current human skeleton information of each frame in the interval between the start frame and the end frame of the current human motion video.

[0047] Specifically, the ST-GCN action recognition network is used for action recognition.

[0048] st-gcn is a combination of TCN and GCN. TCN performs convolution operations on data in the time dimension, while GCN performs convolution operations on data in the spatial dimension.

[0049] like Figure 2 As shown, the graph is divided into three subgraphs, representing the motion characteristics of centripetal motion, centrifugal motion, and stillness, respectively. Each subgraph has one convolution kernel, for a total of three convolution kernels (3, V, V), where V is the number of joints.

[0050] From left to right, the first image connects neighboring nodes that are spatially farther from the center of gravity of the entire skeleton than this node, and includes features of centrifugal motion.

[0051] The second image connects neighboring nodes that are closer to the center of gravity, and includes the characteristics of centripetal motion.

[0052] The third image connects to the root node itself and contains static features.

[0053] The graph convolution expression with K convolution kernels is:

[0054]

[0055] The expression can be represented using the Einstein summation convention as follows (k represents the number of convolution kernels, such as k=3 above; c represents the number of channels; t represents the number of frames; v and w represent the number of feature points or key points):

[0056] kctv, kvw->nctw

[0057] The last three dimensions of the ST-GCN feature map have a shape of (C, T, V), corresponding to the shape (C, H, W) of a typical image feature map. The number of image channels C corresponds to the number of feature map channels C; the image height H corresponds to the number of frames T; and the image width W corresponds to the number of feature points V (joints). Each time a (size, 1) convolution kernel is used, a convolution operation can be performed in the T direction, such as when size = 3.

[0058] The action recognition method used in this embodiment performs action recognition every 24 (greater than 10) frames of the current human action video. When the action recognition result exceeds a set threshold, it is pushed into the corresponding queue according to the action recognition type. Enqueueing conditions can be added, such as requiring at least k consecutively detected frames before being pushed into the queue. Assuming there are n actions, n queues are needed, such as... Figure 3 As shown, assuming the i-th action detects m frames, when m is greater than a threshold (the minimum number of frames for an action), it is determined to be a complete action sequence frame. n, i, and m are all variables greater than or equal to 1.

[0059] 103. Based on action recognition, align the action sequence frames of the current human action video with the standard action sequence frames of the human standard action video using dynamic time warping, and compare the skeletal information of the current human body and the standard action human body.

[0060] Specifically, dynamic time warping (DTW) is performed using the action-identified X, which includes n1 current action sequence frames, and the standard action Y, which includes n2 standard action sequence frames, to calculate the path.

[0061] like Figure 4 As shown, x i y j Let i ∈ (1,2,...,n1) and j ∈ (1,2,...,n2) be vectors composed of v-1 dimensional cosine values, where the cosine values ​​are obtained using the vector cosine formula, such as... Figure 4 As shown. The relevant formulas are as follows: (1) and (2):

[0062] dis,R,D=DTW(X,Y) (1)

[0063] path = find_path(R,D) (2)

[0064] In the formula, v is the number of human joints; dis is the shortest path value; R is the distance accumulation matrix; D is the distance matrix; X = (x1, x2, ..., x n1 Y = (y1, y2, ..., y) n2 ).

[0065] 104. Based on the alignment results, discard inaccurate motion frames to obtain the optimal path, and select the start and end frames of the motion based on the optimal path.

[0066] 105. Perform motion segmentation based on the selected start and end frames.

[0067] Specifically, the rules for find_path(R,D) are as follows:

[0068] By working backwards from R, we can obtain an optimal path (p1, q1);

[0069] Calculate the average distance of p1. If the value at the corresponding position in the distance matrix D is greater than the average distance, discard this path node. After filtering, the path (p2, q2) is obtained, which can be expressed by equations (3) and (4):

[0070] n = len(p1) (3)

[0071] mean_dis = dis / len(p1) (4)

[0072] In the formula, n is the number of path nodes of p1, and dis is the shortest path obtained by Dynamic Time Warping (DTW).

[0073] If p2 has i nodes that correspond to a node q in q2 j If i > 1, then find the sum node q from the i nodes. j The node with the smallest distance is selected as a candidate node, and other nodes are discarded; multiple nodes in q2 correspond to a single node in P2, and the node with the smallest distance is also selected from the i nodes. j The node with the smallest distance is selected as the candidate node, and other nodes are discarded. Finally, the path (p3, q3) is obtained.

[0074] The final output path is (p3, q3), i.e., path = (p3, q3).

[0075] The action is segmented using the action frame sequence corresponding to the first and last path nodes of p3 as the start and end frames of the action.

[0076] In terms of technical implementation, this embodiment is as follows:

[0077] Action recognition and correction model inference use TNN, action sequence frames are managed through queues, and the optimal path is solved using C++. The entire architecture is built using C++.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications to the technical solutions described in the foregoing embodiments, or equivalent substitutions for some or all of the technical features, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time motion detection and motion segmentation on a mobile device, characterized in that, The method includes: Define a standard human motion video and extract the standard human skeleton information for each frame of the standard human motion video; Perform preliminary motion recognition and motion segmentation on the current human motion video, and extract the current human skeleton information of each frame within the start frame and end frame interval of the current human motion video; Based on the action recognition, the action sequence frames of the current human action video are aligned with the standard action sequence frames of the human standard action video using dynamic time warping, and the skeletal information of the current human and the standard action human are compared; wherein, the step of aligning the action sequence frames of the current human action video with the standard action sequence frames of the human standard action video using dynamic time warping includes: performing dynamic time warping (DTW) on the action-recognized X, which includes n1 current action sequence frames, and the standard action Y, which includes n2 standard action sequence frames, to obtain the path; The process of using action recognition to perform dynamic time warping (DTW) on X, which includes n1 current action sequence frames, and standard action Y, which includes n2 standard action sequence frames, to obtain the path includes: It is a vector composed of v-1 dimensional cosine values. The cosine value is obtained by the vector cosine formula, involving formulas such as (1) and (2): ； In the formula, v is the number of human joints; dis is the shortest path value; R is the distance accumulation matrix; and D is the distance matrix. ; Based on the alignment result, inaccurate action frames are discarded to obtain the optimal path. The start and end frames of the action are then selected based on the optimal path, including: By working backwards from R, we can obtain an optimal path (p1, q1). Calculate the average distance of p1. If the value at the corresponding position in the distance matrix D is greater than the average distance, discard this path node. After filtering, the path (p2, q2) is obtained, which can be expressed by equations (3) and (4): ； In the formula, n is the number of path nodes of p1, and dis is the shortest path obtained by Dynamic Time Warping (DTW). If p2 has i nodes that correspond to a node q in q2 j If i > 1, then find the sum node q from the i nodes. j The node with the smallest distance is selected as a candidate node, and other nodes are discarded; multiple nodes in q2 correspond to a single node in P2, and the node with the smallest distance is also selected from the i nodes. j The node with the smallest distance is selected as the candidate node, and other nodes are discarded. Finally, the path (p3, q3) is obtained. The final output path is (p3, q3), i.e., path = (p3, q3). The action frame sequence corresponding to the first and last path nodes of p3 is used as the start and end frames of the action. Action segmentation is performed based on the selected start and end frames.

2. The mobile terminal real-time motion detection and motion segmentation method according to claim 1, characterized in that, The preliminary motion recognition of the current human motion video includes: Action recognition is performed on the current human motion video every 24 frames. When the action recognition result is greater than a set threshold, it is pushed into the corresponding queue according to the action recognition type.

3. The mobile terminal real-time motion detection and motion segmentation method according to claim 2, characterized in that, The step of pushing the action into the corresponding queue based on the action recognition type includes: At least k consecutive frames must be detected before a frame can be pushed into the corresponding queue.

4. The mobile terminal real-time motion detection and motion segmentation method according to claim 2, characterized in that, The preliminary motion recognition of the current human motion video also includes: n actions correspond to n queues; when the i-th action among the n actions detects the m-th frame, and m is greater than the minimum number of frames for the action, the m-th frame is determined to be a complete action sequence frame; n, i, and m are all variables greater than or equal to 1.

5. The mobile terminal real-time motion detection and motion segmentation method according to claim 2, characterized in that, Action recognition is performed using the ST-GCN action recognition network.

Images