Human action recognition method based on distance vector and multi-angle adaptive network

Abstract

The present invention discloses a method for human action recognition based on distance vector and multi-angle adaptive network. The steps of the present invention are as follows: using the Openpose tool to detect human skeleton points on each frame of the video in the data set to generate a video data set 2 with skeleton point coordinates and skeleton point names; inputting the video data set 2 into the multi-angle adaptive network, using the LSTM network and the convolution network to automatically learn and determine the rotation parameters and displacements required for the transition to the optimal viewing angle; according to the rotation parameters and displacements, the skeleton point coordinates are converted into the skeleton point coordinates under the optimal viewing angle; calculating the distance vector between the skeleton points under the optimal viewing angle, including the spatial distance and direction; inputting the skeleton point coordinates, the action category name and the distance vector between the skeleton points under the optimal viewing angle into the spatiotemporal graph convolution network; the spatiotemporal graph convolution network learns the characteristics of time and space, and judges the final action category. The present invention has good robustness and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of action recognition, and in particular to a method for human action recognition based on distance vectors and multi-angle adaptive networks, and belongs to the technical field of computer vision image processing. Background Art

[0002] With the rapid development of Internet technology and the gradual popularization of video surveillance equipment, video has become an important medium in everyone's daily life. The total amount of video data information is growing exponentially, and how to obtain and analyze the content in the video is particularly important. The task of human action recognition has become the focus of research in the field of computer vision, and has been widely used in human-computer interaction, video surveillance, video understanding, virtual reality and other fields.

[0003] Human action recognition based on skeleton point detection can solve this problem well. This method first detects the skeleton points of people in the input video sequence, and then constructs the predicted skeleton points into a graph structure, with the skeleton points as the vertices of the graph and the connections between the skeleton points as the edges of the graph. Then, the action recognition method is used to learn the graph structure features under time series and predict the action category. A large number of previous studies have sent fixed skeleton graphs into the network, which can only capture the local physical relationship between the skeleton points, resulting in the omission of implicit skeleton point correlation. In addition, the feature presentation of the same action is also very different under different heights and directions, which will cause errors in action category recognition. Summary of the invention

[0004] The purpose of the present invention is to provide a human action recognition method based on distance vector and multi-angle adaptive network to solve the problems of missing skeleton point correlation and lack of robustness at different angles in the action recognition process.

[0005] A method for human action recognition based on distance vector and multi-angle adaptive network, comprising the following steps:

[0006] Step (1) obtaining an action recognition dataset 1, using the Openpose tool to detect human skeleton points in each frame of the video in the dataset to generate a video dataset 2 with skeleton point coordinates and skeleton point names;

[0007] Step (2) inputs the video data set 2 into the multi-angle adaptive network, and uses the LSTM network and the convolutional network to automatically learn and determine the rotation parameters and displacements required to transform to the optimal viewing angle;

[0008] Step (3) converting the bone point coordinates into the bone point coordinates under the best viewing angle according to the rotation parameters and the displacement;

[0009] Step (4) calculating the distance vector between the skeleton points under the best viewing angle, including the spatial distance and direction;

[0010] Step (5) inputs the coordinates of the skeleton points under the best viewing angle, the action category name, and the distance vector between the skeleton points into the spatiotemporal graph convolutional network;

[0011] Step (6) The spatiotemporal graph convolutional network learns the temporal and spatial features and determines the final action category.

[0012] Furthermore, the multi-angle adaptive network construction process described in step (2) is as follows:

[0013] The multi-angle adaptive main network is mainly composed of two LSTM branch networks.

[0014] (2-1) Build an LSTM network to determine the displacement d k .

[0015] (2-2) Construct another LSTM network to learn and determine the appropriate viewing angle, that is, to obtain the rotation parameter α k,θ ,β k,θ ,γ k,θ .

[0016] The three parameters are respectively expressed as the angles of rotation around the x, y, and z axes. After obtaining the rotation parameters and the displacement at the specified angle, the skeleton coordinate data is rotated according to the parameters to obtain a new set of skeleton coordinate points. The new set of skeleton coordinate points and the corresponding action labels are then input into a network composed of multiple LSTMs to learn the skeleton data from end to end. After fully connecting the features, the output is finally obtained through the SoftMax function.

[0017] Furthermore, the process of obtaining the coordinates of the skeleton points under the best viewing angle described in step (3) is as follows:

[0018] The displacement d is obtained through the multi-angle adaptive network k and the rotation parameter R k,θ (α k,θ ,β k,θ ,γ k,θ ) After that, the new bone point coordinates are calculated according to formula (1).

[0019] v' k,i,θ =[x' k,i,θ ,y' k,i,θ ,z' k,i,θ ] T =R k,θ (v k,i -d k ) (1)

[0020] Further, the process of calculating the distance vector between the skeleton points in step (4) is as follows:

[0021] (4-1) The distance vector is mainly composed of the distance between the bone points and the relative direction of the bone points. The distance between the bone points can be expressed as:

[0022]

[0023]

[0024]

[0025] Where (x, y, z) is the coordinate of the skeleton point. If the output of human skeleton point detection is a two-dimensional coordinate, the depth information can be set to the same to realize the three-dimensional coordinate. Formula (2) represents the spatial distance between the i-th skeleton point and the j-th skeleton point in the same frame, and formula (3) represents the spatial distance between the i-th skeleton point in the m-th frame and the i-th skeleton point in the n-th frame.

[0026] (4-2) The relative position between skeleton points can be expressed as:

[0027]

[0028]

[0029] Where (x, y, z) is the coordinate of the bone point. is a direction vector. Formula (4) represents the orientation of the jth skeleton point relative to the ith skeleton point in the same frame. Formula (5) represents the orientation of the ith skeleton point in the nth frame relative to the ith skeleton point in the mth frame.

[0030] Furthermore, the process of constructing the spatiotemporal graph convolutional network described in step (6) is as follows:

[0031] (6-1) Let a spatiotemporal graph of a skeleton sequence with N nodes and K frames be G = (V, E), and its node set be V = {v ki |k=1,…,K,i=1,…,N}.

[0032] (6-2) According to the human body structure, the skeleton points of each frame are connected to form a spatial edge Es = {v ki v kj |(i,j)∈B}, connect the same nodes in two consecutive frames into edges, forming a temporal edge Et={v ki v (k+1)i}.

[0033] (6-3) The distance vector between the skeleton points, the coordinate position of the skeleton points, and the action name form a new input feature vector F new(v ti ). Taking the common two-dimensional image convolution operation as a reference, the spatiotemporal graph convolution operation can be written as:

[0034]

[0035]

[0036] Formula (6) represents the graph convolution operation of different skeleton points in the same frame, and formula (7) represents the graph convolution operation of the same skeleton point in different frames. mi ) is the domain constraint of the skeleton point. For the spatial graph convolutional network, B(v mi )={v mj |L(v mj ,v mi )≤D},L(v mj ,v mi ) means from v nj to v mi For the temporal graph convolutional network, K is the convolution kernel size, Γ controls the convolution kernel size in the time domain, that is, sets the maximum number of video frames. p is the sampling function, which obtains the surrounding neighboring skeleton points centered on the skeleton point, w is the weight function, and Z is the normalization term.

[0037] The following is a more detailed description of the spatiotemporal graph convolution operation:

[0038] In the image, the sampling function p is defined on the neighboring pixels with the center position x. In the skeleton point graph, the result of the sampling function refers to a skeleton point v mi The neighbor set B(v mi ), then the sampling function can be written as:

[0039] p(v mi ,v mj )=v mj (8)

[0040] For the weight function w, the neighbor set of the skeleton point is divided into 8 fixed subsets according to the distance. Each subset has a label. The corresponding weight function is obtained according to the label through the mapping function, which can be expressed as formula (9):

[0041] w(v mi ,v mj )=w'(L(v mi ,v mj )) (9)

[0042] Therefore, the graph convolution operation is further expressed as:

[0043]

[0044]

[0045] (6-4) During the testing phase, a video sequence is input and the skeleton points corresponding to each frame of the video are obtained through human skeleton point detection technology. Then, a series of action labels and corresponding scores are obtained through action recognition technology. The action label corresponding to the highest score is the final result.

[0046] The advantages of the present invention are as follows: a method for action recognition based on distance vector and multi-view adaptive network is proposed. First, the multi-view adaptive network is used to automatically determine the best observation points at different heights, and multiple skeleton point coordinate combinations under the same action category are generated, which improves the accuracy while also improving the robustness in specific scenes. Secondly, the distance vector between each skeleton point is used to capture the change information of the action, and rich linkage features can be learned, which improves the recognition accuracy based on the spatiotemporal graph network. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the overall implementation scheme of the present invention;

[0048] Figure 2 It is a schematic diagram of the best viewing angle at different angles (different heights);

[0049] Figure 3 It is a schematic diagram of the skeleton at different angles (different heights);

[0050] Figure 4 It is a multi-angle adaptive network model diagram;

[0051] Figure 5 It is a schematic diagram of the distance vector of the skeleton point;

[0052] Figure 6 It is a schematic diagram of the space-time graph network; DETAILED DESCRIPTION

[0053] The present invention is described in detail below with reference to the accompanying drawings.

[0054] Example 1

[0055] In actual scenes, human motion is affected by various factors. The same motion will be quite different at different angles and the relationship between skeleton points is not fully utilized. This paper proposes a human motion recognition method based on distance vector and multi-angle adaptive network. The flowchart is shown in Figure 1 , including the following steps:

[0056] Step (1) obtaining an action recognition dataset 1, using the Openpose tool to detect human skeleton points in each frame of the video in the dataset to generate a video dataset 2 with skeleton point coordinates and skeleton point names;

[0057] Step (2) inputs the video data set 2 into the multi-angle adaptive network, and uses the LSTM network and the convolutional network to automatically learn and determine the rotation parameters and displacements required to transform to the optimal viewing angle;

[0058] The multi-angle adaptive main network is mainly composed of two LSTM branch networks.

[0059] (2-1) Build an LSTM network to determine the displacement d k .

[0060] (2-2) Construct another LSTM network to learn and determine the appropriate viewing angle, that is, to obtain the rotation parameter α k,θ ,β k,θ ,γ k,θ .

[0061] The three parameters are respectively expressed as the angles of rotation around the x, y, and z axes. After obtaining the rotation parameters and the displacement at the specified angle, the skeleton coordinate data is rotated according to the parameters to obtain a new set of skeleton coordinate points. The new set of skeleton coordinate points and the corresponding action labels are then input into a network composed of multiple LSTMs to learn the skeleton data from end to end. After fully connecting the features, the output is finally obtained through the SoftMax function.

[0062] Step (3) converting the bone point coordinates into the bone point coordinates under the best viewing angle according to the rotation parameters and the displacement;

[0063] The displacement d is obtained through the multi-angle adaptive network k and the rotation parameter R k,θ (α k,θ ,β k,θ ,γ k,θ ) After that, the new bone point coordinates are calculated according to formula (1).

[0064] v' k,i,θ =[x' k,i,θ ,y' k,i,θ ,z' k,i,θ ] T =R k,θ (v k,i -d k ) (1)

[0065] Step (4) calculating the distance vector between the skeleton points under the best viewing angle, including the spatial distance and direction;

[0066] (4-1) The distance vector is mainly composed of the distance between the bone points and the relative direction of the bone points. The distance between the bone points can be expressed as:

[0067]

[0068]

[0069] Where (x, y, z) is the coordinate of the skeleton point. If the output of human skeleton point detection is a two-dimensional coordinate, the depth information can be set to the same to realize the three-dimensional coordinate. Formula (2) represents the spatial distance between the i-th skeleton point and the j-th skeleton point in the same frame, and formula (3) represents the spatial distance between the i-th skeleton point in the m-th frame and the i-th skeleton point in the n-th frame.

[0070] (4-2) The relative position between skeleton points can be expressed as:

[0071]

[0072]

[0073] Where (x, y, z) is the coordinate of the bone point. is a direction vector. Formula (4) represents the orientation of the jth skeleton point relative to the ith skeleton point in the same frame. Formula (5) represents the orientation of the ith skeleton point in the nth frame relative to the ith skeleton point in the mth frame.

[0074] Step (5) inputs the coordinates of the skeleton points under the best viewing angle, the action category name, and the distance vector between the skeleton points into the spatiotemporal graph convolutional network;

[0075] Step (6) The spatiotemporal graph convolutional network learns the temporal and spatial features and determines the final action category.

[0076] (6-1) Let a spatiotemporal graph of a skeleton sequence with N nodes and K frames be G = (V, E), and its node set be V = {v ki |k=1,…,K,i=1,…,N}.

[0077] (6-2) According to the human body structure, the skeleton points of each frame are connected to form a spatial edge Es = {v ki v kj |(i,j)∈B}, connect the same nodes in two consecutive frames into edges, forming a temporal edge Et={v ki v (k+1)i}.

[0078] (6-3) After the output of human skeleton point detection is converted into three-dimensional data and then processed through the above-mentioned spatiotemporal graph, the graph structure data is input into the action recognition network model, and the feature information of the skeleton sequence in time and space and the action label corresponding to the feature information are learned at the same time.

[0079] (6-4) During the testing phase, a video sequence is input and the skeleton points corresponding to each frame of the video are obtained through human skeleton point detection technology. Then, a series of action labels and corresponding scores are obtained through action recognition technology. The action label corresponding to the highest score is the final result.

[0080] Example 2

[0081] As attached Figure 2 , which are the best viewing angles at different heights. It can be seen that this is a bending action sequence. The first row is 4 frames extracted from the original video, and the second row is the original skeleton points and the corresponding human action modeling. The third row is the best viewing angle when the viewing angle and the horizontal angle are 0°. It can be found that the transformed skeleton information is easier to learn, but the action has not changed. The fourth and fifth rows are respectively the viewing angle and the horizontal angle θ 1 With θ 2 The best viewing angle and corresponding human modeling under θ 2 >θ 1 It can be found that the best viewing angles at different heights are also different. By transforming the skeleton points, we can obtain the skeleton points at different heights and viewpoints, and train them together to achieve the purpose of data enhancement.

[0082] Example 3

[0083] As attached Figure 3 , sub-images (a), (b), and (c) are new perspectives at different heights in a certain video frame. (a) is the best perspective determined by the adaptive network, and (b) and (c) are the best perspectives under the added angle constraint. From the 1st frame to the tth frame, the human body posture keeps changing. The multi-view adaptive network can always find the best perspective and improve the height on this basis to learn.

[0084] Example 4

[0085] As attached Figure 4 , the original skeleton coordinates of a certain frame of the video are input into the network. The main network is mainly composed of two LSTM branch networks. The first LSTM network is used to determine the displacement d k , another LSTM network is used to learn and determine the appropriate viewing angle, that is, to obtain the rotation parameter α k,θ ,β k,θ ,γ k,θ. The three parameters are respectively expressed as the angles of rotation around the x, y, and z axes. After obtaining the rotation parameters and the displacement at the specified angle, the skeleton coordinate data is rotated according to the parameters to obtain a new set of skeleton coordinate points. The new set of skeleton coordinate points and the corresponding action labels are then input into a network composed of multiple LSTMs to learn the skeleton data from end to end. After fully connecting the features, the output is finally obtained through the SoftMax function.

[0086] Example 5

[0087] As attached Figure 5 , assuming that (a) is the nth frame image of the video, (b) is the n+1th frame image of the video, for the K of the nth frame 1 The distance vectors of the skeleton points to other skeleton points are represented by dashed arrows, K 1 To K 2 The dotted arrows represent the distance vectors between the same skeleton points in different frames. It can be seen from the figure that at the n+1th frame, the skeleton point K 2 With bone point F 2 By constructing the distance vectors between different skeleton points in the same frame and the same skeleton points in different frames, the change information of the action can be effectively captured and learned through the spatiotemporal graph convolutional network.

[0088] Example 6

[0089] As attached Figure 6 ,When the human body is in motion for a period of time, the skeleton points of each frame are identified by the human skeleton point detection method, such as Figure 6 is the skeleton graph information of several frames. Let a skeleton sequence spatiotemporal graph with N nodes and K frames be G = (V, E), and its node set be V = v ki |k=1,…,K,i=1,…,N}, Figure 6 B k,i That is, it represents the i-th bone point in the k-th frame. According to the human body structure, the bone points of each frame are connected to form a spatial edge Es = {v ki v kj |(i,j)∈B}, where B is the set of all skeleton points in the kth frame, Figure 6 The dotted edge is the spatial edge. Connect the same nodes in two consecutive frames to form an edge, forming a temporal edge Et = {v ki v( k+1 ) i}, Figure 6 The dashed edge with an arrow is the timing edge.

Claims

1. A human action recognition method based on distance vector and multi-angle adaptive network, which is characterized by the following steps: Step (1) obtain action recognition dataset I, use Openpose tool to detect human skeleton points in each frame of the video in dataset I to generate video dataset II with skeleton point coordinates and skeleton point names; Step (2) inputting the video data set II into the multi-angle adaptive network, using the LSTM network and the convolutional network to automatically learn and determine the rotation parameters and displacement required to transform to the optimal viewing angle; Step (3) converting the bone point coordinates into the bone point coordinates under the best viewing angle according to the rotation parameters and the displacement; Step (4) calculating the distance vector between the skeleton points under the best viewing angle, including the spatial distance and direction; Step (5) inputs the coordinates of the skeleton points under the best viewing angle, the action category name, and the distance vector between the skeleton points into the spatiotemporal graph convolutional network; Step (6) The spatiotemporal graph convolutional network learns the characteristics of time and space and determines the final action category; Step (2) is specifically implemented as follows: The multi-angle adaptive main network is mainly composed of two LSTM branch networks; (2-1) Construct an LSTM branch network to determine the displacement d k ; (2-2) Construct another LSTM branch network to learn and determine the best viewing angle, that is, to obtain the rotation parameter α k,θ ,β k,θ ,γ k,θ ; Among them, α k,θ ,β k,θ ,γ k,θ They are respectively expressed as the angles of rotation around the x, y, and z axes; after obtaining the rotation parameters and the displacement at the specified angle, the skeleton coordinate data is rotated according to the parameters to obtain a new set of skeleton coordinate points; then the new set of skeleton coordinate points and the corresponding action labels are input into a network composed of multiple LSTMs, the skeleton data is learned from end to end, the features are fully connected, and finally the output is obtained through the SoftMax function; The process of obtaining the coordinates of the skeleton points under the best viewing angle described in step (3) is as follows: The displacement d is obtained through the multi-angle adaptive network k With the rotation parameter P k,θ (α k,θ ,β k,θ ,γ k,θ ) After that, the new bone point coordinates are calculated according to formula (1); v' k,i,θ =[x' k,i,θ ,y' k,i,θ ,z' k,i,θ ] T =R k,θ (v k,i -d k ) (1) x' k,i,θ ,y' k,i,θ ,z' k,i,θ Respectively represent the coordinates obtained after the rotation parameters of the x, y, and z axes; v k,i Represents the original bone point coordinates; The process of calculating the distance vector between the skeleton points in step (4) is as follows: (4-1) The distance vector is mainly composed of the distance between the bone points and the relative direction of the bone points. The distance between the bone points is expressed as: Where (x, y, z) is the coordinate of the bone point. If the output of human bone point detection is a two-dimensional coordinate, the depth information can be set to the same to realize the three-dimensionalization of the two-dimensional coordinate. Formula (2) represents the spatial distance between the i-th bone point and the j-th bone point in the same frame. Formula (3) represents the spatial distance between the i-th bone point in the m-th frame and the i-th bone point in the n-th frame. (4-2) The relative position between the skeleton points is expressed as: Where (x, y, z) is the coordinate of the bone point. is a direction vector, formula (4) represents the orientation of the jth bone point relative to the ith bone point in the same frame, and formula (5) represents the orientation of the ith bone point in the nth frame relative to the ith bone point in the mth frame; The construction process of the spatiotemporal graph convolutional network described in step (6) is as follows: (6-1) Let a spatiotemporal graph of a skeleton sequence with N nodes and K frames be G = (V, E), and its node set be V = {v ki |k=1,…,K,i=1,…,N}; (6-2) According to the human body structure, the skeleton points of each frame are connected to form a spatial edge Es = {v ki v kj |(i,j)∈B}, connect the same nodes in two consecutive frames into edges, forming a temporal edge Et={v ki v (k+1)i }; (6-3) The distance vector between the skeleton points, the coordinate position of the skeleton points, and the action name form a new input feature vector F new (v ti ), the spatiotemporal graph convolution operation is as follows: Formula (6) represents the graph convolution operation of different skeleton points in the same frame, and formula (7) represents the graph convolution operation of the same skeleton point in different frames; where B(v mi ) is the domain constraint of the skeleton point. For the spatial graph convolutional network, B(v mi )={v mj |L(v mj ,v mi )≤D},L(v mj ,v mi ) means from v mj to v mi The shortest distance of; For the time graph convolution network, K is the convolution kernel size, Γ controls the convolution kernel size in the time domain, that is, sets the maximum interval video frame number; p is the sampling function, which obtains the surrounding neighboring skeleton points centered on the skeleton point, w is the weight function, and Z is the normalization term; The following is a more detailed description of the spatiotemporal graph convolution operation: In the image, the sampling function p is defined on the neighboring pixels with the center position x; in the skeleton point graph, the result of the sampling function refers to a skeleton point v mi The neighbor set B(v mi ), then the sampling function can be written as: p(v mi ,v mj )=v mj (8) For the weight function w, the neighbor set of the skeleton point is divided into 8 fixed subsets according to the distance. Each subset has a label. The corresponding weight function is obtained according to the label through the mapping function, which can be expressed as formula (9): w(v mi ,v mj )=w'(L(v mi ,v mj ))(9) Therefore, the graph convolution operation is further expressed as: (6-4) During the testing phase, a video sequence is input and the skeleton points corresponding to each frame of the video are obtained through human skeleton point detection technology. Then, a series of action labels and corresponding scores are obtained through action recognition technology. The action label corresponding to the highest score is the final result.

Images