Action recognition method based on multi-scale feature fusion

Abstract

The application provides a motion recognition method based on multi-scale feature fusion, first estimates the joint of each frame image in a video by a human reference posture estimation method, enhances the joint, then defines feature information of a skeleton graph, is respectively joint information, skeleton information, and joint motion information, extracts multi-scale features of the joint motion information, inputs the feature information of the skeleton graph into a graph convolution network, dynamically updates a topological relation between nodes, fuses features between related nodes, outputs features for fusion, and recognizes motion of the video. Considering the similarity between graph nodes and the various feature information in the skeleton graph, and effectively eliminating the influence of redundant information when calculating the topological relation between the joints, the detection accuracy is high, the detection efficiency is fast, and the overall performance is better than that of the prior art.

Description

TECHNICAL FIELD

[0001] The application researches the technical field of video action recognition, and proposes an action recognition method based on multi-scale feature fusion, which can quickly and accurately recognize the action of an object in a video. BACKGROUND

[0002] Video action recognition is an important task, which is a task of classifying human actions in videos by receiving video data as input, and is widely used in human-computer interaction, virtual reality and video monitoring. The method based on RGB cannot reliably identify human behavior because it is strongly affected by environmental noise such as background color, light brightness and clothing. Therefore, methods using skeleton modalities have attracted attention because they are not affected by these noises.

[0003] Although the human eye has the ability to recognize human actions in videos, it has the disadvantages of long time consumption and heavy workload, therefore, it is particularly important to use computers to recognize characters in videos. At present, there are many methods of human action recognition based on skeleton graphs, such as Yan et al. modeling the correlation between human joints using graphs, manually setting the topology of the graph on the basis of the original physical connection of the skeleton graph, and then aggregating the joint feature through graph convolution network for classification; Lei, Ye, Zhang et al. adaptively learn the topology of human skeleton through attention or other mechanisms; Lei et al. use the skeleton information of the skeleton graph, propose a dual-flow graph convolution network, increase the feature types during classification, and improve the classification accuracy; Cheng et al. set a separate parameterized topology method for channel groups. These methods consider aspects that are relatively one-sided and do not combine multiple aspects. Therefore, it is impossible to accurately classify the actions of characters in videos, which further leads to increased difficulty in recognition and makes it impossible to achieve high-precision recognition.

[0004] In addition, there are many existing human action recognition methods based on skeleton graphs, which all use skeleton graphs, on the one hand to increase the information types of the skeleton graph, and on the other hand to perfect the topology structure of the skeleton graph and optimize the feature extraction of information. However, the skeleton graph data in the prior art is insufficient in granularity, which further affects the accuracy of action recognition of characters in videos. SUMMARY

[0005] Aiming at the deficiencies of the prior art, a video action recognition method based on multi-scale feature fusion is proposed, which scientifically obtains the topology structure of the skeleton graph by increasing the receptive field of the graph convolution to judge the correlation between different nodes, and defines more kinds of video skeleton graph features to classify the video action from multiple aspects. First, the person in each frame image in the video is estimated for the joint node, the joint node is enhanced, then the feature information of the skeleton graph is defined, respectively the joint node information, the skeleton information, and the joint node motion information, the multi-scale features of them are extracted, the feature information of the skeleton graph is input into the graph convolution network, the graph convolution network dynamically updates the topology relationship between the nodes according to the similarity of the joint nodes, and the features between the related nodes are fused, finally the output features are output, and the output features are fused to obtain the final spatio-temporal features of the video, and the spatio-temporal features are used for action recognition of the video.

[0006] The technical scheme of the application comprises the following steps:

[0007] Step 1, segmenting the video by frames, estimating the joint nodes of the objects in the image using the pose estimation method, obtaining the feature information of the joint nodes, and enhancing the feature information;

[0008] Step 2, extracting features from each frame image through the VGG19 convolution network to obtain a group of feature maps, then extracting confidence and correlation degree from the feature maps using the CNN network, obtaining the two information, using the bipartite graph matching in graph theory to find the partial correlation, connecting the joint nodes of the same person to obtain the skeleton graph and labeling the serial number of the joint nodes;

[0009] Step 3, using the obtained feature information of the joint nodes to calculate the coordinate information of the skeleton joint nodes in each frame to be recognized, the vector information of the skeleton edge, and the joint node motion information of the same joint node between adjacent frames;

[0010] Step 4, building and training the graph convolution network, the graph convolution network first calculates the adjacency matrix of the skeleton graph, i.e. the topology relationship between the joint nodes, then inputs the three types of information obtained in step 3 into the trained graph convolution network for feature extraction, and convolves the information between the nodes according to the topology relationship to obtain the corresponding three types of output feature information;

[0011] Step 5, fusing the output features obtained in step 4 to obtain the final spatio-temporal features of the video to be recognized;

[0012] Step 6, performing action recognition according to the final spatio-temporal features obtained in step 5 to obtain the action recognition result of the video to be recognized.

[0013] The application has the following beneficial effects:

[0014] The detection model of the application considers the similarity between graph nodes and various feature information in the skeleton graph, and effectively eliminates the influence of redundant information when calculating the topological relationship between the joints, so the detection accuracy is high, the detection efficiency is fast, and the overall performance is better than the prior art.

[0015] In order to more accurately calculate the relationship between nodes, the application uses the Gaussian embedding method to convert the features of the nodes to the embedding space to calculate the similarity, and uses the similarity result as the topological relationship between the nodes, so that the graph convolution network can more scientifically aggregate the features between the nodes. The application also performs multi-scale subdivision on the edge features of the skeleton graph, uses 1 / 2 nodes and 1 / 4 nodes on the skeleton edge to replace the information of the skeleton edge, increases the information of the skeleton edge, and reduces the network size. From two aspects, the accuracy of video human motion recognition is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 Algorithm flowchart of the embodiment of the application.

[0017] Figure 2 RGB video image sample.

[0018] Figure 3 Skeleton graph video image sample.

[0019] Figure 4 Skeleton graph schematic diagram with serial numbers in the embodiment of the application.

[0020] Figure 5 Graph convolution network structure schematic diagram in the embodiment of the application.

[0021] Figure 6 The accuracy of the application and other methods for recognizing video motion. DETAILED DESCRIPTION

[0022] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings.

[0023] The embodiment provides a motion recognition method based on a graph convolution network, as shown in Figure 1 The specific steps are described as follows:

[0024] Step 1, segment the video by frames, and use a pose estimation method to estimate the joints of the object in each frame image obtained by the segmentation, obtain the feature information of the joints, and enhance the feature information.

[0025] Specifically, the pose estimation method is used to estimate the key points of the object in the image. Input a video, and assume that the video to be recognized is S trepresents the tth frame of video, the to-be-identified video V has T frames of images, t is an element in (1, 2,..., T), S t = R H*W*C , wherein H, W and C represent the height, width and channel of the image respectively, as shown in Figure 2

[0026] , wherein the to-be-identified video V = {v ti | t = 1, 2,..., T, i = 1, 2,..., N}, i represents the ith joint, v ti represents the feature information of the ith joint in the tth frame;

[0027] Then the information of the joint is enhanced, because the similarity of the joint is different in different embedding spaces, in order to more accurately judge the topological relationship between the joints, the application uses a Gaussian kernel function to convert the information of the joint into a Gaussian embedding space, and then judges the topological relationship between the joints.

[0028] Gaussian kernel function formula:

[0029]

[0030] Step 2, each frame of image is extracted by a VGG19 convolution network to obtain a group of feature maps, and then the feature maps are respectively extracted by a CNN network to obtain confidence and correlation; after obtaining the two information, a bipartite graph matching in graph theory is used to find part of the correlation, and the joints of the same person are connected to obtain a skeleton graph.

[0031] , wherein the to-be-identified video V = {v ti | t = 1, 2,..., T, i = 1, 2,..., N}, i represents the ith joint, v ti represents the information of the ith joint in the tth frame, as shown in Figure 3

[0032] As shown in Figure 4 , and the joint is labeled with a serial number.

[0033] Step 3, using the obtained feature information of the joint, the coordinate information of the skeleton joint in each frame of the to-be-identified object, the vector information of the skeleton edge and the joint motion information of the same joint between adjacent frames are calculated.

[0034] ​​Specifically, according to the serial number of the human skeleton graph joint, the relative positions of the remaining joints to the first joint are calculated as the coordinate information of the joint. According to the serial number of the human skeleton graph joint, the vector direction of the skeleton connection edge is defined according to the distance from the first joint, from near to far, the far joint is taken as the end point of the vector of the skeleton connection edge, and the near joint is taken as the starting point of the vector of the skeleton connection edge. The end point minus the starting point is the vector of the skeleton connection edge, which is taken as the vector information of the skeleton. There is another information, since the motion recognition is performed on a video, the video has a time dimension, that is, the three consecutive frames of video are related. The application assumes that the same joints in the adjacent two frames of images are connected by an edge, and the joint moves from the previous frame of image to the next frame of image along the trajectory of the edge. This kind of information is considered by the application as motion information, which also exists in the form of vector. The joint of the next frame of image is taken as the end point of the motion information, and the joint of the previous frame of image is taken as the starting point of the motion information. The end point minus the starting point is the motion information.

[0035] Coordinate information of the joint:

[0036] v ti

[0037] Vector information of the skeleton, the calculation formula is as follows:

[0038] e ti =v ti -v tj

[0039] Motion information of the joint, the calculation formula is as follows:

[0040] m ti =v t+1i -v ti

[0041] A comparison reveals that the coordinate information of joints is point information, while the vector information of bone edges and the motion information of joints are vector information; these three types of information are different. Some literature suggests using the idea of ​​dual-stream detection, employing two feature extraction networks to extract features from the above information, then performing action recognition based on the extracted features, and summing the probabilities of the recognized actions; the highest probability is the action recognition result for the video. However, this method has a drawback: the network model becomes very large, with a large number of parameters. Therefore, we have invented a new method for multi-scale feature extraction of the motion trajectories of bone edges and joints. Since there are many points on the motion trajectories of bone edges and joints, and two points can determine a straight line, this invention abstracts the motion trajectories of bone edges and joints into four points to represent these two types of information, taking 1 / 2 and 1 / 4 points on the line segment to represent the line segment, respectively. In this way, all three types of information can be represented by points, and this invention only requires one feature extraction network for feature extraction, reducing the number of network parameters.

[0042] Step 4: Build and train the graph convolutional network. The graph convolutional network first calculates the adjacency matrix of the skeleton graph, that is, the topological relationship between the nodes. Then, the three types of information obtained in step 3 are input into the trained graph convolutional network for feature extraction. The information between nodes is convolved according to the topological relationship to obtain the three types of output feature information respectively.

[0043] Specifically, for skeleton graphs, the feature extraction network used in this invention is a graph convolutional network. For example... Figure 5 As shown, where X i C represents the number of input channels, and Z represents the number of input channels. i Let F represent the number of output channels, and let F represent the number of output channels. The connection relationship between nodes is represented by the topological relationship between nodes in the network. The topological relationship between nodes is crucial. Most graph convolutional networks only consider the relationships between nodes in a 1-neighborhood and then superimpose the features of nodes in that 1-neighborhood. This only considers the physical connections between nodes. In reality, many actions cannot be considered solely based on the physical connections between nodes. For example, in the action of clapping, the two hands have a strong relationship, but there is no direct physical connection between them. This invention assumes a virtual connection between the two hands, which is actually transmitted through the physical connection. Therefore, considering only the 1-neighborhood relationship between nodes is incomplete. When designing the adjacency matrix of the skeleton graph, this invention calculates the topological relationship between nodes by considering the 4-neighborhood of the nodes. The larger the neighborhood, the more accurate the relationship, but the more complex the calculation, and the more redundant information will appear. This invention changes all parts greater than 1 in the calculated adjacency matrix to 1 and parts less than 1 to 0, simplifying the calculation of the graph convolutional network and reducing the redundant information generated during calculation.

[0044] The calculation method of the topological relationship between nodes:

[0045]

[0046] Wherein, and are a 1*1 convolution layer, n is n other nodes related to the i-th node, v ti and the relationship between v tj is calculated after being converted to a Gaussian embedding space.

[0047] The graph convolution formula is as follows:

[0048]

[0049] Wherein, H l represents the input of the l-th layer, H l ∈R, W l is the weight parameter matrix of the l-th layer, σ(·) is a nonlinear activation function, A is an adjacency matrix, which is the topological relationship between nodes in the application, is a Laplacian matrix.

[0050] Step 5, the output features obtained in step 4 are fused to obtain the final spatio-temporal features of the video to be recognized.

[0051] Specifically, three kinds of feature information are obtained after the feature extraction of the graph convolution network in step 4, in order to make the recognition accuracy higher, the three kinds of feature information need to be fused, and the three kinds of information are combined to recognize the video action. However, the numerical significance and range of the three kinds of feature information are inconsistent, so the tensor addition method cannot be used to fuse the feature information, and the application selects the connection method, first normalizes the tensors of the three kinds of feature information, and then uses the connection method to fuse the three kinds of feature information to obtain a tensor.

[0052] Fusion of features:

[0053] Z=[v out ||e out ||m out ]

[0054] Wherein, v out is the coordinate information of the node output by the graph convolution network, e out is the vector information of the bone output by the graph convolution network, m out is the motion information of the node output by the graph convolution network, and || is the connection of the matrix.

[0055] Step 6, according to the final spatio-temporal features obtained in step 5, the action recognition is performed, and finally the action recognition result of the video to be recognized is obtained.

[0056] Specifically, the present application first uses the true value graph in a large database to train the neural network, that is, repeats steps 1 to 5, so that the neural network has the ability to recognize the action of the object in the video (input a video into the neural network, and the neural network recognizes that the object in the video is doing the action of a certain label), and then uses the verification set to verify it. The tensor obtained in step 5 (actually obtained by operating once on a to-be-detected video according to steps 1 to 5, that is, after feature extraction and feature fusion are completed in the neural network) is input to the last full connection layer of the network, the fused features are classified, and the action recognition result of the to-be-recognized video is obtained,

[0057] The accuracy of the present application and other methods is compared as shown in the table. Figure 6 As shown in the table, the accuracy of the action recognition method provided by the embodiment can reach 96.7%, which breaks through the accuracy limit of all existing action recognition methods.

[0058] The embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments including components can be made without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.

Claims

1. A motion recognition method based on multi-scale feature fusion, characterized in that, Comprise the following steps: Step 1, the video is segmented by frame, the feature information of the object in the image is estimated by using the pose estimation method of each frame image, and the feature information is enhanced; The specific process is as follows: The method for pose estimation is used for key point estimation of an object in an image, input a video, suppose a to-be-identified video S t represents the t-th frame video, the to-be-identified video V has T frames of images, t ∈ (1, 2,..., T), S t = R H*W*C , wherein H, W and C represent height, width and channel of the image respectively, wherein the to-be-identified video V = {v ti | t = 1, 2,..., T, i = 1, 2,..., N}, i represents the i-th key node, v ti represents feature information of the i-th key node of the t-th frame. The feature information of the joint node is enhanced, the information of the joint node is converted into the Gaussian embedding space by using the Gaussian kernel function, and the topological relationship between the joint nodes is judged; Step 2, each frame image is extracted by VGG19 convolution network to obtain a group of feature maps, then the feature maps are respectively extracted by CNN network to obtain confidence and correlation, after obtaining the two information, the bipartite graph matching in graph theory is used to find the partial correlation, the joint nodes of the same person are connected to obtain the skeleton graph, and the joint node serial number is labeled; Step 3, using the feature information of the obtained joint nodes, the coordinate information of the skeleton joint nodes in each frame to be identified object, the vector information of the skeleton edge and the joint node motion information between adjacent frames are calculated; Step 4, build and train the graph convolution network, the graph convolution network first calculates the adjacency matrix of the skeleton graph, that is, the topological relationship between the joint nodes, then the three types of information obtained in step 3 are respectively input into the trained graph convolution network for feature extraction, the information between the nodes is convolved according to the topological relationship, and the corresponding three types of output feature information are obtained; The method for calculating the adjacency matrix of the skeleton graph by the graph convolution network in step 4 is as follows: The graph convolution formula is as follows: where H l represents the input of the lth layer, H l ∈ R, W l is the weight parameter matrix of the lth layer, σ(·) is a nonlinear activation function, A is an adjacency matrix, i.e., the topological relationship between nodes, is the Laplacian matrix; The calculation method of the topological relationship between the nodes is as follows: where θ and φ is a 1*1 convolutional layer, n is the n other nodes related to the i-th node, v ti and v tj The relationship is to convert to Gaussian embedding space first and then calculate. Step 5, the output features obtained in step 4 are fused to obtain the final spatio-temporal features of the to-be-identified video; Step 6, according to the final spatio-temporal features obtained in step 5, action recognition is carried out, and finally the action recognition result of the to-be-identified video is obtained. 2.The method according to claim 1, wherein, The method for obtaining the coordinate information of the joint node in step 3: according to the serial number of the human skeleton graph joint node, the relative positions of the remaining joint nodes to the first joint node are calculated as the coordinate information v of the joint node ti . 3.The method of claim 2, wherein, The method for obtaining the vector information of the bone edge in step 3: according to the connection condition of the serial numbers of the human body skeleton graph joints, the vector direction of the bone edge is defined according to the distance from the first joint, from near to far, the far joint is taken as the terminal point of the bone edge vector, the near joint is taken as the starting point of the bone edge vector, the terminal point is subtracted from the starting point to obtain the bone edge vector, which is taken as the vector information of the bone, and the calculation formula is as follows: e ti = v ti -v tj . 4.The method of claim 3, wherein, The method for obtaining the motion information of the node in step 3: assuming that the same nodes are connected by an edge in the adjacent two frames of images, the node is moved from the previous frame of image to the next frame of image according to the track of the edge, the information is the motion information of the node, which exists in the form of vector, the node of the next frame of image is taken as the terminal point of the motion information, the node of the previous frame of image is taken as the starting point of the motion information, the motion information is obtained by subtracting the starting point from the terminal point, and the calculation formula is as follows: ti m = v t+1i - v ti . 5.The method of claim 4, wherein, The feature extraction method in step 4 is that there are many points on the skeleton edge and the motion trajectory of the joint node, and because two points can determine a straight line, the motion trajectory of the joint node and the skeleton edge is abstracted into four points to represent the information of the two types, and the 1 / 2 point and the 1 / 4 point on the line segment are taken to represent the line segment. Three types of information can be represented by points, and finally a feature extraction network is used for feature extraction. 6.The method of claim 5, wherein, The specific process of step 5 is as follows: the connected mode is selected, the tensors of the three types of output feature information are normalized, then the three types of output feature information are fused by using the connected mode to obtain a tensor; The fusion of the features is as follows: Z = [v out ||e out ||m out ] wherein v out is the output feature of the coordinate information of the joint point passing through the graph convolution network, e out is the output feature of the vector information of the bone connection passing through the graph convolution network, m out is the output feature of the motion information of the joint point passing through the graph convolution network. 7.The method according to claim 4, wherein, The specific process of step 6 is as follows: First, the true value graph in the database is used to train the neural network composed of steps 1-5, that is, steps 1 to 5 are repeated, after the training, the neural network has the ability to identify the action of the object in the video, and then the verification set is used for verification; The tensor obtained in step 5 is input into the last fully connected layer of the neural network, the fused features are classified, and the action recognition result of the to-be-identified video is obtained.

Images