Action recognition method and system based on graph contrast model
By using a graph comparison model, a time sequence graph of human joint bones is extracted and graph structure is compared, which solves the problem of low accuracy in action recognition in existing technologies and achieves efficient and accurate action recognition.
Patent Information
- Application Number
- CN202310360030.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-06
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2043-04-06
AI Technical Summary
In existing technologies, traditional human motion recognition methods cannot effectively recognize complex sequence features and spatiotemporal features. Deep learning methods have long computation time and a large number of parameters, making it difficult to meet real-time requirements, resulting in low motion recognition accuracy.
A graph-based comparison model is adopted to extract human joint and bone time-series graphs and convert them into graph structures by acquiring comparison videos and videos to be classified. Graph neural networks are then used for cross-graph feature extraction and classification, which reduces the number of training samples and improves recognition accuracy.
The network structure was simplified, the training parameters were reduced, the accuracy of action recognition was improved, and efficient action recognition was achieved with fewer training samples.
Smart Images
Figure CN116311531B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of action recognition technology, and particularly relates to an action recognition method and system based on graph comparison models. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Human motion recognition technology can detect the posture types of people in videos. When people make dangerous movements, the technology can identify the type of dangerous movement and issue early warnings, reducing the probability of accidents. Traditional human motion recognition methods, such as Support Vector Machines, Hidden Markov Models, and Histogram of Oriented Gradients (HGPs), rely on traditional hand-crafted features. These features are processed into feature vectors and then fed into a classifier for learning and training. However, these methods cannot achieve accurate motion recognition due to their inability to recognize complex sequence features and their inability to fully represent the spatiotemporal characteristics of human movements.
[0004] With the continuous development of artificial intelligence technology, deep learning methods have been gradually applied to image and video detection and classification tasks, becoming the mainstream method in the field of action recognition in recent years. In deep learning, the networks mainly used for action recognition are Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). Image-based deep learning action recognition algorithms can be broadly classified into three categories: action recognition algorithms based on two-stream networks, action recognition algorithms based on 3D convolution, and action recognition algorithms based on Long Short-Term Memory Neural Networks (LSTM). However, existing deep learning methods have drawbacks such as long computation time, difficulty in meeting real-time requirements, and the large number of parameters in the network, requiring a large amount of data for training and increasing computational costs. Summary of the Invention
[0005] To overcome the shortcomings of the existing technologies, this invention provides an action recognition method and system based on a graph comparison model. It uses a neural network to extract the time sequence map of human joints and bones in the video, transforms the human joint and bone time sequence map into a graph structure, and then compares the human actions in the video to be recognized with those in known videos to output the classification result. This simplifies the network structure, reduces the number of samples required for training, and improves the recognition accuracy, solving the problems of large number of parameters, difficulty in training, and low accuracy in current human action recognition methods.
[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:
[0007] The first aspect of this invention provides an action recognition method based on a graph comparison model.
[0008] The action recognition method based on graph contrast model includes the following steps:
[0009] Obtain a reference video and a video to be classified, where the reference video is a human movement video of a known category, and the video to be classified is a human movement video of an unknown category;
[0010] The comparison video and the video to be classified are preprocessed to obtain multiple frames of comparison images and multiple frames of images to be classified;
[0011] Multiple frames of reference images and multiple frames of images to be classified are input into the key point recognition network to extract human joint and bone points, and generate graph structures of reference images and images to be classified. The node features adopt the polar coordinate form of the center point.
[0012] Single-image features are extracted from the graph structures of the control image and the image to be classified to obtain single-image features;
[0013] The single-graph features are input into a graph neural network composed of the relationship between the joint and bone points corresponding to the graph structures of the reference image and the image to be classified, and cross-graph features are extracted to obtain cross-graph features.
[0014] Aggregate the distance features of all points across the graph, input the distance features into the multilayer perceptron, and output the predicted distance between the reference video and the video to be classified.
[0015] Based on the predicted distance, the system classifies whether dangerous actions exist in the video to be classified.
[0016] Furthermore, the HRnet key point recognition network is used to extract 25 human skeletal joints from multiple frames of comparison images and multiple frames of images to be classified, and the position information of human skeletal joints in multiple frames of comparison images and multiple frames of images to be classified is obtained. The position information of human bones in each frame is polarized as the feature of the skeletal key nodes.
[0017] Furthermore, for each frame of the reference image and the image to be classified, the human skeletal joints are generated from the actual connection relationship of the skeleton to form an edge structure, thus obtaining the graph structure of the reference image and the image to be classified. Edge structures are generated for multiple frames of reference images and multiple frames of images to be classified, thus generating the human joint skeleton time sequence graph of the reference video and the video to be classified.
[0018] Furthermore, a graph neural network with shared parameters is used in the Siamese network framework. The graph structures of the control image and the image to be classified are input simultaneously, and the node latent variables of the output graph are used as the features of the output graph.
[0019] Furthermore, based on the attributes of human skeletal nodes, a matching matrix is obtained between the graph structure of the reference image and the graph structure of the image to be classified. If the key point in the graph structure of the image to be classified is the same skeletal joint as the key point in the graph structure of the reference image, the element value in the matching matrix is 1; otherwise, the value is 0. The matched edges are connected to form a large graph from the graph structure of the reference image and the graph structure of the image to be classified.
[0020] Furthermore, on the large graph constructed by matching the nodes of the graph structures of the reference image, the image to be classified, and the graph structure of the reference image, all latent variables of the nodes are updated through graph information propagation operations. Cross-graph features are extracted from the graph structures of the reference image and the image to be classified to obtain cross-graph features.
[0021] Furthermore, the graph comparison network uses a soft-margin loss function as its loss function.
[0022] A second aspect of the present invention provides an action recognition system based on a graph comparison model.
[0023] Action recognition systems based on graph comparison models include:
[0024] The video acquisition module is configured to acquire a reference video and a video to be classified, wherein the reference video is a human motion video of a known category, and the video to be classified is a human motion video of an unknown category.
[0025] The preprocessing module is configured to preprocess the comparison video and the video to be classified to obtain multiple frames of comparison images and multiple frames of images to be classified.
[0026] The graph structure generation module is configured to: input multiple frames of reference images and multiple frames of images to be classified into the graph comparison network, extract human joint and bone points, and generate the graph structure of the reference images and images to be classified by polar coordinate transformation;
[0027] The single-image feature acquisition module is configured to extract single-image features from the graph structures of the reference image and the image to be classified, thereby obtaining single-image features;
[0028] The cross-graph feature acquisition module is configured to: input single-graph features into a cross-graph graph neural network composed of the relationship between the joint and bone points of the graph structure of the comparison image and the image to be classified, and perform cross-graph feature extraction to obtain cross-graph comparison features;
[0029] The predicted distance acquisition module is configured to: aggregate the distance features of all points across the map features, input the distance features into the multilayer perceptron, and output the predicted distance between the reference video and the video to be classified;
[0030] The classification module is configured to classify whether the actions in the video to be classified are dangerous actions based on the predicted distance.
[0031] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the action recognition method based on a graph comparison model as described in the first aspect of the present invention.
[0032] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the action recognition method based on a graph comparison model as described in the first aspect of the present invention.
[0033] The above one or more technical solutions have the following beneficial effects:
[0034] 1. When identifying the types of human actions in a video, this invention takes into account the linkage of the overall topological structure of the human body and extracts the temporal sequence map of the human skeleton in the video for motion feature extraction;
[0035] 2. To reduce the required sample size, this invention decomposes the action recognition problem into two parts: the first step is graph representation based on human skeleton data, that is, using deep learning methods to identify human skeletons in videos and generate human skeleton temporal graphs; the second step is to build a graph comparison network based on human skeleton temporal graphs, thereby completing the graph classification problem.
[0036] 3. Compared with previous methods, this invention transforms the action classification problem in videos into graph classification, which requires fewer training parameters and can achieve high classification accuracy with only a small number of training samples.
[0037] 4. The nodes of this invention adopt polar coordinates relative to other nodes, which can effectively capture the relative motion relationships of nodes globally and obtain effective action recognition features in shallow graph neural networks.
[0038] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0039] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0040] Figure 1 This is a flowchart of the method in the first embodiment.
[0041] Figure 2 This is a schematic diagram of the locations of 25 human joints.
[0042] Figure 3 This is a model framework diagram for an action recognition method based on a graph comparison model.
[0043] Figure 4 The timing diagram of the human skeleton used in the model.
[0044] Figure 5 A schematic diagram of a graph representation generated for a single video frame.
[0045] Figure 6 This is a system structure diagram of the second embodiment. Detailed Implementation
[0046] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0047] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.
[0048] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0049] The overall concept proposed in this invention is as follows:
[0050] This invention aims to identify common dangerous actions in daily life. It collects videos of known dangerous actions, and after screening, elimination, and supplementation, obtains a corresponding dataset. When identifying the categories of human actions in the videos, this invention considers the interconnectedness of the overall topological structure of the human body and extracts motion features from the temporal sequence diagram of the human skeleton in the video. To reduce the required sample size, the action recognition problem is decomposed into two parts: the first step is graph representation based on the human skeleton data, that is, using deep learning methods to identify the human skeleton in the video and generate a temporal sequence diagram of the human skeleton; the second step is to build a graph comparison network based on the temporal sequence diagram of the human skeleton to complete the graph classification problem. Finally, it can determine whether there are dangerous actions in the video and provide corresponding warnings. Compared with previous methods, this invention transforms the action classification problem in videos into graph classification, requiring fewer model training parameters and achieving high classification accuracy with only a small number of training samples.
[0051] Example 1
[0052] This embodiment discloses an action recognition method based on a graph comparison model.
[0053] like Figure 1 As shown, the action recognition method based on graph contrast model includes the following steps:
[0054] Obtain a reference video and a video to be classified, where the reference video is a human movement video of a known category, and the video to be classified is a human movement video of an unknown category;
[0055] The comparison video and the video to be classified are preprocessed to obtain multiple frames of comparison images and multiple frames of images to be classified;
[0056] Multiple frames of reference images and multiple frames of images to be classified are input into the key point recognition network to extract human joint and bone points, and generate graph structures of reference images and images to be classified. The node features adopt the polar coordinate form of the center point.
[0057] Single-image features are extracted from the graph structures of the control image and the image to be classified to obtain single-image features;
[0058] The single-image features are input into a cross-image graph neural network composed of the relationship between the joint and bone points corresponding to the graph structure of the comparison image and the image to be classified, and cross-image feature extraction is performed to obtain cross-image comparison features.
[0059] Aggregate the distance features of all points in the cross-image comparison features, input the distance features into the multilayer perceptron, and output the predicted distance between the comparison video and the video to be classified;
[0060] Based on the predicted distance, the video action to be classified is classified as whether it is a dangerous action.
[0061] The following sections describe the steps of the action recognition method based on graph comparison models.
[0062] Step 1:
[0063] The camera acquires human motion videos of known and unknown categories, performs preprocessing, and extracts human joints from each frame of the video. The human motion videos of known categories serve as control samples, while the videos of unknown categories are the actual motion videos that need to be judged.
[0064] Step 2:
[0065] This invention considers common dangerous actions in daily life: shaking to get up, falling, and collecting corresponding datasets. Information is simultaneously captured using three Kinect V2 cameras at the same height but different angles: -45°, 0°, and 45°. The position information of 3D node annotations on each frame is obtained through the Kinect depth sensor, based on... Figure 2Each character has the following skeletal joints: 1. Base of spine, 2. Middle of spine, 3. Neck, 4. Head, 5. Left shoulder, 6. Left elbow, 7. Left wrist, 8. Left hand, 9. Right shoulder, 10. Right elbow, 11. Right wrist, 12. Right hand, 13. Left hip, 14. Left knee, 15. Left ankle, 16. Right knee, 17. Right hip, 18. Right knee, 19. Right ankle, 20. Right foot, 21. Spine, 22. Tip of left hand, 23. Left thumb, 24. Tip of right hand, 25. Right thumb.
[0066] After data collection is complete, the information captured by cameras 2 and 3 is divided into a training set, and the information captured by camera 1 is divided into a validation set.
[0067] according to Figure 2 The 25 key points of the human body shown are categorized as P = (p1, p2, ..., p...). 25 The control video and the video to be classified were preprocessed, and video samples were extracted in segments of 300 frames. Each image frame was uniformly scaled and cropped to 270×480. The ratio of training set to validation set was 2:1, and each frame corresponded to one image.
[0068] For each frame of image, a square grid of size s×s is determined around the center position of its skeletal joint coordinates. The grid probability is calculated using a two-dimensional Gaussian function with the joint coordinates as the center and filled into the heat map. All key points are traversed, and this heat map is used as the supervision signal for recognition.
[0069] Step 3:
[0070] like Figure 3 As shown, the graph comparison model designed in this invention consists of four parts: HRnet network, graph convolutional neural network (GNN), cross-graph feature extraction, and multilayer perceptron (MLP). The graph comparison model is trained based on the training set obtained in step 2:
[0071] For each frame of the video, the image i is labeled y. label The network output is denoted as y. pred The mean squared error is used as the loss function during training.
[0072]
[0073] Considering that human skeletal joint features can be extracted from a relatively small area, and that the limited receptive field of deep convolutional neural networks can adapt well to this problem, this invention uses HRnet to identify human skeletal joints in videos, thereby obtaining joint position information:
[0074] N = HRnet(X),
[0075] Where X∈R 3×p×qp, q represent the width and height of each frame of the image, respectively, and the output is N = [p1, p2, ..., pq]. 25 ], N∈R n×2 , representing the coordinate matrix of the joint point image formed by the identified joints.
[0076] Then, the image coordinates of each pair of nodes are converted to polar coordinates, for example, node p i =(x i ,y i (relative to p) j =(x j ,y j The polar coordinates of are denoted as (ρ, θ), and the calculation process is as follows:
[0077]
[0078] After all the video data is recognized, the final human skeletal joint sequence Q∈R is obtained. n×5000 , where 5000 = 100 * 25 * 2 is the length of the one-dimensional vector formed by splicing the joint coordinates of 100 frames.
[0079] Step 4:
[0080] like Figure 4 As shown, the extracted joint bone points of each frame are used to generate a graph structure, thereby obtaining a temporal graph of human joint bones.
[0081] This invention is based on Figure 2 The provided human skeletal joint information is used to generate an edge structure for the identified human skeletal joints, thereby generating a temporal sequence graph of the human skeletal joints. Simultaneously, based on the arrangement order of the human skeletal joints, the adjacency matrix A of the video is uniquely determined, A∈R. t×x×y This paper employs a deep convolutional network to identify human skeletal joints, outputting the positions of the joints for a set of keypoints P to be identified. This generates a graph structure, where the adjacency matrix of the corresponding feature matrix N is denoted as A. Thus, a semantically meaningful graph representation of the video temporal graph, G(A, Q), is obtained, as shown below. Figure 5 As shown.
[0082] Step 5:
[0083] Latent variables of the time series diagrams of the control samples and the human joints and bones to be classified were obtained by using a graph neural network.
[0084] The graph structure of the video to be identified is represented as G. E (A E Q E E marks the video information to be classified. The graph structure of the collected classified dangerous action videos is represented as G. T (A T QT ), T marks the comparison video image information.
[0085] In the calculation, the control image and the image to be classified have the same structure, using a control structure, denoted as:
[0086] A = A E =A T
[0087] The structure of the control plot is equivalent to the category structure prior. By using the same plot structure, the plot to be classified obtains latent variables similar to the control plot under the prior plot structure.
[0088] The node features in the graph represent the coordinate positions of human skeletal joints, but these coordinates can change due to shifts in the human body's topology. The purpose of this invention is to extract features of human motion from individual samples by establishing a temporal graph model, thereby eliminating the influence of overall human movement on skeletal joint position features between frames. Specifically, graph convolutional layers are used to extract new skeletal joint features:
[0089] Q' = GNN1(A,Q) = f softmax (AQW1)
[0090] GNN1 represents a graph neural network for single-video feature extraction, f softmax W1 is the activation function, and W1 is the parameter to be learned. The training set video is input for training. A graph neural network within the Siamese network framework with shared parameters is used, where both the reference image and the image to be classified are input simultaneously. The latent variables of the nodes in the output single image are used as input for the cross-graph feature extraction step.
[0091] Step 6: Obtain the latent variables of the relative relationship between the two human joint skeleton time series diagrams;
[0092] For the image to be classified and the control image, after obtaining the features of a single image, the two images (G) are used to classify the image and control image. T and G E A graph neural network layer, composed of the relationships between skeletal joints, learns the latent variables representing the relative relationships between two graphs. Based on the attributes of the human skeletal nodes, the matching relationship between the two graphs can be obtained, and the matching relationship matrix is denoted as A. (T E) If the key points in the graph to be classified Key points in the comparison diagram If they are the same skeletal joint, then match the elements in the matrix. Otherwise, the value is 0. These matched edges, along with the reference graph and the graph to be classified, form a large graph. The nodes of this large graph are denoted as Q. match =[Q T Q E The adjacency matrix is denoted as:
[0093]
[0094] In the comparison chart and the unclassified chart (G) T and G E The large graph G(N) generated by constructing the matching relationships between their nodes is as follows: match A match On the graph, all hidden variables are updated through graph information propagation operations:
[0095]
[0096] W2 represents the parameters to be learned in this part.
[0097] Step 7:
[0098] A graph comparison network is used to calculate the distance between two human joint skeleton time series graphs, and finally to realize category judgment and give a reminder.
[0099] After analyzing the distance features of all points on the aggregated graph, the time-series graph of human skeletal joints generated from the video is passed through a multilayer perceptron (MLP) to output the predicted distance between the video of the known category and the video to be classified:
[0100]
[0101] W3 represents the parameters for this part. The final output is g. pred This represents the distance between the image to be classified and the control image. Input the distance between the images of the same category and the images of different categories (G). T and G E ) corresponding to different g pred .
[0102] During the training of the graph contrastive network, the soft-margin loss function, commonly used in contrastive learning, is adopted as the loss function:
[0103]
[0104] For the predicted value, g i For the label value, |g pred | indicates the number of output vectors.
[0105] Based on the output g i Determine whether two videos belong to the same category, and identify the category of human actions in the videos: if the output value is 1, then the person in the video is performing a dangerous action, the dangerous target is identified, and a warning is given; otherwise, there is no dangerous action in the video.
[0106] The loss function l of the action recognition method based on the graph contrast model is the soft-margin loss function commonly used in contrastive learning. The loss function is minimized, and the model parameters W1, W2, and W3 of the action recognition method based on the graph contrast model are updated using stochastic gradient descent to obtain the trained graph contrast model. The trained graph contrast model is then validated using the validation set obtained in step 2.
[0107] Finally, the test set is input into the trained graph contrast model to obtain the prediction results.
[0108] Specifically, the video captured by the camera is preprocessed, and then the preprocessed data is input into a trained graph contrast network to obtain the prediction result. If the output prediction value is 1, the action to be classified and the control sample belong to the same category, and there is a dangerous action in the video, so a warning should be issued; otherwise, no dangerous action is found in the video.
[0109] In practical applications, steps 1-4 constitute the basic feature extraction process for the video. In practice, a test set consisting of 100 video frames is input into the trained network to obtain prediction results. During the human skeleton node extraction process, a visual interface can be created to mark actions in the video to be recognized and to provide corresponding warnings for dangerous actions.
[0110] Example 2
[0111] This embodiment discloses an action recognition system based on a graph comparison model.
[0112] like Figure 6 As shown, the action recognition system based on the graph contrast model includes:
[0113] The video acquisition module is configured to acquire a reference video and a video to be classified, wherein the reference video is a human motion video of a known category, and the video to be classified is a human motion video of an unknown category.
[0114] The preprocessing module is configured to preprocess the comparison video and the video to be classified to obtain multiple frames of comparison images and multiple frames of images to be classified.
[0115] The graph structure generation module is configured to: input multiple frames of reference images and multiple frames of images to be classified into the graph comparison network, extract human joint and bone points, and generate the graph structure of the reference images and images to be classified by polar coordinate transformation;
[0116] The single-image feature acquisition module is configured to extract single-image features from the graph structures of the reference image and the image to be classified, thereby obtaining single-image features;
[0117] The cross-graph feature acquisition module is configured to: input single-graph features into a cross-graph graph neural network composed of the relationship between the joint and bone points of the graph structure of the comparison image and the image to be classified, and perform cross-graph feature extraction to obtain cross-graph comparison features;
[0118] The predicted distance acquisition module is configured to: aggregate the distance features of all points across the map features, input the distance features into the multilayer perceptron, and output the predicted distance between the reference video and the video to be classified;
[0119] The classification module is configured to classify whether the actions in the video to be classified are dangerous actions based on the predicted distance.
[0120] Example 3
[0121] The purpose of this embodiment is to provide a computer-readable storage medium.
[0122] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the action recognition method based on a graph comparison model as described in Embodiment 1 of this disclosure.
[0123] Example 4
[0124] The purpose of this embodiment is to provide an electronic device.
[0125] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the action recognition method based on a graph comparison model as described in Embodiment 1 of this disclosure.
[0126] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.
[0127] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.
[0128] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. An action recognition method based on a graph comparison model, characterized in that, Includes the following steps: Obtain a reference video and a video to be classified, where the reference video is a human movement video of a known category, and the video to be classified is a human movement video of an unknown category; The comparison video and the video to be classified are preprocessed to obtain multiple frames of comparison images and multiple frames of images to be classified; Multiple frames of reference images and multiple frames of images to be classified are input into the key point recognition network to extract human joint and bone points, and generate graph structures of reference images and images to be classified. The node features adopt the polar coordinate form of the center point. Single-image features are extracted from the graph structures of the control image and the image to be classified to obtain single-image features; The single-image features are input into a cross-image graph neural network composed of the relationship between the joint and bone points corresponding to the graph structure of the comparison image and the image to be classified, and cross-image feature extraction is performed to obtain cross-image comparison features. Aggregate the distance features of all points in the cross-image comparison features, input the distance features into the multilayer perceptron, and output the predicted distance between the comparison video and the video to be classified; Based on the predicted distance, the video action to be classified is classified as whether it is a dangerous action.
2. The action recognition method based on graph comparison model as described in claim 1, characterized in that, The HRnet key point recognition network is used to extract 25 human skeletal joints from multiple frames of comparison images and multiple frames of images to be classified, and obtain the position information of human skeletal joints in multiple frames of comparison images and multiple frames of images to be classified. The position information of human bones in each frame is polarized as the feature of skeletal key nodes.
3. The action recognition method based on graph comparison model as described in claim 2, characterized in that, For each frame of the reference image and the image to be classified, the human skeletal joints are generated from the actual connection relationship of the skeleton to form an edge structure, thus obtaining the graph structure of the reference image and the image to be classified. The edge structure is generated for multiple frames of reference images and multiple frames of images to be classified, and the human joint skeleton time sequence graph of the reference video and the video to be classified is generated.
4. The action recognition method based on graph comparison model as described in claim 1, characterized in that, The graph neural network under the Siamese network framework with shared parameters takes the graph structures of the control image and the image to be classified as inputs simultaneously, and outputs the node latent variables of the single graph as the features of the single graph.
5. The action recognition method based on graph comparison model as described in claim 1, characterized in that, Based on the attributes of human skeletal nodes, a matching matrix is obtained between the graph structure of the reference image and the graph structure of the image to be classified. If the key point in the graph structure of the image to be classified is the same skeletal joint as the key point in the graph structure of the reference image, the element value in the matching matrix is 1; otherwise, the value is 0. The matched edges are connected to form a large graph by combining the graph structures of the reference image and the graph structure of the image to be classified.
6. The action recognition method based on a graph comparison model as described in claim 5, characterized in that, On a large graph constructed by matching the nodes of the graph structures of the reference image, the image to be classified, and the graph structures of the reference image and the image to be classified, the latent variables of all nodes are updated through graph information propagation operations. Cross-graph feature extraction is performed on the graph structures of the reference image and the image to be classified to obtain cross-graph comparison features.
7. The action recognition method based on a graph comparison model as described in claim 1, characterized in that, The graph comparison model uses a soft-margin loss function as the loss function.
8. An action recognition system based on a graph comparison model, characterized in that: include: The video acquisition module is configured to acquire a reference video and a video to be classified, wherein the reference video is a human motion video of a known category, and the video to be classified is a human motion video of an unknown category. The preprocessing module is configured to preprocess the comparison video and the video to be classified to obtain multiple frames of comparison images and multiple frames of images to be classified. The graph structure generation module is configured to: input multiple frames of reference images and multiple frames of images to be classified into the graph comparison model, extract human joint and bone points, and generate the graph structure of the reference images and images to be classified by polar coordinate transformation. The single-image feature acquisition module is configured to extract single-image features from the graph structures of the reference image and the image to be classified, thereby obtaining single-image features; The cross-graph feature acquisition module is configured to: input single-graph features into a cross-graph graph neural network composed of the relationship between the joint and bone points of the graph structure of the comparison image and the image to be classified, and perform cross-graph feature extraction to obtain cross-graph comparison features; The predicted distance acquisition module is configured to: aggregate the distance features of all points across the map features, input the distance features into the multilayer perceptron, and output the predicted distance between the reference video and the video to be classified; The classification module is configured to classify whether the actions in the video to be classified are dangerous actions based on the predicted distance.
9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the action recognition method based on the graph comparison model as described in any one of claims 1-7.
10. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the action recognition method based on the graph comparison model as described in any one of claims 1-7.