Fuzzy action recognition method and device for constructing skeleton topological graph by fusing rigid nodes, equipment and medium

By constructing a spatiotemporal topology infographic and introducing attention mechanisms, combined with backbone network training of jump connection modules, the problem of inaccurate fuzzy action recognition in the existing technology is solved, and more efficient human body action recognition is achieved.

CN118968621AActive Publication Date: 2024-11-15NANTONG UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411035703.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-15
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

The existing behavior recognition method based on human skeleton graph structure has the problem of inaccurate recognition when dealing with fuzzy action recognition, especially because it lacks action interaction objects and context information based on bone node coordinate information, which makes it difficult to accurately moves with similar motion trajectories. Identify.

Method used

The method of constructing bone topology maps with fusion rigid nodes is adopted. By constructing a spatiotemporal topology infographic, an attention mechanism is introduced, and the spatiotemporal topology information is represented as joint modal frame sequence and bone modal frame sequence, and a jump connection module is added to the backbone network. , the fuzzy action recognition model is trained using preliminary prediction encoding method.

Benefits of technology

It improves the recognition accuracy of human body movements and the recognition ability of models, can more effectively capture and express the essential features of human body movement, reduces dependence on image quality and lighting conditions, and enhances the calibration ability of fuzzy movements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968621A_ABST
    Figure CN118968621A_ABST
Patent Text Reader

Abstract

The invention discloses a fuzzy action recognition method and device for constructing a skeleton topological graph by fusing rigid nodes, equipment and a medium, and relates to the field of behavior recognition. The fuzzy action recognition method comprises the following steps: constructing a spatio-temporal topological information graph according to skeleton coordinate information of each frame of image in a video; constructing a skeleton topological graph by taking the rigid node as a centroid point; introducing an attention mechanism, and representing the space-time topological information graph as a joint modal frame sequence and a skeleton modal frame sequence according to the skeleton topological graph; according to the joint modal frame sequence and the skeleton modal frame sequence, training an improved backbone network by adopting a preliminary predictive coding mode to obtain a fuzzy action recognition model; and performing action recognition by using the fuzzy action recognition model. According to the invention, effective recognition of human body actions can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of behavior recognition, and in particular to a fuzzy action recognition method, device, equipment and medium for fusing rigid nodes to construct a skeleton topology map. Background Art

[0002] Human action recognition is of great significance in the fields of computer vision and artificial intelligence. In recent years, with the support of deep learning, more effective data sets and more computing resources, action recognition has undergone tremendous technological innovations and has made certain progress in video surveillance, human-computer interaction, virtual reality, health detection and other fields. At present, the methods of action recognition using deep learning can be mainly divided into methods based on RGB images or methods based on human skeletons. However, methods based on RGB images are often limited by factors such as image quality, lighting conditions and complex backgrounds when dealing with action recognition tasks, making it difficult to achieve accurate action recognition and representation. In comparison, action recognition based on human skeletons can effectively avoid the influence of these noises. Skeleton graphs have a more concise representation form, can better capture and express the essential characteristics of human movement, and have higher computational efficiency.

[0003] At present, many existing behavior recognition methods based on human skeleton graph structure mostly rely on the real connection between human skeletons to model the graph structure, but this way of expression is too simple and cannot fully utilize and explore the internal connection between skeletons. On the other hand, since action recognition based on skeleton information only relies on the node coordinate information of the skeleton, it lacks the interactive objects and context information of the action. For many actions with similar motion trajectories, its true meaning cannot be accurately identified, resulting in some fuzzy actions that are easily confused. Summary of the invention

[0004] The purpose of this application is to provide a fuzzy action recognition method, device, equipment and medium that integrates rigid node structure skeleton topology graph, which can realize effective recognition of human body action.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a fuzzy action recognition method for fusing rigid nodes to construct a skeleton topology graph, the fuzzy action recognition method comprising:

[0007] According to the bone coordinate information of each frame image in the video, a spatiotemporal topological information graph is constructed; the spatiotemporal topological information graph takes the joint points of the human body as vertices and the bones as edges;

[0008] A skeleton topology is constructed with rigid nodes as mass centers; the rigid nodes include one or more of the head, chest and hips;

[0009] The attention mechanism is introduced to represent the spatiotemporal topological information graph as joint modality frame sequences and bone modality frame sequences according to the bone topology graph;

[0010] According to the joint modality frame sequence and the skeletal modality frame sequence, the improved backbone network is trained by using a preliminary predictive coding method to obtain a fuzzy action recognition model; the improved backbone network is obtained by adding a skip connection module to the backbone network;

[0011] Action recognition using fuzzy action recognition model.

[0012] In a second aspect, the present application provides a fuzzy motion recognition device that fuses rigid nodes to construct a skeleton topology map. The fuzzy motion recognition device applies the above-mentioned fuzzy motion recognition method. The fuzzy motion recognition device includes:

[0013] A spatiotemporal topological information graph construction module is used to construct a spatiotemporal topological information graph according to the skeletal coordinate information of each frame image in the video; the spatiotemporal topological information graph uses the joints of the human body as vertices and the bones as edges;

[0014] A skeleton topology construction module is used to construct a skeleton topology with rigid nodes as mass centers; the rigid nodes include one or more of the head, chest and hips;

[0015] The modality representation module is used to introduce the attention mechanism and represent the spatiotemporal topological information graph as a joint modality frame sequence and a bone modality frame sequence according to the bone topology graph;

[0016] A training module is used to train the improved backbone network by using a preliminary predictive coding method according to the joint modality frame sequence and the skeleton modality frame sequence to obtain a fuzzy action recognition model; the improved backbone network is obtained by adding a skip connection module to the backbone network;

[0017] The recognition module is used for performing action recognition using a fuzzy action recognition model.

[0018] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned fuzzy motion recognition method of fusing rigid nodes to construct a skeleton topology map.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned fuzzy motion recognition method for fusing rigid nodes to construct a skeleton topology map.

[0020] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0021] The present application provides a fuzzy motion recognition method, device, equipment and medium that integrates rigid nodes to construct a skeleton topology map. On the one hand, the present application can obtain better recognition effect by dividing the rigid nodes, selecting different rigid nodes as the centroid points, and constructing different skeleton topologies respectively. On the other hand, the present application performs model training in a preliminary predictive coding manner, continuously enhances the learning ability of features through comparative learning, enhances the calibration ability of fuzzy motion, and improves the recognition ability of the model. The present application can realize the effective recognition of human body motion.

[0022] This application classifies different joint points according to their distance from the center of mass. Taking into account the symmetry of the human skeleton, such classification can ensure that bone points with the same semantics are always in the same category, such as the left and right hand nodes, left and right knee nodes, etc. of the human body, and maximize the hidden connections between each key point. The data generated by the graph structure constructed with different centers of mass are aggregated in six items to obtain better recognition effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 This is an application environment diagram of a fuzzy action recognition method for fusing rigid nodes to construct a skeleton topology map in one embodiment of the present application;

[0025] Figure 2 A schematic flow chart of a fuzzy motion recognition method for fusing rigid nodes to construct a skeleton topology graph provided in one embodiment of the present application;

[0026] Figure 3 A schematic diagram of a fuzzy motion recognition method for fusing rigid nodes to construct a skeleton topology map provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of the attention mechanism provided in one embodiment of the present application

[0028] Figure 5 A schematic diagram of the structure of a jump connection module provided in one embodiment of the present application;

[0029] Figure 6 An overall framework diagram of a blur motion calibration module provided in an embodiment of the present application;

[0030] Figure 7 A comparison diagram of feature clustering analysis provided in one embodiment of the present application;

[0031] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0034] The fuzzy motion recognition method for fusing rigid nodes to construct a skeleton topology map provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the video for training or recognition to the server 104, and the server 104 receives the skeletal coordinate information of each frame image in the video and constructs a spatiotemporal topological information graph; the spatiotemporal topological information graph takes the joint points of the human body as vertices and the bones as edges; the skeletal topological graph is constructed with rigid nodes as the centroid points; the rigid nodes include one or more of the head, chest and buttocks; the attention mechanism is introduced, and the spatiotemporal topological information graph is represented as a joint modality frame sequence and a skeletal modality frame sequence according to the skeletal topological graph; according to the joint modality frame sequence and the skeletal modality frame sequence, the improved backbone network is trained by preliminary predictive coding to obtain a fuzzy action recognition model; the improved backbone network is obtained by adding a jump connection module to the backbone network; and the fuzzy action recognition model is used for action recognition. The server 104 can feedback the obtained recognition results to the terminal 102. In addition, in some embodiments, the fuzzy motion recognition method of fusing rigid nodes to construct a skeleton topology map can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly process the video, or the server 104 can obtain the video from the data storage system and process the video.

[0035] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0036] In an exemplary embodiment, Figure 2 As shown, a fuzzy action recognition method for fusing rigid nodes to construct a skeleton topology map is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 205. Among them:

[0037] Step 201, constructing a spatiotemporal topological information graph based on the skeletal coordinate information of each frame image in the video; the spatiotemporal topological information graph uses the joints of the human body as vertices and the bones as edges.

[0038] Step 202, constructing a skeleton topology graph with rigid nodes as mass centers; the rigid nodes include one or more of the head, chest and hips.

[0039] Step 203, introducing the attention mechanism, and representing the spatiotemporal topological information graph as a joint modality frame sequence and a bone modality frame sequence according to the bone topology graph.

[0040] Step 204, according to the joint modality frame sequence and the skeleton modality frame sequence, the improved backbone network is trained by preliminary predictive coding to obtain a fuzzy motion recognition model; the improved backbone network is obtained by adding a jump connection module to the backbone network.

[0041] Step 205: Perform action recognition using the fuzzy action recognition model.

[0042] By implementing the above steps 201 to 205, effective recognition of human body movements can be achieved.

[0043] In another exemplary embodiment of the present application, the above step 201 can be implemented by the following steps:

[0044] The embodiments of the present application are trained on three public data sets for skeletal behavior recognition, namely the NTU-RGB+D data set, the NTU-RGB+D120 data set and the Northwestern-UCLA data set. The NTU-RGB+D data set contains a total of 56,880 video clips. These clips cover 60 different action categories from 40 subjects. The NTU-RGB+D120 data set is an extended version of the NTU-RGB+D data set, and is a data set containing more video clips. It consists of approximately 118,562 video clips, covering 120 different action categories from 106 subjects. The Northwestern-UCLA data set contains 1494 video clips in more than 10 categories.

[0045] For the bone coordinate information in the video that has been extracted, the human skeleton information at time node T can be regarded as a graph topology structure. On this basis, the time dimension information is added to construct a spatiotemporal topology information graph, with the joints of the human body as vertices of the graph structure and each bone as an edge.

[0046] For the human motion skeleton information in the acquired video sequence, the skeleton sequence is a collection of two-dimensional skeleton information or three-dimensional skeleton information in continuous time. The human skeleton information at time node T can be regarded as a graph topological structure, on this basis, the time dimension information is added to construct a spatiotemporal topological information graph. The use of a graph structure to represent the human skeleton in two-dimensional space can be defined as G = (V, E), where the joints of the human body are the vertices of the graph structure, V represents the set of joints, each bone is an edge, and E represents the edge set. Expanding the skeleton information to three-dimensional space, the skeleton data can be defined as X∈R C×T×V , C represents the number of feature channels, T represents the number of frames in the feature sequence, and V represents the number of joints in the human skeleton. in The process of GCN processing of ∈X is shown in the formula:

[0047]

[0048] Where F out Represents the output features, Λ represents the normalized diagonal matrix, A represents the adjacency matrix of the connection relationship of the human skeleton in a single frame, the unit matrix I represents the self-connection of the joint, and W represents the weight matrix, which is formed by the superposition of the weight vectors of multiple output channels.

[0049] In another exemplary embodiment of the present application, Figure 3 As shown, the above steps 202 and 203 can be implemented by the following steps:

[0050] Step S2.1: Select the joint points of the head, chest, and hips as the centroid points of the graph structure, construct a skeleton topology graph, and use a dual-stream framework to perform joint training using joint modality and skeleton modality.

[0051] In order to better explore the hidden features in human skeleton information. Considering that the joints (head, neck, chest, abdomen, buttocks) along the spine from the head to the human body during movement can remain relatively stable and not easy to change during movement, they are defined as rigid nodes. In order to better reflect the connection between the various joints during the movement process, the top node, middle node, and bottom node (i.e., head, chest, and hip nodes) of the rigid nodes in the natural state are selected as the centroid points for constructing the bone topology map. For each centroid point, considering that the model itself uses rigid nodes as the topological center points, three different topological maps are constructed. For each constructed topological map, a dual-stream framework is used, that is, joint mode and bone mode are used for joint training. The joint mode is represented by the original bone data X, and the bone mode is represented as follows:

[0052] X bone ={x[:,:,i]-x[:,:,i ne ]|i=1,2,...,N}.

[0053] Among them, i ne Represents the neighboring joint nodes of the ith joint based on a predefined adjacency matrix.

[0054] Step S2.2: For the remaining points except the selected centroid point, a unit distance is defined for the real connection between two adjacent joint points in the natural state of the human skeleton. The remaining bone points are classified according to their distance from the centroid point, and a classification distance attention topology map is built for spatial modeling, and temporal modeling is performed through a timing modeling network.

[0055] After selecting the rigid node as the center point of the topological structure, the remaining skeletal information points need to be effectively classified. The establishment of conventional graph structure topological relationships often only considers the real connection lines of the human skeleton, that is, the connection edge set represented by the real human skeleton. Such connections are effective and meaningful, but they are far from enough for mining deep skeleton information. When the human body is moving, there is still a connection between two skeletal points that do not have a real connection. For example, when making a "clapping" action, there is not only a real skeletal connection, but also an intrinsic connection between the nodes at the hands and the nodes at the shoulders. Based on this consideration, a classification distance attention topological network is proposed.

[0056] Taking the rigid node as the center of mass, the real connection of two adjacent joints in the natural state of the human skeleton is defined as a unit distance. On this basis, the remaining bone points are classified according to their distance from the center of mass. For example, taking the rigid node of the chest as the center of mass of the topological graph, then find the joint points closest to the center of mass, that is, the joint points with a distance of 1 unit, that is, the two points of the left and right shoulders, the neck point and the abdomen point, and classify these four joint points into group 1; then find the joint points that are one joint point away from the center of mass, that is, the joint points with a distance of 2 units, and classify these joint points into group 2, and so on. Under such a construction, nodes with the same semantics, such as the left and right elbow joints, the joints on both hands, and the feet, can be in the same group. Considering that there are only 5 rigid nodes defined and the farthest distance does not exceed 7, the manual tabulation method is used.

[0057] After obtaining K (K≤7) groups classified by distance, we can start building topological relationships. Different from the traditional method of only connecting real bones, more emphasis is placed on discovering the connections between more bone points; for group K (K≤7), the nodes are connected according to different rules to form three types of edge sets. For any group k, which contains n elements, three types of connections are performed on each element. The first is the self-connection of the joint. For each node group, the nodes in the group are connected to themselves, that is, the connection joints between the nodes and themselves; the forward hierarchical connection is to connect the nodes of the first group with the nodes of the second group between two adjacent node groups, and all subsets are connected to group K-1 to form a forward hierarchical connection; the reverse hierarchical connection is to connect the nodes of the second group with the nodes of the first group between two adjacent node groups to form a reverse hierarchical connection; the entire connection can be expressed as For any level ε k , which can be expressed as:

[0058] ε k =[g k ∪g k+1 ,g k → k+1 ,g k+1 → k ].

[0059] Get the skeleton topology representation matrix Afterwards, considering that the attention that should be given to each level in actual motion recognition is not the same, an attention mechanism is introduced in the connection of the levels to fuse the features of different levels. This is equivalent to the aggregation of K+1 levels of different distance classifications. However, the number of subsets contained in different distance groups is different, and the number of edges connected to each node is also different. If the pooling operation is performed directly, scaling deviation will occur. Therefore, the following is performed: Figure 4The processing shown in Dimensionality reduction data processing is performed to form four data branches. The outputs of all four branches will be connected to the channel dimension to form a larger tensor, where the size of the channel dimension is the sum of the sizes of the channel dimensions of the four branch outputs.

[0060] At this time, for the input features Perform the maximum pooling operation on the time dimension T, extract the key information on the time dimension for dimensionality reduction, and then perform downsampling operation on the information. The sampling layer is composed of a 1×1 convolution layer, a normalization layer and an activation function to obtain the corresponding time series information. Then we can perform average pooling. The pooling function is as follows:

[0061]

[0062] In the tensor X s The graph features are extracted and sent to the edge convolution (EGC) module to learn the similarity of these nodes based on the Euclidean distance. The calculation method of the Euclidean distance here is:

[0063]

[0064] Get the adjacent one-dimensional index of each point, and extract the nearest neighbor feature from the feature tensor according to the index by subtracting the difference between the original feature and the nearest neighbor feature, and concatenating the original feature, expressed as:

[0065] X F =cat[(X F -X s ,X s ),dim=3].

[0066] Among them, cat represents the concatenation of tensors along the specified dimension, and dim represents the dimension. F Perform convolution operation to obtain the attention coefficient λ a , calculate the independent attention coefficients of edge graph convolutions at different levels, select the Sigmoid function for normalization, multiply the original input features with the normalized attention coefficients, and sum the time dimension to obtain the final output features.

[0067] In another exemplary embodiment of the present application, the above step 204 can be implemented by the following steps:

[0068] Add a skip connection module to the backbone network to reduce the loss of key information during network transmission.

[0069] like Figure 5As shown, the skip connection module is an effective residual information transmission model inspired by the skip connection structure in CNN. It uses an adaptive pyramid structure to transfer shallow features back to the last layer. The key feature of the network structure is the introduction of skip connections or short-circuit connections between certain layers of the network, so that the network can directly cross several layers of nonlinear transformations, thereby more effectively transferring gradients, alleviating the gradient vanishing problem, and helping to accelerate training and improve model performance. This method helps to reduce the loss of key information during network iteration while reducing the time dimension. In this model, the embodiment of the present application selects the features of the 4th, 7th, and 10th layers for integration, and after obtaining the features of the 7th layer, performs an additional network layer iteration operation on it, and obtains more accurate prediction results by integrating the features of these three layers.

[0070] like Figure 6 As shown, the embodiment of the present application extracts the processing features obtained from the middle layer of the network to perform preliminary predictive coding of the features, and divides these samples into true positive, false positive, false negative, and true negative samples according to the evaluation criteria of the binary classification model for coding processing.

[0071] It is found that the recognition misjudgment rate between some actions is high, so further calibration is performed on these actions. For the predicted action k, the samples involved include all action samples whose real action is k and all action samples identified as action k. The evaluation criteria of the binary classification model are introduced to classify these judgment samples into three categories, namely true positive samples (TruePositive, TP), which means that the predicted result is consistent with the true value. The set containing all true positive samples is represented as The total number of subsets in this collection can be expressed as Prediction errors can be divided into two cases. One is that action k is mistakenly judged as other actions, which is called a false positive sample (False Positive, FP); the other is that an action other than action k is mistakenly judged as action k, which is called a false negative sample (False Negative, FN). Similarly, the set of all samples containing these two errors is defined as The sizes are

[0072] Considering that the features of true positive samples have better intra-class consistency than those of wrong samples, the exponential moving average (EMA) is used for self-feature update. By assigning different weights to each data point in the time series, the changing trend of the data is reflected. The values ​​of true positive samples are updated through EMA. The updating process is as follows:

[0073]

[0074] Among them, E krepresents the prototype of action k, F i is the feature extracted from sample i, and the prototype becomes a stable estimate of the cluster center of action k. The three different sample types are clustered using the k-means clustering algorithm. Each sample should be close to the prototype E of action k. k , and far away from other actions. This close and far similarity relationship is defined as the cosine distance D(·,·) from the error cluster center to the sample prototype.

[0075] Considering that the core of calibration is to make the wrong clustering (FP, FN) closer to the correct clustering, it is necessary to set the penalty term of the wrong clustering so that the sample is closer to the sample prototype in the feature space. Such a penalty term is defined as:

[0076]

[0077] in, is the false negative penalty term for the i-th training sample, F i is the i-th training sample, is the number of false negative samples whose true action is k, is the set of false negative samples whose true action is k, F i' is the i'th false negative sample with the true action k, and D(,) is the cosine distance calculation function;

[0078] is the false positive penalty term for the i-th training sample, is the number of false positive samples whose true action is k, is the set of false positive samples whose true action is k, F i” is the i-th false negative sample whose true action is k.

[0079] When ξ converges to 0, the model can make adjustments to correct false positive samples to action k while preventing the generation of false negative samples.

[0080] For the obtained true positive, false positive, and false negative samples, k-means clustering operation is performed to define their distances. The correct sample clusters are clustered by adding penalty items, and the network is trained using cross entropy loss.

[0081] In order to deeply explore the hidden information in blurred actions and improve the recognition of action features, a strategy of spatiotemporal decoupling is adopted. Specifically, the action features are fed into two parallel branch networks at the same time. Each branch network includes a pooling layer on the corresponding spatial or temporal dimension to retain the average value of the relevant dimension, followed by a 1×1 convolution layer to map the features to a representation space of fixed dimension. After that, a unified representation is obtained by flattening the output features of the two branch networks. In order to achieve spatiotemporal decoupling and refine the feature representation, a contrast loss (CL loss) is introduced at the top of each branch network. The CL loss includes the penalty calculation for FP and FN, where the penalty for the FN class, that is, the false negative contrast loss, can be defined as:

[0082]

[0083] in, is the false negative contrast loss of the i-th training sample, F i is the i-th training sample, E k is the mean value of the skeletal modal shape of the training sample with real action k, E ik is the prediction result of the i-th training sample, that is, the probability score of the action k corresponding to the i-th training sample output by the improved backbone network, is the false negative penalty term for the i-th training sample, E l is the skeletal modal mean of the training samples with real action l, and D(,) is the cosine distance calculation function.

[0084] Similarly, the penalty term for the FP class, that is, the false positive contrast loss, is defined as:

[0085]

[0086] in, is the false positive contrast loss of the ith training sample, is the false positive penalty term for the i-th training sample.

[0087] For the training sample F i The final contrast loss can be expressed as:

[0088]

[0089] Among them, CL() is the sample contrast loss function.

[0090] Finally, the losses of the two branch networks are added together to further refine the spatiotemporal decoupling features, where the training sample features can be expressed as:

[0091]

[0092] in Represents the spatial feature, F t i Represents time characteristics.

[0093] The network is trained using cross entropy loss. The goal of the training is to minimize this difference so that the model's prediction results are as close to the actual situation as possible, as follows:

[0094]

[0095] Among them, y ij is the true score of the i-th training sample for action j, represented by one-hot encoding. If and only if j is the true action of the i-th training sample, y ij =1. ij is the probability that the action corresponding to the predicted i-th training sample is j. Combining the cross entropy loss with the above contrast loss forms a complete learning objective function:

[0096]

[0097] Where L is the loss function, is a hyperparameter, CE is the cross entropy loss, L CL is the comparison loss.

[0098] In order to better reflect the distinction and clustering effect of the method proposed in this application on action features, the t-SNE algorithm is used to draw the clustering effect diagram, and the NMI index is added to quantify the clustering effect of the features. When the NMI value is closer to 1, the clustering effect is better. The algorithm proposed in this application (such as Figure 7 OUR in) and baseline methods (such as Figure 7 The clustering results of CTR-GCR in Figure 7 As shown. Regarding the selection of rigid nodes, different combinations of rigid nodes are selected for testing, and the results are shown in Table 1. From Table 1, it can be seen that when the center of mass points of the topological structure of the head, chest, and hips are selected, the model can obtain the best accuracy after joint training.

[0099] Table 1 Recognition results of selecting different rigid nodes as centroids

[0100] head neck chest Abdomen buttock Ntu_sub(%) √ √ √ 92.90 √ √ √ 93.01 √ √ √ 92.97 √ √ √ 92.98 √ √ √ 92.93 √ √ √ 93.10 √ √ √ 93.14 √ √ √ 93.03 √ √ √ 92.90 √ √ √ 92.87

[0101] Table 2 shows the comparison results of the model obtained by the present technical solution and other models on the NTU RGB+D dataset, NTU RGB+D120 dataset and NW-UCLA dataset. The higher the accuracy value, the better the accuracy of the model. The darkened font represents the highest value, and the underlined font represents the second highest value. It can be seen that the method of the present application has reached the highest value in the two perspectives of the NTU RGB+D 120 dataset, the X-Sub perspective of the NW-UCLA dataset and the NTU RGB+D dataset. The X-View perspective on the NTURGB+D dataset achieved the second highest AUC value of 97.2%, which is only 0.3% away from the highest value.

[0102] Table 2 Comparison results of the technical solution of this application with other solutions

[0103]

[0104] The present application also provides an application scenario, which applies the above-mentioned fuzzy action recognition method for fusing rigid nodes to construct a skeleton topology map. Specifically: The fuzzy action recognition method for fusing rigid nodes to construct a skeleton topology map provided in this embodiment can be applied in behavior recognition. The fuzzy action recognition method for fusing rigid nodes to construct a skeleton topology map provided in this embodiment belongs to the detection link. Specifically, a recognition result is obtained based on the above-mentioned fuzzy action recognition method, and behavior prediction is performed based on the recognition result.

[0105] Based on the same inventive concept, the embodiment of the present application also provides a fuzzy motion recognition device for fusing rigid nodes to construct a skeletal topology map for implementing the fuzzy motion recognition method for fusing rigid nodes to construct a skeletal topology map involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more fuzzy motion recognition devices for fusing rigid nodes to construct a skeletal topology map provided below can be referred to the limitations of the fuzzy motion recognition method for fusing rigid nodes to construct a skeletal topology map above, and will not be repeated here.

[0106] In an exemplary embodiment, a fuzzy motion recognition device for fusing rigid nodes to construct a skeleton topology graph is provided, comprising:

[0107] The spatiotemporal topological information graph construction module is used to construct a spatiotemporal topological information graph according to the skeletal coordinate information of each frame image in the video; the spatiotemporal topological information graph has the joints of the human body as vertices and the bones as edges.

[0108] The skeleton topology construction module is used to construct a skeleton topology with rigid nodes as mass centers; the rigid nodes include one or more of the head, chest and hips.

[0109] The modality representation module is used to introduce the attention mechanism and represent the spatiotemporal topological information graph as a joint modality frame sequence and a bone modality frame sequence according to the bone topology graph.

[0110] The training module is used to train the improved backbone network by preliminary predictive coding according to the joint modality frame sequence and the bone modality frame sequence to obtain a fuzzy motion recognition model; the improved backbone network is obtained by adding a jump connection module to the backbone network.

[0111] The recognition module is used for performing action recognition using a fuzzy action recognition model.

[0112] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data determined by complementarity. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a fuzzy action recognition method for fusing rigid nodes to construct a skeleton topology map is implemented.

[0113] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] In an exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in the above-mentioned fuzzy motion recognition method embodiment of fusing rigid nodes to construct a skeleton topology map are implemented.

[0115] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the steps in the above-mentioned fuzzy motion recognition method embodiment of fusing rigid nodes to construct a skeleton topology graph.

[0116] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0117] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, etc., but is not limited thereto.

[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A fuzzy action recognition method for integrating rigid nodes to construct a skeleton topology graph, characterized in that: The fuzzy motion recognition method comprises: According to the bone coordinate information of each frame image in the video, a spatiotemporal topological information graph is constructed; the spatiotemporal topological information graph takes the joint points of the human body as vertices and the bones as edges; A skeleton topology is constructed with rigid nodes as mass centers; the rigid nodes include one or more of the head, chest and hips; The attention mechanism is introduced to represent the spatiotemporal topological information graph as joint modality frame sequences and bone modality frame sequences according to the bone topology graph; According to the joint modality frame sequence and the skeletal modality frame sequence, the improved backbone network is trained by using a preliminary predictive coding method to obtain a fuzzy action recognition model; the improved backbone network is obtained by adding a skip connection module to the backbone network; Action recognition using fuzzy action recognition model.

2. The fuzzy motion recognition method for fusing rigid nodes to construct a skeleton topology map according to claim 1 is characterized in that: The skeleton topology is constructed with the rigid node as the center of mass, including: The real connection between two adjacent joints in the natural state of human skeleton is defined as a unit distance; Classifying each joint point of the human skeleton according to the number of unit distances between each joint point and the center of mass point in the human skeleton to obtain joint groups of multiple levels; The connection representation of the human skeleton of the joint groups at each level is constructed by using self-connection, forward hierarchical connection and reverse hierarchical connection. Based on the connection representation of the human skeleton of the joint groups at each level, a connection representation of the entire human skeleton is constructed; Generate a bone topology map based on the connection representation of the entire human skeleton.

3. The fuzzy motion recognition method for fusing rigid nodes to construct a skeleton topology map according to claim 2 is characterized in that: The connection representation of the human skeleton of the joint groups at each level is constructed by self-connection, forward hierarchical connection and reverse hierarchical connection, specifically including: Connect each joint point in the k-th level joint group, and perform forward hierarchical connection and reverse hierarchical connection between each joint in the k-th level joint group and each joint in the k+1-th joint group to obtain a connection representation of the human skeleton of the k-th joint group; k=1, 2, ..., K-1, K is the number of joint groups.

4. The fuzzy motion recognition method for fusing rigid nodes to construct a skeleton topology map according to claim 1, characterized in that: The attention mechanism is introduced to represent the spatiotemporal topological information graph as joint modality frame sequences and bone modality frame sequences according to the bone topology graph, including: The attention mechanism is used to determine the attention coefficients of joint groups at each level of the skeleton topology graph; Multiply the attention coefficient spatiotemporal topological information graphs of the joint groups at each level to obtain the spatiotemporal topological information graph after attention processing; According to the skeleton topology map and the spatiotemporal topology information map after attention processing, the joint mode and the skeleton mode of the topology information map of each frame are constructed.

5. The fuzzy motion recognition method for fusing rigid nodes to construct a skeleton topology map according to claim 1, characterized in that: According to the joint modality frame sequence and the skeleton modality frame sequence, the improved backbone network is trained by using a preliminary predictive coding method to obtain a fuzzy action recognition model, which specifically includes: The joint modes in the joint mode frame sequence or the bone modes in the bone mode frame sequence are used as training samples to construct a training sample set; Input each training sample in the training sample set into the improved backbone network to obtain the prediction result of the improved backbone network; Extract intermediate features from the improved backbone network and input them into the binary classification model to obtain preliminary classification results; Calculate false negative penalty and false positive penalty based on the preliminary classification results; Calculate the loss function value based on the prediction results of the improved backbone network, the false negative penalty term, and the false positive penalty term; The parameters of the improved backbone network are adjusted based on the loss function value.

6. The fuzzy motion recognition method for fusing rigid nodes to construct a skeleton topology map according to claim 5 is characterized in that: The formula for calculating the false negative penalty and false positive penalty is: in, is the false negative penalty term for the i-th training sample, F i is the i-th training sample, is the number of false negative samples whose true action is k, is the set of false negative samples whose true action is k, F i' is the i'th false negative sample with the true action k, and D(,) is the cosine distance calculation function; is the false positive penalty term for the i-th training sample, is the number of false positive samples whose true action is k, is the set of false positive samples whose true action is k, F i” is the i-th false negative sample whose true action is k.

7. The fuzzy motion recognition method of fusing rigid nodes to construct a skeleton topology map according to claim 5, characterized in that: The calculation formula of the loss function value is: Where L is the loss function, is a hyperparameter, CE is the cross entropy loss, L CL is the contrast loss, CL() is the sample contrast loss function, F i is the i-th training sample, is the false negative contrast loss of the ith training sample, is the false positive contrast loss of the i-th training sample, E k is the mean value of the skeletal modal shape of the training sample with real action k, E ik is the prediction result of the i-th training sample, that is, the probability score of the action k corresponding to the i-th training sample output by the improved backbone network, is the false negative penalty term for the i-th training sample, E l is the mean value of the skeletal modal shape of the training sample with the real action l, D(,) is the cosine distance calculation function, is the false positive penalty term for the i-th training sample.

8. A fuzzy motion recognition device that integrates rigid nodes to construct a skeleton topology map, characterized in that: The fuzzy motion recognition device applies the fuzzy motion recognition method according to any one of claims 1 to 7, and the fuzzy motion recognition device comprises: A spatiotemporal topological information graph construction module is used to construct a spatiotemporal topological information graph according to the skeletal coordinate information of each frame image in the video; the spatiotemporal topological information graph uses the joints of the human body as vertices and the bones as edges; A skeleton topology construction module is used to construct a skeleton topology with rigid nodes as mass centers; the rigid nodes include one or more of the head, chest and hips; The modality representation module is used to introduce the attention mechanism and represent the spatiotemporal topological information graph as a joint modality frame sequence and a bone modality frame sequence according to the bone topology graph; A training module is used to train the improved backbone network by using a preliminary predictive coding method according to the joint modality frame sequence and the skeleton modality frame sequence to obtain a fuzzy action recognition model; the improved backbone network is obtained by adding a skip connection module to the backbone network; The recognition module is used for performing action recognition using a fuzzy action recognition model.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the fuzzy action recognition method for constructing a skeleton topology graph by fusing rigid nodes as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fuzzy action recognition method for constructing a skeleton topology graph by fusing rigid nodes as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Human body behavior recognition method based on space-time diagram convolutional network

    CN116959094A

  • Behavior recognition method based on skeleton high-order topological graph matching network

    CN118279986A

  • Human body behavior rapid identification method and system based on multi-dimensional skeleton space-time relation graph network

    CN118379787A

  • Human pose prediction method and apparatus, device, and storage medium

    US20200311402A1

  • Methods, apparatus and systems for graph-conditioned autoencoder (GCAE) using topology-friendly representations

    US20230222323A1