Human body behavior recognition method and device

Abstract

The invention relates to a human body behavior recognition method and device. The human body behavior recognition method comprises the steps that joint difference topological representation and joint dependence topological representation are determined based on a human body skeleton sequence, graph topological characteristics are determined based on the joint difference topological representation and the joint dependence topological representation, the human body skeleton sequence is represented as a graph structure, and the graph topological characteristics are represented as a joint structure; the graph topological feature is the sum of general structure topological representation, joint difference topological representation and joint dependence topological representation; based on the human body skeleton sequence and the graph topology features, determining action features of the human body skeleton sequence; reconstructing the topological features of the graph by using a fine-grained prototype to obtain refined action topological representation; and based on the action characteristics of the human body skeleton sequence and the refined action topological representation, determining the action category of the human body skeleton sequence by using a prototype comparison loss function and a cross entropy loss function. By adopting the method and the device, the slight difference change of the action can be better captured, and the behavior characteristics with discriminability can be effectively extracted.

Description

Technical Field

[0001] The present disclosure relates to the field of video analysis and behavior understanding, and more specifically, to a method and device for human behavior recognition. Background Art

[0002] Human behavior recognition is to model the spatiotemporal information of a moving human body, obtain the characteristic representation of human behavior by processing video data, and thus identify the target person's ongoing movement (for example, walking, drinking water, etc.) or the interactive behavior between multiple target persons (for example, handshake, hug, etc.). The spatiotemporal sequence information of movement is the basic component of human behavior. How to establish the semantic mapping between the underlying visual features and the high-level behavior categories is the core problem of the human behavior recognition task. Human behavior recognition technology is widely used in many scenarios in real life, such as intelligent video surveillance, human-computer interaction, virtual reality, motion simulation, etc. Accurate behavior recognition is very critical for these scenarios. Since skeleton sequence data can focus on the motion information most relevant to the action and has good robustness to the complex background of the video, it has received full attention. Human behavior recognition technology based on skeleton sequence data has been widely used in the fields of video analysis and behavior understanding.

[0003] However, in actual application, current human behavior recognition technology has difficulty in accurately distinguishing similar action categories. Human behavior recognition mainly involves two processes: first, identifying the key body parts involved in the movement, and then distinguishing the complex movement patterns of the key parts. Existing methods can capture the key parts involved in the movement very well, but they still face challenges in capturing and distinguishing movement details, and these key subtle information is very important for distinguishing similar actions. At the same time, intra-class differences in action categories (for example, changes in movement amplitude) will inevitably bring noise and interference to the process of obtaining movement patterns. This intra-class difference further increases the difficulty of existing technologies in extracting good movement features, making it more difficult to accurately distinguish similar actions. The lack of the above-mentioned movement details and complex intra-class differences make current technologies unable to accurately identify similar behaviors, and there is a high error rate. Summary of the invention

[0004] In order to solve the above problems, the present disclosure proposes a human behavior recognition method and device, a computing system and a computer-readable storage medium.

[0005] According to one aspect of the present disclosure, a method for human behavior recognition is provided, the method comprising: determining a joint difference topological representation and a joint dependency topological representation based on a human skeleton sequence, and determining a graph topological feature based on the joint difference topological representation and the joint dependency topological representation, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and connection relationships between joints represented by an adjacency matrix, and the graph topological feature is the sum of a general structure topological representation, a joint difference topological representation and a joint dependency topological representation; determining action features of the human skeleton sequence based on the human skeleton sequence and the graph topological features; reconstructing the graph topological features using a fine-grained prototype to obtain a refined action topological representation; determining the action category of the human skeleton sequence based on the action features of the human skeleton sequence and the refined action topological representation using a prototype contrast loss function and a cross entropy loss function.

[0006] Optionally, based on the human skeleton sequence, the step of determining the joint difference topological representation and the joint dependent topological representation includes: utilizing a multi-head attention mechanism to determine the joint dependent topological representation based on the inner product of projection features, and determining the joint difference topological representation based on feature differences between different joints.

[0007] Optionally, based on the human skeleton sequence and the graph topological features, the step of determining the motion features of the human skeleton sequence includes: determining the motion features of the first layer based on the human skeleton sequence, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection; determining the motion features of the current layer based on the motion features of the previous layer, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection.

[0008] Optionally, the human action recognition method also includes: after determining the action features of the last layer, using a global average pooling operation to learn the global features of the human skeleton sequence; determining the predicted action label through a classification network composed of a softmax function and a fully connected layer; and determining the cross entropy loss function based on the predicted action label and the true action label.

[0009] Optionally, the human behavior recognition method also includes: using a mapping network to embed the refined action topology representation into the vector space of common features to obtain contrast features; determining the prototype contrast loss function based on the contrast features, temperature hyperparameters of contrast learning, positive sample prototype aggregation and negative sample prototype aggregation related to the contrast features.

[0010] Optionally, the step of reconstructing the graph topology features using fine-grained prototypes to obtain a refined action topology representation includes: determining the refined action topology representation based on a prototype index matrix, a detailed prototype library, sampling-adjusted hyperparameters and the graph topology features.

[0011] Optionally, the step of determining the refined action topology representation includes: performing feature index mapping on the graph topology features using a prototype index matrix and sampling-adjusted hyperparameters; based on the mapping result, sampling and combining the detail prototype library using a softmax function to determine the refined action topology representation.

[0012] According to another aspect of the present disclosure, a human behavior recognition device is provided, the human behavior recognition device comprising: a motion detail enhancement module, configured to determine a joint difference topological representation and a joint dependency topological representation based on a human skeleton sequence, and to determine a graph topological feature based on the joint difference topological representation and the joint dependency topological representation, wherein the human skeleton sequence is represented as a graph structure, the graph structure comprising a set of joints and connection relationships between joints represented by an adjacency matrix, and the graph topological feature is the sum of a general structure topological representation, a joint difference topological representation and a joint dependency topological representation; an action feature extraction module, configured to determine an action feature of a human skeleton sequence based on the human skeleton sequence and the graph topological feature; a prototype reconstruction module, configured to reconstruct the graph topological feature using a fine-grained prototype to obtain a refined action topological representation; an action feature classification module, configured to determine an action category of the human skeleton sequence based on the action feature of the human skeleton sequence and the refined action topological representation using a prototype contrast loss function and a cross entropy loss function.

[0013] Optionally, the motion detail enhancement module is configured to: utilize a multi-head attention mechanism to determine the joint-dependent topological representation based on the inner product of projection features, and determine the joint-differential topological representation based on feature differences between different joints.

[0014] Optionally, the motion feature extraction module is configured to: determine the motion features of the first layer based on the human skeleton sequence, the normalized adjacency matrix of the adjacency matrix and the learnable weight matrix for feature projection; determine the motion features of the current layer based on the motion features of the previous layer, the normalized adjacency matrix of the adjacency matrix and the learnable weight matrix for feature projection.

[0015] Optionally, the action feature classification module is configured to: after determining the action features of the last layer, use a global average pooling operation to learn the global features of the human skeleton sequence; determine the predicted action label through a classification network composed of a softmax function and a fully connected layer; and determine the cross entropy loss function based on the predicted action label and the true action label.

[0016] Optionally, the action feature classification module is configured to: use a mapping network to embed the refined action topology representation into the vector space of common features to obtain contrast features; determine the prototype contrast loss function based on the contrast features, temperature hyperparameters of contrast learning, positive sample prototype aggregation and negative sample prototype aggregation related to the contrast features.

[0017] Optionally, the prototype reconstruction module is configured to determine the refined action topology representation based on a prototype index matrix, a detailed prototype library, sampling-adjusted hyperparameters and the graph topology features.

[0018] Optionally, the prototype reconstruction module is configured to: perform feature index mapping on the graph topology features using a prototype index matrix and sampling-adjusted hyperparameters; based on the mapping results, perform sampling and combination on the detail prototype library using a softmax function to determine the refined action topology representation.

[0019] According to another aspect of the present disclosure, a computing system is provided that includes at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the human behavior recognition method as described above.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the human behavior recognition method as described above.

[0021] By adopting the present disclosure, it is possible to better capture the subtle differences in actions and effectively extract discriminative behavioral features; significantly amplify key information, obtain refined behavioral features, and thus improve the accuracy of human behavior recognition; it is possible to solve the current problem that similar actions are difficult to accurately recognize, and further improve recognition performance. At the same time, the computational overhead of the present invention is very small, it can be applicable to various scenarios, and it has good generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or other objects and advantages of the present disclosure will become more apparent through the following description of embodiments in conjunction with the accompanying drawings, in which: Figure 1is a flow chart illustrating a method for human behavior recognition according to an exemplary embodiment of the present disclosure; Figure 2 is a schematic diagram showing an example of a human behavior recognition method according to the present disclosure; Figure 3 is a schematic diagram showing an example of a specific process of a motion detail enhancement part and a prototype reconstruction part of a human behavior recognition method according to the present disclosure; Figure 4 is a block diagram illustrating a human behavior recognition device according to an exemplary embodiment of the present disclosure; Figure 5 is a block diagram illustrating a computing system including at least one computing device and at least one storage device storing instructions according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Below, in conjunction with the accompanying drawings, a description of a specific embodiment is provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the method, device and / or system described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and conciseness, the description of features known in the art may be omitted.

[0024] In the existing human skeleton behavior recognition technology, the motion pattern characteristics of the behavior can be understood through the graph convolution method, and the features can be directly compared to distinguish different actions. However, this scheme does not fully consider the key motion details of the action sequence, only focuses on the general structural differences in the motion pattern, and ignores the subtle differences in the motion of specific important parts, so it cannot effectively distinguish similar actions, resulting in a high error rate in practical applications. The existing human skeleton behavior recognition technology is difficult to accurately identify when distinguishing similar action categories, and there is a high error rate. The reason why the existing methods cannot handle this situation well is mainly due to two shortcomings. First, it is difficult for the existing methods to capture the subtle differences in the motion details of key joints. The existing methods use learnable dynamic topology methods to adaptively aggregate the spatiotemporal information in the data sequence and simulate the motion and relationship of the body joints by establishing a joint relationship model. However, the learned dynamic topology graph actually only represents the structured motion pattern of the motion sequence, and is still insufficient in capturing the motion details of the discriminating sequence. For example, when distinguishing between writing and reading actions, the existing technology focuses on the joints related to the hand, but does not handle the motion details of the hand joints well. This key motion detail information is very important for distinguishing similar actions. Ignoring the details makes it impossible for existing technologies to accurately distinguish similar actions. In addition, the intra-class differences of samples also affect the accuracy of recognition. The intra-class differences of such action categories (such as changes in movement amplitude, etc.) will bring noise and interference to the extraction of motion patterns, affect the distinguishability of behavioral features, and increase the difficulty of recognition. The lack of the above motion details and the complex intra-class differences make it impossible for existing methods to accurately identify similar actions.

[0025] Figure 1 is a flowchart illustrating a human behavior recognition method according to an exemplary embodiment of the present disclosure.

[0026] like Figure 1 As shown, in step S101, based on the human skeleton sequence, the joint difference topological representation and the joint dependency topological representation are determined, and based on the joint difference topological representation and the joint dependency topological representation, the graph topological feature is determined, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and connection relationships between joints represented by an adjacency matrix, and the graph topological feature is the sum of the general structure topological representation, the joint difference topological representation and the joint dependency topological representation.

[0027] In the example, a multi-head attention mechanism is used to determine the joint-dependent topological representation based on the inner product of the projected features, and the joint-differential topological representation is determined based on the feature differences between different joints. For example, the input data is A human skeleton sequence of frames, where each human skeleton includes body joints. Due to the structural characteristics of the skeleton, it can be represented as a graph structure , among which Represents a collection of body joints, and the connection relationship between the edges of the joints is used To indicate the connection relationship Through the adjacency matrix to achieve, where Each element of the matrix Reflects the joints and The input and intermediate features of the model can be expressed as ,here is the number of joints, is the number of frames in the sequence, is the dimension of input data and features. In order to enhance the model’s ability to capture complex motion details, the matrix With two parts and To replace. Specifically, represents a learnable structural relationship of motion sequences, shared among all action categories, while depends on a specific input sequence , capturing the subtle movement details of the body joints related to the corresponding action category, which serves as the key information to distinguish similar actions. When calculating the topology, learnable convolution units are used respectively and The input features of each layer Mapped to dimension The query features and key-value features of and The corresponding dimension is , and use the multi-head attention mechanism at the same time, setting the number of multi-heads to , using time pooling operation to compress the time dimension. Compared with the existing direct use of time pooling operation on input features, the human behavior recognition method according to the exemplary embodiment of the present disclosure allows for more extensive context aggregation within different time ranges. Depend on and Calculate and obtain, Contains the association and difference information between body joints. Specifically, it mainly contains two aspects of movement details.

[0028] The first is the interdependence between joints, which is modeled by the inner product of the projected features and can be expressed as: (1) The second is the difference in changes between joints, which can be represented by the feature difference between different joints: (2) Among them, the transformation operation is the first Expand the tensor along a dimension and copy the result along that dimension Second-rate. and Have the same dimensions , and is the activation function used.

[0029] Finally, the topological representation is aggregated. is a learnable graph topology matrix that represents the universal topological structure information of human joints. and Represents the complex motion detailed topological information obtained through the attention mechanism. It has nothing to do with the input data. and It dynamically optimizes the input behavior sequence and further aggregates the three to obtain a more comprehensive topological representation of the motion graph. .

[0030] In step S102, the action features of the human skeleton sequence are determined based on the human skeleton sequence and the graph topology features.

[0031] In the example, the action features of the first layer are determined based on the human skeleton sequence, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection; the action features of the current layer are determined based on the action features of the previous layer, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection.

[0032] For example, for the action feature extractor , in ) layer characteristics and Layer characteristics The conversion function between can be defined as: (3) in, express The normalized adjacency matrix of represents the learnable weight matrix for feature projection, Indicates the use of ReLU activation function, Equivalent to the input data, the number of spatial graph layers is set to 9, the number of temporal graph layers is set to 3, and one temporal graph module is set for every three spatial graph modules.

[0033] Representing the graph topology Input to the action feature extractor , the calculation process corresponding to each layer can be expressed as: (4) In step S103, the graph topology features are reconstructed using fine-grained prototypes to obtain a refined action topology representation.

[0034] The diversity of human joint movements often causes these detailed information to become blurred and difficult to capture effectively during the entire movement process. Therefore, in order to solve this problem, the model's attention needs to be turned to identifying detailed motion features at the joint level, rather than just focusing on structural features at the skeleton level. It does not rely on predetermined human structural relationships (e.g., torso, arms, legs, etc.), but represents human behavior features as a combination of fine-grained prototypes, so that the model can adaptively learn the composition of human actions. In the example, a refined action topology representation is determined based on the prototype index matrix, the detailed prototype library, the sampling adjustment hyperparameters, and the graph topology features. For example, the prototype index matrix and the sampling adjustment hyperparameters can be used to map the graph topology features to feature indexes; based on the mapping results, the softmax function is used to sample and combine the detailed prototype library to determine the refined action topology representation.

[0035] For example, the graph topology representation of the input behavior sequence is set as , the dimension is , adjust it and transform it into , and then express it as a combination of sampled prototypes in the prototype template library, expressed as: (5) in, represents the prototype index matrix, Representation details prototype library, is a sampling adjustment hyperparameter. First use Topological features Perform feature index mapping and then use The function obtains the sampled combination selection. Details prototype library Storage capacity It is also a hyperparameter, which indicates the number of prototype templates stored in the detailed prototype library. The processing details of the prototype library This process actually uses the fine-grained prototype learned adaptively to reconstruct the human behavior characteristics, thereby obtaining a refined action topology representation. , significantly amplifying the movement details of key parts.

[0036] In step S104, based on the action features of the human skeleton sequence and the refined action topological representation, the action category of the human skeleton sequence is determined using the prototype contrast loss function and the cross entropy loss function. Through the supervision constraint of prototype contrast, the distinguishability of the motion pattern is enhanced, and irrelevant interference information is further eliminated, so that the behavior features of different actions have significant discrimination ability, and accurate human behavior recognition is performed.

[0037] In the example, after determining the action features of the last layer, a global average pooling operation is used to learn the global features of the human skeleton sequence; the predicted action label is determined through a classification network composed of a softmax function and a fully connected layer; the cross entropy loss function is determined based on the predicted action label and the true action label. In the example, a mapping network is used to embed the refined action topology representation into the vector space of common features to obtain contrast features; the prototype contrast loss function is determined based on the contrast features, the temperature hyperparameters of contrast learning, the positive sample prototype aggregation related to the contrast features, and the negative sample prototype aggregation.

[0038] The obtained action topology representation It can be seen as The weighted sum of the detailed prototypes retrieved from However, the prototype reconstruction process itself cannot ensure that the adaptive prototype is significantly representative. In order to further highlight the inter-class distinction of human actions, the classification recognition network In the design of cross entropy loss function Compare the loss function with the prototype , introduce more effective regularization constraints and enhance the discriminative ability of behavioral features.

[0039] For example, regarding the cross entropy loss function , action feature extractor The number of spatial module layers is set to 9, and the last layer of feature representation is obtained. Afterwards, a global average pooling operation is used to learn the global features of the sequence. The classification network composed of activation functions and fully connected layers further determines the predicted label ,in, Indicates the number of action categories, and the true action label is The cross entropy loss function for supervised prediction of action categories can be defined as: (6) Before calculating the prototype contrast loss, a mapping network is used to represent the action topology Embed the vector space of common features. The method is to first use the average pooling operation to Compress along the channel dimension to obtain a one-dimensional vector , and then use the fully connected layer to Transformed into contrasting features , represents the dimension of the contrast feature, and It serves as the input information for prototype contrast learning.

[0040] The essence of contrastive learning is to bring positive samples of the same category closer together in the feature space, while moving negative samples of different categories further apart. Since cross-contrast in a batch can enrich the sequence context information, setting the memory bank , the memory bank Elements Represents the The prototype aggregation of the unique attributes of the action categories. Take random initialization and use momentum update strategy for update. Given a batch of input samples, The update method is defined as: (7) in, Indicates that the category labels in the input batch are of Samples The average value of is the corresponding momentum hyperparameter.

[0041] Given contrast features , is the corresponding category label, is the batch index, the prototype contrast loss function can be expressed as: (8) in, and Respectively represent Related positive sample prototype aggregation and negative sample prototype aggregation, is the temperature hyperparameter for contrastive learning.

[0042] Action feature classification and recognition network The overall training objective function can be further expressed as: (9) in, It is a balanced hyperparameter used to adjust the proportional relationship between the cross entropy loss and the prototype contrast loss.

[0043] The computational process of prototype reconstruction enables the model to establish a prototype module library independent of the input data, and reconstruct the action representation by selecting the stored adaptive detail prototypes. Driven by the contrastive learning goal, the corresponding representation will become more discriminative. The amplification of inter-class differences in the feature space enables the adaptively learned prototype to capture representative action detail features. The prototype reconstruction mechanism will automatically explore the relationship between joints and summarize the prototype template that reflects the fine-grained motion characteristics. At the same time, in the process of obtaining key detail information, it will naturally filter out motion features with weaker discrimination ability, thereby suppressing noise and irrelevant information. In general, the human behavior recognition method based on action detail analysis proposed in the present invention represents human behavior characteristics as a combination of fine-grained prototypes, designs prototype contrast learning to understand the details of the movement, further amplifies and compares the key details of the action to distinguish similar actions, and improves the recognition accuracy of the model.

[0044] The human behavior recognition method according to the exemplary embodiment of the present disclosure achieves accurate human behavior recognition by representing human behavior features as a combination of fine-grained prototypes, understanding the details of the motion through a prototype contrast learning method, and amplifying and contrasting key details of the action to distinguish similar actions.

[0045] Figure 2 is a schematic diagram showing an example of a human behavior recognition method according to the present disclosure. Figure 3 It is a schematic diagram showing an example of the specific process of the motion detail enhancement part and the prototype reconstruction part of the human behavior recognition method according to the present disclosure.

[0046] According to the human behavior recognition method disclosed in the present invention, prototype comparison is used to effectively capture the intrinsic characteristics of the action, further understand and reveal the details of human movement, and achieve accurate behavior recognition. On the one hand, a motion detail enhancement module is proposed to analyze the details of the motion sequence through the attention mechanism, providing richer and more expressive topological information for the learning of the motion detail prototype. On the other hand, a prototype reconstruction module is designed to explicitly constrain the action feature representation, and through the combination of adaptive detail prototypes stored in the detail prototype library, a behavior representation with high discrimination ability that more effectively captures the intrinsic characteristics of the action is reconstructed. In addition, under the supervision of the proposed contrast training objective, the design of the prototype contrast loss is equivalent to the regularization constraint of the network, which prompts the model to actively discover unique motion detail prototypes, thereby enhancing the discriminative ability of motion features and further eliminating irrelevant interference information. In summary, the present invention represents human behavior features as a combination of fine-grained prototypes, designs a prototype contrast learning method to understand the details of the motion, amplifies and contrasts the action detail information to distinguish similar actions, and thus achieves accurate human behavior recognition.

[0047] like Figure 2 and Figure 3As shown, the human behavior recognition method according to the present disclosure may include the following process: Motion detail enhancement module Understand the movement details of key parts. Use the attention mechanism to mine the spatiotemporal information of data and adaptively learn the general structure topology. , capturing the subtle differences of actions through feature subtraction and inner product calculation, and learning the joint difference topology of actions and joint-dependent topology Then, the three topological relationships are fused to obtain a more comprehensive and discriminative graph topological feature. , and thus better understand the human behavior sequence; Through the action feature extractor Extract action features of human action sequences. Through graph convolution Extract feature representation of input action data using hierarchical spatial graph module and time graph module To understand the temporal and spatial variation information of the action sequence and integrate the detailed information of the action sequence ,Effectively extract rich and discriminative action features; Rebuilding modules through prototypes Use the learned detail prototypes to reconstruct motion features. In the prototype reconstruction mechanism, the detail prototypes of the action category are first adaptively learned, and then the prototype index matrix Combined reconstruction detail prototype library The prototype template is used to further obtain a refined action representation that can reflect the intrinsic characteristics of the action. ; Identify the network by classifying action features Calculate the supervision loss and classify the action categories. Through the cross entropy loss function Compare the loss function with the prototype ,Under the supervision of the proposed contrastive training objective, a more effective regularization constraint is introduced to enhance the discriminative ability of behavioral features. While significantly amplifying key information, the network can naturally filter noise information, eliminate the interference of irrelevant information, effectively analyze and understand the spatiotemporal information of the behavior sequence, and achieve accurate behavior recognition.

[0048] The whole method process can be summarized as the following steps: First, Using attention mechanism to understand the motion details of key parts , and then by Extract the action features of human behavior sequence, and then pass Use the learned detail prototypes to reconstruct motion features to obtain more refined action representation , and finally through Calculate the loss and impose more effective constraints on different action categories to achieve accurate human behavior action recognition.

[0049] According to the human behavior recognition method disclosed in the present invention, the motion detail information of local important joints is focused on at a finer granularity, and accurate human behavior recognition is achieved by decomposing the feature representation of the motion sequence into a combination of learnable prototypes and further analyzing the refined features obtained by reorganizing the detail prototypes. Through the attention mechanism, we can deeply understand the details of the action, further explore the spatiotemporal change information of the action sequence, capture the subtle differences in the behavior, comprehensively learn the structured and fine-grained topological representation of the sequence, and effectively extract discriminative behavioral features; adaptively learn the detailed prototypes of the action category, reconstruct the motion pattern using the prototype template, more effectively capture the intrinsic characteristics of the action, and make the learned behavioral features more refined through the design of prototype reconstruction, so as to achieve significant amplification of the motion details of key parts; introduce prototype contrast learning loss, design more effective regularization constraints, and prompt the model to actively explore unique motion detail prototypes under the guidance of contrastive learning. While significantly amplifying key information, it focuses on the main characteristics of the action features, further eliminates irrelevant interference information, and enhances the discriminative ability of behavioral features; represent human behavior characteristics as a combination of fine-grained prototypes, understand the details of the movement through prototype contrast learning, amplify and compare the key details of the action to distinguish similar actions, and further improve the recognition accuracy of the model, which has good generalization and robustness.

[0050] Figure 4 is a block diagram illustrating a human behavior recognition apparatus according to an exemplary embodiment of the present disclosure.

[0051] like Figure 4 As shown, according to an exemplary embodiment of the present disclosure, a human behavior recognition device 400 may include: a motion detail enhancement module 401, configured to determine a joint difference topological representation and a joint dependency topological representation based on a human skeleton sequence, and to determine a graph topological feature based on the joint difference topological representation and the joint dependency topological representation, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and a connection relationship between joints represented by an adjacency matrix, and the graph topological feature is the sum of a general structure topological representation, a joint difference topological representation and a joint dependency topological representation; an action feature extraction module 402, configured to determine the action feature of the human skeleton sequence based on the human skeleton sequence and the graph topological feature; a prototype reconstruction module 403, configured to reconstruct the graph topological feature using a fine-grained prototype to obtain a refined action topological representation; an action feature classification module 404, configured to determine the action category of the human skeleton sequence based on the action feature of the human skeleton sequence and the refined action topological representation using a prototype contrast loss function and a cross entropy loss function.

[0052] In the example, the motion detail enhancement module 401 is configured to: utilize a multi-head attention mechanism to determine a joint-dependent topological representation based on the inner product of projection features, and to determine a joint-differential topological representation based on feature differences between different joints.

[0053] In the example, the motion feature extraction module 402 is configured to: determine the motion features of the first layer based on the human skeleton sequence, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection; determine the motion features of the current layer based on the motion features of the previous layer, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection.

[0054] In the example, the action feature classification module 404 is configured to: after determining the action features of the last layer, use a global average pooling operation to learn the global features of the human skeleton sequence; determine the predicted action label through a classification network composed of a softmax function and a fully connected layer; and determine the cross entropy loss function based on the predicted action label and the true action label.

[0055] In the example, the action feature classification module 404 is configured to: use a mapping network to embed the refined action topology representation into the vector space of common features to obtain contrast features; determine the prototype contrast loss function based on the contrast features, the temperature hyperparameters of contrast learning, the positive sample prototype aggregation and the negative sample prototype aggregation related to the contrast features.

[0056] In the example, the prototype reconstruction module 403 is configured to determine a refined action topology representation based on the prototype index matrix, the detailed prototype library, the hyperparameters of sampling adjustment and the graph topology features.

[0057] In the example, the prototype reconstruction module 403 is configured to: perform feature index mapping on graph topology features using the prototype index matrix and sampling-adjusted hyperparameters; based on the mapping result, perform sampling and combination on the detail prototype library using the softmax function to determine a refined action topology representation.

[0058] Combination of the above Figures 1 to 3 The specific operations shown are respectively Figure 4 The operation is performed by corresponding units in the human behavior recognition device 400 shown in the figure, and the specific operation details will not be repeated here.

[0059] In the human behavior recognition device 400 according to the exemplary embodiment of the present disclosure, the motion detail enhancement module 401 is used to further mine the spatiotemporal variation information of the action sequence, and the attention mechanism is combined to deeply understand the details of the action, effectively extracting richer and more expressive behavior features; the prototype reconstruction module 403 further optimizes the behavior features, adaptively learns the detail prototypes of the action category, and uses the detail prototypes to reconstruct the motion pattern, more effectively captures the intrinsic characteristics of the action, and obtains more refined and more discriminative behavior features; in response to the interference of intra-class differences, based on the supervision constraints of prototype contrast learning, a more effective regularization constraint is introduced through the prototype contrast loss function, which prompts the model to actively discover unique motion detail prototypes, which helps to enhance the discriminative ability of behavior features. While significantly amplifying key information, it naturally filters out noise information and focuses on the core characteristics of the action, thereby eliminating the interference of irrelevant information, and can effectively analyze and understand the spatiotemporal information of the behavior sequence.

[0060] Figure 5 is a block diagram illustrating a computing system including at least one computing device and at least one storage device storing instructions according to an exemplary embodiment of the present disclosure.

[0061] like Figure 5 As shown, the computing system 500 provided according to an exemplary embodiment of the present invention includes a computing device 501 and a storage device 502, in which computer executable instructions are stored. When the computer executable instructions are executed by the computing device 501, the human behavior recognition method described in any of the aforementioned embodiments is executed.

[0062] The computing device 501 is deployed in a server or client, and can also be deployed on a node device in a distributed network environment. In addition, the computing device 501 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, a web application or other device capable of executing the above-mentioned instruction set. Here, the computing device is not necessarily a single computing device, but can also be any device or circuit that can execute the above-mentioned instructions (or instruction sets) individually or jointly. The computing device can also be part of an integrated control system or a system manager, or can be configured as a portable electronic device that is interconnected with an interface locally or remotely (for example, via wireless transmission). In the computing device, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor also includes an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0063] According to another aspect of the present disclosure, a computer-readable storage medium storing instructions is provided, and when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the human behavior recognition method described in any of the aforementioned embodiments. Computer-readable storage media include magnetic media such as floppy disks and tapes, optical media (including compact disk (CD) ROM and DVD ROM), magneto-optical media such as floppy disks, hardware devices such as ROM, RAM, and flash memory designed to store and execute program commands. The instructions may include language codes executable by a computer using an interpreter and machine language codes generated by a compiler.

[0064] By adopting the present disclosure, it is possible to better capture the subtle differences in actions and effectively extract discriminative behavioral features; significantly amplify key information, obtain refined behavioral features, and thus improve the accuracy of human behavior recognition; it is possible to solve the current problem that similar actions are difficult to accurately recognize, and further improve recognition performance. At the same time, the computational overhead of the present invention is very small, it can be applicable to various scenarios, and it has good generalization ability.

[0065] The processes, methods or algorithms disclosed herein may be transmitted to or implemented by a processing device, a controller or a computer, which may include any existing programmable electronic control unit or a dedicated electronic control unit. Similarly, the processes, methods or algorithms may be stored in a variety of forms as data and instructions that can be executed by a controller or a computer, including but not limited to information being permanently stored on a non-writable storage medium (such as a ROM device) and information being variably stored on a writable storage medium (such as a floppy disk, a tape, a CD, a RAM device, and other magnetic and optical media). The processes, methods or algorithms may also be implemented in a software executable object. Optionally, the processes, methods or algorithms may be implemented in whole or in part using suitable hardware components (such as an ASIC, an FPGA, a state machine, a controller or other hardware components or devices) or a combination of hardware components, software components and firmware components.

[0066] Although the present disclosure includes specific examples, it will be apparent to those of ordinary skill in the art that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein will be considered to be descriptive only and not for limiting purposes. The description of the features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented with other components or their equivalents in the described systems, architectures, devices, or circuits. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents will be interpreted as included in the present disclosure.

Claims

1. A human behavior recognition method, characterized in that: The human behavior recognition method comprises: Based on a human skeleton sequence, a joint difference topological representation and a joint dependency topological representation are determined, and based on the joint difference topological representation and the joint dependency topological representation, a graph topological feature is determined, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and connection relationships between joints represented by an adjacency matrix, and the graph topological feature is the sum of the general structure topological representation, the joint difference topological representation and the joint dependency topological representation; Determining the motion features of the human skeleton sequence based on the human skeleton sequence and the graph topology features; Reconstructing the graph topology features using fine-grained prototypes to obtain a refined action topology representation; Based on the action features of the human skeleton sequence and the refined action topology representation, the action category of the human skeleton sequence is determined by using the prototype contrast loss function and the cross entropy loss function.

2. The human behavior recognition method according to claim 1, characterized in that: Based on the human skeleton sequence, the steps of determining the joint difference topological representation and the joint dependence topological representation include: using a multi-head attention mechanism to determine the joint dependence topological representation based on the inner product of projection features, and determining the joint difference topological representation based on the feature differences between different joints.

3. The human behavior recognition method according to claim 1, characterized in that: The step of determining the action feature of the human skeleton sequence based on the human skeleton sequence and the graph topology feature comprises: Determining a first layer of action features based on the human skeleton sequence, a normalized adjacency matrix of the adjacency matrix, and a learnable weight matrix for feature projection; The action features of the current layer are determined based on the action features of the previous layer, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection.

4. The human behavior recognition method according to claim 3, characterized in that: The human behavior recognition method also includes: After determining the action features of the last layer, a global average pooling operation is used to learn the global features of the human skeleton sequence; Determine the predicted action label through a classification network composed of a softmax function and a fully connected layer; The cross entropy loss function is determined based on the predicted action label and the true action label.

5. The human behavior recognition method according to claim 1, characterized in that: The human behavior recognition method also includes: Use a mapping network to embed the refined action topology representation into the vector space of common features to obtain contrast features; The prototype contrast loss function is determined based on the contrast feature, a temperature hyperparameter of contrast learning, and a positive sample prototype aggregation and a negative sample prototype aggregation related to the contrast feature.

6. The human behavior recognition method according to claim 1, characterized in that: The step of reconstructing the graph topology features using fine-grained prototypes to obtain a refined action topology representation includes: determining the refined action topology representation based on a prototype index matrix, a detailed prototype library, sampling-adjusted hyperparameters and the graph topology features.

7. The human behavior recognition method according to claim 6, characterized in that: The step of determining the refined action topology representation includes: using the prototype index matrix and the sampling adjustment hyperparameters to perform feature index mapping on the graph topology features; based on the mapping result, using the softmax function to sample and combine the detail prototype library to determine the refined action topology representation.

8. A human behavior recognition device, characterized in that: The human behavior recognition device comprises: A motion detail enhancement module is configured to determine a joint difference topological representation and a joint dependency topological representation based on a human skeleton sequence, and determine a graph topological feature based on the joint difference topological representation and the joint dependency topological representation, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and a connection relationship between the joints represented by an adjacency matrix, and the graph topological feature is the sum of the general structure topological representation, the joint difference topological representation and the joint dependency topological representation; an action feature extraction module, configured to determine the action features of the human skeleton sequence based on the human skeleton sequence and the graph topology features; A prototype reconstruction module is configured to reconstruct the graph topology features using fine-grained prototypes to obtain a refined action topology representation; The action feature classification module is configured to determine the action category of the human skeleton sequence based on the action features of the human skeleton sequence and the refined action topology representation using a prototype contrast loss function and a cross entropy loss function.

9. A computing system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the human behavior recognition method according to any one of claims 1-7.

10. A computer-readable storage medium storing instructions, characterized in that: When the instruction is executed by at least one computing device, the at least one computing device is prompted to execute the human behavior recognition method according to any one of claims 1-7.

Images