Human behavior recognition method and device

Abstract

The present disclosure relates to a method and device for human behavior recognition. The human behavior recognition method includes: determining a joint difference topological representation and a joint dependency topological representation based on a human skeleton sequence, and determining a graph topological feature based on the joint difference topological representation and the joint dependency topological representation, wherein the human skeleton sequence is represented as a graph structure, and the graph topological feature is the sum of the general structure topological representation, the joint difference topological representation and the joint dependency topological representation; determining the action features of the human skeleton sequence based on the human skeleton sequence and the graph topological feature; reconstructing the graph topological feature using a fine-grained prototype to obtain a refined action topological representation; determining the action category of the human skeleton sequence based on the action features of the human skeleton sequence and the refined action topological representation using a prototype contrast loss function and a cross entropy loss function. By adopting the present disclosure, it is possible to better capture subtle differences in actions and effectively extract discriminative behavior features.

Description

Technical Field

[0001] The present disclosure relates to the fields of video analysis and behavior understanding, and more specifically, to a method and device for human behavior recognition. Background Art

[0002] Human action recognition involves modeling the spatiotemporal information of moving humans. By processing video data to obtain characteristic representations of human actions, it can identify the target person's ongoing movements (e.g., walking, drinking), or interactions between multiple target persons (e.g., handshakes, hugs). The spatiotemporal sequence information of movement is a fundamental component of human behavior. Establishing a semantic mapping between low-level visual features and high-level behavioral categories is a core issue in human action recognition. Human action recognition technology is widely used in many real-life scenarios, such as intelligent video surveillance, human-computer interaction, virtual reality, and motion simulation. Accurate action recognition is crucial in these scenarios. Skeleton sequence data has attracted considerable attention because it focuses on the motion information most relevant to the action and is robust to complex video backgrounds. Human action recognition technology based on skeleton sequence data is widely used in video analysis and behavior understanding.

[0003] However, in practical applications, current human action recognition technology struggles to accurately distinguish similar action categories. Human action recognition primarily involves two steps: first, identifying the key body parts involved in the movement, and then discerning the complex motion patterns of these key parts. Existing methods can effectively capture the key parts involved in the movement, but they still face challenges in capturing and discerning motion details, crucial for distinguishing similar actions. Furthermore, intra-class variations within action categories (for example, changes in movement amplitude) inevitably introduce noise and interference into the process of extracting motion patterns. This intra-class variation further complicates the extraction of good motion features by existing technologies, making it even more challenging to accurately distinguish similar actions. This lack of motion details and complex intra-class variation prevent current technologies from accurately identifying similar actions, resulting in high error rates. Summary of the Invention

[0004] In order to solve the above problems, the present disclosure proposes a human behavior recognition method and device, a computing system and a computer-readable storage medium.

[0005] According to one aspect of the present disclosure, a human behavior recognition method is provided, which includes: determining a joint difference topological representation and a joint dependency topological representation based on a human skeleton sequence, and determining a graph topological feature based on the joint difference topological representation and the joint dependency topological representation, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and a connection relationship between the joints represented by an adjacency matrix, and the graph topological feature is the sum of a general structure topological representation, a joint difference topological representation and a joint dependency topological representation; determining the action features of the human skeleton sequence based on the human skeleton sequence and the graph topological feature; reconstructing the graph topological feature using a fine-grained prototype to obtain a refined action topological representation; determining the action category of the human skeleton sequence based on the action features of the human skeleton sequence and the refined action topological representation using a prototype contrast loss function and a cross entropy loss function.

[0006] Optionally, based on the human skeleton sequence, the steps of determining the joint difference topological representation and the joint dependence topological representation include: utilizing a multi-head attention mechanism to determine the joint dependence topological representation based on the inner product of projection features, and determining the joint difference topological representation based on the feature differences between different joints.

[0007] Optionally, based on the human skeleton sequence and the graph topology features, the step of determining the motion features of the human skeleton sequence includes: determining the motion features of the first layer based on the human skeleton sequence, the normalized adjacency matrix of the adjacency matrix and the learnable weight matrix for feature projection; determining the motion features of the current layer based on the motion features of the previous layer, the normalized adjacency matrix of the adjacency matrix and the learnable weight matrix for feature projection.

[0008] Optionally, the human action recognition method also includes: after determining the action features of the last layer, using a global average pooling operation to learn the global features of the human skeleton sequence; determining the predicted action label through a classification network composed of a softmax function and a fully connected layer; and determining the cross entropy loss function based on the predicted action label and the true action label.

[0009] Optionally, the human behavior recognition method also includes: using a mapping network to embed the refined action topology representation into the vector space of common features to obtain contrast features; determining the prototype contrast loss function based on the contrast features, the temperature hyperparameters of contrast learning, the positive sample prototype aggregation and the negative sample prototype aggregation related to the contrast features.

[0010] Optionally, the step of reconstructing the graph topology features using fine-grained prototypes to obtain a refined action topology representation includes: determining the refined action topology representation based on a prototype index matrix, a detailed prototype library, sampling-adjusted hyperparameters and the graph topology features.

[0011] Optionally, the step of determining the refined action topology representation includes: performing feature index mapping on the graph topology features using a prototype index matrix and sampling-adjusted hyperparameters; based on the mapping result, sampling and combining the detail prototype library using a softmax function to determine the refined action topology representation.

[0012] According to another aspect of the present disclosure, a human behavior recognition device is provided, which includes: a motion detail enhancement module, configured to determine a joint difference topological representation and a joint dependency topological representation based on a human skeleton sequence, and determine a graph topological feature based on the joint difference topological representation and the joint dependency topological representation, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and a connection relationship between the joints represented by an adjacency matrix, and the graph topological feature is the sum of a general structure topological representation, a joint difference topological representation and a joint dependency topological representation; an action feature extraction module, configured to determine the action features of the human skeleton sequence based on the human skeleton sequence and the graph topological features; a prototype reconstruction module, configured to reconstruct the graph topological features using a fine-grained prototype to obtain a refined action topological representation; an action feature classification module, configured to determine the action category of the human skeleton sequence based on the action features of the human skeleton sequence and the refined action topological representation using a prototype contrast loss function and a cross entropy loss function.

[0013] Optionally, the motion detail enhancement module is configured to: utilize a multi-head attention mechanism to determine the joint-dependent topological representation based on the inner product of projection features, and determine the joint-differential topological representation based on feature differences between different joints.

[0014] Optionally, the motion feature extraction module is configured to: determine the motion features of the first layer based on the human skeleton sequence, the normalized adjacency matrix of the adjacency matrix and the learnable weight matrix for feature projection; determine the motion features of the current layer based on the motion features of the previous layer, the normalized adjacency matrix of the adjacency matrix and the learnable weight matrix for feature projection.

[0015] Optionally, the action feature classification module is configured to: after determining the action features of the last layer, use a global average pooling operation to learn the global features of the human skeleton sequence; determine the predicted action label through a classification network composed of a softmax function and a fully connected layer; and determine the cross entropy loss function based on the predicted action label and the true action label.

[0016] Optionally, the action feature classification module is configured to: use a mapping network to embed the refined action topology representation into the vector space of common features to obtain contrast features; and determine the prototype contrast loss function based on the contrast features, the temperature hyperparameters of contrast learning, the positive sample prototype aggregation and the negative sample prototype aggregation related to the contrast features.

[0017] Optionally, the prototype reconstruction module is configured to determine the refined action topology representation based on a prototype index matrix, a detailed prototype library, hyperparameters of sampling adjustment, and the graph topology features.

[0018] Optionally, the prototype reconstruction module is configured to: perform feature index mapping on the graph topology features using the prototype index matrix and sampling adjustment hyperparameters; based on the mapping results, perform sampling and combination on the detail prototype library using the softmax function to determine the refined action topology representation.

[0019] According to another aspect of the present disclosure, a computing system is provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the human behavior recognition method as described above.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the human behavior recognition method as described above.

[0021] By adopting the present disclosure, it is possible to better capture subtle differences in actions and effectively extract discriminative behavioral features; significantly amplify key information, obtain refined behavioral features, and thus improve the accuracy of human behavior recognition; it is possible to solve the current problem of similar actions being difficult to accurately recognize, and further improve recognition performance. At the same time, the computational overhead of the present invention is very low, it can be applied to various scenarios, and has good generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or other objects and advantages of the present disclosure will become more apparent through the following description of embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1is a flow chart illustrating a method for recognizing human behavior according to an exemplary embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram illustrating an example of a human behavior recognition method according to the present disclosure;

[0025] Figure 3 is a schematic diagram illustrating an example of a specific process of a motion detail enhancement part and a prototype reconstruction part of the human behavior recognition method according to the present disclosure;

[0026] Figure 4 is a block diagram illustrating a human behavior recognition apparatus according to an exemplary embodiment of the present disclosure;

[0027] Figure 5 is a block diagram illustrating a computing system including at least one computing device and at least one storage device storing instructions according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Below, in conjunction with the accompanying drawings, a description of a specific embodiment is provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and conciseness, descriptions of features known in the art may be omitted.

[0029] Existing human skeleton action recognition technologies use graph convolutional methods to understand the motion pattern characteristics of actions and directly compare these features to distinguish different actions. However, this approach fails to fully consider the key motion details of the action sequence, focusing only on general structural differences in the motion patterns while ignoring the subtle differences in the motion of specific key parts. This results in an inability to effectively distinguish similar actions, leading to high error rates in practical applications. Existing human skeleton action recognition technologies struggle to accurately distinguish similar action categories, resulting in high error rates. This inability of existing methods is primarily due to two shortcomings. First, existing methods struggle to capture the subtle differences in the motion details of key joints. Existing methods use learnable dynamic topology methods to adaptively aggregate spatiotemporal information from data sequences and model the motion and relationships of body joints by building joint relationship models. However, the learned dynamic topology graphs only represent the structured motion patterns of the motion sequence and are insufficient in capturing and distinguishing the motion details of the sequence. For example, when distinguishing between writing and reading, existing technologies focus on the joints associated with the hands but fall short in understanding the motion details of these joints. This crucial motion detail is crucial for distinguishing similar actions. Ignoring this detail prevents existing techniques from accurately distinguishing similar actions. Furthermore, intra-class variability among samples also impacts recognition accuracy. This intra-class variability within action categories (e.g., variations in movement amplitude) introduces noise and interference into the extracted motion patterns, affecting the distinguishability of behavioral features and increasing recognition difficulty. This lack of motion detail and complex intra-class variability prevent existing methods from accurately identifying similar actions.

[0030] Figure 1 is a flowchart illustrating a human behavior recognition method according to an exemplary embodiment of the present disclosure.

[0031] like Figure 1 As shown, in step S101, based on the human skeleton sequence, the joint difference topological representation and the joint dependency topological representation are determined, and based on the joint difference topological representation and the joint dependency topological representation, the graph topological feature is determined, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and the connection relationship between the joints represented by an adjacency matrix, and the graph topological feature is the sum of the general structure topological representation, the joint difference topological representation and the joint dependency topological representation.

[0032] In this example, a multi-head attention mechanism is used to determine the joint dependency topology representation based on the inner product of the projected features, and the joint difference topology representation is determined based on the feature differences between different joints. For example, the input data is A human skeleton sequence of frames, where each human skeleton includes body joints. Due to the structural characteristics of the skeleton, it can be represented as a graph structure , among which Represents a collection of body joints, and the connection relationship between the joint edges is used To express. Connection relationship Through the adjacency matrix to achieve, where Each element of the matrix Reflects the joints and The input and intermediate features of the model can be expressed as ,here is the number of joints, is the number of frames in the sequence, is the dimension of input data and features. In order to enhance the model’s ability to capture complex motion details, the matrix Use two parts and To replace. Specifically, represents a learnable structural relationship of motion sequences, shared across all action categories, while It depends on a specific input sequence , capturing the subtle movement details of the body joints related to the corresponding action category, which serves as the key information to distinguish similar actions. When calculating the topology, learnable convolution units are used respectively. and The input features of each layer Mapped to dimension The query features and key-value features of and The corresponding dimension is , and use the multi-head attention mechanism at the same time, setting the number of multi-heads to , using time pooling operation to compress the time dimension. Compared with the existing direct use of time pooling operation on input features, the human behavior recognition method according to the exemplary embodiment of the present disclosure allows for a wider range of context aggregation in different time ranges. Depend on and Calculated, Contains information about the connections and differences between body joints. Specifically, it contains two aspects of motion details.

[0033] The first is the interdependence between joints, which is modeled by the inner product of projected features and can be expressed as:

[0034] (1)

[0035] The second is the relationship between the changes in joints, which can be expressed by the characteristic differences between different joints:

[0036] (2)

[0037] Among them, the transformation operation is the first Expand the tensor along a dimension and copy the result along that dimension Second-rate. and Have the same dimensions , and is the activation function used.

[0038] Finally, the topological representation is aggregated. It is a learnable graph topology matrix that represents the universal topological structure information of human joints. and Represents the complex motion detail topological information obtained through the attention mechanism. It has nothing to do with the input data. and It dynamically optimizes the input behavior sequence and further aggregates the three to obtain a more comprehensive topological representation of the motion graph. .

[0039] In step S102 , the motion features of the human skeleton sequence are determined based on the human skeleton sequence and the graph topology features.

[0040] In the example, the motion features of the first layer are determined based on the human skeleton sequence, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection; the motion features of the current layer are determined based on the motion features of the previous layer, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection.

[0041] For example, for an action feature extractor , in ) layer characteristics Hedi Layer characteristics The conversion function between can be defined as:

[0042] (3)

[0043] in, express The normalized adjacency matrix of represents the learnable weight matrix for feature projection, Indicates the use of ReLU activation function, Equivalent to the input data, the number of layers of the spatial map is set to 9, the number of layers of the temporal map is set to 3, and one layer of temporal map module is set for every three layers of spatial map modules.

[0044] Representing the graph topology Input to the action feature extractor , the calculation process corresponding to each layer can be expressed as:

[0045] (4)

[0046] In step S103 , the graph topology features are reconstructed using the fine-grained prototype to obtain a refined action topology representation.

[0047] The diversity of human joint movements often makes these detailed information blurred and difficult to capture effectively during the entire movement process. Therefore, to solve this problem, the model's attention needs to be shifted to identifying detailed motion features at the joint level, rather than just focusing on structural features at the skeleton level. Rather than relying on predetermined human structural relationships (e.g., torso, arms, legs, etc.), this approach represents human behavioral characteristics as a combination of fine-grained prototypes, enabling the model to adaptively learn the composition of human actions. In this example, a refined action topology representation is determined based on a prototype index matrix, a library of detailed prototypes, sampling-adjusted hyperparameters, and graph topology features. For example, the prototype index matrix and sampling-adjusted hyperparameters can be used to map graph topology features to feature indices. Based on the mapping results, a softmax function is used to sample and combine the detailed prototype library to determine a refined action topology representation.

[0048] For example, let the graph topology of the input behavior sequence be represented as , the dimension is , adjust it and transform it into , and then express it as a combination of sampled prototypes in the prototype template library, expressed as:

[0049] (5)

[0050] in, represents the prototype index matrix, Representation details prototype library, It is a sampling adjustment hyperparameter. First use Topological features Perform feature index mapping and then use The function gets the sample combination selection. Details prototype library Storage capacity It is also a hyperparameter, which indicates the number of prototype templates stored in the detailed prototype library. Processing details prototype library Sampling combination is performed. This process actually uses the fine-grained prototype learned adaptively to reconstruct the human behavior characteristics, and then obtains a refined action topology representation. , significantly amplifying the movement details of key parts.

[0051] In step S104, the action categories of the human skeleton sequence are determined using the prototype contrast loss function and the cross-entropy loss function based on the motion features of the human skeleton sequence and the refined motion topological representation. The supervised constraints of prototype contrast enhance the distinguishability of motion patterns and further eliminate irrelevant interference information, making the behavioral features of different actions highly discriminative and enabling accurate human action recognition.

[0052] In this example, after determining the action features of the last layer, a global average pooling operation is used to learn the global features of the human skeleton sequence. A classification network consisting of a softmax function and a fully connected layer is used to determine the predicted action label. The cross-entropy loss function is determined based on the predicted action label and the true action label. In this example, a mapping network is used to embed the refined action topology representation into the vector space of common features to obtain contrast features. The prototype contrast loss function is determined based on the contrast features, the temperature hyperparameter of contrastive learning, and the positive and negative prototype aggregations related to the contrast features.

[0053] The obtained action topology representation Can be seen as The weighted sum of the detail prototypes retrieved from However, the prototype reconstruction process itself cannot ensure that the adaptive prototype is significantly representative. In order to further highlight the difference between human actions, the classification and recognition network In the design of cross entropy loss function Comparison loss function with prototype , introducing more effective regularization constraints and enhancing the discriminative ability of behavioral features.

[0054] For example, regarding the cross entropy loss function , action feature extractor The number of spatial module layers is set to 9, and the last layer of feature representation is obtained. After that, a global average pooling operation is used to learn the global features of the sequence. The classification network composed of activation function and fully connected layer further determines the predicted label ,in, Indicates the number of action categories, the real action label is The cross entropy loss function for supervised prediction of action categories can be defined as:

[0055] (6)

[0056] Before calculating the prototype contrast loss, a mapping network is used to represent the action topology Embed the vector space of common features. The method is to first use the average pooling operation to Compress along the channel dimension to obtain a one-dimensional vector , and then use the fully connected layer to Transformed into contrasting features , represents the dimension of contrast features, and It serves as the input information for prototype contrast learning.

[0057] The essence of contrastive learning is to bring positive samples of the same category closer together in the feature space, while moving negative samples of different categories further apart. Since cross-contrast in a batch can enrich the sequence context information, setting up a memory bank , the memory bank Elements Represents the The prototype aggregation of the unique attributes of the action categories. Take random initialization and use momentum update strategy to update. Given a batch of input samples, The update method is defined as:

[0058] (7)

[0059] in, Indicates that the category labels in the input batch are of samples The average value of is the corresponding momentum hyperparameter.

[0060] Given contrast features , is the corresponding category label, For batch index, the prototype contrast loss function can be expressed as:

[0061] (8)

[0062] in, and Respectively represent Related positive sample prototype aggregation and negative sample prototype aggregation, is the temperature hyperparameter for contrastive learning.

[0063] Action feature classification and recognition network The overall training objective function can be further expressed as:

[0064] (9)

[0065] in, It is a balance hyperparameter used to adjust the proportional relationship between cross entropy loss and prototype contrast loss.

[0066] The computational process of prototype reconstruction enables the model to establish a prototype module library independent of the input data, and reconstruct the action representation by selecting the stored adaptive detail prototypes. Driven by the contrastive learning goal, the corresponding representation will become more discriminative. The amplification of inter-class differences in the feature space enables the adaptively learned prototype to capture representative action detail features. The prototype reconstruction mechanism will automatically explore the relationship between joints and summarize the prototype template that reflects the fine-grained motion characteristics. At the same time, in the process of obtaining key detail information, it will naturally filter out motion features with weaker discrimination capabilities, thereby suppressing noise and irrelevant information. In general, the human behavior recognition method based on action detail analysis proposed in the present invention represents human behavior characteristics as a combination of fine-grained prototypes, designs prototype contrast learning to understand the details of the movement, and further amplifies and compares key details of the movement to distinguish similar movements, thereby improving the recognition accuracy of the model.

[0067] The human behavior recognition method according to an exemplary embodiment of the present disclosure achieves accurate human behavior recognition by representing human behavior features as a combination of fine-grained prototypes, understanding the details of the movement through a prototype contrast learning method, and amplifying and contrasting key details of the action to distinguish similar actions.

[0068] Figure 2 is a schematic diagram illustrating an example of a human behavior recognition method according to the present disclosure. Figure 3 3 is a schematic diagram showing an example of the specific process of the motion detail enhancement part and the prototype reconstruction part of the human behavior recognition method according to the present disclosure.

[0069] According to the human behavior recognition method disclosed in the present invention, prototype comparison is used to effectively capture the intrinsic characteristics of the action, further understand and reveal the details of human motion, and achieve accurate behavior recognition. On the one hand, a motion detail enhancement module is proposed to analyze the details of the motion sequence through the attention mechanism, providing richer and more expressive topological information for the learning of motion detail prototypes. On the other hand, a prototype reconstruction module is designed to explicitly constrain the action feature representation, and through the combination of adaptive detail prototypes stored in the detail prototype library, a behavior representation with high discrimination ability that more effectively captures the intrinsic characteristics of the action is reconstructed. In addition, under the supervision of the proposed contrast training objective, the design of the prototype contrast loss is equivalent to the regularization constraint of the network, which prompts the model to actively discover unique motion detail prototypes, thereby enhancing the discriminative ability of motion features and further eliminating irrelevant interference information. In summary, the present invention represents human behavior features as a combination of fine-grained prototypes, designs a prototype contrast learning method to understand the details of the motion, amplifies and contrasts the action detail information to distinguish similar actions, and thus achieves accurate human behavior recognition.

[0070] like Figure 2 and Figure 3 As shown, the human behavior recognition method according to the present disclosure may include the following process:

[0071] Through the motion detail enhancement module Understand the movement details of key parts. Use the attention mechanism to mine the spatiotemporal information of the data and adaptively learn the general structure topology. , by capturing the subtle differences of the action through feature subtraction and inner product calculation, and learning the joint difference topology of the action and joint-dependent topology , and then the three topological relationships are fused to obtain a more comprehensive and discriminative graph topological feature , and thus better understand the human behavior sequence;

[0072] Through the action feature extractor Extract action features of human action sequences through graph convolution Extract feature representation of input action data using a hierarchical spatial graph module and time graph modules To understand the spatiotemporal variation of action sequences and integrate the detailed information of action sequences ,Effectively extract rich and discriminative action features;

[0073] Rebuilding modules through prototypes Use the learned detail prototypes to reconstruct motion features. In the prototype reconstruction mechanism, the detail prototypes of the action category are first adaptively learned, and then the prototype index matrix Combined reconstruction detail prototype library The prototype template is used to further obtain a refined action representation that can reflect the intrinsic characteristics of the action. ;

[0074] Recognition network through action feature classification Calculate the supervision loss and perform classification and recognition of action categories. Through the cross entropy loss function Comparison loss function with prototype Under the supervision of the proposed contrastive training objective, more effective regularization constraints are introduced to enhance the discriminative ability of behavioral features. While significantly amplifying key information, the network can naturally filter out noise information, eliminate the interference of irrelevant information, effectively analyze and understand the spatiotemporal information of behavioral sequences, and achieve accurate behavior recognition.

[0075] The whole method process can be summarized as the following steps: First, Using attention mechanism to understand the motion details of key parts , then by Extract the action features of human behavior sequence, and then Use the learned detail prototypes to reconstruct motion features to obtain more refined action representation , and finally through Calculate the loss and impose more effective constraints on different action categories to achieve accurate human action recognition.

[0076] According to the human behavior recognition method disclosed in the present invention, the motion detail information of local important joints is focused on at a finer granularity, and accurate human behavior recognition is achieved by decomposing the feature representation of the motion sequence into a combination of learnable prototypes and further analyzing the refined features obtained by recombining the detail prototypes. Through the attention mechanism, we deeply understand the details of the action, further explore the spatiotemporal variation information of the action sequence, capture the subtle differences in behavior, comprehensively learn the structured and fine-grained topological representation of the sequence, and effectively extract discriminative behavioral features; adaptively learn the detailed prototypes of the action category, use the prototype template to reconstruct the motion pattern, more effectively capture the intrinsic characteristics of the action, and make the learned behavioral features more refined through the design of prototype reconstruction, so as to achieve significant amplification of the motion details of key parts; introduce prototype contrast learning loss and design more effective regularization constraints, so as to prompt the model to actively explore unique motion detail prototypes under the guidance of contrast learning, while significantly amplifying key information, focusing on the main characteristics of the action features, further eliminating irrelevant interference information, and enhancing the discriminative ability of behavioral features; represent human behavioral features as a combination of fine-grained prototypes, understand the details of the movement through prototype contrast learning, amplify and compare the key details of the action to distinguish similar actions, further improve the recognition accuracy of the model, and have good generalization and robustness.

[0077] Figure 4is a block diagram illustrating a human behavior recognition apparatus according to an exemplary embodiment of the present disclosure.

[0078] like Figure 4 As shown, a human behavior recognition device 400 according to an exemplary embodiment of the present disclosure may include: a motion detail enhancement module 401, configured to determine a joint difference topological representation and a joint dependency topological representation based on a human skeleton sequence, and to determine a graph topological feature based on the joint difference topological representation and the joint dependency topological representation, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and a connection relationship between joints represented by an adjacency matrix, and the graph topological feature is the sum of a general structure topological representation, a joint difference topological representation and a joint dependency topological representation; an action feature extraction module 402, configured to determine the action feature of the human skeleton sequence based on the human skeleton sequence and the graph topological feature; a prototype reconstruction module 403, configured to reconstruct the graph topological feature using a fine-grained prototype to obtain a refined action topological representation; an action feature classification module 404, configured to determine the action category of the human skeleton sequence based on the action feature of the human skeleton sequence and the refined action topological representation using a prototype contrast loss function and a cross entropy loss function.

[0079] In the example, the motion detail enhancement module 401 is configured to: utilize a multi-head attention mechanism to determine a joint-dependent topological representation based on the inner product of the projected features, and to determine a joint-differential topological representation based on the feature differences between different joints.

[0080] In the example, the motion feature extraction module 402 is configured to: determine the motion features of the first layer based on the human skeleton sequence, the normalized adjacency matrix of the adjacency matrix and the learnable weight matrix for feature projection; determine the motion features of the current layer based on the motion features of the previous layer, the normalized adjacency matrix of the adjacency matrix and the learnable weight matrix for feature projection.

[0081] In the example, the action feature classification module 404 is configured to: after determining the action features of the last layer, use a global average pooling operation to learn the global features of the human skeleton sequence; determine the predicted action label through a classification network composed of a softmax function and a fully connected layer; and determine the cross entropy loss function based on the predicted action label and the true action label.

[0082] In the example, the action feature classification module 404 is configured to: use a mapping network to embed the refined action topology representation into the vector space of common features to obtain contrast features; determine the prototype contrast loss function based on the contrast features, the temperature hyperparameters of contrast learning, the positive sample prototype aggregation and the negative sample prototype aggregation related to the contrast features.

[0083] In an example, the prototype reconstruction module 403 is configured to determine a refined action topology representation based on the prototype index matrix, the detailed prototype library, the hyperparameters of the sampling adjustment, and the graph topology features.

[0084] In the example, the prototype reconstruction module 403 is configured to: perform feature index mapping on graph topology features using the prototype index matrix and sampling adjustment hyperparameters; based on the mapping results, use the softmax function to sample and combine the detail prototype library to determine a refined action topology representation.

[0085] Combination of the above Figures 1 to 3 The specific operations shown are respectively Figure 4 The corresponding units in the human behavior recognition device 400 shown in the figure are used for execution, and the specific operation details will not be repeated here.

[0086] In the human behavior recognition device 400 according to the exemplary embodiment of the present disclosure, the spatiotemporal variation information of the action sequence is further mined through the motion detail enhancement module 401, and the details of the action are deeply understood in combination with the attention mechanism, effectively extracting richer and more expressive behavioral features; the behavioral features are further optimized through the prototype reconstruction module 403, and the detail prototypes of the action category are adaptively learned. The motion pattern is reconstructed using the detail prototypes to more effectively capture the intrinsic characteristics of the action and obtain more refined and discriminative behavioral features; in response to the interference of intra-class differences, based on the supervision constraints of prototype contrast learning, a more effective regularization constraint is introduced through the prototype contrast loss function, which prompts the model to actively discover unique motion detail prototypes, which helps to enhance the discriminative ability of behavioral features. While significantly amplifying key information, it naturally filters out noise information and focuses on the core characteristics of the action, thereby eliminating the interference of irrelevant information and effectively analyzing and understanding the spatiotemporal information of the behavior sequence.

[0087] Figure 5 is a block diagram illustrating a computing system including at least one computing device and at least one storage device storing instructions according to an exemplary embodiment of the present disclosure.

[0088] like Figure 5 As shown, the computing system 500 provided according to an exemplary embodiment of the present invention includes a computing device 501 and a storage device 502, wherein the storage device 502 stores computer executable instructions. When the computer executable instructions are executed by the computing device 501, the human behavior recognition method described in any of the aforementioned embodiments is executed.

[0089] The computing device 501 is deployed in a server or client, or can be deployed on a node device in a distributed network environment. In addition, the computing device 501 can be a PC, tablet device, personal digital assistant, smartphone, web application, or other device capable of executing the above-mentioned instruction set. Here, the computing device does not necessarily have to be a single computing device, but can also be any collection of devices or circuits that can execute the above-mentioned instructions (or instruction sets) individually or in combination. The computing device can also be part of an integrated control system or system manager, or can be configured as a portable electronic device that is interconnected with an interface locally or remotely (for example, via wireless transmission). In the computing device, the processor includes a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor also includes an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0090] According to another aspect of the present disclosure, a computer-readable storage medium storing instructions is provided. When the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the human behavior recognition method described in any of the aforementioned embodiments. Computer-readable storage media include magnetic media such as floppy disks and magnetic tapes, optical media (including compact disc (CD) ROMs and DVD ROMs), magneto-optical media such as floppy disks, and hardware devices designed to store and execute program commands, such as ROMs, RAMs, and flash memories. The instructions may include language codes executable by a computer using an interpreter, as well as machine language codes generated by a compiler.

[0091] By adopting the present disclosure, it is possible to better capture subtle differences in actions and effectively extract discriminative behavioral features; significantly amplify key information, obtain refined behavioral features, and thus improve the accuracy of human behavior recognition; it is possible to solve the current problem of similar actions being difficult to accurately recognize, and further improve recognition performance. At the same time, the computational overhead of the present invention is very low, it can be applied to various scenarios, and has good generalization capabilities.

[0092] The processes, methods, or algorithms disclosed herein may be transmitted to or implemented by a processing device, controller, or computer, which may include any existing programmable electronic control unit or a dedicated electronic control unit. Similarly, the processes, methods, or algorithms may be stored as data and instructions executable by a controller or computer in a variety of forms, including but not limited to permanent storage of information on non-writable storage media (such as ROM devices) and mutably storage of information on writable storage media (such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media). The processes, methods, or algorithms may also be implemented in a software executable object. Alternatively, the processes, methods, or algorithms may be implemented in whole or in part using suitable hardware components (such as ASICs, FPGAs, state machines, controllers, or other hardware components or devices) or a combination of hardware, software, and firmware components.

[0093] Although the present disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only and not for purposes of limitation. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order, and / or if the components of the described systems, architectures, devices, or circuits are combined in a different manner and / or replaced or supplemented with other components or their equivalents. Therefore, the scope of the present disclosure is not limited by the specific embodiments, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as included in the present disclosure.

Claims

1. A human behavior recognition method, characterized in that: The human behavior recognition method comprises: Based on a human skeleton sequence, determining a joint difference topological representation and a joint dependency topological representation, and determining a graph topological feature based on the joint difference topological representation and the joint dependency topological representation, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and connection relationships between the joints represented by an adjacency matrix, and the graph topological feature is the sum of the general structure topological representation, the joint difference topological representation, and the joint dependency topological representation; determining a motion feature of the human skeleton sequence based on the human skeleton sequence and the graph topology feature; Reconstructing the graph topology features using fine-grained prototypes to obtain a refined action topology representation; Based on the motion features of the human skeleton sequence and the refined motion topology representation, the prototype contrast loss function and the cross entropy loss function are used to determine the motion category of the human skeleton sequence. The steps of determining the joint difference topological representation and the joint dependency topological representation based on the human skeleton sequence include: using a multi-head attention mechanism to determine the joint dependency topological representation based on the inner product of projection features, and determining the joint difference topological representation based on the feature differences between different joints. Furthermore, the step of reconstructing the graph topology features using fine-grained prototypes to obtain a refined action topology representation includes: performing feature index mapping on the graph topology features using a prototype index matrix and sampling-adjusted hyperparameters; and based on the mapping result, sampling and combining the detailed prototype library using a softmax function to determine the refined action topology representation.

2. The human behavior recognition method according to claim 1, characterized in that: The step of determining the motion features of the human skeleton sequence based on the human skeleton sequence and the graph topology features comprises: determining a first layer of action features based on the human skeleton sequence, a normalized adjacency matrix of the adjacency matrix, and a learnable weight matrix for feature projection; The action features of the current layer are determined based on the action features of the previous layer, the normalized adjacency matrix of the adjacency matrix, and the learnable weight matrix for feature projection.

3. The human behavior recognition method according to claim 2, characterized in that: The human behavior recognition method further comprises: After determining the action features of the last layer, a global average pooling operation is used to learn the global features of the human skeleton sequence; Determine the predicted action label through a classification network composed of a softmax function and a fully connected layer; The cross entropy loss function is determined based on the predicted action label and the true action label.

4. The human behavior recognition method according to claim 1, wherein: The human behavior recognition method further comprises: Use a mapping network to embed the refined action topology representation into the vector space of common features to obtain contrastive features; The prototype contrast loss function is determined based on the contrast feature, a temperature hyperparameter of contrast learning, and a positive sample prototype aggregation and a negative sample prototype aggregation related to the contrast feature.

5. A human behavior recognition device, characterized in that: The human behavior recognition device comprises: a motion detail enhancement module configured to determine a joint difference topology representation and a joint dependency topology representation based on a human skeleton sequence, and determine a graph topology feature based on the joint difference topology representation and the joint dependency topology representation, wherein the human skeleton sequence is represented as a graph structure, the graph structure includes a set of joints and connection relationships between joints represented by an adjacency matrix, and the graph topology feature is the sum of the general structure topology representation, the joint difference topology representation, and the joint dependency topology representation; an action feature extraction module configured to determine action features of a human skeleton sequence based on the human skeleton sequence and the graph topology features; A prototype reconstruction module is configured to reconstruct the graph topology features using fine-grained prototypes to obtain a refined action topology representation; The action feature classification module is configured to determine the action category of the human skeleton sequence using a prototype contrast loss function and a cross entropy loss function based on the action features of the human skeleton sequence and a refined action topology representation. The motion detail enhancement module is further configured to: utilize a multi-head attention mechanism to determine the joint dependency topology representation based on the inner product of the projection features, and to determine the joint difference topology representation based on the feature differences between different joints. Furthermore, the prototype reconstruction module is further configured to: perform feature index mapping on the graph topology features using the prototype index matrix and sampling adjustment hyperparameters; based on the mapping results, perform sampling and combination on the detail prototype library using the softmax function to determine the refined action topology representation.

6. A computing system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the human behavior recognition method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing instructions, characterized in that: When the instruction is executed by at least one computing device, the at least one computing device is prompted to execute the human behavior recognition method according to any one of claims 1 to 4.

Images