Apparatus and Method for Recognizing Behavior based on Skeleton Graph

Abstract

The present disclosure provides a skeleton graph-based behavior recognition device and method capable of accurately recognizing various human behaviors by receiving a skeleton sequence comprising a plurality of skeleton graphs composed of a plurality of nodes and a plurality of edges according to a skeleton topology, classifying a plurality of nodes in each of the plurality of skeleton graphs into a plurality of layers based on the skeleton topology, adding edges connecting nodes of adjacent layers among the nodes of the classified layers to generate a hierarchical skeleton sequence comprising a plurality of hierarchical skeleton graphs, and performing neural network operations on the hierarchical skeleton sequence to recognize behaviors for the skeleton sequence.

Description

Technology Field

[0001] The present disclosure relates to an action recognition device and method, and more specifically, to an action recognition device and method based on a hierarchically separated skeletal graph. Background Technology

[0002] Behavior recognition technology, which recognizes human behavior, is actively applied in various fields such as surveillance and security, as well as virtual reality and 3D content creation. Due to the advancement of artificial neural network technology, current behavior recognition technologies are mostly based on artificial neural networks.

[0003] Initially, artificial neural network-based behavior recognition technology primarily estimated behaviors based on the overall human form extracted from sources such as RGB images. However, estimating behavior based on the overall form presents a problem where the reliability of the estimated behavior is low due to various factors, such as body overlap, shape changes caused by clothing, and noise elements like the background. Consequently, recent behavior recognition technologies mainly utilize skeleton-based behavior recognition techniques that first estimate the human skeleton and recognize behaviors based on the estimated skeleton.

[0004] Skeleton-based behavior recognition technology recognizes human behavior by estimating the position of the skeleton, particularly the joints, and tracking changes in joint position. Among artificial neural networks, Graph Convolutional Networks (GCNs) are primarily used for skeleton-based behavior recognition. GCNs estimate behavior by performing neural network operations on a Skeleton Graph generated by setting the positions of major joints as nodes and the skeletal structures between joints as edges, based mainly on the human body structure.

[0005] However, in existing skeletal graph-based behavior recognition techniques, a skeletal sequence composed of a skeletal graph of multiple frames over time (t) is received to recognize changes in motion for behavior recognition, and behavior is estimated based on the relationship (e.g., position change, etc.) between each joint of the received skeletal sequence and adjacent joints in terms of time (Temporal) and space (Spatial). In other words, human behavior is estimated based on the relationship between a specific node and neighboring nodes connected by edges within a certain number of hops, and corresponding nodes in neighboring frames.

[0006] The aforementioned existing skeletal graph behavior recognition technique could accurately estimate human behavior in many cases. However, depending on the human behavior, the movement of joints that are separated from each other may be more important than the relationship between joints connected by the skeleton. For example, in the case of actions such as clapping, the relationship between the left wrist joint and the right wrist joint, which are not directly connected by edges, can be recognized more easily and accurately than the relationship between the elbow joint and the shoulder and wrist joints connected to the elbow joint.

[0007] However, as mentioned above, existing action recognition techniques using GCNs have a limitation in that they recognize actions based solely on the relationships between adjacent joints within each joint, which is derived from the human skeletal structure; consequently, the recognition performance for certain specific actions is significantly degraded. Furthermore, although edge features are aggregated during the neural network computation process, the contribution of edges is relatively ignored compared to nodes. This makes it difficult to identify key edges for action recognition, resulting in a limitation where optimal performance cannot be achieved. The problem to be solved

[0008] The purpose of the present disclosure is to provide a skeleton graph-based behavior recognition device and method capable of recognizing various human behaviors by hierarchically classifying nodes of a skeleton graph and generating a hierarchical skeleton graph in which nodes between the classified layers are connected by edges.

[0009] The purpose of the present disclosure is to provide a skeleton graph-based behavior recognition device and method capable of accurately recognizing behavior by using a hierarchical skeleton graph to extract hierarchical features for each layer and recognizing the layer to be emphasized based on the extracted hierarchical features. means of solving the problem

[0010] According to one embodiment of the present disclosure, a skeleton graph-based behavior recognition device comprises: a memory; and a processor that executes at least a portion of a neural network model operation according to a program stored in the memory, wherein the processor receives a skeleton sequence comprising a plurality of skeleton graphs composed of a plurality of nodes and a plurality of edges according to a skeleton topology, classifies the plurality of nodes in each of the plurality of skeleton graphs into a plurality of layers based on the skeleton topology, generates a hierarchical skeleton sequence comprising a plurality of hierarchical skeleton graphs by adding edges connecting nodes of adjacent layers among the nodes of the classified layers, and performs neural network operations on the hierarchical skeleton sequence to recognize a behavior for the skeleton sequence.

[0011] The processor can select a first node among the multiple nodes of each of the multiple skeleton graphs to classify it as a first layer, and sequentially classify nodes directly connected by edges to the nodes of the already classified layer according to the skeleton topology as nodes of the next layer.

[0012] The processor can select a node adjacent to the center of mass of the human body among a plurality of nodes of the skeletal graph according to the skeletal topology as the first node.

[0013] The above processor can obtain the hierarchical skeleton graph by generating additional edges that connect all nodes included in adjacent layers among the nodes included in each of the separated multiple layers.

[0014] The processor can receive the hierarchical skeletal sequence, estimate the importance of each layer using neural network operations to extract action features, and classify the extracted action features using neural network operations to determine the action.

[0015] The processor can perform neural network operations on the authorized hierarchical skeleton sequence to extract graph features for a plurality of hierarchical skeleton graphs, estimate the importance of each layer from the graph features through neural network operations to obtain a layer-emphasized graph feature with emphasized importance of each layer, and perform neural network operations on a plurality of layer-emphasized graphs in the time axis direction to obtain the operation feature.

[0016] The processor can receive the graph features, pool features for edges connecting adjacent layers, and estimate the importance of nodes separated by layer using neural network operations on the graph features pooled for edges, thereby obtaining the layer-weighted graph features.

[0017] The above processor can perform pooling by dividing the features of all edges connecting two adjacent layers among a plurality of layers by the sum of the number of nodes included in the two adjacent layers.

[0018] The processor can obtain the layer-weighted graph features by performing neural network operations on node features separated by layer in the edge-pooled graph features as individual feature groups.

[0019] The above processor can obtain graph features for a plurality of hierarchical skeleton graphs in the above hierarchical skeleton sequence using a graph convolution layer.

[0020] A skeleton graph-based action recognition method according to another embodiment of the present disclosure is a method for executing at least a portion of a neural network model operation performed by a processor, comprising: receiving a skeleton sequence including a plurality of skeleton graphs each composed of a plurality of nodes and a plurality of edges according to a skeleton topology, classifying the plurality of nodes in each of the plurality of skeleton graphs into a plurality of layers based on the skeleton topology, and adding edges connecting nodes of adjacent layers among the nodes of the classified layers to generate a hierarchical skeleton sequence including a plurality of hierarchical skeleton graphs; and performing a neural network operation on the hierarchical skeleton sequence to recognize an action for the skeleton sequence. Effects of the invention

[0021] The skeleton graph-based behavior recognition device and method of the present disclosure can recognize various human behaviors by hierarchically classifying the nodes of a skeleton graph and generating a hierarchical skeleton graph in which nodes between the classified layers are connected by edges. Furthermore, by using the generated hierarchical skeleton graph to extract hierarchical features for each layer and identifying the layer to be emphasized based on the extracted hierarchical features, behaviors can be accurately recognized. Brief explanation of the drawing

[0022] FIG. 1 is a schematic diagram showing the configuration of a skeletal graph-based behavior recognition device according to one embodiment, divided according to the operation. Figure 2 shows an example of the detailed configuration of the hierarchical decomposition module of Figure 1. Figure 3 is a diagram illustrating how the hierarchy decomposition module of Figure 2 hierarchically separates the nodes of a skeleton graph. Figure 4 is a diagram illustrating how the inter-layer edge connection module of Figure 2 adds an edge connecting the layered nodes. Figure 5 shows an example of the detailed configuration of the behavior estimation module of Figure 1. Figure 6 shows an example of the detailed configuration of the edge-highlighted feature extraction module of Figure 5. Figure 7 is a diagram illustrating the operation of the pooling module of Figure 6. FIG. 8 illustrates a skeletal graph-based behavior recognition method according to one embodiment. FIG. 9 is a drawing for explaining a computing environment including a computing device according to one embodiment. Specific details for implementing the invention

[0023] Hereinafter, specific embodiments according to the embodiments of the present disclosure will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0024] In describing the embodiments of the present disclosure, detailed descriptions of known technology related to the present invention are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the embodiments. Furthermore, terms described below are defined with consideration of their functions in the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe specific embodiments and should not be limiting. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as “include” or “compose” are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described. Additionally, terms such as “...part,” “...unit,” “module,” and “block” described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0025] FIG. 1 is a schematic diagram showing the configuration of a skeletal graph-based behavior recognition device according to one embodiment, divided according to the operation.

[0026] Referring to FIG. 1, a skeleton graph-based behavior recognition device according to the present disclosure may include a skeleton sequence acquisition module (10), a hierarchy decomposition module (20), and a behavior estimation module (30).

[0027] The skeletal sequence acquisition module (10) acquires a skeletal sequence for an object that is the target of action recognition. As described above, the skeletal sequence may be composed of a skeletal graph of multiple consecutive frames.

[0028] The skeletal sequence acquisition module (10) may be implemented as a memory module that stores already acquired skeletal sequences, or as a communication module that receives skeletal sequences stored in another device, but is not limited thereto. Additionally, the skeletal sequence acquisition module (10) may be implemented as an artificial neural network that acquires a number of consecutive frames of images and performs neural network operations on each frame of the acquired images to extract a skeletal graph. Various pose estimation techniques that extract human skeletal graphs from RGB-based images using artificial neural networks have already been studied and proposed. Accordingly, the skeletal sequence acquisition module (10) may be implemented as an artificial neural network capable of extracting skeletal sequences from images.

[0029] The hierarchical decomposition module (20) hierarchically separates multiple nodes in each of the multiple skeletal graphs of the skeletal sequence obtained from the skeletal sequence acquisition module (10), and generates a hierarchical skeletal graph by adding edges connecting the nodes between the separated layers.

[0030] Each of the multiple skeletal graphs in a skeletal sequence is constructed based on a skeletal topology in which multiple joints are set as nodes according to the human body structure, and the bones between the joints are set as edges. This skeletal topology ensures that the graphs are generated based on the actual structure of the human body, allowing the relationships between joints connected to each bone to be interpreted organically depending on the posture. Consequently, it can exclude unnatural postures or movements that are difficult for humans to assume, thereby demonstrating superior behavior recognition performance. However, in skeletal graphs based on this topology, nodes for joints that are not connected by bones due to the human body structure are not connected to each other via edges. As a result, recognition performance is inevitably degraded for behaviors where the relationships between non-skeletal joints are important, such as clapping.

[0031] To overcome these limitations, the hierarchical decomposition module (20) in the present disclosure constructs additional edges for each of the multiple skeletal graphs. However, if edges that interconnect all nodes of the skeletal graph are added or added arbitrarily, edges are generated regardless of the human body structure, thereby diminishing the significance of generating skeletal graphs based on the skeletal topology. In other words, behaviors regarding unusual or unnatural postures, as well as natural postures resulting from actual human activity, are added as recognition targets, which may actually degrade behavior recognition performance. Furthermore, as the number of edges increases significantly, a decrease in computational performance occurs.

[0032] In order to prevent such problems from occurring and to improve human behavior recognition performance, the hierarchical decomposition module (20) in the present disclosure first hierarchically divides a number of nodes of a skeleton graph into a number of layers sequentially from a central node according to a skeleton connection structure according to the skeleton topology. Then, by ensuring that the nodes of each divided layer are connected by edges only to the nodes of adjacent layers, a number of additional edges are generated that take into account the characteristics of the human body structure according to the skeleton topology, thereby obtaining a number of hierarchical skeleton graphs. That is, a hierarchical skeleton sequence composed of a number of hierarchical skeleton graphs is obtained.

[0033] The detailed operation of the layer decomposition module (20) will be described later.

[0034] The behavior estimation module (30) receives a hierarchical skeletal sequence generated by the hierarchy decomposition module (20) and performs neural network computation to estimate behavior. The behavior estimation module (30) may be implemented based on the neural network of an existing behavior recognition device that receives an existing skeletal sequence and determines behavior. That is, the neural network of the existing behavior recognition device may be configured to receive a hierarchical skeletal sequence, and the neural network may be newly trained or additionally trained based on the hierarchical skeletal sequence to perform behavior recognition. However, in this case, it is difficult to utilize the characteristics of multiple hierarchical skeletal sequences in which multiple nodes are hierarchically separated. Accordingly, the behavior estimation module (30) of the present disclosure recognizes behavior by considering the importance of the edges connecting nodes between each separated layer so that the characteristics of the hierarchical skeletal sequence can be utilized. At this time, human behavior that could not be recognized previously can be accurately recognized not only by adjacent nodes connected by edges but also by added edges.

[0035] Figure 2 shows an example of the detailed configuration of the hierarchy decomposition module of Figure 1, Figure 3 is a diagram explaining how the hierarchy decomposition module of Figure 2 hierarchically separates nodes of a skeletal graph, and Figure 4 is a diagram explaining how the inter-hierarchy edge connection module of Figure 2 adds edges connecting the hierarchically separated nodes.

[0036] Referring to FIG. 2, the layer decomposition module (20) may include a node layer separation module (21) and an inter-layer edge connection module (22).

[0037] The node hierarchy classification module (21) classifies multiple nodes into multiple layers in each of the multiple skeletal graphs of the skeletal sequence, and the layer edge connection module (22) additionally generates edges connecting the nodes classified into multiple layers by the node hierarchy classification module (21). At this time, the node hierarchy classification module (21) classifies the nodes into multiple layers according to the edges connecting the nodes in the skeletal graph, and the layer edge connection module (22) adds edges connecting the nodes of each layer to the nodes of the neighboring layer.

[0038] Referring to FIG. 3, the node hierarchy classification module (21) first selects a center node among multiple nodes of the skeletal graph. At this time, the center node may be selected as the first node (1) corresponding to the Center of Mass (CoM) of the human body. The selected center node may be classified into the first hierarchy (H1). That is, the node set (Set1) of the first hierarchy (H1) includes only one center node (1).

[0039] Then, neighbor nodes (here, nodes 2 and 3) that are directly connected to the central node, which is a node of the first layer (H1) in the skeleton graph, are classified as the second layer (H2). Thus, the set of nodes (Set2) of the second layer (H2) includes the second node (2) and the third node (3).

[0040] Additionally, the node hierarchy classification module (21) classifies neighboring nodes (here, nodes 4 to 8) that are directly edge-connected to the nodes (2, 3) of the classified second hierarchy (H2) into the third hierarchy (H3). Accordingly, the node set (Set3) of the third hierarchy (H3) includes the fourth node (4) to the eighth node (8).

[0041] Thereafter, the node hierarchy classification module (21) repeats the process of classifying nodes connected by edges from each of the nodes of the previously classified hierarchy into nodes of the next hierarchy. In FIG. 3, nodes were classified up to the 5th hierarchy (H5) for convenience, but the node hierarchy classification module (21) repeats the hierarchy classification until all nodes of the skeleton graph are classified.

[0042] As can be seen in FIG. 3, the node hierarchy classification module (21) sequentially classifies nodes connected by edges from the human body center node based on the skeletal topology. Thus, nodes included in the classified multiple layers are connected by edges only to nodes in adjacent layers, and are not connected by edges to nodes in the same layer or to nodes in separated layers.

[0043] As described above, the layer edge connection module (22) creates additional edges connecting nodes included in each of the separated multiple layers, such that each node of each layer is connected to all nodes included in the neighboring layer.

[0044] Referring to FIG. 4, the layered edge connection module (22) can first generate additional edges connecting the nodes of the first layer (H1) and the second layer (H2). However, the first node (1) included in the first node set (Set1) of the first layer (H1) and the second and third nodes (2, 3) included in the second node set (Set2) of the second layer (H2) are already connected as nodes in the skeletal graph. Therefore, the layered edge connection module (22) does not generate additional edges connecting the nodes of the first layer (H1) and the second layer (H2).

[0045] However, in the skeleton graph, among the five nodes (4 to 8) of the third layer (H3), the fourth to sixth nodes (4 to 6) are connected by edges only to the second node (2) among the second and third nodes (2, 3) of the second layer (H2), and the seventh and eighth nodes (7, 8) are connected by edges only to the third node (3). Accordingly, the layer edge connection module (22) can additionally generate three edges connecting each of the fourth to sixth nodes (4 to 6) to the third node (3), while additionally generating two edges connecting the seventh and eighth nodes (7, 8) to the second node (2).

[0046] Additionally, the layer-to-layer edge connection module (22) creates additional edges so that each of the five nodes of the fourth layer (H4) is again connected to each of the five nodes (4 to 8) of the third layer (H3) by an edge.

[0047] That is, in the present disclosure, the layer-to-layer edge connection module (22) adds only edges connecting nodes of neighboring layers. This follows the rule that nodes of a layer distinguished by the node layer distinction module (21) are connected by edges only to nodes of adjacent layers, and are not connected by edges to nodes of the same layer or separated layers. In other words, by adding edges based on the characteristics of the basic skeletal topology, it prevents the addition of edges that are difficult to aid in human behavior recognition, thereby preventing behavior recognition from being hindered. Furthermore, it suppresses the increase in computation required due to the increase in edges.

[0048] The layer edge connection module (22) can add additional edges by checking whether additional edges are created for all nodes of the separated layers.

[0049] FIG. 5 shows an example of a detailed configuration of the behavior estimation module of FIG. 1, FIG. 6 shows an example of a detailed configuration of the edge enhancement module of FIG. 5, and FIG. 7 is a diagram for explaining the operation of the pooling module of FIG. 6.

[0050] Referring to FIG. 5, the behavior estimation module (30) may include a hierarchical graph feature extraction module (31) and a behavior class determination module (32).

[0051] The hierarchical graph feature extraction module (31) receives a hierarchical skeleton sequence from the hierarchical decomposition module (20) and performs neural network operations on the received hierarchical skeleton sequence to estimate changes in temporal characteristics that emphasize the importance of edges according to the layers separated in the multiple hierarchical skeleton graphs of the hierarchical skeleton sequence, thereby extracting operation features.

[0052] The hierarchical graph feature extraction module (31) may include at least one edge-highlighted feature extraction module (41 to 4n) connected in sequence. Each of the at least one edge-highlighted feature extraction module (41 to 4n) may include a graph operation module (51), an Attention-Guided Hierarchy Aggregation Module (hereinafter A-HA) module (52), and a temporal feature extraction module (55), as illustrated in FIG. 6.

[0053] The graph operation module (51) receives the output of the hierarchical skeleton sequence or the previous hierarchical graph feature extraction module (31) and performs neural network operations to extract graph features. The graph operation module (51) can be implemented, for example, as a graph convolution layer, which is a type of operation layer of a neural network.

[0054] The A-HA module (51) extracts hierarchical emphasis graph features by estimating the importance of each hierarchy from the graph features output from the graph operation module (51). The A-HA module (51) may include a pooling module (53) and a hierarchy emphasis module (54).

[0055] The pooling module (53) performs pooling to prevent the features from being biased toward some edges in the graph features. In the case of conventional graph neural networks, particularly graph neural networks for skeletal graphs, average pooling is performed to equalize the importance of all edges in the graph features. As illustrated in FIG. 7, however, in the present disclosure as illustrated in FIG. 3, multiple nodes are classified according to layers, so the number of nodes included in each layer differs based only on the edges based on the skeletal topology. Consequently, the number of edges already differs by layer. Therefore, if average pooling is performed as shown in the right-hand equation of FIG. 7, features for the 2nd to 4th layers (H2~H4), where there are relatively many edges in the graph features, have a significant impact on action recognition, whereas features for the 1st and 5th layers (H1, H5) have a relatively reduced impact on action recognition. This can be a factor in misrecognition of actions. In FIG. 7, N represents the total number of nodes.

[0056] In particular, while FIG. 7 illustrates only the edges according to the skeletal topology for ease of understanding, the present disclosure additionally generates edges connecting all nodes of adjacent layers, as shown in FIG. 4. Therefore, depending on the number of nodes in each layer, a very large difference occurs in the number of edges connecting two adjacent layers. Consequently, because the difference in the number of edges between layers is very large, if average pooling is performed on all edges of the graph features to assign equal importance, it results in a bias where attention is concentrated on the edges connecting two adjacent layers with a large number of nodes. To prevent this problem, the pooling module (53) of the present disclosure performs Representative Spatial Average Pooling (RSAP). As shown in FIG. 7, in RSAP, adjacent layers (H k , Hk+1 The number of nodes of ) (here, N k , N k+1 By performing average pooling by dividing the importance of each edge feature by ), the importance of features for all edges in each layer of the graph features is made equal. Therefore, attention is not biased toward a specific layer.

[0057] The layer enhancement module (54) performs neural network operations on pooled graph features through RSAP so that attention to edges is evenly distributed by layer, thereby enhancing the features of edges between specific layers that require attention. The layer enhancement module (54) can be implemented as an edge convolution layer. In particular, in the present disclosure, the layer enhancement module (54) can be implemented to distinguish and enhance features according to layer, and here, a computation layer implemented to distinguish and enhance features according to layer in this way is referred to as a layered edge convolution layer. The layer enhancement module (54) treats layer-specific features in the pooled graph features as independent nodes, recognizes layers that require attention for action recognition through semantically close edge sets in the embedding space, and obtains layer-enhanced graph features by enhancing the features of the recognized layers.

[0058] The temporal feature extraction module (55) receives layered emphasis graph features from the A-HA module (52) and performs neural network operations in the direction of the time axis to obtain operation features according to the flow of time.

[0059] At least one edge-highlighting feature extraction module (41 to 4n) is provided in multiple numbers and connected sequentially to extract deeper operational features.

[0060] The behavior class determination module (32) receives the final action features obtained from the hierarchical graph feature extraction module (31) and determines the action according to the hierarchical skeletal sequence. The behavior class determination module (32) can recognize the action by obtaining an action score from the action features and identifying the corresponding action class based on the obtained action score. The behavior class determination module (32) can be implemented as, for example, a Fully Connected (FC) layer.

[0061] In the case of existing behavior recognition devices, features are extracted for individual nodes through relationships with temporally and spatially adjacent nodes without distinguishing hierarchies, so actions are recognized only by limited body parts. Additionally, since average pooling is used, specific areas are not emphasized, making it difficult to recognize actions corresponding to movements. In contrast, the present disclosure allows the hierarchical graph feature extraction module (31) to consider not only nodes connected by edges but also relationships with other body parts that may have mutual correlations through hierarchical distinction and additional edges, and can recognize actions by focusing on each body part that requires attention through hierarchical emphasis, thereby improving the accuracy of behavior recognition.

[0062] In the illustrated embodiments, each component may have different functions and capabilities other than those described below, and may include additional components other than those not described below. Additionally, in one embodiment, each component may be implemented using one or more physically separated devices, or by one or more processors or a combination of one or more processors and software, and may not be clearly distinguished in specific operation as in the illustrated examples.

[0063] And the skeletal graph-based behavior recognition device illustrated in FIG. 1 may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or specific-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Additionally, the device may be implemented as a system-on-chip (SoC) including one or more processors and controllers.

[0064] In addition, the skeletal graph-based behavior recognition device may be installed in the form of software, hardware, or a combination thereof on a computing device or server equipped with hardware elements. A computing device or server may refer to various devices that include, in whole or in part, communication devices such as communication modems for communicating with various devices or wired / wireless communication networks, memory for storing data for executing programs, and microprocessors for executing programs to perform calculations and commands.

[0065] FIG. 8 illustrates a skeletal graph-based behavior recognition method according to one embodiment.

[0066] Referring to FIG. 8, the skeletal graph-based action recognition method according to the present disclosure first obtains a skeletal sequence (71). At this time, the skeletal sequence may be obtained by direct input, but in some cases, it may be obtained by performing neural network operations on an image using an artificial neural network. Once the skeletal sequence is obtained, each node of a plurality of skeletal graphs included in the obtained skeletal sequence is hierarchically classified (72). Each skeletal graph consists of a plurality of nodes corresponding to joints and a plurality of edges corresponding to bones connecting the joints, according to a skeletal topology based on the human body structure. Accordingly, among the plurality of nodes, a central node adjacent to the center of mass of the human body is selected as the first node and classified as the first layer (H1), and nodes directly connected to the nodes and edges of the already classified layer are sequentially classified into the next layer (H2, H=, …).

[0067] When a hierarchy is formed for all nodes of the skeleton graph, an edge between the hierarchy is added to obtain a hierarchical skeleton sequence (73). At this time, the additional edge can be created so that all nodes included in adjacent layers among the separated layers are connected to each other, and, as in the skeleton topology, no additional edge is created for nodes between separated layers.

[0068] When a hierarchical skeleton sequence is generated, a neural network operation is performed on the hierarchical skeleton sequence to extract graph features (74). At this time, graph features can be extracted using a graph convolution layer, etc. Once graph features are extracted, pooling is performed on the extracted graph features so that the importance of edges between layers becomes equal for each layer, thereby preventing bias caused by differences in the number of edges between layers (75). At this time, pooling is performed according to the RSAP technique described above, and the features of all edges connecting two adjacent layers are equal to the sum of the number of nodes (N) included in the two adjacent layers. k , N k+1 It can be performed by dividing it into ).

[0069] Then, a neural network operation is performed on graph features in which edge importance is uniformly adjusted according to the layer by pooling to obtain layer-weighted graph features (76). Here, layer-weighted graph features represent graph features that emphasize joints that require attention in order to accurately perceive human posture. Layer-weighted graph features are graph features in which the layer requiring attention among multiple layers is emphasized, and they can be obtained by performing a neural network operation on individual feature groups of node features separated by layer in a graph feature in which edge features are pooled. That is, layer-weighted graph features can be obtained by treating multiple node feature groups separated by layer as individual nodes and performing a neural network operation.

[0070] When a layered emphasis graph feature is obtained, a neural network operation is performed on the obtained layered emphasis graph feature in the time axis direction to obtain a motion feature by extracting a pose feature, that is, a feature of the change in motion over time (77).

[0071] When motion features are extracted, it is determined whether the number of times motion features have been extracted is greater than or equal to a set reference number of repetitions (78). Similar to most techniques using artificial neural networks, the action recognition method of the present disclosure does not perform the process of extracting motion features only once, but rather includes multiple neural network operation blocks, each containing at least one neural network operation layer, to enable accurate recognition of actions by repeatedly extracting deeper features. Accordingly, in the present disclosure, when the number of motion feature extractions is repeated equal to the number of neural network operation blocks, the motion features extracted from the last neural network operation block are finally obtained. If the number of extractions is less than the reference number of repetitions, the graph feature extraction step (74) is repeated again for the extracted motion features.

[0072] When the number of extractions reaches the reference number of repetitions and the motion features are finally extracted, the finally extracted motion features are classified into classes by neural network operation to determine human behavior according to the input skeletal sequence (79).

[0073] Although FIG. 8 describes each process as being executed sequentially, this is merely an illustrative example, and a person skilled in the art can modify and adapt it in various ways without departing from the essential characteristics of the embodiment of the present invention, such as changing the order described in FIG. 8, executing one or more processes in parallel, or adding other processes.

[0074] FIG. 9 is a drawing for explaining a computing environment including a computing device according to one embodiment.

[0075] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those not described below. The illustrated computing environment (90) may include a computing device (91) to perform the skeletal graph-based behavior recognition method illustrated in FIG. 8. In one embodiment, the computing device (91) may be one or more components included in the skeletal graph-based behavior recognition device illustrated in FIG. 1.

[0076] A computing device (91) includes at least one processor (92), a computer-readable storage medium (93), and a communication bus (95). The processor (92) may enable the computing device (91) to operate according to the exemplary embodiment described above. For example, the processor (92) may execute one or more programs (94) stored in the computer-readable storage medium (93). The one or more programs (94) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to enable the computing device (91) to perform operations according to the exemplary embodiment when executed by the processor (92).

[0077] The communication bus (95) interconnects various other components of the computing device (91), including the processor (92) and the computer-readable storage medium (93).

[0078] The computing device (91) may also include one or more input / output interfaces (96) and one or more communication interfaces (97) that provide an interface for one or more input / output devices (98). The input / output interfaces (96) and communication interfaces (97) are connected to a communication bus (95). The input / output devices (98) may be connected to other components of the computing device (91) through the input / output interfaces (96). An exemplary input / output device (98) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (98) may be included inside the computing device (91) as a component constituting the computing device (91), or it may be connected to the computing device (91) as a separate device distinct from the computing device (91).

[0079] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

Claim 1 A skeleton graph-based behavior recognition device comprising: a memory; and a processor that executes at least a portion of a neural network model operation according to a program stored in the memory, wherein the processor receives a skeleton sequence comprising a plurality of skeleton graphs each composed of a plurality of nodes and a plurality of edges according to a skeleton topology, classifies the plurality of nodes in each of the plurality of skeleton graphs into a plurality of layers based on the skeleton topology, generates a hierarchical skeleton sequence comprising a plurality of hierarchical skeleton graphs by adding edges connecting nodes of adjacent layers among the nodes of the classified layers, performs neural network operations on the hierarchical skeleton sequence to recognize a behavior for the skeleton sequence, and the processor selects a first node among the plurality of nodes of each of the plurality of skeleton graphs to classify it into a first layer, and sequentially classifies nodes directly connected by edges to the nodes of the layer already classified according to the skeleton topology into nodes of the next layer. Claim 2 delete Claim 3 A skeleton graph-based behavior recognition device according to claim 1, wherein the processor selects a node adjacent to the center of mass of the human body among a plurality of nodes of the skeleton graph according to the skeleton topology as the first node. Claim 4 A skeleton graph-based behavior recognition device according to claim 1, wherein the processor generates additional edges connecting all nodes included in adjacent layers among the nodes included in each of the separated multiple layers to obtain the hierarchical skeleton graph. Claim 5 A skeleton graph-based behavior recognition device according to claim 1, wherein the processor receives the hierarchical skeleton sequence, estimates the importance of each layer by neural network operation to extract behavior features, and classifies the extracted behavior features by neural network operation. Claim 6 A skeleton graph-based behavior recognition device according to claim 5, wherein the processor performs neural network operations on the authorized hierarchical skeleton sequence to extract graph features for a plurality of hierarchical skeleton graphs, estimates the importance of each layer by performing neural network operations on the graph features to obtain a layer-emphasized graph feature with emphasized importance by layer, and performs neural network operations on the plurality of layer-emphasized graphs in the time axis direction to obtain the action feature. Claim 7 A skeleton graph-based behavior recognition device according to claim 6, wherein the processor receives the graph features, pools features for edges connecting adjacent layers, and estimates the importance of nodes separated by layer using neural network operations on the graph features pooled for edges, thereby obtaining the layer-weighted graph features. Claim 8 In claim 7, the processor is a skeleton graph-based behavior recognition device that performs pooling by dividing the features of all edges connecting two adjacent layers among a plurality of layers by the sum of the number of nodes included in the two adjacent layers. Claim 9 In claim 7, the processor is a skeleton graph-based behavior recognition device that obtains the layer-emphasized graph features by performing neural network operations on node features separated by layer in an edge-pooled graph feature as individual feature groups. Claim 10 In claim 6, the processor is a skeleton graph-based behavior recognition device that acquires graph features for a plurality of hierarchical skeleton graphs in the hierarchical skeleton sequence using a graph convolution layer. Claim 11 A method for executing at least part of a neural network model operation performed by a processor, comprising: receiving a skeletal sequence including a plurality of skeletal graphs each composed of a plurality of nodes and a plurality of edges according to a skeletal topology, classifying the plurality of nodes in each of the plurality of skeletal graphs into a plurality of layers based on the skeletal topology, and adding edges connecting nodes of adjacent layers among the nodes of the classified layers to generate a hierarchical skeletal sequence including a plurality of hierarchical skeletal graphs; and performing a neural network operation on the hierarchical skeletal sequence to recognize an action for the skeletal sequence, wherein the step of generating the hierarchical skeletal sequence comprises selecting a first node among the plurality of nodes of each of the plurality of skeletal graphs to classify it into a first layer, and sequentially classifying nodes directly connected by edges to the nodes of the layer already classified according to the skeletal topology into nodes of the next layer. Claim 12 delete Claim 13 In claim 11, the step of generating the hierarchical skeletal sequence is a skeletal graph-based behavior recognition method in which a node adjacent to the center of mass of the human body among a plurality of nodes of the skeletal graph according to the skeletal topology is selected as the first node. Claim 14 In claim 11, the step of generating the above-mentioned hierarchical skeleton sequence is to generate additional edges connecting all nodes included in adjacent layers among the nodes included in each of the separated multiple layers, thereby obtaining the above-mentioned hierarchical skeleton graph, a skeleton graph-based behavior recognition method. Claim 15 In claim 11, the step of recognizing the behavior is a skeleton graph-based behavior recognition method that receives the hierarchical skeleton sequence, estimates the importance of each layer by neural network operation to extract behavior features, and classifies the extracted behavior features by neural network operation. Claim 16 In claim 15, the step of recognizing the above action is a skeleton graph-based action recognition method comprising: performing neural network operations on the authorized hierarchical skeleton sequence to extract graph features for a plurality of hierarchical skeleton graphs; estimating the importance of each hierarchy from the graph features through neural network operations to obtain a hierarchy-emphasized graph feature with the importance of each hierarchy emphasized; and performing neural network operations on the plurality of hierarchy-emphasized graphs in the time axis direction to obtain the action feature. Claim 17 In claim 16, the step of recognizing the behavior involves receiving the graph features, pooling features for edges connecting adjacent layers, and estimating the importance of nodes separated by layer using neural network operations on the graph features pooled for the edges, thereby obtaining the layer-weighted graph features, a skeleton graph-based behavior recognition method. Claim 18 In claim 17, the step of recognizing the behavior is a skeletal graph-based behavior recognition method that performs pooling by dividing the features of all edges connecting two adjacent layers among a plurality of layers by the sum of the number of nodes included in the two adjacent layers. Claim 19 In claim 17, the step of recognizing the above behavior is a skeletal graph-based behavior recognition method that obtains the layer-weighted graph features by performing neural network operations on node features separated by layer in a graph feature pooled with edge features as individual feature groups. Claim 20 In claim 16, the step of recognizing the behavior is a skeleton graph-based behavior recognition method that uses a graph convolution layer to acquire graph features for a plurality of hierarchical skeleton graphs in the hierarchical skeleton sequence.

Images